web dyno getting OOMKilled

In the last 24 hours, we've had over 20 instances of the dandidav web dyno exceeding its memory limit of 512MB and getting killed by Heroku as a result. See https://dashboard.heroku.com/apps/dandidav/metrics/web?ending=0-hours-ago&starting=24-hours-ago
@yarikoptic @jwodder is 512MB enough for the dandidav server, or should we provision a higher tier dyno?

My best guess is that the high memory usage probably comes from the caching of Zarr manifest files.  However, no more than 16 files are cached at a time, and most manifest files seem to be around 10-20 MB, which doesn't reach 512 MB.

10-20 MB, which doesn't add up to 512 MB.

FWIW some manifest files are much larger than that (e.g. 62MB of ./5c3/7c2/5c37c233-222f-4e60-96e7-a7536e08ef61/4cb549b2e2346bb1a30f493b50fb6a2e-526320--1023396474554.json) but they are in serialized form, so indeed in memory should not take as much.
so should be investigated on how much memory is actually consumed by the process... may be multiple caches for the same thing across threads etc?

FWIW, remains the case. @jwodder - in case might be of use, we do dump logs on drogon which you can inspect:
dandi@drogon:/mnt/backup/dandi/heroku-logs/dandidav$ tail 20240326-0901.log
2024-03-26T13:17:54.887354+00:00 heroku[web.1]: Process running mem=635M(124.1%)
2024-03-26T13:17:54.889073+00:00 heroku[web.1]: Error R14 (Memory quota exceeded)
2024-03-26T13:18:15.933714+00:00 heroku[web.1]: Process running mem=635M(124.1%)
2024-03-26T13:18:15.935432+00:00 heroku[web.1]: Error R14 (Memory quota exceeded)
2024-03-26T13:18:36.885785+00:00 heroku[web.1]: Process running mem=635M(124.1%)
2024-03-26T13:18:36.888306+00:00 heroku[web.1]: Error R14 (Memory quota exceeded)
2024-03-26T13:18:57.096775+00:00 heroku[web.1]: Process running mem=635M(124.1%)
2024-03-26T13:18:57.098827+00:00 heroku[web.1]: Error R14 (Memory quota exceeded)
2024-03-26T13:19:19.324735+00:00 heroku[web.1]: Process running mem=635M(124.1%)
2024-03-26T13:19:19.326660+00:00 heroku[web.1]: Error R14 (Memory quota exceeded)

@yarikoptic

Could you @jwodder add to log line amount of memory that process uses to be provided with each log line

I don't think that's possible.  The closest thing would be to emit dedicated log lines listing the current memory used, either at the start and/or end of each request or periodically.

may be multiple caches for the same thing across threads etc?

That shouldn't be the case.  There should only be one manifest cache in the whole process.

FWIW an idea -- you can "replay" sequence of visits using those logs on a local dandidav instance and monitor memory consumption any way you like

dandidav now logs the current memory usage before & after handling each request as an INFO message of the form "Current memory usage: {int} physical, {int} virtual".

great -- did you pin point what causes that consistent memory consumption growth?
dandi@drogon:/mnt/backup/dandi/heroku-logs/dandidav$ sed -ne '/usage/s,.*usage:,,gp' 20240328-1401.log | uniq -c
      5  201826304 physical, 714584064 virtual
      1  201826304 physical, 714571776 virtual
      1  244264960 physical, 781692928 virtual
      1  244264960 physical, 781680640 virtual
      1  244264960 physical, 781692928 virtual
      1  244264960 physical, 781680640 virtual
      2  244264960 physical, 781692928 virtual
      1  246632448 physical, 781692928 virtual
      1  246632448 physical, 781680640 virtual
      1  246632448 physical, 781692928 virtual
      1  246632448 physical, 781680640 virtual
      1  306573312 physical, 869711872 virtual
      3  306573312 physical, 869699584 virtual
      1  306573312 physical, 869711872 virtual
      9  306573312 physical, 869699584 virtual
      1  308158464 physical, 869711872 virtual
      1  308158464 physical, 869699584 virtual
      1  308158464 physical, 869711872 virtual
      1  308158464 physical, 869699584 virtual
      1  308174848 physical, 869711872 virtual
      3  308174848 physical, 869699584 virtual
      9  308174848 physical, 869711872 virtual
      5  308174848 physical, 869699584 virtual
      1  325824512 physical, 869711872 virtual
      1  325824512 physical, 869699584 virtual
      1  325857280 physical, 869711872 virtual

looking at sample:
dandi@drogon:/mnt/backup/dandi/heroku-logs/dandidav$ grep -B 3 ' 325824512 physical' 20240328-1401.log
2024-03-28T18:11:38.319749+00:00 app[web.1]: 2024-03-28T18:11:38.319732347Z DEBUG request{method=GET uri=/zarrs/9f0/a96/9f0a9681-f4e9-4103-a205-7faecb311246/a77cc8f6ced80f8fb0c712b8b207e14b-55031--94971788668.zarr/4/ version=HTTP/1.1}:outgoing-request{url=https://datasets.datalad.org/dandi/zarr-manifests/zarr-manifests-v2-sorted/9f0/a96/9f0a9681-f4e9-4103-a205-7faecb311246/a77cc8f6ced80f8fb0c712b8b207e14b-55031--94971788668.json method=GET}: reqwest::connect: starting new connection: https://datasets.datalad.org/
2024-03-28T18:11:38.408735+00:00 app[web.1]: 2024-03-28T18:11:38.408630383Z DEBUG request{method=GET uri=/zarrs/9f0/a96/9f0a9681-f4e9-4103-a205-7faecb311246/a77cc8f6ced80f8fb0c712b8b207e14b-55031--94971788668.zarr/4/ version=HTTP/1.1}:outgoing-request{url=https://datasets.datalad.org/dandi/zarr-manifests/zarr-manifests-v2-sorted/9f0/a96/9f0a9681-f4e9-4103-a205-7faecb311246/a77cc8f6ced80f8fb0c712b8b207e14b-55031--94971788668.json method=GET}: dandidav::httputil: Response received status=200 OK
2024-03-28T18:11:38.672798+00:00 heroku[router]: at=info method=GET path="/zarrs/9f0/a96/9f0a9681-f4e9-4103-a205-7faecb311246/a77cc8f6ced80f8fb0c712b8b207e14b-55031--94971788668.zarr/4/" host=webdav.dandiarchive.org request_id=1a482587-988a-4235-b677-13c4b77419f5 fwd="<IP_ADDRESS>" dyno=web.1 connect=4ms service=358ms status=200 bytes=3562 protocol=https
2024-03-28T18:11:38.671895+00:00 app[web.1]: 2024-03-28T18:11:38.671796029Z  INFO request{method=GET uri=/zarrs/9f0/a96/9f0a9681-f4e9-4103-a205-7faecb311246/a77cc8f6ced80f8fb0c712b8b207e14b-55031--94971788668.zarr/4/ version=HTTP/1.1}: dandidav: Current memory usage: 325824512 physical, 869711872 virtual

so it seems about zarrs and their manifests... please review further and see what could be a reason that it hogs on memory

data point - we had a number of days without any memory issues although with traffic:
(dandisets-2) dandi@drogon:/mnt/backup/dandi/heroku-logs/dandidav$ echo -e "filename\t\t#memory\t#lines"; for f in 20240417-*.log; do echo -en "$f\t"; grep -c 'Memory quota exceeded' $f | tr '\n' '\t'; wc -l $f | awk '{print $1;}' ; done
filename                #memory #lines
20240417-0001.log       201     1807
20240417-0101.log       189     1104
20240417-0201.log       198     1197
20240417-0301.log       207     1180
20240417-0401.log       193     2239
20240417-0501.log       187     2433
20240417-0601.log       187     1770
20240417-0701.log       197     1036
20240417-0801.log       198     619
20240417-0901.log       215     724
20240417-1001.log       208     669
20240417-1101.log       210     730
20240417-1201.log       190     643
20240417-1301.log       206     673
20240417-1401.log       27      198
20240417-1501.log       0       249
20240417-1601.log       0       238
20240417-1701.log       0       196
20240417-1801.log       0       236
20240417-1901.log       0       429
20240417-2001.log       0       1128
20240417-2101.log       0       8187
20240417-2201.log       0       7785
20240417-2301.log       100     2731

I've adjusted dandidav's logging to now also log the change in memory usage after each request in order to make investigating the logs easier.

@yarikoptic

The requests that increase the amount of virtual memory[^1] used by the largest amounts are almost all for Zarrs under /zarrs/.

While Heroku has been complaining about excess memory usage constantly, since the start of July, dandidav has only been restarted (presumably always for memory issues) only about once a day:

Script
#!/bin/bash
set -u
printf 'Log\t\t\tQuota\tReboots\tDeploys\tLines\n'
for f in raw/*
do
    complaints="$(grep -cF 'Memory quota exceeded' "$f")"
    restarts="$(grep -cF SIG "$f")"
    if [ "$complaints" -gt 0 ] || [ "$restarts" -gt 0 ]
    then
        deploys="$(grep -cF 'Deploy by' "$f")"
        lines="$(wc -l "$f" | awk '{print $1}')"
        printf '%s\t%d\t%d\t%d\t%d\n' "$f" "$complaints" "$restarts" "$deploys" "$lines"
    fi
done

Output
Log			Quota	Reboots	Deploys	Lines
raw/20240701-0001.log	177	0	0	2260
raw/20240701-0101.log	178	0	0	2211
raw/20240701-0201.log	179	0	0	2182
raw/20240701-0301.log	176	0	0	1981
raw/20240701-0401.log	182	0	0	1993
raw/20240701-0501.log	181	0	0	2175
raw/20240701-0601.log	175	0	0	1616
raw/20240701-0701.log	175	0	0	2489
raw/20240701-0801.log	177	0	0	2785
raw/20240701-0901.log	175	0	0	1506
raw/20240701-1001.log	181	0	0	1599
raw/20240701-1101.log	188	0	0	1676
raw/20240701-1201.log	183	0	0	1535
raw/20240701-1301.log	185	0	0	1618
raw/20240701-1401.log	187	0	0	1317
raw/20240701-1501.log	188	0	0	1212
raw/20240701-1601.log	180	0	0	1986
raw/20240701-1701.log	180	0	0	2328
raw/20240701-1801.log	182	0	0	2450
raw/20240701-1901.log	178	0	0	3585
raw/20240701-2001.log	141	1	0	7133
raw/20240701-2301.log	93	0	0	3310
raw/20240702-0001.log	169	0	0	3539
raw/20240702-0101.log	171	0	0	3242
raw/20240702-0201.log	171	0	0	3602
raw/20240702-0301.log	168	0	0	3985
raw/20240702-0401.log	168	0	0	3761
raw/20240702-0501.log	167	0	0	3636
raw/20240702-0601.log	168	0	0	3589
raw/20240702-0701.log	168	0	0	3932
raw/20240702-0801.log	169	0	0	4231
raw/20240702-0901.log	167	0	0	3458
raw/20240702-1001.log	171	0	0	3928
raw/20240702-1101.log	166	0	0	3677
raw/20240702-1201.log	167	0	0	3589
raw/20240702-1301.log	171	0	0	2559
raw/20240702-1401.log	169	0	0	2885
raw/20240702-1501.log	170	0	0	3002
raw/20240702-1601.log	171	0	0	2880
raw/20240702-1701.log	169	0	0	2343
raw/20240702-1801.log	173	0	0	1725
raw/20240702-1901.log	168	0	0	1592
raw/20240702-2001.log	173	0	0	1509
raw/20240702-2101.log	59	1	0	1557
raw/20240702-2201.log	179	0	0	2201
raw/20240702-2301.log	174	0	0	2350
raw/20240703-0001.log	176	0	0	1578
raw/20240703-0101.log	177	0	0	1558
raw/20240703-0201.log	187	0	0	1640
raw/20240703-0301.log	183	0	0	1616
raw/20240703-0401.log	185	0	0	2131
raw/20240703-0501.log	180	0	0	2242
raw/20240703-0601.log	181	0	0	2482
raw/20240703-0701.log	177	0	0	2746
raw/20240703-0801.log	177	0	0	2988
raw/20240703-0901.log	180	0	0	2437
raw/20240703-1001.log	175	0	0	2546
raw/20240703-1101.log	177	0	0	3419
raw/20240703-1201.log	175	0	0	4224
raw/20240703-1301.log	177	0	0	2327
raw/20240703-1401.log	184	0	0	2752
raw/20240703-1501.log	176	0	0	2505
raw/20240703-1601.log	122	0	0	3114
raw/20240703-1801.log	152	0	0	2029
raw/20240703-1901.log	182	0	0	3215
raw/20240703-2001.log	173	0	0	5082
raw/20240703-2101.log	183	0	0	2937
raw/20240703-2201.log	182	0	0	2131
raw/20240703-2301.log	189	0	0	2072
raw/20240704-0001.log	176	0	0	2462
raw/20240704-0101.log	179	0	0	2657
raw/20240704-0201.log	179	0	0	2619
raw/20240704-0301.log	172	0	0	3320
raw/20240704-0401.log	141	1	0	3181
raw/20240704-0501.log	180	0	0	3346
raw/20240704-0601.log	177	0	0	2648
raw/20240704-0701.log	182	0	0	2146
raw/20240704-0801.log	178	0	0	1943
raw/20240704-0901.log	183	0	0	2088
raw/20240704-1001.log	175	0	0	1900
raw/20240704-1101.log	181	0	0	2395
raw/20240704-1201.log	176	0	0	2672
raw/20240704-1301.log	179	0	0	2483
raw/20240704-1401.log	180	0	0	3105
raw/20240704-1501.log	180	0	0	3142
raw/20240704-1601.log	175	0	0	3443
raw/20240704-1701.log	171	0	0	3422
raw/20240704-1801.log	177	0	0	3165
raw/20240704-1901.log	180	0	0	2734
raw/20240704-2001.log	179	0	0	3173
raw/20240704-2101.log	176	0	0	3011
raw/20240704-2201.log	176	0	0	3198
raw/20240704-2301.log	181	0	0	3185
raw/20240705-0001.log	178	0	0	3494
raw/20240705-0101.log	177	0	0	2611
raw/20240705-0201.log	182	0	0	2300
raw/20240705-0301.log	178	0	0	2230
raw/20240705-0401.log	180	0	0	1932
raw/20240705-0501.log	125	1	0	1635
raw/20240705-0601.log	174	0	0	1848
raw/20240705-0701.log	171	0	0	1476
raw/20240705-0801.log	180	0	0	1267
raw/20240705-0901.log	177	0	0	1085
raw/20240705-1001.log	178	0	0	1322
raw/20240705-1101.log	179	0	0	1195
raw/20240705-1201.log	180	0	0	1091
raw/20240705-1301.log	178	0	0	1080
raw/20240705-1401.log	180	0	0	1252
raw/20240705-1501.log	180	0	0	1041
raw/20240705-1601.log	182	0	0	994
raw/20240705-1701.log	182	0	0	829
raw/20240705-1801.log	181	0	0	819
raw/20240705-1901.log	187	0	0	829
raw/20240705-2001.log	181	0	0	856
raw/20240705-2101.log	191	0	0	809
raw/20240705-2201.log	198	0	0	728
raw/20240705-2301.log	188	0	0	847
raw/20240706-0001.log	185	0	0	636
raw/20240706-0101.log	199	0	0	548
raw/20240706-0201.log	202	0	0	538
raw/20240706-0301.log	202	0	0	588
raw/20240706-0401.log	202	0	0	740
raw/20240706-0501.log	130	1	0	739
raw/20240706-0701.log	81	0	0	942
raw/20240706-0801.log	196	0	0	991
raw/20240706-0901.log	208	0	0	924
raw/20240706-1001.log	206	0	0	1023
raw/20240706-1101.log	205	0	0	1022
raw/20240706-1201.log	195	0	0	915
raw/20240706-1301.log	208	0	0	754
raw/20240706-1401.log	204	0	0	904
raw/20240706-1501.log	212	0	0	884
raw/20240706-1601.log	192	0	0	967
raw/20240706-1701.log	200	0	0	853
raw/20240706-1801.log	200	0	0	902
raw/20240706-1901.log	198	0	0	911
raw/20240706-2001.log	196	0	0	914
raw/20240706-2101.log	205	0	0	939
raw/20240706-2201.log	210	0	0	842
raw/20240706-2301.log	211	0	0	776
raw/20240707-0001.log	214	0	0	753
raw/20240707-0101.log	202	0	0	1386
raw/20240707-0201.log	190	0	0	1064
raw/20240707-0301.log	193	0	0	1335
raw/20240707-0401.log	192	0	0	1244
raw/20240707-0501.log	196	0	0	1467
raw/20240707-0601.log	37	1	0	1054
raw/20240707-0701.log	28	0	0	926
raw/20240707-0801.log	176	0	0	1571
raw/20240707-0901.log	176	0	0	1353
raw/20240707-1001.log	176	0	0	1374
raw/20240707-1101.log	170	0	0	1629
raw/20240707-1201.log	175	0	0	1513
raw/20240707-1301.log	176	0	0	2816
raw/20240707-1401.log	179	0	0	1144
raw/20240707-1501.log	185	0	0	1389
raw/20240707-1601.log	173	0	0	1359
raw/20240707-1701.log	180	0	0	1291
raw/20240707-1801.log	183	0	0	3557
raw/20240707-1901.log	166	0	0	10994
raw/20240707-2001.log	165	0	0	34459
raw/20240707-2101.log	164	0	0	53459
raw/20240707-2201.log	166	0	0	963
raw/20240707-2301.log	192	0	0	744
raw/20240708-0001.log	189	0	0	603
raw/20240708-0101.log	194	0	0	594
raw/20240708-0201.log	199	0	0	683
raw/20240708-0301.log	181	0	0	809
raw/20240708-0401.log	181	0	0	912
raw/20240708-0501.log	186	0	0	972
raw/20240708-0601.log	166	1	0	845
raw/20240708-0701.log	44	1	0	772
raw/20240708-0801.log	196	0	0	895
raw/20240708-0901.log	206	0	0	947
raw/20240708-1001.log	206	0	0	1277
raw/20240708-1101.log	196	0	0	1335
raw/20240708-1201.log	193	0	0	1079
raw/20240708-1301.log	199	0	0	892
raw/20240708-1401.log	206	0	0	876
raw/20240708-1501.log	199	0	0	823
raw/20240708-1601.log	200	0	0	1019
raw/20240708-1701.log	199	0	0	865
raw/20240708-1801.log	205	0	0	960
raw/20240708-1901.log	202	0	0	1026
raw/20240708-2001.log	196	0	0	935
raw/20240708-2101.log	203	0	0	945
raw/20240708-2201.log	196	0	0	887
raw/20240708-2301.log	204	0	0	1560
raw/20240709-0001.log	186	0	0	1043
raw/20240709-0101.log	206	0	0	1513
raw/20240709-0201.log	185	0	0	1422
raw/20240709-0301.log	192	0	0	1570
raw/20240709-0401.log	186	0	0	1550
raw/20240709-0501.log	186	0	0	1500
raw/20240709-0601.log	189	0	0	1298
raw/20240709-0701.log	67	1	0	1442
raw/20240709-0801.log	78	0	0	766
raw/20240709-0901.log	180	0	0	1000
raw/20240709-1001.log	178	0	0	1026
raw/20240709-1101.log	94	0	0	830
raw/20240709-1201.log	174	0	0	1204
raw/20240709-1301.log	57	2	2	1049
raw/20240709-1401.log	36	0	0	12384
raw/20240709-1501.log	164	0	0	22790
raw/20240709-1601.log	166	0	0	14761
raw/20240709-1701.log	171	0	0	2401
raw/20240709-1801.log	180	0	0	1769
raw/20240709-1901.log	172	0	0	1822
raw/20240709-2001.log	175	0	0	1720
raw/20240709-2101.log	180	0	0	2228
raw/20240709-2201.log	96	1	1	2604
raw/20240709-2301.log	188	0	0	2324
raw/20240710-0001.log	188	0	0	2805
raw/20240710-0101.log	188	0	0	3052
raw/20240710-0201.log	187	0	0	2974
raw/20240710-0301.log	185	0	0	3531
raw/20240710-0401.log	187	0	0	3223
raw/20240710-0501.log	188	0	0	2958
raw/20240710-0601.log	186	0	0	3715
raw/20240710-0701.log	186	0	0	3751
raw/20240710-0801.log	184	0	0	3816
raw/20240710-0901.log	185	0	0	3403
raw/20240710-1001.log	166	1	1	3525
raw/20240710-1101.log	169	0	0	3505
raw/20240710-1201.log	167	0	0	4814
raw/20240710-1301.log	168	0	0	5547
raw/20240710-1401.log	147	1	0	5871
raw/20240710-1501.log	173	0	0	7685
raw/20240710-1601.log	175	0	0	4539
raw/20240710-1701.log	176	0	0	2942
raw/20240710-1801.log	175	0	0	2782
raw/20240710-1901.log	180	0	0	2852
raw/20240710-2001.log	181	0	0	3097
raw/20240710-2101.log	179	0	0	2372
raw/20240710-2201.log	175	0	0	3761
raw/20240710-2301.log	175	0	0	3454
raw/20240711-0001.log	175	0	0	3117
raw/20240711-0101.log	178	0	0	2867
raw/20240711-0201.log	178	0	0	2919
raw/20240711-0301.log	179	0	0	5960
raw/20240711-0401.log	174	0	0	3625
raw/20240711-0501.log	177	0	0	2789
raw/20240711-0601.log	190	0	0	6081
raw/20240711-0701.log	188	0	0	1411
raw/20240711-0801.log	184	0	0	5573
raw/20240711-0901.log	102	0	0	3628

[^1]: I'm assuming that virtual memory (rather than physical memory) is what Heroku cares about.  For some reason, the memory values reported by Heroku and the memory logging are very different, though they do appear to always be the same order of magnitude.

Thanks for the investigation!

The fact that it restarts just once a day is good for now, but it might deteriorate quickly whenever service gets used more often, so I would not account on the event to be conveniently sparse in time.
Given that memory consumption does go down some times, but not that much, under assumption that memory management in Rust does pick up all unused correctly (no memory leaks... but running under valgrind might be useful) - my guess would be that it is likely some caching of folders listing since there is so many of them under zarrs, which hogs memory and caching mechanism is not "limiting itself" by amount of free memory?

@yarikoptic The Zarr manifests cache is currently limited simply by the number of manifests it can store at once (set by the code to 16); if a new manifest is added while the cache is full, the least-frequently used manifest in the cache is deleted.  It seems that the cache dandidav uses can be configured to weigh entries by size, but I don't know what would happen if the code tried to cache a manifest larger than the maximum weight; it might refuse to cache it, resulting in a very large manifest being fetched & parsed for every URL in the Zarr requested.

So we might want to either increase memory (IMHO preferable!) on node or decrease to eg 12 from 16 and see how frequently we exhaust memory. Also getting some stats on cache hits logged once in a while would be great to inform us between the two solutions

@yarikoptic

Also getting some stats on cache hits logged

I'm not sure how to do that.  The caching library (moka) uses a different logging system (log) than the rest of the code (tracing), and I couldn't get a supposed bridge between them to work.  Switching the code over to just use log entirely wouldn't work, as the web framework (axum) and multiple other dependencies only use tracing.

Eh, with

and last release half a year ago, for a logging (core used functionally) library... sounds like a ticking bomb...
Log: release two weeks ago and
 but has half of stars , so less adopted I guess.
Not sure what to advise pragmatically on that, but I guess long term projects might need to move to log

@yarikoptic

So we might want to either increase memory (IMHO preferable!) on node or decrease to eg 12 from 16 and see how frequently we exhaust memory.

I believe the amount of memory is determined by the dyno type.  dandidav is currently using a "Basic" dyno, which has the smallest amount of RAM available (512 MB).  See https://devcenter.heroku.com/articles/dyno-types for all the possible dyno types and related information, and see https://devcenter.heroku.com/articles/usage-and-billing for the per-dyno pricing.

@yarikoptic I believe I've figured out how to log a message on every cache hit and/or miss.  Would that be acceptable?  What exactly should be logged?  For cache misses, do you want the log message to be emitted before the Zarr manifest is fetched & parsed, in case an out-of-memory death occurs during the fetching & parsing?

instance: so we would need to jump to Standard-2X to see any difference as it is the first which doubles RAM to 1GB.  I think it would be ok to do for a week or two to see if positive effect.
logging:

would you at the moment of logging know a number of active cache entries and if hitting the cache - which of the instances we hit so to see how recent entry we are hitting?
Also is there logging for cache cleanup, i.e. whenever it is a miss -- would we know that some record was cleaned up to accommodate a new one?
Would probably make sense to emit at the beginning and end of processing cache miss (with some identifier to pair each couple of such messages), so indeed we indeed be able to tell if out-of-memory death is in the middle.
Do we have any control over the cache as if at the end of processing a miss we see that we are close to RAM limit and would go out of it -- clean some LR cache entry?

@yarikoptic

would you at the moment of logging know a number of active cache entries

I can get an approximate count.  (I don't think calling run_pending_tasks() on every cache interaction would be very efficient.)

if hitting the cache - which of the instances we hit so to see how recent entry we are hitting?

I would know which instance was hit.  I'm not sure what you mean by "how recent entry."

Also is there logging for cache cleanup, i.e. whenever it is a miss -- would we know that some record was cleaned up to accommodate a new one?

As I said above, we don't have access to whatever logs moka generates, but we can add our own logging around the cache, and this could include logging of removed items.

Do we have any control over the cache as if at the end of processing a miss we see that we are close to RAM limit and would go out of it -- clean some LR cache entry?

Selectively removing cache items is possible, but checking whether we're near the RAM limit would be problematic as (a) I'm not sure exactly what memory stat Heroku cares about, in part because the memory values that Heroku itself logs don't match the memory values that dandidav gets from memory-stats, and (b) using memory-stats slows down response times by an order of magnitude, so I was hoping we could remove or disable it once we've gotten past this issue.

@yarikoptic I've set dandidav's dyno type to Standard-2X.

I'm not sure what you mean by "how recent entry."

It is LRU cache, so which "index" within ordered by "used" we hit? e.g. if we always hit [0] -- most recent -- there is no need to keep LRU cache of size 10 since we would never hit any other entry.  And on the opposite end, if we tend to hit the highest values, we might want to increase the cache size.

@yarikoptic It's actually LFU, not LRU, but either way there doesn't seem to be a way to get how recently/frequently a given cache entry has been used.

odd though since then we could keep memories of the glorious past - any settings on adding some kind of "recency" effect?  could we switch to LRU altogether? (I do not think LFU without any recency consideration is appropriate here)

@yarikoptic moka has options for LRU and LFU eviction policies, described at https://docs.rs/moka/latest/moka/policy/struct.EvictionPolicy.html.

I don't see any options (like size) described there at all :-/

@yarikoptic I mean that those are the options for what type of eviction policy you can have.  Cache size etc. are set on the CacheBuilder.

@yarikoptic Ping: What exactly should be logged when interacting with the cache?

can you log

cache hit
cache miss
cache eviction
?

what about setting
        // Time to idle (TTI):  5 minutes
        .time_to_idle(Duration::from_secs( 5 * 60))

?

@yarikoptic I can log on those events, yes, but what information should be included in the log messages?  Note that, for cache eviction, we'll only have access to the evicted value and the reason why it was evicted; the approximate number of entries currently in the cache and their total size is only available on hits & misses.

to log

the value (I guess -- path or url)
event: miss_start, miss_end, hit, evict

so on miss, log twice

evict_reason
memory: amount of memory used at that point
timestamp
cache_stats -- the approximate numbers you mentioned

ideally log as json record  so would be easier to parse out (after all this is all for a tool to figure out how cache behaves)

then write a tool which would go through logs, parse those out and would mimic behavior of the cache to tell us how cache is behaving:

miss_duration - mean and max time for miss processing
memory_sizes - mean and max of all memory diffs between miss_start and miss_end
miss_parallel -- mean and max -- number of parallel misses processing happening (so when miss_start for a new value happens before some prior miss_start finishes with miss_end).  I wonder if may be there is some temp memory consumption during miss processing which if multiplied with parallel triggers a kill?
hit_recencies - histogram of how recently accessed before (on miss or hit) each hit was -- pretty much a histogram with int per each of the "size(cache)" entries.   If we see that it is most always recent hits, we can easily drop size of cache.
evict_stats - just counts per each reason.  Might be informative

@yarikoptic

event: miss_start, miss_end

Is that necessary?  It'd be simpler to emit one type of message (not containing the fields you've listed other than value) before interacting with the cache and then emit a log message as you've described afterwards.

memory: amount of memory used at that point

Do you mean the total amount of memory used by the process?  If so, please re-read my last paragraph from this comment.

cache_stats -- the approximate numbers you mentioned

I think the approximate total size of the entries in the cache is only available if we limit the cache by total size rather than by number of entries.  Do we want to do that and, if so, what should the maximum size be?

ideally log as json record

Do you want the "message" portion of the log line to be JSON or do you want all of the logs printed by dandidav to use a JSON format?

@yarikoptic

event: miss_start, miss_end

Is that necessary? It'd be simpler to emit one type of message (not containing the fields you've listed other than value) before interacting with the cache and then emit a log message as you've described afterwards.

It sounds like it would complicate then parsing, e.g. how to associate those separate log messages with the earlier "cache" miss note?

memory: amount of memory used at that point

Do you mean the total amount of memory used by the process? If so, please re-read my last paragraph from this comment.

besides difference in amount, it overall sounds good since it should be just a matter of smth like
❯ time grep -e VmSize -e VmRSS /proc/$$/status
VmSize:	   17740 kB
VmRSS:	   11296 kB
grep --color=auto -d skip -e VmSize -e VmRSS /proc/$$/status  0.00s user 0.00s system 77% cpu 0.003 total

cache_stats -- the approximate numbers you mentioned

~I think the approximate total size of the entries in the cache is only available if we limit the cache by total size rather than by number of entries.~ (EDIT: This isn't true, though getting the total size when the cache isn't tracking it might be a little slow.) Do we want to do that and, if so, what should the maximum size be?

I am not entirely following here, since I was talking simply about stats you can explicitly request cache to report. But now you are hinting on the fact that we can specify max cache size consumption as in MBs? then let's say 100MB and see what happens?

ideally log as json record

Do you want the "message" portion of the log line to be JSON or do you want all of the logs printed by dandidav to use a JSON format?

I was thinking only about these specific ones, but it might make sense to just make them all into json records if that is easier somehow.

@yarikoptic

I am not entirely following here, since I was talking simply about stats you can explicitly request cache to report. But now you are hinting on the fact that we can specify max cache size consumption as in MBs?

There are two ways to get the total size of the entries in a moka cache: either call the weighted_size() method (if a weight function that returns entries' sizes was specified at cache creation) or iterate over all the entries in the cache and add up their sizes.

weighted_size() uses a weight function specified when the cache was built in order to determine the weight of each entry.  You can weigh elements by criteria other than size, and if no weigher was set, then each entry will be treated as weighing one unit.

A moka cache's maximum capacity is actually in weight units.  Without a weight function, the capacity sets the maximum number of entries.  If we set a weight function that returns the size of each entry, then the capacity sets the maximum total size of the entries in the cache.  Note that we cannot limit by both number of entries and total size of entries at once.

I'm pretty sure I've mentioned that moka caches can be limited by entry size before.

@yarikoptic For the record, I've currently gotten dandidav to emit caching-related logs like like the following:
{"timestamp":"2024-07-16T09:32:55.904745-04:00","level":"DEBUG","fields":{"message":"Cache miss for Zarr manifest; about to fetch from repository","event":"manifest_cache_miss_pre","manifest":"001/e3b/001e3b6d-26fb-463f-af28-520a25680ab4/326273bcc8730474323a66ea4e3daa49-113328--97037755426.json","cache_len":0},"target":"dandidav::zarrman","span":{"method":"GET","uri":"/zarrs/001/e3b/001e3b6d-26fb-463f-af28-520a25680ab4/326273bcc8730474323a66ea4e3daa49-113328--97037755426.zarr/","version":"HTTP/1.1","name":"request"},"spans":[{"method":"GET","uri":"/zarrs/001/e3b/001e3b6d-26fb-463f-af28-520a25680ab4/326273bcc8730474323a66ea4e3daa49-113328--97037755426.zarr/","version":"HTTP/1.1","name":"request"}]}
{"timestamp":"2024-07-16T09:32:56.96045-04:00","level":"DEBUG","fields":{"message":"Fetched Zarr manifest from repository","event":"manifest_cache_miss_post","manifest":"001/e3b/001e3b6d-26fb-463f-af28-520a25680ab4/326273bcc8730474323a66ea4e3daa49-113328--97037755426.json","cache_len":0},"target":"dandidav::zarrman","span":{"method":"GET","uri":"/zarrs/001/e3b/001e3b6d-26fb-463f-af28-520a25680ab4/326273bcc8730474323a66ea4e3daa49-113328--97037755426.zarr/","version":"HTTP/1.1","name":"request"},"spans":[{"method":"GET","uri":"/zarrs/001/e3b/001e3b6d-26fb-463f-af28-520a25680ab4/326273bcc8730474323a66ea4e3daa49-113328--97037755426.zarr/","version":"HTTP/1.1","name":"request"}]}
{"timestamp":"2024-07-16T09:32:58.152619-04:00","level":"DEBUG","fields":{"message":"Fetched Zarr manifest from cache","event":"manifest_cache_hit","manifest":"001/e3b/001e3b6d-26fb-463f-af28-520a25680ab4/326273bcc8730474323a66ea4e3daa49-113328--97037755426.json","cache_len":1},"target":"dandidav::zarrman","span":{"method":"GET","uri":"/zarrs/001/e3b/001e3b6d-26fb-463f-af28-520a25680ab4/326273bcc8730474323a66ea4e3daa49-113328--97037755426.zarr/0/","version":"HTTP/1.1","name":"request"},"spans":[{"method":"GET","uri":"/zarrs/001/e3b/001e3b6d-26fb-463f-af28-520a25680ab4/326273bcc8730474323a66ea4e3daa49-113328--97037755426.zarr/0/","version":"HTTP/1.1","name":"request"}]}
{"timestamp":"2024-07-16T09:34:23.873045-04:00","level":"DEBUG","fields":{"message":"Zarr manifest evicted from cache","event":"manifest_cache_evict","manifest":"001/e3b/001e3b6d-26fb-463f-af28-520a25680ab4/326273bcc8730474323a66ea4e3daa49-113328--97037755426.json","cause":"Expired"},"target":"dandidav::zarrman","span":{"method":"GET","uri":"/zarrs/002/f5d/002f5d1d-d7b8-4c7d-97ee-cae835868c77/9dbf62c5f04ddf99b08156a6527b8cdd-97192--48350970234.zarr/","version":"HTTP/1.1","name":"request"},"spans":[{"method":"GET","uri":"/zarrs/002/f5d/002f5d1d-d7b8-4c7d-97ee-cae835868c77/9dbf62c5f04ddf99b08156a6527b8cdd-97192--48350970234.zarr/","version":"HTTP/1.1","name":"request"}]}

@yarikoptic Also, I wrote a utility for determining how much memory a parsed Zarr manifest takes up: https://github.com/jwodder/manifest-size

@yarikoptic

hit_recencies - histogram of how recently accessed before (on miss or hit) each hit was -- pretty much a histogram with int per each of the "size(cache)" entries.

To be clear, you want a report on the durations between each hit and the previous hit or miss for that manifest path, correct?  I'm not sure how to interpret "a histogram with int per each of the "size(cache)" entries".

@yarikoptic Also, do you want the log analysis tool to produce output in a specific format?  Your use of field names in the description suggests that you want JSON or similar.  Is the histogram supposed to be an actual graph produced with matplotlib, or would just a textual list of numbers be sufficient?

@yarikoptic Now that dandidav has been running on a Standard-2X dyno for a week, here's an updated summary of the logs for July:
Date        Requests   MemMsgs  Restarts    Cycles   Deploys
2024-07-01      3497      3607         0         0         0
2024-07-02      6995      3619         0         1         0
2024-07-03      4586      3901         0         1         0
2024-07-04      5930      4254         0         0         0
2024-07-05      4191      4250         0         1         0
2024-07-06      1057      4382         0         1         0
2024-07-07      3668      4176         0         1         0
2024-07-08      2910      4447         0         2         0
2024-07-09      5573      3842         0         1         2
2024-07-10      7001      4197         2         0         2
2024-07-11     10045      4189         1         0         0
2024-07-12     29855      3948         1         1         2
2024-07-13     27899      3460         0         1         0
2024-07-14      6904         0         1         0         2
2024-07-15      3448         0         0         1         0
2024-07-16      4287       148         0         1         0
2024-07-17      3513         0         0         1         0
2024-07-18      5516         4         0         1         0
2024-07-19      6372         0         0         1         0

Script used to generate the summary
#!/usr/bin/env python3
# /// script
# requires-python = ">=3.11"
# dependencies = []
# ///
from __future__ import annotations
from collections import Counter
from datetime import UTC, date, datetime
import re
import sys

def main() -> None:
    days = set()
    requests_per_day: dict[date, int] = Counter()
    quota_msgs_per_day: dict[date, int] = Counter()
    deployments_per_day: dict[date, int] = Counter()
    cyclings_per_day: dict[date, int] = Counter()
    restart_stamps = []
    for fname in sys.argv[1:]:
        with open(fname) as fp:
            for line in fp:
                m = re.match(
                    r"\d{4}-\d\d-\d\dT\d\d:\d\d:\d\d\.\d+[-+]\d\d:\d\d\s*", line
                )
                if not m:
                    # Warn? Error?
                    continue
                timestamp = datetime.fromisoformat(m[0].strip()).astimezone(UTC)
                day = timestamp.date()
                days.add(day)
                line = line[m.end() :].strip()
                if line.startswith("heroku[router]:"):
                    requests_per_day[day] += 1
                elif re.fullmatch(
                    r"heroku\[web\.1\]: Error (?:R14 \(Memory quota exceeded\)"
                    r"|R15 \(Memory quota vastly exceeded\))",
                    line,
                ):
                    quota_msgs_per_day[day] += 1
                elif line == "heroku[web.1]: Cycling":
                    cycling = True
                elif re.fullmatch(
                    r"heroku\[web\.1\]: Stopping (all processes with SIGTERM"
                    r"|process with SIGKILL)",
                    line,
                ):
                    if cycling:
                        cyclings_per_day[day] += 1
                        cycling = False
                    else:
                        restart_stamps.append(timestamp)
                elif line.startswith(("app[api]: Deploy by", "app[api]: Scaled to")):
                    assert not cycling
                    deployments_per_day[day] += 1
                    try:
                        restart_stamps.pop()
                    except IndexError:
                        pass
    restarts_per_day = Counter(ts.date() for ts in restart_stamps)
    print("Date        Requests   MemMsgs  Restarts    Cycles   Deploys")
    for d in sorted(days):
        requests = requests_per_day[d]
        restarts = restarts_per_day[d]
        cycles = cyclings_per_day[d]
        quota_msgs = quota_msgs_per_day[d]
        deployments = deployments_per_day[d]
        print(
            f"{d}  {requests:8d}  {quota_msgs:8d}  {restarts:8d}  {cycles:8d}"
            f"  {deployments:8d}"
        )

if __name__ == "__main__":
    main()

About the summary:

Dates are in UTC
"Requests" is the number of HTTP requests made to the server
"MemMsgs" is the number of log messages emitted about error R14 (Memory quota exceeded) or R15 (Memory quota vastly exceeded)
"Restarts" is the number of restarts not caused by cycling or deployment
"Cycles" is the number of restarts caused by Heroku automatically restarting dynos every 24 hours
"Deploys" is the number of times a new server executable was deployed

so starting from which date exactly? is that the one with 1 restart
2024-07-14      6904         0         1

?  then we did not really tested it in restarts causing conditions which seems to happen whenever there is 10 of thousands of requests... I guess we are not running healthchecks any longer... indeed we do not

here is a diff I am reverting for run.sh
dandi@drogon:~/cronlib/dandisets-healthstatus$ git diff
diff --git a/tools/run.sh b/tools/run.sh
index 585de7d5..c7d3ee63 100755
--- a/tools/run.sh
+++ b/tools/run.sh
@@ -14,11 +14,11 @@ cd "$(dirname "$0")"/..

 "$PYTHON" -m virtualenv --clear venv
 . venv/bin/activate
-pip install ./code
+chronic pip install ./code
 #pip install 'git+https://github.com/fsspec/filesystem_spec'
 #pip install 'git+https://github.com/jwodder/filesystem_spec@rlock-cache'
-dandisets-healthstatus check -m "$MOUNT_PATH" -J 10 "$@"
-dandisets-healthstatus report
+dandisets-healthstatus check -m "$MOUNT_PATH" -J 10 --mode random-outdated-asset-first 000027
+#dandisets-healthstatus report

 # TODO: Uncomment this block when setting up the cronjob:
 #git add .
@@ -28,7 +28,7 @@ dandisets-healthstatus report
 #else echo "No changes to commit"
 #fi

-datalad save -m "Results from a sweep of an archive: $*"
-datalad push
+#datalad save -m "Results from a sweep of an archive: $*"
+#datalad push

 # vim:set et sts=4:

but apparently we are not in `main` branch on drogon for healthchecks
dandi@drogon:~/cronlib/dandisets-healthstatus$ git pull --ff-only
remote: Enumerating objects: 49, done.
remote: Counting objects: 100% (21/21), done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 49 (delta 18), reused 18 (delta 18), pack-reused 28
Unpacking objects: 100% (49/49), 20.18 KiB | 666.00 KiB/s, done.
From github.com:dandi/dandisets-healthstatus
   eef70d99..789e8071  main       -> origin/main
Your configuration specifies to merge with the ref 'refs/heads/traverse-api'
from the remote, but no such ref was fetched.
dandi@drogon:~/cronlib/dandisets-healthstatus$ git branch
  ds-arg
  gh-2
  gh-47
  gh-9
  main
* traverse-api

@jwodder please bring back that repository into "main" line (I do not want to possibly destroy of any other changes you might want to keep) and invoke tools/run_loop.sh under screen so we get healthchecks going again and adding to Requests above

@yarikoptic

so starting from which date exactly?

Starting from the 12th at 3:45 PM EDT.

please bring back that repository into "main" line ... and invoke tools/run_loop.sh under screen

Done.

@yarikoptic

Ping on https://github.com/dandi/dandidav/issues/118#issuecomment-2233243760
Ping on https://github.com/dandi/dandidav/issues/118#issuecomment-2233270697
Are you sure the Zarr manifest cache logs should include the total memory used by the process?  Getting that value slows things down.
Do you want the Zarr manifest cache to continue to be bounded by number of entries, or do you want it to be bounded by the total size of the entries instead?
Should the Zarr manifest cache logs include the current total size of the cache entries?
Should the Zarr manifest cache logs include the size of the entry currently being requested/evicted?
FYI, I ran the tool I wrote for measuring Zarr manifest size on every Zarr manifest in the repository; the largest was https://datasets.datalad.org/dandi/zarr-manifests/zarr-manifests-v2-sorted/5c3/7c2/5c37c233-222f-4e60-96e7-a7536e08ef61/4cb549b2e2346bb1a30f493b50fb6a2e-526320--1023396474554.json, which is 62,943,289 bytes as raw JSON and 86,010,142 bytes when parsed into a Rust structure.

@yarikoptic Also, here is an up-to-date summary of the logs for July after fixing a couple of issues with the script:
Date        Requests   MemMsgs  Restarts    Cycles   Deploys
2024-07-01      3336      3431         0         0         0
2024-07-02      6771      3501         0         1         0
2024-07-03      4390      3737         0         1         0
2024-07-04      5710      4075         1         0         0
2024-07-05      3962      3948         0         1         0
2024-07-06       917      3822         0         1         0
2024-07-07      3467      3816         0         1         0
2024-07-08      2763      3972         0         1         0
2024-07-09      5389      3583         0         1         2
2024-07-10      6788      4051         1         0         2
2024-07-11      9824      4031         1         0         0
2024-07-12     29642      3882         2         1         2
2024-07-13     27674      3326         0         1         0
2024-07-14      6676         0         0         0         2
2024-07-15      3174         0         0         1         0
2024-07-16      4037       130         0         1         0
2024-07-17      3247         0         0         1         0
2024-07-18      5249         4         0         1         0
2024-07-19      7277       367         0         1         0
2024-07-20      5866      2644         0         1         0
2024-07-21      6655         0         0         1         0
2024-07-22      5370         0         0         0         1

Script
#!/usr/bin/env python3
# /// script
# requires-python = ">=3.11"
# dependencies = []
# ///
from __future__ import annotations
from collections import Counter
from datetime import UTC, datetime, timedelta
import re
import sys

# Assume any restarts this close to a deployment are due to it:
DEPLOY_WINDOW = timedelta(seconds=120)

def main() -> None:
    days = set()
    # Sometimes the same log message ends up in more than one log file, so we
    # have to dedup events by timestamp and then, once that's done, bucket them
    # by date.
    requests_stamps = set()
    quota_msgs_stamps = set()
    deployments_stamps = set()
    cyclings_stamps = set()
    restart_stamps = []
    current_deploy_time: datetime | None = None
    for fname in sys.argv[1:]:
        with open(fname) as fp:
            for line in fp:
                m = re.match(
                    r"\d{4}-\d\d-\d\dT\d\d:\d\d:\d\d\.\d+[-+]\d\d:\d\d\s*", line
                )
                if not m:
                    # Warn? Error?
                    continue
                timestamp = datetime.fromisoformat(m[0].strip()).astimezone(UTC)
                day = timestamp.date()
                days.add(day)
                line = line[m.end() :].strip()
                if line.startswith("heroku[router]:"):
                    requests_stamps.add(timestamp)
                elif re.fullmatch(
                    r"heroku\[web\.1\]: Error (?:R14 \(Memory quota exceeded\)"
                    r"|R15 \(Memory quota vastly exceeded\))",
                    line,
                ):
                    quota_msgs_stamps.add(timestamp)
                elif line == "heroku[web.1]: Cycling":
                    cycling = True
                elif re.fullmatch(
                    r"heroku\[web\.1\]: Stopping (all processes with SIGTERM"
                    r"|process with SIGKILL)",
                    line,
                ):
                    if cycling:
                        cyclings_stamps.add(timestamp)
                        cycling = False
                    elif not (
                        current_deploy_time is not None
                        and abs(timestamp - current_deploy_time) < DEPLOY_WINDOW
                    ):
                        restart_stamps.append(timestamp)
                elif line.startswith(("app[api]: Deploy by", "app[api]: Scaled to")):
                    assert not cycling
                    deployments_stamps.add(timestamp)
                    current_deploy_time = timestamp
                    while (
                        restart_stamps
                        and abs(timestamp - restart_stamps[-1]) < DEPLOY_WINDOW
                    ):
                        restart_stamps.pop()
    requests_per_day = Counter(ts.date() for ts in requests_stamps)
    quota_msgs_per_day = Counter(ts.date() for ts in quota_msgs_stamps)
    deployments_per_day = Counter(ts.date() for ts in deployments_stamps)
    cyclings_per_day = Counter(ts.date() for ts in cyclings_stamps)
    restarts_per_day = Counter(ts.date() for ts in set(restart_stamps))
    print("Date        Requests   MemMsgs  Restarts    Cycles   Deploys")
    for d in sorted(days):
        requests = requests_per_day[d]
        restarts = restarts_per_day[d]
        cycles = cyclings_per_day[d]
        quota_msgs = quota_msgs_per_day[d]
        deployments = deployments_per_day[d]
        print(
            f"{d}  {requests:8d}  {quota_msgs:8d}  {restarts:8d}  {cycles:8d}"
            f"  {deployments:8d}"
        )

if __name__ == "__main__":
    main()

@yarikoptic

hit_recencies - histogram of how recently accessed before (on miss or hit) each hit was -- pretty much a histogram with int per each of the "size(cache)" entries.

To be clear, you want a report on the durations between each hit and the previous hit or miss for that manifest path, correct? I'm not sure how to interpret "a histogram with int per each of the 'size(cache)' entries."

not for that manifest path, but rather for that "cache index" if cache index how recently hit index. I.e. how recent we typically hit cache entries in terms of their recency -- do we hit most recent right away within a ms or it takes seconds?
What about least recent -- does it take hours to hit any later entry or it is also hit within seconds (hence we better increase cache size)?

@yarikoptic Also, do you want the log analysis tool to produce output in a specific format? Your use of field names in the description suggests that you want JSON or similar. Is the histogram supposed to be an actual graph produced with matplotlib, or would just a textual list of numbers be sufficient?

textual would be sufficient, could even be just structured json/yaml structure

Are you sure the Zarr manifest cache logs should include the total memory used by the process?  Getting that value slows things down.

if slows down, ok - let's skip, but can we somehow track sizes of records in the cache entries? This relates to those questions below:

Do you want the Zarr manifest cache to continue to be bounded by number of entries, or do you want it to be bounded by the total size of the entries instead?

can we augment eviction policy as e.g. indeed having considered a miss, check total size and possibly trigger eviction until we fall under some threshold in terms of size?

Should the Zarr manifest cache logs include the current total size of the cache entries?
Should the Zarr manifest cache logs include the size of the entry currently being requested/evicted?
FYI, I ran the tool I wrote for measuring Zarr manifest size on every Zarr manifest in the repository; the largest was https://datasets.datalad.org/dandi/zarr-manifests/zarr-manifests-v2-sorted/5c3/7c2/5c37c233-222f-4e60-96e7-a7536e08ef61/4cb549b2e2346bb1a30f493b50fb6a2e-526320--1023396474554.json, which is 62,943,289 bytes as raw JSON and 86,010,142 bytes when parsed into a Rust structure.

so 86MB -- good to know... so we should be able to cache at least a few.

@yarikoptic

not for that manifest path, but rather for that "cache index"

If by "cache index" you mean the specific entry in the cache or its key, that's what I was referring to by "manifest path," as the cache entries are keyed by an identifier for the Zarr manifest.  If that's not what you meant, then I have no idea what you're saying here.

if cache index how recently hit index. I.e. how recent we typically hit cache entries in terms of their recency -- do we hit most recent right away within a ms or it takes seconds?
What about least recent -- does it take hours to hit any later entry or it is also hit within seconds (hence we better increase cache size)?

I can't tell what you're trying to say here.  Could you reword this or otherwise elaborate on exactly what information you want regarding recency?  Could you give an example?

can we somehow track sizes of records in the cache entries?

I can log the size of an individual cache entry whenever it's hit, missed, or evicted.  If you want the total size of all entries, we get that automatically if we choose to limit the cache by total entry size.  Do you want either of these in the logs?

can we augment eviction policy as e.g. indeed having considered a miss, check total size and possibly trigger eviction until we fall under some threshold in terms of size?

If by "augment" you mean that you want the cache to be limited by both the number of entries and their total size, no.  It's one or the other.

@yarikoptic Ping.

not for that manifest path, but rather for that "cache index"

If by "cache index" you mean the specific entry in the cache or its key, that's what I was referring to by "manifest path," as the cache entries are keyed by an identifier for the Zarr manifest. If that's not what you meant, then I have no idea what you're saying here.

if cache index how recently hit index. I.e. how recent we typically hit cache entries in terms of their recency -- do we hit most recent right away within a ms or it takes seconds?
What about least recent -- does it take hours to hit any later entry or it is also hit within seconds (hence we better increase cache size)?

I can't tell what you're trying to say here. Could you reword this or otherwise elaborate on exactly what information you want regarding recency? Could you give an example?

ok, lets play out an example with LRU cache of size 2 and paths 1 2 3 4 , where first I list the path, and then cache where I order cache sorted by hit/recency among entries -- position of path within that list would instruct the "index" I was talking about, and then I add cache-index-hit as what index was hit with this new value in the prior cache:

1 [1] cache-index-hit: None
2 [ 2 1 ] cache-index-hit: None
3 [ 3 2 ] cache-index-hit: None
2 [ 2 3 ] cache-index-hit: 1
2 [ 2 3 ] cache-index-hit: 0
2 [ 2 3 ] cache-index-hit: 0
3 [ 3 2 ] cache-index-hit: 1
2 [ 2 3 ] cache-index-hit: 1

so in this case we would have histogram of index-hits: {None: 3, 0: 2, 1: 3} so we had 3 misses (no index at all), and overall slightly preferred 2nd recent entry.

can we somehow track sizes of records in the cache entries?

I can log the size of an individual cache entry whenever it's hit, missed, or evicted. If you want the total size of all entries, we get that automatically if we choose to limit the cache by total entry size. Do you want either of these in the logs?

but also "total" is just a sum of entries in the cache, right?  so we can log individual sizes and compute by tracking history of miss/evict entries,  right?
Given that we are experiencing out of memory condition, shouldn't we indeed just limit cache by total entry size and not number of entries?

can we augment eviction policy as e.g. indeed having considered a miss, check total size and possibly trigger eviction until we fall under some threshold in terms of size?

If by "augment" you mean that you want the cache to be limited by both the number of entries and their total size, no. It's one or the other.

gotcha... I was thinking about really custom (some callback) handler which upon miss would decide what (if any) entry to evict and/or either to actually cache this new miss?  (e.g. we might not cache any entry which exceeds our memory size, or to not cache anyhow known to be rare but large queries, while favoring caching some other smaller ones)

@yarikoptic

but also "total" is just a sum of entries in the cache, right? so we can log individual sizes and compute by tracking history of miss/evict entries, right?

Yes.  Do you want the logs to contain individual entry sizes, total entry sizes, both, or neither?  Should any of this (or statistics derived therefrom) be included in the output from the log-analysis tool?

Given that we are experiencing out of memory condition, shouldn't we indeed just limit cache by total entry size and not number of entries?

Do you still want the total size to be 100 MB as stated in this comment or a different value?  And is that a decimal MB (= 1,000,000 bytes) or binary MB (= 1,048,576 bytes)?

@yarikoptic Regarding cache indices: There's no guarantee that the logs fed to the analysis tool will start right at the start of a server run, so the tool could encounter records of hits for entries that it never saw previous accesses for, leading to potentially inaccurate cache hit indices.  Do you have a preference for how that should be handled?

@yarikoptic Regarding cache indices: There's no guarantee that the logs fed to the analysis tool will start right at the start of a server run

why? couldn't we restart the server and feed logs since that point? or do you mean that there is log rotation happening so we might need to feed multiple log files for the complete "history"?

@yarikoptic Yes, if we're careful, things will work, but if the tool is run on an arbitrary collection of logs without being careful, the logs will start in the middle of a server run.  We can't count on always being careful.

@yarikoptic Ping on the following questions:

Assuming you want the cache to be limited by total entry size, what should the limit be?  You previously suggested 100 MB.  Should that be the limit, and is "MB" here decimal (= 1,000,000 bytes) or binary (= 1,048,576 bytes)?

Do you want the logs to contain individual entry sizes, total entry sizes, both, or neither?  Should any of this (or statistics derived therefrom) be included in the output from the log-analysis tool?

@yarikoptic Ping.

I didn't really care either it is 1000 or 1024 base. Overall - should be configurable
let's log both if cheap
would we have access to full cache to do periodic (eg each hour) log dump of what is in the cache and sizes? May be could be done on receiving HUP signal - then we could trigger a frequently as we like as well

@yarikoptic

Overall - should be configurable

Do you mean configurable via a CLI option or a constant in the code or something else?

would we have access to full cache to do periodic (eg each hour) log dump of what is in the cache and sizes?

Yes, that's doable.

May be could be done on receiving HUP signal - then we could trigger a frequently as we like as well

That seems a bit superfluous.  Is there even a way to send a SIGHUP on Heroku?

@yarikoptic FYI, the logs for Zarr cache events currently look like this:
{"timestamp":"2024-08-05T10:50:13.135433-04:00","level":"DEBUG","fields":{"message":"Cache miss for Zarr manifest; about to fetch from repository","cache_event":"miss_pre","cache":"zarr-manifests","manifest":"001/e3b/001e3b6d-26fb-463f-af28-520a25680ab4/326273bcc8730474323a66ea4e3daa49-113328--97037755426.json","approx_cache_len":0,"approx_cache_size":0},"target":"dandidav::zarrman","span":{"id":"76b4f503-1be7-4316-8ba1-1daf548d7ce5","manifest":"001/e3b/001e3b6d-26fb-463f-af28-520a25680ab4/326273bcc8730474323a66ea4e3daa49-113328--97037755426.json","name":"get_zarr_manifest"},"spans":[{"method":"GET","uri":"/zarrs/001/e3b/001e3b6d-26fb-463f-af28-520a25680ab4/326273bcc8730474323a66ea4e3daa49-113328--97037755426.zarr/","version":"HTTP/1.1","name":"request"},{"id":"76b4f503-1be7-4316-8ba1-1daf548d7ce5","manifest":"001/e3b/001e3b6d-26fb-463f-af28-520a25680ab4/326273bcc8730474323a66ea4e3daa49-113328--97037755426.json","name":"get_zarr_manifest"}]}
{"timestamp":"2024-08-05T10:50:13.965786-04:00","level":"DEBUG","fields":{"message":"Fetched Zarr manifest from repository","cache_event":"miss_post","cache":"zarr-manifests","manifest":"001/e3b/001e3b6d-26fb-463f-af28-520a25680ab4/326273bcc8730474323a66ea4e3daa49-113328--97037755426.json","manifest_size":18170624,"approx_cache_len":0,"approx_cache_size":0},"target":"dandidav::zarrman","span":{"id":"76b4f503-1be7-4316-8ba1-1daf548d7ce5","manifest":"001/e3b/001e3b6d-26fb-463f-af28-520a25680ab4/326273bcc8730474323a66ea4e3daa49-113328--97037755426.json","name":"get_zarr_manifest"},"spans":[{"method":"GET","uri":"/zarrs/001/e3b/001e3b6d-26fb-463f-af28-520a25680ab4/326273bcc8730474323a66ea4e3daa49-113328--97037755426.zarr/","version":"HTTP/1.1","name":"request"},{"id":"76b4f503-1be7-4316-8ba1-1daf548d7ce5","manifest":"001/e3b/001e3b6d-26fb-463f-af28-520a25680ab4/326273bcc8730474323a66ea4e3daa49-113328--97037755426.json","name":"get_zarr_manifest"}]}
{"timestamp":"2024-08-05T10:50:15.469629-04:00","level":"DEBUG","fields":{"message":"Fetched Zarr manifest from cache","cache_event":"hit","cache":"zarr-manifests","manifest":"001/e3b/001e3b6d-26fb-463f-af28-520a25680ab4/326273bcc8730474323a66ea4e3daa49-113328--97037755426.json","manifest_size":18170624,"approx_cache_len":1,"approx_cache_size":18170624},"target":"dandidav::zarrman","span":{"id":"0bf7db6c-2f1a-4599-bbfa-3c8facc7f5cf","manifest":"001/e3b/001e3b6d-26fb-463f-af28-520a25680ab4/326273bcc8730474323a66ea4e3daa49-113328--97037755426.json","name":"get_zarr_manifest"},"spans":[{"method":"GET","uri":"/zarrs/001/e3b/001e3b6d-26fb-463f-af28-520a25680ab4/326273bcc8730474323a66ea4e3daa49-113328--97037755426.zarr/0/","version":"HTTP/1.1","name":"request"},{"id":"0bf7db6c-2f1a-4599-bbfa-3c8facc7f5cf","manifest":"001/e3b/001e3b6d-26fb-463f-af28-520a25680ab4/326273bcc8730474323a66ea4e3daa49-113328--97037755426.json","name":"get_zarr_manifest"}]}
{"timestamp":"2024-08-05T10:50:29.57411-04:00","level":"DEBUG","fields":{"message":"Zarr manifest evicted from cache","cache_event":"evict","cache":"zarr-manifests","manifest":"001/e3b/001e3b6d-26fb-463f-af28-520a25680ab4/326273bcc8730474323a66ea4e3daa49-113328--97037755426.json","manifest_size":18170624,"cause":"Expired"},"target":"dandidav::zarrman","span":{"id":"8561ffcf-3a2e-4e6e-9375-40c89cace59c","manifest":"003/eb0/003eb053-d45c-42e2-a040-79bca28c84ae/481a2f77ab786a0f45aafd5db0971caa-0--0.json","name":"get_zarr_manifest"},"spans":[{"method":"GET","uri":"/zarrs/003/eb0/003eb053-d45c-42e2-a040-79bca28c84ae/481a2f77ab786a0f45aafd5db0971caa-0--0.zarr/","version":"HTTP/1.1","name":"request"},{"id":"8561ffcf-3a2e-4e6e-9375-40c89cace59c","manifest":"003/eb0/003eb053-d45c-42e2-a040-79bca28c84ae/481a2f77ab786a0f45aafd5db0971caa-0--0.json","name":"get_zarr_manifest"}]}

@yarikoptic

Overall - should be configurable

Do you mean configurable via a CLI option or a constant in the code or something else?

CLI or env var or you could even establish config

May be could be done on receiving HUP signal - then we could trigger a frequently as we like as well

That seems a bit superfluous. Is there even a way to send a SIGHUP on Heroku?

I dunno, didn't check.  Just know that HUP is often used as a signal to trigger some action (like reload of config or log rotation).
logs for cache look good to me.  ( I guess cache size was 1 and that lead to the eviction in the last entry?)

@jwodder could you place that script to process heroku logs on cron somewhere and put pointer to the file here?  I am interested where we are ATM in the status on this

@yarikoptic

Are you referring to the script for generating the Requests/MemMsgs/Restarts/Cycles/Deploys table or the zarr-cache-stats.py script added in #170?

Are you saying that you want a cronjob created that runs the script periodically?  If so, how often, what logfiles should be provided as input, and where should the output be saved?

@yarikoptic

Are you referring to the script for generating the Requests/MemMsgs/Restarts/Cycles/Deploys table or the zarr-cache-stats.py script added in Add logging around Zarr manifest cache interactions #170?

yes

Are you saying that you want a cronjob created that runs the script periodically?

yes

If so, how often,

seems to be doing daily stats, so daily

what logfiles should be provided as input,

whatever you gave to produces those stats.

and where should the output be saved?

It is not for public consumption, so anywhere, e.g.  ~dandi/.cache/dandi/zarr-cache-stats.log. Can even tee it so cron job emails those stats daily

@yarikoptic The first question was an either/or question, not a yes/no question.

I was talking about " the script for generating the Requests/MemMsgs/Restarts/Cycles/Deploys table "

@yarikoptic Done.  The output is saved in /home/dandi/cronlib/dandidav-restarts/report.txt.


Update README.md

Added AppScan and Checkmarx

Thank you @Hipapheralkus,
I will happily merge that. Just one little thing: could we make the description for AppScan a little shorter? Please see my comment there. 😉

Looks good. Merging.


ClrMD 2.0 reports less number of heap objects than ClrMD 1.1 on a 32-bit dump

I have a dump of the 32-bit application.
When I enumerate the heap objects with method heap.EnumerateObjects(), I get different results when using ClrMD 1.1 and ClrMD 2.0.
With ClrMD 1.1, I'm getting 30 666 651 objects. The same number I can get with WinDbg.
ClrMD 2.0 shows me almost three times fewer objects — 11 814 763.

I'm also seeing this on 64-bit dumps.
For ClrMD 1.1.142101
var target = DataTarget.LoadCrashDump( @"F:\w3wp.exe.dmp" )

var clr = target.ClrVersions [ 0 ];

var runtime = clr.CreateRuntime( );

var rootCount = runtime.Heap.EnumerateRoots( ).Count( );
var objectCount = runtime.Heap.EnumerateObjects( ).Count( );

For ClrMD 2.0.161401
var target = DataTarget.LoadDump( @"F:\w3wp.exe.dmp" );

var clr = target.ClrVersions[ 0 ];

using ( var runtime = clr.CreateRuntime( ) )
{
    var rootCount = runtime.Heap.EnumerateRoots( ).Count();
    var objectCount = runtime.Heap.EnumerateObjects( ).Count( );
}

Testing on 3 different dump files:
ClrMD 1.1.142101
================

Dump		Runtime			Root Count		Object Count
----------------------------------------------------------------------------
A		4.7.3701.00		11762			106388187
B		4.8.4311.00		12364			137758778
C		4.8.4311.00		12859			134561991

ClrMD 2.0.161401
================

Dump		Runtime			Root Count		Object Count
----------------------------------------------------------------------------
A		4.7.3701.00		1251			106388187
B		4.8.4311.00		0			1062
C		4.8.4311.0		0			808

Is there a test dump I could use to reproduce this?

I am closing this issue.  Without a dump or repro I could not make progress here.  As far as I know, ClrMD 3.1 accurately reports all objects.  Feel free to open a new issue or reply back here if there is a specific example that I should take a look at.  Thanks!


[docs] Fix /api not being considered for client-side navigation

Pages under /api currently cause a full page reload i.e. are not part of client-side navigation:

goto  https://material-ui.netlify.com/api/app-bar/
click on another api page
notice theme flash indicating a full page reload

This PR fixes this by rewriting as instead of href:

goto https://deploy-preview-20392--material-ui.netlify.com/api/app-bar/
click on another api page
no more flash

Aside:
Once we get getStaticProps for _app we're able to refactor the routing of our docs. It's currently very hard to follow.
The problem is that next needs to map the href to an actual page in the file system. This is not the same what is displayed in the actual href of the anchor tag. This sounds backwards to me. I would find <Link href="/some-arbitrary-url" to="/some/page/under/docs/pages" /> more intuitive since href keeps it's meaning. Redefining an existing prop under a different context makes things needlessly complicate. I think they even plan on implementing the to prop if I recall correctly.

Looks good. @oliviertassinari could you take a look in case I missed some edge case?

Nevermind localized versions produce 404. Exported pages worked locally :(

Not spending more time on 18n routing. Let's wait for getStaticProps on _app and rebuild from scratch. This export vs build vs dev and rewrites vs href vs as vs page vs pathname indirection is not worth it at the moment.


AzurePublicDatasetV2 - Workload categories and VM roles

Hello everyone,
I am using the dataset for research purposes, and I have some questions related to the workload. In vmtable.csv some VMs are labeled as "interactive", other as "delay-insensitive", and most of them as "unknown." I would like to know how this classification has been performed, and what do they mean. E.g., is it safe to think that in the "interactive" workload include web-services?
Related to that, what does a deployment represent? Is this an application? Does it follow the definition of deployments for container strategies?
Thank you very much in advance.

where can I download this dataset? It seems I cannot access the website https://azurecloudpublicdataset2.blob.core.windows.net

hello I am using the dataset for research purposes too and i didn't find a complet description of this dataset if you did find something please contact me i will be gratful for you .

Hi,
Since no one commented on this and offered some description from the data provider, it makes sense to do some EDA over time and plot the workload. However one could ask domain experts in Cloud data about this. Please see the following example are categorized as Delay-insensitive when you filtered their vmid in vmtable.csv:

Fig. 1: vmid= '//20EFdlSE3atYr9P03/9X4nF16d9RXI+JKVFfvpC281ohXWjFoS9L+ldKyb3ple'
Fig. 2: vmid= '/KlNtIK2BCkiGhURgesiA/MQNyTpAgt7daSRsu2kJldWyCBGwnZCbtXR3w+vR4kq'

# Filter the VM_CPU_dataframe to find the category of the given vmid
vmid_to_check = '/KlNtIK2BCkiGhURgesiA/MQNyTpAgt7daSRsu2kJldWyCBGwnZCbtXR3w+vR4kq'
filtered_df = df[df['vmid'] == vmid_to_check]

# Display the filtered dataframe
#display(filtered_df)
print(filtered_df.to_markdown(tablefmt="grid"))
+--------+-------------------+------------------------------------------------------------------+
|        | vmcategory        | vmid                                                             |
+========+===================+==================================================================+
| 267924 | Delay-insensitive | /KlNtIK2BCkiGhURgesiA/MQNyTpAgt7daSRsu2kJldWyCBGwnZCbtXR3w+vR4kq |
+--------+-------------------+------------------------------------------------------------------+

# Filter the VM_CPU_dataframe to find the category of the given vmid
vmid_to_check = '//20EFdlSE3atYr9P03/9X4nF16d9RXI+JKVFfvpC281ohXWjFoS9L+ldKyb3ple'
filtered_df = df[df['vmid'] == vmid_to_check]

# Display the filtered dataframe
#display(filtered_df)
print(filtered_df.to_markdown(tablefmt="grid"))
+---------+-------------------+------------------------------------------------------------------+
|         | vmcategory        | vmid                                                             |
+=========+===================+==================================================================+
| 1011821 | Delay-insensitive | //20EFdlSE3atYr9P03/9X4nF16d9RXI+JKVFfvpC281ohXWjFoS9L+ldKyb3ple |
+---------+-------------------+------------------------------------------------------------------+

without domain knowledge and comment of data providers who collect data, it is difficult to reason.
Here in the left picture, I see some delay especially is clear on the 'avgcpu' at the start but not very clear in the right picture. I am also interested to know why they classed this type of VMs not sensitive to delay or delay-insensitive.

Does the Unknowon class translate if it is not Delay-insensitive or interactive?
Classification is labeled based on which factor during data acquisition??


ExifInterface only supports saving attributes on JPEG formats error when running with react-native-camera

Hi, thank you for the great library. I am getting the following error when use with react-native-camera.
Am I missing something? Thank you in advance.
Bug

console.error: "Unhandled promise rejection", {"framesToPop":1,"nativeStackAndroid":[{"methodName":"saveAttributes","lineNumber":1770,"file":"ExifInterface.java"},{"methodName":"copyExif","lineNumber":397,"file":"ImageEditorModule.java"},{"methodName":"access$500","lineNumber":55,"file":"ImageEditorModule.java"},{"methodName":"doInBackgroundGuarded","lineNumber":282,"file":"ImageEditorModule.java"},{"methodName":"doInBackgroundGuarded","lineNumber":202,"file":"ImageEditorModule.java"},{"methodName":"doInBackground","lineNumber":32,"file":"GuardedAsyncTask.java"},{"methodName":"doInBackground","lineNumber":20,"file":"GuardedAsyncTask.java"},{"methodName":"call","lineNumber":333,"file":"AsyncTask.java"},{"methodName":"run","lineNumber":266,"file":"FutureTask.java"},{"methodName":"runWorker","lineNumber":1162,"file":"ThreadPoolExecutor.java"}],"userInfo":null,"code":"EUNSPECIFIED","line":11371,"column":26,"sourceURL":"http://<IP_ADDRESS>:8081/index.delta?platform=android&dev=true&minify=false"}
error
    index.delta?platform=android&dev=true&minify=false:76687:29
<unknown>
    index.delta?platform=android&dev=true&minify=false:207035:38
<unknown>
    index.delta?platform=android&dev=true&minify=false:5993:26
exports
    index.delta?platform=android&dev=true&minify=false:6427:16
<unknown>
    index.delta?platform=android&dev=true&minify=false:5984:25
_callTimer
    index.delta?platform=android&dev=true&minify=false:37010:17
_callImmediatesPass
    index.delta?platform=android&dev=true&minify=false:37046:19
callImmediates
    index.delta?platform=android&dev=true&minify=false:37265:33
callImmediates
    [native code]
__callImmediates
    index.delta?platform=android&dev=true&minify=false:11811:35
<unknown>
    index.delta?platform=android&dev=true&minify=false:11632:34
__guard
    index.delta?platform=android&dev=true&minify=false:11794:15
flushedQueue
    index.delta?platform=android&dev=true&minify=false:11631:21
flushedQueue
    [native code]
invokeCallbackAndReturnFlushedQueue
    [native code]

Environment info
React native info output:
info
  React Native Environment Info:
    System:
      OS: macOS 10.14.5
      CPU: (4) x64 Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz
      Memory: 454.87 MB / 16.00 GB
      Shell: 2.7.1 - /usr/local/bin/fish
    Binaries:
      Node: 10.13.0 - ~/.nvm/versions/node/v10.13.0/bin/node
      Yarn: 1.12.3 - ~/.nvm/versions/node/v10.13.0/bin/yarn
      npm: 6.4.1 - ~/.nvm/versions/node/v10.13.0/bin/npm
      Watchman: 4.9.0 - /usr/local/bin/watchman
    SDKs:
      iOS SDK:
        Platforms: iOS 12.1, macOS 10.14, tvOS 12.1, watchOS 5.1
    IDEs:
      Android Studio: 3.1 AI-173.4819257
      Xcode: 10.1/10B61 - /usr/bin/xcodebuild
    npmGlobalPackages:
      react-native-cli: 2.0.1
      react-native-git-upgrade: 0.2.7

Library version: ^2.0.1
Steps To Reproduce

Install react-native-image-editor and react-native-camera (^2.11.0)
run the code

takePicture = async () => {
    const data = await this.camera.takePictureAsync({ quality: 0.3, base64: false })
    await ImageEditor.cropImage(data.uri, {
      offset: { x: 10, y: 10 },
      size: { width: 10, height: 10 },
    })
}

render() {
    return (
      <View>
        <RNCamera
              ref={ref => {
                this.camera = ref as RNCamera
              }}
              ratio="4:3"
              captureAudio={false}
              type={RNCamera.Constants.Type.back}
          />
          <Button onPress={this.takePicture} title="Snap" />
      </View>
    )
}

Describe what you expected to happen:

can crop image

Reproducible sample code

Sorry my investigation was not enough.
Catch and show the code then it says ExifInterface only supports saving attributes on JPEG formats.
    try {
      const data = await this.camera.takePictureAsync(options)
      const hoge = await ImageEditor.cropImage(data.uri, {
        offset: { x: 10, y: 10 },
        size: { width: 10, height: 10 },
      })
      console.log(hoge)
    } catch (e) {
      console.error(e)
    }

@acro5piano What is data returned by takePictureAsync function?

Thanks for your reply.
I'm reinstalling it, so wait a moment please.

data is
{ uri: 'file:///data/user/0/com.mydomain.app/cache/Camera/29488e04-034f-4613-9108-29b42e6206d6.jpg',
  height: 960,
  width: 1280,
  pictureOrientation: 1,
  deviceOrientation: 1 }

If I commented out the following line, it seems to work correctly.
https://github.com/react-native-community/react-native-image-editor/blob/master/android/src/main/java/com/reactnativecommunity/imageeditor/ImageEditorModule.java#L282
I would like to create a PR to disable copyExif when given params is set to do so.
Is it okay? I know that is not the ideal solution but should be the current workaround.

@acro5piano Can you use your workaround for now? I need to dig deeper to apply the correct fix.

Thank you. I will use patch-package for now.

In the iOS lib, the cropped picture is being saved as a PNG instead of a JPEG.  I have created a pull request for this.  It may possibly solve this issue. #44

Any updates on the topic ?
I'm facing the same issue, kind in a funny way thought: on Android, depending on the area in a picture I crop, I get the problem or not (working fine on IOS for the same jpg image).  I didn't manage to find the patern that define if it will failed or not thought (I use both offset and width/height) while cropping. I'm not sure my comments helps a lot, but well...
Below a screen shot of different situations where I get the error or not. (Of course I  checked the different parameters for cropping, and when it fails, it's only on Android)

const options = {quality: 0.5, base64: true, fixOrientation: true};
const data = await this.camera.takePictureAsync(options);
fiexOrientation: true solved my problem in rotating image.

Are there any news on this? We're getting the error as well but only on few devices. Can't reproduce this yet but we're using react-native-image-resizer as input for cropping as well. An issue were created there for an related topic too but this was already merged in January and we're running the newest version (just updated from 1.2.3 to 1.2.4 now).
Here the related PR: https://github.com/bamlab/react-native-image-resizer/pull/209
Data looks fine... this is from our logs:
Error while cropping image: ExifInterface only supports saving attributes on JPEG formats.
uri: file:///data/user/0/XXX/cache/Camera/5deb3737-0c94-4925-bb7f-695a83215ef2.jpg, width: 252, height: 320
resImageUri: file:///data/user/0/XXX/cache/1599207653198.JPEG, width: 180, height: 320

2nd line (uri: ...) contains the data input for react-native-image-resizer and 3rd line the resized image data returned by react-native-image-resizer.
Is the only solution currently to modify the code itself as mentioned by @johanschram and additionally add the fix for orientation as pointed out by @MebDevelop or is there already a new configuartion option or anything else I may have missed?

react-native-camera is deprecated. Please use react-native-vision-camera instead


[DOC] GIF Images need to be refined

**What kind of issue you're facing in DOC? **
Well, so far I've used some online tool to convert videos into GIFs but that turned to be ineffective. So, I need some clear GIFs for the README file. And yeah, you're free to use any tools you want! ;)

Duplicate of #29


Disabled dropdown fields receive focus

When tabbing through fields on a form, disabled dropdown fields receive focus and expand the dropdown. See http://jsfiddle.net/trigoesrodrigo/97wbogct/1
However I can't replicate the issue in the docs, the disabled dropdowns behave properly.

Thanks! Helped me catch a typo.  Will be fixed in patch. Disabled fields will still receive keyboard focus but will not open.


[suggestion] Consider making XrpcClient::auth return an owned value

I've written a quick example client to learn about the API. One feature I wanted for my client was to use interior mutability so that the client could refresh its tokens transparently to the consumer, even behind an immutable reference. As you can see, I used a mutex for this - but the fact that auth returns Option<&str> made that pretty difficult. It would also be difficult if you wanted to, for instance, retrieve a token from the environment dynamically in the auth method. My app runs against a branch in my fork that makes the simple change of returning Option<String> instead. You mentioned you were in the middle of refactoring this trait so I thought I'd make this suggestion now.
Thanks for all the work on this! It's been fun to use.

Thanks for the suggestion! I agree exactly, and was just about to change it to Option<String> in the XrpcClient trait update on the branch I'm working on.
https://github.com/sugyan/atrium/commit/f7babd271059c1a9bd7b19998d6972839be220ae#diff-5847c85722dda4e0abdff0065313ae9cd4a2f6a167afda85cb90fa36a4fd3e3aL24-R23
(Other than that, I'm trying to move the dependencies that use serde_json and serde_qs from atrium-api and consolidate them into atrium-xrpc.)


Service  invoke  timeout is  not enought

When I tested 5 times, i  see  dubbo service invoke timeout  is about  10011 , so i suggest it can be extended.

please use the latest version . If you have other questions, please reopen it.


Update test to check codec tag against known FATE suite sample

Updated the test to check against a FATE suite sample as suggested in #750

Fantastic, thanks for this!


Update signalfx-json type to signalfx to avoid confusion

Clean up your branch's commit history

will do. Let me create a new PR


Fix using proper env for platform sidecar statsd

Refer doc: https://hello.atlassian.net/wiki/spaces/MICROS/blog/2022/08/15/1830221772/PSA+We+re+changing+how+platform+sidecar+addresses+are+derived
What's in this PR?
Change to use ENV vars
Why
Because the fix hostname is gonna be deprecated
Added feature flags
N/A
Affected issues
N/A
How has this been tested?
Stage only
Deployed on stage

Traffic still coming through

Whats Next?

Has it been tested in stg?

Do we need a similar change for cryptor?

Has it been tested in stg?

Yup that's the screen shot on description

Do we need a similar change for cryptor?

May or may not. But cryptor is more explicitly declared, so I guess it is okay.


[Source Connector] Timestamp-based Querying Gets Stuck in Postgres

In Postgres, CURRENT_TIMESTAMP (and others) change behavior within a
transaction:

These SQL-standard functions all return values based on the start time
of the current transaction

(see 1)
In 29581b310781ccc52fa0eec3a00e47da91bac767, the JDBC connection for the
source connector had auto-commit disabled, apparently to force cursor
behavior in some JDBC drivers.  When auto-commit is disabled, a
transaction is opened and not committed until commit is called -- and
it's only called in the sink connector.
Because the generic dialect uses CURRENT_TIMESTAMP as its timestamp
query, and the Postgres dialect has no override, the end timestamp used
in timestamp- and timestamp+incrementing-mode is calculated using the
timestamp of the transaction start.  The transaction was started when
the connector started up, so this timestamp always reads the same.
Here are some log lines from one of our production connectors illustrating this.  I have added annotations and omitted days of logs:
[2020-09-29 22:09:01,019] DEBUG Executing prepared statement with timestamp value = 2020-09-25 19:59:12.238 end time = 2020-09-26 23:07:56.308 (io.aiven.connect.jdbc.source.TimestampIncrementingCriteria:170)
[2020-09-29 22:24:01,035] DEBUG Executing prepared statement with timestamp value = 2020-09-25 19:59:12.238 end time = 2020-09-26 23:07:56.308 (io.aiven.connect.jdbc.source.TimestampIncrementingCriteria:170)
[2020-09-29 22:39:01,047] DEBUG Executing prepared statement with timestamp value = 2020-09-25 19:59:12.238 end time = 2020-09-26 23:07:56.308 (io.aiven.connect.jdbc.source.TimestampIncrementingCriteria:170)
[2020-09-29 22:54:01,072] DEBUG Executing prepared statement with timestamp value = 2020-09-25 19:59:12.238 end time = 2020-09-26 23:07:56.308 (io.aiven.connect.jdbc.source.TimestampIncrementingCriteria:170)
--- connector restarted ---
[2020-09-29 22:54:59,136] DEBUG Executing prepared statement with timestamp value = 2020-09-25 19:59:12.238 end time = 2020-09-29 22:53:59.135 (io.aiven.connect.jdbc.source.TimestampIncrementingCriteria:170)
[2020-09-29 23:09:59,531] DEBUG Executing prepared statement with timestamp value = 2020-09-28 19:55:07.147 end time = 2020-09-29 22:53:59.135 (io.aiven.connect.jdbc.source.TimestampIncrementingCriteria:170)
[2020-09-29 23:24:59,545] DEBUG Executing prepared statement with timestamp value = 2020-09-28 19:55:07.147 end time = 2020-09-29 22:53:59.135 (io.aiven.connect.jdbc.source.TimestampIncrementingCriteria:170)
[2020-09-29 23:39:59,561] DEBUG Executing prepared statement with timestamp value = 2020-09-28 19:55:07.147 end time = 2020-09-29 22:53:59.135 (io.aiven.connect.jdbc.source.TimestampIncrementingCriteria:170)
[2020-09-29 23:54:59,574] DEBUG Executing prepared statement with timestamp value = 2020-09-28 19:55:07.147 end time = 2020-09-29 22:53:59.135 (io.aiven.connect.jdbc.source.TimestampIncrementingCriteria:170)
[2020-09-30 00:09:59,586] DEBUG Executing prepared statement with timestamp value = 2020-09-28 19:55:07.147 end time = 2020-09-29 22:53:59.135 (io.aiven.connect.jdbc.source.TimestampIncrementingCriteria:170)
[2020-09-30 00:24:59,609] DEBUG Executing prepared statement with timestamp value = 2020-09-28 19:55:07.147 end time = 2020-09-29 22:53:59.135 (io.aiven.connect.jdbc.source.TimestampIncrementingCriteria:170)
[2020-09-30 00:39:59,673] DEBUG Executing prepared statement with timestamp value = 2020-09-28 19:55:07.147 end time = 2020-09-29 22:53:59.135 (io.aiven.connect.jdbc.source.TimestampIncrementingCriteria:170)
...
[2020-10-01 23:55:04,836] DEBUG Executing prepared statement with timestamp value = 2020-09-28 19:55:07.147 end time = 2020-09-29 22:53:59.135 (io.aiven.connect.jdbc.source.TimestampIncrementingCriteria:170)
[2020-10-02 00:10:04,885] DEBUG Executing prepared statement with timestamp value = 2020-09-28 19:55:07.147 end time = 2020-09-29 22:53:59.135 (io.aiven.connect.jdbc.source.TimestampIncrementingCriteria:170)
[2020-10-02 00:25:04,958] DEBUG Executing prepared statement with timestamp value = 2020-09-28 19:55:07.147 end time = 2020-09-29 22:53:59.135 (io.aiven.connect.jdbc.source.TimestampIncrementingCriteria:170)
[2020-10-02 00:40:05,139] DEBUG Executing prepared statement with timestamp value = 2020-09-28 19:55:07.147 end time = 2020-09-29 22:53:59.135 (io.aiven.connect.jdbc.source.TimestampIncrementingCriteria:170)
[2020-10-02 00:55:05,184] DEBUG Executing prepared statement with timestamp value = 2020-09-28 19:55:07.147 end time = 2020-09-29 22:53:59.135 (io.aiven.connect.jdbc.source.TimestampIncrementingCriteria:170)
[2020-10-02 01:10:05,360] DEBUG Executing prepared statement with timestamp value = 2020-09-28 19:55:07.147 end time = 2020-09-29 22:53:59.135 (io.aiven.connect.jdbc.source.TimestampIncrementingCriteria:170)
[2020-10-02 01:25:05,415] DEBUG Executing prepared statement with timestamp value = 2020-09-28 19:55:07.147 end time = 2020-09-29 22:53:59.135 (io.aiven.connect.jdbc.source.TimestampIncrementingCriteria:170)
[2020-10-02 01:40:05,446] DEBUG Executing prepared statement with timestamp value = 2020-09-28 19:55:07.147 end time = 2020-09-29 22:53:59.135 (io.aiven.connect.jdbc.source.TimestampIncrementingCriteria:170)
[2020-10-02 01:55:05,481] DEBUG Executing prepared statement with timestamp value = 2020-09-28 19:55:07.147 end time = 2020-09-29 22:53:59.135 (io.aiven.connect.jdbc.source.TimestampIncrementingCriteria:170)

In effect, the connector must be continually restarted for
timestamp-based querying to function in Postgres.
There are a couple of possible ways to work around this:

turn auto-commit back on just for postgres (not sure the ideal place
to do this, or if it's too big of a change)
explicitly commit after each statement
make the postgres time query return the actual "current time": PR #55

Hey @NJAldwin
Thank you for the issue report and the PR, I'm looking into them!

Once again, thank you for raising this concern.

turn auto-commit back on just for postgres

This is not possible: Postgres was the reason to introduce it in the first place.

make the postgres time query return the actual "current time"

This will work and would be the easiest fix.
However, there is more to this than stale timestamps. Long-running (effectively, indefinitely running in this case) transactions are generally a bad idea. So

explicitly commit after each statement

will be the way to go. The fix is incoming.

Made a PR https://github.com/aiven/aiven-kafka-connect-jdbc/pull/57


Fix spelling and Grammer issue

Thanks, there's another PR that addresses the same issues so merging that in.


2024 Feb Updates: Bug and Typo Fixes

Overall Review of Changes:
Bug and Typo Fixes
Issue Fixes:
#88, #89, #90, #91, #92 - Thank you @animatco
#94, #96  - Thank you @dennisharder-alight
Enhancements:
N/A
How has this been tested?:
Locally

Thank you @MrSteve81 !


Fix | Fix SQLServerConnection.abort() API behavior to clear resources consistently

As per Java Specifications, executing Connection.abort(Executor executor), must also release all resources and mark the connection closed as Connection.close() would.
The PR fixes the same by managing resource cleanup separately in a common internal method.

Codecov Report

:exclamation: No coverage uploaded for pull request base (dev@78efb57). Click here to learn what that means.
The diff coverage is 100%.

@@          Coverage Diff           @@
##             dev     #983   +/-   ##
======================================
  Coverage       ?   50.57%
  Complexity     ?     2920
======================================
  Files          ?      120
  Lines          ?    27997
  Branches       ?     4679
======================================
  Hits           ?    14159
  Misses         ?    11546
  Partials       ?     2292

Flag
Coverage Δ
Complexity Δ

#JDBC42
50.07% <100%> (?)
2878 <1> (?)

#JDBC43
50.43% <100%> (?)
2912 <1> (?)

Impacted Files
Coverage Δ
Complexity Δ

.../microsoft/sqlserver/jdbc/SQLServerConnection.java
46.49% <100%> (ø)
343 <1> (?)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 78efb57...1007712. Read the comment docs.


Make intrinsic Gateway LoA configurable

Hardcoded LOA identifier in gateway code
https://www.pivotaltracker.com/story/show/90133820

@DRvanR

:+1:


unfound reference: mx.io.ImageRecordIter

Description
after update to the latest version 1.3 mxnet in python 3.6, the IDE catch error as
' Cannot find reference "ImageRecordIter" in "io.py"
when I try to
# construct the training image iterator
import mxnet as mx
trainIter = mx.io.ImageRecordIter(
    path_imgrec=config.Train_MX_REC,
    data_shape=(3,224,224)
    batch_size=batchSize,...)

Environment info
python 3.6
mxnet 1. 3. 0b20
IDE: latest pycharm community

Was that the exact script you used? Seems like you're missing a comma after
    data_shape=(3,224,224)

BTW the doc for ImageRecordIter is at https://mxnet.incubator.apache.org/api/python/io/io.html?highlight=imagerecorditer#mxnet.io.ImageRecordIter. Docs for all Python APIs can be found at https://mxnet.incubator.apache.org/api/python/index.html.

BTW the doc for ImageRecordIter is at https://mxnet.incubator.apache.org/api/python/io/io.html?highlight=imagerecorditer#mxnet.io.ImageRecordIter. Docs for all Python APIs can be found at https://mxnet.incubator.apache.org/api/python/index.html.

BTW the doc for ImageRecordIter is at https://mxnet.incubator.apache.org/api/python/io/io.html?highlight=imagerecorditer#mxnet.io.ImageRecordIter. Docs for all Python APIs can be found at https://mxnet.incubator.apache.org/api/python/index.html.

fix it by simply restarting pycharm.... thanks guys


Add missing includes

Adding a bunch of missing includes.
These were spotted while updating the FreeBSD port.
What type of PR is this? (check all applicable)

[ ] Refactor
[ ] Feature
[ ] Bug Fix
[ ] Optimization
[ ] Documentation Update
[x] Other: Fix build

Description
No user facing changes. Just fixing the build.
Related Issues & Documents
N/A
Documentation (check all applicable)

[ ] This PR requires changes to the Wiki documentation (describe the changes)
[ ] This PR requires changes to the documentation inside the git repo (please add them to the PR).
[x] Does not require documentation changes

I added the missing header in polybar/i3ipcpp#16 and #3052


What's the function of legacy in pillar encoder ?

Hi, I want to know what's the function of legacy in https://github.com/open-mmlab/mmdetection3d/blob/master/mmdet3d/models/voxel_encoders/pillar_encoder.py#L116 ?
Legacy is set True in default, so we will run https://github.com/open-mmlab/mmdetection3d/blob/master/mmdet3d/models/voxel_encoders/pillar_encoder.py#L128:
              f_center = features[:, :, :3]
              f_center[:, :, 0] = f_center[:, :, 0] - (
                  coors[:, 3].type_as(features).unsqueeze(1) * self.vx +
                  self.x_offset)
              f_center[:, :, 1] = f_center[:, :, 1] - (
                  coors[:, 2].type_as(features).unsqueeze(1) * self.vy +
                  self.y_offset)
              f_center[:, :, 2] = f_center[:, :, 2] - (
                  coors[:, 1].type_as(features).unsqueeze(1) * self.vz +
                  self.z_offset)

Here, I find raw features[..., :3] has been changed and features[..., :3] == features[..., -3:], is there a bug ?
Looking forward to your reply.

        legacy (bool, optional): Whether to use the new behavior or             the original behavior. Defaults to True.
Turning on the legacy option actually violates the original paper implementation, because we do not want to make inplace operations while computing the center features. It is a little interesting that previous experiments seem to show that setting legacy to True can yield better performance, you can regard it as a training trick.

Thanks for your quick reply.
I'm still confused. As you mentioned above, inplace operations is an empirical trick to yield better performance.
However, it can cause data redundancy between the first 3 dimensions and the last 3 dimension in the encoded pillar features(10 dimensions), as shown in the following figure.

Here, I debuged an older mmdet3d, where the encoded pillar features are in 9 dimensions, and the first 2 dimensions and the last 2 dimension are the same.  Then, my questions are:

The raw point coordinate information (the first 2 or 3 dimensions) is lost and replaced with relative coordinate. Will it affect the features ?
Why don't you use features[:-2] or features[:-3] instead of features ?

I'm a beginner to 3d detection, and sorry to bother you.
Looking forward to your reply.

You can refer to https://github.com/open-mmlab/mmdetection3d/issues/1150 , maybe it would be helpful.


Websocket client closing handshake to support "force close" after given timetout

RFC455 defines that, generally, a WebSocket client should not close a TCP connection as far as a server is the one who's responsible for doing that. In practice, when I do not control the server, this might lead to connections leak if the server does not comply with the RFC.
RFC455#7.1.1 says

In abnormal cases (such as not having received a TCP Close from the server after a reasonable amount of time) a client MAY initiate the TCP Close.

I suggest to add forceCloseAfterMillis option to WebSocketClientHandshaker (and to WebSocketClientProtocolHandler appropriately) and schedule a channel close when necessary after CloseWebSocketFrame being flushed here. If there's no objections, I'm happy to provide a PR.

@kachayev this sounds good... Its also inline with what we do in SslHandler.

@normanmaurer I'll do a PR for WebSockets first, as I know this code quite well. Will check SslHandler later ;)

Sorry maybe I was not clear here... we have this kind of code already in SslHandler. I just wanted to mention it as it may gives you some good examples

Ah, okay! Now it's clear. I thought you meant that we need to do the same for SSL. Thanks!


All my terminals are gone and I can't get rid of the defaults

Does this issue occur when all extensions are disabled?: Yes/No

VS Code Version: 1.60.0
OS Version: Windows 10

Steps to Reproduce:

Install VS Code Update
Ctrl + Shift + `  to Open Terminal
Terminal dropdown is only "PowerShell", "Command Prompt" and "Javascript Debug Terminal". No idea where these came from but my git-bash is gone.
Clicking "Configure Terminal Settings" just gives the useless UI Settins about Automation Shells and whatever. Editing settings.json doesn't show any of the above choices (I don't want those). I can't get rid of those and I can't add new ones.
Only recourse is uninstall and change career.

Same issue here. Same windows and VS Code Version as @promethyttrium
For more info, I have put below the image of my settings, and the terminal error resulting:

"JavaScript Debug Terminal" ignores every default profile setting (new and deprecated) and always uses PowerShell on Windows now

try add new and set  like this

Same for me. I change default back to Command Prompt but on opening a new window, VS Code starts a PowerShell terminal. This is extremely annoying and is breaking all workflows at the moment.

will be fixed in insider's tomorrow and in the recovery build this week /duplicate #132150


Bump package.json, Fix code to work with next-gen.

Bump package.json.
Fix code to work with next-gen.

Please @joliveros make a tiny review when you can...

I don't want to be picky but naming the branch next-gen would be enough. IMHO there is no need to use a large name like next-gen-marketing-notifications-node. The context will be inferred from the repo itself. What do you think?

Sorry, the "Fix code to work again" gets inherited on the title from the first commit and I didnt notice, was wrong basically, I fixed the title...


Archer C60 Cannot authorize! Error

The Update worked for a short time but now:
TplinkRouter - C1200 - Cannot authorize! Error - 'NoneType' object has no attribute 'get'; Response - {"errorcode":"user conflict","success":false}
Is there any solution of this problem?

I loadet the integration new, and now it works again. Is it a good idea to make in future no updates. Never touch a running system?

It always requires updates as HA changes and it needs to update for compatibility


Cookbook is incompatible with chef client 13

The underlying cause is the same as #16 but reporting it now because it is no longer just a warning that can be ignored. This is what happens:
ArgumentError
Property params of resource chrome overwrites an existing method.
Cookbook Trace:
/var/cache/chef/cookbooks/chrome/resources/default.rb:8:in `class_from_file'
Relevant File Content:
/var/cache/chef/cookbooks/chrome/resources/default.rb:
1:  actions :master_preferences
2:  default_action :master_preferences
3:
4:  attribute :name, kind_of: String, name_attribute: true
5:  attribute :template, kind_of: String, default: 'master_preferences.json.erb'
6:  attribute :local, kind_of: [TrueClass, FalseClass], default: false
7:  attribute :cookbook, kind_of: String, default: nil
8>> attribute :params, kind_of: [Array, Hash], default: []
9:

Released v4.0.0

we are having  same issue when upgrading to the version 13. can we know how did you resolve it.
we have same.
attribute :params, kind_of: [Array, Hash], default: []

Renamed params to parameters: https://github.com/dhoer/chef-chrome/commit/6f8efe8093cd7a40262b557d40a79c2b7d95320c


automatically generate mesos auth flags

PR for #115.
This also adds a task in security-setup to deprecate values that had previously been set. I'd like feedback on that, especially from @stevendborrelli and @keithchambers, but also from anyone who cares about it.

This looks good to me!
@BrianHicks are you ready to merge?
@stevendborrelli are you good with these changes?

I think it's ready, unless Steve has any additions.


I can't get to the amazon log in page through my browser...  Keep getting a http...  I change this to HTTPS

I can't authenticate login on browser.  It says to go to app but i click on the top to proceed to web address.  It then takes me to a http// address.  I switch this to http:// and then get the last screen shot of a 404

Logs:
Show the Homebridge logs here, remove any sensitive information.

Plugin Config:
Show your Homebridge config.json here, remove any sensitive information.

Screenshots:

Environment:

Plugin Version: 2.2 Beta 3
(https://github.com/joeyhage/homebridge-alexa-smarthome/assets/144946144/ae20b6f6-b216-47c2-bac1-8b0aa3622b24)

Homebridge Version:  1.6.1

Node.js Version:  18.17.1

NPM Version:

Operating System:  docker

What URL do you use to manage your homebridge plugins and can you please share the config you are using for this plugin?

I use <IP_ADDRESS>:4444 for homebridge

You need to remove the http:// from clientHost

That made progress...
Now i get this

Just visiting <IP_ADDRESS>:8827 and try again. If that still doesn’t work, please enable 2FA on your Amazon account

That worked thank you!

You ROck


Add ARIA attributes to create blog form

Description
ARIA attributes are crucial for enhancing web application accessibility, benefiting users with disabilities. Currently, our form lack these attributes, hindering proper interpretation by assistive technologies.
Some steps to implement this enhancement-

Identify the form elements in your codebase.
Determine the relevant ARIA attributes to add for each form element.
Add ARIA roles to the forms and their elements (e.g., "form" role for the main form element).
Include concise and descriptive ARIA labels using the aria-label attribute.
Use the aria-describedby attribute to provide additional instructions for the forms.
Implement ARIA states and properties (e.g., aria-required for required fields).
Test the accessibility improvements using assistive technologies.

Please assign me this issue so that I can work on this issue under GSSOC'23.

@Avinash905 go ahead