Only part of photos scanned?!

🐛 Bug Report

[x ] I have looked for similar issues among all the former issues including closed ones.

What Operating system and version is LibrePhotos running on:
I just did a clean install on a Raspberry Pi4 with 8GB RAM. After some initial struggles, it eventually started up and I could set a directory with a few thousand photos (and videos). The initial scan was obviously slow, it took around 24 hours until the system was "responsive" again.
What architecture is LibrePhotos running on:
ARM
How is LibrePhotos installed:
Docker
If running via Docker or Kubernetes please list version including docker-compose:
ubuntu@docker:~$ docker version
Client:
 Version:           20.10.7
 API version:       1.41
 Go version:        go1.13.8
 Git commit:        20.10.7-0ubuntu1~20.04.1
 Built:             Wed Aug  4 22:53:01 2021
 OS/Arch:           linux/arm64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.7
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.8
  Git commit:       20.10.7-0ubuntu1~20.04.1
  Built:            Wed Aug  4 19:07:47 2021
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.5.2-0ubuntu1~20.04.2
  GitCommit:
 runc:
  Version:          1.0.0~rc95-0ubuntu1~20.04.2
  GitCommit:
 docker-init:
  Version:          0.19.0
  GitCommit:

How is you picture library mounted on the host (or in the virtual machine):
Local file system (Type), NFS, or SMB
CIFS
Description of issue:
The defined Scan Directory has almost 2,300 items according to Windows Explorer, with 5 sub-directories containing a few more thousand photos (and videos). The library in Librephotos shows only 504 photos. I understand that some items might be no photos, but there are 1,877 JPG-files in the root directory alone. How come it does not include all the photos in the scan?

(Posting this in #308)
I have had issues with getting photos to all scan (I had first added it as part of the reply to another issue (no photos show up under places #305) as scanning would never complete.  I will also update #305 to keep things together there.
I have tried remove as many layers as possible in the docker setup by moving the test to a Debian9 VM (set with 4vcpu 4GB RAM) as opposed to MacOS which is just using a hidden Linux VM.
NOW I can see an error in the ownphotos.log where before I could not.
---- HERE IS THE COMPLETE LOG FROM A CLEAN TRY ----
021-08-30 12:13:17,086 : views.py : list : 515 : INFO : querying & serializing took 0.01 seconds
2021-08-30 12:13:27,271 : serializers.py : update : 720 : INFO : Updated scan directory for user /data
2021-08-30 12:13:27,273 : serializers.py : update : 726 : INFO : Updated confidence for user 0.1
2021-08-30 12:13:27,274 : serializers.py : update : 736 : INFO : Updated semantic_search_topk for user 0
2021-08-30 12:13:27,275 : serializers.py : update : 742 : INFO : Updated favorite_min_rating for user 4
2021-08-30 12:13:43,409 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_3513.jpg
2021-08-30 12:13:45,702 : photo.py : _generate_captions : 143 : INFO : generated places365 captions for image /protected_media/thumbnails_big/dbd2bd73b451e048584f3082589e2ca71.webp.
2021-08-30 12:13:52,303 : photo.py : _generate_clip_embeddings : 117 : INFO : generated clip embeddings for image /protected_media/thumbnails_big/dbd2bd73b451e048584f3082589e2ca71.webp.
2021-08-30 12:13:53,592 : util.py : mapbox_reverse_geocode : 97 : INFO : mapbox returned status 200.
2021-08-30 12:13:54,889 : photo.py : _extract_faces : 362 : INFO : image dbd2bd73b451e048584f3082589e2ca71: 2 face(s) saved
2021-08-30 12:13:54,891 : directory_watcher.py : handle_new_image : 114 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: image processed: /data/IMG_3513.jpg, elapsed: 11.455016
2021-08-30 12:13:54,906 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_3065.jpg
2021-08-30 12:13:56,146 : photo.py : _generate_captions : 143 : INFO : generated places365 captions for image /protected_media/thumbnails_big/13f966e0d6ab83b3657878097a3ce1bc1.webp.
2021-08-30 12:14:03,017 : photo.py : _generate_clip_embeddings : 117 : INFO : generated clip embeddings for image /protected_media/thumbnails_big/13f966e0d6ab83b3657878097a3ce1bc1.webp.
2021-08-30 12:14:03,049 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_3065.jpg. reason: [Errno 12] Cannot allocate memory
Traceback (most recent call last):
File "/code/api/directory_watcher.py", line 109, in handle_new_image
photo._extract_date_time_from_exif(True)
File "/code/api/models/photo.py", line 211, in _extract_date_time_from_exif
with exiftool.ExifTool() as et:
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter
self.start()
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start
raise oe
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start
self._process = subprocess.Popen(
File "/usr/lib/python3.9/subprocess.py", line 951, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child
self.pid = _posixsubprocess.fork_exec(
OSError: [Errno 12] Cannot allocate memory
2021-08-30 12:14:03,240 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_9411.HEIC.heif
2021-08-30 12:14:05,848 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_9411.HEIC.heif. reason: [Errno 12] Cannot allocate memory
Traceback (most recent call last):
File "/code/api/directory_watcher.py", line 106, in handle_new_image
photo._calculate_aspect_ratio(False)
File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio
with exiftool.ExifTool() as et:
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter
self.start()
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start
raise oe
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start
self._process = subprocess.Popen(
File "/usr/lib/python3.9/subprocess.py", line 951, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child
self.pid = _posixsubprocess.fork_exec(
OSError: [Errno 12] Cannot allocate memory
2021-08-30 12:14:05,913 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_2143.HEIC.heif
2021-08-30 12:14:08,147 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_2143.HEIC.heif. reason: [Errno 12] Cannot allocate memory
Traceback (most recent call last):
File "/code/api/directory_watcher.py", line 106, in handle_new_image
photo._calculate_aspect_ratio(False)
File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio
with exiftool.ExifTool() as et:
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter
self.start()
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start
raise oe
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start
self._process = subprocess.Popen(
File "/usr/lib/python3.9/subprocess.py", line 951, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child
self.pid = _posixsubprocess.fork_exec(
OSError: [Errno 12] Cannot allocate memory
2021-08-30 12:14:08,202 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/567712203.458368public.jpg
2021-08-30 12:14:08,713 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/567712203.458368public.jpg. reason: [Errno 12] Cannot allocate memory
Traceback (most recent call last):
File "/code/api/directory_watcher.py", line 106, in handle_new_image
photo._calculate_aspect_ratio(False)
File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio
with exiftool.ExifTool() as et:
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter
self.start()
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start
raise oe
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start
self._process = subprocess.Popen(
File "/usr/lib/python3.9/subprocess.py", line 951, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child
self.pid = _posixsubprocess.fork_exec(
OSError: [Errno 12] Cannot allocate memory
2021-08-30 12:14:08,836 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_2386.heic
2021-08-30 12:14:11,886 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_2386.heic. reason: [Errno 12] Cannot allocate memory
Traceback (most recent call last):
File "/code/api/directory_watcher.py", line 106, in handle_new_image
photo._calculate_aspect_ratio(False)
File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio
with exiftool.ExifTool() as et:
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter
self.start()
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start
raise oe
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start
self._process = subprocess.Popen(
File "/usr/lib/python3.9/subprocess.py", line 951, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child
self.pid = _posixsubprocess.fork_exec(
OSError: [Errno 12] Cannot allocate memory
2021-08-30 12:14:11,952 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_8176.jpg
2021-08-30 12:14:12,717 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_8176.jpg. reason: [Errno 12] Cannot allocate memory
Traceback (most recent call last):
File "/code/api/directory_watcher.py", line 106, in handle_new_image
photo._calculate_aspect_ratio(False)
File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio
with exiftool.ExifTool() as et:
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter
self.start()
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start
raise oe
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start
self._process = subprocess.Popen(
File "/usr/lib/python3.9/subprocess.py", line 951, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child
self.pid = _posixsubprocess.fork_exec(
OSError: [Errno 12] Cannot allocate memory
2021-08-30 12:14:12,778 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_5830.heic
2021-08-30 12:14:14,758 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_5830.heic. reason: [Errno 12] Cannot allocate memory
Traceback (most recent call last):
File "/code/api/directory_watcher.py", line 106, in handle_new_image
photo._calculate_aspect_ratio(False)
File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio
with exiftool.ExifTool() as et:
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter
self.start()
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start
raise oe
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start
self._process = subprocess.Popen(
File "/usr/lib/python3.9/subprocess.py", line 951, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child
self.pid = _posixsubprocess.fork_exec(
OSError: [Errno 12] Cannot allocate memory
2021-08-30 12:14:14,826 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_7527.heic
2021-08-30 12:14:17,863 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_7527.heic. reason: [Errno 12] Cannot allocate memory
Traceback (most recent call last):
File "/code/api/directory_watcher.py", line 106, in handle_new_image
photo._calculate_aspect_ratio(False)
File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio
with exiftool.ExifTool() as et:
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter
self.start()
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start
raise oe
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start
self._process = subprocess.Popen(
File "/usr/lib/python3.9/subprocess.py", line 951, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child
self.pid = _posixsubprocess.fork_exec(
OSError: [Errno 12] Cannot allocate memory
2021-08-30 12:14:17,922 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_0224.jpg
2021-08-30 12:14:18,552 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_0224.jpg. reason: [Errno 12] Cannot allocate memory
Traceback (most recent call last):
File "/code/api/directory_watcher.py", line 106, in handle_new_image
photo._calculate_aspect_ratio(False)
File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio
with exiftool.ExifTool() as et:
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter
self.start()
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start
raise oe
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start
self._process = subprocess.Popen(
File "/usr/lib/python3.9/subprocess.py", line 951, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child
self.pid = _posixsubprocess.fork_exec(
OSError: [Errno 12] Cannot allocate memory
2021-08-30 12:14:18,627 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_5896.heic
2021-08-30 12:14:21,493 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_5896.heic. reason: [Errno 12] Cannot allocate memory
Traceback (most recent call last):
File "/code/api/directory_watcher.py", line 106, in handle_new_image
photo._calculate_aspect_ratio(False)
File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio
with exiftool.ExifTool() as et:
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter
self.start()
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start
raise oe
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start
self._process = subprocess.Popen(
File "/usr/lib/python3.9/subprocess.py", line 951, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child
self.pid = _posixsubprocess.fork_exec(
OSError: [Errno 12] Cannot allocate memory
2021-08-30 12:14:21,512 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/WP_20151218_13_30_28_Raw__highres.dng
2021-08-30 12:19:54,456 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_3297.heic
2021-08-30 12:19:58,838 : photo.py : _generate_captions : 143 : INFO : generated places365 captions for image /protected_media/thumbnails_big/c8c4c21ce392750f27807efc3e098bcd1.webp.
2021-08-30 12:20:05,393 : photo.py : _generate_clip_embeddings : 117 : INFO : generated clip embeddings for image /protected_media/thumbnails_big/c8c4c21ce392750f27807efc3e098bcd1.webp.
2021-08-30 12:20:06,945 : util.py : mapbox_reverse_geocode : 97 : INFO : mapbox returned status 200.
2021-08-30 12:20:08,988 : photo.py : _extract_faces : 362 : INFO : image c8c4c21ce392750f27807efc3e098bcd1: 1 face(s) saved
2021-08-30 12:20:08,990 : directory_watcher.py : handle_new_image : 114 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: image processed: /data/IMG_3297.heic, elapsed: 14.474443
2021-08-30 12:20:09,022 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/WP_20171127_21_17_10_Pro__highres.jpg
2021-08-30 12:20:10,789 : photo.py : _generate_captions : 143 : INFO : generated places365 captions for image /protected_media/thumbnails_big/576e7b15b9ca9a795945f70f2d6336c31.webp.
2021-08-30 12:20:16,222 : photo.py : _generate_clip_embeddings : 117 : INFO : generated clip embeddings for image /protected_media/thumbnails_big/576e7b15b9ca9a795945f70f2d6336c31.webp.
2021-08-30 12:20:17,072 : util.py : mapbox_reverse_geocode : 97 : INFO : mapbox returned status 200.
2021-08-30 12:20:19,459 : photo.py : _extract_faces : 362 : INFO : image 576e7b15b9ca9a795945f70f2d6336c31: 1 face(s) saved
2021-08-30 12:20:19,460 : directory_watcher.py : handle_new_image : 114 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: image processed: /data/WP_20171127_21_17_10_Pro__highres.jpg, elapsed: 10.394164
2021-08-30 12:20:19,516 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_6338.heic
2021-08-30 12:20:22,407 : photo.py : _generate_captions : 143 : INFO : generated places365 captions for image /protected_media/thumbnails_big/a7569195075a165f4813dc3e6dc2b93a1.webp.
2021-08-30 12:20:27,582 : photo.py : _generate_clip_embeddings : 117 : INFO : generated clip embeddings for image /protected_media/thumbnails_big/a7569195075a165f4813dc3e6dc2b93a1.webp.
2021-08-30 12:20:28,813 : util.py : mapbox_reverse_geocode : 97 : INFO : mapbox returned status 200.
2021-08-30 12:20:29,922 : photo.py : _extract_faces : 362 : INFO : image a7569195075a165f4813dc3e6dc2b93a1: 1 face(s) saved
2021-08-30 12:20:29,923 : directory_watcher.py : handle_new_image : 114 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: image processed: /data/IMG_6338.heic, elapsed: 10.397034
2021-08-30 12:20:29,941 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/WP_20160823_19_25_11_Raw__highres.dng
2021-08-30 12:21:37,015 : photo.py : _generate_captions : 143 : INFO : generated places365 captions for image /protected_media/thumbnails_big/6e3aa849ab5f116973f9220a42f956ff1.webp.
2021-08-30 12:21:43,208 : photo.py : _generate_clip_embeddings : 117 : INFO : generated clip embeddings for image /protected_media/thumbnails_big/6e3aa849ab5f116973f9220a42f956ff1.webp.
2021-08-30 12:21:46,049 : directory_watcher.py : handle_new_image : 114 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: image processed: /data/WP_20160823_19_25_11_Raw__highres.dng, elapsed: 75.926134
2021-08-30 12:21:46,087 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_0404.jpg
2021-08-30 12:21:46,866 : photo.py : _generate_captions : 143 : INFO : generated places365 captions for image /protected_media/thumbnails_big/4f4e6ff4bc394df5d5222216e3fb28401.webp.
2021-08-30 12:21:52,186 : photo.py : _generate_clip_embeddings : 117 : INFO : generated clip embeddings for image /protected_media/thumbnails_big/4f4e6ff4bc394df5d5222216e3fb28401.webp.
2021-08-30 12:21:53,849 : photo.py : _extract_faces : 362 : INFO : image 4f4e6ff4bc394df5d5222216e3fb28401: 1 face(s) saved
2021-08-30 12:21:53,850 : directory_watcher.py : handle_new_image : 114 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: image processed: /data/IMG_0404.jpg, elapsed: 7.750198
2021-08-30 12:21:53,935 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_5584.heic
2021-08-30 12:21:56,280 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_5584.heic. reason: [Errno 12] Cannot allocate memory
Traceback (most recent call last):
File "/code/api/directory_watcher.py", line 106, in handle_new_image
photo._calculate_aspect_ratio(False)
File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio
with exiftool.ExifTool() as et:
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter
self.start()
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start
raise oe
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start
self._process = subprocess.Popen(
File "/usr/lib/python3.9/subprocess.py", line 951, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child
self.pid = _posixsubprocess.fork_exec(
OSError: [Errno 12] Cannot allocate memory
2021-08-30 12:21:56,340 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/SarverHerringWedding-72.jpg
2021-08-30 12:21:56,946 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/SarverHerringWedding-72.jpg. reason: [Errno 12] Cannot allocate memory
Traceback (most recent call last):
File "/code/api/directory_watcher.py", line 106, in handle_new_image
photo._calculate_aspect_ratio(False)
File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio
with exiftool.ExifTool() as et:
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter
self.start()
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start
raise oe
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start
self._process = subprocess.Popen(
File "/usr/lib/python3.9/subprocess.py", line 951, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child
self.pid = _posixsubprocess.fork_exec(
OSError: [Errno 12] Cannot allocate memory
2021-08-30 12:21:57,011 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_6369.heic
2021-08-30 12:21:58,906 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_6369.heic. reason: [Errno 12] Cannot allocate memory
Traceback (most recent call last):
File "/code/api/directory_watcher.py", line 106, in handle_new_image
photo._calculate_aspect_ratio(False)
File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio
with exiftool.ExifTool() as et:
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter
self.start()
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start
raise oe
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start
self._process = subprocess.Popen(
File "/usr/lib/python3.9/subprocess.py", line 951, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child
self.pid = _posixsubprocess.fork_exec(
OSError: [Errno 12] Cannot allocate memory
2021-08-30 12:21:58,930 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/SarverHerringWedding-230.jpg
2021-08-30 12:21:59,486 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/SarverHerringWedding-230.jpg. reason: [Errno 12] Cannot allocate memory
Traceback (most recent call last):
File "/code/api/directory_watcher.py", line 106, in handle_new_image
photo._calculate_aspect_ratio(False)
File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio
with exiftool.ExifTool() as et:
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter
self.start()
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start
raise oe
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start
self._process = subprocess.Popen(
File "/usr/lib/python3.9/subprocess.py", line 951, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child
self.pid = _posixsubprocess.fork_exec(
OSError: [Errno 12] Cannot allocate memory
2021-08-30 12:21:59,561 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_0433.jpeg
2021-08-30 12:22:00,267 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_0433.jpeg. reason: [Errno 12] Cannot allocate memory
Traceback (most recent call last):
File "/code/api/directory_watcher.py", line 106, in handle_new_image
photo._calculate_aspect_ratio(False)
File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio
with exiftool.ExifTool() as et:
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter
self.start()
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start
raise oe
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start
self._process = subprocess.Popen(
File "/usr/lib/python3.9/subprocess.py", line 951, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child
self.pid = _posixsubprocess.fork_exec(
OSError: [Errno 12] Cannot allocate memory
2021-08-30 12:22:00,299 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_3516.jpg
2021-08-30 12:22:01,211 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_3516.jpg. reason: [Errno 12] Cannot allocate memory
Traceback (most recent call last):
File "/code/api/directory_watcher.py", line 106, in handle_new_image
photo._calculate_aspect_ratio(False)
File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio
with exiftool.ExifTool() as et:
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter
self.start()
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start
raise oe
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start
self._process = subprocess.Popen(
File "/usr/lib/python3.9/subprocess.py", line 951, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child
self.pid = _posixsubprocess.fork_exec(
OSError: [Errno 12] Cannot allocate memory
2021-08-30 12:22:01,232 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_6670.jpg
2021-08-30 12:22:01,534 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_6670.jpg. reason: [Errno 12] Cannot allocate memory
Traceback (most recent call last):
File "/code/api/directory_watcher.py", line 106, in handle_new_image
photo._calculate_aspect_ratio(False)
File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio
with exiftool.ExifTool() as et:
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter
self.start()
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start
raise oe
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start
self._process = subprocess.Popen(
File "/usr/lib/python3.9/subprocess.py", line 951, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child
self.pid = _posixsubprocess.fork_exec(
OSError: [Errno 12] Cannot allocate memory
2021-08-30 12:22:01,614 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_5720.heic
2021-08-30 12:22:03,791 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_5720.heic. reason: [Errno 12] Cannot allocate memory
Traceback (most recent call last):
File "/code/api/directory_watcher.py", line 106, in handle_new_image
photo._calculate_aspect_ratio(False)
File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio
with exiftool.ExifTool() as et:
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter
self.start()
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start
raise oe
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start
self._process = subprocess.Popen(
File "/usr/lib/python3.9/subprocess.py", line 951, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child
self.pid = _posixsubprocess.fork_exec(
OSError: [Errno 12] Cannot allocate memory
2021-08-30 12:22:03,819 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_0372.jpg
2021-08-30 12:22:04,380 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_0372.jpg. reason: [Errno 12] Cannot allocate memory
Traceback (most recent call last):
File "/code/api/directory_watcher.py", line 106, in handle_new_image
photo._calculate_aspect_ratio(False)
File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio
with exiftool.ExifTool() as et:
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter
self.start()
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start
raise oe
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start
self._process = subprocess.Popen(
File "/usr/lib/python3.9/subprocess.py", line 951, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child
self.pid = _posixsubprocess.fork_exec(
OSError: [Errno 12] Cannot allocate memory
2021-08-30 12:22:04,417 : directory_watcher.py : handle_new_image : 89 : INFO : job 39b9ca47-340a-4927-8dff-8692774c8db8: handling image /data/IMG_3310.jpg
2021-08-30 12:22:05,177 : directory_watcher.py : handle_new_image : 134 : ERROR : job 39b9ca47-340a-4927-8dff-8692774c8db8: could not load image /data/IMG_3310.jpg. reason: [Errno 12] Cannot allocate memory
Traceback (most recent call last):
File "/code/api/directory_watcher.py", line 106, in handle_new_image
photo._calculate_aspect_ratio(False)
File "/code/api/models/photo.py", line 200, in _calculate_aspect_ratio
with exiftool.ExifTool() as et:
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 410, in enter
self.start()
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 381, in start
raise oe
File "/usr/local/lib/python3.9/dist-packages/exiftool/exiftool.py", line 372, in start
self._process = subprocess.Popen(
File "/usr/lib/python3.9/subprocess.py", line 951, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/lib/python3.9/subprocess.py", line 1754, in _execute_child
self.pid = _posixsubprocess.fork_exec(
OSError: [Errno 12] Cannot allocate memory
2021-08-30 12:23:56,749 : api_util.py : get_location_clusters : 260 : INFO : location clustering took 0.01 seconds

Ahhh. I should have also noted that in this test I allowed the docker image to create the host folder for the volume mapping fo the /data map. It created it with root ownership and then I moved the photos to the folder instead of mapping to an existing folder.

Here is the state of some of the DB tables:
longrunningjobs table and a csv os photo table.
Out of curiosity what exactly is/are clip embeddings?  Just wondering what data this is capturing...

api_photo.csv.txt

I was able to complete my test scan of 33 images by incrementally adding more RAM to the VM and starting clean each time.  Once I reached 12GB it was able to process all 33 images and top showed a max usage of about 9GB.  I have 1 heavy process worker and 2 regular workers configured in the .env file.  That seems like a lot of RAM usage, but if it is expected please make a note for everyone.

I started scan process again and it is now still running. I am at 1,8xx photos, and probably taking another week or so.

PS: The numbers are still hidden as the red is not large enough. Maybe that can be improved?
It seems that a Raspberry Pi is not really up to the job to scan thousands of photos.

@MatthK
You could try creating a swapfile to "boost" your RAM.  True it doesn't really boost it but it works.  I went back to my setup and to conserve RAM from the host needed elsewhere I dropped my VM back to 4GB and added an 8GB swapfile and it worked.  It wasn't as fast but it didn't error out with an out of memory like it does at 4GB or 8GB.

I'm not sure it's a RAM problem. The system uses less than 5GB, with almost 3GB still available. Is there somewhere in Docker that I can allocate more RAM to the container?

@MatthK
Docker by default should have access to all RAM/CPU capacity.  As far as I know the only option available with docker is to limit resource usage.  In my testing the docker container on my VM happily sucked up everything to the extent the VM was unresponsive to SSH or through the VM software 'monitor'... once I found the upper limit of what it would consume in my testing I was able to retain connectivity when it was scanning.
It'd be interesting to see if performance was better or worse if you added another worker and gave it extra swapfile capacity to offset the RAM the worker would use...  Since you seem to have 3GB of capacity left you might be able to squeeze a little more out of the pi that way.  It'd probably be CPU constrained but ...

@n8herring I pushed a new image on dev that should help with your memory issue.

@derneuere that made a huge difference!  I saw the same exact test take max 3GB at it's peak and CPU usage was way down as well - about HALF actually!
-- I realize this is an off topic addition (to a closed topic). what determines if photos show in places? I can search places successfully but the map remains empty.  The tables api_albumplace, api_albumplace_photos and api_photo are populated.  Note that api_photo.exif_gps_lat and .exif_gps_lon are populated but .geolocation_json is just {} in case that matters.

"geolocation_json" has to be set. This is the mapbox response, where we get the cities/countries etc for a given lat/lon. We need this information to cluster the albums nicely.
I guess it doesn't save the photo but still creates the albums for places. I have to investigate this, because it does work for some people.

@derneuere
Cleaned out test system did down/pull/up -d and places is working!  Thanks for fixing that!

@derneuere
Found an issue with how photos are displayed in places.  Will create another issue to separate these threads.


Feature request: set target limits via service discovery labels

Proposal
TL;DR: User should be able to set various scrape-related limits (body_size_limit, sample_limit, label_limit, etc) for individual targets based on its discovered labels.
Use case
Consider a typical prometheus config:
scrape_configs:
- job_name: all-the-targets
  sample_limit: 100
  <some_discovery_config>

This is perfectly readable and works just fine up until some special important target appears. This target contains more than 100 important metrics, and setting different sample_limit is a necessity. However, you have an understanding that there is still a reasonable upper limit. The config grows:
scrape_configs:
- job_name: all-the-targets-but-target1
  sample_limit: 100
  relabel_config:
  - action: drop
    source_labels: [__sd_target_name]
    regex: target1
  <some_discovery_config>
- job_name: target1
  sample_limit: 200
  relabel_config:
  - action: keep
    source_labels: [__sd_target_name]
    regex: target1
  <some_discovery_config>

Not so pretty, but still readable. Time passes, and important endpoint appears. It contains even more metrics. Something like this happens:
scrape_configs:
- job_name: all-the-targets-but-target1-or-target2
  sample_limit: 100
  relabel_config:
  - action: drop
    source_labels: [__sd_target_name]
    regex: target1|target2
  <some_discovery_config>
- job_name: target1
  sample_limit: 200
  relabel_config:
  - action: keep
    source_labels: [__sd_target_name]
    regex: target1
- job_name: target2
  sample_limit: 10000
  relabel_config:
  - action: keep
    source_labels: [__sd_target_name]
    regex: target2
  <some_discovery_config>

Then prometheus 2.27 comes along, and prometheus admin tries to enforce label_value_length_limit. However, there are several endpoints that already have weirdly long labels. They should be allowed to function, but no new endpoints should be allowed to behave that way. This is when hell brakes loose:
scrape_configs:
- job_name: all-the-targets-but-target1-or-target2-and-longlabeltargets
  sample_limit: 100
  label_value_length_limit: 20
  relabel_config:
  - action: drop
    source_labels: [__sd_target_name]
    regex: target1|target2
  - action: drop
    source_labels: [__sd_target_has_long_labels]
    regex: true
  <some_discovery_config>
- job_name: longlabeltargets
  sample_limit: 100
  label_value_length_limit: 100
  relabel_config:
  - action: keep
    source_labels: [__sd_target_has_long_labels]
    regex: true
  <some_discovery_config>
- job_name: target1
  sample_limit: 200
  label_value_length_limit: 20
  relabel_config:
  - action: keep
    source_labels: [__sd_target_name]
    regex: target1
- job_name: target2
  sample_limit: 10000
  label_value_length_limit: 20
  relabel_config:
  - action: keep
    source_labels: [__sd_target_name]
    regex: target2
  <some_discovery_config>

pod_name is used just as an example. various labels would be used in production system, making changes more and more difficult with every change. At some point operator forgets to set some new limit. Those are details. The essence is: config becomes complicated, hard to read and to maintain.
Adjusting some per-target limits via labels will make config dramatically simpler and easier to maintain:
scrape_configs:
- job_name: all-the-targets
  sample_limit: 10
  label_value_length_limit: 100
  <some_discovery_config>

And some target-specific setting will be set via its labels:
[
  {"targets": [ "target1"], "labels": {"__sample_limit__": "200"}},
  {"targets": [ "target2"], "labels": {"__sample_limit__": "10000"}},
  {"targets": [ "longlabeltarget1"], "labels": {"__label_value_length_limit__": ": 100"}},
  {"targets": [ "defaultlimitstarget1"], "labels": {}}
  {"targets": [ "defaultlimitstarget2"], "labels": {}}
 ]

Adding a new limit globally will be as easy as it is supposed to be - just add it to a single scrape_config. Adding exceptions to those default limits would only require setting those labels via SD, which could be done without redeploying new prometheus config.
Similar feature is already present for interval and timeout: https://github.com/prometheus/prometheus/issues/8895
I believe this can be done for other scraper / scrape loop limits and honor* settings as well: https://github.com/prometheus/prometheus/blob/38ef68e27f565b4183b98d5e080076870f99ac65/scrape/scrape.go#L540-L550

A WIP PR to show this is feasible: https://github.com/prometheus/prometheus/pull/10432

Hello from the bug scrub,
closing the issue as there was no further progress after the initial POC. Anyone who would like to work on this, please follow up.

Just in case it helps, my personal gut feeling is we shouldn't do this.
It's very tempting to mirror more and more parts of the scrape config in labels. However, in general there should be one way of doing something. The parts we allowed as labels so far were justified by some notion that targets should have some control of how they wish to be scraped, e.g. if a target "knows" it exposes metrics that change often and make sense to be scraped with high frequency, it makes sense to communicate to Prometheus servers (and possibly other scrapers) via SD that this target wishes to be scraped every second.
However, target limits are inherently limits of the scraper. If a target wishes to limit the metrics it exposes, it can just do so. It doesn't have to be communicated via SD.


validating multiple objects for improved performance

I am facing a scenario where I need to validate a property of multiple objects against a database table. Now instead of checking the database table one by one (one database query per object) in custom validation method in my validator class, it would be better to check it in bulk (single database query) for multiple objects. Is there a way to handle such scenario ?

Hi, in this scenario you’d be better off not using FluentValidation. Perform your bulk validation separately and then merge the results with those from the regular validator (if you need to)


[Admin DS] Refactoring Error Codes

Purpose
Fixes https://github.com/wso2/apk/issues/1055

Codecov Report
Patch coverage has no change and project coverage change: +24.47 :tada:

Comparison is base (ac0484f) 33.62% compared to head (e0b2fa5) 58.10%.

Additional details and impacted files
@@             Coverage Diff             @@
##             main    #1058       +/-   ##
===========================================
+ Coverage   33.62%   58.10%   +24.47%
===========================================
  Files         176       10      -166
  Lines       19985     1012    -18973
  Branches     6172      594     -5578
===========================================
- Hits         6720      588     -6132
+ Misses      13145      424    -12721
+ Partials      120        0      -120

Flag
Coverage Δ

admin-domain-service
58.10% <ø> (?)

idp-domain-service
?

runtime-domain-service
?

Flags with carried forward coverage won't be shown. Click here to find out more.
see 174 files with indirect coverage changes
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


⚠ ClamAV has degraded performance

In 61b70bc, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 64 ms

Resolved: ClamAV performance has improved in 519eb9a.


support for copying custom values from OAuth2TokenContext to JwtEncodingContext

Expected Behavior
In spring-attic/spring-security-oauth, specifically in TokenEnhancer, it was possible to add claims to the JWT token based on the refresh token value, as it was accessible via OAuth2AccessToken.getRefreshToken()
I was expecting it would be possible via OAuth2TokenContext.getAuthorization() but this doesn't seem to be set (always null) for the grants I've tried.
Current Behavior
It doesn't seem like there's any way to access the refresh token value from OAuth2TokenCustomizer<JwtEncodingContext>.
Another option I've tried is to add a custom key/value in OAuth2TokenContext by using DefaultOAuth2TokenContext.Builder.put(key, value), but it seems like custom values are not copied over to the JwtEncodingContext in JwtGenerator.generate(), only known ones.
Would it be possible to add support for something like additionalParameters to handle copying custom values from OAuth2TokenContext to JwtEncodingContext?
Otherwise it seems like the only option would be to have a custom JwtGenerator and use that instead, or did I miss an easier approach?
Context
I'm migrating a project using the existing library, which added a computed claim to the JWT based on the refresh token value.

@emilburzo

it was possible to add claims to the JWT token based on the refresh token value, as it was accessible via OAuth2AccessToken.getRefreshToken()

The default OAuth2RefreshTokenGenerator generates an opaque token so I don't think it will be useful in this case.

I was expecting it would be possible via OAuth2TokenContext.getAuthorization() but this doesn't seem to be set (always null) for the grants I've tried.

The OAuth2RefreshToken is generated after the OAuth2AccessToken.

Would it be possible to add support for something like additionalParameters to handle copying custom values from OAuth2TokenContext to JwtEncodingContext?

Before we consider an enhancement, I'd like to see if we can make things work with the existing customization hooks. Can you please provide more details on your legacy implementation? What are you looking up in the refresh token and using/copying to the access token?

@jgrandja

Before we consider an enhancement, I'd like to see if we can make things work with the existing customization hooks.

FWIW, I was able to proceed with a custom JwtGenerator. You are also correct that I had to generate the refresh token before the access token, but since so far I've only needed it for the unsupported password grant, which must be self-implemented anyway, that wasn't a blocker.
There's code duplication, but now it's more of a question of convenience/maintainability than a necessity.

What are you looking up in the refresh token and using/copying to the access token?

The legacy implementation added a custom identifier based on the refresh token so that it can be uniquely identified from "the outside".
Use-case: users of the refresh token have the ability to request expiry of all refresh tokens except one (e.g. "logout" everywhere except one session)

@emilburzo

The legacy implementation added a custom identifier based on the refresh token so that it can be uniquely identified from "the outside".

I'm assuming the legacy implementation generated a JWT refresh token since it needed to carry a custom identifier as a claim? I'm also assuming you made this work using a custom OAuth2TokenGenerator instead of using the default OAuth2RefreshTokenGenerator:

FWIW, I was able to proceed with a custom JwtGenerator

If my understanding is correct, then the customization you applied is the correct path. Given that OAuth2RefreshTokenGenerator supports opaque token values only, you would need to implement your own OAuth2TokenGenerator to generate a self-contained token value for refresh tokens. Also, there are no plans to provide a OAuth2TokenGenerator implementation for JWT refresh tokens.
It sounds like we can close this issue if my understanding is correct since JWT refresh tokens would need to be a customization applied in the application. I'll leave it open for now until you respond.

@jgrandja

I'm assuming the legacy implementation generated a JWT refresh token since it needed to carry a custom identifier as a claim? I'm also assuming you made this work using a custom OAuth2TokenGenerator instead of using the default OAuth2RefreshTokenGenerator:

sorry, it seems I explained the legacy implementation poorly, let me rephrase:

the legacy implementation generated an opaque refresh token
based on the refresh token value, a unique identifier was added as a claim to the access token JWT using some sort of hashing scheme

(I cannot provide any insights on why it was designed this way, as that knowledge is gone)
The enable for this in the legacy implementation is that the TokenEnhancer had access to the refresh token value, whereas the new JwtGenerator doesn't because, as you mentioned, the refresh token is generated after the access token

Thanks for the explanation @emilburzo. I now understand your issue.
I'm reluctant on modifying the existing implementations OAuth2AuthorizationCodeAuthenticationProvider and  OAuth2RefreshTokenAuthenticationProvider to change the ordering of token generation - it doesn't make sense to generate the refresh token before the access token.
But it sounds like you're not looking for these changes since this is related to the password grant?

FWIW, I was able to proceed with a custom JwtGenerator. You are also correct that I had to generate the refresh token before the access token, but since so far I've only needed it for the unsupported password grant, which must be self-implemented anyway, that wasn't a blocker.

If your case is directly related to a password grant implementation, then you have complete control in the call order of the OAuth2TokenGenerator so you can generate the refresh token first and then associate it to OAuth2Authorization, which then would be available when the OAuth2TokenGenerator is subsequently called for access token generation - you just need to ensure the OAuth2TokenContext.OAuth2Authorization includes the refresh token.
Is this how you implemented it?
Take a look at OAuth2AuthorizationCodeAuthenticationProvider as it has a similar use case where the ID Token generation MAY need access to the access and/or refresh token:
https://github.com/spring-projects/spring-authorization-server/blob/098ab85e2a830966d7ac343d606c1b9d062eb54a/oauth2-authorization-server/src/main/java/org/springframework/security/oauth2/server/authorization/authentication/OAuth2AuthorizationCodeAuthenticationProvider.java#L229

@jgrandja

I'm reluctant on modifying the existing implementations OAuth2AuthorizationCodeAuthenticationProvider and OAuth2RefreshTokenAuthenticationProvider to change the ordering of token generation - it doesn't make sense to generate the refresh token before the access token.

Understood, and since apparently nobody else ran into this problem, it's what I expected.

But it sounds like you're not looking for these changes since this is related to the password grant?

Correct.

If your case is directly related to a password grant implementation, then you have complete control in the call order of the OAuth2TokenGenerator so you can generate the refresh token first and then associate it to OAuth2Authorization, which then would be available when the OAuth2TokenGenerator is subsequently called for access token generation - you just need to ensure the OAuth2TokenContext.OAuth2Authorization includes the refresh token.

Nice, this is actually easier than what I did :point_down:. I had a suspicion this approach might be a problem for the refresh_token grant, since I discovered it's also needed there, but the authorization is already set in the refresh_token grant, so I have access to the refresh token in the JWT customizer -- all good :+1:

Is this how you implemented it?

It looks like I went the harder way, I created a custom OAuth2TokenGenerator<Jwt> with the only change being in the generate() function, to copy the custom id from OAuth2TokenContext to the JwtEncodingContext.
But with your cleaner solution from above, I can delete this part and this github issue can be closed from my side.
Thank you for all the support!

@emilburzo I'm glad I was able to help you find an easier solution for your custom implementation 👍
I'll close this issue.


Better way to query deleted entities

The current way of adding a deleted version for each single query function is cumbersome and ugly. Instead we want to be able to do the follow.
$users = User::withDeleted()->fetchAll();

This might not be feasible, since it would instantiate User or return another (unknown) instantiated object. Returning a class name might be the next best option:
$source = User::withDeleted();
$users = $source::fetchAll();

This does mean that we need to create a new class
class A
{
    protected static $withDeleted = false;

    static function x()
    {
       echo static::$withDeleted ? 'trash' : 'no-trash', "\n";
    }

    static function withDeleted($include = true)
    {
       $classname = get_called_class();
       if (!$include) return $classname;

       $newclass = $classname . '__WITH_DELETED';

       if (!function_exists($newclass)) {
           eval("class $newclass extends $classname { protected static \$withDeleted = true; }");
       }

       return $newclass;
    }
}

After sleeping on it, I believe the above method is to magical. Instead we should just add $opts to the fetch() and fetchAll() methods.
User::fetch(22, Jasny\DB::FROM_TRASH);
User::fetchAll(null, null, null, Jasny\DB::INCLUDE_TRASH);


[Docs] Add missing installs

Hi There, I love this notebook and would love to get it running to create some articles; all goes well with install, but when I try and run the agent, I get the following error:
IndexError                                Traceback (most recent call last)
 in <cell line: 1>()
----> 1 res = await call_agent(
2     "Search on google for the most popular mobile phone models",
3     page,
4 )
5 print(f"Final response: {res}")
12 frames
 in type_text(state)
32     bbox_id = type_args[0]
33     bbox_id = int(bbox_id)
---> 34     bbox = state["bboxes"][bbox_id]
35     x, y = bbox["x"], bbox["y"]
36     text_content = type_args[1]
IndexError: list index out of range
Seems like the boxes are not numbering...


Program Not Running

Hi Dear,
I just downloaded the release v.1.1.1.
I have java installed previously. But when I run the exe or .jar or the start.cmd file, the program is not opening nor showing anything.
I tried to run the .jar file manually through cmd, and it showed like this:
C:\Users\hp>cd C:\java\UgCS-GeoHammer-v.1.1.1
C:\java\UgCS-GeoHammer-v.1.1.1>start.cmd
C:\java\UgCS-GeoHammer-v.1.1.1>java -Xmx2g -cp geohammer-jar-with-dependencies.jar com.ugcs.gprvisualizer.app.MainGeoHammer
Error: Could not find or load main class com.ugcs.gprvisualizer.app.MainGeoHammer
Caused by: java.lang.NoClassDefFoundError: javafx/application/Application
Really appreciate your helps. Thankyou

Hi Dear,
I just downloaded the release v.1.1.1. I have java installed previously. But when I run the exe or .jar or the start.cmd file, the program is not opening nor showing anything. I tried to run the .jar file manually through cmd, and it showed like this:
C:\Users\hp>cd C:\java\UgCS-GeoHammer-v.1.1.1
C:\java\UgCS-GeoHammer-v.1.1.1>start.cmd
C:\java\UgCS-GeoHammer-v.1.1.1>java -Xmx2g -cp geohammer-jar-with-dependencies.jar com.ugcs.gprvisualizer.app.MainGeoHammer Error: Could not find or load main class com.ugcs.gprvisualizer.app.MainGeoHammer Caused by: java.lang.NoClassDefFoundError: javafx/application/Application
Really appreciate your helps. Thankyou

I too have same problem.
com.ugcs.gprvisualizer.app.MainGeoHammer not found

Please check the current version, 1.1.8 - for now it's bundled with Java Runtime 21 custom image for the appropriate platform.


v2.2.0-beta.1 Pop-up background blur sometimes persists after closing popup

Describe the bug
After opening and then closing a pop-up, the background blur will sometimes persist and can only be fixed by refreshing the tab/restarting app.
Expected behavior
Background blur will always be dismissed when pop-up is dismissed.
Screenshots

YAML
Here's YAML from one of my pop-ups if it helps:
type: vertical-stack
cards:
  - type: custom:bubble-card
    card_type: pop-up
    hash: '#living-room'
    sub_button:
      - entity: sensor.living_room_thermostat_temperature
        show_background: false
        show_state: true
      - entity: sensor.living_room_thermostat_humidity
        show_background: false
        show_state: true
    card_layout: normal
    show_header: true
    name: Living Room
    show_icon: false
    scrolling_effect: false
    show_name: false
    styles: |2-
        .bubble-button-card-container {
          background: none !important;
        }
    icon: ''
    tap_action:
      action: none
    double_tap_action:
      action: none
    hold_action:
      action: none
    button_action:
      tap_action:
        action: none
      double_tap_action:
        action: none
      hold_action:
        action: none
    margin_top_mobile: calc(100vh - 325px)
    margin_top_desktop: calc(100vh - 325px)
    shadow_opacity: '0'
    bg_blur: '14'
    bg_opacity: '100'
    hide_backdrop: false
  - type: custom:bubble-card
    card_type: button
    button_type: slider
    entity: light.living_room_light
    name: Light
    card_layout: large
    tap_action:
      action: toggle
    double_tap_action:
      action: none
    hold_action:
      action: more-info
  - type: custom:bubble-card
    card_type: separator
  - type: custom:bubble-card
    card_type: media-player
    entity: media_player.living_room_shield
    name: Shield
    icon: mdi:television
    tap_action:
      action: toggle
    double_tap_action:
      action: none
    hold_action:
      action: more-info
    button_action:
      double_tap_action:
        action: none
      hold_action:
        action: more-info
    show_state: true
    card_layout: large

Informations (please complete the following information):

OS: Windows 11/Android
Browser/App: Chrome/Official App
Bubble Card version: v2.2.0-beta.1
Home Assistant version: 2024.7.4

Thank you for the feedback!

Same on my devices - Android TSV's and iPhone 15 pro. When closing  pop up, the underlying page never shows again.

Same on my device - Android with offiziell app. When closing pop up, the underlying page never shows again. I must restart the app

Same here, on desktop and mobile.

Same for me also, app and browser

Same for me here, but not sometimes. Samsung S23ultra and android official app. Haven't tested on desktop yet.

This is now fixed!
https://github.com/Clooos/Bubble-Card/releases/tag/v2.2.0-beta.2

Thanks Clooos :) working ok here too on my devices. Having problems trying to clear and update the cache on the iOS companion app - but that's an HA thing. Cache seems to always be a problem.

On the iOS app you can go to your HA settings, then on Companion App > Debug > Clear frontend cache.

Yes I did that but it was still showing the previous version. I came back to it 10m or so later and tried again and it was then picking up the new version. Odd.

Odd indeed, the HA cache can be really treaky 😅
Glad that it's fixed!

Fixed, thanks for the update @Clooos !

@Clooos this is happening again on 2.2.1. Seems to be somewhat rare though. It wasn't happening at all after it was fixed in 2.2.0-beta2.


pass TSD pw to tacl_auth as env var

Didn't test
https://github.com/unioslo/tsd-s3cmd/commit/ea3c0d42c1c9db37eb5f6cf595ddf75048806fc8#r47686508

Cool. I'd be happy to merge once you've tested it.

Sorry, have to pass on that one. I'm not using the s3 API. This is as good as it gets.

evidently

PRs welcome.

Um ok, anyways, I will try to find time to test this in the near future.

Great. I only changed two lines. Feel free to review and disagree.


support enum values in ElasticSearch

WHAT:  Add search capability for blueflood so that users can pass in the param "include_enum_values" and be able to retrieve enum values along with the metric names in the result when hitting the /v2.0/:tenantId/metrics/search endpoint.
(see story:  https://jira.rax.io/browse/CMD-741;  Note -- I do not think we should implement the enum version of the DiscoveryWriter class in this story, but should move the writing of that class to the rollup story where the new index will actually be created during rollup).
HOW:

extract ElasticIO.Discovery inner class to standalone Discovery class
extract ElasticIO.ESFieldLabel enum to standalone enum class
add configs for enums elasticsearch module
add EnumElasticIO class with insertDiscovery() and search() methods.
modify SearchResult to store enum values field
modify HttpMetricsIndexHandler to check for include_enum_values
modify HttpMetricsIndexHandler to include enum values in json result

Please refer to the new "enums" index mapping for the index and its fields to be created by metrics rollup of enum values:  https://github.com/VinnyQ/blueflood-chef/blob/add_elasticsearch_mappings_for_enums/site-cookbooks/old-elasticsearch/files/default/enums.json
NOTE:  This deploy will need changes to blueflood.conf (after the code is merged)
To add the following property:
ENUMS_DISCOVERY_MODULES=com.rackspacecloud.blueflood.io.EnumElasticIO

(do this in https://github.com/rackerlabs/blueflood-chef).
And also we will have to create another ElasticSearch alias "metric_metadata_read" that also points to the new "enums" index.  The code to do so is:
curl -XPOST 'http://localhost:9200/_aliases' -d '
{
    "actions" : [
        { "add" : { "index" : "enums", "alias" : "metric_metadata_read" } }
    ]
}'

@GeorgeJahad @tilogaat @goru97 Please review when you have a chance.

Hi @tilogaat @GeorgeJahad @goru97, please start code review again.  I have discussed with tilo and worked with her to reuse her BluefloodEnumRollup class, eliminating the need to create a new EnumMetric class implementing IMetric to store the metric data inc. enum values (we switched to using the PreaggregatedMetric class which is also used by rollups for the same purpose).
I am happy with this code, and uncertain whether to we actually need a writer class akin to DiscoveryWriter at the moment.  I'll look at what's needed when doing the "Check for bad enum metrics" story:  https://jira.rax.io/browse/CMD-749.

@VinnyQ can you make a list of all the chef changes for this story?

@VinnyQ Can you add an integration test that will explain how it all comes together?
Check out
HttpAnnotationsEndToEndTest

for inspiration. Let me know if I missed a similar test though.

+1

@tilogaat re:  end to end integration test similar to HttpAnnotationsEndToEndTest, this is not immediately implementable because the enum_metrics are not ingested and added to ES as indexes similar to other metrics.  There's a rollup step in between.
The closest I have right now is the EnumElasticIOTest search tests (https://github.com/VinnyQ/blueflood/blob/3d6d289712cfcfb84187a5a069632aad08ba8300/blueflood-elasticsearch/src/test/java/com/rackspacecloud/blueflood/io/EnumElasticIOTest.java) where I would create the ES indexes using insertDiscovery() and the search for the results in the test.
However, I am planning or adding the integration end to end test from the rollups creation to the index searching once I added that feature in a different story.  Once the rollup feature is added to create the ES enums indexes during rollup, I'll have an end to end story that will

post/ingest enum metrics
run the rollups
check that the indexes are created in ES

Sounds good?

+1 Sounds good vinny

@GeorgeJahad @tilogaat all issues from code review addressed.  Do you guys want to take a look at it one more time before I merge it?


Regression: Interpolation not compatible with driver name

Issue Type

Bug report

Molecule and Ansible details
ansible --version
ansible <IP_ADDRESS> (detached HEAD 8a7f9beab7) last updated 2018/03/21 16:06:26 (GMT -400)
  config file = None
  configured module search path = [u'/Users/nicholasbering/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/nicholasbering/git/ansible/ansible/lib/ansible
  executable location = /Users/nicholasbering/git/ansible/ansible/bin/ansible
  python version = 2.7.15 (default, May  6 2018, 12:01:05) [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-9<IP_ADDRESS>)]

molecule --version
molecule, version 2.14.0

Molecule installation method (one of):

pip

Ansible installation method (one of):

source

Desired Behavior
I recently upgraded molecule from v2.10.1. I was using environment variable interpolation to select the driver, so that I wouldn't need to manipulate files in CI (where I was ec2 driver) and local (where I use Vagrant).
This was working fine, with the relevant molecul.yml portion looking like this:
driver:
  name: ${MOLECULE_DRIVER:-vagrant}
  provider:
    name: virtualbox
platforms:
  - name: instance
    # virtualbox stuff
    box: debian/jessie64
    # ec2 stuff
    instance_type: t2.micro
    vpc_subnet_id: ${VPC_SUBNET_ID}
    image: ${DEBIAN_AMI}

And a .envrc file with direnv looking this this, to override for EC2.
# General Molecule Settings
# export MOLECULE_DRIVER=vagrant
export MOLECULE_DRIVER=ec2

# AWS settings for EC2 driver
export AWS_REGION=ca-central-1
export AWS_PROFILE=default
export VPC_SUBNET_ID=subnet-xxxxxxxx
export DEBIAN_AMI=ami-a1fe43c5

This worked in v2.10.1.
Actual Behaviour (Bug report only)
When I upgraded to v2.14.0, I received this error:
$ molecule test --all
Traceback (most recent call last):
  File "/usr/local/bin/molecule", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/molecule/command/test.py", line 113, in test
    base.get_configs(args, command_args), scenario_name)
  File "/usr/local/lib/python2.7/site-packages/molecule/command/base.py", line 120, in get_configs
    ) for c in glob.glob(MOLECULE_GLOB)
  File "/usr/local/lib/python2.7/site-packages/molecule/config.py", line 61, in __call__
    obj.after_init()
  File "/usr/local/lib/python2.7/site-packages/molecule/config.py", line 107, in after_init
    self.config = self._reget_config()
  File "/usr/local/lib/python2.7/site-packages/molecule/config.py", line 278, in _reget_config
    env = util.merge_dicts(os.environ.copy(), self.env)
  File "/usr/local/lib/python2.7/site-packages/molecule/config.py", line 186, in env
    'MOLECULE_INSTANCE_CONFIG': self.driver.instance_config,
  File "/usr/local/lib/python2.7/site-packages/molecule/util.py", line 307, in wrapper
    rv = function(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/molecule/config.py", line 170, in driver
    driver.name = driver_name
AttributeError: 'NoneType' object has no attribute 'name'

I traced that down to this function:
https://github.com/metacloud/molecule/blob/ffab458f10bb638106e9fa56431c37a575beee22/molecule/config.py#L145-L172
When I changed my interpolated string back to a plain literal, the error goes away.
driver:
  name: ec2

So my guess would be the it is not interpolating the value from the environment variable before using it in this function.
This used to work... so I'd think of it as a type of regression. However, I understand if this pattern is considered to be unsupported. It is quite convenient though, as it allows team members to use different drivers without changing any source-controlled files.
Either way... it would probably be ideal to raise a meaningful error as an else condition to that long if...elif chain. Other cases - like a simple typo - would also benefit from a better error.

Yeah, this is actually a case where your environment variable name should not be named MOLECULE_.  If you name it something else it will continue to work.  It's a bit of a chicken and egg problem.
In a recent release molecule took ownership of any variable named MOLECULE_ in moleucle.yml so that variables molecule sets (e.g. MOLECULE_SCENARIO_DIRECTORY) could be referenced in molecule.yml.
I suggest using a different env var for the variable.  Too much inception going on.  I'll look into why the schema validator didn't catch this, b/c it should have seen a none value for driver and reported a scehma validation error.
Hope this answer helps.

Cool. I actually wondered about that a little when I saw the changelog entry about MOLECULE_ variables being special somehow. I'll try and some up with some other naming scheme, and try that out.

I've confirmed that using a variable without MOLECULE_ in the name solves my problem.
In looking for an alternative variable prefix that shouldn't conflict with some other program, I found that "molecule" is a synonym for "unit". I always thought the name was appropriate, but I didn't quite get it until I saw them together in a list of words. 😃

Thanks!


Add internationalisation UI to modeladmin

Initial work to fix #7856.
Merely copied modeladmin code from #6230, but it didn't work.
So I added a second commit that made the select show up in the modeladmin index, but using it still doesn't change the listed items... =(

Added the same select used by the snippets to the modeladmin create, and used the ?locale= value when saving the instance. Not sure if I did it the right way...
Didn't changed the edit page, so for now you can only set the locale when creating, but can't edit it later.
The UI flow is a bit awkward, because every time you open the create page the locale will be set to the default one, and if you are adding multiple non default locale items you need to select the right locale every time. Maybe it should maintain the ?locale= value when going back to the index after adding an item. Not sure how to do it...

As I said, the functionality is not perfect, but at least it would make it minimally usable. It would be great if you could add it to the next version. :smiley:
I don't know how robust is the test coverage for this project, but it would be good to make sure this PR isn't breaking anything else, I only marginally tested it. :grin:
Also, if you think it would be mandatory to address the remaining problems I described for it to make to the next version, please confirm if what I did is in the right path and I can try to continue it today.
I understand you must be in a hurry to pack the next version, so I understand if this PR doesn't make it. :wink:

Hey @andresmrm thank you for restarting work on this.
I've made further tweaks in https://gist.githubusercontent.com/zerolab/d50698800aaa25fb9e72011cdccde8c4/raw/3ad13f9b9394d67769dc15003802686855fd08ed/7857.patch -- tidies up the switcher on the create view, adds it to the edit and inspect view, as well as adding a locale column in the index view
Did not have time to write up tests, but they would go in contrib/modeladmin/tests/test_simple_modeladmin.py -- there are tests for the various views. I would add a new test that generates two new languages (e.g. French and Portuguese) and the creates some entries for a chosen model (either Author or Book) then tests the functionality. e.g.

index view: with or without locale filter applied
create view: has locale switcher, then chosen language matches that of the locale GET param
update view: doesn't show the switcher for an instance without translations. only shows the locale for the translation
index view, same as update view

Hello! Merged your patch. I'll try to do the tests.

Changed your patch to avoid error. Not sure if it should be if issubclass(self.model, TranslatableMixin) and self.locale: instead of just if issubclass(self.model, TranslatableMixin): as I did...

For the tests, we need a model with TranslatableMixin, right? If I add it to the Book model in wagtail/tests/modeladmintest/models.py it would could affect the non i18n tests... Should I add a new model to wagtail/tests/modeladmintest/models.py?

@andresmrm ah, you're right. probably a new simple model makes more sense.
on the question of self.locale - the property is on the view, not the model (accessible via self.model). Looks like it did not come through in the patch, but I have self.locale = None in WMABaseView.__init__() which does away with the error

Ops, sorry, my confusion. Deleted my last commit and added self.locale = None.

Added tests for index and create.

update view: doesn't show the switcher for an instance without translations. only shows the locale for the translation
index view, same as update view

Not sure about what do you mean with "instance without translations", or "only ... the translation". Is the TranslatableMixin.translation_key used to link multiple translations of the same "thing"? When editing a post a see the "translate" button, but it isn't present when editing a modeladmin...

Say you create a Book in English and leave it at that. Book.get_translations() (https://github.com/wagtail/wagtail/blob/093d59d4e3f8057f4bdb281f9d8cf223c90cf0c5/wagtail/core/models/i18n.py#L133) will return an empty queryset. So when you go to edit it, there should not be any switcher, just the current language
Having a "Translate" button when there are no translations would be pretty neat, but it expands the scope of this PR. Plus, if you edit a page without any translations, there is no button to start the translation. So I would say let's keep that for a follow up PR

Not sure if I understand it...
At the moment the edit page doesn't show the top right locale switcher for me. That seems right, since my items aren't linked to other items, as translations.
But without the "translate" button, it seems there is no way to link items as translations, so the switcher shouldn't appear anyway (unless someone adds then programmatically...).
Should we leave it this way? Or should I try to add the switcher when there are translations?

https://github.com/wagtail/wagtail/pull/7857/files#diff-f369e817eb6a93bb0b23639713e726aec704aba1b4342d1aae8afeb625530ed9R1 and https://github.com/wagtail/wagtail/pull/7857#discussion_r788959001 will ensure the locale switcher shows in the edit view if you have any translations for the instance you are currently editing.
so the test should be something like:
book = TranslatableBook.objects.first()  # the english book

# get edit view, check no switcher shows

book.copy_for_translation(locale=Locale.objects.get(language_code='fr'))

# get edit view, check locale switcher shows

Hi, again!
Added the test. Had to add https://github.com/wagtail/wagtail/pull/7857/files#diff-131da701ebff77c03de56bb7ff4b43d307531db394c9905efbc57ca418ce6a59R177 without it I was getting TypeError: Direct assignment to the reverse side of a related set is prohibited. Use +.set() instead.. Not sure if this is the right way to fix it.

Added postgres_index_entries to the excluded fields, looks like it worked. :smiley:

Hi! Any update on this? It has been more than a month...

Hey @andresmrm,
First off, sorry for the delay. It did not make the cut for 2.16, but it is on my list to review.
Would you be able to rebase your branch on latest main, and lint it as well as we added black linting. https://github.com/wagtail/wagtail/wiki/Updating-PRs-for-linting-changes has some instructions
And I commit to reviewing the PR this weekend.

@andresmrm tomorrow I have dedicated Wagtail core time and this is first on my list (I did promise I will review after the rebase, so that'll happen)

Hi! Any updates? Sooner or later I'll need this feature in a website I'm developing. :grin:

Hey @andresmrm,
well, the PR is a pinned tab in my browser. Aside from that, I've been ill and out of action, then playing catch up.
On the other hand this will make it the next release in a bit over a month -- and we'll get that in,
I am sorry I promised and not got to it, especially this being your first contribution to Wagtail.

Thanks for the quick reply!
I'm sorry to know you've been will... I wish you the best recover possible!
But I'm glad to know this feature will make it to the next release, that should be good enough for us. :smiley:

Thank you very much for this @andresmrm!
Merged in 483b7d2 and scheduled for release in 3.0

Great news! Thanks! :smile:


Add parameter to enable adding Hangouts/Meet link toevent

Don't forget me, I'm necessary for adding Hangouts/Meet links to events. That old fogey Ross left me out of the earlier PR.


Boyer-Moore Majority Voting Algorithm

Is your feature request related to a problem? Please describe.
Implementation of Boyer-Moore algorithm to solve majority element.
Describe the solution you'd like
I will solve it for finding majority element 1 (occuring more than 2 times), and majority element 2 (occuring more than 3 times)
Additional context
In which section should I add this? under sub-arrays.
@Kumar-laxmi
assign me this in c++

Assigned!
@Anmol-Baranwal : C++

@Kumar-laxmi Please assign me in c.

@Kumar-laxmi Please assign it to me in Python.

Assigned!
@Sachittarway : C
@stiwari-ds : Python

@Sachittarway What is your status on this issue

@Sachittarway What is your status on this issue

Ya actually working on one more algorithm. Will commit this by 20th.

Is anyone working on this issue in C++?

Is anyone working on this issue in C++?

I have already merged the solution through PR in c++.


New placeholders that show expiry time of permissions/groups but in seconds.

Description
I was happing to use this info inside a script to compare permission nodes' expiry time and then have the output depend on which permission node has the longest expiry time. Just haven't been able to figure out how to get the time in seconds.
As it is right now, the only placeholder that shows such information, show it in a human-readable format

This information isn't helpful when wanting to use it in a script etc. and since I'm unsure of how else I could get the expiry time in seconds, I am unable to compare it to another permission node, inside of my Scripts / JavaSciprt placeholders etc.
Proposed Behaviour
The suggested placeholders could follow this format.

Placeholder
Argument
Description

%luckperms_expiry_time_in_seconds%
<permission>
Gets the duration remaining on a temporary permission assigned directly to the player in seconds.

%luckperms_inherited_expiry_time_in_seconds%
<permission>
Gets the duration remaining on a temporary permission assigned directly to or inherited by the player in seconds.

%luckperms_group_expiry_time_in_seconds%
<group name>
Gets the duration remaining on a temporary group membership assigned directly to the player in seconds.

%luckperms_inherited_group_expiry_time_in_seconds%
<group name>
Gets the duration remaining on a temporary group membership assigned directly to or inherited by the player in seconds.

Extra Details
n/a

This is a good suggestion, however in the mean time you could probably look into something like PlaceholderAPI's Formatter placeholder Expansion to change the output of it.
https://github.com/PlaceholderAPI/PlaceholderAPI/wiki/Placeholders#formatter


fuzzing: Add more fuzzers

Signed-off-by: AdamKorcz<EMAIL_ADDRESS>Please provide a description of this PR:
Adds more fuzzers.

@AdamKorcz: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

lint_istio
fbd37e80bc43564873ca31530ab321149ea199c3
link
true
/test lint

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@AdamKorcz: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

lint_istio
fbd37e80bc43564873ca31530ab321149ea199c3
link
true
/test lint

gencheck_istio
fbd37e80bc43564873ca31530ab321149ea199c3
link
true
/test gencheck

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@AdamKorcz: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

lint_istio
fbd37e80bc43564873ca31530ab321149ea199c3
link
true
/test lint

gencheck_istio
fbd37e80bc43564873ca31530ab321149ea199c3
link
true
/test gencheck

integ-ds_istio
fbd37e80bc43564873ca31530ab321149ea199c3
link
true
/test integ-ds

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.


[intel] Add TargetInfo

TargetInfo was added upstream, and used in NVidia and AMD backends. This PR adds TargetInfo for intel backend to be more synchronous with other backends.

@prathams417 @whitneywhtsang this PR has no associated issue and no explanation on the reason/rationale these changes are necessary.  Can you retroactively open an issue, explain the rationale for this PR, and  link it here.

@prathams417 @whitneywhtsang this PR has no associated issue and no explanation on the reason/rationale these changes are necessary. Can you retroactively open an issue, explain the rationale for this PR, and link it here.

Done.


Updating the go version to 1.22 in the docker file

What type of PR is this?
maintenance
What this PR does / why we need it?
Based on the existing supported versions of OCP, https://github.com/openshift/ocm-agent should atleast support Go 1.22 ahead till version 4.17.
Which Jira/Github issue(s) this PR fixes?
Fixes #OSD-25318

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 47.94%. Comparing base (9c8017f) to head (c6e2d62).

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #112   +/-   ##
=======================================
  Coverage   47.94%   47.94%
=======================================
  Files          20       20
  Lines        1508     1508
=======================================
  Hits          723      723
  Misses        745      745
  Partials       40       40

Flag
Coverage Δ

47.94% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

/lgtm

/approve


Allow to copy spiders between projects

Added:
https://github.com/scrapinghub/portia/commit/6011a6a394cffb756f50a919a0b88a3458c62710
https://github.com/scrapinghub/portia/commit/0d0bba9a2edc5422d31be61e9b19b2c4169117bd
https://github.com/scrapinghub/portia/commit/076d58df5c41aff389c3dc49c13f2260f564b80c
https://github.com/scrapinghub/portia/commit/0c9db103c3a359cd49c906dccf7ee5667a65e126


Cannot open Java file

I try ti open Java files and it said " unsupported file"

What is the encoding of the file? It's likely an encoding issue.

ISO-8859-1


COMP3710 StyleGAN Implementation for ADNI Brain Image Generation

Hello COMP3710 Team,
This fork of the topic-recognition branch contains an implementation of the StyelGAN deep neural network architecture. Which was implemented to try and solve the problem of both generating brain scan images based on the ADNI dataset, while also having the Discriminator and Generator be able to distinguish between the two classes, being Cognitive Normal (CN) and Alzheimer's Disease (AD).  This also includes the functionality to plot the UMAP Embeddings based on the feature map output of the Generator, excited by the class labels it receives.
For further information regarding the entire implementation of this StyleGAN architecture, it's performance, and it's dependencies, please don't hesitate to view the README.md file contained within this pull request. For all files pertaining to the StyleGAN model, including the README.md, go to the following directory recognition/StyleGAN-46988133 within the topic-recognition branch.
Here is a brief summary of each item contained within this pull request:

Assets\: Used to store all figures and images required for the README.md file. This includes training loss plots, real vs fake image comparisons and UMAP Embedding plots.
dataset.py: Loads the required ADNI dataset images, and performs simple torchvision transforms on them to get them to the desired resolution and normalisation range. Follow the dataset structure comments within this file to ensure your ADNI dataset is structured in the required way.
environment.yml: Contains the definitions for the Conda environment used to run this program. This can be used to install the same Conda environment used during development of this StyleGAN model. Information for how to perform this action is explained in the README.md, in the section labelled 'How To Run'.
hyperparameters.py: Contains all hyperparameters and configuration constants used to train and visualise the StyleGAN model. To change the dataset location to one that you have, change the ROOT constant within this file.
modules.py: Contains the entire implementation of this StyleGAN, including the classes for the Discriminator, Generator and Mapping Network. Plus any helper functions and classes necessary to them.
README.md: Contains helpful documentation about the entire StyleGAN development process, results, dependencies and even possible future improvements that could be applied. If you have any questions after reading this pull request description, please head there for further information.
predict.py: Loads both the Discriminator and Generator models after being trained through various resolutions, and is used to validate that they work. This is done by plotting a UMAP Embedding based on the Generator's feature map output, and also a comparison of the real ADNI dataset images compared to the ones created by the Generator.
train.py: Implements the main training loop for this StyleGAN implementation, which uses a baseline ProGAN model to incrementally increase the resolution of the training image. Once each resolution is trained fully, various plots and figures are saved to the /saved_output folder.
utils.py: Contains a helper function , specifically one to help plot the necessary figures after every resolution is trained within the training loop.

Thank you,
Matthew Lockett [46988133]

Recognition problem: StyleGAN on ADNI ---- Hard Difficulty

Algorithm solves the problem: (4.5 Marks)
Somewhat not reasonably clear generated images (32x32) -0.5;

Algorithm demonstrated and functions: (3 Marks)

Design: (1 Mark)
Modularized design, good programming practice using utils script

Commenting: (1 Mark)
Sufficient docstrings and detailed comments

Algorithm above Normal Difficulty: (5 Marks)

Algorithm is Hard difficulty: (5 Marks)

Suggestions:

Firstly, I do think the progressive training framework adapted from ProGAN should work, yet you mentioned:

In terms of image generation this StyleGAN implementation was not able to fully progress through the chain of ProGAN image sizes, being 8x8, 32x32, 64x64, 128x128 and lastly 256x256.

It would be good if you can clarify the reason. If this is due to memory issue, you can try to decrease the batch size. Also, the final 256x256 might not be a good idea as the original image size is 256x240. Anyway, I'm afraid 32x32 is not sufficient to present "reasonably clear images", much information would be lost and performance would be jeopardized (think about it, images of higher resolutions were not essentially used to train the model given this progressive training approach). I'm happy to give the 0.5 mark back if you can justify;

It seems you only used the training set to train the model, while you could make use of both training and test sets since there's no testing for this task ---- this might improve the current performance;
The UMAP was plotted on the feature map extracted from the last layer of the generator, while ideally it should be plotted on the style space (see the reference paper from the task sheet). I'm not taking marks off for it as it seems you are aware of what you are doing based on your justification. I just need to point out that this is not the intention of this task, and by plotting the UMAP on the style code, it's expected to observe better clustering as your model is conditional;
Scripts environment.yml and hyperparameters.py are not needed for this project, but good practice for the purpose of modularization and reproducibility;
For the Future Improvements, I'm still convinced progressive training should be beneficial given sufficient resources, and the Wasserstein loss makes sense if it can converge for higher resolutions;
Very good attempt to introduce conditions into the StyleGAN;
Very elaborative documentation and good referencing practice.
Very good project, showing effort put :)

This is an initial inspection, no action is required at this point

Solves problem: Nice documentation and introduction of the network. All generated images are the same, which is unreasonable. The loss plot of 32*32 resolution is provided; it would be better to have the loss plot for each resolution. Nice UMAP plot, clear two clusters. Nice records for experiment steps.
Implementation: Very well structured.
Design: Good
Comment: Very nice and clear, sufficient and detailed comments.
Difficulty: Hard

Marking

Good/OK/Fair Practice (Design/Commenting, TF/Torch Usage)
Good design and implementation.
Spacing and comments.
Header blocks.
Recognition Problem
OK solution to problem. Low image quality.-1
Driver Script present.
File structure present.
Good Usage & Demo & Visualisation & Data usage.
Module present.
Commenting present.
No Data leakage found.
Difficulty : Hard. StyleGAN (Hard Difficulty)
Commit Log
Good Meaningful commit messages.
Good Progressive commits.
Documentation
Readme :Acceptable. Some images inconsistancies.-1
Model/technical explanation :Good.
Description and Comments :Good.
Markdown used and PDF submitted. PDF Checked.
Pull Request
Successful Pull Request (Working Algorithm Delivered on Time in Correct Branch). ON time.
No Feedback required.
Request Description is good.
TOTAL-2

Marked as per the due date and changes after which aren't necessarily allowed to contribute to grade for fairness.
Subject to approval from Shakes

Good work!


Sampling parameters client-side config

First off, love the project, one thing has been bothering me was that I couldn't change the sampling parameters on client-side. With llama3.1 there's more repetition and temperatures above 0.2 are difficult to work with in that if you don't give the AI a clear goal it might go on a tangent.

Added the most popular sampling parameters into glados_config.yml to allow users to customize the sampling on client-side. This is helpful for a few reasons:

Given different scenarios benefit from different parameters, it's good to allow customization.

For smart, mostly accurate and helpful GlaDOS turn temperature and various other parameters down. Turn them up if you want more witty jokes and creativity at risk of dreaming up stuff.

Ability to use one LLM server for multiple tasks (GlaDOS, Coding, ST and RAG) when each client sends their sampling parameters along with each request.

Testing/Edge cases

When user comments out all the parameters, the (llama.cpp ¹ ) safe defaults from class LlamaGenConfig are used, preventing any errors.

Although this could've been an if statement, in edge cases like the server being configured for coding tasks by default, it's better to use the safe defaults from this.

TODOs

Implement some of the less-important sampling parameters. I've decided not to do it since the defaults are usually best there, although some people might want repetition penalty for long conversations..?

Notes

llama.cpp safe defaults ¹ Since LLama.cpp defaults differ between their API descriptions and readme descriptions, I've opted to use the default values from llama.cpp's chat UI (while running LLama3.1 8b)

Summary by CodeRabbit

New Features

Introduced a new configuration class, LlamaGenConfig, enhancing the control over Llama generation parameters.
Added a YAML configuration section allowing users to customize Llama generation settings.

Improvements

Enhanced the VoiceRecognition class to support new parameters for improved Llama generation.
Updated the main execution flow to load configurations from YAML for dynamic adjustments.

Much appreciated!
I will be leaving today for an overseas holiday, I can review this when I'm back next month.

I will be leaving today for an overseas holiday, I can review this when I'm back next month.

Thanks for the quick response and enjoy your holiday!


Splits do not redistribute space evenly when added

This works perfectly in iterm2 but is clunky in ghostty.
Adding new splits should divide the space evenly, not cut the focused one in half:
Do a split, split twice:
Iterm2:

Ghostty:

I'm not really sure that because another terminal does it that way Ghostty needs to do it too?   personally I'd rather not have an existing split or window shrink down just because I'm adding another split or multiples and affecting all my existing sizes automatically.

Yeah the question is really whether you are splitting the whole window or splitting the split. Perhaps a new command could redistribute space evenly across one axis for the current split's position.

Mmm, rationale isn't "iterm2 does it therefore it's right". Iterm2 just has far more development and has implemented these refinements, it's obviously the right way to do it because 1 - the user wants 3 columns, almost always would want them even width, and not doing this means the user has to drag it every time to get the most commonly desired layout. Not wanting them even could be desired but that's an edge-case.


Doc

spelling fixes and added pcs compatibility commands
I'm also not sure why you renamed mysql_prm to mysql when setting downloading the resource agent. Keeping the mysql_prm I thought would have been a good differentiator.
Thanks for the documentation and agent.

I think this is good, one thing I'd suggest changing though is:
pcs constraint location  reader_vip_1  avoids  readable eq 0
pcs constraint location  reader_vip_2  avoids  readable eq 0

to:
pcs constraint location  reader_vip_1  avoids  readable lt 1
pcs constraint location  reader_vip_2  avoids  readable lt 1

So the it closely matches the original doc and if in any case, any negative value is used for the variable in the future.


mhm

Coverage decreased (-0.4%) to 84.321% when pulling 4e83b6b1da174a99693d99659f25f20e6d8fdec9 on StefanBranch into a0c27083da9b732609e01bfa696ed3da5757a12a on master.

Coverage decreased (-0.4%) to 84.321% when pulling 4e83b6b1da174a99693d99659f25f20e6d8fdec9 on StefanBranch into a0c27083da9b732609e01bfa696ed3da5757a12a on master.

Coverage decreased (-0.5%) to 84.278% when pulling ef2e41bd01e4aad7f8924e8e6c07a46cbaec4356 on StefanBranch into a0c27083da9b732609e01bfa696ed3da5757a12a on master.

Coverage increased (+2.7%) to 87.436% when pulling 54f018ba4567843f5b987bb6fdc4bc00ffb9b60a on StefanBranch into a0c27083da9b732609e01bfa696ed3da5757a12a on master.


Add Shopaholic technology

Adding https://shopaholic.one/
This is extension for October CMS

I fixed it up, thank you!


SecurityPortfolioManager.TotalPortfolioValue looks like a bug

https://github.com/QuantConnect/Lean/blob/master/Common/Securities/SecurityPortfolioManager.cs#L342
I think you need to subtract the totalHoldingsCostWithoutForex instead of adding it.

No, I misunderstood, never mind.


Add numericPrecision method to DataType

This is mostly a fixed number, but can change for the numeric type, and will
be used in type compatibility checks in #15525

@Mergifyio squash title+body

@romseygeek btw, since this is gonna be used in https://github.com/crate/crate/pull/15525 which is a bug fix (no?) maybe already backport this to 5.6?


Code clean up

@CannonLock I changed the base to the rework-editing-tools branch. Really, this should have all been done within the UW-Macrostrat repository


Unable to run ace on Mac OSX 10.12.2

Hi,
I installed Adonis in my Mac, with node = v4.5.0, npm = 4.0.5, with yarn package. When I run ace for any operation, i am getting the following error. Running with flag --harmony_proxies also gives the same error. I am unable to run any commands. Please help.
Error: proxies not supported on this platform. On v8/node/iojs, make sure to pass the --harmony_proxies flag

Hey @swethaphatke !
Did you read the documentation ?

Older versions of Node.js requires --harmony_proxies flag to add support for ES2015 Proxies. If you are using Node.js < 6.0, make sure to make following changes.
...

@swethaphatke You need to write --harmony_proxies on the first line of the ace file

Closing since not actionable


Vaccine time series Data stopped getting updated after 11-22-22

Noticed that the time series data for vaccinations is no longer getting updated. Was wondering if this was a known issue or some change with the frequency of updates for this data?
see https://raw.githubusercontent.com/govex/COVID-19/master/data_tables/vaccine_data/global_data/time_series_covid19_vaccine_global.csv and https://raw.githubusercontent.com/govex/COVID-19/master/data_tables/vaccine_data/us_data/time_series/time_series_covid19_vaccine_us.csv

Yeah,we are facing the same issue. Could this be updated soon?

https://github.com/govex/COVID-19/commit/e9bc27a624101caeb8ab750ac6e4fc923e51d392 has imported data up to 2022-11-27.  🤞 for today's update.