raw.py ERROR

#!/usr/bin/python3

# Configure a raw stream and capture an image from it.
import time

from picamera2 import Picamera2, Preview

picam2 = Picamera2()
picam2.start_preview(Preview.QTGL)

preview_config = picam2.create_preview_configuration(raw={"size": picam2.sensor_resolution})
print(preview_config)
picam2.configure(preview_config)

picam2.start()
time.sleep(2)

raw = picam2.capture_array("raw")
print(raw.shape)
print(picam2.stream_configuration("raw"))

ERROR
[72:40:04.172678509] [7765]  INFO RPI raspberrypi.cpp:1374 Registered camera /base/soc/i2c0mux/i2c@1/imx477@1a to Unicam device /dev/media0 and ISP device /dev/media2
{'use_case': 'preview', 'transform': <libcamera.Transform 'identity'>, 'colour_space': <libcamera.ColorSpace 'sYCC'>, 'buffer_count': 4, 'main': {'format': 'XBGR8888', 'size': (640, 480)}, 'lores': None, 'raw': {'format': 'SBGGR12_CSI2P', 'size': (4056, 3040)}, 'controls': {'NoiseReductionMode': <NoiseReductionModeEnum.Minimal: 3>, 'FrameDurationLimits': (100, 83333)}, 'display': 'main', 'encode': 'main'}
[72:40:05.956939345] [7764]  INFO Camera camera.cpp:1035 configuring streams: (0) 640x480-XBGR8888 (1) 4056x3040-SBGGR12_CSI2P
[72:40:05.957404833] [7765]  INFO RPI raspberrypi.cpp:761 Sensor: /base/soc/i2c0mux/i2c@1/imx477@1a - Selected sensor format: 4056x3040-SBGGR12_1X12 - Selected unicam format: 4056x3040-pBCC
qt.qpa.xcb: QXcbConnection: XCB error: 148 (Unknown), sequence: 192, resource id: 0, major code: 140 (Unknown), minor code: 20
(3040, 6112)
{'format': 'SBGGR12_CSI2P', 'size': (4056, 3040), 'stride': 6112, 'framesize': 18580480}

Dear developer, I would like to save raw images

Hi,  I'm not quite sure what you're trying to do, the following line captures a raw image and stores
the result in a numpy array:
raw = picam2.capture_array("raw")

If you want to save a raw image to a file that can be loaded in an image editor capture_dng.py or capture_dng_and_jpeg.py might be more handy for you.

thanks chris. and how can i save .raw file?

capture_dng.py will save you a DNG file, which is a raw image format that can be loaded into image editing software ( https://en.wikipedia.org/wiki/Digital_Negative )

from picamera2 import *
from time import sleep
from fractions import Fraction

with picamera2.Picamera2() as camera:

        config = camera.create_still_configuration(main={"size": (600, 400)},raw={}, display=None)
        camera.configure(config)

        camera.set_controls({"ExposureTime": 120, "AnalogueGain": 1, "ColourGains": (1.8,1.8)})
        camera.start()

        r = camera.switch_mode_capture_request_and_stop(config)
        r.save("main", "full.jpg")
        r.save_dng("full.dng")

        print(camera.capture_metadata())

        camera.stop()

        camera.close()

how can i change the resolution? Its always 2028x1080?
why does it not print metadata?
kidn regards

Hi, DNGs are fundamentally different in that we're saving the exact data that came out of the image sensor. Generally speaking (and certainly for all Raspberry Pi supported sensors) image sensors expose only a limited number of "modes", and you can't get any other resolutions in a DNG file beyond those that correspond to one of these sensor modes.
You can find out exactly what sensor modes are available with
from picamera2 import Picamera2
from pprint import *

picam2 = Picamera2()
pprint(picam2.sensor_modes)

These is some further explanation in the "Raw Streams" section on page 19 of the manual.

ok great, but how can i set teh different formats in code?
```

with picamera2.Picamera2() as camera:
		config = camera.create_still_configuration(main={'format': 'SRGGB10', 'size': (600, 400)}, raw={}, display=None)
		camera.configure(config)

		camera.set_controls({"ExposureTime": 60, "AnalogueGain": 1, "ColourGains": (1.8,1.8)})
		camera.start()

		r = camera.switch_mode_capture_request_and_stop(config)
		r.save("main", "1.tiff")
		r.save_dng("1.dng")
		camera.stop()

		camera.close()

 self.check_stream_config(camera_config["main"], "main")
  File "/usr/lib/python3/dist-packages/picamera2/picamera2.py", line 555, in check_stream_config
    raise RuntimeError("Bad format " + format + " in stream " + name)
RuntimeError: Bad format SRGGB10 in stream main

great thank you david, I managed to set the SRGGB10 format, which is great, because it reaches 31us shutter speed!
Is there any way to catch metadata equal the way you showed me with jpg ?
	with picamera2.Picamera2() as camera:

			config = camera.create_still_configuration(main={"size": (600, 400)}, raw={"format": "SRGGB10"}, display=None)
			camera.configure(config)

			camera.set_controls({"ExposureTime": 31, "AnalogueGain": 1, "ColourGains": (1.0,1.0)})
			camera.start()

			print(camera.capture_metadata())
			r = camera.switch_mode_capture_request_and_stop(config)
                        r.save("main", 2.jpg")
			r.save_dng("2.dng")

			buffers, metadata = camera.switch_mode_and_capture_buffers(config, ["main", "raw"])
			camera.helpers.save(camera.helpers.make_image(buffers[0], config["main"]), metadata, "2.jpg")
			camera.helpers.save_dng(buffers[1], metadata, config["raw"], "2.dng")
			print(metadata)

                       camera.stop()
                      camera.close()

unfortunalty it does print nothing. i wanted to get sensortemperature etc.. :]

Hi, there are a few things going on here, let me go through them:

You're only using one configuration here, so I think there's no point switching camera modes (it will only take time and switch to the same mode again). So I'd use just capture_request instead of switch_mode_capture_request_and_stop.
The capture that you have after that probably doesn't work because the camera is now stopped; you'd have to restart it. You'd also have to release the request r because still configurations by default only use 1 request, so you'll never get another image until you release it. But in any case, I don't think you need this capture at all.
I often use dcraw to have a quick look at raw files. Use dcraw -w 2.dng (sudo apt install dcraw if you don't have it).

To summarise, I'd probably try some code like this:
import picamera2

with picamera2.Picamera2() as camera:
    config = camera.create_still_configuration(main={"size": (600, 400)}, raw={"format": "SRGGB10"})
    camera.configure(config)
    camera.set_controls({"ExposureTime": 31, "AnalogueGain": 1, "ColourGains": (1.0,1.0)})
    camera.start()

    r = camera.capture_request(config)
    r.save("main", "2.jpg")
    r.save_dng("2.dng")
    print(r.get_metadata())
    r.release()

hey David how are you? - thank your for you snippet, that helped me alot!
In any case its great that I can use raw camera sensor data, because image compression algorithm like jpeg introducing nonlinearities in the pixel values, which affects the process of estimating solar irradiance.
Is it actually also possible to get .tga files from the raw modes?

Python's PIL module actually supports TGA files, so what you want is almost very easy. Except that I think there are some problems with the fact that you would need to save 16-bit pixels if you don't want to lose any of the raw data. Kind of like this:
from picamera2 import Picamera2
from PIL import Image
import numpy as np

picam2 = Picamera2()
config = picam2.create_still_configuration(raw={"format": "SBGGR10"})
picam2.configure(config)
picam2.start()

array = picam2.capture_array("raw")
array_16bit = array.view(np.uint16) * 64 # convert 10-bit values to 16
im = Image.fromarray(array_16bit, 'I;16L')
im.save("test.tga")

The catch is that the save operation fails - PIL's TGA plugin appears not to support 16-bit formats. I'm not sure if there's anything else out there that does, so I don't really know what else to try. But how is it you want to use these files? DNG seems fairly standard for raw files, and Python's rawpy module will read them quite happily.

ahh damn that would have been great solution. I use a c programm for transforming the fisheye images. maybe i have to modify the code. https://github.com/Bra1nsen/fish2skyplane
when I use:
libcamera-raw -t 1000 --ev -10 --awbgains 1.0,1.0 --gain 1.0 --analoggain 1.0 --denoise off --rawfull 1 --shutter 9000
I get a .raw file. How do I get that srggb format in python?
I'm so thankful for your support david, I really don't know what I would do without you.

The ".raw" files saved by libcamera-raw are flat binary files with no formatting data whatsoever. There's a console message that tells you the exact dimensions and format, but then it's up to you.
You can make like slightly easier by adding --mode 4056:3040:12:U to the command which will force the raw file to contain unpacked pixels, that is, each pixel (still with a 12-bit dynamic range) will occupy 2 bytes, as opposed to "packed" format where 2 pixels occupy 3 bytes.
Anyway, once you've got your unpacked format raw file (let's assume you have a single frame which should be about 24MB), you can load it into Python simply with
import numpy as np
array = np.fromfile("test.raw", dtype=np.uint16).reshape((3040, 4056))

Looking back at the earlier code snippet, this actually gives you the equivalent array to
...
array = picam2.capture_array("raw").view(np.uint16)

libcamera-raw -t 1000 --ev -10 --awbgains 1.0,1.0 --gain 1.0 --analoggain 1.0 --denoise off --rawfull 1 --shutter 90 --mode 4056:3040:12:U -o packed.raw
import numpy as np

array = np.fromfile("packed.raw", dtype=np.uint16).reshape((3040, 4056))

print(array)

ValueError: cannot reshape array of size 49418240 into shape (3040,4056)
packed.raw ~ 100MB
https://ufile.io/ph03hi1n

Looks like you've captured multiple raw images into the same file - which is the default behaviour, it writes them all to the same file. You can use -o test%d.raw --segment 1 on the command line if you want each written to a separate file.

it created automatically test.raw test0.raw test1.raw test2.raw and test3.raw        O.O

How big is test0.raw? It should be a single raw frame, which in SBGGR12 format should be about 24MB. After loading with array = np.fromfile("test0.raw", np.uint16) you should find that len(array) is about 12 million. What numbers are you seeing?

yep 23,4Mb and  ~12 million..
ValueError: cannot reshape array of size 12354560 into shape (4056,3040)

What Iam looking at ;D?

https://ufile.io/f/fkr77

Goood Morning David, I hope you slept well. In fact it did work (3040,4064).
Some questions have come up again:

On the one hand we have the de-bayered image formats (XBGR8888, XRGB8888, RGB888, BGR888 and YUV420) and on the other hand the ones listed as "raw streams" which has not yet been de-bayered.

I would like to know how I set exactly which image format is currently being used. (with app nad if possible python)
All formats are .raw files, aren't they? With libcamera python you couldn't generate a .raw file at all, right?
pi@allskyLA:~ $ libcamera-raw -t 1000 --ev -10 --awbgains 1.0,1.0 --gain 1.0 --analoggain 1.0 --denoise off --rawfull 1 --shutter 90 --mode 4056:3040:12:U -o test%.raw --segment 1

Hi, not totally sure I understand all the questions, but let me answer what I can.

libcamera-raw only outputs these flat binary raw (Bayer) files, nothing else. You can force the use of a particular raw sensor mode with the --mode parameter. You can choose either packed or unpacked output.
The other libcamera-apps output processed (non-Bayer) images, though libcamera-still can output the raw Bayer image to a DNG file as well.
In Picamera2 you have (up to) three streams, "main", "lores" and "raw". The "main"  and "lores" are always processed non-Bayer, and the "lores" must further be a YUV and not an RGB format. This really is how the hardware on the Pi works (diagram on page 15 of the manual).
The "raw" stream is always a raw Bayer image. Actually you have very little choice over the format, it must be one of the formats listed in picam2.sensor_modes. The only further choice you have is whether the pixels are packed or unpacked.
Probably worth re-reading the "Raw streams" section on page 19 of the manual if you haven't done so.

In Python I expect you can easily write the numpy raw image array to a ".raw" file , I think the tofile method (the opposite of fromfile which we used earlier) would do it.
Let me know if there's anything I haven't answered!

Unfortunalty DNG pecifications are not public and I havent found a complete third party programming API.
The only difference between XRGB8888 and RGB888 and uncompressed TGA is a header in TGA that defines the
dimensions, bit depth and some other parameters.
I would like to know, how to get a raw non-Bayer image like XRGB8888. And how can I check the format after the image was taken?

I wonder if there's a bit of confusion over the use of the word "raw". I always use "raw" to mean the raw Bayer image coming straight off the image sensor. Unfortunately, there's also a historic use of the word "raw" to mean "uncompressed" or "unencoded" (video people often talk about "raw video" when they mean "uncompressed video", for example).  In such circumstances I avoid "raw" and try to use the most appropriate alternative.
I also use the word "processed" meaning an image that has been through the ISP (Image Signal Processor). The only thing that converts a raw image to a processed (i.e. non-raw) image is the ISP, and that "raw", at least in the way that Picamera2 uses it, actually has nothing to do with "uncompressed".
So to be clear, do you want raw (i.e. Bayer), or are you after something else? If you want a processed RGB image but wish to avoid lossy compression, PNG is quite often a good choice. There are also options that might involve changing what the ISP does to give you different kinds of processed image. Can you explain a little more about what you're after, perhaps?
(The DNG specification is here, but it is quite a complicated file format based on TIFF which means you don't want to be writing your own code to handle it!)

Youre absolut right, thanks for making that clear. So currently iam after a processed [R,G,B] image like RGB888.
The idea is to debayer it externally to transform it afterwards with fish2skyplane (where dng arent fitting yet, its just too difficult)

I would like know how to get a XBGR8888 and/or XRGB888 and/or RGB888 file

Picamera2 will easily give you RGB888 images on its "main" stream, and you can save these as JPEG, PNG, BMP. From what I said earlier, I think even tga might "just work". Or you could save them as unformatted binary files using numpy.tofile. Does this cover what you need?

Ok thanks, yes it does, sorry if I'm a bit difficult. I guess I'm still missing the basics.
As we tried .tga didnt work or?
The problem with image compression algorithm like jpeg, png, bmp is that there are introducing nonlinearities in the pixel values, which then affects the process of estimating solar irradiance. So i was looking for a "raw" RGB888 file. Or am I wrong ?

tga didn't work when trying to save a (true) raw Bayer file, but once you have a non-raw file I think it would:
from picamera2 import Picamera2
picam2 = Picamera2()
picam2.start()
picam2.capture_file("test.tga")

You're right that processed images have an inverse gamma transform applied to them, making the non-linear in terms of brightness. You can stop the ISP from doing that, but I wouldn't say that's a good idea if you want to use the images as "normal" images later.
I seem to recall discussing the "Lux" metadata value before. Now that is always a linear measure, though it's for the whole image and you'd have to calibrate what those units mean for you. But still, double the illumination and the lux value will double too.
Other options might include undoing the inverse gamma transform, this wouldn't be too hard. Or you could take the (true) raw data and de-Bayer it - the number of operations required could be really quite minimal, depending on what you want.

Maybe I should give you a bit more context. The idea is to feed a neuronal network with the sky images labeled with solar irradiance.

The artifical network ( i use VertexAI from google) will work best with raw values because they capture the radiometric characteristics of the scene (which iam interessted in)

Thanks for the info. First of all, let me see if I understand!
You're going to want 3-channel RGB images (top row of pictures) which you will crop and de-fisheye (middle row). Then you're going to want these in quite a low resolution (last row) to feed to a neural network. Finally, the neural network will output a "solar irradiance" value? (Just for my curiosity, is that a single number or something?)
If I've got that right then I'll follow up with some suggestions on these two approaches...

yes exactly, one way will be through generating a hdr image via exposure fusion, another one will be through feeding the neuronal network with a tensor of all images taken.
The target is DNH - global radiation which is measured in W/m²
Moreover I will add weather data to the model like wind direction/speed to further improve the prediction. Its about seconds to minutes.

Here you can see the solarpower of some days, if there were no clouds teh should be a perfect hill. Abrupt drops in the performance of a photovoltaic system are problematic for the grid.

You're going to want 3-channel RGB images

It would also be very interesting to use a non raw Bayer Matrix (like RGB888) image. We pretend it's a monochrome picture.
Of course for us humans the image would probably make no sense, but I feel like the AI will see it differently.

OK, thanks!
So I think your first idea was to use the raw Bayer image. It sounds like you'd have to "de-Bayer" this yourself in order to make it work with your other software. Here are the steps you would need - you'd have to think which of these are important.

Black level removal. This is easy, just set each pixel to max(0, pixel - 64) for 10-bit modes, or max(0, pixel - 256) for 12-bit modes.
Lens shading correction. This is more difficult, but if your lenses don't show too much of this, and you crop only a portion from the centre of each image, you may not need it.
Denoise. Again, you may not need this if you're down-scaling quite a lot.
De-Bayer. You could trivially replace each 2x2 BGGR quad by a single (R, (G1+G2)/2, B) pixel. There's a one-pixel shift of some of the colour channels, but if you're downscaling quite a bit it shouldn't matter. This procedure would also give you a quarter resolution image.
Colour matrix. If you want proper colours (sRGB typically) then you'll need to apply a suitable 3x3 colour matrix to each RGB pixel.
After this the ISP would normally do the gamma transform and scaling.

I think your other plan involved letting the ISP do the de-Bayering, which would do all the above for you. You could set the format to 'RGB888' (or maybe 'BGR888') and choose whatever output resolution you want. If these images are to be consumed only by the neural network, you could turn off the inverse gamma transform completely. Try this:
from picamera2 import Picamera2

tuning = Picamera2.load_tuning_file("imx477.json")
contrast = Picamera2.find_tuning_algo(tuning, "rpi.contrast")
contrast["ce_enable"] = 0
contrast["gamma_curve"] = [0, 0, 65535, 65535]

picam2 = Picamera2(tuning=tuning)
picam2.start(show_preview=True)

from picamera2 import *

tuning = Picamera2.load_tuning_file("imx477.json")
contrast = Picamera2.find_tuning_algo(tuning, "rpi.contrast")
contrast["ce_enable"] = 0
contrast["gamma_curve"] = [0, 0, 65535, 65535]

ExposureTime_1 = 31
ExposureTime_2 = 120
ExposureTime_3 = 350
ExposureTime_4 = 650
ExposureTime_5 = 1000
ExposureTime_6 = 1500

def exposure_bracket():

	with picamera2.Picamera2(tuning=tuning) as camera:

			config = camera.create_still_configuration(main={"size": (4056, 3040),"format": "RGB888"})
			camera.configure(config)
			camera.set_controls({"ExposureTime": ExposureTime_6, "AnalogueGain": 1, "ColourGains": (1.0,1.0)})
			camera.start()
			camera.capture_file("#6.tga")

			camera.stop()
			camera.close()

exposure_bracket()

Thats the current snippet, Iam wondering if it does matter which image format I choose (RGB888, BGR888, XBGR8888, XRGB8888) when it gets transformed to .tga or .jpg

Would it be possible to use sensor mode SRGGB10 while using main? Because so one could further decrease shutter speed to 31us

And for the debayering approach, which way would you save these bayer matrixes/images - with numpy.tofile?

The 24 bit formats will use slightly less memory. For jpeg I'd expect no difference in the saved file, for tga the 32 bit images may get saved with an alpha channel.
No problem using the 10 bit sensor mode.

Well, once you've debayered the image and got regular RGB you can take your pick... jpeg, png, tga, "tofile", whatever is easiest!

Yes sure, but first I have to provide my friend a RGB888 or XRGB8888 file for debayering.

how can I set the 10 bit sensor mode in my snippet david?

The raw Bayer image just isn't RGB888, you have only one sample at each pixel location. You could save it as DNG, or using tofile (in this case the receiver would have to know what to expect).
You could even pretend it's a greyscale image and you might be able to save that as a 16 bit PNG, but it would depend on whether the libraries you're all using support that.

How would it look like to get a raw Bayer matrix as DNG file in RGB888. I mean I cant use RGB888 in raw format and with main I cant get raw images I thought..
		   	with picamera2.Picamera2() as camera:

		    config = camera.create_still_configuration(main={"size": (600, 400), "format": "RGB888"}, raw={"format": "SRGGB10"})
		    camera.configure(config)
		    camera.set_controls({"ExposureTime": ExposureTime_1, "AnalogueGain": 1, "ColourGains": (1.0,1.0)})
		    camera.start()
		    r = camera.capture_request(config)
		    r.save("main", "2.dng")
		    r.save_dng("2.dng")
		    metadata= r.get_metadata()
		    print(metadata)
		    print(metadata["Lux"])
		    r.release()

		    camera.close()

I worry a bit that we're not talking about pixel formats in quite the same way.
When I talk about RGB888 or BGR888 images, each pixel has 3 values (R, G and B). If you capture such an image and ask for its shape, it will report (height, width, 3). You can save these as normal images such as JPEG, PNG, BMP or TGA.
There are also the XBGR8888 formats. Here there's a 4th channel which is set to a fixed value (255, I think). You can normally save these as regular images too. Some formats (JPEG for example) will drop the 4th channel. Other formats may preserve it as an alpha channel.
Then there are the SBGGR10 type formats. These are raw Bayer formats. They all begin with S, the BGGR may change to indicate the Bayer pattern order and the 10 is the bit depth of the samples. Capturing one of these images and querying the shape will yield (height, stride) - only 2 dimensions this time and not 3. These are not RGB888 images and you can't save them as a normal image. You can save them as DNG files, or you can "roll your own format" using numpy.tofile.
Does that make things clearer? The code you have above looks to me like it will set up a 600x400 RGB888 main stream, and a raw stream which will probably choose the 1332x990 10-bit mode (the console log messages will confirm). Saving the "main" stream images as .dng doesn't really make sense - maybe you'll get a JPEG file with a funny name, I'm not sure. But the code looks to be saving the "raw" stream image as I would expect, in DNG format.

ahh okay my question was obvisouly not thought through. A raw Bayer Matrix has only 2 dimensions and cant be RGB888 format.
one question remains david, how can I set the 10bit sensor mode for my .tga file snippet in code? by default its srggb12

from picamera2 import *

tuning = Picamera2.load_tuning_file("imx477.json")
contrast = Picamera2.find_tuning_algo(tuning, "rpi.contrast")
contrast["ce_enable"] = 0
contrast["gamma_curve"] = [0, 0, 65535, 65535]

ExposureTime_1 = 31
ExposureTime_2 = 120
ExposureTime_3 = 350
ExposureTime_4 = 650
ExposureTime_5 = 1000
ExposureTime_6 = 1500

def exposure_bracket():

	with picamera2.Picamera2(tuning=tuning) as camera:

			config = camera.create_still_configuration(main={"size": (4056, 3040),"format": "RGB888"})
			camera.configure(config)
			camera.set_controls({"ExposureTime": ExposureTime_6, "AnalogueGain": 1, "ColourGains": (1.0,1.0)})
			camera.start()
			camera.capture_file("#6.tga")

			camera.stop()
			camera.close()

exposure_bracket()

Thats the current snippet, Iam wondering if it does matter which image format I choose (RGB888, BGR888, XBGR8888, XRGB8888) when it gets transformed to .tga or .jpg

The line you had previously:
config = camera.create_still_configuration(main={"size": (600, 400), "format": "RGB888"}, raw={"format": "SRGGB10"})

should give you the 10-bit sensor mode. As I said, the console messages will confirm exactly what it chose, but it looks right to me. The sensor only has one 10-bit mode IIRC, which is the 1332x990 one. All other modes will give you 12 bits. (Use, for example, libcamera-hello --list-cameras to see exactly what modes are supported.)
If you capture one of the raw frames as an array (camera.capture_array("raw")) then you should find all the even columns have values from 0-255, and the odd columns only 0-3 (being just the top two bits of the 10-bit values).

	with picamera2.Picamera2(tuning=tuning) as camera:

			config = camera.create_still_configuration(main={"size": (4056, 3040),"format": "RGB888"}, raw={"format": "SRGGB10"})
			camera.configure(config)
			camera.set_controls({"ExposureTime": ExposureTime_6, "AnalogueGain": 1, "ColourGains": (1.0,1.0)})
			camera.start()
			camera.capture_file("#6.tga")

			camera.stop()
			camera.close()

[190:58:21.599431415] [32174]  INFO Camera camera_manager.cpp:293 libcamera v0.0.0+3866-0c55e522
[190:58:21.629259862] [32177]  INFO RPI raspberrypi.cpp:1374 Registered camera /base/soc/i2c0mux/i2c@1/imx477@1a to Unicam device /dev/media0 and ISP device /dev/media2
[190:58:21.633988892] [32174]  INFO Camera camera.cpp:1035 configuring streams: (0) 4056x3040-RGB888 (1) 4056x3040-SBGGR12
[190:58:21.634475252] [32177]  INFO RPI raspberrypi.cpp:761 Sensor: /base/soc/i2c0mux/i2c@1/imx477@1a - Selected sensor format: 4056x3040-SBGGR12_1X12 - Selected unicam format: 4056x3040-BG12

#6.tga is (4056,3040), Where is my misconception?

Ahh okay I had to change the resolution in main to make it work :)!
config = camera.create_still_configuration(main={"size": (1332, 990),"format": "RGB888"}, raw={"format": "SRGGB10"})

But the picture is cropped, which of course is no fun. I thought the picture stays the same and only the resolution changes -.-

If you don't give a resolution for the raw stream it will guess what it thinks is the "best" one for you. Originally, because the main stream resolution was so large it was selecting a bigger (non 10 bit) mode.
By selecting a lower main stream resolution it decided the 10 bit mode was close enough and gave you that. You can force the choice of sensor mode by setting  size for the raw stream (as well as the format), and then it will use that regardless of the main stream resolution.
The other thing to say is that the 10 bit mode is quite heavily cropped. If you do this
from picamera2 import Picamera2
camera = Picamera2()
camera.sensor_modes

It will list all the available camera modes and exactly what field of view (crop from the full pixel array) you will get.

David your support is great .I appreciate it very much.
The following idea is still going through my head, namely:
I would like to load a Bayer matrix as a numpy array.
Can you tell me how? Basically a (4056,3040) matrix in which each entry corresponds exactly to the strength of the incident radiation. Nothing else - just the intensity. (!!!)

Or maybe even different size, just no cropping =)!

If you want the raw unprocessed Bayer image just use, for example
from picamera2 import Picamera2
picam2 = Picamera2()
config = picam2.create_still_configuration(raw={"format": "SBGGR12"})
picam2.configure(config)
picam2.start()
array = picam2.capture_array("raw")

This will default to the full resolution, full field of view mode. The shape of the array should be something like (3040, 8192) because it's one value per pixel (so no 3rd dimension). The width is 8192 because it's 2 bytes per pixel (12-bits of dynamic range) plus some extra padding on the end of each row.
As we've discussed previously, you could save that as a flat binary file (using tofile), or you could capture and save as a DNG. Some of the processing steps to think about are the ones I outlined in an earlier post.
I would also still consider the version where you use a processed image but disable the gamma (by tweaking the tuning). This would have the benefit of doing all the processing for you, give you any resolution you want and an output in standard sRGB but with linear pixel intensities.

Perhaps this is going too far, but where exactly does the information processing process begin?

When photons hit the CMOS sensor (IMX477), an energetic interaction occurs, which releases electrons.
These are amplified and measured. then quantified into a binary code using a transistor I guess.
What is the "raw" array I see here? Is the ISP feed with that?
[215:37:58.510468279] [32471] INFO RPI raspberrypi.cpp:761 Sensor: /base/soc/i2c0mux/i2c@1/imx477@1a - Selected sensor format: 4056x3040-SBGGR12_1X12 - Selected unicam format: 4056x3040-BG12
[[ 4 1 12 ... 0 0 0]
[11 1 3 ... 0 0 0]
[ 9 1 11 ... 0 0 0]
...
[10 1 2 ... 0 0 0]
[ 8 1 7 ... 0 0 0]
[10 1 4 ... 0 0 0]]
(3040, 8128)

Yes, I think that's pretty much it. The signal that comes off the image sensor is saved directly to memory. The only thing that can happen to it at this point is the optional unpacking, because that's handled by the CSI-2 (camera interface) receiver which actually writes to memory. This block of memory is exactly what you get fed as the raw stream images, it's also exactly what gets fed to the ISP (as per the diagram I pointed out in an earlier post).

The direct output of the sensor is  a binary file in format of that matrix (3040, 8128)?
There is no tool to show this matric or? I would like to analyse the whole matric.

I could use numpy I guess

The direct output of the sensor is that (3040, 8128) array. Of course the sensor outputs to the chip's CSI-2 receiver via a serial link, so to the sensor it's just 3040x4056x12 bits. It's the CSI-2 receiver that lays them out in memory as a (3040, 8128) array of uint8s.
As regards visualising this array... well that's kind of what the ISP is for. You could also view the array as a (3040, 4064) array of uint16s then you could pretend it's a single channel 16-bit greyscale image.  It would look a bit weird (you'd notice the alternating colours in the Bayer pattern very obviously).
One benefit of processing through the ISP is that you get something "standard" out - an sRGB image. The raw Bayer image is particular to each kind of sensor. If you trained a neural network on these raw images it would never work with anything but imx477. Depends if that's an issue for you or not.

Once the neuronal network "understood", its can be containerized as pretrained network architecture for an exact same camera setup. I would be able to send you an identical camera setup which is already capalbe  measuring and predicting Solar Radiation in W/m² . It would get even more accurate with time .
I want that Matrix with alternating colours in the Bayer pattern in an uncompressed Bitmap (.bmp).
Moreover I would like to take n-matrices for generating a tensor (alternative to multi exposure fusion).
Of course I also will test sRGB images, maybe they perfrom even better. How can one output that Bayer pattern looking image?

class numpy.ubyte[source]
Unsigned integer type, compatible with C unsigned char.

Character code
'B'

Alias on this platform (Linux x86_64)
numpy.uint8: 8-bit unsigned integer (0 to 255).

Do I need something like that?

Once you have the 2d raw Bayer array (for example using array = camera.capture_array("raw")) then you can view it as 16-bit samples using array = array.view(np.uint16). I would expect that OpenCV would be able to save that directly as a PNG. Normally I would expect PNGs to support 16-bit values and greyscale images (which is what it would think this is). I don't have the means to test that at the moment, but I'd have expected something like cv2.imwrite("test.png", array) to work. (BMP may make your life harder as I suspect it may only support 8-bit pixels.)
(Disclaimer: I haven't actually tried this...)

If I create an array of say (30,30,3) and save it as a jpg, how does image compression play a role when the computer processes that jpg file?
Or to put it another way: Can I save an array as a jpg, which has absolutely nothing to do with the jpeg image compression algorithm?

import json
import numpy as np
import cv2

from picamera2 import Picamera2
picam2 = Picamera2()
config = picam2.create_still_configuration(raw={"format": "SBGGR12"})
picam2.configure(config)
picam2.set_controls({"ExposureTime": 100, "AnalogueGain": 1, "ColourGains": (1.8,1.8)})
picam2.start()
array = picam2.capture_array("raw")
print(array)
print(array.shape)

array = array.view(np.uint16)

cv2.imwrite("test0.png", array)

This Image should represent the pure radiometric characterstics in greyscale, right?
But how do we get the alternating colour Bayer pattern image?

The code you have there looks OK to me, though this image is a bit strange. It seems to be 32-bit and not 4056x3040 in size, perhaps you processed it before posting it?
But in any case, it looks like what I would expect. The pixel values are exactly what came off the sensor, though you must as a minimum subtract the black level to get a linear signal.
Once you've got that array, another thing you could do is this:
rgb_array = np.zeros((1520, 2028, 3))
rgb_array[:, :, 0] = array[1::2, 1::2] # red
rgb_array[:, :, 1] = (array[0::2, 1::2] + array[1::2, 0::2]) / 2 # green
rgb_array[:, :, 2] = array[0::2, 0::2] # blue

and then you've got a 3-channel RGB image. It would look gloomy and pale, but might be helpful, and you could save it as anything that supports 16-bit pixels (that might just be PNG, not sure).
As regards JPEG files, there is a lossless version of JPEG but I don't know how widely implemented it is. I'll bit JPEG implementations mostly only handle 8-bit pixels too.

that cant be the result?

Ah, maybe use rgb_array = np.zeros((1520, 2028, 3), dtype=np.uint16). I didn't specify the datatype because I was worried about overflows, but the pixels only have 12 bits of range so the G calculation can't overflow. (You have to be a bit careful with the datatype of the array as the value for "white" is often inferred simply from that - 255 for uint8, 65535 for uint16 and 1.0 for floats which may be what happened to you here.)

Interesting. The thing to remember about those raw images is that there's been no processing done to them at all. Not even the colour gains have been applied, so mostly stuff will probably look pale, dark and green. Also remember that the pixels are 12-bit, so you might need to scale everything up by 16x if you want to treat it as a 16-bit image. And don't forget to subtract the black level!

I would also still consider the version where you use a processed image but disable the gamma (by tweaking the tuning). This would have the benefit of doing all the processing for you, give you any resolution you want and an output in standard sRGB but with linear pixel intensities.
--> Back to that approach

from picamera2 import *

tuning = Picamera2.load_tuning_file("imx477.json")
contrast = Picamera2.find_tuning_algo(tuning, "rpi.contrast")
contrast["ce_enable"] = 0
contrast["gamma_curve"] = [0, 0, 65535, 65535]

ExposureTime_1 = 315
ExposureTime_2 = 5120
ExposureTime_3 = 7350
ExposureTime_4 = 11650
ExposureTime_5 = 25000
ExposureTime_6 = 50000

def exposure_bracket():

	with picamera2.Picamera2(tuning=tuning) as camera:

			config = camera.create_still_configuration(main={"size": (4056, 3040),"format": "RGB888"})
			camera.configure(config)
			camera.set_controls({"ExposureTime": ExposureTime_6, "AnalogueGain": 1, "ColourGains": (1.0,1.0)})
			camera.start()
			camera.capture_file("#6.tga")

			camera.stop()
			camera.close()

exposure_bracket()

You said these images are also good for viewing the radiation intensity, is there anything else which could be important here?

I can't think of anything specific. They will be fully processed images with the colour gains that you requested, but without the gamma transform so the pixel levels will be linear with regard to the incident light intensity. You can also ask for any output resolution (pretty much) that you want, but use raw={"size": (4056, 3040)} in the configuration if you want to guarantee using the full res mode all the time, even if the output resolution is much reduced.
There are some other processing blocks you might want to consider disabling. For example, there's probably not much point in applying any image sharpening, though I doubt it hurts either. The colour matrix that it uses will mix the different colour channels together so you might prefer  not to do that. If you want to make several edits to the tuning file it might be easier actually to edit it, rather than to load it in Python and tweak it (as we did previously). If you're interested in trying any of that, let me know and I can explain (it's not hard).

Ok lets edit the tuning file improving raw radiometric characterstics (!!!)
I would like to apply the changes you are suggesting David. Could you maybe provide a reasoned tuning file. The camera sensor represents an eye, a sense, for the artificial neural network. We are currently configuring this sense.
Iam exicted to generate the first dataset of 30 days solarpower labeld sky images for VertexAI (machine learning). Transforming .tga to .bmp would be lossless or?

Try this tuning file: https://drive.google.com/file/d/1-olj82VusLTb6o68oQopO4ypYsS-iLGM/view?usp=sharing
It's the same as the standard one but with the gamma transform and colour matrix set to the identity, and sharpening turned off. Load and use it as we did before, for example:
...
tuning = Picamera2.load_tuning_file("/path/to/downloaded/file")
picam2 = Picamera2(tuning=tuning)
...

I expect you'll still want to set the colour gains to 1.0 to stop AWB from doing anything. Still captures by default produce regular RGB888 images which you can save as JPEG, PNG, BMP, TGA. Conversions between lossless formats (that's all of them except JPEG) should be, well, lossless!

Today I will start generating the first dataset for training a neuronal network to learn measuring global radiation [W/m²] Solarpower.

back to the raw:
libcamera-raw -t 1000 --ev -10 --awbgains 1.0,1.0 --gain 1.0 --analoggain 1.0 --denoise off --rawfull 1 --shutter 9000 --mode 4056:3040:12:U -o /mnt/rd/test%05d.raw --segment 1
Why did it combine multiple raw "images" last time?
Why do I need this 'test%05d.raw --segment1'?

I will probaly write a bash script to generate mutliple .raw one with different shutter speeds.

libcamera-raw by default saves all the raw files back-to-back, as many as come out of the camera, so there is always likely to be a frame or so of variability. That stuff with --segment 1 tells it to split the single output file into a new file every 1ms (so each frame would get a new file). The %05d gets turned into an incrementing counter.
There's no particularly easy way to get just a single frame. Really, that's what Picamera2 is for. You can start the camera system and configure the camera just once, and then do "set exposure - start - capture - stop" multiple times. I don't think there'd be any need for a 1s delay anywhere, either.

So its a challenge to get exact 10 .raw files with different exposure times...
contrast = Picamera2.find_tuning_algo(tuning, "rpi.contrast")
contrast["ce_enable"] = 0
contrast["gamma_curve"] = [0, 0, 65535, 65535]

I can comment that out, since you modified this already in thje imx477.json file right?

Well, it's a challenge to get exactly 10 .raw files. You would generally have some extra ones lying on your disk that you didn't need!
And yes, that tuning file I posted has that change already (you can look inside to check, it's not hard to understand, just search for "gamma_curve").

Yeah I will definitly dive depper into that tuning file. Some questions come up about resolution scaling algorithm.
i) The raw array is (3040, 4064) but we set for (3040,4056) - why?
ii) How are the pixel downscaled, for example if I reduce the image size to (120,90), is it bilinear interpolated?
Moreover the its hard to downscale in mathmatical correct ratio. Using an Image aspect ratio and resizing calculator it should be W:H =1,3342105263157895:1 which would be (120,89.94)
(https://red-route.org/code/image-resizing-calculator)

If I downscale the image of          (4056,3040) [12.300.000pixel]       to           (120,90) [10.800pixel].
Every Pixel would be average of  1138 Pixel - this time insane, these MP camerasensors are world class.
I would like to be able to the save the images (like .tga) as flat binary file. My intension is to save the "images" in a time series database like influxdb. Using UTC Timestamps for automatical label matching. Any Thoughts?

That sounds like a good plan to me. I guess I don't see too many TGA files around these days, but they have a simple file format which perhaps makes them a good choice. BMP would have the same kind of advantages.
 It's so huge you might consider using the 2x2 binned mode if you wanted higher framerates or a slightly lower minimum exposure (you'd need a configuration with raw={"size": (2028, 1520)}). By the time it's been shrunk to 120x90 I don't think you'd see any difference (though you would notice a drop in quality at higher output resolutions).

Thats a very good point, I will go for the 2x2 binning to lower the exposure time.
Where do I find the math behind the applied debayer algorithmn? How is the raw Bayer Matric convertd/transformed to the RGB Matric?

I still think the best way would be to take the array("raw") transforming the dimension (4064,3040,1) to (4064,3040,3) -->splitting R,B,Gb&r.
and then downsizing (4064,3040,3) to (120,90,3).
But I dont know, if the debayer algorithmus is straight forward the neuro network should also work fine with the standard RGB images. Just thinking loud... :)

There are lots and lots of debayering algorithms, many of them really quite sophisticated. The hardware block inside the Pi's ISP has its own algorithm which is definitely at the more complex end of the spectrum! The simplest algorithm is the one referred to as "bilinear interpolation", though the results, especially along straight vertical or horizontal edges is quite poor.
The good news is that, in your case, because of the huge downscale afterwards, almost anything you do will make basically no difference (so long as you're anti-aliasing to some extent). I would probably still just convert each 2x2 BGGR quad into (R, G, B), where G = (Gr + Gb) / 2, as you were suggesting. That will leave you with (2032, 1520, 3), but the downscale to (120, 90, 3) is still huge!

if i use the libcamer-raw app, the shutter speed is in ms. So one wouldnt be able to use the full potential of the camerasensor, since the imx477 is capable of 60us..
libcamera-raw --ev -10 --awbgains 1.0,1.0 --gain 1.0 --analoggain 1.0 --denoise off --rawfull 1 --shutter 1 --mode 4056:3040:12:U -o shutter1.raw --segment 1
And when there are multiple segments combined to one file are alle of these raw images identical? Or would it make a difference if I go for --segment4 for example?
Currently Iam on to create a second dataset in parallel to our first approach with the tuning file.

The --shutter parameter is always in microseconds, I believe. The --segment time is in milliseconds (it's not a frame count), but we have no cameras that run close to 1000fps so there is no practical limitation here!
When you get multiple frames in a single file, they are  simply successive frames that came out of the camera. So they're all different (even just by noise if the scene doesn't change).

 https://www.raspberrypi.com/documentation/accessories/camera.html#libcamera-raw
Ok interesting, they are all different. If I go for --shutter 50260 I get 259Mb file, so exactly 11 frames. And only the last one should be the one with the set shutterspeed of ~ 5s?

All the libcamera-apps set the exposure time (and gain) before starting the camera, as you do in your Python scripts. So all the frames will be at 50ms.

Ok lets finish this one, so that we can generate two datasets with both approaches.
from picamera2 import Picamera2
picam2 = Picamera2()
config = picam2.create_still_configuration(raw={"format": "SBGGR12", "size": (2032, 1520)})
picam2.configure(config)
picam2.set_controls({"ExposureTime": 100, "AnalogueGain": 1, "ColourGains": (1.0,1.0)})
picam2.start()
array = picam2.capture_array("raw")
print(array)
print(array.shape)

array16 = array.view(np.uint16)
cv2.imwrite("greyscale.png", array16)

#safe as greyscale and as rgb

rgb_array = np.zeros((2032, 1520, 3), dtype=np.uint16)
rgb_array[:, :, 0] = array16[1::2, 1::2] # red
rgb_array[:, :, 1] = (array16[0::2, 1::2] + array16[1::2, 0::2]) / 2 # green
rgb_array[:, :, 2] = array16[0::2, 0::2] # blue

cv2.imwrite("rgbraw.png", rgb_array)

So I expected that the generated .png file should store the raw radiometric intensities between 0-4095 for [R,G,B]
Something like that:

but if I print the array I get this:

Hi, so I've modified your script a bit to get something that works. Let me just explain what I changed:

For me the exposure time of 100us yields almost completely black images, so I commented that out and put a short sleep in to be sure the AGC is completed settled.
I multiplied array16 by 16 to bump up the dynamic range to 16 bits. (Having 12-bit numbers in 16-bit pixels is like having normal 8-bit images where every number is less than 16.)
I also found I had to be careful adding the two green values because they overflow the allowed 16-bits (even though the result gets divided back down).
Finally I added some checking code that shows how to reload the saved files. The two saved files look as expected in the default Pi image viewer.

Here's my code:
import time
import numpy as np
import cv2
from picamera2 import Picamera2

picam2 = Picamera2()
config = picam2.create_still_configuration(raw={"format": "SBGGR12", "size": (2032, 1520)})
picam2.configure(config)
# Exposure time of 100us makes everything black...
#picam2.set_controls({"ExposureTime": 100, "AnalogueGain": 1, "ColourGains": (1.0,1.0)})
picam2.start()
time.sleep(0.5)
array = picam2.capture_array("raw")

array16 = array.view(np.uint16)
array16 *= 16  # Bump 12-bit range up to 16
cv2.imwrite("greyscale.png", array16)

rgb_array = np.zeros((760, 1016, 3), dtype=np.uint16)
rgb_array[:, :, 0] = array16[1::2, 1::2] # red
# Must avoid overflowing 16-bits in the addition here:
rgb_array[:, :, 1] = (array16[0::2, 1::2].astype(int) + array16[1::2, 0::2]) / 2 # green
rgb_array[:, :, 2] = array16[0::2, 0::2] # blue

cv2.imwrite("rgbraw.png", rgb_array)

# Load images back and check errors are zero!
grey_check = cv2.imread("greyscale.png", cv2.IMREAD_ANYDEPTH | cv2.IMREAD_ANYCOLOR)
rgb_check = cv2.imread("rgbraw.png", cv2.IMREAD_ANYDEPTH | cv2.IMREAD_ANYCOLOR)
print(np.max(np.abs(array16 - grey_check)))
print(np.max(np.abs(rgb_array - rgb_check)))

hej david the first datasets getting generated and saved in google drive. it will be so clean. :)
import time
import numpy as np
import cv2
from picamera2 import Picamera2

t1 = time.perf_counter()

def exposure_bracket():

	picam2 = Picamera2()
	config = picam2.create_still_configuration(raw={"format": "SRGGB12", "size": (2032, 1520)})
	picam2.configure(config)

	exposures = [50, 100, 150, 250, 400, 650, 1050, 1700, 2750]
	#exposures = [50]

	for exp in exposures:

		picam2.set_controls({"ExposureTime": exp, "AnalogueGain": 1, "ColourGains": (1.0,1.0)})
		picam2.start()
		time.sleep(0.5)
		array = picam2.capture_array("raw")
		#picam2.stop()
		#picam2.close()

		array16 = array.view(np.uint16)
		array16 *= 16  # Bump 12-bit range up to 16
		cv2.imwrite(f"{exp}_greyscale.png", array16)

		rgb_array = np.zeros((760, 1016, 3), dtype=np.uint16)
		rgb_array[:, :, 0] = array16[1::2, 1::2] # red
		# Must avoid overflowing 16-bits in the addition here:
		rgb_array[:, :, 1] = (array16[0::2, 1::2].astype(int) + array16[1::2, 0::2]) / 2 # green
		rgb_array[:, :, 2] = array16[0::2, 0::2] # blue

		cv2.imwrite(f"{exp}_rgbraw.png", rgb_array)

		# Load images back and check errors are zero!
		grey_check = cv2.imread(f"{exp}_greyscale.png", cv2.IMREAD_ANYDEPTH | cv2.IMREAD_ANYCOLOR)
		rgb_check = cv2.imread(f"{exp}_rgbraw.png", cv2.IMREAD_ANYDEPTH | cv2.IMREAD_ANYCOLOR)
		print(np.max(np.abs(array16 - grey_check)))
		print(np.max(np.abs(rgb_array - rgb_check)))

exposure_bracket()

t2 = time.perf_counter()
print(f"Finished in {t2-t1} seconds")

Traceback (most recent call last):
File "/home/pi/test_rawbyer.py", line 47, in
exposure_bracket()
File "/home/pi/test_rawbyer.py", line 22, in exposure_bracket
picam2.start()
File "/usr/lib/python3/dist-packages/picamera2/picamera2.py", line 872, in start
self.start_()
File "/usr/lib/python3/dist-packages/picamera2/picamera2.py", line 835, in start_
raise RuntimeError("Camera already started")
RuntimeError: Camera already started
Whats wrong? :/

If I've read the code right, I think you might just need to put the picam2.stop() back (it seems to be commented out). But you're right that you don't need the picam2.close()

crazy i was so close, it works now!

Iam still not sure about the file format. Is .png lossless, and its best suited for 16 bit right?

Yes thats great. I was thinking about adding a e-mail notification if the check back returns an error, since iam not constantly checking the monitor :)

print(np.max(np.abs(rgb_array - rgb_check)))
ValueError: operands could not be broadcast together with shapes (760,1016,3) (1520,2032)
that would be an error hm?

Not sure what's going on there. The script that you have just above, once you re-instate the picam2.stop() runs fine for me. It prints a pair of zeros after each capture (the errors) and finishes in just under 15 seconds.
You must be running some different code? It looks a bit like your rgb_check array has loaded the greyscale image, and they're completely different dimensions and sizes.

Yes the code works fine! What about .tiff files. Your check back printed the following: (saving as .tiff)
[44:07:00.218389133] [10181]  INFO RPI raspberrypi.cpp:761 Sensor: /base/soc/i2c0mux/i2c@1/imx477@1a - Selected sensor
format: 2028x1520-SBGGR12_1X12 - Selected unicam format: 2028x1520-BG12
65520
65528
65520
65528
65520
65528
65520
65528
65520
65528
65520
65528
65520
65528
65520
65528
65520
65528
Finished in 13.754075371020008 seconds

https://drive.google.com/file/d/19nTkt17TvdINnvFwvyOwOyeCrLDtfcrX/view?usp=sharing, https://drive.google.com/file/d/1x6Xeg8oFu2UbamtlxztKlLmj14zbdz_i/view?usp=sharing

It looks like a sign overflow thing. You're subtracting two uint16s and it gives you the result as a uint16 as well, which is a problem if it was actually negative. You might need this instead
		print(np.max(np.abs(array16.astype(int) - grey_check)))
		print(np.max(np.abs(rgb_array.astype(int) - rgb_check)))

But in any case, there do appear to be some errors. I never much like tiff as a format because it's really a container for something else and I never quite know what I might get.

What about .exr image format? Any experience with that?

	for exp in exposures:

		picam2.set_controls({"ExposureTime": exp, "AnalogueGain": 1, "ColourGains": (1.0,1.0)})
		picam2.start()
		time.sleep(0.5)
		array = picam2.capture_array("raw")
		picam2.stop()
		#picam2.close()

		array16 = array.view(np.uint16)
		array16 *= 16  # Bump 12-bit range up to 16
		arrayX = array.astype("float32")
		cv2.imwrite(f"{exp}_greyscale.exr", arrayX)
		#image = pyTGA.Image(data=array16)
		#image.save(f"{exp}_greyscale", original_format=True)

		rgb_array = np.zeros((760, 1016, 3), dtype=np.uint16)
		rgb_array[:, :, 0] = array16[1::2, 1::2] # red
		# Must avoid overflowing 16-bits in the addition here:
		rgb_array[:, :, 1] = (array16[0::2, 1::2].astype(int) + array16[1::2, 0::2]) / 2 # green
		rgb_array[:, :, 2] = array16[0::2, 0::2] # blue
		rgb_arrayX = rgb_array.astype("float32")
		cv2.imwrite(f"{exp}_rgbraw.exr", rgb_arrayX)
		#image = pyTGA.Image(data=rgb_array)
		#image.save(f"{exp}_rgb")

		# Load images back and check errors are zero!
		#grey_check = cv2.imread(f"{exp}_greyscale.exr", cv2.IMREAD_ANYDEPTH | cv2.IMREAD_ANYCOLOR)
		#rgb_check = cv2.imread(f"{exp}_rgbraw.exr", cv2.IMREAD_ANYDEPTH | cv2.IMREAD_ANYCOLOR)
		#print(np.max(np.abs(array16.astype("float32") - grey_check)))
		#print(np.max(np.abs(rgb_array.astype("float32") - rgb_check)))

i dont know how to adjust the check back accordingly, but it does write the .exr files

Ok in Photoshop theres nothing to see, guess the way i did it is wrong..:/

I can honestly say I've never seen an exr file and know nothing about them! Slightly surprised that OpenCV even supports it...
Perhaps the question is what features you need from the saved files that you don't get from tga or png?

the imagetransformation code is written in C. currently the programm only support 8bit tga and jpeg. now iam chatting with the author to upgrade the image transformation tool for 16/24/32 bit..
he asked for .exr files, because of their high dynamic range properties..
the problem with tga was, that i can only save them as 8bit.. (imageio)

		picam2.set_controls({"ExposureTime": exp, "AnalogueGain": 1, "ColourGains": (1.0,1.0)})
		picam2.start()
		time.sleep(0.5)
		array = picam2.capture_array("raw")
		picam2.stop()
		#picam2.close()

		array16 = array.view(np.uint16)
		array16 *= 16  # Bump 12-bit range up to 16

		imageio.imwrite(f'{exp}g.tga', array16)
		matplotlib.image.imsave(f'{exp}_matplot.tga', array16)

		rgb_array = np.zeros((760, 1016, 3), dtype=np.uint16)
		rgb_array[:, :, 0] = array16[1::2, 1::2] # red
		# Must avoid overflowing 16-bits in the addition here:
		rgb_array[:, :, 1] = (array16[0::2, 1::2].astype(int) + array16[1::2, 0::2]) / 2 # green
		rgb_array[:, :, 2] = array16[0::2, 0::2] # blue

		imageio.imwrite(f'{exp}rgb.tga', rgb_array)
		#matplotlib.image.imsave(f'{exp}_matplot.tga', rgb_array) only 8unt

		# Load images back and check errors are zero!
		grey_check = cv2.imread(f"{exp}g.tga", cv2.IMREAD_ANYDEPTH | cv2.IMREAD_ANYCOLOR)
		rgb_check = cv2.imread(f"{exp}rgb.tga", cv2.IMREAD_ANYDEPTH | cv2.IMREAD_ANYCOLOR)
		print(np.max(np.abs(array16 - grey_check)))
		print(np.max(np.abs(rgb_array - rgb_check)))

it seems like matplotlib can save 16bit greyscale .tga
what about the conversion loss, are the 16 bit scaled to the 8 bit imagewisely correct?
Lossy conversion from uint16 to uint8. Losing 8 bits of resolution. Convert image to uint8 prior to saving to suppress this warning.
Lossy conversion from uint16 to uint8. Losing 8 bits of resolution. Convert image to uint8 prior to saving to suppress this warning.
Traceback (most recent call last):
  File "/home/pi/testrawbayer/pyTGA/tests/xxy.py", line 61, in <module>
    exposure_bracket()
  File "/home/pi/testrawbayer/pyTGA/tests/xxy.py", line 57, in exposure_bracket
    print(np.max(np.abs(array16 - grey_check)))
TypeError: unsupported operand type(s) for -: 'int' and 'NoneType'

Where's that Lossy conversion warning coming from, is it imageio? I'm sure it converts 16 bits to 8 "properly", but you can't get round the fact that it will lose the bottom 4 bits of your original 12-bit images.
Not sure about that unsupported operand error. Maybe OpenCV won't load the files back? Seems like it needs some debugging...

yes imageio reports that message
matplotlib seems to safe greyscale 16bit, which is pretty decent. how could I check that right? Which way would you look up big matrices in numbers? Getting an impression/intuition from the images soley through the numbers ;D
so the .tga are not loaded by opencv, ok so have to comment them out for now ...

I'm afraid I don't really know too much about some of these other file formats and exactly which libraries support them. One problem, as I think you've discovered, is that libraries support formats but "not fully". For example, the developers often think that no one needs more than 8-bit images so they leave out 16-bit support. Mostly that's true, but not always.
That's the main reason why I often use PNG. It is one of the more common formats, which means support for it (including 16-bit) is generally quite good. I've never written code to read a PNG, only to write them (here), but the code is simply copied from the web. I expect there will be more examples out there of how to read a PNG than to write one.

How is it possible that I get pixel values beyond 4095 in our generated raw bayer array? For example [4096 4116 4092]...

rgb_array[:, :, 1] = (array16[0::2, 1::2].astype(int) + array16[1::2, 0::2]) / 2 # green
should I add the brackets so that the green values are added first before dividing them?
rgb_array[:, :, 1] = ((array16[0::2, 1::2].astype(int) + array16[1::2, 0::2])) / 2 # green

It's a 12-bit camera, that's 0 to 2^11 - 1 = 4095. So you can't get anything bigger than that (other than by scaling the numbers, which I think I did in one of the earlier examples, but then the bottom bits are zero instead of the top ones).
If you were feeling super ambitious you could do several Bayer captures and simply add them together.That would give you higher dynamic range!

I'm not making that up. Last night I saved the array as .npy and printed it...
		array = picam2.capture_array("raw")
		picam2.stop()

		array16 = array.view(np.uint16)
		array16 *= 16  # Bump 12-bit range up to 16
		save(f'{exp}_grey.npy', array16)

I'm travelling this weekend, but take a look at https://github.com/raspberrypi/picamera2/blob/next/examples/stack_raw.py

sure thank you for letting me know david, have fun! =)

I'd like to close this thread as it's got crazily long, and in any case has wandered way off the original questions. Obviously please feel free to open another issue when you have other things that you want to ask about. Does that sound OK?

just one last thing, any idea why I got values beyond 4095? (please take a look at the last screenshot)

In the last screen shot, the code just before saving the file contained the line
array16 *= 16  # Bump 12-bit range up to 16

I'm assuming this is what gives you the larger values? Prior to that, you have numbers from 0 to 4095 (though nearly all will be at least 256 because that's the black level), with a granulatity of 1. After that scaling operation you have numbers from 0 to 65535 (nearly all will be at least 256*16 = 4096) with a granularity of 16. Does that make sense?

ok yes understood. any why did you recommend bumping up to 16 bit?

Mostly because if you view the file with some image viewer, without that scaling it will just look black. The largest value would be the equivalent of 16 in an 8-bit image which you would scarcely see.

ok interesting, would be an image viewer imaginable, which is capable of showing a 12 bit image in a proportionate way?
I mean why would be the largest value equivalent to 16 in a 8bit image, if we get values between 0-4095...
so if its only for image viewer standard-purposes, I can remove that bumping for the neuronal network....

What I'm saying is that an image viewer that understands 16-bit images will expect "white" to be 65535. But if the largest value you have is 4095 then that's only 1/16 of the white value, so everything will be very dark. Now, in an 8-bit image the equivalent value would be 1/16th of 255, or about 16. Or to put it in another way, in a 16-bit image, 4095 will have the same brightness as the value 16 does in an 8-bit image.
Other than for viewing, there's no particular reason to rescale the images. You could create an image viewer that expects 12-bit numbers. To be honest I'd load the image using cv2, multiply everything by 16 and then use imshow! Just be aware that if you use an off-the-shelf viewer for such a file (where the 16-bit pixels are never larger than 4095), it will all look simply black.

Okay very informative, thanks. I'll probably open an issue again soon.. best regards

@Bra1nsen were you ever able to save a RAW image from PiCam2 that was the array and metadata combined that could be read by raw software? Feel free to comment here #602


Switched broken pypip.in badges to shields.io

Hello, this is an auto-generated Pull Request. (Feedback?)
Some time ago, pypip.in shut down. This broke the badges for a bunch of repositories, including gallows. Thankfully, an equivalent service is run by shields.io. This pull request changes the badges to use shields.io instead.

Coverage remained the same at 98.667% when pulling 9a0c9ddfb9fe4e760f2cc2d50a16cb21ea20f960 on movermeyer:fix_badges into e6268e6bf4a6e0a2fc70a43d160da047cc113f6c on Chennaipy:master.

Coverage remained the same at 98.667% when pulling 9a0c9ddfb9fe4e760f2cc2d50a16cb21ea20f960 on movermeyer:fix_badges into e6268e6bf4a6e0a2fc70a43d160da047cc113f6c on Chennaipy:master.

Coverage remained the same at 98.667% when pulling 9a0c9ddfb9fe4e760f2cc2d50a16cb21ea20f960 on movermeyer:fix_badges into e6268e6bf4a6e0a2fc70a43d160da047cc113f6c on Chennaipy:master.


Lint error on -> console log

cc: @dsamojlenko

This should also pop up a warning in IDEs such as WebStorm.


trailing-comma for single line regression between 3.3.0 and 3.5.0?

Since upgrading to 3.5.0, I'm getting errors reported for single lines missing a trailing comma, even though the rules are set up like:
"trailing-comma": [true, {
  "singleline": "never",
  "multiline": "always"
}],

Here is an example line of code which now fails lint under 3.5.0, which used to pass in 3.3.0 with the same rules:
return expectBothTypesEqual(form, {
  'recurringAmount': 'recurringAmount:typeError'
})

it now requires a comma after the second line, which seems incorrect.

Is this a regression or a change in the spec of this rule?

This is a change in the spec of the rule, or rather a correction of it. The intent of the trailing-comma rule is to make code diffs more precise. Multiline literals should always have trailing commas because the addition of a member should only produce a one-line diff. In your example, a 2-line diff would be produced if you added a field, like this:
return expectBothTypesEqual(form, {
  'recurringAmount': 'recurringAmount:typeError',
  'somethingElse': 'foo'
})

Full details here https://github.com/palantir/tslint/issues/856

Got ya, that makes perfect sense. I thought it was because single comma
objects should never contain more than one item, so you wouldn't need to be
there as you'd never append another item. The diff thing is actually more
useful :) Will update my code, thanks
On 26 February 2016 at 15:04, Adi Dahiya<EMAIL_ADDRESS>wrote:

This is a change in the spec of the rule, or rather a correction of it.
The intent of the trailing-comma rule is to make code diffs more precise.
Multiline literals should always have trailing commas because the addition
of a member should only produce a one-line diff. In your example, a 2-line
diff would be produced if you added a field, like this:
return expectBothTypesEqual(form, {
'recurringAmount': 'recurringAmount:typeError',
'somethingElse': 'foo'
})
Full details here #856 https://github.com/palantir/tslint/issues/856
—
Reply to this email directly or view it on GitHub
https://github.com/palantir/tslint/issues/999#issuecomment-189315537.

It's also a lot easier when you're trying to reorder

Indeed, sorry should have closed that myself!

On 26 Feb 2016, at 22:20, Jason Killian<EMAIL_ADDRESS>wrote:
It's also a lot easier when you're trying to reorder
—
Reply to this email directly or view it on GitHub.


need to some improvements for work on latest Smartgesture

I have smartgesture version 4.0.24 (link). Current bat won't work because it starts only AsusTPCenter.exe and not starts the utility itself (even tray icon won't appear).
If I am correct, it is easy to fix. You just need to change line
start AsusTPCenter.exe
to
start AsusTPLoader.exe
Also you can make .bat run automatically after Windows resumes from sleep / hibernation. For that you need to create scheduler task like described in this ASUS forum thread.
As described in the thread you need to set Single trigger "On Event" + basic + log: System + Source: Kernel-Power + Event ID: 107. Also don't forget to enable option "Run with highest privileges" and make sure you unchecked "Start only if the computer is on AC power".

Thanks I will check it

Actually now that I remember it was intended to start without the utility and the try icon neither was present.
This was done to prevent some bug and made the software run smoother but wont be an issue to add it.


Adhoc in subqueries

I have a dashboard with adhoc filter set to some customer. I want to plot rps for top X things on that customer. This doesn't work, because adhoc filters are only applied to outer queries, which is fair.
Let's have a variable that I can manually plug into the subquery.
cc @vavrusa

Hi @bobrik @vavrusa
Thank you for contribution. The issue will be closed on release

Pls see https://github.com/Vertamedia/clickhouse-grafana/releases/tag/1.7.0


Option to make overlays non-modal

Overlays (vaadin-overlay) opened by various Vaadin components (e.g. combo-box, date-picker, etc) are effectively modal, although they render no modality curtain, as they set pointer-events:none; on the page body, which prevents the users from interacting with the rest of the page while the overlay is open.
E.g. while a combo-box dropdown is open, focus cannot be moved to another field by clicking it, as the initial click will only close the overlay, but not move focus to the clicked field.
While this is in line with native html elements like <select>, it may be undesired in some applications.
Solution proposal:
A simple solution, that would not require any additional API in components, could be to use a custom css property (with the currently hardcoded none as default value) for the pointer-events value set to the page body, so that it would be easy to override just by applying a single line of css.
E.g. if vaadin-overlay applied pointer-events: var(--vaadin-overlay-page-body-pointer-events); the modality  could be removed from all overlays with a single line of global css: --vaadin-overlay-page-body-pointer-events: auto;
If a more scoped override is desired, e.g. to only remove modality from combo-boxes, this could be achieved by a css selector targeting combo-box-specific overlays.

Have quickly prototyped this option. There are several things that I would like to discuss. The major one is the css property name.
In the logic of entering modal state overlay is also applying pointer-events  to other overlays and I guess we want to change that as well, so then word body  in --vaadin-overlay-page-body-pointer-events won't represent the whole functionality. In addition, not sure what does word page represent 🤔
Another thing is the usage of custom css property for that purpose. I guess the reason is to avoid “any additional API in components”, but in reality from the options only auto and none would make sense (i.e. SVG options cannot be applied, initial doesn’t make difference, inherit would most probably confuse because of the contents teleportation logic)
If only two values makes sense we can continue usage of the modeless state and propagate it to the overlay from the component.
<vaadin-combo-box modeless placeholder="Placeholder"></vaadin-combo-box>

Pros:

Would be more visible for the users as public API than custom css prop.
Would not confuse with all other possible options, like fill / stroke / inherit / unset  etc.
Can be scoped to one specific component without classes/attributes/ids used on :host
Will not change current behavior, but instead add simple functionality on top.

Cons:

Cannot be applied to a group of overlays at once if not default (i.e. scoping all combo-boxes on the page or all overlays). That one should probably stay opt-in feature due to the reasons described here.
Implementation is not difficult, but requires changes to all overlay components.

Yeah the name was just what first came to mind, as an example, not a proposal for the actual property name.
As for whehter a css property is appropriate, my assumption is that you'd specifically want to apply it to all components with invisibly modal overlays (i.e. overlay input fields, context-menu, menu-bar), as opposed to just one. The rationale being that you'd want all such overlays to work the same -- otherwise the behavior would be inconsistent between different component types and instances.
That being said, if there's a better way to achieve that, that would be avoid the issues mentioned, that would be even better. I just don't know what that would be.

Actually, one way to make the custom property approach less weird would be to hide the fact that it's using pointer-events, like so:
--nonmodal-overlays: all;
and
pointer-events: var(--nonmodal-overlays, none);
That gives us:
--nonmodal-overlays undefined => pointer-events: none; (fallback value used, no overlays are non-modal)
--nonmodal-overlays: none; ==> pointer-events: none; (no overlays are non-modal)
--nonmodal-overlays: alll; ==> pointer-events: all; (has no effect, falls back to  auto, i.e. all overlays are non-modal)


Study Time Stats fails to load - "No module named 'PyQt5'"

Bug/Issue Description
Study Time Stats fails to load, with a mention of "No module named 'PyQt5'" in the debug screen.
To Reproduce
Steps to reproduce:

Install Study Time Stats v2.0.6.
Restart Anki.
Error appears and Study Time Stats fails to load.

System/Debug Info
Anki 23.10.1 (fac9e0ee) Python 3.9.15 Qt 6.6.0 PyQt 6.6.0
Platform: Windows-10-10.0.19045
Flags: frz=True ao=True sv=3
Add-ons, last update check: 2023-11-10 19:13:44
When loading Study Time Stats:
Traceback (most recent call last):
File "aqt.addons", line 245, in loadAddons
File "C:\Users\Carter David\AppData\Roaming\Anki2\addons21\1247171202_init_.py", line 8, in
from .src import toolbar
File "C:\Users\Carter David\AppData\Roaming\Anki2\addons21\1247171202\src\toolbar.py", line 12, in
from .options import TimeStatsOptionsDialog
File "C:\Users\Carter David\AppData\Roaming\Anki2\addons21\1247171202\src\options.py", line 41, in
from ..res.ui.cell_item import Ui_CellWidget
File "C:\Users\Carter David\AppData\Roaming\Anki2\addons21\1247171202\res\ui\cell_item.py", line 11, in
from PyQt5 import QtCore, QtGui, QtWidgets
ModuleNotFoundError: No module named 'PyQt5'
Additional Context
May be related to #37 but I do not see a mention of this particular error there. Note that no other version of Anki or other Qt applications are installed (to my knowledge) - this may have been missed due to other users having a version of PyQt5 present on their machine.

had the same issue. @iamjustkoi do you need help updating it?

Hey, sorry to hear, and thanks for reporting!

had the same issue. @iamjustkoi do you need help updating it?

I really appreciate the offer! Honestly, I've been having to deal with so many recent life things that've popped up that I haven't really been able to code as much as I would've liked to for this or some other projects.
If a PR was submitted for this it'd really help a lot and I could push an update out really quick after taking a look!
Either way I'll try to see what I can do when I can, too, if nothing else!

I don't currently have time for helping much, but wanted to point this out for anyone else looking into doing a PR and to make sure @iamjustkoi saw this before doing future updates:
# Form implementation generated from reading ui file 'cell_item.ui'
#
# Created by: PyQt5 UI code generator 5.15.7
#
# WARNING: Any manual changes made to this file will be lost when pyuic5 is
# run again.  Do not edit this file unless you know what you are doing.

I don't know for sure if just regenerating the Python using the PyQt6 version of the UI code generator would fix everything, but at the least it's necessary here, and that'll need to happen for any future updates. (I'm not sure if this means that there would need to be separate version for Qt5 and Qt6 since there are still two Anki versions...which could also be the source of this bug being missed, if iamjustkoi is using the Qt5 version for testing.)


Excessive replication in walproposer because of lagging truncate_lsn

Steps to reproduce
Looks at the logs for some active timeline and try to find there Failed to open WAL file. This shouldn't happen in the normal work, but occurs in some our environments.
What happens here is truncate_lsn on all safekeepers is seriously outdated, and that causes walproposer to start downloading WAL from a relatively small LSN. That LSN can be less than backup_lsn, and the requested WAL segment can be already removed from the disk. It makes safekeeper to redownload it from S3 and write
Expected result
No Failed to open WAL file ... logs from safekeepers.
Actual result
2023-01-12T11:47:11.471093Z  INFO {tid=296850}:WAL acceptor{ttid=eaea7d36435df6474696d0bb61633419/f1c664a0cd057613cfac654fd7bc175e}: processed greeting from walproposer 254D2DCBFCE7F4752B477D6B9D5CA, sending term 8135
2023-01-12T11:47:11.471492Z  INFO {tid=260601}:WAL acceptor{ttid=eaea7d36435df6474696d0bb61633419/f1c664a0cd057613cfac654fd7bc175e}: processed greeting from walproposer 254D2DCBFCE7F4752B477D6B9D5CA, sending term 8135
2023-01-12T11:47:11.471595Z  INFO {tid=269567}:WAL acceptor{ttid=eaea7d36435df6474696d0bb61633419/f1c664a0cd057613cfac654fd7bc175e}: processed greeting from walproposer 254D2DCBFCE7F4752B477D6B9D5CA, sending term 8135
2023-01-12T11:47:11.471582Z  INFO elected for backup eaea7d36435df6474696d0bb61633419/f1c664a0cd057613cfac654fd7bc175e: elected 2 among [(NodeId(1), 0/776EDB8), (NodeId(2), 0/776EDB8), (NodeId(39), 0/776EDB8)] peers, with 3 of them being caughtup
2023-01-12T11:47:11.471679Z  INFO WAL backup task{ttid=eaea7d36435df6474696d0bb61633419/f1c664a0cd057613cfac654fd7bc175e}: started
2023-01-12T11:47:11.472659Z  INFO {tid=296850}:WAL acceptor{ttid=eaea7d36435df6474696d0bb61633419/f1c664a0cd057613cfac654fd7bc175e}: processed VoteRequest for term 8136: VoteResponse { term: 8136, vote_given: 1, flush_lsn: 0/776EDB8, truncate_lsn: 0/6F32000, term_history: ... [(8135, 0/7769078), (8134, 0/7769078), (8133, 0/77636F8), (8132, 0/77636F8), (8131, 0/775E150), (8130, 0/775E150), (8129, 0/7758F30), (8128, 0/7758F30), (8127, 0/7754108), (8126, 0/7754108), (8125, 0/774E220), (8124, 0/774E220), (8123, 0/7748768), (8122, 0/7748768), (8121, 0/7743070), (8120, 0/7743070), (8119, 0/773DD38), (8118, 0/773DD38), (8117, 0/7738DD8), (8116, 0/7738DD8)], timeline_start_lsn: 0/149F0F0 }
2023-01-12T11:47:11.472743Z  INFO {tid=269567}:WAL acceptor{ttid=eaea7d36435df6474696d0bb61633419/f1c664a0cd057613cfac654fd7bc175e}: processed VoteRequest for term 8136: VoteResponse { term: 8136, vote_given: 1, flush_lsn: 0/776EDB8, truncate_lsn: 0/6F32000, term_history: ... [(8135, 0/7769078), (8134, 0/7769078), (8133, 0/77636F8), (8132, 0/77636F8), (8131, 0/775E150), (8130, 0/775E150), (8129, 0/7758F30), (8128, 0/7758F30), (8127, 0/7754108), (8126, 0/7754108), (8125, 0/774E220), (8124, 0/774E220), (8123, 0/7748768), (8122, 0/7748768), (8121, 0/7743070), (8120, 0/7743070), (8119, 0/773DD38), (8118, 0/773DD38), (8117, 0/7738DD8), (8116, 0/7738DD8)], timeline_start_lsn: 0/149F0F0 }
2023-01-12T11:47:11.472896Z  INFO {tid=260601}:WAL acceptor{ttid=eaea7d36435df6474696d0bb61633419/f1c664a0cd057613cfac654fd7bc175e}: processed VoteRequest for term 8136: VoteResponse { term: 8136, vote_given: 1, flush_lsn: 0/776EDB8, truncate_lsn: 0/6F32000, term_history: ... [(8135, 0/7769078), (8134, 0/7769078), (8133, 0/77636F8), (8132, 0/77636F8), (8131, 0/775E150), (8130, 0/775E150), (8129, 0/7758F30), (8128, 0/7758F30), (8127, 0/7754108), (8126, 0/7754108), (8125, 0/774E220), (8124, 0/774E220), (8123, 0/7748768), (8122, 0/7748768), (8121, 0/7743070), (8120, 0/7743070), (8119, 0/773DD38), (8118, 0/773DD38), (8117, 0/7738DD8), (8116, 0/7738DD8)], timeline_start_lsn: 0/149F0F0 }
2023-01-12T11:47:11.478511Z  INFO {tid=296852}:WAL sender{ttid=eaea7d36435df6474696d0bb61633419/f1c664a0cd057613cfac654fd7bc175e}: Start replication from 0/6F32000 till Some(0/776EDB8)
2023-01-12T11:47:11.478803Z  INFO {tid=296852}:WAL sender{ttid=eaea7d36435df6474696d0bb61633419/f1c664a0cd057613cfac654fd7bc175e}: segment download about to start from remote path RemotePath("eaea7d36435df6474696d0bb61633419/f1c664a0cd057613cfac654fd7bc175e/000000010000000000000006") at offset 15933440
2023-01-12T11:47:11.478781Z  WARN {tid=296852}:WAL sender{ttid=eaea7d36435df6474696d0bb61633419/f1c664a0cd057613cfac654fd7bc175e}: Failed to open WAL file "/storage/safekeeper/data/eaea7d36435df6474696d0bb61633419/f1c664a0cd057613cfac654fd7bc175e/000000010000000000000006"
2023-01-12T11:47:11.603579Z  INFO {tid=296850}:WAL acceptor{ttid=eaea7d36435df6474696d0bb61633419/f1c664a0cd057613cfac654fd7bc175e}: received ProposerElected ProposerElected { term: 8136, start_streaming_at: 0/776EDB8, term_history: ... [(8136, 0/776EDB8), (8135, 0/7769078), (8134, 0/7769078), (8133, 0/77636F8), (8132, 0/77636F8), (8131, 0/775E150), (8130, 0/775E150), (8129, 0/7758F30), (8128, 0/7758F30), (8127, 0/7754108), (8126, 0/7754108), (8125, 0/774E220), (8124, 0/774E220), (8123, 0/7748768), (8122, 0/7748768), (8121, 0/7743070), (8120, 0/7743070), (8119, 0/773DD38), (8118, 0/773DD38), (8117, 0/7738DD8)], timeline_start_lsn: 0/149F0F0 }
2023-01-12T11:47:11.603864Z  INFO {tid=269567}:WAL acceptor{ttid=eaea7d36435df6474696d0bb61633419/f1c664a0cd057613cfac654fd7bc175e}: received ProposerElected ProposerElected { term: 8136, start_streaming_at: 0/776EDB8, term_history: ... [(8136, 0/776EDB8), (8135, 0/7769078), (8134, 0/7769078), (8133, 0/77636F8), (8132, 0/77636F8), (8131, 0/775E150), (8130, 0/775E150), (8129, 0/7758F30), (8128, 0/7758F30), (8127, 0/7754108), (8126, 0/7754108), (8125, 0/774E220), (8124, 0/774E220), (8123, 0/7748768), (8122, 0/7748768), (8121, 0/7743070), (8120, 0/7743070), (8119, 0/773DD38), (8118, 0/773DD38), (8117, 0/7738DD8)], timeline_start_lsn: 0/149F0F0 }
2023-01-12T11:47:11.604466Z  INFO {tid=260601}:WAL acceptor{ttid=eaea7d36435df6474696d0bb61633419/f1c664a0cd057613cfac654fd7bc175e}: received ProposerElected ProposerElected { term: 8136, start_streaming_at: 0/776EDB8, term_history: ... [(8136, 0/776EDB8), (8135, 0/7769078), (8134, 0/7769078), (8133, 0/77636F8), (8132, 0/77636F8), (8131, 0/775E150), (8130, 0/775E150), (8129, 0/7758F30), (8128, 0/7758F30), (8127, 0/7754108), (8126, 0/7754108), (8125, 0/774E220), (8124, 0/774E220), (8123, 0/7748768), (8122, 0/7748768), (8121, 0/7743070), (8120, 0/7743070), (8119, 0/773DD38), (8118, 0/773DD38), (8117, 0/7738DD8)], timeline_start_lsn: 0/149F0F0 }
2023-01-12T11:47:11.615034Z  INFO {tid=296850}:WAL acceptor{ttid=eaea7d36435df6474696d0bb61633419/f1c664a0cd057613cfac654fd7bc175e}: start receiving WAL since 0/776EDB8
2023-01-12T11:47:11.642132Z  INFO {tid=269567}:WAL acceptor{ttid=eaea7d36435df6474696d0bb61633419/f1c664a0cd057613cfac654fd7bc175e}: start receiving WAL since 0/776EDB8
2023-01-12T11:47:11.644397Z  INFO {tid=260601}:WAL acceptor{ttid=eaea7d36435df6474696d0bb61633419/f1c664a0cd057613cfac654fd7bc175e}: start receiving WAL since 0/776EDB8

Proposed fix
Try to update truncate_lsn more often. Perhaps we should persist control file on compute disconnect, to store actual data here in 99% cases.

Should be mostly fixed by https://github.com/neondatabase/neon/pull/3753, need to check the logs to verify.
It's probably still can be reproduced by restarting safekeepers before flushing peer_horizon_lsn (truncate_lsn) to the disk, we should try to flush state to disk more often. To do that, we can trigger flushing when timeline becomes inactive (compute disconnects), on graceful shutdown, etc.
Before the fixes there were many timelines with MAX(backup_lsn) - MIN(disk_peer_horizon_lsn) around 16MB, and that triggered S3 download. We can try to get MIN(flush_lsn) - MIN(disk_peer_horizon_lsn) close to zero, should be not hard to do with additional flushes.

We still have some similar logs on staging:
2023-03-15T02:46:53.706337Z  INFO {cid=27899}:WAL sender{ttid=279b7473cf885a7ae024c2fdf92bf099/646a14b237d79cf52ae2283f4dd0bdcd}: segment download about to start from remote path RemotePath("279b7473cf885a7ae024c2fdf92bf099/646a14b237d79cf52ae2283f4dd0bdcd/000000010000000000000002") at offset 0
2023-03-15T02:46:53.303655Z  INFO {cid=27899}:WAL sender{ttid=279b7473cf885a7ae024c2fdf92bf099/646a14b237d79cf52ae2283f4dd0bdcd}: segment download about to start from remote path RemotePath("279b7473cf885a7ae024c2fdf92bf099/646a14b237d79cf52ae2283f4dd0bdcd/000000010000000000000001") at offset 4845808
2023-03-14T19:59:29.906014Z  INFO {cid=2939}:WAL sender{ttid=0ab678e010193c6bdfc2c6db87abb9fa/072aef4e42a6bd7bfd3fed9e38cae4c4}: segment download about to start from remote path RemotePath("0ab678e010193c6bdfc2c6db87abb9fa/072aef4e42a6bd7bfd3fed9e38cae4c4/000000010000000000000001") at offset 4845880
2023-03-14T19:59:29.040308Z  INFO {cid=2937}:WAL sender{ttid=0ab678e010193c6bdfc2c6db87abb9fa/072aef4e42a6bd7bfd3fed9e38cae4c4}: segment download about to start from remote path RemotePath("0ab678e010193c6bdfc2c6db87abb9fa/072aef4e42a6bd7bfd3fed9e38cae4c4/000000010000000000000001") at offset 4845880

That's probably because of the frequent restarts. I'll create a new issue with next steps I wrote in https://github.com/neondatabase/neon/issues/3317#issuecomment-1458286332.


chore: add more Nuxt templates powered by Cloudflare

Summary

https://github.com/Flosciante/nuxt-image-gallery (Pages + R2): 100+ stars
https://github.com/ra-jeev/hub-chat (Pages + A1): just launched but promising (see demo)

Documentation checklist

[x] The documentation style guide has been adhered to.

Thanks Kody for the quick merge and kind words ❤


Feedback on API security requirements

•	The Software Defined Networking (SDN) controller must let the administrator determine what application programming interface (API) objects can be used to safely and securely manage the SDN controller through the management interface.
•	The SDN controller must let the administrator specify how the allowed API objects can be manipulated.
These two seem rather vague. What is a practical application of this requirement?  What is meant by API "object" here?   Is it expected to have a granual RBAC policy to require the administrator define  all possible actions (create, read, update, delete) on each and every object managed by the API?  Would it be acceptable to simply have a pre-defined RBAC policy where you essentially have a full access role where all objects can be manipulated and another role where the objects are read-only?  Then you just assign the user to either the admin or read-only role, and that defines the operations which can be performed on the entire set of objects under the scope of the API.
We need to consider the various types of "APIs" that can be used and describe these requirements in terms of the allowed mechanisms. When I think of network management "APIs", a common one that is widely used is NETCONF which is an IETF approved "protocol", not an API per se.  Other solutions may use a RESTful API for management, etc.
•	The SDN controller must have API templates that define how API calls must be performed to access API objects.
Again, how do we describe this in terms that would apply to a protocol?  For example, NETCONF uses YANG templates to describe data models which define the capabilities of the protocol.  Is this a correct interpretation?
•	The SDN controller must perform validation of API calls against the API templates.
Exactly what validation checks are required?  Basic semantic validation, preventing command injection, we need to be precise here regarding success/fail criteria.
•	The SDN controller must let the administrator be able to allowlist API templates so that configuration attempts from all other APIs not in the allowlist is blocked.
What is meant by API templates here?  Device configuration templates? or the templates (see YANG) that define the API/protocol parameters used to manipulate config objects?  Why would whitelisting be needed over normal RBAC?  Usually API templates are defined in source code and an administrator would not have this level of control over which "API templates" are available for use. Unless we are talking about a differnet meaning of the word.
•	The SDN controller must provide an API user role and API administrator role for role-based access control
What if RBAC is not enforced natively by the API but through other access means?  for example, a CLI user would be presented with the config commands/objects that are available based on the role and will not process the command (or proceed to make an API call) without first validating the user is in the correct role.
•	The SDN must provide analytics on the API logs to detect API usage that does not comply with policies
This is also very vague.
•	The SDN controller must have maximum auditing and logging off all unauthorized API usage, especially attempts API usage from users that are not in the API roles.
Should this be N/A if no non-API users exist?  Again it is hard to apply when we are talking about protocols where the user access enforcement may not be directly at the API layer.
It is hard to interpret and give feedback the requirements as written without some type of pass/fail criteria or real-world examples of what is expected for each.

Thank you for your insights @kenlasoski ! See below for clarifications:
For the below requirement:
• The Software Defined Networking (SDN) controller must let the administrator determine what application programming
interface (API) objects can be used to safely and securely manage the SDN controller through the management interface.
This above requirement does refer to RBAC, and yes, pre-defined roles is fine. This specific requirement though refers to the SDNC allowing the admin to specify which API endpoints are allowed to be accessed by the SDNC user/admin.
For the below requirement:
• The SDN controller must let the administrator specify how the allowed API objects can be manipulated.
This above requirement does refer to RBAC, and yes, pre-defined roles is fine. This specific requirement though refers to the SDNC allowing the admin to which CRUD operations can be performed on the API endpoints that the given user/admin has access to.
For the below requirement:
• The SDN controller must have API templates that define how API calls must be performed to access API objects.
This above requirement refers to the management interface ("northbound")/GUI of the SDNC. Any protocol that the SDNC supports for the management interface is fine, whether REST API, RESTCONF, etc. This requirement though refers to the SDNC requiring the API call to conform to a structure like JSON, XML, etc. This requirement also covers the types of input for each parameter, e.g. only dates for date fields, only positive integers for positive integer fields, only boolean for boolean fields. The API call, then, would need to be in the required structure, and data input into fields would need to be in the required type, e.g. true/false for boolean fields, etc.
For the below requirement:
• The SDN controller must perform validation of API calls against the API templates.
This above requirement refers to "input validation" in general. Basically, API calls that have any bad data, unwanted input, any type of malicious "injection" payload, etc. should be blocked. API calls that have data in the wrong "type" should be blocked (e.g. putting arbitrary string data in a boolean-only field, negative integers in a postive-integer field, etc.).
Since input validation is a general mitigation that is typically needed in all types of IT systems, not just SDNC, we'll take a look at other PPs, and CC in general, to see if any other examples of "input validation" requirements were already done, and if so, we'll look into letting input validation be covered by those previous PP/CC input validation requirements. In other words, no need  to reinvent.
For the below requirement:
• The SDN controller must let the administrator be able to allowlist API templates so that configuration attempts from all other
APIs not in the allowlist is blocked.
Yes, this above requirement refers to the API/protocol parameters used to manipulate config objects. Per your feedback, since normal RBAC covers this (and so eliminating the need for an allowlist), and since this is usually defined in source code, we'll go ahead and remove this requirement. Thank you for helping provide guidance on this.
For the below requirement:
• The SDN controller must provide an API user role and API administrator role for role-based access control
Yes, if RBAC is not enforced natively by the API but through other access means, that is fine.
For the below requirement:
• The SDN must provide analytics on the API logs to detect API usage that does not comply with policies
PP/CC already has existing requirements spelled out for handling auditing/logging. We'll go ahead and pull from the existing wording for handling auditing/logging. The existing auditing/logging requirements in PP/CC also uses a table that lists exactly which "Auditable Events" should be captured. We'll pull from that table, and then add additional rows for any additional things that an SDNC should audit/log. This should help make this requirement less vague/more specific. Thanks for your input on this.
For the below requirement:
• The SDN controller must have maximum auditing and logging off all unauthorized API usage, especially attempts API usage
from users that are not in the API roles.
This requirement refers to the SDNC logging usage of the SDNC management interface API. This requirement refers to generating a log if a malicious actor tried to perform an API call, whether that actor was authenticated or unauthenticated. It also covers a legitimate user/admin attempting to perform an API call that they are not allowed to do, whether they had an API role or not. For example, if someone who is not an SDNC user/admin tries to send an REST API request to the SDNC, it should be blocked since they are not an SDNC user/admin, and the SDNC should generate a log that shows this attempted unauthorized attempt. Or, if an SDNC user/admin has a read-only role, and tried to send a POST request via REST API to attempt to perform a write operation, it should be blocked, even though the user/admin is a legitimate SDNC user/admin, and there should still be a log generating showing this attempted write operation from a read-only role. Or, if a SDNC user/admin in a non-API role tried to craft a REST API request to read data from an API endpoint of the SDNC, this attempt should be blocked, and logged, even though the user/admin is a legitimate SDNC user/admin, because they are a non-API role.
Please let us know any thoughts. Thank you for helping with the SDNC SFRs feedback!

This statement seems conflicting:
This above requirement does refer to RBAC, and yes, pre-defined roles is fine. This specific requirement though refers to the SDNC allowing the admin to specify which API endpoints are allowed to be accessed by the SDNC user/admin.
I interpret pre-defined roles as a hard coded set of authorized functions/attributes (in this case, APIs) that are assigned to difference categories of acccess roles (Admin, Network user, Operator, etc.).   But the second part of the statement  (and the following response about CRUD) implies that the SDN is expected to have a UI that allows granular definition of specific API endpoints that can be manually assigned to customized roles.  To do so would mean that the admin user is expected to read through the entire API specification to understand the security impacts of each capability before configuring RBAC.
Consider an example SDN product with three pre-defined roles, the Admin user has access to all endpoints, Network User would have read/write to only Network related endpoints, Operator would have read-only to a certain subset.
Wouldn't the ability to define the role for each user inherently meet the requirement of defining the APIs they are allowed to access?
What if the capability for granular role definition and enforcement is not in the SDN controller itself, but rather in an external Management component?  would that disqualify the product from meeting the SDN PP?  If not, which of these requirements can be met by the MC vs the SDNC?

Thank you for elaborating @kenlasoski !
To clarify, the SDNC must allow setting which endpoints can be accessed/modified, or at least have pre-defined roles. So, if the SDNC has pre-defined roles, that would satisfy this requirement. If there is no pre-defined roles, then at least let the admin have the custom roles.
For MC vs SDNC definition/enforcement, the requirement must be satisfied at wherever the management interface of the SDNC is. So, if the SDNC's management interface is in the external MC, then the external MC must satisfy this requirement. This would of course mean that the SDNC should not allow management directly, since it is handled via MC. Regardless of how the SDNC is deployed (all-in-one SDNC appliance, separate MC for administering the SDNC, etc.), the requirement is the same, and the test is the same. Now, if the MC is just an optional external component that can be used for managing the SDNC, and the SDNC still has it's own management interface, then this type of optional external component is out-of-scope.
Thanks for continuing to help with feedback. Please let us know any additional thoughts.


switch terminal action breaks (and creates...?) terminal editors when switching between terminal editors in different groups

Open latest insiders
Create new terminal editor
Split it using editor split button on top right.
Switch from one terminal to the other (several times) using CMD+P term listing.
I hope that my message is clear enough. If not don't hesitate to tell me, and I'll do a video for you on Monday.
from https://github.com/microsoft/vscode/issues/10546#issuecomment-873168851

for the same reason as https://github.com/microsoft/vscode/issues/126469
sharing the terminal editor input between groups does not work. I think we need a terminal editor for each editor group so the setInput call/detaching of the instance does not break terminal editors in other groups

I believe this is caused by us not tracking the group index to which the editor input belongs

Switching from terminal A to terminal B results in terminal A's group picking up a new terminal, and terminal B breaking.

Can repro:

It actually breaks the left side but only because it gets moved to the side. I suspect this is related to the recent SIDE_GROUP change. I don't think this meets the bar to push the release back as it can only happen when changing groups via ctrl/cmd+p and the fragility of that code.

FWIW, the SIDE_GROUP change was reverted.


disable opacity on treemap mouseover

Thanks for this great lib,
Is there an easy way to remove opacity effect on treemap mouseover ?
thanks again

.data({"opacity": 1})

thanks


ocamlPackages.mirage-crypto*: 0.8.1 → 0.8.3

Motivation for this change
New minor release, should now build on power pc.
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[x] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

ocsigen-start fails in this PR, but it also does so on master because oscigen_server is broken currently (resolved in #93612).

@GrahamcOfBorg build dune-release jackline  ocamlPackages.irmin-unix ocamlPackages.opium

@GrahamcOfBorg build jackline dune-release


Yasks does not wait for the Istio sidecar

Hi all!
We have a yaks feature that uses the presto sql driver and therefore needs to load the dependency first before it can work properly. Sometimes the sidecar needs longer than normal to boot up and the yaks tries to resolve the dependency before a connection thru the sidecar can be fully established. Therefore the yaks test fails, because it can not resolve the dependency. Should we just include a wait in a pre hook, or is this a bug?
We tested this with the newest version of Istio.
Thanks for you help!

Hi Nils,
Is this a sidecar of the YAKS test pod?
From your description I would not consider this as a bug. I rather think we need to find a way to properly wait for the service to be available. Maybe we can change the pattern to an init container if this is a sidecar on the YAKS pod. As an alternative the test could wait for a Http Url to return 200 OK or the test could wait for another Pod to log some specific message. WDYT?

Hi Christoph,
Yes, I was talking about the sidecar of the YAKs test pod. I tried out the alternative that you mentioned (should have tried that out on my own) and this method works fine for my use case.
Thank for you help!


gardenlet: Stop deleting Ingress resources for Seed via the extensions/v1beta1 API

How to categorize this PR?

/area control-plane
/kind cleanup
What this PR does / why we need it:
From https://kubernetes.io/docs/reference/using-api/deprecation-guide/#ingress-v122:

The extensions/v1beta1 and networking.k8s.io/v1beta1 API versions of Ingress is no longer served as of v1.22.

Migrate manifests and API clients to use the networking.k8s.io/v1 API version, available since v1.19.

From https://github.com/gardener/gardener/blob/v1.57.1/docs/usage/supported_k8s_versions.md:

The minimum version of a seed cluster that can be connected to Gardener is 1.20.x. Please note that Gardener does not support 1.25 seeds yet.

Having in mind both of the conditions, gardenlet can stop deleting Ingress resources for Seeds via the extensions/v1beta1 API. This was only required for Seeds with K8s < 1.19.
Which issue(s) this PR fixes:
Fixes #6846
Special notes for your reviewer:
GKE for example keeps track of removed APIs usage and against K8s v1.23 GKE Seed gardenlet tries to delete the Ingress resources for a Seed  via the extensions/v1beta1 API:

Release note:

gardenlet no longer tries to delete Ingress resources for a Seed via the `extensions/v1beta1` API (no longer served as of K8s 1.22). As Gardener supports only Seed clusters with K8s >= 1.20, it is enough to delete the Ingress resources via the `networking.k8s.io/v1` API (available since v1.19).

/assign

Thanks!
/lgtm
/approve

/test pull-gardener-integration

/test pull-gardener-e2e-kind-migration