SkipPattern for Management Endpoints doesn't work with context-path

The skip pattern for actuator endpoints contains the context-path, but the TracingFilter tries to match the skip pattern without context-path.
The following is added to the configured default skip pattern:
/user/api/management/(health|health/.*|info|info/.*)
The method TracingFilter#isTraced removes the context-path from the request URI:
httpServletRequest.getRequestURI().substring(httpServletRequest.getContextPath().length())
Maybe I get something wrong, but I suspect that this filter never works when Spring has a context-path configured.

cc)  @larchanjo @ask4gilles  could you please have a look. You worked on the actuator part.
Does the context path affect the actuator endpoints?
I think we should remove the context path from the actuator skip pattern and add e2e test with configured context-path.
Would you be interested in submitting a PR?

I'm trying to understand the unit tests and found this while debugging SkipEndPointsWithContextPathAndBasePathTest#testTraceHealthCareEndpoint:

The TestRestTemplate expands the URL automatically by context-path (RestTemplate#execute).
When you prefix your URL to call for tests with the context-path
private static final String HEALTHCARE = CONTEXT_PATH + "/healthcare";
it will be there twice and therefore can be removed once.
I think, the unit test is wrong and hides the actual error I mentioned in my initial post.
I would not like to submit a PR because it affects a lot of unit test methods and I can't estimate the impact.

@pavolloffay @ancore I'll have a look at this tomorrow.

@ancore In order to gain time, can you please provide a minimal project sample to reproduce your issue? Thanks!

You will see the effect when you remove CONTEXT_PATH + from
private static final String HEALTHCARE = CONTEXT_PATH + "/healthcare"

@ancore What's "the effect"?
If I removed it, I still see it traced, which is normal.
What's your issue exactly? An actuator endpoint traced which should not or the other way around?
When you say:

The following is added to the configured default skip pattern:
/user/api/management/(health|health/.|info|info/.)

Did you add this configuration yourself or is it the result that you see?
It would be easier to have a simple example with your configuration. Thanks

Module: io.opentracing.contrib:opentracing-spring-web-starter:2.1.3-SNAPSHOT
Class: SkipEndPointsWithContextPathAndBasePathTest
Let's focus one one test:
private static final String INFO_ACTUATOR = CONTEXT_PATH + ACTUATOR_PATH + INFO_PATH;

@Test
public void testSkipInfoEndpoint() {
   invokeEndpoint(INFO_ACTUATOR);
   assertNoSpans();
}

Debugging:  Breakpoint at line 271 in io.opentracing.contrib.web.servlet.filter.TracingFilter
Inspection of httpServletRequest.getRequestURI shows /myapp two times.
When using a TestRestTemplate, the configured context-path is added automatically (RestTemplate line 669). We must not add it to the test URL (INFO_ACTUATOR).
So, adding CONTEXT_PATH to the test URLs is wrong in these unit tests. The test fails (rightly so) if you setup the test URL correctly:
private static final String INFO_ACTUATOR = ACTUATOR_PATH + INFO_PATH;
The following happens:

test starts with '/actuator/info'
rest template prefixes /myapp, so you get '/myapp/actuator/info'
TracingFilter.isTraced removes '/myapp' again
no match of skippPattern because the pattern contains a context-path for actuator: '/myapp/actuator/info' (shortened the fragment of pattern)

The problem is the skip pattern, it should not contain the context-path.

@ancore Thanks, makes sense. See attached PR.


How to pass Scopes ?

I'm trying to authenticate but getting an error

Invalid permissions requested
Requested scopes cannot be blank

I'm not sure how to go about attaching the Scopes to your provider object ?

As anticipated I picked up the answer elsewhere within minutes of posting this but after an hour or so of looking. sigh
$noSlack->getAuthorizationUrl([
'scope' => 'channels:read chat:write:bot chat:write:user im:write users:read']
));


Feature Request: Texture like widget

Discussed on #21
pdf_render has page level widget and we need such widget.

0.4.23 introduces PdfDocumentViewBuilder and PdfPageView.
@jezell
I think PdfPageView can be used like Texture widget but I'm not sure whether it's what you want or not...
Example code:
PdfDocumentViewBuilder.asset(
  'assets/sample.pdf',
  builder: (context, document) => ListView.builder(
    itemCount: document?.pages.length ?? 0,
    itemBuilder: (context, index) {
      return Container(
        margin: const EdgeInsets.all(8),
        height: 240,
        child: Column(
          children: [
            SizedBox(
              height: 220,
              child: PdfPageView(
                document: document,
                pageNumber: index + 1,
              ),
            ),
            Text('${index + 1}'),
          ],
        ),
      );
    },
  ),
),

@espresso3389 looks good, I'll give it a shot!


Use the available Quartznet repository and evaluate it on chosen datasets

[ ] Evaluate Quartznet15x15 on ASR dataset. Measure the WER and time required for each prediction. Think/read about a time metric, like inference_time/1s_of_sample.
[ ] Evaluate Quartznet15x15 using the same method on the Snips-SmartSpeaker dataset, which will be available from HF datasets as soon as #37 is closed
[ ] Add some white, and real-world noise to both datasets above and measure performance on them once again
[ ] Create a nice-looking comparison, that everyone can look at.

Use the Nemo ASR collection from Nvidia. I'll post an example in a comment.
https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/asr/api.html

quartznet15x5: WER 0.0447, 2.46it/s
At first it appears being worse than the Conformer, but in further task we can experiment with it

It also seems to be a little bit slower than the small conformer. But Iterations/sec doesn't tell us much. @kordc please do some measurement on how long inference on one second of audio takes.
Also difference of 1% in WER is not that much. For the comparison to be more reliable we should test it also with some noisy data.

@Tymon-Dydowicz look at the updated description

example use of the Nemo:
import nemo.collections.asr as nemo_asr
asr_model = nemo_asr.models.EncDecCTCModelBPE.from_pretrained(model_name="stt_en_conformer_ctc_small")
asr_model.transcribe(paths2audio_files=[r"C:\polibuda\audio\data_prepared\en\librispeech\audios\nzDKVo9jMkSQBeqIZ4imPoivp839vI3Tp5fHOVMDrGwSdiLeVMwov0CpJotVNSkbcOedL53JgUAcohPSPSlo23CB6qQrZLJ4srS3BWa0uYp6W7.wav"])

or using csv from dataset
librispeech_path = r"C:\polibuda\audio\data_prepared\en\librispeech\\"
df = pd.read_csv(librispeech_path+r"test-clean.csv", sep="\t")
preds = asr_model.transcribe(paths2audio_files=[
    librispeech_path+row["filepath"] for _, row in df.iterrows()])
targets = [row["text"] for _, row in df.iterrows()]

About WER metric:
from torchmetrics import WordErrorRate
wer = WordErrorRate()

wer(preds, targets)

Allegedly 15x15 is worse than 15x5 and there is no pretrained versions of 15x15 so use 15x5

Try tu use dataset #37 to evaluate


Adding Polaris000 as mentor for some project ideas

I would like to participate as a mentor for CHAOSS during GSoC'20 for some grimoirelab projects. I have added myself as a mentor for these. @valeriocos @GeorgLink please have a look.
Thanks

LGTM, thanks @Polaris000

Thanks @Polaris000 and @valeriocos


Surface errors from child resources in the CR status.

Right now when a statefulset for amalthea is invalid or fails to start we do not surface this error in the amaltheaSession status.
But we should. Especially for cases that are not transient and require intervention from the user/creator of the session to fix.

Followup from #748


integration w habit tracker

I am looking to integrate a chat app with a habit tracker.  Any documentation or suggestions you can point me to?
Specifically would like users to be able see how people are doing on habits and communicate around it

Hi @bobavery - no suggestions, our developer documentation is available at the following links but we don't have anything around habit tracking.

https://api.mattermost.com/
https://developers.mattermost.com/
https://docs.mattermost.com/guides/developer.html


Custom dataset

Congrats on your great work and thank you for releasing the code.
I am curious to train my custom dataset but get stuck in the dataset preparation process. I also notice that you mention in the paper: "We refine the initial GPS-derived camera poses in the Mill 19 and UrbanScene3D datasets and the estimates provided in the Quad 6k dataset using PixSFM."
I have the following questions:

How are the initial camera poses derived from drone GPS and IMU sensors? How are the initial GPS-derived camera poses refined using PixSFM? Could you kindly provide the script that can generate camera poses?
Can the camera poses estimated by COLMAP work with Mega-NeRF? Or should also be refined by PixSFM?

Thanks in advance and really looking forward to your reply!
Xingyi

Hi @xingyi-li - thanks for the kind words. To answer your questions:
Re: the initial camera poses from the GPS and IMU - we're just using the GPS geotags in the image metadata. Often this is embedded in the JPEG metadata itself (for example https://developer.parrot.com/docs/pdraw/photo-metadata.html for ANAFI photos). We also used to use the camera orientation back when we were refining poses with a commercial solution, but didn't use it for our PixSFM poses.
The PixSFM repo explains their process in detail but our usage of it is pretty straightforward - https://gist.github.com/hturki/b3ae2e020bcc2a08dbba402786f1fcd3 should give you an idea of how we refine the poses with it. The main deviation is that we used a custom GPS matcher which limits the image matching to only other images in its vicinity (https://gist.github.com/hturki/5a76b1ee54b8cd23e34e075f8a8b87e0) using the metadata found in the Mill19 and UrbanScene3D images. If your dataset is relatively small you might be able to get away with just using exhaustive matching.
Re: Using COLMAP instead of PixSFM - PixSFM uses COLMAP as part of their pipeline and does additional refinement before and after bundle adjustment, and they actually write the final result in COLMAP format. I just pushed a script that should convert COLMAP output (from PixSFM or just using COLMAP) at https://github.com/cmusatyalab/mega-nerf/blob/main/scripts/colmap_to_mega_nerf.py. As a word of caution, Mega-NeRF (and NeRF in general) is extremely sensitive to good camera poses, and we observed differences of over 6 PSNR when comparing PixSFM poses to that of a commercial photogrammetry software we initially used. So I'd strongly recommend using PixSFM if possible. And in either case, you will likely want to geo-register your result dataset using COLMAP's model_aligner utility (https://colmap.github.io/faq.html#geo-registration) so that Mega-NeRF's --ray_altitude_range parameter works properly. If you don't have geotags for your image you might be able to get away with manhattan world alignment (https://colmap.github.io/faq.html#manhattan-world-alignment) but it will probably not give you perfect bounds.
Hope that this helps!

Hi @hturki, really appreciate your quick reply. I will try your suggestions as soon as possible and will report if I can successfully train my custom dataset. Temporarily closing this issue.
Thanks again!
Xingyi

Hi @hturki, just want to make sure that I understand you. I first summarize the total steps to obtain the refined camera poses Mega-NeRF expects:

Run the script at https://gist.github.com/hturki/b3ae2e020bcc2a08dbba402786f1fcd3, where in line 38 of it, pairs_from_gps actually corresponds to the script at https://gist.github.com/hturki/5a76b1ee54b8cd23e34e075f8a8b87e0. In pairs_from_gps, you derive the initial camera poses from the drones metadata. Then the initial camera poses are used to generate 'pairs-sfm.txt'. This limits the image matching to only other images in its vicinity. After this step, the refined camera poses using PixSFM have been obtained.
Use COLMAP's model_aligner utility (https://colmap.github.io/faq.html#geo-registration) so that Mega-NeRF's --ray_altitude_range parameter works properly.
Use mega-nerf/scripts/colmap_to_mega_nerf.py to convert from PixSFM/COLMAP output to the format Mega-NeRF expects.

Am I right?
I also have two questions:

It seems that in pairs_from_gps, you only make use of latitude, longitude, and altitude, without using things like roll, pitch, and yaw (or, orientation). Is that true?
As far as I am concerned, the so-called initial camera poses are only used to generate 'pairs-sfm.txt'. The camera poses that Mega-NeRF uses are actually estimated by PixSFM. Is that true?
Although you point out in README that the Scale factor maps from the real-world unit (ie: meters) to [-1, 1] range, I still get confused about how the value of the --scale parameter in mega-nerf/scripts/colmap_to_mega_nerf.py is decided. Is it making camera_in_drb[:, 3] to be between [-1, 1]?

Thanks and wish you all the best!
Xingyi

Yes, we're not making use of roll/pitch/yaw for our PixSFM poses (we explored using them when using a different commercial solution)
You're correct in that they're mainly used to minimize the image pairs that pixsfm/colmap have to evaluate (using exhaustive matching is prohibitive when your dataset is thousands of images). We also use these GPS readings to geo-reference the final poses (via colmap's model_aligner utility)
Yes, it's just a scaling factor to make sure all the poses fall within [-1, 1]. It should be pretty straightforward to modify the provided script to automatically determine a good scaling factor.

I don't have a very scientific way of doing this - sometimes I'd start off with a wide range, then quickly train a cheap model (10-20k iterations without using spatial partitioning), and then see what sort of range looks good in mega-nerf-viewer. In general you want ray altitude range to not extend too far below ground level while still containing your camera poses
Just commented on the ticket - would recommend giving https://github.com/NVlabs/instant-ngp a spin

@xingyi-li Have you successfully trained your custom dataset?

@pengzhao-life are you able to prepare the dataset as mentioned ?

@hturki https://gist.github.com/hturki/b3ae2e020bcc2a08dbba402786f1fcd3
explain how to use this and gps_matcher together. What i have understood is use install pixsfm run this script https://gist.github.com/hturki/b3ae2e020bcc2a08dbba402786f1fcd3 but i am facing some issue please check the comment on this link then use colmap to load the files generated from pixsfm apply aligner . Use this to colmap to mega_nerf direclty  use this for training ?

Hi @hturki, I have two more questions:

I have followed Custom dataset #6 (comment) to generate the refined camera poses Mega-NeRF expects, but how --ray_altitude_range is decided?
As mentioned in Demo camera trajectory #14 (comment), I am also interested in generating videos through my camera trajectory :-)

Many thanks!

@xingyi-li have you figured out a way to estimate ray_altitude_range? what is this term represents ?

Hi @hturki, just want to make sure that I understand you. I first summarize the total steps to obtain the refined camera poses Mega-NeRF expects:

Run the script at https://gist.github.com/hturki/b3ae2e020bcc2a08dbba402786f1fcd3, where in line 38 of it, pairs_from_gps actually corresponds to the script at https://gist.github.com/hturki/5a76b1ee54b8cd23e34e075f8a8b87e0. In pairs_from_gps, you derive the initial camera poses from the drones metadata. Then the initial camera poses are used to generate 'pairs-sfm.txt'. This limits the image matching to only other images in its vicinity. After this step, the refined camera poses using PixSFM have been obtained.
Use COLMAP's model_aligner utility (https://colmap.github.io/faq.html#geo-registration) so that Mega-NeRF's --ray_altitude_range parameter works properly.
Use mega-nerf/scripts/colmap_to_mega_nerf.py to convert from PixSFM/COLMAP output to the format Mega-NeRF expects.

Am I right?
I also have three questions:

It seems that in pairs_from_gps, you only make use of latitude, longitude, and altitude, without using things like roll, pitch, and yaw (or, orientation). Is that true?
As far as I am concerned, the so-called initial camera poses are only used to generate 'pairs-sfm.txt'. The camera poses that Mega-NeRF uses are actually estimated by PixSFM. Is that true?
Although you point out in README that the Scale factor maps from the real-world unit (ie: meters) to [-1, 1] range, I still get confused about how the value of the --scale parameter in mega-nerf/scripts/colmap_to_mega_nerf.py is decided. Is it making camera_in_drb[:, 3] to be between [-1, 1]?

Thanks and wish you all the best! Xingyi

Hi, for step 2  "Use COLMAP's model_aligner utility", what 'alignment_type' did you use? ecef or enu or something else, how do you decide it? Thanks!

Hi, for step 2 "Use COLMAP's model_aligner utility",

what 'alignment_type' did you use? 'ecef 'or 'enu' or something else, how do you decide it?
for '--ref_images_path' or '--database_path', I used the database.db file created from step 1 i.e. pixsfm reconstruction function. But 'colmap model_aligner' throws error 'SQLite error: colmap/src/base/databsase.cc: unable to open database file.

@pengzhao-life  did you figure it out?

Hi @xingyi-li - thanks for the kind words. To answer your questions:
Re: the initial camera poses from the GPS and IMU - we're just using the GPS geotags in the image metadata. Often this is embedded in the JPEG metadata itself (for example https://developer.parrot.com/docs/pdraw/photo-metadata.html for ANAFI photos). We also used to use the camera orientation back when we were refining poses with a commercial solution, but didn't use it for our PixSFM poses.
The PixSFM repo explains their process in detail but our usage of it is pretty straightforward - https://gist.github.com/hturki/b3ae2e020bcc2a08dbba402786f1fcd3 should give you an idea of how we refine the poses with it. The main deviation is that we used a custom GPS matcher which limits the image matching to only other images in its vicinity (https://gist.github.com/hturki/5a76b1ee54b8cd23e34e075f8a8b87e0) using the metadata found in the Mill19 and UrbanScene3D images. If your dataset is relatively small you might be able to get away with just using exhaustive matching.
Re: Using COLMAP instead of PixSFM - PixSFM uses COLMAP as part of their pipeline and does additional refinement before and after bundle adjustment, and they actually write the final result in COLMAP format. I just pushed a script that should convert COLMAP output (from PixSFM or just using COLMAP) at https://github.com/cmusatyalab/mega-nerf/blob/main/scripts/colmap_to_mega_nerf.py. As a word of caution, Mega-NeRF (and NeRF in general) is extremely sensitive to good camera poses, and we observed differences of over 6 PSNR when comparing PixSFM poses to that of a commercial photogrammetry software we initially used. So I'd strongly recommend using PixSFM if possible. And in either case, you will likely want to geo-register your result dataset using COLMAP's model_aligner utility (https://colmap.github.io/faq.html#geo-registration) so that Mega-NeRF's --ray_altitude_range parameter works properly. If you don't have geotags for your image you might be able to get away with manhattan world alignment (https://colmap.github.io/faq.html#manhattan-world-alignment) but it will probably not give you perfect bounds.
Hope that this helps!

Hello, when i use the scripts to refine camera pose,but when run the pairs_from_gps, i got the error:
##### <Element '{http://www.w3.org/1999/02/22-rdf-syntax-ns#}Description' at 0x7feb8705b860>
Traceback (most recent call last):
  File "/home/paper/Desktop/meganerf/pixs.py", line 53, in <module>
    main(_get_opts())
  File "/home/paper/Desktop/meganerf/pixs.py", line 39, in main
    pairs_from_gps.main(sfm_pairs_path, images_path, references, 100, 4)
  File "/home/paper/Desktop/pixel-perfect-sfm/Hierarchical-Localization/hloc/pairs_from_gps.py", line 100, in main
    lat, long, alt, R, date = get_gps_pos(image_dir / image_id)
  File "/home/paper/Desktop/pixel-perfect-sfm/Hierarchical-Localization/hloc/pairs_from_gps.py", line 59, in get_gps_pos
    lat = float(root[0][0].get('{http://www.dji.com/drone-dji/1.0/}GpsLatitude'))
TypeError: float() argument must be a string or a number, not 'NoneType'

How can i fix it ? Thanks a lot.

Hi @xingyi-li - thanks for the kind words. To answer your questions:
Re: the initial camera poses from the GPS and IMU - we're just using the GPS geotags in the image metadata. Often this is embedded in the JPEG metadata itself (for example https://developer.parrot.com/docs/pdraw/photo-metadata.html for ANAFI photos). We also used to use the camera orientation back when we were refining poses with a commercial solution, but didn't use it for our PixSFM poses.
The PixSFM repo explains their process in detail but our usage of it is pretty straightforward - https://gist.github.com/hturki/b3ae2e020bcc2a08dbba402786f1fcd3 should give you an idea of how we refine the poses with it. The main deviation is that we used a custom GPS matcher which limits the image matching to only other images in its vicinity (https://gist.github.com/hturki/5a76b1ee54b8cd23e34e075f8a8b87e0) using the metadata found in the Mill19 and UrbanScene3D images. If your dataset is relatively small you might be able to get away with just using exhaustive matching.
Re: Using COLMAP instead of PixSFM - PixSFM uses COLMAP as part of their pipeline and does additional refinement before and after bundle adjustment, and they actually write the final result in COLMAP format. I just pushed a script that should convert COLMAP output (from PixSFM or just using COLMAP) at https://github.com/cmusatyalab/mega-nerf/blob/main/scripts/colmap_to_mega_nerf.py. As a word of caution, Mega-NeRF (and NeRF in general) is extremely sensitive to good camera poses, and we observed differences of over 6 PSNR when comparing PixSFM poses to that of a commercial photogrammetry software we initially used. So I'd strongly recommend using PixSFM if possible. And in either case, you will likely want to geo-register your result dataset using COLMAP's model_aligner utility (https://colmap.github.io/faq.html#geo-registration) so that Mega-NeRF's --ray_altitude_range parameter works properly. If you don't have geotags for your image you might be able to get away with manhattan world alignment (https://colmap.github.io/faq.html#manhattan-world-alignment) but it will probably not give you perfect bounds.
Hope that this helps!

Hello, when i use the scripts to refine camera pose,but when run the pairs_from_gps, i got the error:
##### <Element '{http://www.w3.org/1999/02/22-rdf-syntax-ns#}Description' at 0x7feb8705b860>
Traceback (most recent call last):
  File "/home/paper/Desktop/meganerf/pixs.py", line 53, in <module>
    main(_get_opts())
  File "/home/paper/Desktop/meganerf/pixs.py", line 39, in main
    pairs_from_gps.main(sfm_pairs_path, images_path, references, 100, 4)
  File "/home/paper/Desktop/pixel-perfect-sfm/Hierarchical-Localization/hloc/pairs_from_gps.py", line 100, in main
    lat, long, alt, R, date = get_gps_pos(image_dir / image_id)
  File "/home/paper/Desktop/pixel-perfect-sfm/Hierarchical-Localization/hloc/pairs_from_gps.py", line 59, in get_gps_pos
    lat = float(root[0][0].get('{http://www.dji.com/drone-dji/1.0/}GpsLatitude'))
TypeError: float() argument must be a string or a number, not 'NoneType'

How can i fix it ? Thanks a lot.

I guess the images you use don't have meta data that the program can read.

Hi @xingyi-li - thanks for the kind words. To answer your questions:
Re: the initial camera poses from the GPS and IMU - we're just using the GPS geotags in the image metadata. Often this is embedded in the JPEG metadata itself (for example https://developer.parrot.com/docs/pdraw/photo-metadata.html for ANAFI photos). We also used to use the camera orientation back when we were refining poses with a commercial solution, but didn't use it for our PixSFM poses.
The PixSFM repo explains their process in detail but our usage of it is pretty straightforward - https://gist.github.com/hturki/b3ae2e020bcc2a08dbba402786f1fcd3 should give you an idea of how we refine the poses with it. The main deviation is that we used a custom GPS matcher which limits the image matching to only other images in its vicinity (https://gist.github.com/hturki/5a76b1ee54b8cd23e34e075f8a8b87e0) using the metadata found in the Mill19 and UrbanScene3D images. If your dataset is relatively small you might be able to get away with just using exhaustive matching.
Re: Using COLMAP instead of PixSFM - PixSFM uses COLMAP as part of their pipeline and does additional refinement before and after bundle adjustment, and they actually write the final result in COLMAP format. I just pushed a script that should convert COLMAP output (from PixSFM or just using COLMAP) at https://github.com/cmusatyalab/mega-nerf/blob/main/scripts/colmap_to_mega_nerf.py. As a word of caution, Mega-NeRF (and NeRF in general) is extremely sensitive to good camera poses, and we observed differences of over 6 PSNR when comparing PixSFM poses to that of a commercial photogrammetry software we initially used. So I'd strongly recommend using PixSFM if possible. And in either case, you will likely want to geo-register your result dataset using COLMAP's model_aligner utility (https://colmap.github.io/faq.html#geo-registration) so that Mega-NeRF's --ray_altitude_range parameter works properly. If you don't have geotags for your image you might be able to get away with manhattan world alignment (https://colmap.github.io/faq.html#manhattan-world-alignment) but it will probably not give you perfect bounds.
Hope that this helps!

Hi, @hturki Thanks for your clear explanation. I wonder how did you finally get a whole model for a scene? I tried your code on Mill 9 dataset but I got multiple small models.


Powershell Script Html tags automatically capitalized and quotes removed from output

Prerequisites

[X] Write a descriptive title.
[X] Make sure you are able to repro it on the latest released version
[X] Search the existing issues.
[X] Refer to the FAQ.
[X] Refer to Differences between Windows PowerShell 5.1 and PowerShell.

Steps to reproduce
I am using below powershell script to read an html file and wanted to modify certain tags with missing attributes.
$Source = Get-Content -path example.html -raw
$HTML = New-Object -Com "HTMLFile"
$HTML.IHTMLDocument2_write($Source)

$html.all.tags('HTML') |
	ForEach-Object{
		$_.document.documentElement.OuterHTML
	} | Out-File "test1.html"

I am facing a strange issue in which all tags gets capitalized and quotes are missing in final html file generated.
e.g

 becomes

  becomes

I tried setting encoding as well but it didn't work as well
$content | Out-File -Encoding utf8 "test1.html"
Any idea how can I avoid the capitalization of HTML tags and also keep the attribute Quotes intact?
Expected behavior
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">

Actual behavior

<META content="text/html; charset=UTF-8" http-equiv=Content-Type>
<META name=Generator content="Microsoft Word 15 (filtered)">


8.3-rc-3: The value for property 'languageVersion' is final and cannot be changed any further

Current Behavior
When using Java plugin with org.graalvm.buildtools.native:
java {
    toolchain {
        languageVersion.set(JavaLanguageVersion.of(20)) // <-- the error occurs here
    }
}

graalvmNative {
    binaries {
        named("main") {
            javaLauncher.set(javaToolchains.launcherFor {
                languageVersion.set(JavaLanguageVersion.of(20))
            })
        }
    }
}

I got an error:
* What went wrong:
The value for property 'languageVersion' is final and cannot be changed any further.

Expected Behavior
It was working fine in 8.2.1.
Context (optional)
No response
Steps to Reproduce
Using Java plugin with "org.graalvm.buildtools.native".
Gradle version
8.3-rc-3
Build scan URL (optional)
No response
Your Environment (optional)
No response

Thank you for providing a valid report.
The issue is in the backlog of the relevant team and is prioritised by them.

I made a reproducer here: https://github.com/eskatos/issue-reproducers/tree/main/gradle/issue-26061
Build Scan for the failure: https://scans.gradle.com/s/5q2vmspnm7r2o/failure#1
@Eng-Fouad, please next time provide a valid complete reproducer to make it easier for us to triage issues.

We found that reverting that PR fixes it

https://github.com/gradle/gradle/pull/25126

Related

https://github.com/gradle/gradle/issues/25966

As mentioned above, https://github.com/gradle/gradle/pull/25126 is indeed the cause of this behavior change. A brief explanation of the cause can be found in a related issue, here. After careful consideration, we have decided to accept this change in behavior for Gradle 8.3.
The root cause of this change was the removal of an internal mechanism, beforeLocking, which provided Configurations a means of lazily calculating attributes. https://github.com/gradle/gradle/pull/25126 removed our internal usage of this API, by migrating to the public, lazy, provider-accepting methods on AttributeContainer to configure Configuration attributes. In particular, the migrated attributes were org.gradle.libraryelements and org.gradle.jvm.version.
The specific root cause of this issue is the org.gradle.jvm.version, which has its value calculated by realizing the javaCompile task, and extracting the target JVM value for the configured test suite. Given the recent changes, querying the value of org.gradle.jvm.version will lock in the configured test suite, as is evident by this issue report. (Note, this is not necessarily desired behavior, but changing that is an even larger breaking change).
Digging deeper, the GraalVM plugin is indeed querying the values of these the runtimeClasspath's attributes -- at configuration time -- before the user has the opportunity to configure the toolchain. This effectively locks-in the toolchain when the plugin is applied. Note that this issue uncovers an existing problem with the GraalVM plugin, where previously it was silently not copying these two attributes, as they were only present after beforeLocking was called.
We have decided to accept this behavior change, primarily due to the fact that it will need to happen at some point -- and because there are workarounds (see below). beforeLocking as it was designed inherently blocks important optimizations that we plan to include in future versions of Gradle, so reverting this change will only be a step backwards.
In terms of solutions to this issue, the fix must be done by the GraalVM plugin. We recommend the following options, with our most recommended option first. (Note, we will have a detailed description of this issue in the upgrade guide upon the release of 8.3)

Declare the attributes for the native configuration explicitly, without copying them from runtimeClasspath. These attributes are well-defined and can be hard-coded when the configuration is created.
Copy the attributes in the same manner as currently, except use the attributeProvider to lazily query the attributes of the source Configuration (runtimeClasspath).
Wrap the creation of the Configuration in an afterEvaluate. While this is a solution, we highly discourage it. Kotlin is experiencing a similar problem, as it also copies attributes in a similar fashion.

As for your own solutions, I see two options currently

Wait for GraalVM to fix this issue. Remain on Gradle 8.2 until the fix is released
Configure the toolchain in your own custom plugin, which is applied before the GraalVM plugin.

I agree that solution 2 is the best. However, I would like to express my dissatisfaction about the process. Because while 2 is the best solution, this is again a new API which didn't exist before, which must be used. Using that new API is not possible when you want to support multiple Gradle plugin versions. Of course, you will say "actually it is", but it's extremely cumbersome. I am currently in vacation and will work on a fix when I'm back, but this will involve reflection so that for older version of Gradle this works, while also on newer releases. It would really be better if Gradle didn't break backwards compatibility. This isn't about binary compatibility but runtime one.

For me, upgrading org.graalvm.buildtools.native plugin to the latest helped.

For me, upgrading org.graalvm.buildtools.native plugin to the latest helped.

for me it did not unfortunately :-( I am using gradle 8.4 and "org.graalvm.buildtools.native" version "0.9.28"
and I get the error...

Anyone finding this in the future: if you're not using GraalVM, you can also try to change io.micronaut.application plugin to io.micronaut.minimal.application which removes the transitive GraalVM plugin application.


This branch implements the CSV and native HA support for Hyper-V

For design related details refer to https://cwiki.apache.org/confluence/display/CLOUDSTACK/iSCSI+and+HA+support+in+Hyper-V.
This is similar to VmWare in design and implementation.

Hi @anshul1886,
Thanks for the PR :-) Looks like quite some work!
I don't have a Hyper-V setup to test it.. Will ask around and see who can help test it.

@anshul1886 looks like it failed the tests. Can you check what's happening and correct accordingly ?

@runseb These are false failures. The changes on which these tests are failing are in Hyper-V agent so can be safely ignored as these tests are not testing Hyper-V Agent.

I'll take a look and get back if I have any queries

Rebased the code to latest master

@anshul1886 Can you update the PR? It doesn't merge clean.
Anyone able to review Hyper-v?

@remibergsma,
Rebased the branch to latest master

Removed the MigrateCompleteCommand as per review comments

Unit test which is failing is testing KVM code and is not related to any changes in this file

Updated the code as per review comments

Changes looks good to me. @remibergsma Are you ok with these changes? I'll go ahead and merge them then.

@devdeep Did you LGTM on this? Were you able to test / verify it with a hyper-v cluster?
@anshul1886 Can you look at the commits and see if you can squash some of them? I'm not saying it should be one commit, but I do want them to be atomic. Also, I don't see any tests for this? How are we going to make sure this keeps working once it is merged?
Pinging @karuturi @DaanHoogland to see if they can review the code as I'm not a Java guru. Thanks!

@remibergsma Either we have to squash all commits to one commit or leave it as it is. In squashing all commits to one I will put all the relevant messages. These commits don't have in order dependency. So what would you prefer one commit or leave it as it is?
Implementation have very little impact in core CloudStack so will not affect functionality of other Hypervisors. Also I have done testing on my end to make sure everything works fine after these commits.
Existing unit tests will be able to identify the issues in management server code.

@anshul1886 Then I'd say squash them into one so you commit the whole functionality as one commit. Also, when you add new code, I'd expect new unit tests as well? Otherwise the test coverage of ACS goes down.

I don't see any test, we can not release this to the public without limitation of warranty. extensive testing needs to be implemented to give any degree of certainty on the functionality. there is not a single unit test and no reference to applicable integration tests or #!human scripts.

@DaanHoogland Updated the code with review comments.
Regarding unit tests I am identifying modules for which unit tests can be written. But be aware that unit tests are meant for development not for functional testing.

@anshul1886 if the purpose is to ensure that the path returned doesn't change, this seems perfectly aceptable to me. If people want to change the path in the future they must be aware and change the unit tests as well. That said they seem a litle trivial but I'm alright with that. It is mainly the integrity class hierarchy of the resources that is being tested this way.

@anshul1886 please re-base against latest master, update on current status of your work/PR

@swill @rhtyd Updated the PR against the latest master. It was conflicting in imports. Who is going to test this? AFAIK I am the only guy working in Hyper-V.

Can you run the following tests on this PR and post the results?  I will run my KVM CI against it to verify nothing is broken for other hypervisors.
echo "Running tests with required_hardware=true"
nosetests --with-marvin --marvin-config=${marvinCfg} -s -a tags=advanced,required_hardware=true \
smoke/test_password_server.py \
smoke/test_vpc_redundant.py \
smoke/test_routers_iptables_default_policy.py \
smoke/test_routers_network_ops.py \
smoke/test_vpc_router_nics.py \
smoke/test_router_dhcphosts.py \
smoke/test_loadbalance.py \
smoke/test_internal_lb.py \
smoke/test_ssvm.py \
smoke/test_vpc_vpn.py \
smoke/test_privategw_acl.py \
smoke/test_network.py

echo "Running tests with required_hardware=false"
nosetests --with-marvin --marvin-config=${marvinCfg} -s -a tags=advanced,required_hardware=false \
smoke/test_routers.py \
smoke/test_network_acl.py \
smoke/test_reset_vm_on_reboot.py \
smoke/test_vm_life_cycle.py \
smoke/test_service_offerings.py \
smoke/test_network.py \
component/test_vpc_offerings.py \
component/test_vpc_routers.py

CI RESULTS
Tests Run: 81
  Skipped: 0
   Failed: 0
   Errors: 1
 Duration: 8h 06m 18s

Summary of the problem(s):
ERROR: test suite for <class 'integration.smoke.test_internal_lb.TestInternalLb'>
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/nose-1.3.7-py2.7.egg/nose/suite.py", line 209, in run
    self.setUp()
  File "/usr/lib/python2.7/site-packages/nose-1.3.7-py2.7.egg/nose/suite.py", line 292, in setUp
    self.setupContext(ancestor)
  File "/usr/lib/python2.7/site-packages/nose-1.3.7-py2.7.egg/nose/suite.py", line 315, in setupContext
    try_run(context, names)
  File "/usr/lib/python2.7/site-packages/nose-1.3.7-py2.7.egg/nose/util.py", line 471, in try_run
    return func()
  File "/data/git/cs1/cloudstack/test/integration/smoke/test_internal_lb.py", line 296, in setUpClass
    cls.template.download(cls.apiclient)
  File "/usr/lib/python2.7/site-packages/marvin/lib/base.py", line 1350, in download
    elif 'Downloaded' in template.status:
TypeError: argument of type 'NoneType' is not iterable
----------------------------------------------------------------------
Additional details in: /tmp/MarvinLogs/test_network_XMQTJ1/results.txt

Associated Uploads
/tmp/MarvinLogs/DeployDataCenter__May_17_2016_19_58_42_BEN56X:

dc_entries.obj
failed_plus_exceptions.txt
runinfo.txt

/tmp/MarvinLogs/test_network_XMQTJ1:

failed_plus_exceptions.txt
results.txt
runinfo.txt

/tmp/MarvinLogs/test_vpc_routers_QO25FZ:

failed_plus_exceptions.txt
results.txt
runinfo.txt

Uploads will be available until 2016-07-18 02:00:00 +0200 CEST
Comment created by upr comment.

I have seen this error with other PRs.  I am not sure exactly where it comes from.  Out of curiosity, is the PR up-to-date in terms of rebasing?
I think this is a clean CI result considering I am seeing this same issue in some other PRs.

@swill Yes, PR is up to date. It was updated yesterday.

Ok thanks.  I just wanted to verify.  I don't think I can merge this without integration tests against HyperV being run.  My tests just show it does not break other stuff, but we still don't have any proof it works for HyperV at this point.

@swill @rhtyd I am facing permission denied exceptions for some APIs in UI. Seems like they are after the merge of PR #1489.

@rhtyd can you work with @anshul1886 to understand what is going on here?  This is important as we are about to freeze master and cut 4.9.  Thanks...

@swill yeah I've asked @anshul1886 few questions and waiting for answers. Btw, since we're in a release/branch freeze, this feature PR need to be held until 4.9.0 is released.

ACS CI BVT Run
Sumarry:
Build Number 142
Hypervisor xenserver
NetworkType Advanced
Passed=70
Failed=3
Skipped=3
Link to logs Folder (search by build_no): https://www.dropbox.com/sh/yj3wnzbceo9uef2/AAB6u-Iap-xztdm6jHX9SjPja?dl=0
Failed tests:

test_vpc_vpn.py

test_01_vpc_remote_access_vpn Failed

test_volumes.py

test_06_download_detached_volume Failed

test_vm_life_cycle.py

test_10_attachAndDetach_iso Failed

Skipped tests:
test_vm_nic_adapter_vmxnet3
test_static_role_account_acls
test_deploy_vgpu_enabled_vm
Passed test suits:
test_deploy_vm_with_userdata.py
test_affinity_groups_projects.py
test_portable_publicip.py
test_over_provisioning.py
test_global_settings.py
test_scale_vm.py
test_service_offerings.py
test_routers_iptables_default_policy.py
test_routers.py
test_reset_vm_on_reboot.py
test_snapshots.py
test_deploy_vms_with_varied_deploymentplanners.py
test_login.py
test_list_ids_parameter.py
test_public_ip_range.py
test_multipleips_per_nic.py
test_regions.py
test_affinity_groups.py
test_network_acl.py
test_pvlan.py
test_nic.py
test_deploy_vm_root_resize.py
test_resource_detail.py
test_secondary_storage.py
test_disk_offerings.py

@anshul1886 @rhtyd it appears that there was a push to get this merged for the 4.9 release, but a conflict may have occurred with the Dynamic Role-based API Checker enhancement.  Were you able to resolve the issue?  It also appears that @bvbharat had 3 test failures and 3 test skips.  Where does resolution of these test issues stand?

@anshul1886 can you rebase against latest master and push -f ?

Rebased the branch against current master.

tag:needslove

ACS CI BVT Run
Sumarry:
Build Number 383
Hypervisor xenserver
NetworkType Advanced
Passed=105
Failed=0
Skipped=7
Link to logs Folder (search by build_no): https://www.dropbox.com/sh/yj3wnzbceo9uef2/AAB6u-Iap-xztdm6jHX9SjPja?dl=0
Failed tests:
Skipped tests:
test_01_test_vm_volume_snapshot
test_vm_nic_adapter_vmxnet3
test_static_role_account_acls
test_11_ss_nfs_version_on_ssvm
test_nested_virtualization_vmware
test_3d_gpu_support
test_deploy_vgpu_enabled_vm
Passed test suits:
test_deploy_vm_with_userdata.py
test_affinity_groups_projects.py
test_portable_publicip.py
test_over_provisioning.py
test_global_settings.py
test_scale_vm.py
test_service_offerings.py
test_routers_iptables_default_policy.py
test_loadbalance.py
test_routers.py
test_reset_vm_on_reboot.py
test_deploy_vms_with_varied_deploymentplanners.py
test_network.py
test_router_dns.py
test_non_contigiousvlan.py
test_login.py
test_deploy_vm_iso.py
test_list_ids_parameter.py
test_public_ip_range.py
test_multipleips_per_nic.py
test_regions.py
test_affinity_groups.py
test_network_acl.py
test_pvlan.py
test_volumes.py
test_nic.py
test_deploy_vm_root_resize.py
test_resource_detail.py
test_secondary_storage.py
test_vm_life_cycle.py
test_routers_network_ops.py
test_disk_offerings.py

@kiwiflyer why does this need TLC? or is that remark deprecated?

It had been untouched for a while, so now that it's moving forward again, you can consider this tag depreciated.

@karuturi Could you see if you could get some folks from Accelerate to help review this PR? We'd really like to get this one moved forward. There are some comments above regarding some requested unit testing to make sure this doesn't break functionality. Help would be appreciated!

Hi there, I found this PR after listening to a recording of Sowmya's presentation "Hyper-V Clustering with CloudStack".
The Accelerite Blog implies CSV's and Failover Clustering is enabled in CloudStack but when I built a test environment I could not select PreSetup when adding Primary Storage as per the presentation slides.
It appears to me the presentation was based on functionality provided in this PR, is there any chance of this ever getting merged?

@anshul1886 please rebase and re-open if still relevant


Order entry via backend -> email with payment request, cannot ship order

We entered order entry via backend
Payment method: email with payment request
The status is "Payment Review"
But we need to give the products today .... problem is that I cannot set the status to shipped
We cannot ship order (probably until after payment)
I think we should be able to ship the order if we need to

Hi Sean,
Yes, it is waiting for the payment the we set the order to processing. This is how all our payment methods work there needs to be a payment before creating the invoice. Shipment is not part of our solution only if it is a virtual product then we create a shipment but only after the invoice is created. I don't know if it is even possible to create a shipment before an invoice is created and this will screw up the whole workflow. This is not something we want to be involved in so I think you can best use a third-party module to change the status of an order or create something manually.
Regards,
Rik
Adyen

The request is not to create a shipment
The request is to not block the ability to create a shipment (method
canShip() should return true)
On Wed, Jul 15, 2015 at 12:57 PM, Rik ter Beek<EMAIL_ADDRESS>wrote:

Hi Sean,
Yes, it is waiting for the payment the we set the order to processing.
This is how all our payment methods work there needs to be a payment before
creating the invoice. Shipment is not part of our solution only if it is a
virtual product then we create a shipment but only after the invoice is
created. I don't know if it is even possible to create a shipment before an
invoice is created and this will screw up the whole workflow. This is not
something we want to be involved in so I think you can best use a
third-party module to change the status of an order or create something
manually.
Regards,
Rik
Adyen
—
Reply to this email directly or view it on GitHub
https://github.com/Adyen/magento/issues/367#issuecomment-121579198.

Hi Sean,
This is the common magento flow. This is only available when invoice is created.
Regards,
Rik
Adyen


Dose MQTTnet support Native AOT?

Describe the feature request
i will generate a dll lib via native aot, Dose MQTTnet support Native AOT?

I don't know if there is anything we can actively do to support it. I assume you need to just give it a try if it works or not.


Improve RealityTile loading priority calculation

This PR modifies the function computing the loading priority of RealityTiles.
Previously, the priority corresponded to the distance from the view point of tile tile center.
This PR gives the tiles in the center of the screen a better priority than the ones next to the sides, the user being more likely to look at the center of the viewport.
The new priority value is a mix between the normalized distance to the frustum center axis and the old distance to view point, giving to the later to stronger weight.
The distance from the center of the screen is get by computing the distance in NPC of the tile location without the z coordinate (the depth) to the point (0.5, 0.5).

@DStradley can you email @TitouanLeDuigou instructions for running ImageTests against his branch?

@TitouanLeDuigou please move your changes to a branch in the itwinjs-core repository so that we can run ImageTests against it. This also simplifies the PR process.

I've moved @TitouanLeDuigou's changes to this branch and enqueued an ImageTests run.

ImageTests produced no visual differences. Minor fluctuations in tile loading time and GPU memory usage averaged out to ~ net zero change.

@RBBentley may have comments.

@Mergifyio update

@calebmshafer please point us to documentation regarding how to get this built and merged into itwinjs-core repo.

@calebmshafer please point us to documentation regarding how to get this built and merged into itwinjs-core repo.

@pmconne sure, it's a pain at the moment. I'll send details.  Also looks like the extract api build is failing.

@TitouanLeDuigou please do:
rush build --to core-frontend
cd core\frontend
npm run extract-api
git commit -am"extract-api"
git push

@RBBentley may have comments.

This looks like a good optimization to me... I believe Cesium may use a similar prioritization.

@Mergifyio update

@TitouanLeDuigou please run rushchange.bat from the repo's root directory and push the result (assuming you are on WIndows).

GitHub checks passed; ADO builds queued.


Recovery from power outage and shoot above a lux threshold (Construction Cam)

Love the project. Not really an issue, but wondering if you have any suggestion in setting it up so it only records during the daylight between sunrise and sunset. I would like to use your project for a small construction project that is going to last about 3 months.
Thanks !

Hi, the code Is not linked with a calendar, therefore it isn't possible to start shooting at sunrise and stop at sunset.
The code accepts a start time, a stop time, and eventually the number of days to repeat the shooting between start and stop.
During the construction, I would recommend to approach the Raspberry Pi from time to time, and discharge the pictures or the video clips. This to backup the data, preventing total loss in case of a any issue, for instance the microSD getting corrupted.

Appreciate the suggestions. I did find a script that will look at images and determine if they are black. I think I am having some trouble though with my settings. I am hoping to set my timelapse to run every day and take a picture at 600 seconds for three months. I was thinking about rebooting the device nightly as well(or might change it to the week) Thinking about recovery if a power outage was to occur and where it could pick up where its at.
{
"preview": "False",
"erase_pics": "False",
"erase_movies": "False",
"local_control": "False",
"start_now": "True",
"period_hhmm": "01:00",
"start_hhmm": "00:01",
"end_hhmm": "23:59",
"interval_s": "600",
"days": "120",
"rendering": "False",
"fix_movie_t": "False",
"movie_time_s": "20",
"fps": "24",
"overlay_fps": "False",
"overlay_text": "",
"camera_w": "1920",
"camera_h": "1080",
"hdr": "True",
"autofocus": "True",
"focus_dist_m": "0.1",
"date_folder": "True",
"folder": "timelapse_pics",
"parent_folder": "/home/weedy/shared",
"pic_name": "picture",
"pic_format": "jpg",
"rotate_180": "False",
"display": "True",
"modified_disp": "False",
"disp_preview": "False",
"disp_image": "False",
"disp_rotation": "90",
"disp_bright": "100",
"disp_width": "240",
"disp_height": "240",
"disp_offsetL": "0",
"disp_offsetT": "0"
}

Hi, getting the code starting automatically at the boot is rather easy, I'll follow up.
The current code is not meant to restart from where it left, but it makes a lot of sense in case of power outage.
Once back from holiday I'll  modify the code.
When is the construction going to start?
When will you have the setup ready for testing?
Considering the expected large variation of light condition, strongly suggested a camera with HDR function, so the V3 normal or the V3 wide.
Also recommended to set a couple of wifi addresses at the Raspberry Pi, one of them with your laptop hotspot (Windows allows it if the PC has a wifi connection, that could be to your phone hotspot)
For the settings, there is a bit more explanation  at the Instructables project page.
Considering your scope, I'd set start_hhmm when the daily activities are expected to start, let say 0700. Similarly, the stop__hhmm when the daily activities are supposed to stop, let say 1900.
I'd also reduce the interval_s to 300 seconds, and eventually double the frames per second of the video clip, for a smoother result. Total quantity of images are still manageable, so the memory.
I have a few more notes about settings, but those will come later

Thanks again for your replies.
My construction project is starting the end of next month. I was starting to play around with it to get a feel for the size of pictures and how often I would need to get grab them and how well it would work on recovery of power loss if their was any(don’t have a way to put a UPS outside)
I am using this camera https://www.adafruit.com/product/5658 -- I tried a higher resolution 4608 × 2592 pixels but it was not happy about that.
From: Andrea @.>
Sent: Sunday, July 28, 2024 3:10 PM
To: AndreaFavero71/timelapse @.>
Cc: weedy171 @.>; Author @.>
Subject: Re: [AndreaFavero71/timelapse] Construction Cam (Issue #2)
Hi, getting the code starting automatically at the boot is rather easy, I'll follow up.
The current code is not meant to restart from where it left, but it makes a lot of sense in case of power outage.
Once back from holiday I'll modify the code.
When is the construction going to start?
When will you have the setup ready for testing?
Considering the expected large variation of light condition, strongly suggested a camera with HDR function, so the V3 normal or the V3 wide.
Also recommended to set a couple of wifi addresses at the Raspberry Pi, one of them with your laptop hotspot (Windows allows it if the PC has a wifi connection, that could be to your phone hotspot)
For the settings, there is a bit more explanation at the Instructables project pagehttps://www.instructables.com/Timelapse-With-Raspberry-Pi-4b-and-PiCamera-V3-wid/.
Considering your scope, I'd set start_hhmm when the daily activities are expected to start, let say 0700. Similarly, the stop__hhmm when the daily activities are supposed to stop, let say 1900.
I'd also reduce the interval_s to 300 seconds, and eventually double the frames per second of the video clip, for a smoother result. Total quantity of images are still manageable, so the memory.
I have a few more notes about settings, but those will come later
—
Reply to this email directly, view it on GitHubhttps://github.com/AndreaFavero71/timelapse/issues/2#issuecomment-2254631191, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ACQNONYQLIIPCLAEK6HCL73ZOVF3FAVCNFSM6AAAAABLLMXR5SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJUGYZTCMJZGE.
You are receiving this because you authored the thread.Message ID<EMAIL_ADDRESS>

Hi, that camera supports the HDR (strongly suggested to use it) by reducing the resolution to the HD format. Depending on the subject, the JPG file might take from 250Kb (large uniform areas) to 600Kb (small uniform areas).
I believe, by mid of August, I'll be able to add the info for the automatic start after booting, and the code change to proceed with pictures' names from where it was before an eventual power outage....

Thanks for looking into this while you are on holiday. I updated the latest py and updated my settings to the following. I noticed it created the folder "timelapse_pics" inside my shared but don't see any pictures in it. I was able prior to get the autostart the py while and the old py was generating images.
{
"preview": "True",
"erase_pics": "False",
"erase_movies": "False",
"local_control": "False",
"start_now": "False",
"period_hhmm": "8760:00",
"start_hhmm": "06:00",
"end_hhmm": "19:00",
"interval_s": "180",
"days": "120",
"rendering": "False",
"fix_movie_t": "False",
"movie_time_s": "20",
"fps": "24",
"overlay_fps": "False",
"overlay_text": "",
"camera_w": "1920",
"camera_h": "1080",
"hdr": "True",
"autofocus": "True",
"focus_dist_m": "0.1",
"date_folder": "False",
"folder": "timelapse_pics",
"parent_folder": "/home/weedy/shared",
"pic_name": "picture",
"pic_format": "jpg",
"rotate_180": "False",
"display": "False",
"modified_disp": "False",
"disp_preview": "False",
"disp_image": "False",
"disp_rotation": "90",
"disp_bright": "100",
"disp_width": "240",
"disp_height": "240",
"disp_offsetL": "0",
"disp_offsetT": "0"
}

Hi, when the code is started at the boot (crontab), all the prints will be
saved in the text file timelapse_terminal.log.
Everytime the Raspberry Pi boots, the same file will be used (over written)
Please boot the system again, and check the content of that file within the
expected shooting period of the day.
In case or errors indications, share the complete file to
@.***
Perhaps good to share it anyhow.
I'm going to be driving the coming days, therefore I'll be rather limited
... yet good for me to start thinking what goes grong and why.
On Thu, 8 Aug 2024, 03:30 weedy171, @.***> wrote:

Thanks for looking into this while you are on holiday. I updated the
latest py and updated my settings to the following. I noticed it created
the folder "timelapse_pics" inside my shared but don't see any pictures in
it. I was able prior to get the autostart the py while and the old py was
generating images.
{
"preview": "True",
"erase_pics": "False",
"erase_movies": "False",
"local_control": "False",
"start_now": "False",
"period_hhmm": "8760:00",
"start_hhmm": "06:00",
"end_hhmm": "19:00",
"interval_s": "180",
"days": "120",
"rendering": "False",
"fix_movie_t": "False",
"movie_time_s": "20",
"fps": "24",
"overlay_fps": "False",
"overlay_text": "",
"camera_w": "1920",
"camera_h": "1080",
"hdr": "True",
"autofocus": "True",
"focus_dist_m": "0.1",
"date_folder": "False",
"folder": "timelapse_pics",
"parent_folder": "/home/weedy/shared",
"pic_name": "picture",
"pic_format": "jpg",
"rotate_180": "False",
"display": "False",
"modified_disp": "False",
"disp_preview": "False",
"disp_image": "False",
"disp_rotation": "90",
"disp_bright": "100",
"disp_width": "240",
"disp_height": "240",
"disp_offsetL": "0",
"disp_offsetT": "0"
}
—
Reply to this email directly, view it on GitHub
https://github.com/AndreaFavero71/timelapse/issues/2#issuecomment-2274740743,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AZYVQEYTX2DXMNLROQVE6J3ZQLCZZAVCNFSM6AAAAABLLMXR5SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENZUG42DANZUGM
.
You are receiving this because you commented.Message ID:
@.***>

I just checked this morning and it started taking pictures at the start period.  I will have to play with it later in the day to see if a reboot of the device makes it still work.

I think I figured it out, I had the end time set earlier in the night compared to when i was working on the script. It appears to be working. I will play around with it some more. Do you think the resolution I have set could be any better or you think that is good for the Camera 3?

Hi, sounds good.
1920x1080 is the max resolution for the V3 camera, when HDR is set True.
The video clip will be HD.

Hi, the just release commit (https://github.com/AndreaFavero71/timelapse/commit/f435cf39e1a4b105b8052582787631e12a74205c) solves a couple of bugs from the previous commit.
Additionally, it adds the possibility to only shoot when the camera estimated Lux is above a threshold; This should answer the original request, based on illuminance instead of sunrise and sunset time.
Below snippet is from the top part of the "main" function:
    ################    mix lux level check (optional)   ###########################################

    # Lux levels from https://en.wikipedia.org/wiki/Daylight
    # 120000        Brightest sunlight
    # 111000        Bright sunlight
    # 109880        AM 1.5 global solar spectrum sunlight (= 1,000.4 W/m2)[3][circular reference]
    # 20000         Shade illuminated by entire clear blue sky, midday
    # 1000 – 2000   Typical overcast day, midday
    # 400           Sunrise or sunset on a clear day (ambient illumination)
    # <200          Extreme of thickest storm clouds, midday
    # 40            Fully overcast, sunset/sunrise
    # <1            Extreme of thickest storm clouds, sunset/rise

    lux_check = False                          # flag to suject the shooting to a min Lux value
    min_lux_threshold = 30                     # minimum Lux threshold, when lux_check is set True
    # ##############################################################################################

If the lux_check is set True (default is False) the code only grabs a picture if the camera's estimated Lux is above the min_lux_threshold. Remaining functionality does not change.
I don't have experience to judge how well the PiCamera V3 estimates the Lux.
As a test, yesterday evening I set the camera to start shooting today from 3.00am and it started at 5:40am ...  literally when half sun was above the horizion, according to internet sunrise info where I live. The min_lux_threshold was set to 30.
The min_lux_threshold also works to stop shooting once the illuminance drop at sunset.
For the Contruction Cam I'd advise to set the start_hhmm not too much earlier than the expected time there will be construction activity (let's say 7.00), and the stop_hhmm not too much later than the expected time the construction activity are going to finish (let's say 21.00). This will prevent to have excess of similar pictures ....
Of course, the Lux control can be used to take pictures as long as the camera sees enough illuminance.

I noticed powering off the device and powering it back on it did not recover.  I am guessing tomorrow it will start over.
Local control is disabled
Picture resolution: 1920x1080
Picture format: jpg
Picture file size 0.28 Mb  (picture size varies on subject and light!)
Free disk space: 111297 Mb
Rough max number of pictures: 397489
Day 4 of 120
Shooting starts:     10:13:00
Shooting ends:       21:00:00
Shooting starts in:  0:00:17
Shooting every:      180 seconds
Camera will take:    216 pictures today
Timelapse video render not activated
Display at Raspberry Pi not activated
##############################################################################
##############################################################################
14 Aug 2024 10:12:55     frame: 00000 *************************************************
14 Aug 2024 12:42:53     frame: 00050 **************

Hi, could you elaborate a bit more?
Was the power off within your time 10:13:00 and 21:00:00?
Any chance the powering on happened at about your time 10:13:00? I'd expect this as a start_hhmm 10:13:00 seems defined by the code when powered after the intended start_hhmm (i.e. 07:00).
Was the power off a hardware disconnection?

Looks like around 1:24PM. I unplugged the cable. The last picture was taken on 8/14/2024
The start and stop isn't different from my last settings I posted but whats interesting is I rebooted normally and I was going to review the settings and now it says its open by a PID 1447. I ran top but could not find the process tied to it.
Sent from Mailhttps://go.microsoft.com/fwlink/?LinkId=550986 for Windows

From: Andrea @.>
Sent: Wednesday, August 14, 2024 4:45:41 PM
To: AndreaFavero71/timelapse @.>
Cc: weedy171 @.>; Author @.>
Subject: Re: [AndreaFavero71/timelapse] Recovery from power outage and shoot above a lux threshold (Construction Cam) (Issue #2)
Hi, could you elaborate a bit more?
Was the power off within your time 10:13:00 and 21:00:00?
Any chance the powering on happened at about your time 10:13:00? I'd expect this as a start_hhmm 10:13:00 seems defined by the code when powered after the intended start_hhmm (i.e. 07:00).
Was the power off a hardware disconnection?

Reply to this email directly, view it on GitHubhttps://github.com/AndreaFavero71/timelapse/issues/2#issuecomment-2289973715, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ACQNON5KUDTKVQMQ3EOOP2DZRPFYLAVCNFSM6AAAAABLLMXR5SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOBZHE3TGNZRGU.
You are receiving this because you authored the thread.Message ID: @.***>

Have you checked if picture were saved in folder, from 1.25pm onward, and
the terminal.log file?
On Thu, 15 Aug 2024, 00:36 weedy171, @.***> wrote:

Looks like around 1:24PM. I unplugged the cable. The last picture was
taken on 8/14/2024
The start and stop isn't different from my last settings I posted but
whats interesting is I rebooted normally and I was going to review the
settings and now it says its open by a PID 1447. I ran top but could not
find the process tied to it.
Sent from Mailhttps://go.microsoft.com/fwlink/?LinkId=550986 for Windows

From: Andrea @.>
Sent: Wednesday, August 14, 2024 4:45:41 PM
To: AndreaFavero71/timelapse @.>
Cc: weedy171 @.>; Author @.>
Subject: Re: [AndreaFavero71/timelapse] Recovery from power outage and
shoot above a lux threshold (Construction Cam) (Issue #2)
Hi, could you elaborate a bit more?
Was the power off within your time 10:13:00 and 21:00:00?
Any chance the powering on happened at about your time 10:13:00? I'd
expect this as a start_hhmm 10:13:00 seems defined by the code when powered
after the intended start_hhmm (i.e. 07:00).
Was the power off a hardware disconnection?

Reply to this email directly, view it on GitHub<
https://github.com/AndreaFavero71/timelapse/issues/2#issuecomment-2289973715>,
or unsubscribe<
https://github.com/notifications/unsubscribe-auth/ACQNON5KUDTKVQMQ3EOOP2DZRPFYLAVCNFSM6AAAAABLLMXR5SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOBZHE3TGNZRGU

.
You are receiving this because you authored the thread.Message ID:
@.***>

—
Reply to this email directly, view it on GitHub
https://github.com/AndreaFavero71/timelapse/issues/2#issuecomment-2290036002,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AZYVQE42CSC7GINO3WI3QQDZRPLW5AVCNFSM6AAAAABLLMXR5SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJQGAZTMMBQGI
.
You are receiving this because you commented.Message ID:
@.***>

In pervious days yes they were onwards after 1:25pm. The terminal log is what I posted prior. It appears after a couple a sudo reboots its started working on its own last night.
From: Andrea @.>
Sent: Wednesday, August 14, 2024 11:10 PM
To: AndreaFavero71/timelapse @.>
Cc: weedy171 @.>; Author @.>
Subject: Re: [AndreaFavero71/timelapse] Recovery from power outage and shoot above a lux threshold (Construction Cam) (Issue #2)
Have you checked if picture were saved in folder, from 1.25pm onward, and
the terminal.log file?
On Thu, 15 Aug 2024, 00:36 weedy171<EMAIL_ADDRESS>wrote:

Looks like around 1:24PM. I unplugged the cable. The last picture was
taken on 8/14/2024
The start and stop isn't different from my last settings I posted but
whats interesting is I rebooted normally and I was going to review the
settings and now it says its open by a PID 1447. I ran top but could not
find the process tied to it.
Sent from Mailhttps://go.microsoft.com/fwlink/?LinkId=550986 for Windows

From: Andrea<EMAIL_ADDRESS>Sent: Wednesday, August 14, 2024 4:45:41 PM
To: AndreaFavero71/timelapse<EMAIL_ADDRESS>Cc: weedy171<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Subject: Re: [AndreaFavero71/timelapse] Recovery from power outage and
shoot above a lux threshold (Construction Cam) (Issue #2)
Hi, could you elaborate a bit more?
Was the power off within your time 10:13:00 and 21:00:00?
Any chance the powering on happened at about your time 10:13:00? I'd
expect this as a start_hhmm 10:13:00 seems defined by the code when powered
after the intended start_hhmm (i.e. 07:00).
Was the power off a hardware disconnection?

Reply to this email directly, view it on GitHub<
https://github.com/AndreaFavero71/timelapse/issues/2#issuecomment-2289973715>,
or unsubscribe<
https://github.com/notifications/unsubscribe-auth/ACQNON5KUDTKVQMQ3EOOP2DZRPFYLAVCNFSM6AAAAABLLMXR5SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOBZHE3TGNZRGU

.
You are receiving this because you authored the thread.Message ID:
<EMAIL_ADDRESS>
—
Reply to this email directly, view it on GitHub
https://github.com/AndreaFavero71/timelapse/issues/2#issuecomment-2290036002,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AZYVQE42CSC7GINO3WI3QQDZRPLW5AVCNFSM6AAAAABLLMXR5SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJQGAZTMMBQGI
.
You are receiving this because you commented.Message ID:
<EMAIL_ADDRESS>
—
Reply to this email directly, view it on GitHubhttps://github.com/AndreaFavero71/timelapse/issues/2#issuecomment-2290534809, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ACQNONYOFAWUP2MKZ7CXCG3ZRQSZDAVCNFSM6AAAAABLLMXR5SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJQGUZTIOBQHE.
You are receiving this because you authored the thread.Message ID<EMAIL_ADDRESS>

Hi, after a power outage, the code is started automatically at the boot, and if you reconnect to the Raspberry Pi you won't see the prints made by code to the terminal ... yet prints are redirected to a text file: /timelapse/timelapse_terminal.log (this is what I meant as "terminal.log file").
So you can open that text file, to see what kind of feedback the code is printing out, further than checking the timestamp of the pictures in folder.
Be aware the boot takes a couple of minutes and the first picture will be taken at about the next minute chance; When waiting, without doing anything lse, this time seems even longer and it might let you think the code not being active ....
Having the mini display on the Raspberry Pi makes life easier :-)
On the tests I performed, everything was working fine.
Overall, I have hard time to follow the chain of events at your end, I'm afraid it requires more detailed and clear info from you.

Thanks for the replies. I was playing around with it tonight and doing a couple of power outage simulations and do not see the same results like I reported prior. I haven’t had a chance to play around with the daylight setting. I might just leave it alone. I plan on checking the camera at least once a week and grab the pictures and feel comfortable with what I have setup. Appreciate the code fix and the contributions for this project. I know some linux stuff but I am not around it lot, so its fun to play around this stuff too.
From: Andrea @.>
Sent: Thursday, August 15, 2024 1:37 PM
To: AndreaFavero71/timelapse @.>
Cc: weedy171 @.>; Author @.>
Subject: Re: [AndreaFavero71/timelapse] Recovery from power outage and shoot above a lux threshold (Construction Cam) (Issue #2)
Hi, after a power outage, the code is started automatically at the boot, and if you reconnect to the Raspberry Pi you won't see the prints made by code to the terminal ... yet prints are redirected to a text file: /timelapse/timelapse_terminal.log (this is what I meant as "terminal.log file").
So you can open that text file, to see what kind of feedback the code is printing out, further than checking the timestamp of the pictures in folder.
Be aware the boot takes a couple of minutes and the first picture will be taken at about the next minute chance; When waiting, without doing anything lse, this time seems even longer and it might let you think the code not being active ....
Having the mini display on the Raspberry Pi makes life easier :-)
On the tests I performed, everything was working fine.
Overall, I have hard time to follow the chain of events at your end, I'm afraid it requires more detailed and clear info from you.
—
Reply to this email directly, view it on GitHubhttps://github.com/AndreaFavero71/timelapse/issues/2#issuecomment-2291944073, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ACQNON6MISIL4IRKD366MNLZRTYMLAVCNFSM6AAAAABLLMXR5SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJRHE2DIMBXGM.
You are receiving this because you authored the thread.Message ID<EMAIL_ADDRESS>

Hi, I think I've discovered the reason we have different results after recovering from a power outage.
Background: The timing methd used to take the pictures is based on the EPOCH time, to prevent shifting over time.
In my case, once the Pi boots it will also find a network and an NTP (Network Time Protocol) and this might not be always the case or your case.
When the Pi boots and does not find a NTP, the internal clock proceeds from where it left: The pictures won't be taken as per intended schedule.
To overcome this issue, I see two options:

Adding a RTC module, with backup battery, to the the Pi (i.e. search for 'DS3231 Real Time Clock Module RTC Clock Memory Module 3.3V/5V for Raspberry Pi').
Changing the shootting approach (settings) to "start_now": "True" with a very long period "period_hhmm": "3000:00"  (covers 125 days) and by bounding the shooting to the illuminance level.

Option 1 is rather clean yet, further than the hardware, it requires to properly set the Pi in order to use the hardware clock after a power outage. Time alignment with NTP happen once the Pi will connect to a network. I don't have personal experience with these modules. With this option the pictures time stamp are always correct.
Option 2 should work too, yet with some caviets. In case of power outage and booting without a network available (no NTP), the pictures will be tken as per interval_s and Lux but the timestamp will be wrong.
I have now available a Pi with a screen alowing for tests with and without network connection: Tests on option 2 confirms the above notes.

I just installed the device last night and noticed the issue of the time changing and I had to manually set it. I will find out in a couple days how things are going.
From: Andrea @.>
Sent: Wednesday, August 21, 2024 3:47 PM
To: AndreaFavero71/timelapse @.>
Cc: weedy171 @.>; Author @.>
Subject: Re: [AndreaFavero71/timelapse] Recovery from power outage and shoot above a lux threshold (Construction Cam) (Issue #2)
Hi, I think I've discovered the reason we have different results after recovering from a power outage.
Background: The timing methd used to take the pictures is based on the EPOCH time, to prevent shifting over time.
In my case, once the Pi boots it will also find a network and an NTP (Network Time Protocol) and this might not be always the case or your case.
When the Pi boots and does not find a NTP, the internal clock proceeds from where it left: The pictures won't be taken as per intended schedule.
To overcome this issue, I see two options:

Adding a RTC module, with backup battery, to the the Pi (i.e. search for 'DS3231 Real Time Clock Module RTC Clock Memory Module 3.3V/5V for Raspberry Pi').
Changing the shootting approach (settings) to "start_now": "True" with a very long period "period_hhmm": "3000:00" (covers 125 days) and by bounding the shooting to the illuminance level.

Option 1 is rather clean yet, further than the hardware, it requires to properly set the Pi in order to use the hardware clock after a power outage. Time alignment with NTP happen once the Pi will connect to a network. I don't have personal experience with these modules. With this option the pictures time stamp are always correct.
Option 2 should work too, yet with some caviets. In case of power outage and booting without a network available (no NTP), the pictures will be tken as per interval_s and Lux but the timestamp will be wrong.
I have now available a Pi with a screen alowing for tests with and without network connection: Tests on option 2 confirms the above notes.
—
Reply to this email directly, view it on GitHubhttps://github.com/AndreaFavero71/timelapse/issues/2#issuecomment-2302984075, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ACQNON5AU7QC37PQWJPVST3ZST4CPAVCNFSM6AAAAABLLMXR5SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMBSHE4DIMBXGU.
You are receiving this because you authored the thread.Message ID<EMAIL_ADDRESS>

Hi, I've just released an update, that includes:

Recovering from power outage also when the shooting approach is based on "start_now" and related "period_hhmm".
Moved the "lux_check" and "lux_threshold" to the settings.txt.
Some minor improvements.
Wrt your target, I think you'll benefit from setting "start_now" as True, a very long "period_hhmm" (i.e 3000:00 will cover 125 days) and by bounding the shooting to the illuminance level: Set "lux_check" as True and "lux_threshold" to 30.
In case the system is not into a network nor having a RTC module, by setting "start_now" as True and the illuminance control, the shooting  schedule won't mess up in case a power outage will happen.

I was able to grab three days worth prior to the start of the construction and everything looked great. Then it appeared it worked for a day after then stopped. I missed two important destruction so kinda bummed about that. Not sure what happened. I tried to implement some of your suggestions below and now I did not see it try to take pictures after a reboot. Can you look at the settings I posted above and make sure I am not missing anything. Do you think because I am not rendering the file that is an issue?
From: Andrea @.>
Sent: Sunday, August 25, 2024 7:11 AM
To: AndreaFavero71/timelapse @.>
Cc: weedy171 @.>; Author @.>
Subject: Re: [AndreaFavero71/timelapse] Recovery from power outage and shoot above a lux threshold (Construction Cam) (Issue #2)
Hi, I've just released an update, that includes:

Recovering from power outage also when the shooting approach is based on "start_now" and related "period_hhmm".
Moved the "lux_check" and "lux_threshold" to the settings.txt.
Some minor improvements.
Wrt your target, I think you'll benefit from setting "start_now" as True, a very long "period_hhmm" (i.e 3000:00 will cover 125 days) and by bounding the shooting to the illuminance level: Set "lux_check" as True and "lux_threshold" to 30.
In case the system is not into a network nor having a RTC module, by setting "start_now" as True and the illuminance control, the shooting schedule won't mess up in case a power outage will happen.

—
Reply to this email directly, view it on GitHubhttps://github.com/AndreaFavero71/timelapse/issues/2#issuecomment-2308806980, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ACQNONZDAFI2VEXEEF2EXHLZTHCWFAVCNFSM6AAAAABLLMXR5SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMBYHAYDMOJYGA.
You are receiving this because you authored the thread.Message ID<EMAIL_ADDRESS>

Hi, I can understand the frustration from getting a system not working when it counts.
The issue is not related to the render process, but without the timelapse_terminal.log file content is difficult to assess what went wrong. I think the problem relates to the Pi time as not reflecting the reality when it doesn't find a network (NPT).
Anyway, I have one timelapse system that keeps working consistently after restoring power.
It uses a slightly different shooting approach: It uses of "start_now", "period_hhmm" and "lux_check".
By setting "start_now" as True, it won't anymore relate to start_hhmm and stop_hhmm, but it will shooting every ïnterval_s"as long as there is enough light.
This system has the release V0.5.
After updating the python file, you have to run it once first to get the additional parameters added to the settings ("lux_check" and "lux_threshold") and afterward you can edit the settings (change the "lux_check" to True).
To change the shooting process you have to change 3 parameters to below values:

"start_now" : "True"
"period_hhmm" : "3000"
"lux_check": "True"

Here all the settings with some comments:
{
"preview": "False",
"erase_pics": "False", (Must remain False, to prevent pictures from being overwritten after a recovery from power outage)
"erase_movies": "False", (ok)
"local_control": "False", (ok)
"start_now": "True", (Must be changed to True)
"period_hhmm": "3000:00", (Must cover the hours of the complete period, therefore 24h x days. 3000h =125 days)
"start_hhmm": "00:01", (this time is not taken into account when "start_now"is set True)
"end_hhmm": "23:59", (this time is not taken into account when "start_now"is set True)
"interval_s": "600", (I would consider 300, or 180, to increase video smootness, You can increase FPS to shorten the video)
"days": "120", (this time is not taken into account when "start_now"is set True)
"rendering": "False", (ok)
"fix_movie_t": "False", (not relevant when rendering is set False)
"movie_time_s": "20", (not relevant when rendering is set False)
"fps": "24", (not relevant when rendering is set False)
"overlay_fps": "False", (not relevant when rendering is set False)
"overlay_text": "", (not relevant when rendering is set False)
"camera_w": "1920", (ok)
"camera_h": "1080", (ok)
"hdr": "True", (ok)
"autofocus": "True", (ok)
"focus_dist_m": "0.1", (not relevant when autofocus is set True)
"lux_check": "True",
"lux_threshold": "30", (on my system 30 lux is about when pubblic street lights are turned on/off where i live)
"date_folder": "False", (Must be set as False, to allow recovery from power outage)
"folder": "timelapse_pics", (ok. This folder will contain the pictures. This older will be appended to the parent_folder)
"parent_folder": "/home/weedy/shared", (ok)
"pic_name": "picture", (ok)
"pic_format": "jpg", (ok)
"rotate_180": "False", (depends if the camera is suspended upside down)
"display": "True", (depends if you have added the display to the Raspberry Pi)
"modified_disp": "False",
"disp_preview": "False",
"disp_image": "False",
"disp_rotation": "90",
"disp_bright": "100",
"disp_width": "240",
"disp_height": "240",
"disp_offsetL": "0",
"disp_offsetT": "0"
}
I hope this is clear enough.

Do you have any recommendations on how I should take pictures off, I left the last picture on so it would keep with the numbering. It seems to work for another day. Since I don’t use a display do you think I need that to be true. I noticed that was on last night.
From: Andrea @.>
Sent: Tuesday, August 27, 2024 12:46 PM
To: AndreaFavero71/timelapse @.>
Cc: weedy171 @.>; Author @.>
Subject: Re: [AndreaFavero71/timelapse] Recovery from power outage and shoot above a lux threshold (Construction Cam) (Issue #2)
Hi, I can understand the frustration from getting a system not working when it counts.
The issue is not related to the render process, but without the timelapse_terminal.log file content is difficult to assess what went wrong. I think the problem relates to the Pi time as not reflecting the reality when it doesn't find a network (NPT).
Anyway, I have one timelapse system that keeps working consistently after restoring power.
It uses a slightly different shooting approach: It uses of "start_now", "period_hhmm" and "lux_check".
By setting "start_now" as True, it won't anymore relate to start_hhmm and stop_hhmm, but it will shooting every ïnterval_s"as long as there is enough light.
This system has the release V0.5.
After updating the python file, you have to run it once first to get the additional parameters added to the settings ("lux_check" and "lux_threshold") and afterward you can edit the settings (change the "lux_check" to True).
To change the shooting process you have to change 3 parameters to below values:

"start_now" : "True"
"period_hhmm" : "3000"
"lux_check": "True"

Here all the settings with some comments:
{
"preview": "False",
"erase_pics": "False", (Must remain False, to prevent pictures from being overwritten after a recovery from power outage)
"erase_movies": "False", (ok)
"local_control": "False", (ok)
"start_now": "True", (Must be changed to True)
"period_hhmm": "3000:00", (Must cover the hours of the complete period, therefore 24h x days. 3000h =125 days)
"start_hhmm": "00:01", (this time is not taken into account when "start_now"is set True)
"end_hhmm": "23:59", (this time is not taken into account when "start_now"is set True)
"interval_s": "600", (I would consider 300, or 180, to increase video smootness, You can increase FPS to shorten the video)
"days": "120", (this time is not taken into account when "start_now"is set True)
"rendering": "False", (ok)
"fix_movie_t": "False", (not relevant when rendering is set False)
"movie_time_s": "20", (not relevant when rendering is set False)
"fps": "24", (not relevant when rendering is set False)
"overlay_fps": "False", (not relevant when rendering is set False)
"overlay_text": "", (not relevant when rendering is set False)
"camera_w": "1920", (ok)
"camera_h": "1080", (ok)
"hdr": "True", (ok)
"autofocus": "True", (ok)
"focus_dist_m": "0.1", (not relevant when autofocus is set True)
"lux_check": "True",
"lux_threshold": "30", (on my system 30 lux is about when pubblic street lights are turned on/off where i live)
"date_folder": "False", (Must be set as False, to allow recovery from power outage)
"folder": "timelapse_pics", (ok. This folder will contain the pictures. This older will be appended to the parent_folder)
"parent_folder": "/home/weedy/shared", (ok)
"pic_name": "picture", (ok)
"pic_format": "jpg", (ok)
"rotate_180": "False", (depends if the camera is suspended upside down)
"display": "True", (depends if you have added the display to the Raspberry Pi)
"modified_disp": "False",
"disp_preview": "False",
"disp_image": "False",
"disp_rotation": "90",
"disp_bright": "100",
"disp_width": "240",
"disp_height": "240",
"disp_offsetL": "0",
"disp_offsetT": "0"
}
I hope this is clear enough.
—
Reply to this email directly, view it on GitHubhttps://github.com/AndreaFavero71/timelapse/issues/2#issuecomment-2313169840, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ACQNON35KKRVWLJLVPRWCMLZTS3NBAVCNFSM6AAAAABLLMXR5SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMJTGE3DSOBUGA.
You are receiving this because you authored the thread.Message ID<EMAIL_ADDRESS>

Yes, keeps at least the last picture in folder.
Keeps "display" : "False" to prevent issues in case you don't have installed the driver for the display.

Hi, just released one more update: Solved a bug that in V0.5 was preventing shooting, when start_now=true and lux_check=false.

So it was working good for two whole day(plus the night before a couple hours before ) then it stopped taking pictures around 7:30 pm. I went to check it tonight and I have noticed it does not want to connect to the access point. I have to power cycle for it to connect to the access point that I bring over. Then i notice it starts taking pictures. Not sure what to check for on that issue. I am debating creating a nightly reboot.

Hi, any chances it stopped at about 7.30pm because of too low illuminance?
Sometimes the Raspberry Pi boards aren't often searching for the AP, requiring quite some waiting time; In a few cases I went for a power disconnection, as my patience was over.
It isn't a good idea to disconnect the power, without a proper shut down, as there are open files than can get corrupted ... and could cause the Pi to don't boot or not toworking as expected.

Based on the last picture it took, their was plenty of light I am more concerned it didn’t turn back up the following days. I couldn’t really put the battery option because I had it already sealed up and mounted on a high pole.
How long do you think I should wait for the AP? I am thinking maybe the pi froze up and that’s why it stopped because I noticed as soon as I power cycled it started taking pictures.
From: Andrea @.>
Sent: Tuesday, September 3, 2024 4:06 PM
To: AndreaFavero71/timelapse @.>
Cc: weedy171 @.>; Author @.>
Subject: Re: [AndreaFavero71/timelapse] Recovery from power outage and shoot above a lux threshold (Construction Cam) (Issue #2)
Hi, any chances it stopped at about 7.30pm because of too low illuminance?
Sometimes the Raspberry Pi boards aren't often searching for the AP, requiring quite some waiting time; In a few cases I went for a power disconnection, as my patience was over.
It isn't a good idea to disconnect the power, without a proper shut down, as there are open files than can get corrupted ... and could cause the Pi to don't boot or not toworking as expected.
—
Reply to this email directly, view it on GitHubhttps://github.com/AndreaFavero71/timelapse/issues/2#issuecomment-2327436516, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ACQNON7627JKRATVXJVRLCTZUYQCPAVCNFSM6AAAAABLLMXR5SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMRXGQZTMNJRGY.
You are receiving this because you authored the thread.Message ID<EMAIL_ADDRESS>

Hi, if the Pi freezes .... waiting for the connection to the Ap is not a good strategy.
It might be helpfull to add a led, and blinking it via the timelapse code as feedback it is working: This approach is informative, but again not resolving a frozen Pi.
Differently, it looks like the Raspberry Pi can be set to automatically switch off and on in case it freezes.
I didn't know about this feature, and just read about it at https://raspberrypi.stackexchange.com/questions/99584/rpi-freezes-every-now-and-then-how-to-fix-it-with-a-watchdog.
I've just followed Berto's instruction and it works :-)
sudo nano /etc/systemd/system.conf
Uncomment these two rows:
RuntimeWatchdogSec=10
ShutdownWatchdogSec=10min
Ctrl + X, then Y, then Enter to save with the same name by overwriting the file
.....And you're done.
If the Pi gets unresponsive for 10seconds, it will automatically power off and after 10 minutes it powers on.
Setting the WatchDog is really easy, the difficult part for me it has been for force it unresponsive.... Anyhow it works well
By using this technic, the Pi reboots only in case it freezes


LL-1086 Account for a reset event to delete all observed devices

Linked to https://github.com/LedgerHQ/ledgerjs/pull/303 , once merged react-native-hid will broadcast a reset event that we need to handle on the mobile end to clear the device list. This is that handling.

we'll re-discuss about this but i'm not sure about this semantic.
to me it should "remove" all the past emitted "add" if it's what a "reset" means.

After the discussion, there'll be no reset event since we can just delete the device, single device otg support for now since I can't test multiples.

I'm concerned here @gre if we remove the unsubscribe from mobile too, will it break the ble part?


RHCLOUD-35760: updates dockerfile and deployment to allow for easier config

PR Template:
Describe your changes

updates the dockerfile to:

create a path for config vs using /usr/local/bin so that a config can be managed externally and not overwrite the directory
updates the entrypoint command for new config path

adds a default config to ephemeral deployment
updates the pod spec to mount the configmap as a file

Purpose:
The config is currently hard backed into the dockerfile which makes it difficult to provide different configs for every env, especially with some using auth, some not, and having different auth endpoints.
For ephemeral, the configmap is created with the deployment, allowing for local edits/testing
For stage/prod, the configmap will be deployed via App interface and is expected to exist to run. This will allow for configuring values that are env specific.
The config is still copied over to the image for the sake of local testing, but when using kubernetes, the configmap will be required after this change.
Ticket reference (if applicable)
Fixes #
Checklist

[ ] Are the agreed upon acceptance criteria fulfilled?

[ ] Was the 4-eye-principle applied? (async PR review, pairing, ensembling)

[ ] Do your changes have passing automated tests and sufficient observability?

[ ] Are the work steps you introduced repeatable by others, either through automation or documentation?

[ ] If automation is possible but not done due to other constraints, a ticket to the tech debt sprint is added
[ ] An SOP (Standard Operating Procedure) was created

[ ] The Changes were automatically built, tested, and  - if needed, behind a feature flag - deployed to our production environment. (Please check this when the new deployment is done and you could verify it.)

[ ] Are the agreed upon coding/architectural practices applied?

[ ] Are security needs fullfilled? (e.g. no internal URL)

[ ] Is the corresponding Ticket in the right state? (should be on "review" now, put to done when this change made it to production)

[ ] For changes to the public API / code dependencies: Was the whole team (or a sufficient amount of ppl) able to review?

/retest


ZFS support

Is your feature request related to a problem? Please describe.
It's unclear to me whether k3os already supports ZFS after or if this feature is on the roadmap, after skimming the docs. It seems to be supported on rancheros
Describe the solution you'd like
Ideally support and documentation to use persistent storage via ZFS with k3os.
Describe alternatives you've considered
Well, I could live without it or consider using RancherOS for that usecase.
Thanks for your time and work!

I spent a little time back in (wow, 2017!) getting ZFS kernel modules loaded into a simple Linuxkit/moby OS build: https://metahertz.co.uk/2017/09/15/using-zfs-with-linuxkit-and-moby/
Out of interest, what are your usecases?

The K3OS image itself installed onto a ZFS root?
The ability to have a separate persistence disk(s) loaded via ZFS
(for example, for data resilience in a single node cluster environment?)
Something i've not thought of?

K3OS builds on linuxkit so i'm happy to take a stab at it, tying to understand the usecase!

Hi @matjohn2, thanks for your reply!
My use case would primarily be your second one, persistent storage via ZFS for my volumes. Installing k3os on the same zfs would be awesome but not necessarily needed.

@phaer @matjohn2 To avoid licensing issues, we currently do not build/ship the ZFS kernel modules (our kernel is basically upstream Ubuntu). (See https://github.com/rancher/k3os-kernel/blob/4.15.0-72.81-rancher1/scripts/build#L16).That said, I would also like to see ZFS support with a working example.

@phaer Could be as simple as running the k3os-kernel drone pipeline with do_zfs=true in https://github.com/rancher/k3os-kernel/blob/master/scripts/build#L16
Would then need alpine compatible/compiled userspace binaries added to the k3os userspace image if this is the case. Will have a play with this tomorrow (as i need to understand the K3OS build process anyway, this is a good excuse!)

Just wanted to post an update here that I've been able to build zfs (modules) from source during the k3os build process and adding the alpine zfs package to get the zfs userspace binaries, however I need to modprobe zfs manually on boot.
I'll try and refine the steps and try and break it out into it's own build step so the userspace binaries from the source build can be used to avoid version incompatibility between modules and userspace binaries.

Here's my branch (https://github.com/zimme/k3os/tree/zfs) with initial zfs support.
After you've built the iso and started the system you will need to sudo modprobe zfs to load zfs modules.
There's a lot of refinement needed before this could potentially be put up as a PR and I'm thinking that the changes being made to Makefile.in using sed  should really be a PR to openzfs.
Try it out and see if it works for you and feel free to improve anything.

Great work @zimme I'm going to try this out quite soon and combine this with:
@asteven 's  https://github.com/asteven/zfs-provisioner
The idea would be to get https://kubernetes.io/docs/concepts/storage/volume-snapshots/ implemented using zfs snapshots behind the scenes locally. And in a later step use zfs replication to send the backups to other places.
The whole idea is that with longhorn we don't really need the HA part on the edge, and zfs underneath would already greatly support our usecases, we just need to cover the gap of creating crd's to trigger underlying zfs operations.
Since linux's mdadm is already included, how high are the chances that we can get zfs in the main k3os as well?
Would it be considered as a possible replacement of the default local path provisioner if we can work out a poc to benefit all k8s volumes using zfs providing lightweight snapshotting?

Just saw this issue in k3s, https://github.com/rancher/k3s/issues/66, in light of k3s not including the zfs snapshotteer I'm not sure how useful it would be to include zfs in k3os?


Files related to hg19 mirbase changed names - installation or upgrade crashes

Hi,
A recent update of cloudbiolinux updated the URLs of mirbase srnaseq files for hg19 and hg38. One of these files, here:
https://github.com/chapmanb/cloudbiolinux/blob/cbba4e96c2f45ccc5d57c96d310b7dbeb18db580/ggd-recipes/hg19/mirbase.yaml#L45
has changed name from mirna_mature.txt.gz to mirna_context.txt.gz. Both the old and the new files still exist under the new URL at https://www.mirbase.org/ftp/21/database_files/. I'm not sure of the significance and impact of this change. However, when updating a current bcbio-nextgen install, the GGD recipe still references the old name, here:
https://github.com/chapmanb/cloudbiolinux/blob/cbba4e96c2f45ccc5d57c96d310b7dbeb18db580/ggd-recipes/hg19/mirbase.yaml#L59
which causes the update process to crash:
2021-11-25 10:45:11 URL:https://www.mirbase.org/ftp/21/database_files/mirna_context.txt.gz [212869/212869] -> "mirna_context.txt.gz" [1]
Traceback (most recent call last):
  File "/path/to/bcbio/bin/bcbio_nextgen.py", line 228, in <module>
    install.upgrade_bcbio(kwargs["args"])
  File "/path/to/bcbio/anaconda/lib/python3.6/site-packages/bcbio/install.py", line 107, in upgrade_bcbio
    upgrade_bcbio_data(args, REMOTES)
  File "/path/to/bcbio/anaconda/lib/python3.6/site-packages/bcbio/install.py", line 359, in upgrade_bcbio_data
    args.cores, ["ggd", "s3", "raw"])
  File "/path/to/bcbio/tmpbcbio-install/cloudbiolinux/cloudbio/biodata/genomes.py", line 354, in install_data_local
    _prep_genomes(env, genomes, genome_indexes, ready_approaches, data_filedir)
  File "/path/to/bcbio/tmpbcbio-install/cloudbiolinux/cloudbio/biodata/genomes.py", line 480, in _prep_genomes
    retrieve_fn(env, manager, gid, idx)
  File "/path/to/bcbio/tmpbcbio-install/cloudbiolinux/cloudbio/biodata/genomes.py", line 875, in _install_with_ggd
    ggd.install_recipe(os.getcwd(), env.system_install, recipe_file, gid)
  File "/path/to/bcbio/tmpbcbio-install/cloudbiolinux/cloudbio/biodata/ggd.py", line 31, in install_recipe
    _move_files(tmpdir, base_dir, recipe["recipe"]["full"]["recipe_outfiles"])
  File "/path/to/bcbio/tmpbcbio-install/cloudbiolinux/cloudbio/biodata/ggd.py", line 74, in _move_files
    (out_file, tmp_dir))
AssertionError: Did not find expected output file srnaseq/mirna_mature.txt.gz in /path/to/bcbio/genomes/Hsapiens/hg19/txtmp

Would you be able to fix this small issue? I didn't propose a PR as I don't do mirna analysis, so someone experienced with these files and analyses should review and update this recipe.
Thank you in advance, and thank you for all the work on bcbio-nextgen.
Best,
-- Paulo

Thanks for catching this, @pfpjs
I think this is because we require mirna_mature.txt.gz in the output of the recipe:
https://github.com/chapmanb/cloudbiolinux/blob/master/ggd-recipes/hg19/mirbase.yaml#L59
I've updated it to mature https://github.com/chapmanb/cloudbiolinux/blob/master/ggd-recipes/hg19/mirbase.yaml#L45
at least it should run smoothly now.

Thank you @naumenko-sa, I confirm the issue is solved and I was able to upgrade successfully.
Best,
-- Paulo

Hi @naumenko-sa,
sorry, I can't comment on these files since I don't have experience with mirbase neither.