v0.8 - timelapse creation failed

The first night / morning after updating to 0.8, a timelapse was produced no problem.
I noticed that it was heavily compressed and quite blocky in places, so I looked at how to improve this - my first thought was the bitrate.
In the timelapse script the bitrate is set to 2MB/s so I changed this line and only this line from :-b:v 2000k to :-b:v 10000k \ #bitrate default 2000k
This morning, there's a keogram etc but no timelapse.
In the log, there's nothing about the timelapse, it goes from taking the final night image to the first day image. The only message is 'Processing end of night data'. I could be wrong but on v0.7 wasn't the entire process of timelapse creation recorded in the log? I don't have a copy of the previous log to check but I'm sure I've seen it?
I'm going to ssh into the pi and see if I can run the script manually before I set the bitrate back to 2000k, but the main question I suppose, is what is the maximum bitrate I can set? Is there another was to improve the timelapse quality? I'm happy with a lossless video, space / bandwidth etc. isn't an issue.
From past experience, 20MB/s is a good bitrate for high def video files, but this timelapse is near 3K resolution (whatever the image resolution is).

Hmmm...running the script manually has returns this error:

I'm stuck now so will return the timelapse script to default (re download the file from github) and see what happens.

I've replace the timelapse.sh containing my small modification, with the default as per the original file (from 0.8).
Running manually I get the same error message as above.
I've also compared the scripts from v0.7 and v0.8 and the first few lines are the same, so I'm confused as to why it fails at the first line?
I suppose I could put the 0.7 timelapse.sh file in and see if that runs manually?

D'oh - I've just learnt that python3 is for .py files, not .sh.....
It needs to be made an executable sudo chmod +x timelapse.sh then run manually using sh timelapse.sh
Ignore me, I'm off to hide in shame!
But before I do that...
This is now the error it returns...

Looks like you're close. Give this a go:
terminal
cd allsky
./scripts/timelapse.sh 20210807
i've had issues runing it manually from the scripts folder in the past, found it needs to be run from the root of the Allsky folder.  Let me know how that goes for you.

Ok I did what you suggested and it worked - the timelapse file was created.

I changed the bitrate in the script from 2000k to 10000k and ran it again...
...and it worked fine, resulting vide0 a lot better (size approximately 45MB vs 8MB). I'll try maybe 20MB/s too, giving approx 100MB filesize.
I will also check tomorrow to see if it automatically creates one for tonight's images and report back, but manually, yes it seems to do as it should, even copying the video in to the 'website location', in my case a local NAS.
Thanks for the tip.

Jon,
Running it as "sh timelapse.sh" runs it in the standard shell ("sh"), which doesn't know about the "source" command.  Had you done "bash timelapse.sh" it would have worked, but no need to prepend the "bash" since the first line of the file is "#!/bin/bash" which tells the operating system to run the file using /bin/bash.  "sh" also doesn't know about the "-en" option to "echo", which is why you saw "-en" in the log file as well.
At the end of every night, the "endOfNight.sh" script is run.  It will write an entry to the log for every sub-command it executes, depending on what options you have in config.sh.  You should see:
endOfNight.sh: Generating Keogram
endOfNight.sh: Processed 1000 keogram files       (or whatever number were on your system)
endOfNight.sh: Generating Startrails
eendOfNight.sh: Processed 1000 startrails files.  Summary:
Minimum .....
endOfNight.sh: Generating Timelapse
timelapse.sh: Creating symlinks to generate timelapse
timelapse.sh: Created 1000 links total                 (or whatever number were on your system)
timelapse.sh: Deleting sequence
timelapse.sh: Timelapse was created

What's tricky, is that there will likely be lots of other log entries between the entries above, so you may need to search for the script name in the log file (endOfNight.sh, timelapse.sh, etc.).  These log entries could span dozens of minutes, depending on how many images you take.
Please look again in your log file for the timelapse.sh messages and let me know if you find them.
Here's what I plan to do to allow the user to specify the bitrate.
In timelapse.sh change the line     "-b:v 2000k "    to  "-b:v ${TIMELAPSE_BITRATE:-2000k} "
In config.sh, add:
Set the bitrate of the timelapse video.  Higher numbers will produce higher quality but larger files.
TIMELAPSE_BITRATE="2000k"
Then change the 2000 in config.sh to whatever you want.
Let me know how that goes.
Eric
Eric

Would this error be USB related ?
I upgraded the night of 8/7/3021, entered my ZWO ASI178 camera and Location, double checked the location then captured no images.
Log.txt shows
Starting Night Time Capture
Saving auto exposed night image
ERROR Failed getting image …….
Status 11 ASI Error Timeout
scripts/copy_notification_image.sh: Permission Denied

It's a permissions error with one of the newer updated files that somehow didn't get the correct permissions on install.
Open the terminal
sudo chmod u+x /home/pi/allsky/scripts/copy_notification_image.sh
Not sure if the 'u' is needed, but it works for me so I've just run with it. 😅
That should get the permissions error fixed and let the rest of the script continue as normal.

The permissions error was fixed yesterday so anyone getting the new version won't see the problem.
For the "timeout" message, try increasing your USB Bandwidth from 40 to, for example, 90.   Let me know if that fixes the problem.
Eric

Eirc,
I've looked through the log again and this is what I find (I've removed lots of lines in between, take a note of the timestamps for the end of night.sh related lines):

Aug  8 05:46:08 allsky allsky.sh[20548]: Processing end of night data
Aug  8 05:46:08 allsky allsky.sh[20548]: endOfNight.sh: Removing bad images

Aug  8 05:59:10 allsky allsky.sh[20548]: removeBadImages.sh: No bad files found.

This took 13 minutes, nothing to worry about there I guess?
Next...

Aug  8 05:59:10 allsky allsky.sh[20548]: endOfNight.sh: Generating Keogram
Aug  8 06:04:18 allsky allsky.sh[20548]: endOfNight.sh: Processed 1063 /home/pi/allsky/tmp/keogramTMP.txt keogram files

This took 5 minutes. Again, nothing to worry about.
Next...

Aug  8 06:04:18 allsky allsky.sh[20548]: endOfNight.sh: Generating Startrails
Aug  8 06:08:28 allsky allsky.sh[20548]: endOfNight.sh: Processed 1065 /home/pi/allsky/tmp/startrailsTMP.txt startrails files.  Summary:
Aug  8 06:08:28 allsky allsky.sh[20548]: Minimum: 0.112967 maximum: 0.386757 mean: 0.197037 median: 0.176065

This took 2.5 ish minutes. good, no problem.
Then...

Aug  8 06:08:28 allsky allsky.sh[20548]: timelapse.sh: #033[0;32mCreating symlinks to generate timelapse#033[0m
Aug  8 06:08:31 allsky allsky.sh[20548]: STARTING EXPOSURE at: 2021-08-08 06:08:31
Aug  8 06:08:32 allsky allsky.sh[20548]:   > Saving image 'image.jpg' that started at 2021-08-08 06:08:31  (11 Âµs)
Aug  8 06:08:32 allsky allsky.sh[20548]:   > Sleeping from histogram exposure: 5000 ms
Aug  8 06:08:32 allsky allsky.sh[20548]: timelapse.sh: Created 1063 links total
Aug  8 06:08:32 allsky allsky.sh[20548]: [NULL @ 0x18238b0] Unable to find a suitable output format for ' #bitrate'
Aug  8 06:08:32 allsky allsky.sh[20548]:  #bitrate: Invalid argument
Aug  8 06:08:32 allsky allsky.sh[20548]: ./timelapse.sh: line 76: -pix_fmt: command not found
Aug  8 06:08:32 allsky allsky.sh[20548]: ./timelapse.sh: line 77: -movflags: command not found
Aug  8 06:08:32 allsky allsky.sh[20548]: #033[0;31m*** timelapse.sh: ERROR: ffmpeg failed with RET=127
Aug  8 06:08:32 allsky allsky.sh[20548]: Links in '/home/pi/allsky/images/20210807/sequence' left for debugging.
Aug  8 06:08:32 allsky allsky.sh[20548]: Remove them when the problem is fixed.#033[0m

This clearly shows the timelapse creation failing, after all I did was change the bitrate and comment that the original was 2000k.
Was the # comment the issue? i.e. my fault?
:-b:v 10000k \ #bitrate default 2000k
If so, great, this can be closed. If not?

Jon,
Yes, the "#bitrate comment is the problem.  Put it above or below the ffmpeg command.  The "" tells the shell that the next line is part of the current line.  The errors below #bitrate are a result of the #bitrate error.
The other output is exactly as expected.
Removing bad frames is pretty slow, which is why there's a comment in config.sh saying it's slow.  Other than taking time and CPU, it doesn't hurt though.
Eric

Jon,
Did you change your USB bandwidth, and if so, did it fix the problem?  What did you set it to?

I 'may' have set it to auto after the timeout issue was noticed, I don't remember but I haven't had the issuse since I reported. I don't really want to set it back to the default 40 now it's working just in case, but I will if it helps?
The only issue I've had since regarding the actual camera, is 2 frames last night, one was pure purple and the next was pure green:

I have seen this before, so not sure what would cause it. Bandwith maybe, could be a bug in the ASI driver.
I'll close this issue about the timelapse as that is now sorted.

Jon,
If your USB Bandwidth is working, leave it.   Mine's at "auto" and I don't remember seeing a problem in a while, so I'm not going to change.  I will put in an update to set the default to "auto" and "80".
I get a purple picture once or twice a day it seems.  I'll have to check if I've seen one at night.  Haven't seen a green one (yet).  I'd really like to figure out a way for "removeBadImages.sh" to remove them.  I assume it's a bug in the ASI driver.  I may try setting auto-white balance and see if that works, but I think the last time I tried auto balance I didn't like the colors.


activeThumb Undefined

Duplicates

[X] I have searched the existing issues

Latest version

[X] I have tested the latest version

Current behavior 😯

It should provide me an index of Label, but It's coming out to be undefined.

Expected behavior 🤔
I am using a material UI Range slider, in that when I'm using the onChange function, activeThumb is coming out to be undefined, which according to documentation should be defined and equal to the Index of the label which is changed.
Steps to reproduce 🕹
Steps: Using Range Slider of material UI
Context 🔦
No response
Your environment 🌎
No response

Please provide a CodeSandbox reproducing the issue. The issue template is a good starting point.

@AAYUSHAIRAN I had the same issue.
Don't forget to import the Slider from "@mui/material/Slider"
I imported the Slider from "@material-ui/material/Slider" which caused this issue.


Adds option and try sequencing

Adds implicit class for List[Option[T]] to allow .sequence. Includes tests
Adds missing test for the OptionList class.
Adds fairly specific sequencing methods for Trys

I'm a fan. I made one comment, but not adding or removing the implicit classes is fine. :shipit:

@2rs2ts Any comments?

@kionka So you're saying the implicit classes for the Options should just be functions like the try sequences? Because I think I like that better. I was mostly following the pattern in the file.

@theanti9 Ah. Interesting. I was just commenting that they were different, which was disappointing. Up to you if you want to do something about it, like just using methods.

Just using methods might require more type annotations though.

I like the implicit classes. We should however not use implicit methods.

@kionka @theanti9 In my experience defining it as a(n explicit) method of some object doesn't require extra type annotations for something this simple. It doesn't look that bad either, as you can wrap your code in a sequence { } block. The problem is that you can't use sequence without overloading if you intend on implementing for many different higher kinds.
Personally I like implicit classes when it's not a conceptual stretch to add them. I approve of this as it is.

Nevermind. Type annotations are not needed in these cases.

Oddly, this is actually making the http tests not compile, even though travis said it passed.

[error] cascade/http/src/main/scala/com/paypal/cascade/http/resource/resource.scala:227: not found: type A
[error]   implicit class RichTryHalt[A](t: Try[A]) extends RichEitherThrowableHalt(t.toEither)

Which was likely caused by
[error] cascade/http/src/main/scala/com/paypal/cascade/http/resource/resource.scala:227: illegal cyclic reference involving class RichTryHalt
[error]   implicit class RichTryHalt[A](t: Try[A]) extends RichEitherThrowableHalt(t.toEither)

@theanti9 clean. I just had that happen to me.

Looks good to me. :shipit: when you're comfortable with it.


Taxonomy: Tags should add on blur

Right now in the tag input if you type in a new value and hit tab instead of enter or the spacebar it doesn't save the new tag. It should. The initial PRs for taxonomy were doing a lot at once, this just slipped through. Fixing ahead of release.

Work for taxonomy history ticket 9073 will completely redo the taxonomy provider. Hold off on this ticket until that work is complete. The new work being done may also solve this issue.


Bluemap Support

Bluemap support: https://bluemap.bluecolored.de/

I'd like to support bluemap, but I'm well aware that'd be a significant undertaking since it isn't leaflet based like all other supported map types.
Maybe one for the future


[Feature] Connected Guest Devices Count

The problem
Is it possible to get a count of devices connected to the guest network only?
This can be used to monitor if visitors are at home or not.
Specific device(s)?
No
Firmware type
No

This is not implemented directly yet, but can possibly be. How do you see this feature? As a sensor connected_guest_5_1 numeric sensor? Or just total number of clients on all the guest networks?
I would need a proper context on the way you would like to use it, so that I can suggest the best solution. Maybe, it can already be done with the existing integration features, maybe something new should be implemented

Hello
I‘m very interested in this feature too.
For me it would be perfect if I get the total number on clients on all guest networks.

I am now trying the same with automation but it is impossible since you don't provide the correct event data when guest devices disconnect (guest flag is omitted). Can you at least make sure that guest: true is provided when a guest device disconnects? I presume you keep enough state to do so...

of course it would be best if your solution provided the counter

I am now trying the same with automation but it is impossible since you don't provide the correct event data when guest devices disconnect (guest flag is omitted). Can you at least make sure that guest: true is provided when a guest device disconnects? I presume you keep enough state to do so...

I created a separate issue for that here: #843


:preferredCountries don't work

 <vue-tel-input
v-model="phone"
 :required="true"
:preferredCountries="['us','uk','fr','de','fi','au','mx','br','sw','es','is']">
</vue-tel-input>

result:

That's weird, I copy your code to test in the demo file and it works.
Are you missing something? also, the code for UK is 'gb', you can take a look at the assets/all-countries.js to see all the code we are using.

@alexeydg Try with :preferred-countries instead of :preferredCountries


release: v0.4.1

Description

feat: support setting context on ListObjects
feat: support setting context and contextual tuples on ListRelations
feat: add retries to OAuth2 Client Credentials request
feat: support modular models metadata
fix: avoid clone of object mapper

References
Generated from https://github.com/openfga/sdk-generator/pull/343
Review Checklist

[x] I have clicked on "allow edits by maintainers".
[ ] I have added documentation for new/changed functionality in this PR or in a PR to openfga.dev [Provide a link to any relevant PRs in the references section above]
[x] The correct base branch is being used, if not main
[ ] I have added tests to validate that the change in functionality is working as expected

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 28.51%. Comparing base (988c3b4) to head (1afb414).

Additional details and impacted files
@@            Coverage Diff            @@
##               main      #73   +/-   ##
=========================================
  Coverage     28.51%   28.51%
  Complexity      721      721
=========================================
  Files           145      145
  Lines          5277     5277
  Branches        567      567
=========================================
  Hits           1505     1505
  Misses         3714     3714
  Partials         58       58

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


elfshaker clone fails when server does not return Content-Length

When cloning from a server, the server can either return Content-Length or Transfer-Encoding.
However elfshaker doesn't handle the latter case correctly because it tries to draw a progress bar which of course fails when there's no Content-Length:
thread 'main' panicked at 'assertion failed: (0..=100).contains(&percent_complete)', src/bin/elfshaker/utils.rs:130:5
stack backtrace:
   0: rust_begin_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:575:5
   1: core::panicking::panic_fmt
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/panicking.rs:65:14
   2: core::panicking::panic
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/panicking.rs:115:5
   3: elfshaker::utils::create_progress_bar
   4: elfshaker::utils::create_percentage_print_reporter::{{closure}}
   5: std::io::Write::write_all
   6: std::io::copy::stack_buffer_copy
   7: elfshaker::repo::remote::update_remote_pack
   8: elfshaker::repo::repository::Repository::update_remote_pack
   9: elfshaker::repo::repository::Repository::extract_entries
  10: elfshaker::repo::repository::Repository::extract_snapshot
  11: elfshaker::extract::run
  12: elfshaker::main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

I've hacked up the source code to just not draw the progress bar when Content-Length is 0 which gets it to work correctly. But it's just a hack.

I suggest storing the filesize in the esi file as well. when the server doesn't send back Content-Length we can use that size instead.
when the server does send back a Content-Length the two should match, and the final downloaded binary should match the original size.
If maintainers agree I can implement this.

Uh, storing the size inside the file could modify its size? It seems to me you would really want that information to exist out-of-band.
But also the esi is tiny, so having a progress bar for downloading it doesn't seem terribly important to me - we should gracefully fall back to showing a spinner if we don't know the size.

Hmm? What do you mean? The esi stores the link to the index and the package itself, including their hashes.
The issue isn't with showing a progress for the esi file, but that for downloading the packages.  So I don't get how adding a field to the esi to show the sizes (along side the hashes) would change anything on the packages themselves, if it did the hashes would also be broken :)
for the esi file download itself I don't see any progress bar code.

ProgressWriter::new does not require the size to be specified. I suggest we use that and avoid changing the .esi format.

Ah, apologies @Mistuke, from our previous offline conversation I somehow mistakenly came to think we were crashing on the esi file itself rather than the packs, but I see from the stack trace in this issue that this is happening to the packs themselves.
I think the priority should be on not crashing, followed by niceties such as showing an accurate progress bar. I would prefer not to have different esi formats floating around unless it's unavoidable.
So my preferences are:

First don't crash.
If we can't show the expected amount, at least show the amount downloaded.
Via some other means, determine the amount. But we should discuss and think hard before having multiple esi formats floating around.


Reworked README for improved clarity

fixes: #1001

Thanks for the pull request, @that-ar-guy!
What's next?
Please work through the following steps to get your changes ready for engineering review:
:radio_button: Get product approval
If you haven't already, check this list to see if your contribution needs to go through the product review process.

If it does, you'll need to submit a product proposal for your contribution, and have it reviewed by the Product Working Group.

This process (including the steps you'll need to take) is documented here.

If it doesn't, simply proceed with the next step.

:radio_button: Provide context
To help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:

Dependencies

This PR must be merged before / after / at the same time as ...

Blockers

This PR is waiting for OEP-1234 to be accepted.

Timeline information

This PR must be merged by XX date because ...

Partner information

This is for a course on edx.org.

Supporting documentation
Relevant Open edX discussion forum threads

:radio_button: Submit a signed contributor agreement (CLA)
:warning: We ask all contributors to the Open edX project to submit a signed contributor agreement or indicate their institutional affiliation.
Please see the CONTRIBUTING file for more information.
If you've signed an agreement in the past, you may need to re-sign.
See The New Home of the Open edX Codebase for details.
Once you've signed the CLA, please allow 1 business day for it to be processed.
After this time, you can re-run the CLA check by adding a comment below that you have signed it.
If the CLA check continues to fail, you can tag the @openedx/cla-problems team in a comment for further assistance.

:radio_button: Get a green build
If one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green.
:radio_button: Let us know that your PR is ready for review:
Who will review my changes?
This repository is currently maintained by @bmtcril.  Tag them in a comment and let them know that your changes are ready for review.
Where can I find more information?
If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources:

Overview of Review Process for Community Contributions
Pull Request Status Guide
Making changes to your pull request

When can I expect my changes to be merged?
Our goal is to get community contributions seen and reviewed as efficiently as possible.
However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:

The size and impact of the changes that it introduces
The need for product review
Maintenance status of the parent repository

:bulb: As a result it may take up to several weeks or months to complete a review and merge your PR.

And we will need that signed CLA as noted in the webhook comment in order to be able to accept the PR. Let me know if you need any help with that process!

And we will need that signed CLA as noted in the webhook comment in order to be able to accept the PR. Let me know if you need any help with that process!

Done

Thank you to all the maintainers for reviewing and merging my changes! As someone new to open-source, I’m eager to learn and improve. If you have any feedback or advice on my contribution or how I can better approach such projects, I’d truly appreciate it.
Looking forward to contributing more!


add more NF stuff

depends on #1536 and #1539 for Jenkins to perfectly do his dance
Everything updated to match the official list: https://dl.dropboxusercontent.com/u/44754370/ckan.txt

#rebuild


fixed rates feedforward

We want to have a feedforward from our desired rates to the desired torque. The transfer function from angular rates to angular acceleration (~torque) is given by the derivative operator. Therefore, we have to feedforward the derivative of the desired rates into the actuator controls.

Test results?

My apologies - here is the link. We should have put that one in straight away.

http://dash.oznet.ch/view/eGNadtBR5vVCFRPmYsYfGc

The first set of motions is with FF. It has significantly less overshoot. Still room for improved tuning.

What's going on with pitch angle/rate? Were you only testing roll?
We really need a log event for parameter changes

The pitch is bad because the copter is pretty unbalanced: https://pixhawk.org/platforms/multicopters/ir-discovery

+1 for a log event for parameter changes. What would be the best approach
to take?
On Sat, May 23, 2015 at 2:46 PM Lorenz Meier<EMAIL_ADDRESS>wrote:

The pitch is bad because the copter is pretty unbalanced:
https://pixhawk.org/platforms/multicopters/ir-discovery
—
Reply to this email directly or view it on GitHub
https://github.com/PX4/Firmware/pull/2217#issuecomment-104889400.

There is already a parameter update uORB topic. All it takes would be to graph a label similar to the flight mode background drawing once it shows up in the log. We might need to add it there.


Port tests to rspec

This branch ports the few existing unit tests to rspec, and adds several tests for the Client class, leveraging the mock support provided by Savon (the SOAP Client gem).
The tests for the Client class could be more complete, but the tests that are there serve as a starting point for building a more comprehensive test suite.

Note that rspec tests can be run using the following command:
bundle exec rake spec

And code coverage reports will be written to the coverage directory.

looks good


Optimize dir globbing

Globbing through test directories may take a lot of time.
I would suggest to skip them and scan only .rb files for now.

@exAspArk thx for this. I am super busy at work today so probably won't get to this until tonight or this weekend.

I am closing this because this is no longer applicable now that alt has been rewritten.


Librarian-ansible support for git references e.g. tag/branch/SHA or SHA Prefix

Hi there I have a feature request is it possible to give support for any
thing that Git will recognize as a ref. This includes any branch name, tag name, SHA, or SHA unique prefix.
This will allow to do specific version locking when a playbook does not have the version entry in their meta/main.yml file.
Cheers
@actionjack

Hey so this is actually already supported, but it could maybe use a little more documentation.
You can do the following:
role "maxb.smokeping", git:"https://github.com/max-b/ansible-smokeping.git", ref:"maxb"

where the ref could be a branch name or tag name or any valid git reference.

@max-b good suggestion, I'll update the documentation.
Did this do the trick for you @actionjack?

Thanks a lot!


#2503 push SNAPSHOT Maven artifacts to github Packages

Fixes #2503
Description
Publish Maven artifacts in GitHub Packages
Type
Feature
Checklist

Tests done https://github.com/icrc-fdeniger/android-fhir
Maven packages are deployed here: https://github.com/icrc-fdeniger?tab=packages&repo_name=android-fhir ( will be done on google/android-fhir)
maven-repository.zip artifact is deployed as usual ( not sure about the usage of this artifact but not changes).
See https://github.com/icrc-fdeniger/android-fhir/actions/runs/8798380904/job/24145256498 for a full deployment

Versions

When the maven artifacts are deployed in GitHub Packages the SNAPSHOT prefix is used to follow Maven coventions.
When the maven artifacts are deployed in official maven repositories, the SNAPSHOT prefix is not used ( as it's done actually as far as I can understand)
In the PR, the artifact versions has been updated to next expected release version:

for instance, commonmodule will be automatically deployed with the version 0.1.0-alpha06-SNAPSHOT in GitHub Packages and with 0.1.0-alpha06 in the next release.

Suggestion/question
why not removeing beta/alpha as versions like  0.X.Y are considered as development version ?

Current Version
 Used in this PR
 Proposition

common:0.1.0-alpha05
 common:0.1.0-alpha06
 common:0.1.1

engine:1.0.0
 engine:1.0.1
 engine:1.0.1

data-capture:1.1.0
 data-capture:1.1.1
 data-capture:1.1.1

workflow:0.1.0-alpha04
 workflow:0.1.0-alpha05
 workflow:0.1.1

ontrib-barcode:0.1.0-beta3
 ontrib-barcode:0.1.0-beta4
 ontrib-barcode:0.1.1

contrib-locationwidget:0.1.0-alpha01
 contrib-locationwidget:0.1.0-alpha02
 contrib-locationwidget:0.1.1

knowledge:0.1.0-alpha03
 knowledge:0.1.0-alpha04
 knowledge:0.1.1

if a dev Team wants to use an alpha/beta version it can use a dependency to a SNAPSHOT version ( the "unstable version created from the last build").
it's also possible to specify the artifact of a build with by example:  engine-1.0.1-20240423.083712-3

Hello @santosh-pingle, @ktarasenko
please tell me if this PR need to be modified. I understand that the version updates are the "tricky" part and will be happy to modify this. My point was to update them to the next version as GitHub Actions should produce SNAPSHOT of next versions.
Thanks

@icrc-fdeniger FYI the artifact version number changes introduced here will cause build failures once you rebase this after #2533 is merged. The required fix (after you have rebased it, not right now) is as simple as changing those version numbers also in the (new, TBD) docs/use/api.md file; hopefully that's trivial. Please do shout if unclear!
@MJ1998 @santosh-pingle @ktarasenko merge this only after #2533.

@vorburger @santosh-pingle @ktarasenko  should be ok now

@vorburger @santosh-pingle @ktarasenko should be ok now

LGTM! I say let's merge this... @santosh-pingle @ktarasenko @MJ1998 no objects, right?

Thanks for this PR @icrc-fdeniger. LGTM.
About the version proposal we would wanna keep the alpha/beta naming conventions. Reason - Using alpha and beta suffixes aligns with established conventions in semantic versioning and open-source software development. So we would like to stick with it.

@santosh-pingle  any news on that PR ?

OK, now onto github packages - with this change, do we always overwrite the SNAPSHOT? so if I have a PR in development, and I push a new commit it will overwrite the github packages?

Yes it will overwrite.

Another thing we might need to be careful about is just any incurred cost - https://github.com/features/packages#pricing. With our repo under google organisation I'm hoping that we'll be covered by the google organisation quota/allowance. But I will investigate.

I doubt if there will be a problem but we will informed by Google in case the quota reaches limit..

Thansk @icrc-fdeniger for this contribution! 🙏
For some background on the maven-repository.zip artifact, see this PR from @yigit: #249. We also have some documentation and I just realised it's a bit messed up after our recent documentation migration to this page: google.github.io/android-fhir/contrib but for your information the original text is this:

Use unreleased GitHub build (for development/testing only)
Each GitHub build of the project also contains a maven repository. You can access unreleased features by downloading it and including it in your project. To acquire the maven repository, go the the actions page, and click on the build you want. In the build artifacts dropdown, you will find a maven-repository.zip file. After downloading and extracting it to a local folder, you can add it to your gradle setup via:
repositories {
  maven {
    url "file:///<path to the unzipped folder>"
  }
}

These artifacts are versioned as a combination of the current version + buildid (e.g. 0.1.0-alpha01-build_123). You can find the version in the zip file by checking the contents of it and update your build file to match that version.

OK, now onto github packages - with this change, do we always overwrite the SNAPSHOT? so if I have a PR in development, and I push a new commit it will overwrite the github packages?
Another thing we might need to be careful about is just any incurred cost - github.com/features/packages#pricing. With our repo under google organisation I'm hoping that we'll be covered by the google organisation quota/allowance. But I will investigate.

the old version will be kept as you can see here:
https://github.com/icrc-fdeniger/android-fhir/packages/2129825
It's a kind of feature as we can use a specific SNAHPSHOT Version.
So yes it could be an issue with the storage.  It seems there are no perfect solution for this point:
https://github.com/orgs/community/discussions/48971

Thansk @icrc-fdeniger for this contribution! 🙏
For some background on the maven-repository.zip artifact, see this PR from @yigit: #249. We also have some documentation and I just realised it's a bit messed up after our recent documentation migration to this page: google.github.io/android-fhir/contrib but for your information the original text is this:

Use unreleased GitHub build (for development/testing only)
Each GitHub build of the project also contains a maven repository. You can access unreleased features by downloading it and including it in your project. To acquire the maven repository, go the the actions page, and click on the build you want. In the build artifacts dropdown, you will find a maven-repository.zip file. After downloading and extracting it to a local folder, you can add it to your gradle setup via:
repositories {
  maven {
    url "file:///<path to the unzipped folder>"
  }
}

These artifacts are versioned as a combination of the current version + buildid (e.g. 0.1.0-alpha01-build_123). You can find the version in the zip file by checking the contents of it and update your build file to match that version.

OK, now onto github packages - with this change, do we always overwrite the SNAPSHOT? so if I have a PR in development, and I push a new commit it will overwrite the github packages?
Another thing we might need to be careful about is just any incurred cost - github.com/features/packages#pricing. With our repo under google organisation I'm hoping that we'll be covered by the google organisation quota/allowance. But I will investigate.

the old version will be kept as you can see here: https://github.com/icrc-fdeniger/android-fhir/packages/2129825 It's a kind of feature as we can use a specific SNAHPSHOT Version.
So yes it could be an issue with the storage. It seems there are no perfect solution for this point: https://github.com/orgs/community/discussions/48971

is there a way to publish a github package on whatever is on HEAD? i feel like that's much more useful?
@icrc-fdeniger have you tested that the current local packages (what i mentioned above) are generated in your forked repo? I'm not seeing any successful github build actions?

See https://github.com/icrc-fdeniger/android-fhir/actions/runs/8798380904/job/24145256498 for a full deployment
this run seems to have failied?
also can you update the description to clarify when the packages are generated? is it on every github action run? will different branches and prs overwrite each other's packages?

also can you update the description to clarify when the packages are generated? is it on every github action run? will different branches and prs overwrite each other's packages?

How about, instead of capturing this here, contributing (in this PR) clarifications for any questions related to all this in a snapshot.md (or whatever) doc in https://github.com/google/android-fhir/tree/master/docs/contrib? (And link it in mkdocs.yaml.

Thansk @icrc-fdeniger for this contribution! 🙏
For some background on the maven-repository.zip artifact, see this PR from @yigit: #249. We also have some documentation and I just realised it's a bit messed up after our recent documentation migration to this page: google.github.io/android-fhir/contrib but for your information the original text is this:

Use unreleased GitHub build (for development/testing only)
Each GitHub build of the project also contains a maven repository. You can access unreleased features by downloading it and including it in your project. To acquire the maven repository, go the the actions page, and click on the build you want. In the build artifacts dropdown, you will find a maven-repository.zip file. After downloading and extracting it to a local folder, you can add it to your gradle setup via:
repositories {
  maven {
    url "file:///<path to the unzipped folder>"
  }
}

These artifacts are versioned as a combination of the current version + buildid (e.g. 0.1.0-alpha01-build_123). You can find the version in the zip file by checking the contents of it and update your build file to match that version.

OK, now onto github packages - with this change, do we always overwrite the SNAPSHOT? so if I have a PR in development, and I push a new commit it will overwrite the github packages?
Another thing we might need to be careful about is just any incurred cost - github.com/features/packages#pricing. With our repo under google organisation I'm hoping that we'll be covered by the google organisation quota/allowance. But I will investigate.

the old version will be kept as you can see here: icrc-fdeniger/android-fhir/packages/2129825 It's a kind of feature as we can use a specific SNAHPSHOT Version.
So yes it could be an issue with the storage. It seems there are no perfect solution for this point: github.com/orgs/community/discussions/48971

is there a way to publish a github package on whatever is on HEAD? i feel like that's much more useful?
@icrc-fdeniger have you tested that the current local packages (what i mentioned above) are generated in your forked repo? I'm not seeing any successful github build actions?

See icrc-fdeniger/android-fhir/actions/runs/8798380904/job/24145256498 for a full deployment
this run seems to have failied?
also can you update the description to clarify when the packages are generated? is it on every github action run? will different branches and prs overwrite each other's packages?

it's failing because of the feature "git pages" that I didn't activate on my repo. I tested it on April  on my branch master-deploy-repo. To do the PR  I had to revert my "test setup" :
https://github.com/icrc-fdeniger/android-fhir/commit/82934301f1b8301f5fa6b21a3ce15bffefaa3a54
You can have a look here:
https://github.com/icrc-fdeniger/android-fhir/actions/runs/8798380904/job/24145256498
and here:

The publication is working. If you want I can modify the github actions on my branch to show that the action will be launched for every push on master-deploy-repo and it's ok.
If the PR is accepted as  is on the official repo,  The publish task is launched automatically  for all push to master only ( so when a PR is merged on master branch).  For PR, the task will be skipped:
https://github.com/icrc-fdeniger/android-fhir/blob/6cc76eec4476188bce55a455755968fa97a38cfb/.github/workflows/build.yml#L79

also can you update the description to clarify when the packages are generated? is it on every github action run? will different branches and prs overwrite each other's packages?

How about, instead of capturing this with comments here, contributing (in this PR) clarifications for any questions related to all this in a snapshot.md (or whatever) doc in master/docs/contrib? (And link it in mkdocs.yaml.

@vorburger I should create a new file shapshot.md in docs/contrib  and add a line related to this in mkdocs.yml and in the sections "Contributors". Am I correct ?

@vorburger I should create a new file shapshot.md in docs/contrib  and add a line related to this in mkdocs.yml and in the sections "Contributors". Am I correct ?

Yup, exactly; that's how you can contribute documentation! (If you want to, this is a suggestion.) You can name the MD file with any filename like, obviously.

OK, now onto github packages - with this change, do we always overwrite the SNAPSHOT? so if I have a PR in development, and I push a new commit it will overwrite the github packages?
Another thing we might need to be careful about is just any incurred cost - github.com/features/packages#pricing. With our repo under google organisation I'm hoping that we'll be covered by the google organisation quota/allowance. But I will investigate.

@jingtang10: a simpler solution could be to use jfrog.io or https://oss.sonatype.org as a maven repository as it's free for opensource and easier to user: we don't need to provide a Git Token to download artifacts ( see https://github.com/icrc/openmrs-android-fhir/blob/main/README.md#setup).

another solution could be to use jfrog.io or https://oss.sonatype.org/ as a maven repository

Does the build just produce a "standard Maven repo layout" on the filesystem - with a directory structure à la https://repo1.maven.org/maven2/? Or could it be made to? If so, then why not simply copy that into the site/ directory during the build, and make it show up and available on https://google.github.io/android-fhir/ ? Just a thought.

another solution could be to use jfrog.io or oss.sonatype.org as a maven repository

Does the build just produce a "standard Maven repo layout" on the filesystem - with a directory structure à la repo1.maven.org/maven2? Or could it be made to? If so, then why not simply copy that into the site/ directory during the build, and make it show up and available on google.github.io/android-fhir ? Just a thought.

But will the artifcats be available via a Maven Repository like: https://maven.google.com/web/index.html?#com.google.android.fhir:engine ?
my objective is just to be able to use latest android-fhir artifacts in our Android Application in a usual way and via dependencies declaration :)

Thansk @icrc-fdeniger for this contribution! 🙏
For some background on the maven-repository.zip artifact, see this PR from @yigit: #249. We also have some documentation and I just realised it's a bit messed up after our recent documentation migration to this page: google.github.io/android-fhir/contrib but for your information the original text is this:

Use unreleased GitHub build (for development/testing only)
Each GitHub build of the project also contains a maven repository. You can access unreleased features by downloading it and including it in your project. To acquire the maven repository, go the the actions page, and click on the build you want. In the build artifacts dropdown, you will find a maven-repository.zip file. After downloading and extracting it to a local folder, you can add it to your gradle setup via:
repositories {
  maven {
    url "file:///<path to the unzipped folder>"
  }
}

These artifacts are versioned as a combination of the current version + buildid (e.g. 0.1.0-alpha01-build_123). You can find the version in the zip file by checking the contents of it and update your build file to match that version.

OK, now onto github packages - with this change, do we always overwrite the SNAPSHOT? so if I have a PR in development, and I push a new commit it will overwrite the github packages?
Another thing we might need to be careful about is just any incurred cost - github.com/features/packages#pricing. With our repo under google organisation I'm hoping that we'll be covered by the google organisation quota/allowance. But I will investigate.

the old version will be kept as you can see here: icrc-fdeniger/android-fhir/packages/2129825 It's a kind of feature as we can use a specific SNAHPSHOT Version.
So yes it could be an issue with the storage. It seems there are no perfect solution for this point: github.com/orgs/community/discussions/48971

is there a way to publish a github package on whatever is on HEAD? i feel like that's much more useful?
@icrc-fdeniger have you tested that the current local packages (what i mentioned above) are generated in your forked repo? I'm not seeing any successful github build actions?

See icrc-fdeniger/android-fhir/actions/runs/8798380904/job/24145256498 for a full deployment
this run seems to have failied?
also can you update the description to clarify when the packages are generated? is it on every github action run? will different branches and prs overwrite each other's packages?

it's failing because of the feature "git pages" that I didn't activate on my repo. I tested it on April on my branch master-deploy-repo. To do the PR I had to revert my "test setup" : icrc-fdeniger@8293430
You can have a look here: https://github.com/icrc-fdeniger/android-fhir/actions/runs/8798380904/job/24145256498 and here:
The publication is working. If you want I can modify the github actions on my branch to show that the action will be launched for every push on master-deploy-repo and it's ok.
If the PR is accepted as is on the official repo, The publish task is launched automatically for all push to master only ( so when a PR is merged on master branch). For PR, the task will be skipped: https://github.com/icrc-fdeniger/android-fhir/blob/6cc76eec4476188bce55a455755968fa97a38cfb/.github/workflows/build.yml#L79

thanks for the detailed explanation - this sounds all good to me.

OK, now onto github packages - with this change, do we always overwrite the SNAPSHOT? so if I have a PR in development, and I push a new commit it will overwrite the github packages?
Another thing we might need to be careful about is just any incurred cost - github.com/features/packages#pricing. With our repo under google organisation I'm hoping that we'll be covered by the google organisation quota/allowance. But I will investigate.

@jingtang10: another solution could be to use jfrog.io or https://oss.sonatype.org as a maven repository as it's free for opensource project and it will be easier for other projects to integrate android-fhir as a SNAPSHOT dependency : we don't need to provide a Git Token to download artifacts ( see https://github.com/icrc/openmrs-android-fhir/blob/main/README.md#setup).

@bashir2 do you push any project to maven central? i can't remember.

OK, now onto github packages - with this change, do we always overwrite the SNAPSHOT? so if I have a PR in development, and I push a new commit it will overwrite the github packages?
Another thing we might need to be careful about is just any incurred cost - github.com/features/packages#pricing. With our repo under google organisation I'm hoping that we'll be covered by the google organisation quota/allowance. But I will investigate.

@jingtang10: another solution could be to use jfrog.io or https://oss.sonatype.org as a maven repository as it's free for opensource project and it will be easier for other projects to integrate android-fhir as a SNAPSHOT dependency : we don't need to provide a Git Token to download artifacts ( see https://github.com/icrc/openmrs-android-fhir/blob/main/README.md#setup).

i have been considering this actualy - perhaps we can have a discussion at the next android fhir sdk developers call. let me konw if you can attend @icrc-fdeniger (also @vorburger).

actually one last thing before merging - i feel we're really publishing a snapshot of the HEAD, instead of any particular versions... I couldn't find a way to do this but when we're publishing github packages can we just use the build id or something that is not tied to our version number for public releases?
@icrc-fdeniger @vorburger

actually one last thing before merging - i feel we're really publishing a snapshot of the HEAD, instead of any particular versions... I couldn't find a way to do this but when we're publishing github packages can we just use the build id or something that is not tied to our version number for public releases?
@icrc-fdeniger @vorburger

I guess this is better because someone might be using a particular snapshot in their dependency and could upgrade it to the latest build snapshot which is available. On the other hand, using version will overwrite the snapshot and the the developer may get confused that even though they are using the latest snapshot available they are not getting all the features added.
Only problem I see is that each PR will push a snapshot and this can grow huge... So we might need another GitHub action (if possible) to delete old snapshots?

@jingtang10 @vorburger documentation added.

OK, now onto github packages - with this change, do we always overwrite the SNAPSHOT? so if I have a PR in development, and I push a new commit it will overwrite the github packages?
Another thing we might need to be careful about is just any incurred cost - github.com/features/packages#pricing. With our repo under google organisation I'm hoping that we'll be covered by the google organisation quota/allowance. But I will investigate.

@jingtang10: another solution could be to use jfrog.io or https://oss.sonatype.org as a maven repository as it's free for opensource project and it will be easier for other projects to integrate android-fhir as a SNAPSHOT dependency : we don't need to provide a Git Token to download artifacts ( see https://github.com/icrc/openmrs-android-fhir/blob/main/README.md#setup).

@bashir2 do you push any project to maven central? i can't remember.

Sorry forgot to reply this earlier; yes we do: https://central.sonatype.com/search?q=com.google.fhir.gateway
@icrc-fdeniger I have not read all the discussions on this PR but IIUC you want to use SNAPSHOT versions of the android-fhir libraries in your dependent code. One approach that I have seen people use is to check-out/build locally if SNAPSHOT version of a dependency is needed. IIRC, OpenMRS is also set up that way. So for example, if I needed to change something in openmrs-core and then use that in development of one of OpenMRS modules, I would checkout core and do a mvn install locally (can be built as part of CI process too). Would that solve this problem?

OK, now onto github packages - with this change, do we always overwrite the SNAPSHOT? so if I have a PR in development, and I push a new commit it will overwrite the github packages?
Another thing we might need to be careful about is just any incurred cost - github.com/features/packages#pricing. With our repo under google organisation I'm hoping that we'll be covered by the google organisation quota/allowance. But I will investigate.

@jingtang10: another solution could be to use jfrog.io or oss.sonatype.org as a maven repository as it's free for opensource project and it will be easier for other projects to integrate android-fhir as a SNAPSHOT dependency : we don't need to provide a Git Token to download artifacts ( see icrc/openmrs-android-fhir@main/README.md#setup).

@bashir2 do you push any project to maven central? i can't remember.

Sorry forgot to reply this earlier; yes we do: central.sonatype.com/search?q=com.google.fhir.gateway
@icrc-fdeniger I have not read all the discussions on this PR but IIUC you want to use SNAPSHOT versions of the android-fhir libraries in your dependent code. One approach that I have seen people use is to check-out/build locally if SNAPSHOT version of a dependency is needed. IIRC, OpenMRS is also set up that way. So for example, if I needed to change something in openmrs-core and then use that in development of one of OpenMRS modules, I would checkout core and do a mvn install locally (can be built as part of CI process too). Would that solve this problem?

Thanks @bashir2  for your answer.
You are right: we want to  use SNAPSHOT versions of android-fhir in our dependent code.
For OpenMRS ( given as an example but true for all projects :)) there are 2 main use cases:

the situation you describe: you want to modify openmrs-core and  so you have  to check-out/build locally. This use case is clear.
you want to test your application with the latest version of openmrs-core. In this case, the usual way is to use SNAPSHOT version to avoid redoing the full check-out/build process.

For the second point, for sure developpers can check-out/build the source  locally but:

It's a long process: I must be sure that all developpers in our team performed a pull/build locally of android-fhir. Maybe some of them will forget or a commit in android-fhir can break some of our features ( and we will have to checkout an earlier commit) , etc...We will have to deal with all these situations.
With a SNAPSHOT dependency, the update will be automatically done and we can force it with the option --refresh-dependencies to be sure we have the latest. If a commit in android-fhir breaks our application, we can modify our dependencies to use a pinned snapshot version temporarely ( until the commit is fixed in android-fhir or our application is modified) and so all developpers can go on.
Impossible or very complex to integrate this approach "check-out/build" in a CI Tool to launch integration tests or to generate demo "apk". This point is surely questionable but no choice for now if we want to use  the latest improvements  in FHIR engine concerning OpenMRS. As soon as a FHIR engine version is released and contains all features we need we will use this version and then we will stop using SNAPHSOT for sometimes to release our application. And then we will prepare a v2 and use SNAPSHOT versions again to be sure we are up-to-date to latest dev ...

IMHO, deploying SNAPSHOT versions in Maven Central will be better than using GitHub packages ( no token needed and no extra cost). For now this PR is using GitHub Packages ( and documentation is done for this) but will be happy the modify it to use Maven Central.

to use Maven Central

Maven Central is... erm, "evolving" ("cough"), and I think they won't support SNAPSHOT repos in the future.
Just FYI, https://github.com/vorburger/ch.vorburger.exec/issues/212 (unrelated, just FYI).

to use Maven Central

Maven Central is... erm, "evolving" ("cough"), and I think they won't support SNAPSHOT repos in the future.
Just FYI, vorburger/ch.vorburger.exec#212 (unrelated, just FYI).

hi @vorburger - can you pls add more details on maven central's support for snapshot?

hi @vorburger - can you pls add more details on maven central's support for snapshot?

My advice is to stay clear of Maven Central, for SNAPSHOT.
I advise to either use GitHub packages (with some auto clean), or simply a SNAPSHOT repo on the docs site.

tweaked a few things in the doc.
thanks so much @icrc-fdeniger for this! excited for this change!

tweaked a few things in the doc.
thanks so much @icrc-fdeniger for this! excited for this change!

thanks this change will help us :)
Do I have something to do or just wait for the workflow to go on ?

tweaked a few things in the doc.
thanks so much @icrc-fdeniger for this! excited for this change!

thanks this change will help us :)
Do I have something to do or just wait for the workflow to go on ?

No I've enabled auto merge so hopefully it'll go in once tests pass


how to update transformer module  in miniLLM

The transformer module should be manually updated because we have several modifications on it. We will consider updating it to the latest version recently.


kpt cfg commands don't work on top-level lists

If the yaml file is a list kpt cfg fmt and other commands won't work on it as they expect a map at the top level.

It looks to me that from code, there is a rule for fmt work on a yaml file that is a sequence (correct me if I misunderstood it)
that is:
the sequence have to have
".spec.template.spec.containers": "name",
".webhooks.rules.operations":     "",
in path.

To repro, you can use a List as return by a kubectl:
kubectl get namespaces -o yaml

@jijiew I think your test looks correct. Can we simplify it a bit (mostly just reduce the number of fields in each of the list elements) and add a test for it to kyaml? Or maybe there is a test already?
And then I think we can just close this issue.


Index page trailing slash 200

Describe the bug
Index page returns 200 on homepage with trailing slash.
Reproduction
https://kit.svelte.dev/

Logs
No response
System Info
any

Severity
annoyance
Additional Information
No response

I don't think this is an issue, since stripping the trailing slash

I don't think this is an issue on / pages, you'll notice if you go to other pages such as /docs/introduction/ you are getting a redirect. Let me know if I am wrong :pray:

@ghostdevv hello, Google expects the response code that explicitly indicates redirect.
https://developers.google.com/search/blog/2010/04/to-slash-or-not-to-slash
Otherwise it counts it as duplicate content, as I understand from the article.  Correct me if I'm wrong.
And I guess it wouldn't be a problem for some/page/down/below/with-low-seo-weight. But index page has the highest amount of weight (importance).
Or you can download sitechecher Google extension that will mark it as error and reduce your website overall tech audit score on each internal page because of index page.

@ghostdevv hello, Google expects the response code that explicitly indicates redirect.
https://developers.google.com/search/blog/2010/04/to-slash-or-not-to-slash
Otherwise it counts it as duplicate content, as I understand from the article.  Correct me if I'm wrong.
And I guess it wouldn't be a problem for some/page/down/below/with-low-seo-weight. But index page has the highest amount of weight (importance).
Or you can download sitechecher Google extension that will mark it as error and reduce your website overall tech audit score on each internal page because of index page.

From the same article:

Rest assured that for your root URL specifically, https://example.com is equivalent to https://example.com/ and can't be redirected even if you're Chuck Norris.

From the same article:

Rest assured that for your root URL specifically, https://example.com is equivalent to https://example.com/ and can't be redirected even if you're Chuck Norris.

Yes, the second grade doesn't mean defective lol

@fehnomenal as I understand, it doesn't mean it's the best behavior. Of course google will be able to catch it after some time for old domains. But it is still duplicated content on main page that can very well increase SEO budget for new sandboxed websites.


Adding matched_queries to search hits

Queries/sub-queries that match named queries are returned as matched_queries in the Elasticsearch response. Matched queries aren't currently populated in SearchHits. This PR adds a new field to SearchHit and populates the new field.
https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-named-queries-and-filters.html

Coverage increased (+0.09%) to 58.027% when pulling 8c3611982fed7bf8a8307957f36165afbcb09aa8 on nirajpatel:matched-queries-parsing into f0c0e77fccc7469dc8d6689b3833d0d20f922ffe on searchbox-io:master.

Hey @johnlister @ferhatsb! Is there any way we can get this PR merged in please? Currently blocked on this. Thanks in advance.

Thanks @ferhatsb :)

@ferhatsb can we do a cut/release please?


Update slack invite link

Note: DefectDojo is now on Python3 and Django 2.2.1 Please submit your pull requests to the 'dev' branch as the 'legacy-python2.7' branch is only for bug fixes. Any new features submitted to the legacy branch will be ignored and closed.
When submitting a pull request, please make sure you have completed the following checklist:

[x] Your code is flake8 compliant
[x] Your code is python 3.5 compliant
[x] If this is a new feature and not a bug fix, you've included the proper documentation in the ReadTheDocs documentation folder. https://github.com/DefectDojo/Documentation/tree/master/docs or provide feature documentation in the PR.
[x] Model changes must include the necessary migrations in the dojo/dd_migrations folder.
[x] Add applicable tests to the unit tests.

Fixes #1636

@Aniket21mathur Please re-submit against the dev branch and not the master branch. Thanks.

@madchap thanks, I will do that.

#1687


Resolve unexpected character encodings in behavior importer

eg:
failed to convert 143 unexpected invalid character encodings for ["\x93", "\x94", "\xF3", "\xED", "\xE1", "\xFA", "\xC1", "\x95", "\xA8", "\xF1", "\x97", "\x96", "\x85", "\xA0"]

Closed for duplicate https://github.com/studentinsights/studentinsights/issues/1808.


Add Eq & Ord instances to ForeignPtr newtypes. (#42)

This PR adds Eq and Ord instances for every newtype that wraps around a ForeignPtr. These instances convert the foreign pointer to a regular pointer just like the instances in base, then work on the pointer's contents instead of the address or pointer itself.

closes https://github.com/haskell-cryptography/libsodium-bindings/issues/42

I've resolved all the past conversations since I've redone the entire implementation. This, hopefully, is better than before.

@kozross on BSD the last argument is supposed to be a size_t but the error message says it has gotten an unsigned long. Might be related? I'm not super clear on casting rules…

No wait, that could be a red herring. How about changing the definition to string.h memcmp?

Haha yes, perfect.


Delete system tests

Rather than spending an inordinate amount of time trying to fix the system tests, perhaps it's time to just remove them altogether.
Pros:

Less time consuming
Most of the commands that are covered by the system tests are either going to be removed/modified anyway
Deadline is fast approaching and there are many other things that we need to prioritise

Cons:

Less code coverage
Higher risk of regressions

Vote for this by approving this PR. I'll merge it after we get 3 approving votes (including my own)

Well...

I'm sure others have done worse, so I think we're fine


return None and raise warning when transform_geom dst_ogr_geom is NULL

Related to: https://github.com/mapbox/rasterio/issues/1170

All transformer tests pass. The failure is an unrelated test failure:
E           AttributeError: 'UnparsedPath' object has no attribute 'scheme'

@snowman2 can you rebase this on the maint-1.9 branch ?

@snowman2 oddly, this new test fails for me with GDAL 3.0.2. There's no warning and no failure to transform.

Huh, that's interesting. I wonder what changed 🤔


git clone over ssh, gitea log says user does not exist.

Gitea version (or commit ref):
Git version:
Operating system:

Database (use [x]):

[x] PostgreSQL
[ ] MySQL
[ ] MSSQL
[ ] SQLite

Can you reproduce the bug at https://try.gitea.io:

[ ] Yes (provide example URL)
[x] No

Log gist:

on server side (journalctl /usr/bin/local/gitea)
May 21 19:53:00 gittea gitea[6005]: 2021/05/21 19:53:00 Started GET /api/internal/serv/command/1/git:gabe/resume?mode=1&verb=git-upload-pack for [::1]:47464
May 21 19:53:00 gittea gitea[6005]: 2021/05/21 19:53:00 ...ters/private/serv.go:108:ServCommand() [E] Unable to get repository owner: git:gabe/resume Error: user does not exist [uid: 0, name
May 21 19:53:00 gittea gitea[6005]: 2021/05/21 19:53:00 Completed GET /api/internal/serv/command/1/git:gabe/resume?mode=1&verb=git-upload-pack 500 Internal Server Error in 1.815752ms

client side (git clone ssh://git@venberg.xyz/git:Gabe/lidarutils.git)
Cloning into 'lidarutils'...
Gitea: Internal Server Error
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Description

...
Screenshots

Your git clone format is wrong. it should be<EMAIL_ADDRESS>or git@venberg.xyz:Gabe/lidarutils.git

Ahh, I see it. just a PEBKAC issue. (Problem Exists Between Keyboard And Chair)


Group custom settings by extension

Problem: as part of #4551 we need to open Extensions page from the Preferences correspondant tab. It was impossible because we reserved Extensions tab for showing extension custom settings.
Solution: grouping custom preferences by extension and show them as separate tabs. It is also more meaningful now, because previously settings from all extensions were listed in one page.

https://user-images.githubusercontent.com/9607060/148756019-35dd2a85-3d50-4ada-9623-7ead22ea35ce.mov

This PR will be refactored a bit after https://github.com/lensapp/lens/pull/4671 will be merged. Since it's heavily uses AppPreferenceRegistry and now it's converted as a dependency.

Closing this in favor of https://github.com/lensapp/lens/pull/5284


Merge proxy into main repo

This is a proposal to merge istio/proxy into the main repo.
PROs:

shared ownership, we have a decline in the contributions to istio/proxy and issues spanning envoy are not getting enough attention;
faster development, we can avoid the two-step commit process for envoy changes;
majority of istio/proxy code should be either eventually upstreamed or spawned into wasm-specific repositories;
Istio can benefit from faster tests - relying on k8s to run basic datapath testing does not give us enough confidence at scale; testing without containers can scale much higher.

CONs:

cherry pick disruption;
large CI tests - that can be guarded by a basic git diff on the proxy directory;
bazelification - this is purely aesthetics, there is no need to look at bazel files if not developing proxy.

cc @ericvn (thanks for the diligent retesting of tsan failures ;) )

@kyessenov do you have concerns with all istio tests running against any change to proxy?
The inverse (only run proxy tests if proxy/ change can be done fairly simply), but that will increase the tests run on any proxy change. This may be a good thing since we get more test coverage, though.
If we do this, we should consider moving ztunnel as well. Doesn't have to be decoupled. Ztunnel is a bit worse since ztunnel CI is <10 minutes, so the overhead of istio tests (slowest is ~40min - which is bad) is high. For proxy I think the tests are slow there so the extra tests may not be too bad (?).
If we do it, I would imagine:
proxy/ - has all proxy dumped here
ztunnel/ - has all of ztunnel dumped here
... rest of istio code remains ...

CI jobs can use run_if_change: proxy/ and run_if_change: ztunnel/ for proxy/ztunnel specific jobs.
We may still want 2 different build-tools jobs (although we could consider just having a gigantic shared one). We could do this in a few ways:

Main folder uses normal image, proxy users set env var manually to use proxy image. Main Makefile.core.mk can import proxy/makefile.
Main folder uses normal, proxy users have their own copy of all the Make infra under proxy/. Must be in proxy/ folder to do any proxy operations
something else - probably lot of ways to do it

Overall I have no concerns with this - as long as proxy maintainers want it, of course

Oh totally forgot about that. That may be an issue. Prow dependencies support is basically non-existent. We can do some hacks but its not great.
Even if we did have a DAG execution, always building proxy as step 0 for istio tests would at a lot of time to most PRs. If we somehow would use a precompiled one if proxy/ folder is unchanged it could work, although that is a bit odd (for example, PR 1 changes proxy and merges. In postsubmit it is building proxy. Now PR 2 starts testing... does it wait for postsubmit to finish building...?)

I haven't been here long, but FWIW I'm concerned about istio/istio having an even higher barrier to entry spanning 3 main languages and toolchains: Go, C++, and Rust (if ztunnel moves - I agree with John that it should move if proxy does). The new developer onboarding experience for Istio is already a bit intimidating; I worry this would exacerbate that. There are definitely benefits, but I wanted to call out the onboarding DX here

@keithmattix We certainly shouldn't make knowledge of C++ a pre-requisite for new developers, and I think if we merge, the CI will simply skip proxy tests without any changes there. On the other hand, it would be beneficial to the project if contributors had more hands-on experience with the core technology instead of treating it as a black box, and Envoy is certainly a big part of Istio.

it would be beneficial to the project if contributors had more hands-on experience with the core technology instead of treating it as a black box, and Envoy is certainly a big part of Istio.

I definitely don't disagree with this point. I'm thinking more of the toolchain than anything else

I think ztunnel release/testing should be as decoupled as possible. It was a huge mistake to tie proxy and istiod - resulting in many upgrade problems because the tests assumed they are in sync.

This is in particular critical for ambient - since waypoints and control planes may be operated independently from ztunnel, and the data plane may be implemented in different ways. Protocol and API stability is very important.

Also don't think ztunnel should move into the main repo.
If working on changes across two repos is hard:

We have a CI gap
We have a build/deploy/test gap

A monorepo will not solve those problems.

I am not proposing anything about ztunnel. This is just about Envoy L7 stuff. Let's not mix the two discussions.

Overall I'm not a fan of the merge. It would make the istio repo even more complex for newcomers as Keith pointed out. The structure as it is now is a barrier of entry as it is and I think adding istio/proxy and another language and toolset would just make it even more complicated. This is a huge con, imho.
With shared ownership, we have a decline in the contributions to istio/proxy and issues spanning envoy are not getting enough attention; I don't see how changing the repo the code is in would change the contributions. Maybe there's another way to gain focus on the issues that need attention that affect istio/proxy?


🌐 Add German translation for docs/deployment/index.md

@mawassk , are you still proceeding with this? I'd be happy to review. But it seems the englisch version has already changed a lot.

Due to the big changes in the english version i will close this PR. It seems to be better to work on this in another PR


Possible athenad memory leak

It was using 15% of memory, causing low memory events. I had not rebooted the device in a few days iirc and had uploaded many rlogs.
I tried to attach with memray, but instead the process just crashed so I didn't get any information about this. @deanlee could this be similar to the leak you found in athena previously?

It's hard to identify the exact issue, as the code is unclear and difficult to understand. However, multiple factors could be causing the resource leak. One possibility is that the athenad's implementation might cause garbage collector to accumulate objects over time, leading to memory issues during long runs.
additionally, thread race conditions related to queue and other global variable usage, as well as bad practices like not calling task_done() after queue.get(), could also contribute to issues.

The following issues and fixes address potential resource leaks and long-term accumulation risks:

Unclosed WebSocket: Not closing the WebSocket after a main loop restart can cause resource leaks by keeping connections open. https://github.com/commaai/openpilot/pull/34085
Thread Safety in Upload Handling: Improper synchronization can lead to race conditions, potentially causing threads to terminate and forcing the main loop to restart. This may result in invalid or inconsistent data being stored in the upload queue, leading to data corruption or queue bloat.  https://github.com/commaai/openpilot/pull/34084
Memory Bloat in cur_upload_items[tid]: The cur_upload_items dictionary isn't properly cleared of invalid or duplicate entries. If threads restart with new IDs, the dictionary can grow indefinitely, causing memory bloat over time.

This is still happening after uploading files

I see this after uploading rlogs:
lock                            120       +47
cell                            828       +47
builtin_function_or_method     1685       +43
list                           1776       +38
tuple                          4715       +30
ReferenceType                  2736       +30
Event                            75       +15
socket                           24       +15
Condition                        54       +14
deque                            62       +14
method                          109       +13
function                       7974       +13
SSLContext                       14        +8
WebSocket                        13        +8
sock_opt                         13        +8
frame_buffer                     13        +8
continuous_frame                 13        +8
SplitResult                      14        +8
SSLSocket                        14        +8
_SSLSocket                       14        +8

The issue may not just be with the upload process—WebSocket instance growth could indicate deeper resource management problems.

@sshane: Do we really need the ws_manage thread? It doesn’t handle exceptions, and changing socket options while the WebSocket is sending or receiving could cause race conditions or undefined behavior. If socket options need to be set, a better approach would be to close the connection, reconnect, and configure the socket before restarting ws_recv/ws_send threads.
Also, storing sock = ws.sock at the start of the thread and assuming it's valid throughout the loop may not be safe.

The current athenad implementation lacks consideration for concurrent data access safety in a multi-threaded environment. Besides https://github.com/commaai/openpilot/pull/34084, which fixes race conditions for global variables, the UploadQueueCache class also has multi-threading issues. UploadQueueCache may be accessed by multiple threads simultaneously, but it is not thread-safe.

@sshane : The athenad implementation does not maintain the order between received commands and their corresponding responses. Responses may be sent in a different order than the commands were received. Is this behavior expected and by design, or should it be addressed as a issue?

@sshane: The current implementation does not preserve the order between received commands and their responses. Responses may be sent out of order.
Commands are placed in the recv_queue and processed by multiple jsonrpc_handler threads, with responses added to the send_queue after handling. Due to the multithreaded nature, the order of responses may differ from the order of received commands.
Is this behavior by design, or should it be considered an issue?

The order of responses does not matter since the JsonRPC spec handles this. The 'requester' sends a request with a unique ID which could just be a ns timestamp for example. When the 'responder' (athena in this case) responds, it will also send the ID in the payload. https://www.jsonrpc.org/specification#response_object


full_accounts needs list of assets created by that account

The get_full_accounts api call needs to return a simple list of asset ids of the assets that were created by this account

This issue was moved to bitshares/bitshares-core#101


Add support for qwen2.5-coder

Try https://ollama.com/library/qwen2.5-coder (1.5b, 7b) with FIM, but it doesnt work. It show in output some tokens e.g. <fim_suffix>. It would be nice to add special template for it.

Do you know which template it should be?

And what autocomplete (FIM) model is the best and tested with twinny?

I found that codellama-code or deepseek-base models work the best.

Thanks

I've experimented a bit after seeing the odd template of
❯ ollama show --template qwen2.5-coder:1.5b
{{- if .Suffix }}<|fim_prefix|>{{ .Prompt }}<|fim_suffix|>{{ .Suffix }}<|fim_middle|>
...

and the base models doesn't have much of a template. So I assumed you have to use the instruct model and you have to fill the suffix parameter in api/generate and lo and behold
❯ echo -e $(curl http://localhost:11434/api/generate -d '{
  "model": "qwen2.5-coder:1.5b",
  "prompt": "def fib(", "suffix": " "
}' | jq -s 'map(.response) | join("")')
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  6223    0  6149  100    74   4721     56  0:00:01  0:00:01 --:--:--  4775
"n):
 \"\"\"
 Compute the nth Fibonacci number.

 Args:
 n (int): The position in the Fibonacci sequence to compute.

 Returns:
 int: The nth Fibonacci number.
 \"\"\"
 if n <= 1:
 return n
 "

while when leaving out the suffix or even setting it to empty string I get
❯ echo -e $(curl http://localhost:11434/api/generate -d '{
  "model": "qwen2.5-coder:1.5b",
  "prompt": "def fib(", "suffix": ""
}' | jq -s 'map(.response) | join("")')
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 26463    0 26390  100    73   6107     16  0:00:04  0:00:04 --:--:--  6125
" It looks like you're trying to define a function named `fib` in Python. However, there seems to be an issue with the syntax of your code. The `fib` function should be properly defined using parentheses and arguments.

Here's a corrected version of your code:

```python
def fib(n):
 # Base cases for the first two Fibonacci numbers
 if n == 0:
 return 0
 elif n == 1:
 return 1

 # Recursive case to calculate Fibonacci sequence
 else:
 return fib(n-1) + fib(n-2)

In this code:

The function fib takes one argument n.
The base cases handle the first two Fibonacci numbers (0 and 1).
The recursive case calculates the Fibonacci number by summing the two preceding ones.

You can call this function with an integer argument to compute the corresponding Fibonacci number. For example:
print(fib(5)) # Output: 8 (since 5th Fibonacci number is 8)

This will print 8, which is the fifth Fibonacci number in the sequence."

which is not what we want.

Do you have any guidance on how to make this work with Twinny?

I can get similar results with
echo -e $(curl http://localhost:11434/api/generate -d '{
  "model": "qwen2.5-coder:1.5b",
  "prompt": "def fib(", "template": "<|fim_prefix|>{{ .Prompt }}<|fim_suffix|>{{ .Suffix }}<|fim_middle|>"
}' | jq -s 'map(.response) | join("")')

by overriding the template

Ok for future googlers - my workaround for now is:
cat <<EOF > modelfile
FROM qwen2.5-coder:1.5b
TEMPLATE "{{ .Prompt }}"
EOF
ollama create qwen2.5-coder:1.5b-plain -f modelfile
rm modelfile
echo -e $(curl http://localhost:11434/api/generate -d '{
  "model": "qwen2.5-coder:1.5b-plain",
  "prompt": "<|fim_prefix|>def fib(<|fim_suffix|><|fim_middle|>"
}' | jq -s 'map(.response) | join("")')

set the custom fim template to
<|fim_prefix|>{{{prefix}}}<|fim_suffix|>{{{suffix}}}<|fim_middle|>

and Twinny completes like a champ when selecting qwen2.5-coder:1.5b-plain model.
Go for 7b if you can.

Hey @kirel is it possible to get the same result by using the deepseek fim template? Seems maybe just the model needs to be plain? Thanks.

There qwen models are trained on the template above. There is no reason to use a different one. The problem is that with ollama's default twinny isn't passing it so that it's compatible.

https://github.com/twinnydotdev/twinny/blob/e5d2f6ceafa15affb24419238c3cf303dd26d413/src/extension/fim-templates.ts#L96
Thanks, sorry I thought deepseek's template was the same. Seems we already have this template declared? Will this work?

Thanks @kirel.  FYI Further to this I just released v3.17.11 which adds proper support for codeqwen2.5.  There is no need to create a new model using a model file as suggested and you can just use the base model e.g qwen2.5-coder:7b-base, it works perfectly fine.  The issue was actually the extra context which was added to the suffix in the original template. @kv-gits if you now choose this model with the codeqwen fim template all should work perfectly.  Many thanks.

Cool, thanks! No dirty workarounds 🤝