[SPARK-36533][SS] Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches

What changes were proposed in this pull request?
This change creates a new type of Trigger: Trigger.AvailableNow for streaming queries. It is like Trigger.Once, which process all available data then stop the query, but with better scalability since data can be processed in multiple batches instead of one.
To achieve this, this change proposes a new interface SupportsTriggerAvailableNow, which is an extension of SupportsAdmissionControl. It has one method, prepareForTriggerAvailableNow, which will be called at the beginning of streaming queries with Trigger.AvailableNow, to let the source record the offset for the current latest data at the time (a.k.a. the target offset for the query). The source should then behave as if there is no new data coming in after the beginning of the query, i.e., the source will not return an offset higher than the target offset when latestOffset is called.
This change also updates FileStreamSource to be an implementation of SupportsTriggerAvailableNow.
For other sources that does not implement SupportsTriggerAvailableNow, this change also provides a new class FakeLatestOffsetSupportsTriggerAvailableNow, which wraps the sources and makes them support Trigger.AvailableNow, by overriding their latestOffset method to always return the latest offset at the beginning of the query.
Why are the changes needed?
Currently streaming queries with Trigger.Once will always load all of the available data in a single batch. Because of this, the amount of data a query can process is limited, or Spark driver will run out of memory.
Does this PR introduce any user-facing change?
Users will be able to use Trigger.AvailableNow (to process all available data then stop the streaming query) with this change.
How was this patch tested?
Added unit tests.

Test build #142557 has started for PR 33763 at commit 5d0aee8.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47059/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47059/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47059/

Test build #142557 has finished for PR 33763 at commit 5d0aee8.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class FakeLatestOffsetMicroBatchStream(source: MicroBatchStream)
class FakeLatestOffsetSource(source: Source)
class FakeLatestOffsetSupportsTriggerAvailableNow(source: SparkDataStream)
case class OneBatchExecutor() extends TriggerExecutor
case class MultiBatchExecutor() extends TriggerExecutor

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142557/

Test build #142578 has started for PR 33763 at commit 6c017c7.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47079/

Test build #142582 has started for PR 33763 at commit c2223f0.

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47079/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47079/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47084/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47084/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47084/

Test build #142578 has finished for PR 33763 at commit 6c017c7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142578/

Test build #142601 has started for PR 33763 at commit 6d9253e.

Test build #142582 has finished for PR 33763 at commit c2223f0.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class FakeLatestOffsetMicroBatchStream(source: MicroBatchStream)
class FakeLatestOffsetSource(source: Source)
class FakeLatestOffsetSupportsTriggerAvailableNow(source: SparkDataStream)
case class OneBatchExecutor() extends TriggerExecutor
case class MultiBatchExecutor() extends TriggerExecutor

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142582/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47101/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47101/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47101/

Test build #142614 has started for PR 33763 at commit c2223f0.

Test build #142601 has finished for PR 33763 at commit 6d9253e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142601/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47114/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47114/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47114/

Test build #142614 has finished for PR 33763 at commit c2223f0.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class FakeLatestOffsetMicroBatchStream(source: MicroBatchStream)
class FakeLatestOffsetSource(source: Source)
class FakeLatestOffsetSupportsTriggerAvailableNow(source: SparkDataStream)
case class OneBatchExecutor() extends TriggerExecutor
case class MultiBatchExecutor() extends TriggerExecutor

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142614/

@HeartSaVioR, @brkyvz, could you review this? Thanks!

cc @viirya

Btw, thanks for the great contribution! Nice feature indeed.

Test build #142720 has started for PR 33763 at commit e0ad3eb.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47220/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47220/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47220/

Test build #142720 has finished for PR 33763 at commit e0ad3eb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142720/

Thanks for the work. I will review in next few days.
I think we also need to update the document. But I don't see doc change included yet.

Thanks @viirya! Could you point me to some places where I need to update the doc?

The structured streaming programming guide: https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#triggers
The source is at docs/structured-streaming-programming-guide.md.

Test build #142755 has started for PR 33763 at commit 10abc21.

This looks awesome! Left some comments. It would also be great to add tests around the progress reporting (not overwriting the Source description with that of the wrapper class, etc)

Thanks! Added assertions for the source descriptions in TriggerAvailableNowSuite.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47255/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47255/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47255/

Test build #142755 has finished for PR 33763 at commit 10abc21.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class AvailableNowDataStreamWrapper(source: SparkDataStream)
class AvailableNowMicroBatchStreamWrapper(source: MicroBatchStream)
class AvailableNowSourceWrapper(source: Source)
case class SingleBatchExecutor() extends TriggerExecutor

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142755/

Test build #142853 has started for PR 33763 at commit d13807a.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47356/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47356/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47356/

Test build #142853 has finished for PR 33763 at commit d13807a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142853/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47358/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47358/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47358/

Test build #142861 has started for PR 33763 at commit 0824910.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47364/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47364/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47364/

Test build #142861 has finished for PR 33763 at commit 0824910.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #142855 has finished for PR 33763 at commit 85cb90d.

This patch fails from timeout after a configured wait of 500m.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142855/

retest this please

Test build #142865 has started for PR 33763 at commit 0824910.

Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47368/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47368/

Test build #142866 has started for PR 33763 at commit c96d60d.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47369/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47369/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47369/

Test build #142865 has finished for PR 33763 at commit 0824910.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142865/

retest this, please

@viirya Would you mind revisiting this? The PR looks OK to me, but I'd like to see your approval as you're reviewing the change. Thanks in advance!

Test build #142872 has started for PR 33763 at commit c96d60d.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47375/

Test build #142866 has finished for PR 33763 at commit c96d60d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142866/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47375/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47375/

@viirya Would you mind revisiting this? The PR looks OK to me, but I'd like to see your approval as you're reviewing the change. Thanks in advance!

Also @brkyvz could you review again for the changes you requested?

Test build #142872 has finished for PR 33763 at commit c96d60d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142872/

Test build #142896 has started for PR 33763 at commit b6c26e2.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47399/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47399/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47399/

Test build #142896 has finished for PR 33763 at commit b6c26e2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142896/

Thanks! Merging to master!

Thanks @bozhang2820 for the great contribution and thanks all for reviewing! I just merged this into master.

@HeartSaVioR @viirya @bozhang2820 is there a way to call this from Pyspark? I can't figure it out

Nice catch. I realized it was missing. I'm going to address this soon.


Remove redundant badges from readme

Codecov is only tracking JS coverage, so displaying a coverage figure for the whole project is misleading
wagtail-slack.now.sh appears to be dead

Thanks!


Fix logging error

Thank you for contributing to Pipenv!
The issue
What is the thing you want to fix? Is it associated with an issue on GitHub? Please mention it.
Fixes this issue which I've created: closes #6245.
Trying to install a library results in an error.
Always consider opening an issue first to describe your problem, so we can discuss what is the best way to amend it.  Note that if you do not describe the goal of this change or link to a related issue, the maintainers may close the PR without further review.
If your pull request makes a non-insignificant change to Pipenv, such as the user interface or intended functionality, please file a PEEP.
https://github.com/pypa/pipenv/blob/master/peeps/PEEP-000.md

The fix
How does this pull request fix your problem? Did you consider any alternatives? Why is this the best solution, in your opinion?
There are just some missing brackets which look like they were removed during a refactor.
The checklist

[x] Associated issue
[ ] A news fragment in the news/ directory to describe this fix with the extension .bugfix.rst, .feature.rst, .behavior.rst, .doc.rst. .vendor.rst. or .trivial.rst (this will appear in the release changelog). Use semantic line breaks and name the file after the issue number or the PR #.

@oz123 is this fix upstreamed? I just installed pipenv and the bug was still there.

Merged but not released. Have you installed from the main branch?

no, from pip I think.


function sample for authorizer for IDCS in python

Hi Team can we create function sample for authorizer but in python format?

Hi @heljhumenad ... Acknowledging your request.
cc: @WunderlichRd

Thank you @sachin-pikle  I'm learning right now the serveless function for oracle native need some good references though :)


skip any recieved done_rows amounts and avoid creating extra resultsets

closes #66

I would remove this part of statement in the test: "set nocount on". Test should pass without it if the fix is correct.

I would remove this part of statement in the test: "set nocount on". Test should pass without it if the fix is correct.

Nevermind, I jsut didn't notice there is a test with "nocount off" too...

In which version and when will it be available, will this be included in? :D

0.4.8 is out and should have this fix:
https://github.com/prisma/tiberius/blob/master/CHANGELOG.md


--surfreg in FastSurfer VINN 2.0.0

Question/Support Request

Hi I just ran the new version of the pipeline and the results and the processing times are outstanding. However I noticed two minor issues.

can we do manual edits to the mask and then run your spectral spherical projection, just for minor edits. Only in those areas where the meninges are really close to the grey matter.
The sphere.reg file was not generated even when I did not used the --no_surfreg flag. is this a normal behaviour? can I generate this file for further registrations? in the readme it says that: "For surface group analysis, sphere.reg is also generated automatically by default. "

Screenshots

...

Environment

FastSurfer Version: Docker Build CPU FastSurfer Image v2.0.0
FreeSurfer Version: Docker Build CPU FastSurfer Image v2.0.0
OS: Ubuntu mate 1.20.1
CPU: 15

Execution

build
cd ..
docker build --rm=true -t my_fastsurfer:cpu -f ./Docker/Dockerfile_CPU .

run
docker run -v ${T1w}:/data/${idBIDS}_t1w.nii.gz \
        -v "${SUBJECTS_DIR}":/output \
        -v "${fs_licence}":/fs_license/license.txt \
        --user "$(id -u)" --rm my_fastsurfer:cpu \
        --t1 /data/${idBIDS}_t1w.nii.gz \
        --sid ${idBIDS} --sd /output \
        --fs_license /fs_license/license.txt \
        --parallel --threads ${threads} --device cpu \
        --no_fs_T1 --vox_size min

Thanks for trying the new version. These are actually 2 separate issues. If the solution to 1. below works, we can keep it like this, if not, please open a separate issue for the editing where we can discuss this further.

in version 2.0 you should be able to do this by running the segmentation and recon-surf steps separately. The segmentation will create  its output under /home/user/my_fastsurfeer_analysis/subjectX/mri/aparc.DKTatlas+aseg.deep.mgz, mask.mgz, and orig.mgz . You can then edit mask.mgz and run the surface stage without specifying inputs. It should not re-create those files but use them for the surface creation.

missing sphere.reg could be a bug and needs replication. Due to the OHBM deadline tonight, this will probably be looked at next week.

Hey,
I ran the command with the docker from our docker hub (deepmi/fastsurfer:cpu-v2.0.0) and it automatically produced the ?h.sphere.reg:
docker run -v $PWD:/data -v /fs732:/fs732 \                  --user "$(id -u)" --rm deepmi/fastsurfer:cpu-v2.0.0 \                  --t1 /data/IXI/IXI002-0828-T1.nii.gz \                  --sid IXI002-0828 --sd /data/surfreg \                  --fs_license /fs732/.license \                  --parallel --device cpu \                  --no_fs_T1 --vox_size min
Can you share your log-file (recon-surf.log)?

Hi, I build the container from the stable branch and git pull it right before building it. And I followed the example from the README file inside  /docker.
However,  I notice a difference now, your docker container was downloaded from the Docker hub deepmi/fastsurfer:cpu-v2.0.0 and mine build from my local github which created this image my_fastsurfer:cpu.
Do you think this may lead to the different results? I am now downloading this deepmi/fastsurfer:cpu-v2.0.0 and I'll re-run the tests.

The name should not matter. Basically the name that you put after the -t flag in the docker build command sets the name for the image. The content is identical (if the same Dockerfile is used, which should be the case here).
But it is strange, that stable does not include the correct changes for you. Maybe you can git pull origin stable again to make sure all changes are included? Or you can check if line 28 in recon-surf is equal to v2.0.0 (fssurfreg=1           # run FS surface registration to fsaverage, if 0 omit this step; https://github.com/Deep-MI/FastSurfer/blob/stable/recon_surf/recon-surf.sh#L28)

Any progress on this? @rcruces did it work with our distributed docker images? Also if you build your own images, make sure you checkout our stable branch into a fresh directory. That way you avoid any git related things (like checking from your own fork, rather than from ours, or merging accidentally with a local branch with other changes etc).

For the mask edit, we also put some info on how to run the two stages separately here:
https://github.com/Deep-MI/FastSurfer/blob/dev/recon_surf/README.md#manual-edits
Other edits are currently not easily possible.
Closing this issue for now. Requests for manual editing options should be started as a new issue.
And the missing surfreg could be resolved by using our released containers and by building fresh ones from stable.


Unique row IDs for repeating groups

Description
The main headline here is that app-frontend now uses (and requires) the altinnRowId property in repeating group rows, as generated automatically by either frontend (when rows are added by the user) or backend (when rows are added by custom code, prefill, etc). These row IDs are now used when generating a JsonPatch so that we're 100% sure which row needed a change (regardless of the row index).
This also includes:

Using the row UUID in internal state for which group row we're editing. This means the backend can sort the rows, and that app-frontend will keep the correct row open for editing after.
Some code cleanup, moving component definitions out from other components (ExtraRows in RepeatingGroupTable)
A few improvements to ttd/frontend-test

Adding a new page for 'Pets' that can be used to test large repeating groups (with lots of rows), along with sorting rows. The page is intentionally ugly to point out some styling problems in our apps. :wink:
Using a gateway that makes it possible to jump straight to a given task, skipping the previous ones. This should make it much easier to use ttd/frontend-test when working locally (I'm sick of filling out the changename form just to get to the repeating groups).

Adding support for CustomButton responses returning validation issues (which it now does when the data model is updated).
Showing a custom error page when the backend did not give us row IDs. This page should nudge app developers into reading the docs and running the migration tool (altinn-studio upgrade ...)
Adding a cy.waitUntilSaved() function that will wait until the next save request has finished, and the page is no longer marked as having unsaved changes.
Alerting the user if you try to close the page (or navigate away) when there are unsaved changes in the form.
Updating tests, especially shared expression tests
Removing previous workarounds in the frontend-test/group.ts Cypress test, as JsonPatch should now work fully with repeating groups.
Minimum version of nuget packages are now v8.0.0-rc1

Related Issue(s)

closes #1750
#1812

Verification/QA

Manual functionality testing

[x] I have tested these changes manually
[ ] Creator of the original issue (or service owner) has been contacted for manual testing (or will be contacted when released in alpha)
[ ] No testing done/necessary

Automated tests

[x] Unit test(s) have been added/updated
[x] Cypress E2E test(s) have been added/updated
[ ] No automatic tests are needed here (no functional changes/additions)
[ ] I want someone to help me make some tests

UU/WCAG (follow these guidelines until we have our own)

[ ] I have tested with a screen reader/keyboard navigation/automated wcag validator
[x] No testing done/necessary (no DOM/visual changes)
[ ] I want someone to help me perform accessibility testing

User documentation @ altinn-studio-docs

[ ] Has been added/updated

[x] No functionality has been changed/added, so no documentation is needed
[ ] I will do that later/have created an issue

Support in Altinn Studio

[ ] Issue(s) created for support in Studio

https://github.com/Altinn/altinn-studio/issues/12277

[ ] This change/feature does not require any changes to Altinn Studio

Sprint board

[x] The original issue (or this PR itself) has been added to the Team Apps project and to the current sprint board
[ ] I don't have permissions to do that, please help me out

Labels

[x] I have added a kind/* label to this PR for proper release notes grouping
[ ] I don't have permissions to add labels, please help me out

Some tests are failing now, and it seems to be a problem with v8-rc1. Thread here: https://altinndevops.slack.com/archives/C045EB3JA9X/p1709039345902389

I skipped the failing tests. They should be possible to run again after this is merged, released, and ttd/frontend-test is deployed with the new version:

https://github.com/Altinn/app-lib-dotnet/pull/462

This should be ready for review now. :+1:


Fix quorum related issues and a race condition

This commit is cherry-picked from https://github.com/openebs/jiva/commit/59b33519b68a85ee573a094d1056367d4bc87130

Fix quorum related issues and a race condition
1)Reply with success when IOs are written to atleast 51% of replicas.
Add delays in through signals in read/write code to simulate the above case.
2)Attempt replica close before trying to connect to controller.
Replica can only connect to controller in closed state.
3)Add delays through signals in replica addition path to simulate race condition.
Add test case for 3 replicas and reduce R/W timeout and retry values

Signed-off-by: Utkarsh Mani Tripathi<EMAIL_ADDRESS>

This cherry-pick is not required, so closing the PR


docker swarm mode nginx not get client realip

in docker swarm mode ,run nginx container get the client ip is not real ip
for example
docker service ps nginx

everything get <IP_ADDRESS>, no client ip
but no swarm mode is ok

That is how swarm mode works. You'll need to file an issue with them to see if they have a way to forward the client IP in something like X-Forwarded-For so that Nginx could pickup the proper ip when using endpoint mode of vip (which is the default).


error handling with ORA codes

Hi,
I would like to do error handling by reacting to the returned error codes from Oracle.
I have created a short demo project here, in which several attempts are made to create a table. From the second call I would like to handle the code ORA00955, but I don't know how to get the code with the lib.
Could you please take a look at the project or possibly extend the documentation?
https://github.com/volker-raschek/golang-purego-oracle
Here are the steps
git clone https://github.com/volker-raschek/golang-purego-oracle
cd golang-purego-oracle
make start-oracle-xe
sleep 180 # wait some seconds until database is ready
make run

Volker

actual I return the error as a message come from the server. and still I didn't make a special error class for the project
you can take the error text and split it to get the code.
As there is still missing important feature I concentrate on adding it and didn't start making specific error class
the error message is like this
ORA-01403 no data found
also the error code is return in the network structure and it is present in class SummaryObject.RetCode under network namespace

I add OracleError class that carry error message from the server
the class contain ErrCode and ErrMsg


Update translations

I just released 3.0beta7, the last beta for Spyder 3.0. So now it has come the time to update our translations for the final 3.0 release (at last!!).
This is a call to all our translators to update their respective translations. Please create separate pull requests for each language.

French

@Nodd
@SylvainCorlay

Japanese

@sohnishi

Portuguese (Brazilian)

@manipuladordedados

Russian

@burrbull
@Qlogin

Spanish

@ccordoba12
@goanpeca

Important note: I can only wait for you until September 21 for your updates. I need to release Spyder 3.0 that day. I hope you can make it!
Thanks all for your help and cooperation ;-)

I am wondering whether a German translation (down the line, I'm not volunteering for next week) would have a sufficient impact on broadening the userbase to make it worth the effort. Do you have any numbers regarding anaconda installations and spyder usage by country?

Hi @wtheis I dont think we have enough stats for that but any new translation definitely has a positive impact on the spyder user base :-)

Nop, sorry. But there's no rush to send a new translation, this is just for current translations.
You can send a German translation whenever you like, and I'll release a minor version of Spyder (e.g. 3.0.1) to include it :-)

Hello, @ccordoba12! There are some mismatches in translation template. May be you will update 'spyder.pot' file to prevent its changes in each PR.

Will do for beta8, and will give you some extra days to finish the translations :-) Sorry for not noticing before, silly me!

@Qlogin and others, I just updated our translation files. I really don't have time to release a new beta, so please update your translations based on the files on master.
Thanks :-)

And you have until Friday to update your translations :-) I'll release 3.0 final on Saturday.

@Qlogin added some missing strings to be translated, so you need to rebase or merge your efforts with master to update your translations.
Thanks :-)

Ok, I'm moving this to 3.0.1 (to be released in two or three weeks) to give more time to our translators.
@manipuladordedados, the Portuguese translation is the most outdated one. If you don't update it by 3.0.1, I'm going to disable it. This doesn't mean a removal, just disabling it from our Preferences until you update it.
@PierreRaybaut, could you help me with the French translation? It shouldn't take that much time because you update it some months ago :-)
@sohnishi, I hope you have time to update the Japanese translation by then :-)
@Qlogin, I fixed some errors which were preventing us to get all translation strings. So there are more strings to translate, but unfortunately (as said before) I have to release Spyder today, so please update the Russian translation for 3.0.1.

@ccordoba12, Sorry for the delay.
Check the pull requests.

Thanks a lot for the quick response. Your update will be part of 3.0 (not 3.0.1 :-)

Hey guys, sorry I missed this in 3.0.1. But we need your help now, for 3.0.2 :-)
This is (most probably) the version that's going to be added to the next version of Ubuntu (17.04), so we'd like you to help us to maintain very good translations for the languages you speak.
Important recommendation: Please do your pull requests against the 3.x branch, not master. That's because 3.x is our maintenance branch now, and we create new releases from it instead of from master.
You have until Friday to send us your corresponding pull requests. Thanks a lot for your help!

@Qlogin and @manipuladordedados, please don't forget to send us your pull requests. You're the most active translators :-)

@ccordoba12 done

Ok, last call for French translations: @stonebig, could you help us with updating them? I'd need a PR from you today to release 3.0.2, else I'll disable them because they are too outdated.

Most translations have been updated for 3.0.2. Thanks a lot for your help!!


Instantiate a Generic Parameter

For example
class GenericClass<T>
{
     T getInstance()
     {
           //This does not work
           return new T();
     }
}

You should be able to instantiate a generic, maybe extra syntax should be added to generics for example.
class GenericClass<T with new()>

This would mean the generic type should have an empty constructor.

Duplicate of issue 3633.


grip_touch controller signal not happening on Quest2 and Pico4

The XRPositionalTracker.button_pressed signal never emits a "grip_touch" touch event on the Quest2 or Pico4.
It's detecting all the other button touches in clicks in my benchmark example.
See the picture where I'm holding the trigger and grip buttons fully in (hence the input_float_changed is registering 1 on both of them).

My working demo is at:
https://github.com/goatchurchprime/godot-demo-projects/tree/openxr_seeallfeatures/xr/OpenXRHandTrackingDemo

The same code is giving me xr_interface.xr_play_area_mode = 0 and xr_interface.get_play_area() = [ ].  I'll file this as another ticket if it helps.  (I haven't checked it on the Rift yet)

Ok, so this is not a bug, it's supposed to work this way.
Remember the default action map is just there to mimic the behaviour we have in Godot 3. It presents each input on the controller as an input for Godot, 1:1.
That means that an action will only work, if the action is mapped to an input.  And action is only mapped to an input in the default map, if the input exists on the controller.
We can't magically make features of a controller appear that aren't there.
On an index we see all grip related actions mapped:

While on the Quest no such input:

This is the danger of mapping inputs 1:1 and really how the action map isn't supposed to be used. You should create an action map for your game with the actions your game needs, and then map the available inputs accordingly.
Even then it is possible you use a controller that doesn't have the required inputs for your game of instance:

WMR controllers or original Vive wands that don't have A/B/X/Y buttons,
WMR controllers and index controllers have both touchpad and joystick, while others only have one or the other,
So if you're game relies on one of those to function, you're in trouble.

For XR Tools this for instance means that we should create our own XR Tools action map. We didn't because for a long time we wanted to keep compatibility between Godot 3 and Godot 4, and now it's a major breaking change. But officially we should have actions for each feature, and map them to the required buttons in the action map, instead of selecting the action from a dropdown on the function.
So say for our jump function, instead of having:

We should remove the "Jump Button Action" and instead have an action called "Jump" in the action map, and have that action mapped to the best input on each controller. Right now jump is not possible on for instance a WMR controller because there is no A/X button on those devices, but when using actions properly, we could map the Jump action to one of the touchpad corners (it can function as a D-pad, though we need to implement an extension for that).
To be exact, XR Tools completely breaks right now if a user decides to make their own action map and strays away from the actions XR Tools assumes are there.

Actually looking through all the profiles, I don't think any controllers even support touch on the grip button, only grip force which is only available on Index.

Removing the dud action here: https://github.com/godotengine/godot/pull/85048

Closing this issue as resolved through the above PR.


Add permissions to slashcomamnds

Slash commands have their own permissions. Allowing some commands to only certain users/roles might become crucial. Implementing an elegant way of defining permission will be needed in the future.

Closed with #215

Closed in #215


Maki version mismatch

Bug: T302861

Great work, thanks!

Note that I didn't manage to get a new package-lock.json compiled. I believe this must be done before we tag this release.


Move Instagram feed further up my Shopify homepage

Hi there! Just wondering how I get the Instafeed to appear further up the page instead of at the bottom above the footer, which is where it automatically got added when activating Instafeed.

@Cazmic This isn't an Instafeed issue, you should contact the developer of your theme.

No problem - already done soon after I emailed you.
Thanks for your reply :)
Caz Hoad
CAZMIC DESIGN
+61 412 745 280
http://www.cazmicdesign.com.au
On 23 Mar 2018, at 9:43 am, Liam Merlyn <notifications<EMAIL_ADDRESS>wrote:
@Cazmichttps://github.com/cazmic This isn't an Instafeed issue, you should contact the developer of your theme.

You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/stevenschobert/instafeed.js/issues/565#issuecomment-375481610, or mute the threadhttps://github.com/notifications/unsubscribe-auth/Aj1pex0b-ZZE2wjFHDsYZVNZjkr2joT_ks5thCkjgaJpZM4SxrLe.


Add a check job in the CI+CD workflow

Problem

The CI+CD workflow should invoke Django checks.
$ python manage.py check

Description

The Django check command can catch a number of problems in the codebase and act as an effective way to identify possible bugs.
Alternatives

This is just an addition, we already have some unit and integration tests to catch most major issues.
Implementation

[ ] 🙋 I would be interested in implementing this feature.

TIL! This is would be a great addition!


Render never runs within synchronous loop

Since the internal render function is only called on the tick of a setInterval it will never get rendered if looping through a set of synchronous actions.
A better approach would be to to keep track of when the bar was rendered last, and only attempt to re-render when update is called. That would give the same optimization as re-rendering when there is a change but also works with synchronous code.

Dear Jimmy,
do you have an example for such blocking synchronous code ? i thought the setInterval callback should by executed by the v8 engine when the update() function is called, because it's non-blocking in this moment ?!

Below is a small example, based on the usage details from your readme.
var _progress = require('cli-progress');

// create a new progress bar instance
var bar1 = new _progress.Bar();

// Some set of data
var data = [];

// start the progress bar with a total value of 200 and start value of 0
bar1.start(data.length, 0);

for (var i = 0; i < data.length; i++)
{
  // Some operation that uses the data (synchronously)
  useData(data[i]);

  // update the current value in your application..
  bar1.update(1);
}

// stop the progress bar
bar1.stop();

No updates will happen until after the data is iterated. Take a look at this post by John Resig for nitty gritty, but the basic idea is that synchronous operations (iterating sequentially) does not allow a timeout or interval to execute a callback.

Dear Jimmy,
thanks for your hint. the whole for-loop is executed as "synchronous" block, therefore the interval queue is not triggered. i've created a simple testcase to reproduce the issue:
var _progress = require('./main');
var limit = 2000;

// long running, recursive function (blocking operation)
function fibonacci(n) {
    if (n < 2) {
        return 1;
    }else {
        return fibonacci(n - 2) + fibonacci(n - 1);
    }
}

// create new progress bar using default values
var b1 = new _progress.Bar();
b1.start(limit, 0);

for (var i = 0; i < limit; i++) {
    var r = fibonacci(i);
    console.log("Step ", i, " #", r);
    b1.update(i);
}

b1.stop();

i will fix this problem asap.

I actually came up with a nice workaround, but it could only be done with a different progress bar library. As long as you have a stop or done command, you actually don't need a timer at all. Here is a simplified version of how I made it work:
const Bar = require('progress');

const TICK_DELAY = 600; // Can be calculated based on FPS (Should just be 1000/fps).

class ProgressBar {
    constructor(total) {
        // Create a simple progress bar, that
        this._bar = new Bar(' :bar :percent | ETA: :etas'), {
            total,
            complete: ' ',
            incomplete: '█',
            width: 40
        });

        // Let's keep track of the total ticks that must be applied.
        this.tickAmount = 0;
        // Let's also store the millisecond timestamp from the last call.
        this.prevTick = 0;
    }

    tick(amount) {
        // Auto-increment by 1 if not specified...
        if(typeof amount !== 'number'){
            amount = 1;
        }

        // Add the amount to the total ticks that must be applied...
        this.tickAmount += amount;

        // Get the time difference, in milliseconds...
        let now = Date.now();
        let diff = now - this.prevTick;

        // If the time since the last successful tick is more than the threshold...
        if(diff > TICK_DELAY) {
            // Tick the full expected amount and reset the variables...
            this._bar.tick(this.tickAmount);
            this.prevTick = now;
            this.tickAmount = 0;
        }
    }

    complete(){
        // If there is any unapplied ticks, apply them now.
        if(this.tickAmount){
            this._bar.tick(this.tickAmount);
        }
    }
}

module.exports = ProgressBar;
```

this solution might work for long running operation but causes issues related to the ETA calculation.
in case the tick() function is called in the following way:
throttle time: 25ms

Elapsed Time
Update-Value
Rendering Status

10ms
5
updated

20ms
20
rejected

50ms
30
updated

80ms
50
updated

100ms
290
rejected

4s
408
updated

the update on 100ms is rejected and no update will happen until the next call. to avoid such problem, you have to run a timer in the background which will trigger the re-rendering (maybe on 2*throttle_time ?)
i know, this example is very theoretical, but i think a new version should also work under such conditions ?

I've added synchronous operation support to the current "v1.1.0" release.


Update README.md

@PoUpgini quick example needed, why would I use that custom loss. Best one - poisson, for targets where target var is the Poisson process, like count of smth in a period of time


Revisit build process, build artifacts

Currently, as a vestige of earlier prototype days by a single developer, the built files are created and committed manually, on an as-needed basis and before publishing. It's a good time to revisit what should be done
Some potential options:

Build artifacts should be committed on every commit

:heavy_check_mark: can use tip of tree build artifact from git, have a built file for every commit
:x: messy git history
will need to set up a github action to build this via CI or something

Build artifacts should be committed only on publish

:heavy_check_mark: cleaner git history, can easily grab latest published/working build from git
:x: can't get tip of tree build artifacts
Build step can be a part of the publish

Build artifacts should not be committed

:heavy_check_mark: Cleanest history
:x: Requires npm or build step to use
Can create a build during publish as part of an npm package that never hits git history.

My preference is for committing builds only on publish, or not committing them at all -- I don't know how much value it is to have tip-of-tree built artifacts.
Additionally, what kind of output files should be supported? I wouldn't mind only a ES module export, and leave minification up to consumers, unless we want to get in the business of hosting (via unpkg or other CDN solutions) a canonical polyfill URL.

My $0.02 is that I don't think a single canonical URL is useful;  I would never include that in my web app because when it gets updated, my web page may break.  I would always include a known version. Canonical URLs that are versioned would be useful, though.
Including a minified or at least packaged file seems useful;  tossing a webxr polyfill at the top of your html file seems like a normal pattern, doesn't it?
Aren't there three models:  ES6 modules (I have all code on my site), a collection of packaged js files (I include three.js/babylon.js/aframe.js/webxr-polyfill.js from my site, at the top of my html), or npm (I package everything in a blob).  The former and later would just pull from npm, I guess, but the middle seems to need a file.
I personally am moving to ES6, but not everyone will.
I agree that we don't need tip of tree;  only devs would care, and then can just use npm or build right?
Alternatively, it would be reasonable to move to a two branch system:  develop branch for ongoing work, where we build on every commit (which also serves the purpose of making it easy to test), and then do "releases" by moving to master and adding a tag.


Feat: Show private IP and port in installation script for easier Raspberry PI installation.

Changes

Feat: Show private server IP and port URL in the installer so the user can choose how to open the Coolify dashboard. Also good if the server does not have a public IP.

Issues

fix #2850

Thank you for the PR!


Disable downloads if a node is offline or doesn't not have a valid SSL certificate

Is your feature request related to a problem? Please describe

If the node doesn’t have a valid SSL certificate:

Without the HTTPS patch: Chrome will block it with the “Mixed Content” error and nothing appears
With the HTTPS patch: Chrome returns a  “Privacy error” page

Regardless, if a node’s certificate expires, we need a way for the UI to disable downloads to avoid it breaking with a “Privacy error”.
Describe the solution you'd like

A function has to detect if the dataset’s node has a valid SSL certificate to enable file downloads. Otherwise, it is disabled.
Add prompt about behavior for downloads and behavior variation.
Add button for user to copy and paste into another tab (optional, for HTTP)
Describe alternatives you've considered

Additional context

My opinion is this is not high priority.  Broken download links and SSL warning pages is not desireable, but I recommend bumping down the priority to medium or low.
Before we can start disabling links, need to ensure that the API providing the info has a very low false-positive rate.  I am yet to see that with the Prometheus black box exporter that provides the info now.

We may want to revisit converting http to https links as many sites may have trouble with the former.  The SSL errors should be reported and can help us police sites that don't maintain their certs.


QoL-Support albums as singles

Requested Feature
If an album is linked and it is a single, fetch the metadata of the song instead of failing with "Unsupported URL / URI"
Possible implementation
Check if there is only one song in the album, then fetch metadata from that one song instead of rejecting

I've just tested this and it works? I suppose you're using a spotify link? You could just copy the song instead if the album only has 1 song in it. The song also has the album meta tag ^.^

Need example links.
Open a new bug if this is an issue. Not a feature reuqest


Use libucontext

The POSIX context switching functions are deprecated, and on some platforms (like in Apple Silicon and Raspberry Pi) are not functional.
Add a dependency on the libucontext library.

Fixed on 2023-06-13.


IE9 javascript error

In IE9, I get the following error when choosing a file from the file explorer:
Unable to get value of the property 'length': object is null or undefined
angular-file-upload.js?body=1, line 210 character 29
However, everything work fine in IE10 and chrome. I haven't tested other browsers.
Also, drag and drop doesn't work and FileAPI.flash.swf + FileAPI.min.js are never loaded by the browser, even if they are in the same directory as angular-file-upload.js file.
Any idea how to solve this?

You are probably in compatibility mode or have another library that defines FormData object for IE9.

I had the same issue when using angular-file-upload.js. Switching to angular-file-upload-all.js solved the issue for me.

The problem was on my side. Path to FileAPI.min.js / FileAPI.flash.swf was wrong.


Send TargetType as a parameter in output in NetworkLoggerPlugin

NetworkLoggerPlugin currently has an output closure to print logs. I think the TargetType should be added in the output.
Reason for this:

Able to print logs according to TargetType. I would like to print logs for only particular API targets.

     NetworkLoggerPlugin.reversedPrint
...
    }

    public func willSend(_ request: RequestType, target: TargetType) {
        if let request = request as? CustomDebugStringConvertible, cURL {
            output(target, separator, terminator, request.debugDescription)
            return
        }
        outputItems(logNetworkRequest(request.request as URLRequest?), target: target)
    }

    public func didReceive(_ result: Result<Moya.Response, MoyaError>, target: TargetType) {
        if case .success(let response) = result {
            outputItems(logNetworkResponse(response.response, data: response.data, target: target), target: target)
        } else {
            outputItems(logNetworkResponse(nil, data: nil, target: target), target: target)
        }
    }

    fileprivate func outputItems(_ items: [String], target: TargetType ) {
        if isVerbose {
            items.forEach { output(target, separator, terminator, $0) }
        } else {
            output(target, separator, terminator, items)
        }
    }
}

Hey @hasan-monument. Apologies for the late response. This proposal seems great, the only question is if we can do that in a safely manner and for all plugin methods. Would you like to explore this idea further and come up with a PR maybe?


Set default file format to parquet

resolves #363
Description
We already set this default here, but it's clearly not flowing through properly, because AdapterConfig doesn't really seem to be working in dbt-core (https://github.com/dbt-labs/dbt-core/issues/5236):
https://github.com/dbt-labs/dbt-spark/blob/8744cf1faa0b57fe9e797a32a109ba4e7a056e76/dbt/adapters/spark/impl.py#L38
Let's set the default explicitly in the macro, so that if the user hasn't provided a file_format, dbt-spark uses Parquet. This prevents weird issues with query rewrite in Databricks SQL warehouses, too (https://github.com/dbt-labs/dbt-spark/issues/236).
Update: If the user is connecting via ODBC (= Databricks), let's use delta as the default file format instead. This helps out Databricks SQL warehouse connections, which can only write Delta-formatted tables.
Checklist

[x] I have signed the CLA
[ ] I have run this code in development and it appears to resolve the stated issue
[ ] This PR includes tests, or tests are not required/relevant for this PR
[ ] I have updated the CHANGELOG.md and added information about my change to the "dbt-spark next" section.

Running into more issues on ODBC cluster + endpoint than I'm able to debug right now. I'm going to close this PR for now; someone else could pick up this work and try to carry it over the finish line.


add tagsText example type

Add tags Text example type

Warnings

:warning:

You should add at least 1 more reviewer to the PR

:warning:

The README has not been updated. Please update the README.

Messages

:book:

Awesome! Thanks for the well-formatted PR!

  Generated by :no_entry_sign: dangerJS


Server up but, validate tab says its not

Started a PROOF server. Got the email saying PROOF server is ready. Tried to validate a workflow but validate tab says my server is not ready. Logging out and re-logging back in solved the issue.

Thanks Sita, i'll see if I can replicate and then ping back here

@sckott one thing that is important is that the PROOF shiny app says when a server is live when a job is started on a gizmo node for the server.  But the server takes a minute to start up before the API is available for communication with the app.
One thing that might be useful is somehow registering both when a server job is kicked off AND when the API is open for business.  This could be an occasional polling of the API until a valid response is obtained (that's what I used to use the function that checks for the cromwell version for - just to see if the API was available).  It could poll until it gets a valid response and THEN tell the user the server is on and send that email.

Also, @sitapriyamoorthi  and @sckott  the above behavior is often not actually solved by "a user doing x" it's actually solved by time passing and the API becoming available to hit with the app.

@vortexing i think thats a good approach. Because that might be true representation of that the server is ready to accept jobs when API is open for business!

A few thoughts

The behavior of the PROOF API isn't always the same as we've all experienced. So this plays a role here probably
We do currently check that the Cromwell API started by the PROOF API is up. The same set of checks are used in the modal that is used to start a server and throughout the app to make sure the API is up.
@sitapriyamoorthi could you share the exact sequence of events that made this happen?

You logged in to the app?
Then went to server tab?
Did you wait for the modal to go away after starting a server?
etc.

Hey @sckott following up here is the series of events:

Logged into app
Started server
Waited for server to start up (aka the details of the file paths etc get loaded up)
Waited for me to get the email
Headed to the validation tab to upload the wdl and json
Hit validate
Said my server wasnt ready
logged out and logged back in
tried the validate step again ... and it worked!

Curious if you could explain what you mean by the behavior of the PROOF API isnt always the same?

Thanks for that Sita. That sequence of events makes me think the cause was this issue of the PROOF infrastructure not always behaving the same - not totally sure, but it smells like that. I haven't been able to reproduce this issue, so hard to say for sure 🤷🏽

Curious if you could explain what you mean by the behavior of the PROOF API isnt always the same?

Since PROOF runs on shared infrastructure I've heard from Dan that sometimes things are slower than normal. If e.g., PROOF was all run on a dedicated server we'd expect behavior to always be the same.

Issue persists


Add Icon

We should add an icon to the repo. How about something in the general direction of:

Maybe a little less wide and with rounded off arrows?

From what I can tell, the repository's icon/logo/thumbnail cannot actually be changed. If you go to https://github.com/KIT-MRT/arbitration_graphs, you will always see the mrt logo in the top left corner.
What can be changed is the social media preview. I.e. the thumbnail that's visible when sending a link to the readme via e.g. a messenger. I do not think that this use case is worth the effort to come up with an icon.
So either we would also utilize the icon elsewhere, i.e. at the top of the readme or we don't and simply close this issue.
@orzechow What do you think?

For now, I don't see us creating an icon in the near future... 😔


KAFKA-15562: ensure commit request manager handles errors correctly

The current commitRequestManager lacks logic handling various failures.  In the patch I addressed the following gap:

Network disconnection should disconnect the coordinator node
Handling and testing various of fatal and retriable errors scenarios
Ensure the time returned in the poll results is Long.MAX or the minimum of all the inflight request's remainingBackoffMs.  Because we need to respect the backoff.

Thanks @philipnee, I left some comments

Hi @lucasbru - Thanks for taking the time to review my PR.  I addressed all but 2 comments:

Mockito: Could you be more specific on how you expect to mock the response object?
Error handling: Essentially all errors in continueHandlePartitionErrors can only happen in the response. I understand there are some redundancy there and can be a bit confusing.  But the response and both throw a hard failures (response = null and throwable = non-null) or a server side error (response = non-null).  That is why it was kept separated.  If you find it unclear - how do I make it more readable?

Changes look good to me. Did I understand david correctly that we want to add more changes to this PR?

Mockito: Could you be more specific on how you expect to mock the response object?

That was just "out of interest", since you are already using mockito to mock things, why not use it for the response object as well. But I realize that the object is a pretty plain data container, so the answer I guess is that it's easier.

Error handling: Essentially all errors in continueHandlePartitionErrors can only happen in the response. I understand there are some redundancy there and can be a bit confusing.  But the response and both throw a hard failures (response = null and throwable = non-null) or a server side error (response = non-null).  That is why it was kept separated.  If you find it unclear - how do I make it more readable?

That's not what I meant. I just found the flow onResponse a bit hard to read, so I suggested structuring it a bit differently. In particular, continueHandlePartitionErrors sounds like that it's determining whether to continue handling errors, but it's actually completing the future inside.
I think the control flow may also lead to logging in a way that may not be intended? For fencing, we get one generic log.error message and one very similar log.info message with some specific details. For topic authorization errors, we get one log.error message for each partition, and then another aggregated log.error message, listing the partitions again. However, if we have 3 unauthorized topic errors and then one other error, we get three generic error messages for the topics, but we do not get the aggregated error message. Is that trying to be consistent with the old consumer?
I imagined something like:
if (error == Errors.NONE) {
    log.debug("OffsetCommit {} for partition {}", offset, tp);
} else if (error == Errors.TOPIC_AUTHORIZATION_FAILED) {
    // Collect all unauthorized topics before failing
    unauthorizedTopics.add(tp.topic());
} else if (error instanceof RetriableException) {
    log.warn("OffsetCommit failed on partition {} at offset {}: {}", tp, offset, error.message());
    handleRetriableError(error);
    retry(responseTime);
    return;
} else {
    log.error("OffsetCommit failed on partition {} at offset {}: {}", tp, offset, error.message());
    handleFatalError(error);
    return;
}

But I'm not super familiar with the code style in the new consumer, and consistency with the rest of the code and the old consumer is also important. So I just wanted to give this to you as an inspiration, but you are probably in the best position to come up with the best way to implement it.

Hi @lucasbru - Thanks for the response. To your first question: Yes, what is left is correcting the error handling at the user API level.  At the time I implemented this, I wrapped all errors in KafkaException (I forgot the reason why), however, we should throw the exact exception instead.  So I will follow up with a second part PR to make this Jira issue completed.
I get what you meant for onResponse and I think your suggestion makes a lot of sense. I'll patch it accordingly.

@lucasbru - Thanks for taking time reviewing this PR. I addressed your previous comment. Let me know if there's anything unclear on this PR.

Test failures seem irrelevant, but retriggering the test as there's a failed build

Test failures unrelated


bug: here some text out of box

It's server side issue ig, locally it's work fine

yeah it seems to be a server side issue

Yep, as @4kshi7 & @Surajsuthar said. It's a server side bug


wrong name of root node

Hey,
you link this page which states that you should name the root node coverage, but when generating the xml file the root node is called testExecutions, which is answered with an error by sonarqube.
Would it be possible to change this according to sonarqubes expectations?
Thanks

I'm sorry... I just realised that I've looked at the wrong part of the page 😓


[REQUEST] ShuffleCipherSuites in Request Block

The title kinda says it all.
Being able to randomize ciphersuites will help bypassing CF and alot more.

You can already do this (on linux only, with recent OpenSSL library)
BLOCK:ConstantList
  value = ["TLS_AES_128_GCM_SHA256", "TLS_CHACHA20_POLY1305_SHA256", "TLS_AES_256_GCM_SHA384", "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256", "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA", "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA", "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA", "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA", "TLS_RSA_WITH_AES_128_GCM_SHA256", "TLS_RSA_WITH_AES_256_GCM_SHA384", "TLS_RSA_WITH_AES_128_CBC_SHA", "TLS_RSA_WITH_AES_256_CBC_SHA", "TLS_RSA_WITH_3DES_EDE_CBC_SHA"]
  => VAR @suites
ENDBLOCK

BLOCK:Shuffle
  list = @suites
ENDBLOCK

BLOCK:HttpRequest
  useCustomCipherSuites = True
  customCipherSuites = @suites
  TYPE:STANDARD
  $""
  "application/x-www-form-urlencoded"
ENDBLOCK


 Thanks.

Not currently, I'd be happy if you can create one.

Got a weird action here, a single selection on a target in the main window or treeview will trigger 3 time of @select bound function

somehow, the event seems to be loaded three times,
Probably, we should register the event once per vuefinder instance.


tilt: 0.33.6 -> 0.33.10

Description of changes
Updates Tilt from version 0.33.6 to 0.33.10.

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[x] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Version 0.33.10 was released in the meantime, so I updated the PR to include that

@Pyxels, thanks for the update, I did a quick smoke test and it works for me :+1:

another pr went through

@Pyxels @anton-dessiatov might be a silly question but how did you all get the updated hash for fetchYarnDeps? I'm trying to test v0.33.12 right now and can't get the build past that point after I update the fetchFromGitHub hash


Does it predict face pose?

I'd like to know if the model also outputs face pose angles, or is it only detecting the area where it finds a face?
Thank you!

No

Ok thank you.


Add OnCreateSiteInstanceForNavigation

This is attempt 2 because commit's went weird

@zcbenz These builds are all green and with my changes in https://github.com/electron/electron/pull/10221 the tests that were failing are green as well.  Can we merge this to get the ball rolling on getting this into Electron 😄

This looks good to me. /cc @alexeykuzmin This PR is going to affect chrome 59 and 61 updates.


Fix syntax error

There should be a space between the sharp and the title in GitHub Markdown syntax

Q
A

Bug fix?
yes

New sample?
no

Related issues?
N/A

What's in this Pull Request?
Add a space between the sharp and title, such that GitHub can render the title correctly.

Thanks for catching this!


Question about inference process

In the paper, there is a sentence which talk about limitation of manual prompt, "Furthermore, the manual design assumes knowing the task in advance, which limits its applicability to lifelong learning where the next task to learn may not be known in advance.".
However, the model takes a task-specific generation token as input, which can give information about the next task to the model. Is it also kind of above limitation?
Also, apart from above question, when the model inferences with respect to certain domain of certain task, does the model take the task-specific generation token corresponding to the domain as input?

Hi, sorry for the late reply. The task-specific generation token is for current task, not for next task. You can not know it in advance!
For your next question, No. All domains of the same type share the same generation token.


VM Networking improvements

With contributions from @tomribbens

[ ] 1. IP ranges used by VMs should be updated because they are not following RFC1918

[ ] 2. Network configuration commands are changed from iptables to nft

[ ] Port forwarding - from a public IPv4 port on the host to a port on the guest
Dedicated issue: https://github.com/aleph-im/aleph-vm/issues/148
Related: https://github.com/aleph-im/aleph-message/pull/30

[ ] IPv6 access to every virtual machine
Dedicated issue: https://github.com/aleph-im/aleph-vm/issues/221

[ ] A Wireguard VPN between virtual machines from the same user, across hosts.

Notes

Using the same IPv4 range inside VMs on all tap networks might be possible.
Using the same Tap interface for all VMs of a user’s cluster would be ideal.

Include (and improve) the cleanup of network configuration on VM stop and on supervisor crash/start.

Solving this issue as well would be handy: https://github.com/aleph-im/aleph-vm/issues/18

I've been looking at point 1, on making the ip addresses RFC1918 compliant. The easy way obviously would be to limit the vm_id generated to be limited to 4096  (assigning /24 networks out of <IP_ADDRESS>/12 gives 12 bits to work with) and then modifying the guest_ip and host_ip functions in firecracker/microvm.py to f"172.{16 + self.vm_id // 256}.{self.vm_id % 256}.1".
There are 2 issues with that:

This puts a hard limit on amount of VMs at 4096, which I understand is bordering on potentially not enough
This is not flexible at all if in the future a change to another address pool is wanted

I would propose as a solution to make two new settings, VM_ADDRESS_POOL and VM_NETWORK_SIZE. This would make everything more flexible on a host per host basis.
However, this would mean passing those settings to the MicroVM class, which doesn't seem to be cleanest either. Therefor I would propose to move the calculation of the ip addresses out of firecracker/microvm.py and into functions called by the create_a_vm() function. This seems cleaner as the VM logic itself has no business determining the IP addresses, but should be a function in the supervisor anyway.
Additionally, this would make it easier in the future when looking at using a single tap interface per user, as at that point more knowledge about other VMs is required to determine which IP address to hand out, which is clearly a supervisor task.

Did some research into sharing the tap interface between multiple hosts. As I understand it currently, this would require the tap interface to be set with the option multi_queue, but this currently is not supported by Firecracker:
https://github.com/firecracker-microvm/firecracker/issues/750


Meta-Issue: Usage Stats

Title (Goal)
Gather usage stats on binary resources via Piwik

Primary Actor
Repository admin

Scope
Architecture

Level
High

Story
As a repository admin, I want to be able to gather usage/analytics data via Piwik. There is already a Piwik integration module for D8; this use case is more about being able to track usage of binary resources associated with an Islandora node.

Incidentally, there is a Docker image for a Piwik server, which will lower the barrier to setting up a Piwik instance.

Additional info: we discussed this at Islandoracon, and Piwik was specifically chosen because its client-side (no caching issues) and conforms to European privacy laws which is of interest to an international projects.

@mjordan @bryjbrown FYI, formerly known as PIWIK, now named Matomo!

Groan. Their rationale for changing the name is that it "will allow users to take a fresh look at what we’ve become today and acknowledging all of the community’s hard work over the past 10 years." Srsly?
Awesome platform though.

True. Name changes&market-needs. But: I deployed today the latest version of Matomo over my old 2.7 PIWIK (full of data) and migration was pain-less + UX is super nice and fast. So, I can't complain really. Also: opensource and naming stuff: maybe someone needs to make a conference to discuss this for the future, so needed 🙄

@DiegoPino As a PiWik/Matomo user, how flexible is it in terms of sending custom variables? For instance, could you configure it to send parent collection or certain RDF values to it so that you could facet out certain data or create reports?
A good example of this would be creating a report of views/downloads for ETDs.

@bryjbrown totally flexible. I made some slight modifications to 7.x Piwik module/js to send custom dimensions instead. So using that we manage a segmentation based "collection owners" (we have many), each collection owner has many islandora collections and Matomo/Piwik reports two custom dimensions directly: Islandora Object (separating drupal page and title from what an object is) and Collection membership. This allows to actually have many PIWIK/Matomo users that see only their own stats but also generate separate reports. Dimensions are cool. https://matomo.org/docs/custom-dimensions/

Notes from 2/7/2018 CLAW call:

Setting up standardized custom variables for Islandora objects could correlate with standardized reports on the Piwik server. Pre-made out of the box reports could go a long way towards conveying the value of the repository to external stakeholders like authors/department heads/etc in an IR setting.
Setting up a Piwik API that takes an Islandora object ID and returns some data about its usage could allow you to display stats from inside a template, similar to what some are doing with Islandora Usage Stats Callbacks in D7.
Setting up user accounts on the Piwik server would be a quick way to allow certain people to see visualizations of relevant reports. You could also set up custom APIs to give that report data back to Drupal in some format (JSON?) and visualize it in Drupal as special user dashboards.

Additional info: Implementing Piwik in CLAW would also go a long way towards satisfying behavior #10 in COAR's Next Generation Repositories repot: http://ngr.coar-repositories.org/behaviour/exposing-standardized-usage-metrics/
Piwik would be harvesting the usage stats in a standardized can consistent way, and would allow you set up an API that a third party could query to aggregate views/downloads.

@bryjbrown Little late returning to this, but I can def make a COAR NGR label and slap it on the issues you've made.

Converting this use case to a meta-issue to track usage stats development.  Please refer to this issue in any subsequent issues to link them.
At this point in time, it looks like using https://www.drupal.org/project/matomo and providing an ansible installation of a Matomo server should fulfill the requirements of this use case.

@dannylamb You can pull roles from any git repo as here.  I've created a branch: https://github.com/Natkeeran/claw-playbook/commit/71fb8b54f6526ad5bf14707c6496865520ae6c06.
Sure, we can transfer it.  I'll convert some of the config settings to variables and ping you.  The one thing I am not sure is how to configure the apache in RedHat.

@dannylamb I've made it more configurable.  If you are good, please pull into the Islandora-Devops namespace, and I'll do a PR to the claw-playbook.
Then we can do a PR to install and configure the module in drupal separately.

vagrant up

@Natkeeran I've given you the permissions to transfer the repo using the Danger Zone :tm:
Works as advertised!

We know officially have Ubuntu support for including Matomo during the installation process.  I've added another ticket to add CENTOS support, and after that, it's maybe some configuration and we can consider this accomplished.

So I've been working on Matomo quite a bit over the past few days and here are some outcomes worth sharing:
Configuration & Custom Variables
The Islandora 7 Ansible VM doesn't have much configuration in terms of the Matomo integration. It has the link to the Matomo server, which is the minimum you need for it to work, but I have a few recommendations.
First, the Drupal Matomo module has the ability to not track users of a certain role, and I think most of us would probably agree that administrators shouldn't count. This is pretty easy to set up through the Admin page.
Second, the current build isn't using any custom variables. I did a lot of thinking about what custom variables we should be using since any Matomo plugin that implements a custom API will be dependent on not only having the same custom variables, but having them named the same thing and in the same order. Matomo's custom variable API calls them by number, not by name, so if we are to share custom Matomo report APIs, we need to rally around a standard configuration of the Drupal Matomo module.
With regards to the custom variables we send to Matomo, I think all we really need to send is a unique ID. Node ID is good enough if you are only running one Drupal, but if you are running multiple Drupals that all phone home to the same Matomo server and you want to be able to create reports that pull data from all of them, something truly unique like a UUID or Fedora URI would be better. So long as we have a unique ID transmitted as a custom variable to Matomo, we can set up a custom API in Matomo that returns all the data available about that ID (similar to the API Islandora Usage Stats Callbacks had).
Thats really all we actually need, because if we want to create other more robust types of reports (eg, "Sum all the downloads of ETDs" or "Sum the views of this collection and all its children recursively") we could do that at the Drupal layer with views. Create a view that results in a list of nodes, hit the Matomo API to get data for each node by unique ID, then process that data back Drupal and display the final result. No need to duplicate all that node data in Matomo in custom variables as far as I can see.
Custom Matomo APIs
Matomo's internal API is kind of tricky and, like Drupal, it requires you to think about the data in a specific way. It is pretty flexible though, and it also allows you to very easily create custom plugins that can implement their own outwards-facing APIs. I've done just that with a Matomo module created for the LDbase project that can be modified to meet similar needs in the Islandora community. The module is at https://github.com/ldbase/LDbaseReports, but the real meat of what it is doing is in https://github.com/ldbase/LDbaseReports/blob/master/API.php#L22-L49. If you hit this API with a site ID and a node ID, you get a response like this:
[{
	"nid": "2",
	"page_visits": 5,
	"page_hits": 9,
	"download_visits": 4,
	"download_hits": 6
}]

As far as I can tell, "hits" are individual page loads and "visits" are hits deduplicated by unique visitors within 30 minutes of each other. I can see situations where you might want one or the other, so I provided both. This is an extremely simple module right now, but it could be expanded to provide more data on individual nodes, or even new types of reports if we decide we want them.
I've created a skeleton for an Islandora version at https://github.com/fsulib/IslandoraReports in case the community decides they like this direction. Right now it is the vanilla plugin & API skeleton generated by the Matomo console script, but I can update it to behave like LDbaseReports once the community decides on a unique ID to use as the custom variable.

After reading https://github.com/Islandora-CLAW/CLAW/issues/1073, it seems like the optimal ID that should be sent to Matomo would be one that represents the object conceptually, uncoupled from any specific instance of that object. This would allow you to track an object that is represented in multiple Drupals, whereas the same object in two different Drupals would have two different UUIDs or Node IDs. The field that seems to fit this use case the most in the current iteration of Islandora 8.x (at least to me) is PID, since thats identifying the same conceptual object regardless of being represented in Islandora 7.x or 8.x. From this point of view, it seems like a good candidate for reuse across multiple Islandora 8.x instances because it avoids trying to assign duplicate UUIDs or Fedora URIs to multiple nodes which has a fair chance of causing all sorts of chaos.

The following are usage stat capabilities outlined in the Islandora 8 IR Delta. I don't know where all of these stand in current priorities, but they represent desired functionality for institutional repository usage stats, so I thought they should at least get mentioned here.

track if the page is an Islandora object, also the parent object, the type of object, and the UUID (in order to discern when the same object is loaded via different URL paths)
Extend Matomo module to include custom dimensions as a feature, or have a submodule that adds this feature, or rewrite a new updated version of the Matomo module
Create a custom Matomo API module that runs on the Matomo server that Islandora 8 features can query in an object-centric way (likely by UUID) and get a response with all the relevant information for that object
Allow for the inclusion of a CSV file with legacy usage data that can be dynamically included in the API response
Create a simple “Views and Downloads” block that can query this API and get the aggregate total of all views and downloads for a given Islandora object which can be included as part of that object’s display
Create a standardized set of custom dimensions for a standardized Islandora 8 configuration so that all Islandora 8 sites report similar data on usage to Matomo
View blocks that query Islandora Matomo API and display common data such as most popular objects, most popular files, most popular collections, views over time, summary analytics by current owner/user, collection, topic, etc; can export the analytics


Support SDK modularization on a service basis

A common use case may be to use some boto functions in a client app frozen with pyinstaller, in this case botocore will add 30MB to the install size.
Is there any plan to document how to trim the install size? For example:

Specify which services you are going to be using with setuptools install_extras
Archive old service definitions outside pypi package

As of right now we don't have any official guidance on slimming down the package size to remove unused services.
We wouldn't be able to do the second one as it's possible to specify that you want to use older API versions.
This is effectively a feature request to be able to modularize the the Python SDK in a similar fashion as some of the other SDKs (Java, ruby, etc). Marking as a feature request.

Any updates?
With aws lambda unpacked limit being 250 mb, the 42mb of botocore are really a lot

Yeah any updates on this? I'm not using AWS Lambda so the generic "you don't need to import boto3 for lambda" doesn't fix my issue. The image is increasing size way too much (a 80mb increase is a bit too much) and botocore is responsible for the majority of this increase on size.

 It would be great to have the same for Boto3 which soon will be unusable for AWS Lambdas.

For those looking for a short-term solution.
A customer wrote an article demonstrating how to selectively discard services you are sure you don't use
https://blog.cubieserver.de/2020/building-a-minimal-boto3-lambda-layer/

Any kind of hint or indication that this would be worked on?
 28K	__pycache__
 20K	_distutils_hack
1.3M	boto3
 44K	boto3-1.17.54.dist-info
 55M	botocore
132K	botocore-1.20.54.dist-info
696K	dateutil
4.0K	distutils-precedence.pth
148K	jmespath
 40K	jmespath-0.10.0.dist-info
928K	pkg_resources
 32K	python_dateutil-2.8.1.dist-info
552K	s3transfer
 36K	s3transfer-0.4.0.dist-info
 24K	six-1.15.0.dist-info
 36K	six.py
824K	urllib3
116K	urllib3-1.26.4.dist-info```

Following up as we're not at 64M - I understand this would be a huge undertaking considering how these are created today, so I'm primarily interested in hearing whether the team is considering a modularization in the future.

+1

Copying my comment from https://github.com/boto/botocore/issues/2842#issuecomment-1367484168

On further inspection it looks like about 70MB out of that 72.5MB is just the data/ directory. :exploding_head:
I'm sure there's options here. One could be to split each service into an individual package (e.g. botocore-a-la-carte-s3, etc...). > botocore-a-la-carte would contain core code with an extra per service (e.g. botocore-a-la-carte[s3, cloudfront]) and to > maintain backwards-compatibility botocore would simply be botocore-a-la-carte[all].

From https://github.com/thejcannon/botocore-a-la-carte I've started publishing botocore-a-la-carte with an additional packager per service provided as an extra on the main package.
E.g. botocore-a-la-carte just has the Python code and core resources. botocore-a-la-carte[s3] also install the S3 data, etc...

What about different versions of API, are the old versions are still needed?

$ du -hs *

1,2M    boto3
48K     boto3-1.28.44.dist-info
83M     botocore
220K    botocore-1.31.44.dist-info

83m! where in (not so unusual) venv all ~30 libs takes 133M (so 63% is botocore) with only cryptography 41.0.3 even close to botocore at 14M.
This is so so so much, alpine linux is whole system with required disk size of 130M.
And looking how the size is evolving we will see 100M in 2024 probably :disappointed: :cry:

I used this myself:
          function join_elements {
            local prefix="-path */" separator=" -o "
            local prefixed=("${@/#/$prefix}")
            local rest=("${prefixed[@]:1}")
            local separated=$(printf "%s" "${prefixed[0]}${rest[@]/#/$separator}")
            echo "${separated[@]}"
          }
          function remove_but_latest {
            local latest=$(ls -1 "botocore/data/$1" | sort -r | head -1)
            find "botocore/data/$1" -mindepth 1 -maxdepth 1 -type d -not -path "*/${latest}" -exec rm -vr '{}' +
          }
          function keep_components {
            local component

            find_params=$(join_elements "$@")
            set -o noglob
            find botocore/data -mindepth 1 -maxdepth 1 -type d -not \( $find_params \) -prune -exec rm -vr '{}' +
            set +o noglob
            for component in "$@"; do remove_but_latest "$component"; done
          }
          keep_components cloudformation dynamodb ec2 elbv2 ssm sso sts

It seems to work, not sure if it can cause issues.

Thanks! that's useful overview.
I ended up with dead simple cp + rm only solution. I'm only using s3 and it seems that everything (or most of things) which ends with *json is also used by the common code.
RUN mkdir /tmp/data \
    && cp -r /usr/local/lib/python3.11/site-packages/botocore/data/s3 /tmp/data/ \
    && cp -f /usr/local/lib/python3.11/site-packages/botocore/data/*.json /tmp/data \
    && rm -rf /usr/local/lib/python3.11/site-packages/botocore/data \
    && cp -r /tmp/data /usr/local/lib/python3.11/site-packages/botocore/ \
    && rm -rf /tmp/data

Seems to work just fine, probably until it doesn't.
So having official support would be a nice thing.


x64 build crashes everytime I press F9

...Or if I check the corresponding box in the Options window -> Planetarium.
Logs_Orbiter-x64.zip
While reading the log, I noticed that the new PlanetTexDir entry is not logged.

Can confirm. Happens to me too. But the problem is that it doesnt happen in Release build or the one that we download from github releases. In debug it happens every time.

I can't reproduce this crash in either release or debug builds, so it would help if you could debug it for me. Thanks!

I'm not at my pc now.
I wrote "build" in the title, but now I don't know if it's correct.
What I downloaded and tested are the x86/x64 zip files under "Assets".
The 3 or 4 previous versions I tried never showed this bug.
Il ven 25 nov 2022, 23:41 mschweiger @.***> ha scritto:

I can't reproduce this crash in either release or debug builds, so it
would help if you could debug it for me. Thanks!
—
Reply to this email directly, view it on GitHub
https://github.com/orbitersim/orbiter/issues/311#issuecomment-1327918367,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AUM2X7LXLXXU7PJ4ALW7IOLWKE6BHANCNFSM6AAAAAASKM4F6Y
.
You are receiving this because you authored the thread.Message ID:
@.***>

Ok that's unfortunate. I am still new to the code but I will try to debug to the best of my abilities Dr. Schweiger.

Add line Planetarium = 0 to OrbiterNG.cfg. This will turn off all planetarium elements.
Check that Ctrl-F9 no longer crashes
If that works, open the planetarium configuration page (F6 | Planetarium) and turn on elements one by one until the crash happens. Since the last thing added were the grid labels, I suspect that turning on any of the grids should trigger the crash.
Once you have a minimum configuration that consistently crashes, put a breakpoint at the beginning of D3D9CelestialSphere::Render and try to locate the offending statement or the system call that doesn't return.

Add line Planetarium = 0 to OrbiterNG.cfg. This will turn off all planetarium elements.
Check that Ctrl-F9 no longer crashes
If that works, open the planetarium configuration page (F6 | Planetarium) and turn on elements one by one until the crash happens. Since the last thing added were the grid labels, I suspect that turning on any of the grids should trigger the crash.
Once you have a minimum configuration that consistently crashes, put a breakpoint at the beginning of D3D9CelestialSphere::Render, step through it and try to locate the offending statement or the system call that doesn't return.

Thank you very much for the detailed debugging instructions Dr. Schweiger. Will debug and figure out what might be causing the problem.

After setting Planetarium=0, I can confirm that enabling any of the grids always crashes Orbiterx64 (and Orbiterx86, only in D3D9).

OK. Are you in a position to debug (or at least compile) the code?

Also, I just noticed from the log file you posted that you have redirected your TextureDir and HightexDir locations. Any particular reason for this? For the development builds this is probably not a good idea since you will be missing any additional textures added during development. If you didn't copy the new gridlabel.dds file from the local Textures directory to your Orbiter2016\Textures directory, it will not be found, which may well be the reason for the crash. What happens if you don't redirect these folder locations?
The only directory that should be redirected is the PlanetTexDir folder, since that doesn't exist in the development build.

I have now pushed a PR to main which should deal with a missing grid label texture file more gracefully. Instead of crashing, it should just not display the labels and leave an error message in the log file instead. Does this make any difference to the observed behaviour for both of you?

The only directory that should be redirected is the PlanetTexDir folder, since that doesn't exist in the development build.
I guess I did this mistake and by only redirecting PlanetTexDir the issue seems to be solved on my side even on the debug builds. Sorry for the late reply Dr. Schweiger

Also, I just noticed from the log file you posted that you have redirected your TextureDir and HightexDir locations.

That was it! Redirecting only PlaneTexDir and leaving the other dirs at their default results in NO crashes whatsoever.
Thanks and sorry for my misleasding bug report.

Thanks for the feedback. Can you also confirm that with the latest build you don't get crashes anymore even if you still redirect the TextureDir and HightexDir locations?

Yes, confirmed, no more crashes with textures dirs redirection, both x86 and x64, BUT now solar panels flicker when in external visuals (default DG docked at ISS scenario).

Sorry, are you saying that this latest problem has only started happening with the latest commit (111577b) and only if you redirect TextureDir and HighTexDir?
If that isn't what you are saying, can you please open another issue for that problem, since it doesn't seem to be connected to what was discussed here so far. Also, please provide as much details as possible (which clients, which screen modes, and ideally the commit hash when this started to happen). Thanks!

Hi, the new dawn took the flickering away.
No flickering whatsoever in ext visuals now, redirecting one texture dir at the time.
Must have been an error on my side, sorry for my crappy bugreport.

No worries. Closing this issue.


Updated showcase-intro4 to reflect date changes

Fixes description of showcase, updates links, and adds new Open Call Form
Changes:

Switch description from 2020 showcase to 2021 showcase Open Call
Updated links of current curator and Open Call Form
Commits on June 14th are all tests for PR done with my mentor because we were trying to find the PR on Github desktop

@Qianqianye @outofambit
Looking to get PR merged, any advice/ what would be my next steps? Waiting to officially launch p5.js 2021 showcase Open Call, cannot do so until the website is updated with changes

Hi @kchan2000 it looks great! quick question: are you planning to use showcase.p5js.org or p5js.org/showcase for the final showcase? I  saw that you updated the p5js.org/showcase  page, but this page is not linked in the p5js.org sidebar menu or the 'Check out the new p5.js 2020 showcase!' hypertext (see screenshot below)on the homepage. It would be helpful to let us know what's the relationship between the two pages.
@outofambit are there any previous conversations about the two pages?
Thanks!

@Qianqianye @outofambit
I'm planning on using showcase.p5js.org for the showcase, but for the Open Call Form I was hoping to use p5js.org/showcase. I am also not too sure about where the p5js.org/showcase page sits right now because I also noticed that it is not attached to the sidebar anymore...

I think the decision was made to replace p5js.org/showcase with showcase.p5js.org and p5js.org/showcase was only left for legacy reasons (so that existing links does not 404) so was not meant to be updated or linked to anymore. Maybe @connieliu0 can add a bit more on this?

@limzykenneth @Qianqianye @outofambit
Ohh I see - that makes sense. Would it be possible to add the form under the Get Involved tab? And then I will add a section at the top of showcase.p5.js.org for when people click showcase : )

Hi! I think that would be perfect! Basically p5js.org/showcase is only accessible from showcase.p5js's 2019 page, so if you wanted to advertise the new submissions I would put it on showcase.p5js.org and on the p5js.org's homepage


Sparse system matrix

The error on AppVeyor is:
Build execution time has reached the maximum allowed time for your plan (60 minutes).

But I do not know how to reduce compilation time


[TS-757] Added check repeating group flag

Please, close this request. Should keep order in the result by default if we don't strictly check the order


Decode json with hyphen (-) char cause errors

I tried to decode a json generated with this link https://pokeapi.co/api/v2/pokemon/ditto
The json contains some key with the hyphen character, the classes generated have this name PokemonSpritesVersionsGeneration-ii and this causes compiler errors.
Moreover, in the entity helper class all the methods to convert from or two json are without the right side of the hyphen

This issue has been fixed and a new version is expected to be released this week


Only provide allowed voting methods in poll dialog

Show only allowd voting types in poll dialog and hide types which cannot used for electronic voting.
Currently the "start voting" button is missing - and there is no hint that this type is not supported.
E.g. all "votecollector" voting types cannot be used for election method 'YN' or for elections with more than one open posts.

E.g. all "votecollector" voting types cannot be used for election method 'YN' or for elections with more than one open posts.
I think this is the only case. All electronic voting modes support every poll method. Just the VC is restrictive in the given case: YN(A) with >1 candidate.
I'll create a PR.


Automated Restructured Text Converter

There is a Restructured Text Converter listed here:
http://jekyllrb.com/docs/plugins/#converters
On a rainy day I will test it out. If anyone knows Jekyll better and wants to beat me to it, please go ahead.

Raising the @doismellburning bat/jekyll signal here.

Thanks very much.

I will give a beer for first person who gets it to work :)

Closing, since this is covered by https://github.com/PyconUK/pyconuk.org.
:beer: please!


tests: Increase unit-test coverage for list.go

Increase unit-test coverage on list.go to 98.6%.
Once https://github.com/containers/virtcontainers/pull/355 lands and is
re-vendored into the runtime, coverage will be 100% for this file.
Fixes #404.
Signed-off-by: James O. D. Hunt<EMAIL_ADDRESS>

Coverage increased (+3.6%) to 69.781% when pulling e487ab444184089c0018029aa2a7845d01595b94 on jodh-intel:increase-list-test-coverage into 9c8d40ef06ac5d849c4ffb6e049c84b007568da6 on clearcontainers:master.

lgtm


Add MIGRATION Guide

This document should describe the necessary steps for existing repositories to migrate from classic eslint 8.x.x to eslint 9.x.x using @iobroker/eslint-config.

I've now migrated two project following this guide.
Please review and if Ok make it available at main branch.
If some link from README is desired, please let me know.

@foxriver76 FYI

Thanks for review. Should be fixed.


NER can be quite off sometimes

e.g. the abstract for https://www.researchgate.net/publication/273349137_Realization_of_narrowband_thermal_emission_with_optical_nanostructures
seems to have a lot of inaccuracies

another example:
https://link.springer.com/article/10.1007/s11708-009-0009-x

Looks like most of these errors are related to the application tag, which is one of our worst performing tags. There aren't a lot of papers like these in our NER training set.

@LeighWeston86 is the NER training set available for download somewhere?

Yes the training set is here: https://figshare.com/articles/Materials_Science_Named_Entity_Recognition_train_development_test_sets/8184428

Thanks @computron!

@LeighWeston86 is it possible to share the code used for training? i tried to reproduce the result but i only got like 76% accuracy, while in the paper it is mentioned abt 87%.
Thanks!


Add actions

Hey @ggmichael,
this PR adds a github workflow that runs the test suite, reports the coverage of the tests, and runs the demo command for the cli and then uploads the figures to a zip file using the upload-artifact action.
Normally the owner of the repository needs to authorize/create workflows for a github repository. You may see a prompt to authorize the action/any and all actions to run either on the PR or in the actions tab for the project near the top of the page.

ok - I'll merge it - thanks.

hmm things are not quite right yet with the setup.py and the layout of the project. I'll look into it but maybe @rbeyer will spot first...

Things look fine to me (maybe Greg fixed them with post-PR commits).
Most importantly, after I pulled, I was able to pip install --no-deps -e . in the project directory, which added the craterstats program to my path, and I was able to traipse to a new directory, and run craterstats -demo and it worked exactly as expected!


armhf support.

Just wanted to let you know I was able to get this working on armhf devices with very little modification. I can PR it in if you want to make e.g. an armhf branch.
For context, this was to get our baseimage-docker-based kobo-docker project running on Rasberry Pis.

I'm glad to hear that baseimage-docker is being used on Raspberry Pi! As far as actively maintaining an armhf branch, if that is something you would like to contribute to, you are more than welcome to submit your new branch! Thank you!

@Theaxiom, I don't have permissions to create a new branch on your repo. Regarding maintaining the branch, I've been thinking of setting up Jenkins on a Pi to automate image rebuilding and updating my branch, but there are no immediate plans for this.

@esmail You can submit your branch via a PR. When you get the CI setup for that branch, definitely let us know.

A PR into what branch, master?

I can change the banch after you submit your PR.

Interesting. I wasn't aware of that.

Initial PR #368 submitted for other armhf users with the caveat that I have no immediate plans to actively maintain it.

Thank you!


Zone control in PEA-RP140GAA

Just upgraded to the latest release. Many thanks.
Questions:
Are any zone controls available for ducted units in the Echonet protocol?
If so, is there any possibility of adding those?

Hi Dekks,
Sorry but ECHONET doesn't support zone controls - believe me I have been asked this question before!


[HUDI-4042] Support truncate-partition for Spark-3.2

What is the purpose of the pull request
In spark-3.2 (compiling with mvn -P3.2), test of TestTruncateTable("Test Truncate single-partition table' partitions") fails. The reason is HoodieSpark3Analysis seems forgot to transform TruncatePartition to TruncateHoodieTableCommand.
Brief change log

Add a branch to transform TruncatePartition to TruncateHoodieTableCommand in HoodieSpark3Analysis;
Replace some checking of "isInstance" by case-class pattern matching;

Verify this pull request
This pull request is already covered by existing tests, such as TestTruncateTable.
Committer checklist

[ ] Has a corresponding JIRA in PR title & commit

[ ] Commit message is descriptive of the change

[ ] CI is green

[ ] Necessary doc changes done or have another open PR

[ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

Hi @leesf
I touched you code, please give a check when you have time. Thanks a lot !