[2.3] 404 after installation

While deploying this role (commit 7900bc196a028e2f899b47d7a0461573e439f58b) on a test server I see that BBB is not successfully installed.
This is how I deploy this role:
https://github.com/PrivacyLx/devops/tree/bbb/v2.3/ansible/roles/bigbluebutton
# bbb-conf --check
BigBlueButton Server 2.3.1 (2216)
                    Kernel version: 5.4.22
                      Distribution: Ubuntu 18.04.5 LTS (64-bit)
                            Memory: 8161 MB
                         CPU cores: 4

/etc/bigbluebutton/bbb-web.properties (override for bbb-web)
/usr/share/bbb-web/WEB-INF/classes/bigbluebutton.properties (bbb-web)
       bigbluebutton.web.serverURL: https://testing.bbb.privacylx.org
                defaultGuestPolicy: ALWAYS_ACCEPT
                 svgImagesRequired: true

/etc/nginx/sites-available/bigbluebutton (nginx)
                       server_name: testing.bbb.privacylx.org
                              port: 80, [::]:80
                              port: 443 ssl

/opt/freeswitch/etc/freeswitch/vars.xml (FreeSWITCH)
                       local_ip_v4: <IP_ADDRESS>
                   external_rtp_ip: stun:testing.bbb.privacylx.org
                   external_sip_ip: stun:testing.bbb.privacylx.org

/opt/freeswitch/etc/freeswitch/sip_profiles/external.xml (FreeSWITCH)
                        ext-rtp-ip: $${local_ip_v4}
                        ext-sip-ip: $${local_ip_v4}
                        ws-binding: :5066
                       wss-binding: <IP_ADDRESS>:7443

/usr/local/bigbluebutton/core/scripts/bigbluebutton.yml (record and playback)
                     playback_host: testing.bbb.privacylx.org
                 playback_protocol: https
                            ffmpeg: 4.2.4-1ubuntu0.1bbb2~18.04

/etc/bigbluebutton/nginx/sip.nginx (sip.nginx)
                        proxy_pass: <IP_ADDRESS>
                          protocol: https

/usr/local/bigbluebutton/bbb-webrtc-sfu/config/default.yml (Kurento SFU)
                        kurento.ip: <IP_ADDRESS>
                       kurento.url: ws://<IP_ADDRESS>:8888/kurento
                    kurento.sip_ip: <IP_ADDRESS>
                    localIpAddress: <IP_ADDRESS>
               recordScreenSharing: true
                     recordWebcams: true
                  codec_video_main: VP8
               codec_video_content: VP8

/usr/share/meteor/bundle/programs/server/assets/app/config/settings.yml (HTML5 client)
                             build: 1720
                        kurentoUrl: wss://testing.bbb.privacylx.org/bbb-webrtc-sfu
                  enableListenOnly: true
                    sipjsHackViaWs: false

/etc/bigbluebutton/turn-stun-servers.xml (STUN Server)
                              stun: testing.bbb.privacylx.org

# Potential problems described below
grep: /opt/freeswitch/var/log/freeswitch/freeswitch.log: No such file or directory

#
# Warning: Failed to verify STUN server at testing.bbb.privacylx.org 3478 with command
#
#    stunclient --mode full --localport 30000 testing.bbb.privacylx.org 3478

# stunclient --mode full --localport 30000 testing.bbb.privacylx.org 3478
Binding test: fail
Behavior test: fail
Filtering test: fail

# journalctl -u freeswitch.service
systemd[1]: Starting freeswitch...
systemd[30108]: freeswitch.service: Failed to set up CPU scheduling: Operation not permitted
systemd[30108]: freeswitch.service: Failed at step SETSCHEDULER spawning /opt/freeswitch/bin/freeswitch: Operation not permitted
systemd[1]: freeswitch.service: Control process exited, code=exited status=214
systemd[1]: freeswitch.service: Failed with result 'exit-code'.
systemd[1]: Failed to start freeswitch.

I am deploying from the latest commit https://github.com/ebbba-org/ansible-role-bigbluebutton/commit/7900bc196a028e2f899b47d7a0461573e439f58b and will test this.

PLAY RECAP **************************************************************************************************************************************************************************
meet.bbb.ansibletest.safeserver-support.de : ok=274  changed=170  unreachable=0    failed=0    skipped=355  rescued=0    ignored=0

Yes these are a lot more steps since I use more roles for ssh, zabbix yada yada.
root@meet:~# bbb-conf --status
nginx —————————————————► [✔ - active]
freeswitch ————————————► [✔ - active]
redis-server ——————————► [✔ - active]
bbb-apps-akka —————————► [✔ - active]
bbb-fsesl-akka ————————► [✔ - active]
mongod ————————————————► [✔ - active]
bbb-html5 —————————————► [✔ - active]
bbb-webrtc-sfu ————————► [✔ - active]
kurento-media-server ——► [✔ - active]
etherpad ——————————————► [✔ - active]
bbb-web ———————————————► [✔ - active]

Oh?
root@meet:~# systemctl list-units --failed
UNIT                         LOAD   ACTIVE SUB    DESCRIPTION
●<EMAIL_ADDRESS>loaded failed failed BigBlueButton HTML5 service, backend instance 1
●<EMAIL_ADDRESS>loaded failed failed BigBlueButton HTML5 service, backend instance 2
●<EMAIL_ADDRESS>loaded failed failed BigBlueButton HTML5 service, frontend instance 1
●<EMAIL_ADDRESS>loaded failed failed BigBlueButton HTML5 service, frontend instance 2

Hmmm yes this is a race condition, I only use a very cheap VPS for testing.
root@meet:~# systemctl restart bbb-html5-*
root@meet:~# systemctl list-units --failed
0 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.

And no issues for me.
https://meet.bbb.ansibletest.safeserver-support.de/b/adm-rs3-dgv-4sc

Wait...
systemd[1]: Starting freeswitch...
systemd[30108]: freeswitch.service: Failed to set up CPU scheduling: Operation not permitted
systemd[30108]: freeswitch.service: Failed at step SETSCHEDULER spawning /opt/freeswitch/bin/freeswitch: Operation not permitted
systemd[1]: freeswitch.service: Control process exited, code=exited status=214
systemd[1]: freeswitch.service: Failed with result 'exit-code'.
systemd[1]: Failed to start freeswitch.

Did you set this? Are you running inside LXD/C?
LXD/LXC compatibility

To run BigBlueButton in unprivileged LXD/LXC containers, you have to set bbb_container_compat to true.

@BrutalBirdie Thank you for helping out.

Did you set this? Are you running inside LXD/C?

Yes, and I have set the bbb_container_compat to true.

@anadahz
I suspect this part in the tasks/config.yml
- name: configure CPU and IO Scheduling in FreeSWITCH service
  lineinfile:
    path: /lib/systemd/system/freeswitch.service
    regexp: '{{ item.regexp }}'
    line: '{{ item.line }}'
    state: '{{ item.state }}'
    insertbefore: '^CPUSchedulingPriority='
  with_items:
    - line: 'CPUSchedulingPolicy=rr'
      regexp: '^#?CPUSchedulingPolicy='
      state: '{{ "present" if not bbb_container_compat | bool else "absent" }}'
    - line: 'IOSchedulingClass=realtime'
      regexp: '^#?IOSchedulingClass='
      state: '{{ "present" if not bbb_container_compat | bool else "absent" }}'
  notify:
  - restart freeswitch
  - reload systemd

Can you edit your /lib/systemd/system/freeswitch.service file and remove the erroring parts described in the error message?
systemd[1]: Starting freeswitch...
systemd[30108]: freeswitch.service: Failed to set up CPU scheduling: Operation not permitted
systemd[30108]: freeswitch.service: Failed at step SETSCHEDULER spawning /opt/freeswitch/bin/freeswitch: Operation not permitted
systemd[1]: freeswitch.service: Control process exited, code=exited status=214
systemd[1]: freeswitch.service: Failed with result 'exit-code'.
systemd[1]: Failed to start freeswitch.

Closing it is as invalid and not related issue to this role.

@anadahz are you sure? What about that part I posted?

@anadahz are you sure? What about that part I posted?

I first installed the previous version of the role without resetting the server.
Running the role in a clean server works as expected.


Regenerate antlr code with 4.10.1

Update ANTLR generated files to be compatible with version 4.10.1.

ANTLR 4.10 does not support java 8.

We are unable to move to antlr 4.10.0 because the runtime was compiled against java 11 (even though the target and source were set to Java 8).
This manifests in the following error when running our tests in rewrite-java-8:
Caused by:
        java.lang.NoSuchMethodError: java.nio.CharBuffer.flip()Ljava/nio/CharBuffer;

I've submitted a PR to the antlr project to fix this issue.
https://github.com/antlr/antlr4/pull/3648

Update: ANTLR is pinned back to 4.9.3.
There are several downstream projects that have dependencies that are not compatible with antlr 4.10.1.

rewrite-maven-plugin: checkstyle
graphql-java

It appears checkstyle is now updated.
https://github.com/checkstyle/checkstyle/commit/1e862ff74bcfd34eaa10d8e0bb7c11abf26d8927

So the only downstream dependency that is preventing this is the graphql project?

So the only downstream dependency that is preventing this is the graphql project?

Yes. checkstyle was updated pretty quickly, but graphql-java has essentially stated it's not a priority.

I (probably) ran into this issue, using the org.openrewrite.recipe:rewrite-recipe-bom:1.12.3:
ANTLR Tool version 4.9.3 used for code generation does not match the current runtime version 4.10.1
ANTLR Runtime version 4.9.3 used for parser compilation does not match the current runtime version 4.10.1

Overriding the Antlr version to 4.9.3 helps.
Are there any plans to upgrade to a newer Antlr version?


Updated the emission-line notebooks to work with SPARCL

The notebook works well with the SPARCL. I still have to update the username to demo00 before the final merging.

Hi @Ragadeepika-Pucha I looked through and ran both notebooks and they look awesome! I agree with Robert's comments and don't have any additional remarks.

Hi @rnikutta,
Thank you for the comments. I have updated the notebooks and modified the section names.

For the plot_sdss_spectra() function in the stacking notebook, I wanted to have a function that directly plots the spectra given a specobjid. In the outliers notebook, I have it differently in the sense that the function takes in the spectra to plot. I wanted to show both the functions in the different notebooks. It is mainly because sometimes plotting is the main thing and one extra line can be avoided in the code.
I am not sure I understand your rectangles comment. I define the rectangles only once (in cell 23) and select random sources within each rectangle only once as well (in cell 25).

Let me know if you have any more questions/comments.
Thanks,
Raga

Hi @Ragadeepika-Pucha , thanks for addressing my comments.

Regarding plot function vs get_spectrum function: you avoid a line in calling a function... but now you have to define basically the same function twice... It's no problem, but it's not efficient either, hence my original suggestion.
About defining the 4 rectangles/squares: you define the box edge values twice: in cell 22 in the Rectangle() instantiations. And then again in cell 23 (same numbers). If you now decided you want to change the exact boxes, you'd have to change the numbers in two places. Wouldn't it be better to define them only once, and then both create the Rectangle() instances in cell 22, and getting the index arrays in cell 23, using the only-once defined limits?
Thanks!

Hi @Ragadeepika-Pucha and @stephjuneau Bumping the discussion on this PR; what do you think of my last comments from 12 days ago? Thanks!

Hi @rnikutta, Raga told me about these comments and that she would address them during one of the astro-coding sessions. Now that I read through, I'll take a look at the revisions as well so thanks for bumping them to make sure that they're still on the radar!

@rnikutta Thank you for explaining the comments, and I am sorry for the delay in submitting them.
I have separated the accessing and plotting the spectra functions and also updated the docstrings of the functions.
I have also added an extra cell to define the rectangles that are used for plotting and accessing the objects.
Let me know if any other changes need to be made. Thanks!
@stephjuneau please take a look at the updated notebooks. Thank you.


Moved FormsModule and HttpModule to app.modules.shared

Moved FormsModule and HttpModule from from app.module.client to app.module.shared

@AmrineA,
Thanks for having already signed the Contribution License Agreement. Your agreement was validated by .NET Foundation. We will now review your pull request.
Thanks,
.NET Foundation Pull Request Bot

Thanks for contributing this!
Based on @MarkPieszak's advice, I extended your PR a bit to make the shared module into a real @NgModule. This is now committed in f3b7103c83ef38d787c2e66bc761a1c4c562937a.


Depend on serving and eventing master.

Depend on serving and eventing master instead of specific revisions.
Originally posted by @grantr in https://github.com/GoogleCloudPlatform/cloud-run-events/pull/35#discussion_r301226846

Also add a comment to overrides This is a preemptive override https://github.com/GoogleCloudPlatform/cloud-run-events/pull/35#discussion_r301221119


[SPARK-33832][SQL] Add an option in AQE to mitigate skew even if it c…

…auses an new shuffle

What changes were proposed in this pull request?

This PR adds spark.sql.adaptive.force.if.shuffle config option which, if set to true, causes OptimizeSkewedJoin to perform skew mitigation even if it requires a new shuffle to be performed.
For example, current behavior is not perform skew mitigation in the following query
Project
|-Group By key1
  |-SMJ on key1 = key2

With spark.sql.adaptive.force.if.shuffle=true, the SMJ will become skew=true and a new shuffle will be added for the Aggregate.  New shuffle itself will be processed by AQE and coalesced, etc if needed.
Why are the changes needed?

It enables AQE to apply in cases where the cost of extra shuffle is less than the cost of unmitigated skew.
Does this PR introduce any user-facing change?

No, with spark.sql.adaptive.force.if.shuffle=false, which is the default
How was this patch tested?

New UTs + enhancements of existing ones

Can one of the admins verify this patch?

This failed with [error] (Javaunidoc / doc) sbt.inc.Doc$JavadocGenerationFailed which other unrelated PRs failed with.
Could someone restart this please?  cc @dongjoon-hyun

Jenkins, retest this please

Jenkins, retest this please

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Hi,
@maryannxue , @cloud-fan, @HyukjinKwon  could one of you take a look please?  Or perhaps suggest someone else who could comment

Hi,
@maryannxue , @cloud-fan, @HyukjinKwon  could one of you take a look please?  Or perhaps suggest someone else who could comment

ok to test

ok to test

It's hard to calculate the cost of a shuffle and compare it with the benefit of skew join handling. We need some ways to tune it manually. But I don't understand why this patch is so complicated. Isn't it simply skip counting the shuffles at the end of OptimizeSkewedJoin?

It's hard to calculate the cost of a shuffle and compare it with the benefit of skew join handling. We need some ways to tune it manually. But I don't understand why this patch is so complicated. Isn't it simply skip counting the shuffles at the end of OptimizeSkewedJoin?

Test build #133913 has started for PR 30829 at commit c9c236b.

Test build #133913 has started for PR 30829 at commit c9c236b.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38500/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38500/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38500/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38500/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38500/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38500/

Test build #133913 has finished for PR 30829 at commit c9c236b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #133913 has finished for PR 30829 at commit c9c236b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133913/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133913/

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38520/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38520/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38520/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38520/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38520/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38520/

I agree that automatically computing the cost of the new shuffle is hard.  That is why this first step uses an option to enable this behavior - based on previous runs of a query or report, for example.  I think the mechanics of adding a new shuffle have to be built first before it can become a cost based decision.
I'm not sure what you mean by "skip counting"
OptimizeSkewedJoin runs on the part of the plan bounded by Exchange above (call it E) and completed QueryStageExec leaves below.
So if OptimizeSkewedJoin makes a change that needs a new shuffle, it happens in the Nth recursive invocation of createQueryStages.  So it figures out where the new Exchange should be and throws ShuffleAddedException to pop back out to the top, add the new Exchange in the right place and start createQueryStages again from the top.  Now that there is a new Exchange (call it E') is in the plan and below E, and the same QueryStageExec leaves, newQueryStage(E') is run.
This seems like the easiest way to preserve the flow of the existing algorithm.
The
        var continue = true
        while (continue) {

in getFinalPhysicalPlan is because in a complex plan there may be several new stages created during one iteration of
while (!result.allChildStagesMaterialized) { and each may potentially introduce a new shuffle.
The processing the final stage (before any of my changes) is slightly different - finalStageOptimizerRules to preserve properties of the output, so handleFinalStageShuffle does the same.

I agree that automatically computing the cost of the new shuffle is hard.  That is why this first step uses an option to enable this behavior - based on previous runs of a query or report, for example.  I think the mechanics of adding a new shuffle have to be built first before it can become a cost based decision.
I'm not sure what you mean by "skip counting"
OptimizeSkewedJoin runs on the part of the plan bounded by Exchange above (call it E) and completed QueryStageExec leaves below.
So if OptimizeSkewedJoin makes a change that needs a new shuffle, it happens in the Nth recursive invocation of createQueryStages.  So it figures out where the new Exchange should be and throws ShuffleAddedException to pop back out to the top, add the new Exchange in the right place and start createQueryStages again from the top.  Now that there is a new Exchange (call it E') is in the plan and below E, and the same QueryStageExec leaves, newQueryStage(E') is run.
This seems like the easiest way to preserve the flow of the existing algorithm.
The
        var continue = true
        while (continue) {

in getFinalPhysicalPlan is because in a complex plan there may be several new stages created during one iteration of
while (!result.allChildStagesMaterialized) { and each may potentially introduce a new shuffle.
The processing the final stage (before any of my changes) is slightly different - finalStageOptimizerRules to preserve properties of the output, so handleFinalStageShuffle does the same.

@ekoifman ah I get your point. By design, queryStageOptimizerRules should not add/remove shuffles. We probably need to move the rule OptimizeSkewedJoin to queryStagePreparationRules.

@ekoifman ah I get your point. By design, queryStageOptimizerRules should not add/remove shuffles. We probably need to move the rule OptimizeSkewedJoin to queryStagePreparationRules.

@cloud-fan I don't think you can move OptimizeSkewJoin to queryStagePreparationRules for 2 reasons

OptimizeSkewJoin logically has to run after CoalesceShufflePartitions.  (Also, comments on queryStageOptimizerRules state the same)
queryStagePreparationRules may run before all child QueryStageExec of the stage have materialized (called from reOptimize()) but OptimizeSkewJoin has to know the input from both sides.

@cloud-fan I don't think you can move OptimizeSkewJoin to queryStagePreparationRules for 2 reasons

OptimizeSkewJoin logically has to run after CoalesceShufflePartitions.  (Also, comments on queryStageOptimizerRules state the same)
queryStagePreparationRules may run before all child QueryStageExec of the stage have materialized (called from reOptimize()) but OptimizeSkewJoin has to know the input from both sides.

OK, then can you briefly explain your idea about how to allow queryStageOptimizerRules to add shuffles?

OK, then can you briefly explain your idea about how to allow queryStageOptimizerRules to add shuffles?

A Rule in queryStageOptimizerRules adds a shuffle indirectly.

OptimizeSkewJoin runs and adds a new shuffle below e in a newQueryStage(e: Exchange) call
We can't keep this optimizedPlan because the new shuffle should become the new root of a stage so instead it remembers the location in the plan where the new shuffle needs to be and throws ShuffleAddedException with the location back to the main loop in getFinalPhysicalPlan()  and val z = insertShuffle(e.sp, currentPhysicalPlan) properly inserts the new shuffle in currentPhysicalPlan.  Call the new shuffle e'
now it restarts createQueryStages(currentPhysicalPlan) from the top but with new shuffle added.  When newQueryStage finds e', it becomes the root of the stage with the same leaves as e.  Now when OptimizeSkewJoin runs on this stage, it can do skew mitigation but it no longer needs to add a new shuffle to do so, since e' "protects" e

A Rule in queryStageOptimizerRules adds a shuffle indirectly.

OptimizeSkewJoin runs and adds a new shuffle below e in a newQueryStage(e: Exchange) call
We can't keep this optimizedPlan because the new shuffle should become the new root of a stage so instead it remembers the location in the plan where the new shuffle needs to be and throws ShuffleAddedException with the location back to the main loop in getFinalPhysicalPlan()  and val z = insertShuffle(e.sp, currentPhysicalPlan) properly inserts the new shuffle in currentPhysicalPlan.  Call the new shuffle e'
now it restarts createQueryStages(currentPhysicalPlan) from the top but with new shuffle added.  When newQueryStage finds e', it becomes the root of the stage with the same leaves as e.  Now when OptimizeSkewJoin runs on this stage, it can do skew mitigation but it no longer needs to add a new shuffle to do so, since e' "protects" e

This seems like a big change to the AQE framework. If it's rare to force apply the skew join handling, how about we start with a sub-optimal solution: we just add the extra shuffle and do not create a query stage for this shuffle. Under the hood, ShuffleQueryStageExec submits a map stage, which can contain shuffle as well.

This seems like a big change to the AQE framework. If it's rare to force apply the skew join handling, how about we start with a sub-optimal solution: we just add the extra shuffle and do not create a query stage for this shuffle. Under the hood, ShuffleQueryStageExec submits a map stage, which can contain shuffle as well.

@cloud-fan  I think introducing an "unhandled" shuffle is conceptually a bigger change.  Currently all the code assumes (and tests assert it) that that there are no "unhandled" shuffles.  That is the reason I made this PR such that when a queryStageOptimizerRules rule wants to add a shuffle, the algorithm pops back out  to the top to restart createQueryStages(currentPhysicalPlan) from the top.  This preserves the main logic of finding and creating new QueryStages as close as possible to what it was before my change.
I think the mechanics of adding a shuffle in the middle of stage creation would be just as complicated.  Currently, queryStagePreparationRules which includes EnsureRequirements run on the whole plan (i.e. currentPhysicalPlan not a specific stage).  If ShuffleQueryStageExec is the place where a new shuffle is added, should it just run EnsureRequirements on the current stage or all of queryStagePreparationRules?  Should it be done before or after other rules in queryStageOptimizerRules are to run?  As additional rules are added to queryStagePreparationRules or queryStageOptimizerRules, would OptimizeSkewJoin ability to add shuffles make it more complicated?
I'm raising these questions to illustrate why I don't think adding an "unhandled" shuffle makes it easier.  Please let me know if you think they have easy answers.

@cloud-fan  I think introducing an "unhandled" shuffle is conceptually a bigger change.  Currently all the code assumes (and tests assert it) that that there are no "unhandled" shuffles.  That is the reason I made this PR such that when a queryStageOptimizerRules rule wants to add a shuffle, the algorithm pops back out  to the top to restart createQueryStages(currentPhysicalPlan) from the top.  This preserves the main logic of finding and creating new QueryStages as close as possible to what it was before my change.
I think the mechanics of adding a shuffle in the middle of stage creation would be just as complicated.  Currently, queryStagePreparationRules which includes EnsureRequirements run on the whole plan (i.e. currentPhysicalPlan not a specific stage).  If ShuffleQueryStageExec is the place where a new shuffle is added, should it just run EnsureRequirements on the current stage or all of queryStagePreparationRules?  Should it be done before or after other rules in queryStageOptimizerRules are to run?  As additional rules are added to queryStagePreparationRules or queryStageOptimizerRules, would OptimizeSkewJoin ability to add shuffles make it more complicated?
I'm raising these questions to illustrate why I don't think adding an "unhandled" shuffle makes it easier.  Please let me know if you think they have easy answers.

OptimizeSkewJoin already runs EnsureRequirements inside it, the only change we need is to not give up the optimization even if extra shuffles are added.
Query stage is quite self-contained. queryStagePreparationRules can't see the query plan of query stages. During re-optimization, query stage becomes a leaf node LogicalQueryStage.
One problem is, queryStageOptimizerRules can't assume that query stage doesn't have shuffles in the middle. We need to revisit these rules.

OptimizeSkewJoin already runs EnsureRequirements inside it, the only change we need is to not give up the optimization even if extra shuffles are added.
Query stage is quite self-contained. queryStagePreparationRules can't see the query plan of query stages. During re-optimization, query stage becomes a leaf node LogicalQueryStage.
One problem is, queryStageOptimizerRules can't assume that query stage doesn't have shuffles in the middle. We need to revisit these rules.

After a second thought, I think adding "unhandled" shuffle is a bit risky, but allowing the stage optimization phase to add new shuffles is too complicated.
I'd like to revisit the idea of putting the skew join optimization rule in the stage preparation phase. For the two points you gave:

I think it's not true now. The comment is stale. If you look at the classdoc of OptimizeSkewedJoin, it says when this rule is enabled, it also coalesces non-skewed partitions like CoalesceShufflePartitions does. So I don't think OptimizeSkewedJoin needs to be run after CoalesceShufflePartitions.
We can add some checks and only trigger OptimizeSkewedJoin if the related shuffle stages are all materialized.

After a second thought, I think adding "unhandled" shuffle is a bit risky, but allowing the stage optimization phase to add new shuffles is too complicated.
I'd like to revisit the idea of putting the skew join optimization rule in the stage preparation phase. For the two points you gave:

I think it's not true now. The comment is stale. If you look at the classdoc of OptimizeSkewedJoin, it says when this rule is enabled, it also coalesces non-skewed partitions like CoalesceShufflePartitions does. So I don't think OptimizeSkewedJoin needs to be run after CoalesceShufflePartitions.
We can add some checks and only trigger OptimizeSkewedJoin if the related shuffle stages are all materialized.

RE 1, I see this comment on OptimizeSkewedJoin but I don't see the code actually doing any coalesce.  optimizeSkewJoin() uses ShuffleStage.unapply to either take CoalescedPartitionSpec from a CustomShuffleReaderExec child node or creates CoalescedPartitionSpec if the child is ShuffleQueryStageExec.  In the latter case, all the specs created are CoalescedPartitionSpec(i, i + 1).  As far as I can tell, the rest of optimizeSkewJoin() either handles a skewed partition or uses CoalescedPartitionSpec from ShuffleStageInfo.  Is there some other place OptimizeSkewedJoin performs coalesce?
Also, I'd like to clarify one thing in the in the current PR.  Stage optimization doesn't actually add the shuffle.  OptimizeSkewedJoin in the stage optimization runs and returns the modified plan with the new shuffle in newQueryStage(e) but this plan is not used.   Once this happens, recursive call stack of createQueryStages() stops and control goes back to the top of currentPhysicalPlan in the main while (!result.allChildStagesMaterialized) { of getFinalPhysicalPlan.  This is done via ShuffleAddedException  which serves as a "request" to add a shuffle.  So with respect to createQueryStages(), the insertion of the shuffle is no different than if the new shuffle was caused by something in reOptimize(logicalPlan).
So in fact, the new shuffle is added during Query Stage preparation as you suggest,  but it looks different in the code because OptimizeSkewedJoin decides where the new shuffle should be added, but it asks higher logic to add it.  In some sense this can be thought of as search space exploration with backtracking: query stage optimization  (specifically OptimizeSkewedJoin) says I can't do what you want unless you give me a plan with a new shuffle in it and causes query stage creation to backtrack, add the shuffle and restart query stage creation/optimization on the new currentPhysicalPlan with the new shuffle.

RE 1, I see this comment on OptimizeSkewedJoin but I don't see the code actually doing any coalesce.  optimizeSkewJoin() uses ShuffleStage.unapply to either take CoalescedPartitionSpec from a CustomShuffleReaderExec child node or creates CoalescedPartitionSpec if the child is ShuffleQueryStageExec.  In the latter case, all the specs created are CoalescedPartitionSpec(i, i + 1).  As far as I can tell, the rest of optimizeSkewJoin() either handles a skewed partition or uses CoalescedPartitionSpec from ShuffleStageInfo.  Is there some other place OptimizeSkewedJoin performs coalesce?
Also, I'd like to clarify one thing in the in the current PR.  Stage optimization doesn't actually add the shuffle.  OptimizeSkewedJoin in the stage optimization runs and returns the modified plan with the new shuffle in newQueryStage(e) but this plan is not used.   Once this happens, recursive call stack of createQueryStages() stops and control goes back to the top of currentPhysicalPlan in the main while (!result.allChildStagesMaterialized) { of getFinalPhysicalPlan.  This is done via ShuffleAddedException  which serves as a "request" to add a shuffle.  So with respect to createQueryStages(), the insertion of the shuffle is no different than if the new shuffle was caused by something in reOptimize(logicalPlan).
So in fact, the new shuffle is added during Query Stage preparation as you suggest,  but it looks different in the code because OptimizeSkewedJoin decides where the new shuffle should be added, but it asks higher logic to add it.  In some sense this can be thought of as search space exploration with backtracking: query stage optimization  (specifically OptimizeSkewedJoin) says I can't do what you want unless you give me a plan with a new shuffle in it and causes query stage creation to backtrack, add the shuffle and restart query stage creation/optimization on the new currentPhysicalPlan with the new shuffle.

Ah sorry for my bad memory, so the comment in OptimizeSkewedJoin is actually the stale one...
I get your point that it's like backtracking, but it does make the control flow more complicated. And using exceptions in control flow is anti-pattern (see here), we need more effort to refactor the code of the AQE loop.
So I'd like to avoid changing the control flow if possible. I don't see any blockers to run OptimizeSkewedJoin in the stage preparation. We can update CoalesceShufflePartitions to make it work even if there are skewed partition specs, or do the coalesce in OptimizeSkewedJoin.

Ah sorry for my bad memory, so the comment in OptimizeSkewedJoin is actually the stale one...
I get your point that it's like backtracking, but it does make the control flow more complicated. And using exceptions in control flow is anti-pattern (see here), we need more effort to refactor the code of the AQE loop.
So I'd like to avoid changing the control flow if possible. I don't see any blockers to run OptimizeSkewedJoin in the stage preparation. We can update CoalesceShufflePartitions to make it work even if there are skewed partition specs, or do the coalesce in OptimizeSkewedJoin.

ok, let me think about this - I'll be back

ok, let me think about this - I'll be back

The main loop in AdaptiveSparkPlanExec.getFinalPhysicalPlan() has this check
        if (newCost < origCost ||
            (newCost == origCost && currentPhysicalPlan != newPhysicalPlan)) {

Why does it check for new shuffles?  Under what circumstances can reOptimize(logicalPlan) add a new shuffle (w/o changes proposed in this PR)?
If we move OptimizeSkewedJoin to queryStagePreparationRules and it adds a new shuffle, there is no good way that I see to know which new shuffles are "good" and which are "bad".

I think it's just a very conservative check. We can skip this check if the config to force-apply skew join optimization is turned on.

@ekoifman This check is to guard against re-planning that turns an already shuffle-materialized SMJ to BHJ while causing extra shuffles downstream.
One way around this might be to cost a skew join differently so it counters the increased cost of the extra shuffle.

Hi @cloud-fan, @maryannxue
sorry it took me longer than I expected to get back to this.  I opened a new PR https://github.com/apache/spark/pull/31653 for the proposed changes to make it easier to compare.

@cloud-fan do you have any more thoughts about this or #31653?


[Feature request] Allow send instance as backend

First off: amazing idea! I've wanted this sort of functionality for a while. Once my server is up and running, I'm thinking of implementing a "send" option, so the notes can be uploaded to a send instance, and then a link is copied to the user clipboard.

Thanks!
Can you explain what you mean by a send instance? I'm unfamiliar with this.
FYI, you can already host your own instance of the Noteshare.space backend and web viewer, see the readme at https://github.com/mcndt/noteshare.space for more info.

Send: https://github.com/timvisee/send
It's a tool originally built by mozilla for file safe sharing.

Oh cool, I didn't know there's a community fork of Firefox Send!
I'll be happy to merge a PR for Send compatibility, as long as it can be implemented as a drop-in replacement for the current backend.


Add Support for Ultramarine Linux

Just adding support for Ultramarine Linux. It's a Linux distro based on Fedora.
Please let me know if anything needs to be adjusted.

Thanks so much for doing this for us! It'll be nice to have this in our favourite shell

Sorry for the late reply and thank you for the pull request!

I currently have build failures of master related to this, there are a few bits missing from that PR:
error[E0599]: no variant or associated item named `Ultramarine` found for enum `Type` in the current scope
   --> os_info/src/linux/file_release.rs:137:49
    |
137 |                     "ultramarine" => Some(Type::Ultramarine),
    |                                                 ^^^^^^^^^^^ variant or associated item not found in `Type`
    |
   ::: os_info/src/os_type.rs:8:1
    |
8   | pub enum Type {
    | ------------- variant or associated item `Ultramarine` not found for this enum


Add addresses in Birmingham, AL

Status:

[x] Research sources and propose best source (and it's data license)
[ ] Review proposed source
[ ] Approve source based on review comments
[ ] Add to OpenAddresses

NOTE: Proposed source notes should include basic column mapping to OpenAddress data model.

Sent e-mail requesting data (09/27/2016)

We have data for Jackson County - can close this issue

I don't see it in the Alabama folder: https://github.com/openaddresses/openaddresses/tree/master/sources/us/al

**Sorry, I'm a idiot - if it in Jefferson County.  https://github.com/openaddresses/openaddresses/commit/fb890d04cd8efca86dd02d70b63fd4c957f72619

Birmingham is in Jefferson County.

Wahoo!


Deduplicate mentor note

This accompanies https://github.com/exercism/website-copy/pull/828. Let's discuss whether the (newer) example code from the mentor note makes a better example.
cc @exercism/r

The idea of linking to the example code from the mentor notes seems like a good one.
And the proposed solution here is certainly much more concise and readable. It is worth noting however, that it seems to be less performant by roughly a factor of 20 for lengthy factor vectors (see below). I'd be happy to accept this as is, but maybe this performance/readability tradeoff is worth noting in the mentor notes?
I haven't really given much thought to this problem or explored what a 'good' solution for this is - so there may be a different approach that's better than both of these.
sum_of_multiples <- function(factors, limit) {
  get_mults <- function(factor) {
    max_mult <- ceiling(limit / factor) - 1
    if (max_mult > 0) {
      new_multiples <- factor * 1:max_mult
    }
  }
  multiples <- unique(unlist(lapply(factors, FUN = get_mults)))
  return(sum(as.numeric(multiples)))
}

sum_of_multiples2 <- function(factors, limit) {
  func <- function(x) seq(0, limit - 1, x)
  sum(unique(unlist(lapply(factors, func))))
}

library(microbenchmark)

microbenchmark(
  sum_of_multiples(1:1000,10),
  sum_of_multiples2(1:1000,10)
)

Unit: microseconds
                          expr       min         lq      mean    median        uq      max neval
  sum_of_multiples(1:1000, 10)   946.723   986.3395  1104.015  1007.277  1031.295  7983.91   100
 sum_of_multiples2(1:1000, 10) 20015.815 20829.7245 22101.764 21530.939 22990.639 44784.12   100

In that case, we don't have to change the example at all ;-)
I mostly didn't want to drop the newer one silently.

I don't think this needs to be closed persé; the discussion here is very valuable. See https://github.com/exercism/website-copy/pull/828#issuecomment-469447926 for why the other PR was closed.

I'm failing to extract generally useful comments from community solutions. Please feel free to take over adding some comments.

Maybe the completed Sum of Multiples exercises that each of us mentored are a better source for comments & notes?

Thanks for persevering with this @katrinleinweber!
To answer your earlier question:

In that case, let's leave the example code as is, but maybe add a few comments to explain the efficiency (it's ceiling(limit / factor), right?).

The main reason the example solution is more efficient is because seq() is an S3 generic - so there is a (slight) bit of overhead in using it due to the method dispatch. This overhead is non-negligible when calling the function repeatedly (within lapply). You can check this by replacing seq() with seq.int():
microbenchmark(
  sum_of_multiples_seq(1:1000,100),
  sum_of_multiples_seq_int(1:1000,100)
)

Unit: milliseconds
                                  expr       min        lq      mean    median        uq       max neval
     sum_of_multiples_seq(1:1000, 100) 20.707143 21.403223 23.175072 21.677880 23.730201 82.729626   100
 sum_of_multiples_seq_int(1:1000, 100)  1.224661  1.287063  1.366065  1.320318  1.349056  3.797145   100

In terms of general comments for mentoring, I think the current mentor notes already cover the key things, namely:

that for loops should be avoided in favour of the apply or map function families due to R's interpreted nature, and
that this exercise is great one to introduce packages like microbenchmark and discuss the merits of experimentation to assess the impacts of different functional choices on performance.


Add Screen Change Helper Methods

Right now, the GUI manually switches between JPanels (game->menu, menu->help). Build helper methods that change between screens to clean up code.
Estimated ~100 points

m16 okay

F16 OK (100 pts)


define scope and audience of paper earlier

Casper:

CJ The reviewer asks for 2 things: 1) define the SCOPE of the workflow early (e.g., abstract and/or opening paragraph), 2) thoroughly discuss limitations. I think you have only addressed point 2). Make sure to also address point 1).

And Casper at a different point:

CJ When you define the scope of the workflow, you can also define the readership for this paper. That would help manage expectations. Your paper cannot be everything to everyone - but what is it trying to be, and for whom!?


Firestore: The datastore operation timed out, or the data was temporarily unavailable

Environment details

OS: Tested on both Linux and Windows 10
.NET version: .Net Core 2.0
Package name and version: Google.Cloud.Firestore, Version 1.1.0 (latest stable release)

Steps to reproduce

Fails to query when the property happens to be boolean, works fine if the property is string for example
Tested on Firestore Emulator with boolean property and it works just fine.
Tested the same code base running on .Net Core in Linux Container in GCP Kubernetes and it timesout. Tried connecting to the GCP Firestore from local debuger (windows 10) as well without much success.

       CollectionReference FoosRef = FirestoreDb.Collection(FooKind);
        Query query = FoosRef.WhereEqualTo("IsGoodFoo", true);
        // QuerySnapshot querySnapshot = await query.Offset(offset).GetSnapshotAsync();
       // Searching for String property works.
        // Query query = FoosRef.WhereEqualTo("FooName", "p,ezbnR33GU_");
        QuerySnapshot querySnapshot = await query.GetSnapshotAsync();
        DocumentSnapshot documentSnapshot = querySnapshot.Documents.FirstOrDefault();

In the interim is there any other way to query for Boolean property with Offset to skip, please suggest.
Stack Trace:
Grpc.Core.RpcException: Status(StatusCode=Unavailable, Detail="The datastore operation timed out, or the data was temporarily unavailable.")
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Grpc.Core.Internal.ClientResponseStream`2.<MoveNext>d__5.MoveNext() in T:\src\github\grpc\src\csharp\Grpc.Core\Internal\ClientResponseStream.cs:line 61
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Linq.AsyncEnumerable.<ForEachAsync_>d__174`1.MoveNext() in D:\a\1\s\Ix.NET\Source\System.Interactive.Async\ForEach.cs:line 141
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Google.Cloud.Firestore.Query.<GetSnapshotAsync>d__54.MoveNext() in T:\src\github\google-cloud-dotnet\releasebuild\apis\Google.Cloud.Firestore\Google.Cloud.Firestore\Query.cs:line 619
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at FooLib.Library.FirestoreMappings.FooFirestore.<GetNextUnparsedFooInfo>d__4.MoveNext() in C:\Foo\FooLibFirestore\FooLib.Library\FirestoreMappings\FooFirestore.cs:line 51
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at FooLib.Library.Repositories.FooRepository.<GetNextUnparsedFooInfo>d__9.MoveNext() in C:\Foo\FooLibFirestore\FooLib.Library\Repositories\FooRepository.cs:line 102
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at FooLib.WebApi.Controllers.FoosController.<GetNextUnparsedFooInfo>d__8.MoveNext() in C:\Foo\FooLibFirestore\FooLib.WebApi\Controllers\FoosController.cs:line 83
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at lambda_method(Closure , Object )
   at Microsoft.Extensions.Internal.ObjectMethodExecutorAwaitable.Awaiter.GetResult()
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeActionMethodAsync>d__12.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeNextActionFilterAsync>d__10.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeInnerFilterAsync>d__14.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeNextResourceFilter>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeFilterPipelineAsync>d__17.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeAsync>d__15.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Builder.RouterMiddleware.<Invoke>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.<Invoke>d__7.MoveNext()

Making sure to follow these steps will guarantee the quickest resolution possible.
Thanks!

Thanks for the report - I'll look into it tomorrow.

In the meantime, if you could tell me what you're observing, that would help. Does the code show an exception? Fail to find anything? Fail to filter out anything?

I am trying to filter a collection based on a Property (of type boolean) and the query times out instead of finding the document
I updated the original with stack trace.

That's very helpful, thanks.

I can't currently reproduce this. Here's a console application which sounds like it's doing what your code is doing
using Google.Cloud.Firestore;
using System;
using System.Threading.Tasks;

class Program
{
    static async Task Main(string[] args)
    {
        var db = FirestoreDb.Create(args[0]);

        var collection = db.Collection(Guid.NewGuid().ToString());

        await collection.AddAsync(new { Name = "First true entry", Include = true });
        await collection.AddAsync(new { Name = "First false entry", Include = false });
        await collection.AddAsync(new { Name = "Second true entry", Include = true});
        await collection.AddAsync(new { Name = "Second false entry", Include = false });

        var query = collection.WhereEqualTo("Include", true);
        var querySnapshot = await query.GetSnapshotAsync();
        foreach (var document in querySnapshot)
        {
            Console.WriteLine(document.GetValue<string>("Name"));
        }
    }
}

Output:
First true entry
Second true entry

Could you try this code locally, and see whether it works for you? If this works, please could you try amending it to query your existing collection?

Could you try this code locally, and see whether it works for you? If this works, please could you try amending it to query your existing collection?

Hi John I tried your code by creating a new Console app and it worked with Cloud Firestore (not emulator, as I have credential file already), it created a new Collection with guid and got results properly.
I just changed the code to point to the collection I have and with boolean flag and it times out.  I have around 200K documents, I am expecting that shouldn't be any concern, but can it be?
       var db = FirestoreDb.Create("projectmusic-id");

        var collection = db.Collection("CollectionName");
        var query = collection.WhereEqualTo("SomeFlag", true);
        var querySnapshot = await query.GetSnapshotAsync();
        foreach (var document in querySnapshot)
        {
            Console.WriteLine(document.GetValue<string>("Name"));
        }

I tried querying for string type and it works on the same collection.

Okay, that's good to know, thank you. I shall have to ask the Firestore team about that.

One thing to clarify - how many of those 200,000 documents would match the filter? I'm trying to work out whether this is a matter of "it takes a long time to get back a snapshot for a huge number of documents" or whether it's really about the filtering itself. I'll try to reproduce tomorrow with 200k documents - if I know how many of those should match, that would be great.

99.9% of the documents have the flag.
Yeap,  its must be timing out as there are lot of documents that match the filter.  When I tried to find a document which has null value (for example) the query didn't time out as it is minority.

Right. Out of interest, do you see the same timeout if you just get a snapshot of the collection, unfiltered? I honestly don't know whether you should expect it to work.

 var collection = db.Collection("CollectionName");
 var querySnapshot = await collection.GetSnapshotAsync();

The above does fail.
What else should I do if I want to fetch documents in paginated manner, skip some and fetch some,  I am interested in no more than 100 documents at a time.

I'll have to look at what the options are - I'm not deeply familiar with using the Firestore API (as I'm not a specialist in specific APIs, just the client libraries). I'll have a look tomorrow. I believe you'll want to specify a limit on one call, then specify a "start at..." cursor for the next query.

Note: if your app is something new to help with the impact of the current health emergency, let me know and I'll try to look tonight. Otherwise, it'll be tomorrow (UK time).

Note: if your app is something new to help with the impact of the current health emergency, let me know and I'll try to look tonight. Otherwise, it'll be tomorrow (UK time).

Not related to emergency

I'll have to look at what the options are - I'm not deeply familiar with using the Firestore API (as I'm not a specialist in specific APIs, just the client libraries). I'll have a look tomorrow. I believe you'll want to specify a limit on one call, then specify a "start at..." cursor for the next query.

Thank you, the following worked,  do you see any issues
       var db = FirestoreDb.Create("projectid");

        var collection = db.Collection("CollectionName").Limit(10);
        var query = collection.WhereEqualTo("SomeFlag", true);
        var querySnapshot = await query.Offset(10).GetSnapshotAsync();
        foreach (var document in querySnapshot)
        {
            Console.WriteLine(document.GetValue<string>("Name"));
        }

I suspect using Offset like that will become very slow when you get to much later "pages" of the results. I'll write up a complete test of this later on.

(Removed the bug label as I suspect there's no bug here, just a matter of the best API usage. Currently populating a large collection...)

Okay, I've now had time to put together a complete program, which is shown below. It shows three ways of fetching the query results:

Using the StreamAsync method. If you're using C# 8 and can depend on the 2.0.0-beta01 or 2.0.0-beta02 versions, this is the simplest option to use (with caveats)
Executing queries multiple times, using a DocumentSnapshot as a cursor
Executing queries multiple times, using an offset

Results:

StreamAsync worked well to start with, fetching 210K records in just over a minute - but it then timed out. I'm not entirely sure why; I suspect something somewhere is asserting a one minute timeout, but I don't know what.
Querying with cursors: this worked very well, fetching nearly as fast as streaming, but completing - it counted 910183 items in 330s
Querying with offsets: this slowed down really quickly. I stopped it after a while - after five minutes it had only counted 60,000 items, and was getting slower and slower

Increasing the limit (to perform fewer queries) would no doubt help make the offset approach work better, but using cursors is clearly better. If your code wouldn't make it easy to use DocumentSnapshot, you can create a cursor in other ways, but it'll take more work. (You'll need to specify that you're ordering by document ID, then specify the document ID as the cursor.)
I'm going to close this issue now as I believe I've addressed everything - please add a comment if you need more help though.
using Google.Cloud.Firestore;
using System;
using System.Diagnostics;
using System.Linq;
using System.Threading.Tasks;

class Program
{
    static async Task Main(string[] args)
    {
        var db = FirestoreDb.Create(args[0]);

        // Note: comment this out after the first run!
        // It actually failed for me due to too much contention, but only after it had written about 900K documents.
        var collection = db.Collection("Issue4561-attempt2");

        await PopulateCollectionAsync(collection);
        await CountViaStreaming(collection);
        await CountViaCursors(collection);
        await CountViaOffsets(collection);
    }

    static async Task CountViaStreaming(CollectionReference collection)
    {
        Console.WriteLine("Counting using query streaming");
        var stopwatch = Stopwatch.StartNew();
        int count = 0;
        var stream = collection.WhereEqualTo("Include", true).StreamAsync();
        await foreach (var item in stream)
        {
            count++;
            MaybeLogCount(count);
        }
        stopwatch.Stop();
        Console.WriteLine($"Counted {count} items in {(int) stopwatch.Elapsed.TotalSeconds}s");
    }

    static async Task CountViaCursors(CollectionReference collection)
    {
        Console.WriteLine("Counting using cursors");
        int limit = 1000;
        var query = collection.WhereEqualTo("Include", true).Limit(limit);
        var stopwatch = Stopwatch.StartNew();
        int count = 0;

        DocumentSnapshot lastDocument = null;
        while (true)
        {
            var queryWithCursor = lastDocument is null ? query : query.StartAfter(lastDocument);
            var querySnapshot = await queryWithCursor.GetSnapshotAsync();
            foreach (var document in querySnapshot)
            {
                count++;
                MaybeLogCount(count);
                lastDocument = document;
            }
            if (querySnapshot.Count != limit)
            {
                break;
            }
        }
        stopwatch.Stop();
        Console.WriteLine($"Counted {count} items in {(int) stopwatch.Elapsed.TotalSeconds}s");
    }

    static async Task CountViaOffsets(CollectionReference collection)
    {
        Console.WriteLine("Counting using offsets");
        int limit = 1000;
        var query = collection.WhereEqualTo("Include", true).Limit(limit);
        var stopwatch = Stopwatch.StartNew();
        int count = 0;

        while (true)
        {
            var queryWithOffset = query.Offset(count);
            var querySnapshot = await queryWithOffset.GetSnapshotAsync();
            foreach (var document in querySnapshot)
            {
                count++;
                MaybeLogCount(count);
            }
            if (querySnapshot.Count != limit)
            {
                break;
            }
        }
        stopwatch.Stop();
        Console.WriteLine($"Counted {count} items in {(int) stopwatch.Elapsed.TotalSeconds}s");
    }

    static void MaybeLogCount(int count)
    {
        if (count % 10000 == 0)
        {
            Console.WriteLine($"{DateTime.UtcNow:HH:mm:ss} Fetched {count} items");
        }
    }

    static async Task PopulateCollectionAsync(CollectionReference collection)
    {
        // Execute 100 parallel threads, each inserting 10,000 items in batches of 100.
        var tasks = Enumerable.Range(0, 100)
            .Select(chunk => InsertChunkAsync(chunk))
            .ToList();

        await Task.WhenAll(tasks);

        async Task InsertChunkAsync(int chunk)
        {
            for (int batch = 0; batch < 100; batch++)
            {
                var writeBatch = collection.Database.StartBatch();
                for (int i = 0; i < 100; i++)
                {
                    var id = i + (batch * 100) + (chunk * 10_000);
                    bool included = id % 1000 != 0;
                    var item = new { Name = $"Entry {i}", ItemId = id, Include = included };
                    writeBatch.Create(collection.Document(), item);
                }
                await writeBatch.CommitAsync();
                Console.WriteLine($"Completed batch {chunk} / {batch}");
            }
        }
    }
}

Thank you Jon for all the samples, very helpful, here are my thoughts
Tried CountViaOffsets and when the offset is huge, it timesout eventually.
CountViaStreaming: Looks very clean, but my concern is Firestore charges based on read operations, and as we can't offset directly, may get charged for all
CountViaCursors:  Seems to be better option.

Guys,
i have a collection of 350.000 documents, i want to update 1 value of all documents and i'm tryint this to get all collection:
ApiFuture<QuerySnapshot> future = db.collection("DI/FACL/elitecustomers").get();         List<QueryDocumentSnapshot> documents = future.get().getDocuments();
But, i have this error:
Caused by: java.util.concurrent.ExecutionException: com.google.api.gax.rpc.UnavailableException: io.grpc.StatusRuntimeException: UNAVAILABLE: The datastore operation timed out, or the data was temporarily unavailable.
Can you help me please

@DiegoECQ: Right, so you'd need to use one of the techniques shown above. Pulling all 350K documents into memory in one query as you're currently trying to do simply isn't going to work.

@PopeJuliusII: Due to the timeouts of the service, I believe. (Note that Diego's message was about Java anyway, so is really off-topic for this repo...)

@jskeet Thanks for answering! Was just curious! :)


Shadowing I not working

By default I means the identity matrix, but we should be able to shadow/obscure it with a parameter. That is,
given
I ∈ ℝ²

I_1+I_2

should compile the same as
given
X ∈ ℝ²

X_1+X_2

instead the former gives the error:
Error on line 4 at column 1. You can't use 'I' with subscript since it has been defined before.
I_1+I_2
^

I believe we haven't got to the indexing rule when implementing this.
The simple case will work.
given
I ∈ ℝ

I + I

I'll fix later

Fixed


warn deprecated<EMAIL_ADDRESS>request has been deprecated, see https://github.com/request/request/issues/3142

Please kindly asking you to help me to use your app. For first time to install the ewelink-api on Node.js 12.16.2 (x64) I got this issue and so on:
warn deprecated<EMAIL_ADDRESS>see the logs attached or link
https://drive.google.com/open?id=1RdjJI_8KTgY-HBs7kVfsp04O8SVyQX8I
Also please sorry I'm beginner any tutorial will be great
_logs.zip

thanks for the notice @xo555
request library will be replaced with node-fetch on version 3.0


add --debug to testenv operations run by buildbot

Issues:
Fixes #609
Problem: testenv logs generated by buildbot are lost
after the run
Analysis: by adding the debug flag we set testenv to
send logs to stdout where they will be caught by the
buildbot reporting tool
Tests: The output of the testenv commands will now show up in
nightly.

Do we intend to ever remove this? Could the output be too much to reasonably read through?

It will increase the information in the log file.

The idea is that you run this manually and you don't see any issue that you suspect may occur in nightly? So you're putting this here to figure out problems when things actually go wrong?


Fix the links to the examples for <IP_ADDRESS> UDP and JSON formats.

Quick fix to the URLs for the UDP and JSON formats for the using the API for <IP_ADDRESS>.

Thanks for catching this @evanderkoogh. I'm gonna go ahead and merge it into production. :)


Select without parsing dates

My example:
var query = 'SELECT * FROM mytable;';
new sql.Request().query(query).then(function(recordset) {
  if(callback) {
	callback(null, recordset);
  }
  console.log("finished");
}).catch(function(err) {
  console.log(err);
  callback(err, null);
});

If I have an SQL Server column as Date, mssql converts it to the JavaScript Date object. Am I right?
Is there a way to get everything as string, 'as-is'?

You can define this in your statement. For example: select convert(varchar, getdate(), 126).

Let me explain my situation.
I have about 10 databases which I'm importing from SQL Server to MongoDB on a regular basis. All I can do from these databases is SELECT * FROM db1 ... dbN. I don't know which attributes they have, how many attributes they have, etc.
Is there a quickfix to just ignore every type-checking and just select everything as a string?

You convert did the job. I created a SQL View and now I'm selecting the view instead of table but it will still be nice to have an option to turn-off type-checking and conversion to JavaScript Date.
Once again, thank you. :)

Just an addendum/comment:
SQL Dates aren't strings though.. they're numeric representations of a point in time based on the implementation type specifics.  Beyond that the date may be stored with or without timezone specific information and without may either be the SQL server's offset or from the application server, or corrected to UTC... without specifics on the application or from the database, there's no way of knowing generically.  So even in converting to strings, you may well see other issues.


Removed unused enter/exit methods of MySQL's CursorWrapper.

Unused since their introduction in e1d839237f7ce38ef078b7f09cc3a1aeaacc02f0.

@manfre, do you remember anything about this? I didn't see an explanation in the original PR: https://github.com/django/django/pull/2232. Not sure if test coverage is missing or if this is not needed.

@timgraham What do you mean by unused? Isn't it used if an enduser does use:
with mysql_cursor:
   cursor.execute(…)

?

There are with connection.cursor() calls in the test suite -- that does not execute these lines. If there's some other way to obtain mysql_cursor, then please elaborate and I'll add a test. I think there's some confusion that this CursorWrapper class is similar to or takes the place of django.db.backends.utils.CursorWrapper but that's not the case.

Oh, yeah I mixed those up as well :/


Message GeoSpatialValue Retreival

From @ericbmartin on May 8, 2017 17:50
I am using the NOAA grib for Apparent Temp located here:
ftp://tgftp.nws.noaa.gov/SL.us008001/ST.opnl/DF.gr2/DC.ndfd/AR.conus/VP.001-003/ds.apt.bin
Based on the "Iterating lat/lon/value:" example on the main page, I have this code fragment:
                using (GribFile gribFile = new GribFile(m_file_name))
                {
                    foreach (GribMessage hour in gribFile)
                    {
                        foreach (GeoSpatialValue geoLocValue in hour.GeoSpatialValues)
                        {
                            if (geoLocValue.IsMissing) { continue; }

                            // Read and convert the Lat and Lon
                            double lat = geoLocValue.Latitude;
                            // GRIB2 Spec says Longitude is between 0 and 360, so subract from 180 to get to a usable component
                            double lon = 180D - geoLocValue.Longitude;

                            DateTime time = hour.Time;

                            double apt = geoLocValue.Value;
                        }

When I run this code, geoLocValue.Value returns the same "value" for every Lat/Lon, for example 308.7000, which I had the impression this would be the apparent temp (in Kelvin per the spec) for that lat/lon, so it should vary between most geo locations - this seems to be the file's MAX VALUE entry if I print out the keys/values as shown in the main page example code.  I also tried using the index/RAW VALUES code example on the main page withe the same results.  There is similar weirdness using other NOAA grib files in the above mentioned FTP site where the values are not specifically repeated, but values that are far out of reasonable range for the data set.  Is there some other mechanism I should be using to retrieve data at a given geo location?  I am using GribAPI.NET <IP_ADDRESS> code.
Copied from original issue: 0x1mason/GribApi.NET#57

From @ericbmartin on June 12, 2017 10:23
Has any one else been able to successfully retrieve data values using a similar mechanism?

From @IdahoSixString on June 12, 2017 17:15
@ericbmartin I have. However, if you are having issues could you upload your specific grib file and when I have some time I can look into it for you.

From @ericbmartin on June 12, 2017 18:20
The file I have from NOAA is a 34MB file, so I cannot upload it here, however, you can retrieve a similar file from the NOAA website directly:
ftp://tgftp.nws.noaa.gov/SL.us008001/ST.opnl/DF.gr2/DC.ndfd/AR.conus/VP.001-003/ds.apt.bin
Look at the geoLocValue.Value as shown in my original post to see if it is the same value for every entry.  If you are able to get variying values, I'd like a code sample to try.
Thanks!


Update Namespace names to follow DNS label format

Related issues: #8010 #6666
cc @bgrant0607 @thockin

cc @ArtfulCoder

Comments addressed. PTAL @thockin

ping

@thockin: Code updated.

a couple nits

Fixed.

LGTM, merge on green.  Self-merge if I forget.


When I install and use ccls, and do :LspInfo, will error.

Problem description
I use bashls, cmake, jsonls, pytight, sumneko_lua, they can work fine. But when open a cpp file, and do LspInfo, will error.Like nvim a.cpp and :LspInfo
E5108: Error executing lua vim/shared.lua:0: s: expected string, got function
stack traceback:
        [C]: in function 'error'
        vim/shared.lua: in function 'validate'
        vim/shared.lua: in function 'gsplit'
        vim/shared.lua: in function 'split'
        ...packer/start/nvim-lspconfig/lua/lspconfig/ui/lspinfo.lua:68: in function 'make_config_info'
        ...packer/start/nvim-lspconfig/lua/lspconfig/ui/lspinfo.lua:213: in function <...packer/start/nvim-lspconfig/lua/lspconfig/ui/lspinfo.lua:14
7>
        .../site/pack/packer/start/nvim-lspconfig/lua/lspconfig.lua:19: in function <.../site/pack/packer/start/nvim-lspconfig/lua/lspconfig.lua:18>

        [string ":lua"]:1: in main chunk

Why do you think this is an issue with nvim-lsp-installer?
I read the error message, I think this might be an issue about nvim-lsp-installer and nvim-lspconfig.
Neovim version (>= 0.7)
NVIM v0.8.0-dev
Build type: RelWithDebInfo
LuaJIT 2.1.0-beta3
Operating system/version
Linux HzBook <IP_ADDRESS>-microsoft-standard-WSL2 #1 SMP Fri Apr 2 22:23:49 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
I've manually reviewed the Nvim LPS client log (:LspLog) to find potential errors

[X] Yes

I've recently downloaded the latest plugin version of both nvim-lsp-installer and nvim-lspconfig

[X] Yes

Affected language servers
ccls
Steps to reproduce

nvim a.cpp
:LspInfo

Actual behavior
error.
Expected behavior
show be show the floating window.
LspInfo
E5108: Error executing lua vim/shared.lua:0: s: expected string, got function
stack traceback:
        [C]: in function 'error'
        vim/shared.lua: in function 'validate'
        vim/shared.lua: in function 'gsplit'
        vim/shared.lua: in function 'split'
        ...packer/start/nvim-lspconfig/lua/lspconfig/ui/lspinfo.lua:68: in function 'make_config_info'
        ...packer/start/nvim-lspconfig/lua/lspconfig/ui/lspinfo.lua:213: in function <...packer/start/nvim-lspconfig/lua/lspconfig/ui/lspinfo.lua:14
7>
        .../site/pack/packer/start/nvim-lspconfig/lua/lspconfig.lua:19: in function <.../site/pack/packer/start/nvim-lspconfig/lua/lspconfig.lua:18>

        [string ":lua"]:1: in main chunk

LspLog
[START][2022-07-16 16:50:14] LSP logging initiated
[WARN][2022-07-16 16:50:14] .../lua/vim/lsp.lua:926     "server_request: no handler found for"  "workspace/diagnostic/refresh"
[WARN][2022-07-16 16:50:16] .../lua/vim/lsp.lua:926     "server_request: no handler found for"  "workspace/diagnostic/refresh"
[START][2022-07-16 16:51:30] LSP logging initiated
[WARN][2022-07-16 16:51:30] .../lua/vim/lsp.lua:926     "server_request: no handler found for"  "workspace/diagnostic/refresh"
[WARN][2022-07-16 16:51:30] .../lua/vim/lsp.lua:926     "server_request: no handler found for"  "workspace/diagnostic/refresh"
[START][2022-07-16 16:56:50] LSP logging initiated
[WARN][2022-07-16 16:56:50] .../lua/vim/lsp.lua:926     "server_request: no handler found for"  "workspace/diagnostic/refresh"
[WARN][2022-07-16 16:56:50] .../lua/vim/lsp.lua:926     "server_request: no handler found for"  "workspace/diagnostic/refresh"

Healthcheck
WARNING: The following servers have not been enhanced by nvim-lsp-installer:
cmake
ccls
jsonls
pyright
sumneko_lua
bashls
Make sure you set up nvim-lsp-installer before you set up these severs.

Screenshots or recordings
No response

Hello! This is a problem with the lspconfig plugin, I've created a PR over there to fix this.


Greenkeeper/serverless plugin typescript 1.1.9

Coverage remained the same at 100.0% when pulling 831859c5fddd6b111f93a5a6ef77ed98e7ab08a7 on greenkeeper/serverless-plugin-typescript-1.1.9 into 1130b42fc05f03a51da0f7fd18e5b5ccd30ca954 on master.


The influence of the "gardening" module on other mods

In addition to dealing with cacti and reeds, the mod affects other mods as well

Hmm... based on the current implementation, this is intentional. Could add a config for it though...


Suffix model version

Codecov Report

Merging #122 (4967653) into main (90d3692) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #122   +/-   ##
=======================================
  Coverage   81.97%   81.97%
=======================================
  Files          56       56
  Lines        3384     3384
=======================================
  Hits         2774     2774
  Misses        610      610

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 90d3692...4967653. Read the comment docs.


Create CDN to images

Imgur blocked Discord, this makes all image commands down for an hour.
Create a way to host the images by myself, maybe in base64 to put them all in a database, and cache them to send them faster

Created in https://github.com/MenheraBot/MenheraAlbum


<IP_ADDRESS> Two General Linux Guides.

Main Changes.

New Section for some Linux stuff, with two new guides.

Sudo rights
NTP Sources
Main meta.json edited to show the new section.

Minor work on docker-cli guides

Possible Issues/Questions.
I'm not entirely sure where to put the new guides, General Linux seems appropriate?
I'm not quite following style guide and other guides, I have added a section or two that may be removed.
The new section doesn't have an intro yet, as I don't think it needs one.

Possible Issue.
vim as the editor in guides.


What is the Mircosoft's roadmap for app center development

Does Microsoft have a new application in the pipeline has there haven't been any updates for nearly a year to the app center  , particularly the annoying feature of not being able to refresh an intune distribution token not being resolved #407

Hi @UniperMaster
At the moment, our team is focused on the goals of the roadmap.


[Hive] Use SQL to limit fetch size is not working

I am running Hive on Spark.
I need to use hive with option hive.fetch.task.conversion=none . So all SQL commands will run within an Apache Spark application.  Howerver, the Data Editor will run SELECT * FROM [table] without LIMIT clause and this take a long time to run this job.
Therefore, I have enabled Use SQL to limit fetch size in Data Editor setting but the SELECT * query still didn't append LIMIT clause. I think this is a bug of Dbeaver.

Hi @akizminet ,
unfortunately, we've got no environment to test for now. Cannot reproduce it against postgresql.
Could you check that there is no limit clause in Query Manager? Thanks in advance

@uslss Yeah. I have also tested with PostgreSQL and it worked well. PostgreSQL SELECT query had LIMIT clause. Hive still didn't have.

thanks for the bug report and clarification

Fixed.

verified


The unit of intensity

I want to know if you use:

mag/arcsec^2/mag/pixel^2 (with -2.5log)
W/m^2/sr (without -2.5log)

for $I(r)$ in Sersic profile:
$$
I(r) = I(0)\exp(r/r_0)^{1/4}
$$

You can see the exact equation used when we say "sersic" here: https://github.com/pysersic/pysersic/blob/main/pysersic/rendering.py#L716
You can see that our amplitude normalization is tied to the total flux (note: some define the reference point as the flux at $r_e$, we do not).  Also note that we do not require (or assert) that the pixel units of the input image be "real" in any sense, and we do not carry out any internal conversions -- you will get out a model that fits the pixel counts in a given image, whether your pixels have been zero-pointed to real units or not.


Fix yarn command in git upgrade blog post

The yarn global add needs to have global before add, it will otherwise just install in in the local project together with the global package.
https://yarnpkg.com/en/docs/cli/global

LGTM

@jeremistadler Thanks for the PR.
@ncuillery Thanks for the review.


Longformer Encoder/Decoder model pretraining query

I'm really interested in the work you've done to adapt the longformer to an encoder/decode architecture. I'd like to use it in some work I'm doing. My main motive is the memory reductions of both the linear scaling of the longformer and the built-in gradient checkpointing in your version as I'm constrained to using 12GB GPUs which make using the standard BART large difficult with a reasonable batch size.
The questions is are the listed models i.e. https://ai2-s2-research.s3-us-west-2.amazonaws.com/longformer/longformer-encdec-base-16384.tar.gz pretrained for summarisation or just converted from the original BART models? My use case is general text generation so would want to use the original BART weights. Just thought I'd check as I see you have a conversion script as well which I can use if they are for summarisation.

It is the original BART.


there is an error when trained on ResNet-50 model

I trained the network on the pre-trained model Resnet 50, which is downloaded from http://data.mxnet.io/models/imagenet/resnet/50-layers/resnet-50-0000.params.
However, there is an error "AssertionError: conv1_weight not initialized".
I think it may be the problem of the pre-trained model Resnet 50.
Can anyone share the useful pre-trained model Resnet 50 with me, or help me find the problem?
Thank you very much.

    def init_weight(self, cfg, arg_params, aux_params):
        for k in self.arg_shape_dict:
            if 'weight' in k and k not in arg_params:
                arg_params[k] = mx.random.normal(0, 0.01, shape=self.arg_shape_dict[k])
            if 'bias' in k and k not in arg_params:
                arg_params[k] = mx.nd.zeros(shape=self.arg_shape_dict[k])
            elif k not in arg_params:
                arg_params[k] = mx.random.normal(0, 0.01, shape=self.arg_shape_dict[k])

just do like this


The "shell" command is very slow to connect

Running lfr sh display Connecting to localhost:11311... for a couple of minutes before giving back the Gogo Shell.

I don't know where the slowness comes from. I did some tests only using the standard libraries, and it sill hangs for a minute after a terminal type negotiation.
I'm marking this as won't fix but still hope to solve this one day.


Find a diagram element on a DRD using text search (ctrl-f)

Is your feature request related to a problem? Please describe.
This is related to the issue described in https://github.com/camunda/camunda-modeler/issues/950, which occurs when lots of DRD elements are modeled on the same DRD, making it hard to find a specific one quickly.
Describe the solution you'd like
Apart from the solution using boxed contexts as described in https://github.com/camunda/camunda-modeler/issues/950, we would also like to be able to search for a specific DRD element, similar to the bpmn diagram search, i.e.:

ctrl-f -> display text box
type text -> display drop-down list with matching diagram elements by label or id.
hoover over drop-down list item -> select/highlight DRD element on diagram + properties panel
select item

So you basically want the same functionality as in bpmn-js?

This is totally understandable. Unfortunately dmn-js is a bit behind in terms of features in the DRD view.


[FIX AVAILABLE] Sometimes when creating a shape and extruding it, you only see the outlines until you rebuild

Seems as if CSG brushes aren't updated at all while extruding for some reason, but only some of the time.
Still investigating

Fixed in version 1.202, available on beta site

Fix available on asset store!


Change yaml files to support Citadel monitoring

Citadel monitoring is enabled by https://github.com/istio/istio/pull/5015.

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: ijsnellf
Assign the PR to them by writing /assign @ijsnellf in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

install/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@myidpt: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-presubmit.sh
8ab881e7e04734fa4fb2986e273fe6706a5b1860
link
/test istio-presubmit

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@myidpt: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/e2e-bookInfoTests-v1alpha3.sh
5da32d81537adb6f30e0d873bcd9036bedeab3d4
link
/test e2e-bookInfo-envoyv2-v1alpha3

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@myidpt: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/e2e-bookInfoTests-v1alpha3.sh
5da32d81537adb6f30e0d873bcd9036bedeab3d4
link
/test e2e-bookInfo-envoyv2-v1alpha3

prow/istio-unit-tests.sh
cf0b3fcee32a8a195ea29dc8d57400a263211b6a
link
/test istio-unit-tests

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@myidpt: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/e2e-bookInfoTests-v1alpha3.sh
5da32d81537adb6f30e0d873bcd9036bedeab3d4
link
/test e2e-bookInfo-envoyv2-v1alpha3

prow/istio-unit-tests.sh
cf0b3fcee32a8a195ea29dc8d57400a263211b6a
link
/test istio-unit-tests

prow/istio-pilot-e2e.sh
cf0b3fcee32a8a195ea29dc8d57400a263211b6a
link
/test istio-pilot-e2e

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@myidpt: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-unit-tests.sh
cf0b3fcee32a8a195ea29dc8d57400a263211b6a
link
/test istio-unit-tests

prow/istio-pilot-e2e.sh
cf0b3fcee32a8a195ea29dc8d57400a263211b6a
link
/test istio-pilot-e2e

prow/e2e-bookInfoTests-v1alpha3.sh
cf0b3fcee32a8a195ea29dc8d57400a263211b6a
link
/test e2e-bookInfo-envoyv2-v1alpha3

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@myidpt: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-pilot-e2e.sh
cf0b3fcee32a8a195ea29dc8d57400a263211b6a
link
/test istio-pilot-e2e

prow/e2e-bookInfoTests-v1alpha3.sh
954a1a55dae04421be9b9ee548b3b9069179ea57
link
/test e2e-bookInfo-envoyv2-v1alpha3

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.


module number

Hi,
PyWGCNA can determine the module numbers automatically now, how to adjust the number if I wanna increase or decrease the modules ?

Hi,
Every parameter has a default value similar to the original WGCNA. you can change them to what suits you!
for example, one of the parameters related to the number of modules is MEDissThres. right now the default is 0.2, so if you want more modules, you can increase this number.


Vulcan Oculus Quest 2 capture hangs at ~10-20% mark

Description
Pressing the capture button when using the Vulkan API freezes the game and hangs the progress bar at the 10-20% mark. Vulkan is stated as "Active" in RenderDoc initially, but ~5 seconds into the capture, it shifts to "Not Presenting." Launching the game from the RenderDoc Launch Menu also results in "OVRMetricsTool keeps stopping." popping up repeatedly if running OVRMetricsTool in the Oculus Quest. Force quitting it allows the game to load, but the above vulkan capture issue persists. I've tried this in the latest stable, nightly RenderDoc, and in the RenderDoc For Oculus 1.16 fork. Uninstalled and reinstalled numerous times. And enabled and disabled the following player settings in Unity: Optimize Buffer Discards (Vulkan), Phase Sync, Subsampled Layout (Quest 2/Vulkan), Late Latching (Vulkan). Disabling/enabling all of them and/or a combination does not seem to fix the issue.
Steps to reproduce

Build my Unity game for Android with the following settings:

Texture Compression: ASTC
ETC2 fallback: 32bit
Export Project: disabled
Build App Bundle: disabled
Create Symbols.zip: disabled
Development Build: enabled
Autoconnect Profiler: enabled
Deep Profiling Support: disabled
Script Debugging: disabled
Scripts Only Build: disabled
Compression Method: LZ4

Open Renderdoc 1.16 (stable)/1.17 (nightly)
Switch replay context to Oculus Quest 2
Wait for quest 2 server to start
Once connected, launch recent Quest 2 game build
If running OVRMetricsTool, force quit 3-4 times until game resumes loading
Once game has loaded in, press capture button
Game freezes and capture progress halts at 10-20% mark.

Environment

RenderDoc version: 1.16/1.17
Operating System: Windows 11
Graphics API: Vulkan

GPU is Nvidia RTX 3090 with the latest Nvidia Driver. CPU is AMD Ryzen 7 5800X, Memory: 64gb 3200mhz ram.

Please share your program so that I can run it. If you don't, it is impossible for me to follow your steps to reproduce.

Here it is! Looking forward to any potential findings :)
https://drive.google.com/file/d/1CR351VCcoKeoEP_LDkINhnbeGRegffC_/view?usp=sharing

Hey Abdel!
Could you package the APK with MSAA set to 1x (off) and both see if that fixes, and also send us the updated no-MSAA APK?
For what it's worth, subsampled isn't really supported properly on Renderdoc-Vulkan yet and should be off.

Hey @tuxerr, I've been trying to get my build running so that I can test that for you, but it's not loading. I'll get you that build when I get it to load successfully.

@tuxerr disabled MSAA, but oddly there is now this new issue with running the project through RenderDoc. It seems like some texture is transposed right in front of the camera. This issue doesn't appear when running the app through Unity's build and run or through the Oculus 'unknown apps' menu. Also, it still allows one screenshot, although possibly with incorrect information. It now freezes as I initially described upon the second screenshot.. bizarre. Here's a link to the latest build: https://drive.google.com/file/d/14aVrMVb6OByqxMKExi4iR0LgsjQ4_ozL/view?usp=sharing

I'm not currently able to run any programs on my quest 2 so I can't investigate this, hopefully either tuxerr can help or I can fix whatever is stopping me running programs.

OK I'm finally able to run programs again. The original repro seems to be hanging in the driver, but after it runs out of memory anyway (so even if it didn't hang it would likely either crash, or at best fail to capture/produce a broken capture). The new version does capture for me, however I notice that both are built as 32-bit programs. I know the memory available on mobile devices is limited so this won't make a ton of difference, but you still may have VA exhaustion or fragmentation issues so I'd suggest building as 64-bit.
The capture that I can make from the second one does crash on load - it seems like there's some missing code to remove the subsampled bit on replay which should be fixed in that commit, after which it does load and replay correctly for me:

I don't see any texture issues, but maybe if it's subtle and I'm not sure what it's supposed to look like I'm missing it. Can you explain what you mean about a transposed texture near the camera? do you mean the ground texture is reversed or something?
I also don't understand at all what you mean about screenshots. Is there some screenshot functionality which isn't working when used with RenderDoc?

Wish I could get Vulcan working with 64 bit, but building on it with the latest Oculus XR leads to the game build loading infinitely, but I guess that's an issue for another team to handle. But even the 32 bit was running semi-okay on it own. Does Renderdoc use a lot of the Quest's memory?
Glad you resolved the capture crashing on load. Also my apologies, poor choice of words earlier. What I meant about the textures is that I get this strange issue where the textures are zoomed up very close to the eyes to the point where all I see is green or blue or white as I turn my head. Strange thing is that only happens when loading the game with RenderDoc. Running the build from the Oculus Quest menu worked fine.
Also by screenshot, I meant capture :'] Even with the textures plastered on my eyes, I could still capture a frame one, but the second time would result in RenderDoc hanging and my app crashing. But if RenderDoc is consuming a lot of memory on it's own and the same could be said for my project, I could be running out of memory on the second capture (but if the first works fine, and the second crashes, could that be indicative of a memory leak?)

@AbdelNabut, do you mind doing a few things to figure out/fix the bug on the Unity end? Also, we should probably move this conversation to Unity or Oculus forums, not Renderdoc, now that it's no longer a Renderdoc issue.
Can you:

Disable post-processing in your Unity app, and then create a new 64-bit build? You look to be doing quite a bit of post-processing in there, not to mention that this will be extremely expensive since it doesn't use the Vulkan subpass system anyways, but that's a separate topic.
Ideally you won't get the crash if you disable post-processing entirely. I'd strongly recommend removing it entirely as it probably is eating away an absurd amount of your total per-frame GPU budget here.
If 1) fails, can you share an apk of your app after 1), i.e. still with no post-processing at all?

And in general, are you doing anything special with the SRP, like a lot of SRP code changes, or mostly just standard stuff? Are you trying to use the Vulkan/SRP subpass API, etc. or something like that?
To avoid cluttering this Renderdoc forum though, you might want to create a forum post either on the Unity or Oculus forums and we can have this conversation there.

The memory overhead is variable and depends on what the application is doing, but yes there is a certain amount of overhead so if the application is using all available memory it will fail. Yes I would definitely recommend checking your code to see why 64-bit builds are failing, and following the suggestions above.
By VA exhaustion I meant either through fragmentation or through high memory use you might have no more space in the process's virtual address (VA) space left to allocate, and so memory allocations may fail. On 32-bit even if more physical memory is available it may not be possible to map any more since you only have (at absolute most) 4GB of VA available, and in practice a fair bit less considering ranges that may be reserved.
I don't see any kind of obvious visual corruption. When I run from RenderDoc I get just a normal view of some heightmap terrain and two hand models for the controllers. I'm not sure why you'd be seeing anything broken but I can't reproduce that.
I do get a failed capture the second time, but it's also out of memory. That could be fragmentation on a 32-bit build, it's not necessarily a memory leak.
Since the only RenderDoc issue seems to be fixed, I'm closing this.


docs/usage: New file

Document offline USB upgrades.

Just noting here for ourselves...right now we have this super simplistic model where everything is in a container image, and that's it.
But in the future, we will need a more sophisticated mirroring setup to add things like https://github.com/containers/bootc/issues/22 and https://github.com/containers/bootc/issues/128 - like OpenShift, we may end up needing something like bootc image copy registry:quay.io/exampleos/someos:latest oci:example.oci.


wasm: set upper limit for maximum Wasm memory size when using V8.

Signed-off-by: Piotr Sikora<EMAIL_ADDRESS>

@lizan PTAL. This blocks Proxy-Wasm C++ Host updates, since V8 flags are out-of-sync between Proxy-Wasm C++ Host and wee8_compile. I'm going to move wee8_compile to Proxy-Wasm C++ Host later, but I want to unblock it first.

/lgtm deps