Gold Price Prediction

Gold Price Prediction is a project that involves forecasting future gold prices using historical data and various machine learning or time series analysis techniques. The goal of this project is to build a model that can accurately predict the price of gold, which is influenced by a range of economic, financial, and geopolitical factors.
Please assign to me under GSSOC24.

Proposed Solution
The proposed solution for the Gold Price Prediction project involves a comprehensive workflow that includes data acquisition, preprocessing, feature selection, model training, and evaluation. Initially, historical gold price data will be sourced from reliable financial databases or APIs. The dataset will be enhanced with relevant economic indicators that may impact gold prices.
Data preprocessing steps will include cleaning the dataset, dealing with missing values through interpolation or imputation, and normalizing the numerical features to improve model training. Exploratory data analysis (EDA) will help visualize trends, seasonal patterns, and correlations between variables, guiding feature selection for the predictive model.
For the model, various algorithms will be considered, including:
Linear Regression: For establishing a baseline predictive performance.
Random Forest: To capture nonlinear relationships and interactions between features.
LSTM Networks: For leveraging the sequential nature of time series data and capturing temporal dependencies.
Once the models are trained, performance will be evaluated using metrics such as Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE). Additionally, backtesting strategies will be implemented to assess how the model would have performed in real market scenarios.
ADD LABELS GSSOC EXT 24 AND hacktoberfest
ASSIGN ME THIS WORK

I have already worked on this project . Please do assign me this project with suitable labels .
Below is a detailed readme of my work
Gold-Price-Analysis-with-(Time--1950-to-2020)-Time-Series-Forecasting-
Applied statistical and machine learning techniques including Linear Regression, Naive Model, and Exponential Smoothing. Visualized historical trends and seasonal patterns to generate accurate future predictions. Achieved robust forecasting performance with a comprehensive evaluation of model accuracy.
OVERVIEW:
This project involves a comprehensive analysis and forecasting of gold prices from January 1950 to August 2020. Using various statistical and machine learning techniques, we aim to understand historical trends and make future predictions.
Dataset The dataset contains monthly gold prices spanning from January 1950 to August 2020. The data was processed and analyzed using Python libraries such as Pandas, NumPy, Matplotlib, Seaborn, and Statsmodels.
METHODOLOGY:
Data Preprocessing:
Loaded the dataset and inspected its structure. Converted the 'Date' column to datetime format and set it as the index. Conducted exploratory data analysis (EDA) to visualize trends and patterns. Exploratory Data Analysis (EDA):
Visualized gold prices over time using line plots. Created box plots to analyze price distribution across years and months. Generated descriptive statistics to summarize the data. Resampled the data to visualize yearly, quarterly, and decadal trends. Time Series Decomposition:
Decomposed the time series data to identify trend, seasonal, and residual components. Modeling and Forecasting:
Linear Regression: Built a linear regression model to forecast future prices. Naive Model: Used the last observed value as the forecast for future periods. Exponential Smoothing: Applied the Holt-Winters method to account for trends and seasonality in the data. Evaluated models using Mean Absolute Percentage Error (MAPE) to compare accuracy.
KEY RESULTS:
Visualizations:
Line plots and box plots provided insights into price trends and distributions over the years. Resampling revealed the long-term trends and seasonal patterns in gold prices. Forecasting Models:
Linear Regression: Achieved a MAPE of 29.760%. Naive Model: Achieved a MAPE of 19.380%. Exponential Smoothing: Achieved a MAPE of 17.241%.
FINAL FORECAST:
The final forecast model using Exponential Smoothing provided a robust prediction of future gold prices, considering both trend and seasonality.
CONCLUSION:
This project demonstrates the use of statistical and machine learning techniques for time series analysis and forecasting. The visualizations and models developed provide valuable insights into historical gold price trends and offer reliable predictions for future prices.
Tools and Libraries:
Programming Language: Python Libraries: Pandas, NumPy, Matplotlib, Seaborn, Statsmodels, Scikit-learn
Future Work:
Incorporate external economic indicators to improve forecasting accuracy. Explore more advanced models such as ARIMA, SARIMA, and machine learning approaches like LSTM for time series forecasting.


Add phpstan to github actions workflow

Fixes https://github.com/symfony/demo/issues/1260
I tried checking different repos under https://github.com/symfony to see how those solve running phpstan and tried to apply that here. I saw that all the repos seem to install phpstan in the actions workflows instead of adding them to composer.json,so I did that, too. Also it seems like all of the projects I checked out are using the phpstan level 5, so that's what I was using. What I also found out is that most of them configured phpstan to use some phpunit autoload file, that's also why phpunit has to be installed in the workflow before running phpstan.
Most of the errors reported with this configuration are probably not that hard to solve, but I didn't want to include any unrelated code changes here, so I added a baseline file. I can also try to fix them if you think that should be part of this PR.
I think adding phpstan to the lint workflow sounds right, but I wasn't too sure if phpstan should be added to the 'linters' job or if it should get its own job. I decided to add it to the existing one to avoid having to duplicate all the composer/php setup steps into another job.
I hope that's something like what you expected to see regarding running phpstan for this project :)

Why not use this repository?
https://github.com/OskarStark/phpstan-ga
As it is done here:
https://github.com/symfony/demo/blob/main/.github/workflows/lint.yaml#L21

Why not use this repository? https://github.com/OskarStark/phpstan-ga
As it is done here for PHP CS Fixer: https://github.com/symfony/demo/blob/main/.github/workflows/lint.yaml#L21

@seb-jean sure, we could do that. What I did was searching through the symfony organization to see how it's done in other repos here, and I found e.g. these three which all install phpstan through shivammathur/setup-php@v2:

https://github.com/symfony/webpack-encore-bundle/blob/eeb893dec94ce5c471592588c945e364fe3ead09/.github/workflows/static.yml#L13
https://github.com/symfony/mercure/blob/aac27352a8f17ce867a6b63c4f64906afb787dc4/.github/workflows/static-analysis.yml#L15
https://github.com/symfony/panther/blob/ec2ff073e80d3efb986022a2937a9cd94310237d/.github/workflows/ci.yml#L31

So I thought solving it the same way it's already done in other symfony repos is probably a good idea, isn't it?

Why not use this repository? https://github.com/OskarStark/phpstan-ga
As it is done here for PHP CS Fixer: https://github.com/symfony/demo/blob/main/.github/workflows/lint.yaml#L21

@seb-jean sure, we could do that. What I did was searching through the symfony organization to see how it's done in other repos here, and I found e.g. these three which all install phpstan through shivammathur/setup-php@v2:

https://github.com/symfony/webpack-encore-bundle/blob/eeb893dec94ce5c471592588c945e364fe3ead09/.github/workflows/static.yml#L13
https://github.com/symfony/mercure/blob/aac27352a8f17ce867a6b63c4f64906afb787dc4/.github/workflows/static-analysis.yml#L15
https://github.com/symfony/panther/blob/ec2ff073e80d3efb986022a2937a9cd94310237d/.github/workflows/ci.yml#L31

So I thought solving it the same way it's already done in other symfony repos is probably a good idea, isn't it?

I prefer that more experienced people answer because I don't know what is the best solution.

@burned42 thanks for this contribution. Do you consider that it's "mergeable" or is there anything pending? Thanks!

@burned42 thanks for this contribution. Do you consider that it's "mergeable" or is there anything pending? Thanks!

@javiereguiluz if you don't expect me to try and fix the reported phpstan errors within this PR, then yes, I consider this mergeable as is :)


[FEATURE][DocBot] Move cluster bootstrapping into a class

Is your feature request related to a problem?
With cluster bootstrap expanding like it has we should consider moving this into a class to be referenced by other parts of the code.
The ordering of the bootstrapping pipeline should be handled in init instead of main. The class should expose attributes for each of our variables in model state. For example -> model_id will probably turn into self.embedding_model and our new language model should live in something like self.language_model attributes.
Encapsulate all of cluster bootstrap into a class that can be used and referenced in other parts of the code.

I would like to take this issue

@dtaivpp I would like to take this issue.

Closed with #86


Docs for websocket "Keeping the connection alive" are outdated

It says to return a number for idle_timeout as 4th tuple item, it seems it's a map now.
Did a quick search and couldn't find an issue for this, feel free to close if there's already one.
https://ninenines.eu/docs/en/cowboy/2.0/guide/ws_handlers/#_keeping_the_connection_alive

Forgot to update the site! Will do now.

Done! Thanks.


Add documentation for new OPA plugin

Review

@reviewer
Summary
OPA plugin docs. Pulling from docs-private, https://github.com/Kong/docs.konghq.com-private/pull/581.
Reason
New plugin for 2.4.
Testing
TBA

LGTM. Thank you for taking care of this plugin!

Thank you @hbagdi and @lena-larionova !

Direct review link: https://deploy-preview-2773--kongdocs.netlify.app/hub/kong-inc/opa/
I need to add a note about 2.4 EE since it won't show up until that branch is released.


An Ordered Matrix

Description
Problem statement : Create an ordered 2D array (matrix). A matrix is ordered if its (0, 0) element is 1, its (0, 1) element is 2, and so on. Your function needs to create an a × b matrix where a is the first argument and b is the second.
Problem Link : https://edabit.com/challenge/qbpHHZYQrJnNYjtjn
Choose one Category - Medium
Question category : Arrays, Math
Program Name -  An Ordered Matrix
Issue No.- #73

Type issue number after hash(#)

Checklist(self check)

[x] Yes, this task assigned to me.
[x] My code follows the style guidelines(Clean Code) of this project
[x] I have performed a self-review of my own code
[x] My Filename is in camelCase format
[x] I have added Problem statement, Problem link in file.

I was unable to add labels so kindly add from your side.

@aayushi-droid  You can merge and close the issue.


Bootstrap-Portfolio

#Bootstrap-Portfolio
##Developer Story
To recreate the portfolio site with items using bootstrap, to include the following components:

A Navigation bar, with a navigation menu at the top that include links that are applicable and navigate to the appropriate sections
A hero section
A jumbotron featuring picture, name, and any other information.
A work section using a section displaying the work in grid and a placeholder image https://placehold.co/, with a bootstrap cards for each project with a description brief overview of the work.
A skills section, listing the skills expected to learn from the bootcamp.
An about / contact section, with the "About Me" section in the same row.

##Acceptance Criteria

A footer section.
All hyperlinks should have a hover effect.
All buttons should display a box shadow upon hover.
The Bootstrap solutions should minimize use of media queries.
Deploy the Bootstrap-powered portfolio to GitHub Pages.

Website solution completed. All sections fulfiled.


(Resubmission) Improve Flatten to avoid using dynamic shapes when possible

Fixes issues @ebrevdo pointed out with #30380
I also added a unit test for the int32 overflow situation.
See original PR for explanation of overall changes.

Thanks for fixing!

Thanks for the improvements @ebrevdo . Let me know if you have any ideas to shorten the code since it's gotten a bit long now.

Hey @ebrevdo it looks like the PR was reverted again?

yes :(  for some reason the shape inference code on the python side is
truncating in windows (this affects the unit test).  i'll follow up with
our internal team on how we can replicate this in windows.  if you have a
windows box, let us know as you may be able to do some debugging by just
running the unit test with your PR in.
On Mon, Aug 12, 2019 at 1:54 PM Trevor Morris<EMAIL_ADDRESS>wrote:

Hey @ebrevdo https://github.com/ebrevdo it looks like the PR was
reverted again?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/pull/31450?email_source=notifications&email_token=AANWFG3K7VB3YDX27WAEPUTQEHEZJA5CNFSM4IKMVIM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4DZOLI#issuecomment-520591149,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AANWFG22UYA3EW6PVITPI33QEHEZJANCNFSM4IKMVIMQ
.

My suggestion for now is to disable the unit test in windows; easiest way to check:
import platform
is_windows = any(platform.win32_ver())
and self.skipTest if is_windows.

@mrry know who might have time to figure out why shape inference code (either the c++ or swig) is truncating int32 values in windows?  i tried debugging it but ran into too many issues with the tooling.

@trevor-m sorry for the delay.  i'm hoping that final change will allow this PR to stick :)

ebrevdo@- I'm not clear on what you are suggesting. Should we disable the unit test on windows? Or are you saying you've uncovered a different path to make the PR stick?

disabling the unit test on windows is the best choice right now.


How to train?

Hello, I wanna to know how to train and which dataset used?

The implementation details for training each module are provided in the main paper and supplementary material. You can refer to these documents at the following link: https://jeolpyeoni.github.io/stylecinegan_project/
To answer briefly, we trained StyleGAN2 and feature-style encoder model on LHQ dataset,
and trained pix2pixHD model on both Sky-Timelapse and Eulerian dataset as a motion generator.
For the mask predictor, we trained datasetGAN with a few (in our case, 10) selected samples from LHQ dataset, making pairs with manually annotated masks obtained using Segment Anything.


markdown to html fails on links on relatively simple markdown doc

As in #125, using https://sdsawtelle.github.io/attachments/resume.md, shiva conversion to html gives:
Sonya Sawtelle
======

#### Data scientist with a strong background in math, science and programming.
###### [ [website](http://sdsawtelle.github.io) ] . [ [linkedin](www.linkedin.com/in/sonya-sawtelle-data-scientist) ] . [ [github](https://github.com/sdsawtelle) ] . [ [sonya.sawtelle<EMAIL_ADDRESS>] . [<PHONE_NUMBER> ]

Skills
------

to
<!DOCTYPE html>

Sonya Sawtelle

Data scientist with a strong background in math, science and programming.

[ website

Skills

Renders result

As a reference, comrak on the same input renders correctly;

confirm
are you ready to fix it?

No sorry, I don't know this crate well enough and don't have availability right now. If this doesn't get resolved and I find the time I'll let you know.


Move from submodule to subtree

I suggest we move to use subtree instead of submodule.
We can add ".sh" files that assist updating a subtree from commit A to commit B.
Advantage

users no longer need to manually run submodule update/init/sync
no dependency on availability of other git repositories

Disadvantage

maintainers need to learn about subtree method

Example of a .sh file that would update Sylvan:
git subtree pull --prefix=sylvan https://github.com/utwente-fmt/sylvan master --squash -m "sylvan: update to HEAD of master branch"
This updates the "sylvan" subdirectory to the latest version. --squash will mean it will squash all commits since previous subtree pull into 1 commit. It will then add a commit with the included message (-m "...") that merges the squashed commit that modifies the subtree with the checked out branch of the main repository.

Another advantage of this approach is that custom code in the checked out repository is better integrated with the main repository. Currently, if we customize a subrepository, this needs changes on the repository of the submodule. As far as I know, the subtree approach allows customization without modifying the submodule.

I agree, we'll need to decide whether we want this in version 2.1.1 or 2.2.

Yes, a related request for the Sylvan submodule/subtree is to remove
the generated files from the repository, in particular:
modified:   configure
modified:   m4/libtool.m4
modified:   m4/ltoptions.m4
modified:   m4/ltversion.m4
modified:   tools/ltmain.sh
--
Prof.dr. J.C. van de Pol,  chair Formal Methods and Tools
Universiteit Twente, P.O. Box 217, 7500 AE  Enschede, The Netherlands
<EMAIL_ADDRESS>   http://www.cs.utwente.nl/~vdpol
tel: +31 53 489 3017  fax: +31 53 489 3247  secr: +31 53 489 3767

For the related request (next time, please create separate a issue) I have prepared a commit on my personal repository, for which I created a pull request.
This removes the configure file, the aclocal.m4 file, the m4 directory and the tools directory from the sylvan tree.

Example of changing submodule sylvan to subtree sylvan
To remove the submodule:
git submodule deinit sylvan
rmdir sylvan
vim .gitmodules  # to remove sylvan
git commit -a

To add the subtree:
git remote add sylvan git@github.com:utwente-fmt/sylvan
git subtree add -P sylvan --squash sylvan/master

You will now find a new commit in the tree that merges the squashed sylvan commit.

Sylvan is still a module?


select on attribute returning only first element

If I try to select on a particular attribute (such as value in the below snippet), the returned array of the select call is of length 1, whereas there should be 2 matches
  var xpath = require('xpath')
    , dom = require('xmldom').DOMParser

  var xml = "<book><title value='1'>Harry Potter</title><title value='2'>ba2ra</title></book>"
  var doc = new dom().parseFromString(xml)
  var nodes = xpath.select("//title/@value", doc)
  console.log(nodes.length)

However, if I select on the node name, and then return the attribute via getAttribute, then I get an array of length 2
  var xpath = require('xpath')
    , dom = require('xmldom').DOMParser
  var xml = "<book><title value='1'>Harry Potter</title><title value='2'>ba2ra</title></book>"
  var doc = new dom().parseFromString(xml)
  var nodes = xpath.select("//title", doc)
  nodes = nodes.map(function(x) { return x.getAttribute("@value"); });
  console.log(nodes.length)

I created a pull request with added unit tests for this issue

Thank you for pointing this out. It should be fixed now.


please update to 1.21

plz update.

Use the latest alpha versions:

https://modrinth.com/plugin/pv-addon-discs/version/1.1.3
https://modrinth.com/plugin/pv-addon-lavaplayer-lib/version/1.1.1

If you still have any issues, please open a new issue and provide server logs.


[BUG] ServiceBusReceiverClient client never returning after receiveMessages method call (maxWaitTime is seto to 1 sec)

Describe the bug
I am using the the latest beta version of SB 7.15.0-beta.4. Using ServiceBusReceiverClient client. create like:
ServiceBusClientBuilder()
                    .connectionString(sbTopicConnUri)
                    .receiver()
                    .topicName(azureConfig.getSbTopicName())
                    .subscriptionName(azureConfig.getSbTopicSubscription())
                    .disableAutoComplete()
                    .prefetchCount(500)
                    .buildClient();

Using this client I am pulling messages from subscription periodically. For doing so I have a thread pool of 5 threads which internally uses the same receiver client object to receive messages from SB subscription.
While doing so I noticed that as long as the messages are available in subscription the client keep returning them. Once no messages are available the these thread stuck while calling receiveMessages(batch, Duration.ofSeconds(1)). Although the wait time is 1 sec still these threads never returns after the call.
Exception or Stack Trace
As such I dont see any stack trace or exception during this.
To Reproduce
Create multiple threads sharing the same receiver client.
All threads should actively try to read messages using same client at the same time.
Once the messages are exhaust on subscription, then threads must return without any message.
Code Snippet
Add the code snippet that causes the issue.
Creating the receiver client (as above)
AzureSBMsgProcessor[] jobs = { new AzureSBMsgProcessor(barrier, this.messageReceiver),
                new **AzureSBMsgProcessor**(barrier, this.messageReceiver),
                new AzureSBMsgProcessor(barrier, this.messageReceiver),
                new AzureSBMsgProcessor(barrier, this.messageReceiver),
                new AzureSBMsgProcessor(barrier, this.messageReceiver)

AzureSBMsgProcessor is a class which pull messages from sb using the receiver client.
while(recordChangeList.size() < batchSize && (maxPullIter-- != 0) && isMsgPresentForDp) {

            List<FutureTask<List<RecordChange>>> futureTaskList  = new ArrayList<>();
            for (AzureSBMsgProcessor job: jobs) {
                futureTaskList.add((FutureTask<List<RecordChange>>) service.submit(job));
            }
            List<RecordChange> tempRecordChangeList = new ArrayList<>();
            try {
                barrier.await();
                futureTaskList.forEach(msgList -> {
                    try {
                        tempRecordChangeList.addAll(msgList.get());
                    } catch (InterruptedException | ExecutionException ex) {
                        log.error("Error occurred while fetching values from future tasks", ex);
                        if (ex instanceof InterruptedException)
                            Thread.currentThread().interrupt();
                    }
                });
            } catch (InterruptedException | BrokenBarrierException e) {
                log.error("Error occurred while waiting & processing future tasks", e);
                if (e instanceof InterruptedException)
                    Thread.currentThread().interrupt();
            }

            if (tempRecordChangeList.isEmpty())
                isMsgPresentForDp = false;
            else
                recordChangeList.addAll(tempRecordChangeList);

            barrier.reset();
        }

Expected behavior
It is expected that when no messages are available with the SB subscription then all the thread must immediately return after the maxWaitTime, which is 1 second in my case.
Screenshots
If applicable, add screenshots to help explain your problem.
Setup (please complete the following information):

OS: [e.g. iOS] ubuntu
IDE: [e.g. IntelliJ] InteliJ
Library/Libraries: [e.g. com.azure:azure-core:1.16.0 (groupId:artifactId:version)]

 <dependency>
    <groupId>com.azure</groupId>
    <artifactId>azure-messaging-servicebus</artifactId>
    <version>7.15.0-beta.4</version>
</dependency>

Java version: [e.g. 8] java 17
App Server/Environment: [e.g. Tomcat, WildFly, Azure Function, Apache Spark, Databricks, IDE plugin or anything special]
Frameworks: [e.g. Spring Boot, Micronaut, Quarkus, etc]  springboot 2.7

If you suspect a dependency version mismatch (e.g. you see NoClassDefFoundError, NoSuchMethodError or similar), please check out Troubleshoot dependency version conflict article first. If it doesn't provide solution for the problem, please provide:

verbose dependency tree (mvn dependency:tree -Dverbose)
exception message, full stack trace, and any available logs

Additional context
Add any other context about the problem here.
Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

[ ] Bug Description Added
[ ] Repro Steps Added
[ ] Setup information Added

Hello @dakshme, thanks for reaching out. I want to clarify the repro steps, is the below steps looks good?

Create a topic and prefill it with 1000 messages.
Create one ServiceBusReceiverClient instance.
Create a Callable instance that uses ServiceBusReceiverClient instance (in step2) to receive a batch of message with timeout as 1 sec (the Callable returns this received batch of messages as list).
Create an ThreadPoolExecutor with 5 Threads.
Submit the Callable instance (in Step3) to the ThreadPoolExecutor 5 times.
Wait for all submission to return messages (i.e., wait via Future.get() for list).
Repeat Step 5 and 6 in a loop, eventually all the 1000 messages prefilled in step1 should be pulled.
Once the topic is empty, we expect Future.get() from Callable to return empty list.

In step8, what you’re observing is, Future.get() do not return because receiveMessages() is stuck once topic is empty?
Could you let me know the value for batch in receiveMessages(batch, Duration.ofSeconds(1))?

Hello Anu Thomas,
Thank you for your response.
Yes, the steps you mentioned are correct. The batch size I am passing is
100. So basically each of the 5 threads tries to pull max of 100 msg with
max wait time 1 sec (I even tried varying wait time to lesser and higher
values).
Theoretically, with the above case you mentioned, in the first
execution cycle 5 threads must read 500 msgs (approx), then next 500 (near
about), then if anything remains in subscription and then finally an empty
list.
Thanks,
Mahesh Daksha
On Fri, Nov 10, 2023 at 12:57 AM Anu Thomas Chandy @.***>
wrote:

Hello @dakshme https://github.com/dakshme, thanks for reaching out. I
want to clarify the repro steps, is the below steps looks good?

Create a topic and prefill it with 1000 messages.
Create one ServiceBusReceiverClient instance.
Create a Callable instance that uses ServiceBusReceiverClient
instance (in step2) to receive a batch of message with timeout as 1 sec
(the Callable returns this received batch of messages as list).
Create an ThreadPoolExecutor with 5 Threads.
Submit the Callable instance (in Step3) to the ThreadPoolExecutor 5
times.
Wait for all submission to return messages (i.e., wait via
Future.get() for list).
Repeat Step 5 and 6 in a loop, eventually all the 1000 messages
prefilled in step1 should be pulled.
Once the topic is empty, we expect Future.get() from Callable to
return empty list.

In step8, what you’re observing is, Future.get() do not return because
receiveMessages() is stuck once topic is empty?
Could you let me know the value for batch in receiveMessages(batch,
Duration.ofSeconds(1))?
—
Reply to this email directly, view it on GitHub
https://github.com/Azure/azure-sdk-for-java/issues/37594#issuecomment-1804482192,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ALR2LFJ2DRVKXQPRSB2GRXLYDUVBJAVCNFSM6AAAAAA7ESYPTGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMBUGQ4DEMJZGI
.
You are receiving this because you were mentioned.Message ID:
@.***>

Hello @dakshme, I investigated this.
The synchronous receiveMessages(,) API is designed to be invoked "serially". Underneath, each receiveMessages(,) synchronous invocation enqueue a work to pull the message. Each such work is dequeued and processed "serially", i.e. a work-1 pulls messages from broker into list (or timeout), after the "Completion" of work-1 only work-2 will "Start" to pull messages into list. So, on the library level, there is no real parallelism, even if it’s called from multiple threads. This is a design choice made to guarantee the message ordering.
It looks like by calling from multiple threads, the above serial invocation assumption is broken resulting in the problem. I think there are some improvements we can make to fail gracefully, (and Javadoc this requirement/assumption).
So, in your case you want to pull 1000 messages in one go, instead of pulling 500 and 500 messages using, say, two threads (which will be translated to serial message pulling anyway). Then you want to split that received 1000 messages into two sub-batches (each with 500), hand over each sub-batch to separate threads for actual business processing (responsible for converting to RecordChange) then join the results. Repeat this process in a loop, so synchronous receiveMessages(,) guaranteed to invoked serially.

Hello @anuchandy , Thank you for you response, I have studied through the details. Please check the below point and help me with your response.

Fixing the hang issue in concurrent receiver messages call - I have gone thru this PR. So now with this fix I should not find any application threads in hang state (while requesting msgs with a max wait time). Also please confirm which version of SDK this change will be available? I believe, this will at-least solve one part of our reported issue i.e. application threads getting freeze while waiting for messages and never timed out.
Coming to the our scenario or fashion of pulling msgs using receiver client, With our initial implementation we were using single (main) thread for pulling and ack msgs. But we were not happy with the SB performance We noticed that it takes almost 4-5 mins to pull and ack 5000 msgs from subscription (using the this sync client receiver). So we thought to optimize this (using above mentioned implementation). This landed us in current situation. The approach you suggested above may not significantly help the application as the application's msg processing time (after receiving from SB) is just smaller fraction of the overall time of pulling and completing message to SB. Hence we thought to parallelized the pulling and acking of msgs to SB. So, with this receiver client can you confirm the SDK performance number (considering single thread). Performance numbers as in how many msgs pulled and acked with a minute or so.
Also confirm me, what if I create separate receiver client for each thread (unlike current implementation of re-using the same receiver client). So if each concurrent thread using its own receiver client to pull/receive messages will this change the sdk or SB server behavior in anyways or if it will help us in achieving any type optimization or this will not make any impact as such?. Please confirm.
Can you suggest any other way or using this or any other available SB receiver client to optimize our use case. In our case we have a constraint of pulling msgs, process them. Due to other application limitations we do not want to go with push based receiver clients.

Kindly suggest on the above points.

Hello @dakshme,
I’ve got to prepare and run more tests. Once it passes and proves change can be shipped, I’ll update plan around ship date and version of SDK.
Thanks for sharing the additional context around the reason to try the split approach.
The rate at which messages can be pulled and completed also depends on the tier, the network condition and how close the consumer and broker are. After completing the previously mentioned tests, I’ll try set up an environment to get a general idea.
Expect this to take 1-2 weeks.
I agree with your thoughts about using separate Receiver Client instance per Thread. With this approach you should be able to perform concurrent pulling (assuming egress does not throttle the service), given message arrival will no longer be serialized via a single channel (Each Receiver Client uses separate channel).
I wish we could use the ProcessorClient which has out-of-the-box support for threading but given there is a no-push-model application constrain, we cannot use ProcessorClient. The synchronous Receiver is the only client type for pull-model.

@dakshme, btw, could you let me know the service tier?

@dakshme, btw, could you let me know the service tier?

Pricing tier is Standard for us (at least in the env I am working).

Hello @dakshme, I was trying to find an official documentation about Standard tier and came across this.
My understanding from the above documentation about Standard tier is –

For Standard tier Service Bus namespace, there is a limit on the number of operations that can be performed on a specific period. This limiting logic follows a credit-based algorithm.
A namespace has 1000 credits available every second. Each operation performed against namespace has a credit cost associated with it.
For example, each time a message is received, one credit is subtracted from the available credits. Once the credits exhaust, subsequent receive attempts will be throttled until the start of the next second.

My understanding here is - let’s say, application received 1000 messages in 50ms since the start of the current second i, then any receive attempt made in the remaining 50ms of the second i will be throttled. Once that second i elapses, and next second i+1 starts with 1000 credit.
I don’t see the credit cost of complete|abandon in the doc or it’s part of the receive cost. Any other operations (e.g. Send) performed consumes its own credit. Different operations have different credit costs.

@anuchandy
Thank you for point out the documentation. In my case for now I am not seeing any related issues indicating credit limit exhausted. I understand that in standard tier there is limit to the operations one can perform across a namespace. In my case we can hit this limit if constantly we receive loads of messages (which is not the case in my test setup).
As you pointed out, so this is confirmed that even if application receives 100 msgs in single receive msg call will still cause reducing the credit by 100 and not by 1 (as application made single read operation to SB). Please assert this understanding of mine.
Also as I discussed with you in the last thread, I tried reading messages using separate sync receiver client for each thread. With my initial testing it looks like even this wont work as there may be some internal limitations within the sdk implementation. Please find the below multiple errors application observed while reading/complete/abandon messages to SB (each thread using its own receiver client): The occurrences of these err msgs were multiple i.e. error for each msg complete()/abandon(). I am just listing single err for your reference.

Cannot process the disposition request to set the state as 'Modified{deliveryFailed=null, undeliverableHere=null, messageAnnotations=null}' for the delivery with delivery tag (id) '698c7658-81b8-45cc-a887-ed625554004b'. Reason: The delivery with the delivery tag does not exist in the link's DeliveryMap.
Cannot process the disposition request to set the state as 'Rejected{error=Error{condition=com.microsoft:dead-letter, description='null', info={}}}' for the delivery with delivery tag (id) 'd8cad3ff-6fc1-47f3-93c7-251e655e000b'. Reason: The delivery with the delivery tag does not exist in the link's DeliveryMap.
The lock supplied is invalid. Either the lock expired, or the message has already been removed from the queue. For more information please see https://aka.ms/ServiceBusExceptions . Reference:385a80a4-3132-41fc-afcd-e5d9c943bd00, TrackingId:3576e1a3-c040-48ba-8080-55f1f10dac06_B38, SystemTracker:osdu-r3mvp-dp1de-077y-bus:topic:recordstopic~79|wke-listener, Timestamp:2023-11-15T15:54:38, errorContext[NAMESPACE: osdu-r3mvp-dp1de-077y-bus.servicebus.windows.net. ERROR CONTEXT: N/A, PATH: recordstopic/subscriptions/wke-listener/$management, REFERENCE_ID: recordstopic/subscriptions/wke-listener-mgmt:receiver, LINK_CREDIT: 44]
Cannot process the disposition request to set the state as 'Accepted{}' for the delivery with delivery tag (id) '698c7658-81b8-45cc-a887-ed625554004b'. Reason: The delivery with the delivery tag does not exist in the link's DeliveryMap., message=Cannot process the disposition request to set the state as 'Accepted{}' for the delivery with delivery tag (id) '698c7658-81b8-45cc-a887-ed625554004b'. Reason: The delivery with the delivery tag does not exist in the link's DeliveryMap. cause=com.azure.core.amqp.implementation.handler.DeliveryNotOnLinkException: Cannot process the disposition request to set the state as 'Accepted{}' for the delivery with delivery tag (id) '698c7658-81b8-45cc-a887-ed625554004b'. Reason: The delivery with the delivery tag does not exist in the link's DeliveryMap., reason=GENERAL_ERROR
The lock supplied is invalid. Either the lock expired, or the message has already been removed from the queue. For more information please see https://aka.ms/ServiceBusExceptions . Reference:6e926b7f-b7e7-4152-9c14-85a6c0394138, TrackingId:168a0c30-1c78-44e8-a98f-9a30326a35bf_B45, SystemTracker:osdu-r3mvp-dp1de-077y-bus:topic:recordstopic~15|wke-listener, Timestamp:2023-11-15T15:54:38

Actually these lock expiry messages seems to be false positive here as message lock duration is set to 5 minutes in my env. Application is immediately trying to mark the msg complete() after receiving it from subscription. Could this be side effect of using multiple receiver client?. Please suggest more on the above encountered error based in the internal implementation of sdk.
I can try out few more testing with this implementation but I am afraid if this is not going to help me. Also if single sync receiver client can not be used concurrently across thread then it will be become challenging for us to use pull based receiver client to satisfy our performance numbers.
One more question on the side line, do we have api to bulk ack list of messages using sync receiver client. As currently, application need to complete() individual msg to SB and its very time consuming. I mean i see the application spend almost 70-80 percentage of the time in acknowledging the read messages one by one. In the older SB sdk (which is marked as deprecated now) there was an api available to ack list of messages (which happened to be a missing in this new sdk). Can you please comment on this as well.
Sorry for long update, this behavior of SB is hurting us and impacting our production environments hence wanted to fix/optimize this as early as possible.
Thanks

Hello @dakshme, receiving 100 messages will reduce the namespace-throttling-credit by 100 not by 1. Even though it’s a single receiveMessages call in SDK, in response to that call, over the wire there are 100 messages flowing, this number 100 is what decrement the namespace-throttling-credit.
The bulk ack is not supported in the service level, so there is no such APIs in any of the SDKs (across languages). From what I heard; this was a decision made when designing new SDKs due to not having service support.
Regarding the errors you’re seeing with One Receiver Instance per One Thread model, let me take look. Could you confirm the following –

What is the Prefetch per Receiver Client instance?
What is the batch-size and timeout?
In the code, can you double check if you ever try to complete|abandon a message using a Receiver Client other than the one that returned the message?
Regarding the Runnable that each thread runs, how the Runnable looks like? Is there a while loop that calls receiveMessages in a loop? If so, what’s the exit criteria for that while loop?
Can you confirm that once a thread finish executing a Runnable only next submission using the same Receiver Client happens?

Hello @dakshme, to answer your question on lock expiry messages
A reason for seeing "the lock expired" is: Consider application called receiveMessages(100, 5-sec). The Client will send 100 AMQP-Credit to service. Let’s say service returned available 40 messages and client-side 5 sec time out elapsed. The application will process those 40 messages. But the server has no idea about client-side timeout, from service perspective there is another 60 AMQP-Credit pending. As soon as some messages are available (e.g. 20), the service will send it (pending AMQP-Credit is now 40), this gets buffered in the Client. Now assume application called receiveMessages(100, 5-sec) after 5 minutes. A part of this request will be served from the client’s buffer, which has 20 messages in it, it is possible that these messages are already expired while sitting in the buffer.
This buffering of message can happen either due to Prefetching (the Prefetch-Count settings, a.k.a. explicit Prefetch) or due to the above explained scenario (a.k.a. implicit Prefetching). This documentation on Prefetch Prefetch messages from Azure Service Bus - Azure Service Bus | Microsoft Learn is applicable for implicit and explicit prefetch, though doc does not specifically call out the above scenario.

I wrote the test code (as shown below) that uses "One Receiver Client per One Thread model". I ran it for some time, with this approach I was able to receive and complete ~5000 messages in ~70 seconds. I prefilled the subscription with ~40,000 messages, and measured time for 4 sets with each set drains ~5000 messages, each set took ~70 seconds.
Note that I disabled Prefetch and opted-in for the V2 stack (upcoming engine for service bus, currently beta). See the code. Also, used separate TCP Connections, by using separate builder instances.

test-code
package com.contoso.sb;

import com.azure.core.util.ConfigurationBuilder;
import com.azure.core.util.IterableStream;
import com.azure.messaging.servicebus.ServiceBusClientBuilder;
import com.azure.messaging.servicebus.ServiceBusReceivedMessage;
import com.azure.messaging.servicebus.ServiceBusReceiverClient;
import io.netty.util.concurrent.DefaultThreadFactory;

import java.time.Duration;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.*;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.stream.Collectors;

public final class SyncReceiveMeasuring {
    public static void main(String[] args) throws InterruptedException {
        // uses beta.4
        //  <dependency>
        //      <groupId>com.azure</groupId>
        //      <artifactId>azure-messaging-servicebus</artifactId>
        //      <version>7.15.0-beta.4</version>
        //  </dependency>
        measure1();
    }

    private static void measure1() {
        final int concurrency = 5;
        final ArrayList<ServiceBusReceiverClient> clients = new ArrayList<>(concurrency);
        for (int i = 0; i < concurrency; i++) {
            // Build 5 clients, each from dedicated builder, so each gets it's own TCP connection.
            final ServiceBusReceiverClient receiverClient = new ServiceBusClientBuilder()
                .connectionString(System.getenv("CON_STR"))
                // Enabling new stack via configuration.
                .configuration(new ConfigurationBuilder()
                    .putProperty("com.azure.messaging.servicebus.nonSession.syncReceive.v2", "true")
                    .build())
                .receiver()
                .topicName(System.getenv("TOPIC_NAME"))
                .subscriptionName(System.getenv("SUBSCRIPTION_NAME"))
                .disableAutoComplete()
                .buildClient();
            clients.add(receiverClient);
        }

        final List<Work> works = new ArrayList<>(concurrency);
        for (int i = 0; i < concurrency; i++) {
            final ServiceBusReceiverClient client = clients.get(i);
            // 5 Callable (Work) to be submitted to the executer service later.
            works.add(new Work(i, client));
        }

        final ExecutorService executorService = Executors.newFixedThreadPool(concurrency, new DefaultThreadFactory("worker-thread"));
        while (true) {
            final List<FutureTask<WorkResult>> futureTaskList = new ArrayList<>();
            for (int i = 0; i < concurrency; i++) {
                futureTaskList.add((FutureTask<WorkResult>) executorService.submit(works.get(i)));
            }

            final AtomicInteger dispositionFailureCount = new AtomicInteger(0);
            final AtomicInteger messagesCount = new AtomicInteger(0);
            final long beginTime = System.currentTimeMillis();
            futureTaskList.forEach(workResultFuture -> {
                try {
                    final WorkResult result = workResultFuture.get();
                    dispositionFailureCount.addAndGet(result.getDispositionFailureCount());
                    messagesCount.addAndGet(result.getMessagesCount());
                    // For each thread, Print the number of messages received, completion failed and time taken.
                    result.print();
                } catch (InterruptedException | ExecutionException ex) {
                    System.out.println();
                    if (ex instanceof InterruptedException) {
                        Thread.currentThread().interrupt();
                    }
                }
            });
            final long endTime = System.currentTimeMillis();
            final long elapsedTime = endTime - beginTime;
            // Across all threads, Print the total number of messages received, total completion failed and time taken.
            System.out.println("[Threads-Joined]:"
                + " Messages-Count=" + messagesCount.get()
                + " Completion-Error=" + dispositionFailureCount.get()
                + " Duration (sec)=" + TimeUnit.SECONDS.convert(elapsedTime, TimeUnit.MILLISECONDS));
        }
    }

    static final class WorkResult {
        private final String threadName;
        private final int messagesCount;
        private final int dispositionFailureCount;
        private final long elapsedSeconds;
        WorkResult(String threadName, int messagesCount, int dispositionFailureCount, long elapsedSeconds) {
            this.threadName = threadName;
            this.messagesCount = messagesCount;
            this.dispositionFailureCount = dispositionFailureCount;
            this.elapsedSeconds = elapsedSeconds;
        }

        public int getDispositionFailureCount() {
            return dispositionFailureCount;
        }

        public int getMessagesCount() {
            return messagesCount;
        }

        public void print() {
            System.out.println("   [" + threadName + "]:"
                + " Messages-Count=" + messagesCount
                + " Completion-Error=" + dispositionFailureCount
                + " Duration (sec)=" + elapsedSeconds);
        }
    }

    static final class Work implements Callable<WorkResult> {
        private final int id;
        private final ServiceBusReceiverClient client;

        Work(int id, ServiceBusReceiverClient client) {
            this.id = id;
            this.client = client;
        }

        @Override
        public WorkResult call() {
            int dispositionFailureCount = 0;
            final long beginTime = System.currentTimeMillis();
            // Use a coarse timeout e.g, 5sec or more. Using too low values like 1 sec may sometime cause timeout to happen even before connection is established.
            final IterableStream<ServiceBusReceivedMessage> messagesStream = client.receiveMessages(100, Duration.ofSeconds(5));
            final List<ServiceBusReceivedMessage> messagesList = messagesStream.stream().collect(Collectors.toList());
            for (ServiceBusReceivedMessage message : messagesList) {
                try {
                    client.complete(message);
                } catch (RuntimeException e) {
                    dispositionFailureCount++;
                }
            }
            final long endTime = System.currentTimeMillis();
            long elapsedTime = endTime - beginTime;
            return new WorkResult(Thread.currentThread().getName(),
                messagesList.size(), dispositionFailureCount,
                TimeUnit.SECONDS.convert(elapsedTime, TimeUnit.MILLISECONDS));
        }
    }
}

sample-output

   [worker-thread-1-1]: Messages-Count=100 Completion-Error=0 Duration (sec)=7
   [worker-thread-1-2]: Messages-Count=100 Completion-Error=0 Duration (sec)=7
   [worker-thread-1-3]: Messages-Count=100 Completion-Error=0 Duration (sec)=7
   [worker-thread-1-4]: Messages-Count=100 Completion-Error=0 Duration (sec)=7
   [worker-thread-1-5]: Messages-Count=100 Completion-Error=0 Duration (sec)=7
[Threads-Joined]: Messages-Count=500 Completion-Error=0 Duration (sec)=7

   [worker-thread-1-1]: Messages-Count=100 Completion-Error=0 Duration (sec)=7
   [worker-thread-1-2]: Messages-Count=100 Completion-Error=0 Duration (sec)=7
   [worker-thread-1-3]: Messages-Count=100 Completion-Error=0 Duration (sec)=7
   [worker-thread-1-4]: Messages-Count=100 Completion-Error=0 Duration (sec)=7
   [worker-thread-1-5]: Messages-Count=100 Completion-Error=0 Duration (sec)=7
[Threads-Joined]: Messages-Count=500 Completion-Error=0 Duration (sec)=7

......

Now I've the above code, you can ignore my earlier questions around repro code.

@anuchandy
Yes you test program looks close to what I am doing here. Just additional I have used cyclic barrier on top of regular executor service. So we are good with it.
Also, as suggested I removed prefect count (while init client) and observing better performance now. At least not seeing those err (reported yesterday). I am also observing around 5k msgs processing in a approx 1 min. I will continue more testing around and keep you posted.
Can you also please explain me more about this V2 stack:
configuration(new ConfigurationBuilder()
                    .putProperty("com.azure.messaging.servicebus.nonSession.syncReceive.v2", "true")
                    .build()

How this can help or optimize the client ? or if you can share some documentation around it. I assume this feature is available in 7.15.0-beta.4.
Thanks.

Hello, @dakshme, good to hear that the initial testing is positive.
The reason for no or less occurrence of error on complete call is likely is due to disabling the "Explicit Prefetch" (while building client), which reduces the chances of buffering there by expiration of message lock while in buffer.  Also, beware of the "Implicit Prefetch" we discussed earlier.
We both seems to observe the same measurement in "One Receiver Client per One Thread model". :)
Two questions –

Did you use one TCP connection per Receiver Client approach like I tried in the test program (By choosing dedicated builder, not shared)? Dedicated builder per client instance reduces the internal noise compared to shared builder, which is sometime good.
Are you testing with 7.15.0-beta.4?

V2 stack is the new upcoming underlying engine of messaging libraries. We addressed many known issues as part of V2 stack. Some of the main improvements are – removal of locks in critical paths, made the AMQP-Credit calculation more stable, optimized internal buffering that were causing complete() to timeout, addressed out of order message delivery in processor clients, optimized use of threads (so less thread hoping, context switch), and there are more, improving overall stability.
We call the underlying engine of currently GA-ed versions V1 stack. The V2 stack today exists in the beta releases (e.g. 7.15.0-beta.4), but it exists side by side with V1 stack. The library allows opt-in or opt-out V1 or V2 Stack.
For example, in latest beta, the ProcessorClient (session unaware one) by default uses V2 stack and it can be opted-out to use V1 stack. Similarly, the pull based synchronous client by default is on V1 stack and can be opted-in to use V2 stack. The configuration you saw in the test program is for opting-in the synchronous client to use V2 stack. The plan is to eventually phase out the V1 stack, so all clients run on V2. Hope this clarifies.

Hello @anuchandy,
To answer above question:

Yes, I am doing with the approach of each thread having its own receiver client. So basically if I have 5 thread then I build 5 separate sync receiver client.
Yes I am also using the same  7.15.0-beta.4 version.

When the final version of above beta will be released, any tentative time lines? Also whether it will also address the thread blocking issue (in case of concurrent threads using same receiver client) ?
Some updates:
With my further testing I am seeing better results now with this implementation approach. For now we seems to be good with this. Just need a small confirmation from your side once the final version of this beta is out along with the fix (i reported originally with this issue).
I am able to see 100,000 messages processing within 22 mins (with the thread pool config I am using). Which seems to be fine to us for now.
Also, during my testing I observed the below messages (from sdk) getting logged intermittently. Just wanted to know if these are just info msgs or if there is anything concerning in my implementation:
{"az.sdk.message":"The mediator is active.","messageFlux":"mf_736ba5_1700158420500","connectionId":"MF_172210_1700158420579","linkName":"teststopic/subscriptions/test-listener_9f915f_1700158420494","entityPath":"teststopic/subscriptions/test-listener"}

{"az.sdk.message":"Setting next mediator and waiting for activation.","messageFlux":"mf_10fa96_1700158420500","connectionId":"MF_348ca4_1700158420579","linkName":"teststopic/subscriptions/test-listener_3f33a3_1700158420494","entityPath":"teststopic/subscriptions/test-listener"}

Please look at them let confirm it to me once. Otherwise I am good with this issue once we get the fixed version of this lib.
One more query on the side notes:
Is there any way I can mock this.messageReceiver.receiveMessages(batch, Duration.ofMillis(500)); method to return an IterableStream<ServiceBusReceivedMessage>. I need it for writing junit for mu receiver class (in case of this sync receiver clinet).
During mock on receiverMessages method I want to return an iterable stream obj of ServiceBusReceivedMessage. Where the iterable list contains some dummy messages. Unfortunately, I am unable to create obj of ServiceBusReceivedMessage class as its final and constructors are defined with default access (hence can not init it).
I request if you can provide a way or reference code to mock or write junit for this receiver method so it able to return an iterable obj having some actual dummy msgs.
Thanks.

Hello @dakshme,

The beta line is currently going through customer and in-house testing. Overall, the feedback is positive, and beta is stable. If all goes well, we plan the first GA in early next year (first quarter).
We released 7.15.0-beta.5 today (11/22), which has an additional fix to reduce memory consumption when sending messages. Unsure, if your use case includes send or is limited to receive.
We’ll target the multi-thread blocking fix, this needs additional validation, I hope we get bandwidth to include this in the first GA. But with your new model (One Receiver One Thread), IIUC, you’re not blocked by this, right?
That info log messages appears each time there is an attempt to create a receive channel to the service bus. The channel is usually long-lived, unless the client is closed, or the channel is idle for long time (I guess ~10 min), server throttles, network errors. It is safe to ignore these log lines. These lines are showing that underneath channel is recovering.
One more thing, I would suggest using a coarse timeout value for receive(), say 3-5 seconds or above. When the first receive call happen on the client or when channel recover underneath (see point 4), there are a series of operations - bidirectional handshake, authentication, then credit to written to the channel, once service receive the credit, it sends messages which client needs to decode and deliver to the application. If the timeout is too small, it can result in cancelling one of these operations in the middle, only for application to retry again, resulting in all or subset of above operations to trigger again.
Regarding the general Mocking question, please see this discussion .

Hello @anuchandy ,
In continuation with the above implementation, in my implementation (this.messageReceiver.receiveMessages(batch, Duration.ofMillis(**500**))) I have kept max wait time as 500 miliseconds. With this configuration the receiver client works wells most of the time, but in certain case I am observing issue that the SB server stops returning the messages to receiver call (though the messages are stacked). This happens only to specific data partition in this environment. Although, the other dp(s) continue receiving msgs normally.
 The application continues pulling msgs (using this async client with max wait time as 500 mili secs). Then suddenly from one point it stops receiving messages. Following is the are series of error (sdk err logs) I captured in the application logs:

1/6/2024, 12:26:14.764 PM
{"az.sdk.message":"The request was terminated because the namespace osdu-r3mvp-dp3ps-1u86-bus is being throttled. Error code : 50009. Please wait 2 seconds and try again. To know more visit https://aka.ms/sbResourceMgrExceptions and https://aka.ms/ServiceBusThrottling TrackingId:1dd6c549004bf3740000bede65824d7e_G36_B16, SystemTracker:G36:Recv:421806148:637577493003800000:<dp details, subscription name>:F5:C521076, Timestamp:2024-01-06T12:24:39, errorContext[NAMESPACE: . ERROR CONTEXT: N/A, PATH: < subscription name>, REFERENCE_ID: BD62A2280004610A0004611065824D7E_G36, LINK_CREDIT: 5]","exception":"The request was terminated because the namespace <dp details, subscription name> is being throttled. Error code : 50009. Please wait 2 seconds and try again. To know more visit https://aka.ms/sbResourceMgrExceptions and https://aka.ms/ServiceBusThrottling TrackingId:1dd6c549004bf3740000bede65824d7e_G36_B16, SystemTracker:G36:Recv:421806148:637577493003800000:<dp details, subscription name>:F5:C521076, Timestamp:2024-01-06T12:24:39, errorContext[NAMESPACE: osdu-r3mvp-dp3ps-1u86-bus.servicebus.windows.net. ERROR CONTEXT: N/A, PATH: recordstopic/subscriptions/wke-listener, REFERENCE_ID: BD62A2280004610A0004611065824D7E_G36, LINK_CREDIT: 5]","entityPath":"<dp details, subscription name>","linkName":"<dp details, subscription name>"}
1/6/2024, 12:33:01.288 PM
{"az.sdk.message":"Operation not successful.","entityPath":"<dp details, subscription name>","status":"SERVICE_UNAVAILABLE","description":"The request was terminated because the namespace <dp details, subscription name> is being throttled. Error code : 50009. Please wait 2 seconds and try again. To know more visit https://aka.ms/sbResourceMgrExceptions and https://aka.ms/ServiceBusThrottling TrackingId:f964b0c0-3d32-4891-88ee-080bb45b8003_B21, SystemTracker:<dp details, subscription name>, Timestamp:2024-01-06T12:33:01 Reference:1ecffb84-bac0-4a3e-be48-5e62e49b0777, TrackingId:93a77f86-93eb-4b04-be59-e832b8cfa550_G82, SystemTracker:NoSystemTracker, Timestamp:2024-01-06T12:33:01","condition":"com.microsoft:server-busy"}
1/6/2024, 12:44:36.896 PM
{"az.sdk.message":"Transient error occurred.","exception":"The link '<dp details, subscription name>' is force detached. Code: aggregate-link354861. Details: AmqpMessagePartitioningEntityMessageCache.IdleTimerExpired: Idle timeout in Seconds: 900., errorContext[NAMESPACE: osdu-r3mvp-dp3ps-1u86-bus.servicebus.windows.net. ERROR CONTEXT: N/A, PATH: <dp details, subscription name>, REFERENCE_ID: BD62E5A000056A2200056A2D658251E6_G22, LINK_CREDIT: 0]","linkName":"n/a","entityPath":"n/a","attempt":1,"retryAfter":4511}

As per the logs I understand that it throttled the requests few a times, it worked also in between during that time. It also throttle requests with "status":"SERVICE_UNAVAILABLE". Finally at 1/6/2024, 12:44:36.896 PM (the last log above) It gave this error and after this it never recovered to receive any message. After this point it was the same log for this particular dp receiver client.
With earlier implementation we never got this issue (when we used single receiver client). It started coming with multiple receiver client for the same dp. The above issue is reproducible.
Need you suggestion/inputs what could be the cause of this observation
Thanks.

@anuchandy can you please help me with above query,

Hello @dakshme, for production deployments, I would strongly suggest using a coarse timeout in the range 3-5 seconds. If you remember, earlier we discussed the importance of coarse timeout, and problem with low timeout that can overwhelm the system during recovery phases. https://github.com/Azure/azure-sdk-for-java/issues/37594#issuecomment-1823677260
I’ll take a look and see if the low timeout put the system in a non-recoverable state and how we can improve here.

@anuchandy
Thank you for your response. I already made the max wait time to 5 seconds but I see that with this config the said issue i.e.   AmqpMessagePartitioningEntityMessageCache.IdleTimerExpired: Idle timeout in Seconds: 900 delays but it still occurs after few days.
I need to understand the following points:

In response to my this original issue your earlier fix (https://github.com/Azure/azure-sdk-for-java/pull/37665) is now available in any latest version of the lib ?. If yes, please help me with the GA version of the lib.

why even after 5 second I am continuously hitting this issues, do we need to consider a higher value such as 10 seconds or something. Would that guarantee that this issue wont occur in that case?.

Also I see whenever sdk gives this error/exception: the receiver client stuck and the sdk never throw any exception or return until we see this err? Shall we consider writing a workaround of having a Timertask on this client thread and to interrupt it after a certain period it it doesn't return?
{"az.sdk.message":"Transient error occurred.","exception":"The link 'recordstopic/subscriptions/test-listener_3b405a_1705500283844' is force detached. Code: aggregate-link61730040. Details: AmqpMessagePartitioningEntityMessageCache.IdleTimerExpired: Idle timeout in Seconds: 900., errorContext[NAMESPACE: test-dp3env-testing-bus.servicebus.windows.net. ERROR CONTEXT: N/A, PATH: recordstopic/subscriptions/wke-listener, REFERENCE_ID: 83C5B50603ADECEC03ADECF865A7DE7B_G22, LINK_CREDIT: 0]","linkName":"n/a","entityPath":"n/a","attempt":1,"retryAfter":4511}

Is it not possible to deal with smaller max wait time internally within the sdk, that is whatever underlying activity/operations needed by SB server should not be consider under the max wait time given by the application ?. Otherwise in the sdk there must be a check or validation on the minimum max wait time value which should be passed by any application. This is important from application development side otherwise it causing lots of confusion and increasing the overall development time also.

Please suggest with the above points.
Thanks,
Mahesh Daksha

Hi @dakshme, thank you for the additional context.
Looking at the below log message you shared,

{"az.sdk.message":"Transient error occurred." …

It seems the environment is loading 7.14.x version or v1 code in 7.15.0-beta.x. When we both worked on designing the multi-receiver solution back in November-2023, we used 7.15.0-beta.x version. The above log message is coming from the class ServiceBusReceiveLinkProcessor. In 7.15.0-beta.x, with the configuration-value "com.azure.messaging.servicebus.nonSession.syncReceive.v2" set, we no longer use this class.
The 7.15.0-beta.x line has many receiver reliability improvements, that include better recoveries when connection goes idle.

It could be that, in your solution, somewhere else in dependency tree it gets downgraded to 7.14.x or it indeed uses 7.15.0-beta.x but without the v2 configuration-value "com.azure.messaging.servicebus.nonSession.syncReceive.v2" set. Could you check this?
I understand that there are separate receiver-client instances. Like discussed last time, could you ensure each of these clients are built from independent builder as well? I.e., create a new builder instance and obtain a client instance from it, repeat this (new_builder -> new_client pattern) for each client.

I’m working to GA 7.15.0, it should happen in few days. The #37665 will not be included in the first GA of 7.15.x, but all reliability improvements in beta so far. I hope with the current design of One_Receiver:One_Thread approach with serial access, we won’t hit 37665

@anuchandy
Since the 7.15 dev was in beta mode so we purposefully switched back to earlier one. But after your last reply I change the implementation to use the latest beta of 7.15. Unfortunately even after that we are encountering IdleTimerExpired  error. We tried with max wait time as 7 second. It only delayed the problem but eventually the receiver client stopped pulling the msgs (after 2/3 days).
We event tried keeping the ServiceBusClientBuilder object same for whole application running time (keeping separate obj of builder for each thread) and creating the ServiceBusReceiverClient (using builder obj) during each polling cycle. Even this result in to the same hang state where the azure lib do no release the application call. So even this approach did not work.
Now with this instability of the sdk it seems we do not have any option other than creating the ServiceBusClientBuilder along with ServiceBusReceiverClient every polling cycle to make it work properly. I understand creating Builder obj every time may be a expensive operation but we do no see any other possible way also. Please suggest.
Quick suggestion will help as our development of certain critical fixes are stuck.
Please note all the above combinations of implementation tried with latest beta of 7.15 version of sdk.
Thanks,
Mahesh D.

Hello @ dakshme, I started an offline discussion with you on this, so we can look into actual usage and discuss/plan how to resolve this.

Hello @anuchandy,
Just for the information and future reference I am noting down the below points we discussed during our offline meeting. As discussed over the call we expect these issues to be addressed in the upcoming version of lib in order to bring more stability in its operation:

Application thread when calling this.messageReceiver.receiveMessages(batch, Duration.ofMillis(500)) stuck/hangs with in SB sdk and never returns. This blocks the application execution. This is already identified and acknowledged by you.
When using multiple threads to receive or pull msgs using the same receiver client connection then also the above problem is encountered. This is also observed if we have dedicated connection builder for each object and open and close the client while receiving messages using their respective connection builder object.
Whenever there is any type of error or exception within the sdk/lib call then it should return or throw the same exception back to application. Currently it is observed that the library silently suppress the error and return the empty IterableStream. This causes false behavior at the application side as it has no way to recover from this erroneous state.
When using multiple thread having their own connection builder and receiver client (created and closed every time while receiving message). In this case the application able to work fine for few days, after that it is observed that application unable to receive any message from SB (though the messages are available). In this case also it is observed that the library keeps returning the empty object of IterableStream of messages and not throwing any error in this case as well.
If there is a hard requirement of keeping the value of max wait time in receiveMessages greater than 5 second or so then I suggest to either have a validation check on this or should be documented under best practice along with validation.

Please add any other point if I have missed on anything.
Need your help to have an early resolution to above mentioned issues as its impacting our development activity and eventually affecting customer environments.
Thanks,
Mahesh Daksha

Hello @anuchandy,
Just for the information and future reference I am noting down the below points we discussed during our offline meeting. As discussed over the call we expect these issues to be addressed in the upcoming version of lib in order to bring more stability in its operation:

Application thread when calling this.messageReceiver.receiveMessages(batch, Duration.ofMillis(500)) stuck/hangs with in SB sdk and never returns. This blocks the application execution. This is already identified and acknowledged by you.
When using multiple threads to receive or pull msgs using the same receiver client connection then also the above problem is encountered. This is also observed if we have dedicated connection builder for each object and open and close the client while receiving messages using their respective connection builder object.
Whenever there is any type of error or exception within the sdk/lib call then it should return or throw the same exception back to application. Currently it is observed that the library silently suppress the error and return the empty IterableStream. This causes false behavior at the application side as it has no way to recover from this erroneous state.
When using multiple thread having their own connection builder and receiver client (created and closed every time while receiving message). In this case the application able to work fine for few days, after that it is observed that application unable to receive any message from SB (though the messages are available). In this case also it is observed that the library keeps returning the empty object of IterableStream of messages and not throwing any error in this case as well.
If there is a hard requirement of keeping the value of max wait time in receiveMessages greater than 5 second or so then I suggest to either have a validation check on this or should be documented under best practice along with validation.

Please add any other point if I have missed on anything.
Need your help to have an early resolution to above mentioned issues as its impacting our development activity and eventually affecting customer environments.
Thanks, Mahesh Daksha

@anuchandy Any updates on the released version of the sdk addressing these points (as we discussed last time). To make the sdk more stable.
Thanks.


Missing code

2 months ago you said:
Note: This repo is under construction and within the next few days we will release the code and more details on this Readme. Please stay tuned!
Any news on when it will be available?

Any updates here?

Is the code being added here?

Sorry about the delay, @SrivatsaMarichi - we wanted to make sure all the code worked with updates that've been made since Connect();, and we wanted to craft a better readme that walks through the entire setup of everything this demo required. It's in here now, and we hope you enjoy it!


Project status questions after Mozilla layoffs

Just want to ask if this project will be continued after the Mozilla layoffs and closing the servo team.
I hope this continues.

Nobody is actively working on this project anymore.


Added an option to reset the bus volume to 0db

Fixes #10631

Oh, do you think it's possible to add a shortcut, like Ctrl+Click or Alt+Click?

@vnen I was just trying to assign a Ctrl+Click or Alt+Click but it gets tricky since the Ctrl/Alt doesn't reach the bus input function unless you're focusing it, and the mouse button event doesn't have an get_alt() function. Maybe it would be better to assign a double Right Click to the slider to reset it too?

No, I think that zeroing when Right Clicking would be counter-productive since it would happen a lot by mistake.


Middleware crashes with serverless-http

Expected behavior
The bugsnag middlewares should work with serverless-http that is a middleware wrapper for serverless use (AWS Lambda), compatible with express / connect middlewares.
Observed behavior
Every request returns an error:
TypeError: connection.address is not a function
    at requestInfo (/var/task/node_modules/bugsnag/lib/request_info.js:5:44)
    at Bugsnag.requestHandler (/var/task/node_modules/bugsnag/lib/bugsnag.js:205:25)
    at Layer.handle [as handle_request] (/var/task/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/var/task/node_modules/express/lib/router/index.js:317:13)
    at /var/task/node_modules/express/lib/router/index.js:284:7
    at Function.process_params (/var/task/node_modules/express/lib/router/index.js:335:12)
    at next (/var/task/node_modules/express/lib/router/index.js:275:10)
    at expressInit (/var/task/node_modules/express/lib/middleware/init.js:40:5)
    at Layer.handle [as handle_request] (/var/task/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/var/task/node_modules/express/lib/router/index.js:317:13)
    at /var/task/node_modules/express/lib/router/index.js:284:7
    at Function.process_params (/var/task/node_modules/express/lib/router/index.js:335:12)
    at next (/var/task/node_modules/express/lib/router/index.js:275:10)
    at query (/var/task/node_modules/express/lib/middleware/query.js:45:5)
    at Layer.handle [as handle_request] (/var/task/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/var/task/node_modules/express/lib/router/index.js:317:13)
    at /var/task/node_modules/express/lib/router/index.js:284:7
    at Function.process_params (/var/task/node_modules/express/lib/router/index.js:335:12)
    at next (/var/task/node_modules/express/lib/router/index.js:275:10)
    at Function.handle (/var/task/node_modules/express/lib/router/index.js:174:3)
    at EventEmitter.handle (/var/task/node_modules/express/lib/application.js:174:10)
    at finish.then (/var/task/node_modules/serverless-http/serverless-http.js:36:13)

Steps to reproduce
Setup a simple serverless-http service on AWS Lambda:
const express = require('express');
const serverless = require('serverless-http');
const bugsnag = require('bugsnag');

const app = express();

// Initialize bugsnag
bugsnag.register(process.env.BUGSNAG_TOKEN, {
  releaseStage: process.env.NODE_ENV,
  notifyReleaseStages: ['staging']
});

app.use(bugsnag.requestHandler);

app.post('/', auth, (req, res, next) => {
  return res.json({});
});

app.use(bugsnag.errorHandler);

module.exports.handler = serverless(app);

Version

Node 6.10
serverless-http 1.5.2
bugsnag 2.1.2

Hi @jguepin, thanks for the report. I'll get this looked into – I'm sure the workaround on our part is pretty simple.
Do you think this it's also worth posting a bug on dougmoscrop/serverless-http? I would imagine a fair amount of people/middleware for connect/express are using req.connection.address() in some way. It looks like it stubs a socketobject here to pass to the HTTP IncomingMessage class which is incomplete…
I guess the inbound TCP connection is not available to a Lambda function, but is there something else useful we could snag here?

Hello @bengourley, thanks for the reply!
Indeed, I think the error could be fixed on serverless-http side, I just opened an issue on the repo.


Unload observer for multiple components

I would like to use the unload observer to unregister components (my application uses an event bus, and I would like my components to be removed right away on close instead of waiting for the detach event).
It would be nice to be able to register/unregister components to be notified when navigating.  Maybe just a coding example would do.

UnloadObserver is intended to provide a callback to the server when the user closes the tab (or browser window) or manually navigates away. Navigation lifecycle is a standard feature of Vaadin, as described here: https://vaadin.com/docs/flow/routing/tutorial-routing-lifecycle.html - so I would expect what you are trying to achieve can be done through that.
I admit I have not tested the component well enough, so it is very well possible that the event triggers when navigating to somewhere within the app or in some other weird cases. I will investigate and update this ticket accordingly.

The "onbeforeleave" navigation events don't trigger when the window is
closed.  So the components on a page stay there until the onDetach triggers
a few minutes later.  I need something that will push the registrations on
a stack and pop them when navigating to another page, or when the unload of
the window occurs.
Le lun. 22 juin 2020 à 15:45, Miki<EMAIL_ADDRESS>a écrit :

UnloadObserver is intended to provide a callback to the server when the
user closes the tab (or browser window) or manually navigates away.
Navigation lifecycle is a standard feature of Vaadin, as described here:
https://vaadin.com/docs/flow/routing/tutorial-routing-lifecycle.html - so
I would expect what you are trying to achieve can be done through that.
I admit I have not tested the component well enough, so it is very well
possible that the event triggers when navigating to somewhere within the
app or in some other weird cases. I will investigate and update this ticket
accordingly.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/vaadin-miki/super-fields/issues/170#issuecomment-647734266,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAFFWB4FMSNWBF4E74KYDILRX6YHFANCNFSM4OE26CIQ
.

Alrighty. I think that with the fix of #173 it should be possible to get the parent component in which UnloadObserver resides. In that case you should be able to detach components manually e.g. by removing them from the layout or taking them from some other globally available place (not sure if there is a better approach, and I think having each component listen to unload events is in general a bad idea).
I will try to come up with changes to the demo app to support multiple views (see #174) and at the same time I can have the unload trigger e.g. log the components in the current view.

Agreed.
Le mar. 23 juin 2020 à 07:16, Miki<EMAIL_ADDRESS>a écrit :

Alrighty. I think that with the fix of #173
https://github.com/vaadin-miki/super-fields/issues/173 it should be
possible to get the parent component in which UnloadObserver resides. In
that case you should be able to detach components manually e.g. by removing
them from the layout or taking them from some other globally available
place (not sure if there is a better approach, and I think having each
component listen to unload events is in general a bad idea).
I will try to come up with changes to the demo app to support multiple
views (see #174 https://github.com/vaadin-miki/super-fields/issues/174)
and at the same time I can have the unload trigger e.g. log the components
in the current view.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/vaadin-miki/super-fields/issues/170#issuecomment-648080076,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAFFWB5JA3QXDUK3G3KPKGDRYCFKLANCNFSM4OE26CIQ
.

While I am working on #173, I had an idea. Currently the only config for UnloadObserver is to either:

display a confirmation window and register an attempt (both successful or not)
do nothing

I think it would make sense to inform server also when the unload happened. I think this is what would help you achieve your goal of unloading components when the app is closed - no need for query here. Is my thinking correct?

I will need to manage a list of all objects that I need to clean up when the UI they being to closes.  I will need to manage a map<UI,Registration> So I need to know which UI is closing.
jfl, de mon téléphone

Le 24 juin 2020 à 12:25, Miki<EMAIL_ADDRESS>a écrit :

While I am working on #173, I had an idea. Currently the only config for UnloadObserver is to either:
display a confirmation window and register an attempt (both successful or not)
do nothing
I think it would make sense to inform server also when the unload happened. I think this is what would help you achieve your goal of unloading components when the app is closed - no need for query here. Is my thinking correct?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

Alright. I did some testing. It is impossible to contact the server on unload event, as it is not available. However, it is possible to call server without displaying a prompt to the user. So, UnloadObserver will do this by default, and the event now has a flag isBecauseOfQuerying which will be true when the user was shown prompt, and false when the event just happened and the page has been unloaded (unless there has been something else preventing it from unloading, of course).
This has a drawback: it is impossible to find out if the page has been closed because user has agreed to leave it. I think though it is acceptable as is. The drawback can be fixed by #176 if there is anyone willing to work on it, I am not skilled in JS enough to fix it.
Anyway, the above behaviour should fit your use case @jflamy - when the event is called the UnloadObserver is attached to some parent, so you can probably figure out everything you need. This will be included in version 0.7.1 which I hope to release early next week.

Excellent.  I will be looking at it.
------ Original Message ------
From: "Miki"<EMAIL_ADDRESS>To: "vaadin-miki/super-fields"<EMAIL_ADDRESS>Cc: "Jean-François Lamy"<EMAIL_ADDRESS>"Mention"
<EMAIL_ADDRESS>Sent: 2020-06-26 5:35:24 PM
Subject: Re: [vaadin-miki/super-fields] UnloadObserver should always
send an event before unloading (#170)

Alright. I did some testing. It is impossible to contact the server on
unload event, as it is not available. However, it is possible to call
server without displaying a prompt to the user. So, UnloadObserver will
do this by default, and the event now has a flag isBecauseOfQuerying
which will be true when the user was shown prompt, and false when the
event just happened and the page has been unloaded (unless there has
been something else preventing it from unloading, of course).
This has a drawback: it is impossible to find out if the page has been
closed because user has agreed to leave it. I think though it is
acceptable as is. The drawback can be fixed by #176
https://github.com/vaadin-miki/super-fields/issues/176 if there is
anyone willing to work on it, I am not skilled in JS enough to fix it.
Anyway, the above behaviour should fit your use case @jflamy
https://github.com/jflamy - when the event is called the
UnloadObserver is attached to some parent, so you can probably figure
out everything you need. This will be included in version 0.7.1 which I
hope to release early next week.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/vaadin-miki/super-fields/issues/170#issuecomment-650413735,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAFFWB2BDM6CV2QHPIOQCODRYUIBZANCNFSM4OE26CIQ.


python311Packages.evohome-async: 0.4.8 -> 0.4.9

Diff: https://github.com/zxdavb/evohome-async/compare/refs/tags/0.4.8...0.4.9
Description of changes

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 267022 run on x86_64-linux 1

2 packages built:

python311Packages.evohome-async
python311Packages.evohome-async.dist


Flask App factory PR #1

CATEGORY
Choose one

[ ] Bug Fix
[ ] Enhancement (new features, refinement)
[x] Refactor
[ ] Add tests
[ ] Build / Development Environment
[ ] Documentation

SUMMARY
WIP - This is the first of several PRs which addresses SIP-24
ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Changes UI
[ ] Requires DB Migration.
[ ] Confirm DB Migration upgrade and downgrade tested.
[ ] Introduces new feature or API
[ ] Removes existing feature or API

REVIEWERS

Read through, LGTM. This is many steps in the right direction.

Codecov Report

Merging #8418 into master will increase coverage by 0.08%.
The diff coverage is 84.13%.

@@            Coverage Diff             @@
##           master    #8418      +/-   ##
==========================================
+ Coverage    66.5%   66.59%   +0.08%
==========================================
  Files         450      456       +6
  Lines       23117    23269     +152
  Branches     2451     2452       +1
==========================================
+ Hits        15375    15496     +121
- Misses       7593     7624      +31
  Partials      149      149

Impacted Files
Coverage Δ

superset/utils/core.py
88.02% <ø> (-0.79%)
:arrow_down:

superset/forms.py
95% <ø> (-0.46%)
:arrow_down:

superset/tasks/celery_app.py
0% <0%> (-100%)
:arrow_down:

superset/extensions.py
100% <100%> (ø)

superset/utils/cache_manager.py
100% <100%> (ø)

superset/tasks/schedules.py
79.89% <100%> (ø)
:arrow_up:

superset/__init__.py
100% <100%> (+24.03%)
:arrow_up:

superset/sql_lab.py
77.93% <100%> (ø)
:arrow_up:

superset/tasks/cache.py
73.68% <100%> (ø)
:arrow_up:

superset/utils/cache.py
46.15% <71.42%> (+4.48%)
:arrow_up:

... and 18 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 78f41b3...92e7922. Read the comment docs.

@craig-rueda
jinja2.exceptions.TemplateNotFound: superset/welcome.html
class SupersetIndexView(IndexView):
@expose("/")
def index(self):
return redirect("/superset/welcome")

@craig-rueda we ran into an issue with Celery using the configuration defined here related to running outside of an app context and need to use,
celery worker --app=superset.tasks.celery_app.app

instead. I was wondering whether you could update UPDATING.md and possibly CONTRIBUTING.md to  reflect the change.


KTX Basis textures fail to load in editor in Unity 2021.1.0f1 & 2021.2.0a10.1508

Hey again @atteneder, making sure you have plenty of bugs to work on when you get back :)
So in Unity 2021.1.0f1 (the latest stable version from the Unity Hub at the time of this post), .basis textures fail to load in the Editor when loading with KTXUnity 1.0.

You can see in the image above, I'm using the KTXUnity-Tester github, and upgraded it to 2021.1.0f1 on my local instance for testing.
After that I just press the play button on the testing scene to encounter the error.
Oh and it looks like 2021.1.0f1 has a bug with WebRequest.downloadHandler.data always having a size of zero. That's completely unrelated, but just thought you should know. (You can test this by removing the KTX scripting define symbol in the project, and the UITextureLoader.cs will swap the url's for .png images instead.). I'll be creating a new bug report for Unity for that one as well once I confirm that it's not already squashed in the latest alpha version.

Oh, and just tried it on Unity 2021.2.0a10.1508, which was the latest alpha from Unity hub today. The same error exists for KTXUnity in the editor.

Sorry for the late reply.
Great that you found the problem. Might be useful to know for the future.
Thanks!


[FEATURE REQ] Extract Diagnostic Information per API

Please describe the feature.
When specifying custom diagnostic information on API's, the configuration therefore is currently not extracted.
Example configuration:

This should be supported from my POV, to be able to process a complete API extraction & deployment.

Thanks for reporting, I am also experiencing this issue.
Also, in sample, can you include full syntax of api and diagnostics I would request in sample config file full properties of API be listed.
I am having difficulty publishing diagnostics as well, though extract pipeline is not generating diagnostics.

Not sure if I get you right, but here's a full response body from a configured applicationInsight diagnostic of an API:
GET https://management.azure.com/subscriptions/***/resourceGroups/***/providers/Microsoft.ApiManagement/service/***/apis/***/diagnostics/applicationinsights?api-version=2022-08-01
{
  "id": "/subscriptions/***/resourceGroups/***/providers/Microsoft.ApiManagement/service/***/apis/***/diagnostics/applicationinsights",
  "type": "Microsoft.ApiManagement/service/apis/diagnostics",
  "name": "applicationinsights",
  "properties": {
    "alwaysLog": "allErrors",
    "httpCorrelationProtocol": "W3C",
    "verbosity": "information",
    "logClientIp": true,
    "loggerId": "/subscriptions/***/resourceGroups/***/providers/Microsoft.ApiManagement/service/***/loggers/ApplicationInsightsLogger",
    "sampling": {
      "samplingType": "fixed",
      "percentage": 100
    },
    "frontend": {
      "request": {
        "headers": [
          "My-Correlation-Id"
        ],
        "body": {
          "bytes": 8192,
          "sampling": null
        }
      },
      "response": {
        "headers": [
          "My-Correlation-Id"
        ],
        "body": {
          "bytes": 8192,
          "sampling": null
        }
      }
    },
    "backend": {
      "request": {
        "headers": [
          "My-Correlation-Id"
        ],
        "body": {
          "bytes": 8192,
          "sampling": null
        }
      },
      "response": {
        "headers": [
          "My-Correlation-Id"
        ],
        "body": {
          "bytes": 8192,
          "sampling": null
        }
      }
    },
    "tags": null
  }
}

Thank you, that was an oversight on our part. We'll address this.

@guythetechie,
I was looking at the commit, and I didn't see support for API Diagnostics override functionality for the publisher. I am sorry, is it missed or am I reading it wrong?


The github plugin failed to collect tidb on postgres.

Describe the bug
postgres unsupported Unicode escape sequence.
To Reproduce
Steps to reproduce the behavior:

run github by (pingcap/tidb)
subtask collectApiIssues error occured

Screenshots

change type to text


Connection Flow Animation not fluid

Preflight Checklist

[x ] I agree to follow the Code of Conduct that this project adheres to.
[x ] I have searched the issue tracker for a feature request that matches the one I want to file, without success.

Describe the bug
When using the flow animation property or the flow plugin, the connection is not animated in a fluid way
To Reproduce
Steps to reproduce the behavior:

Click on a connection
Click on 'Property'
Scroll down to 'Flow Animation' and enable
See error
URL for replicating.

Expected behavior
The animation behaves fluidly without any stutter
Screenshots

draw.io version (In the Help->About menu of the draw.io editor):

draw.io version: 21.2.8

Desktop (please complete the following information):

OS: Windows
Browser Chrome, Edge, Brave

Additional context
The stuttering is worse with longer connection patterns but present in all. The stuttering seems to be present in all views

has this been fixed ?

Hey guys, I just figured out that this "stuttering" only happens if you change the line pattern from "solid" to "dashed" on the "style" tab and then enable the "flow animation" option.
If you left the line as "solid" and enable the "flow animation" option, it will automatically turn into a "dashed" behind the scenes and you will have a fluid experience.

Fix in 24.0.1. Thanks for report.


built in keyboard and touchpad  are not recognized after sleep

after connecting an external keyboard and mouse and after the mac goes to sleep mode the built in keyboard and touchpad are no longer recognize as a device and can't be use.
restart dose not working just resetting the SMC and NVRAM fix the problem
(https://support.apple.com/en-il/HT201295, https://support.apple.com/en-il/HT204063)
BTW,

Disable the built in keyboard while one of the following selected device is connected - no device is checked.
after uninstalling Karabiner Elements this issue does not occur.

I've experienced similar problems, though in my case it seems that restarting does seem to fix the problem.

i went back to ver 0.90.90 and now I'm having issue similar to what @marcreynolds describes and restart does helps but it's different from the case i've wrote that occurs on version 0.90.92

KE Version: 0.91.12
MacOS: 10.12.6
I believe I'm experiencing a similar thing. It doesn't happen after every "sleep", but does happen enough that it is annoying. The symptoms are that my computer thinks my "Shift" key is continuously pressed, and only Quiting and Relaunching karabiner-elements helps alleviate the issue.
 I was using it to get the Caps-Lock key to act like an escape key.  I've since discovered that that functionality is build into OS X 😄.

I've experienced this issue twice after upgrading karabiner to 12.0.0. The second incident has happened right now in about 2 weeks after the first one.
I might be just speculating, because there's no evident correlation between those things. But from the other perspective I really don't have any other software that does what karabiner does.


Can't PlainClone a Local Repository

Hey All,
During testing I am creating local repositories on my filesystem and wanting to treat them as "remote".
I've tried both types of what I believe should be valid URLs
file://var/folders/zg/7727fl_x1n906p44h9b5mtxh0000gn/T/git-repo310648135
AND
/var/folders/zg/7727fl_x1n906p44h9b5mtxh0000gn/T/git-repo310648135
When calling Head() I basically end up getting.
reference not found

Listing References on the repo does give me back HEAD. Am I doing something incorrectly here?

Seems like it had to be:
file:/// instead of file://
Seems like either should work? Also error was misleading but switched to repository not found

@zachgersh This is how standard URL works: file:// is the scheme part, the third / is the first slash of an absolute UNIX path. file://var/folders/zg/7727fl_x1n906p44h9b5mtxh0000gn/T/git-repo310648135 actually means var/folders/zg/7727fl_x1n906p44h9b5mtxh0000gn/T/git-repo310648135 relative to your current path. Note that this is by design.
We could definitely improve the error, but that will have to wait until a pending major overhaul of error handling in go-git probably based on https://github.com/src-d/go-errors but that will probably not happen in the short term.


xterm text corruption

With wlc 0.0.3 as found in Archlinux, xterm under both orbment and sway randomly draws some characters partially. No such issue when running the same xterm under Weston.

Can you demonstrate this in a screenshot, or at least how to reproduce this visual(?) corruption?

Start a vanilla (no Xresources) or custom (personal Xresources) uxterm and run ls to print some text. If none of the characters are incomplete, either run ls a few more times or start and exit mc. The way it looks is that the right half or upper right part of some of characters is invisible. Given that the vanilla uxterm exhibits the same issue, it's safe to assume it's unrelated to font config.

First of all, never use uxterm, use -u8 instead as uxterm screws with your locale settings.
As for xterm -u8 with no X resources file, I can reproduce this.  It's by default using the fixed font.
Screenshot:

I didn't know about uxterm's locale bug. I'll change my bindings. Isn't there an X resource I can configure to achieve -u8 by default (zero args passed to xterm)?

What terminal do others run under Wayland? weston-terminal doesn't really work for me and st's Wayland fork is also very buggy in addition to st's general incompatibilities (compared to xterm).

According to xterm(1) you can set the class utf8 to 2 to always use UTF-8 encoding regardless of locale while disallowing it from being disabled.  Other options are 0 (false), 1 (true) and 3 which is the default and based on the locale class.

I have that already set to 1, that seems sufficient then. Thanks for the tip.

It seems that whole columns of pixels are missing. Is there a mismatch between how wide xterm thinks it is, and how wide wlc considers it to be?

I think I've found another program that experiences the same rendering problem: The help system for SWI Prolog

Does this still happen? I don't think I've seen this in xterm for a while now. I mean, maybe it's my config or it was actually fixed like the Firefox drop-down menu repaint bug has been.


Update Composer dependencies (2021-03-15-00-12)

Visual regression test passed!
View the visual regression test report


Fix missing audio artwork metadata on multiple exports

Q
A

Bug fix?
yes

New feature?
no

BC breaks?
no

Deprecations?
no

Fixed tickets
-

Related issues/PRs
#281

License
MIT

What's in this PR?
This commit fixes this issue by copying (by value) the metaArr
property to a local variable each time the filter is applied.
Why?
When the AddMetadataFilter is applied on an Audio media, the artwork
parameter was being unset from the metaArr array.
Saving the same Audio instance multiple times generated the
correct commands for the first export, but the subsequent commands were
missing the artwork parameter.
Example Usage
$ffmpeg = new FFMpeg\FFMpeg::create();

$audio = $ffmpeg->open('track.wav');

$audio->addMetadata([
    "title" => "Some Title",
    "album" => "Some Album",
    "artwork" => "/path/to/some/file.jpg"
]);

// Now we can do

$mp3 = new FFMpeg\Format\Audio\Mp3();

// The first time we export a 320kbps MP3

$mp3->setAudioKiloBitrate(320);
$audio->save($mp3, 'track-hd.mp3');

// The second time we export a 128kbps MP3

$mp3->setAudioKiloBitrate(128);
$audio->save($mp3, 'track-sd.mp3');

Nice one. I'll merge it.


use dask.array functions instead of numpy

I had hoped for some performance gains, but this is not so clear.

NB:
Most of the dask functions you have changed are just calls to the same numpy functions using ufunc, see the "direct" entries here: https://docs.dask.org/en/stable/array-numpy-compatibility.html#compatibility-with-numpy-functions .
Changing does give a bit of consistency, I'm not opposed.


Clarify import from go2rtc to hass generic camera

The protocol is not set by default. According to my tests, only TCP works.

This is not necessary option. Important not to select UDP.

You are right, I just tested here. Ok, I updated the PR. Can you take a look again, @AlexxIT?

Don't think this really necessary, but anyway

Ok, sorry. I could have closed the PR too. Have a great day/night anyway.