Breadcrumbs & context wrapping broken with native promises

I've been pulling my hair out trying to get breadcrumbs working in my Serverless Lambda services, but I can never seem to get it working. Like not even a little bit...
I've gone over the docs multiple times, and it seems like I'm doing everything as recommended. At this point I don't know what else to try.
Background

Code written in ES6
Built with Webpack v1 via serverless-webpack plugin
Compiled using babel-loader
Auto-breadcrumbs are disabled
Using latest v1.1.1 of raven-node

Example Code
Here's a simplified example of one of my functions.
// Lambda function handler
import Raven from 'raven'
Raven.config('my-dsn', { /* my config */ }).install()

import MyClass from './my-class'

export const handler = Raven.wrap((event, context, callback) => {
  new MyClass(event, callback).startDoingStuff()
})

// my-class.js
import Raven from 'raven'
import AWS from 'aws-sdk'
const ddbClient = new AWS.DynamoDB.DocumentClient()

export default class MyClass {
  constructor (event, callback) {
    this.event = event
    this.callback = callback
    Raven.setContext({ extra: this.event })
  }

  startDoingStuff () {
    Raven.captureBreadcrumb({ message: 'started doing stuff' })
    return this.doTheStuff()
      .then(this.itWorked.bind(this))
      .catch(this.itBroke.bind(this))
  }

  doTheStuff () {
    return new Promise((resolve, reject) => {
      ddbClient.query({ /* query params */ }, (error, data) => {
        if (error) {
          Raven.captureBreadcrumb({ message: 'lol nope' })
          return reject(error)
        }
        resolve(data)
      })
    })
  }

  itWorked (data) {
    this.callback(null, data)
  }

  itBroke (error) {
    Raven.captureException(error, (sentryError, eventId) => {
      this.callback(error)
    })
  }
}

Results
If something goes wrong in doTheStuff(), the events that get logged to Sentry don't have any of the extra context I tried setting in the constructor, and don't have any breadcrumbs other than the final exception...
Also, my Lambda logs on AWS are full of entries like<EMAIL_ADDRESS>alert: getContext called without context; this may indicate incorrect setup - refer to docs on contexts.
So it seems like a problem with the Node.js domain API. Maybe because raven-node seems to rely on implicit binding?
I'm not really familiar with the domain API, but if a different library also tried to use domains (say maybe the AWS SDK), would changing the active domain break raven-node since it relies on domain.active.sentryContext? IDK if that's even how it works...
Anyway, is there a way to disable node domains all together and just run everything in a global context? Domains aren't really useful in an AWS Lambda environment IMO (also, they're deprecated).

Hey @adambiggs, thanks for opening this. I also created a proof-of-concept example here demonstrating our failure to carry context across promise resolution boundaries with some comments on exactly what's going on.
In short, our contexts don't carry across promises like we'd like them to, and that's mostly the source the problem you're running into. With the additional background you've provided here, I think I think I can propose the following:

We accept/merge the change in #252 to make globally-scoped breadcrumbs be captured
We make any other changes necessary to stop treating the global context as a thing that should be avoided
You can just throw everything in the global context and avoid using wrap/context methods (which lets you dodge the issue of contexts not propagating across promise resolution boundaries)
Your breadcrumbs should start working due to bullet 1 and you should stop seeing strange error messages due to bullet 2
Especially in the Lambda use case, leaning on global context isn't a big deal since we don't have to worry about juggling multiple contexts anyway: each lambda run is sort of a one-off execution

Longer term - I might be able to come up with a clever way to make our contexts propagate through Promises (initial impression: cautiously optimistic but need to experiment to see how viable, will investigate), or we might just note that our contexts don't work with Promises until async_hooks lands and we migrate our context implementation to be based on async_hooks instead of based on domains.
I'll write more on our take on domains being noted as "deprecated" and our usage of them in #264. The implicit binding bit is related but a little bit of a separate topic - there's a sort of rift between promises and all other async stuff in Node because of how the V8 microtask queue works. It's an ongoing issue in Node core that we're following, but maybe in the meantime we can find a better workaround than what we have now (just being broken).

Thanks for the quick reply @LewisJEllis. And thanks for the validation that I'm not going crazy!
All your points seem sensible, and it looks like merging #252 would fix my problems. As a user, global context is what I would expect as default behaviour anyway, and wrap/context would seem more of an "advanced" use case instead of a best-practice (especially given the current async gotchas).

@adambiggs it just occurred to me while writing up my response to #264 that if you use Bluebird instead of native promises (just throw var Promise = require("bluebird"); at the top of your script), everything should work for you. It's obviously not ideal to introduce an additional dependency, but bluebird is quite reliable and it's probably the easiest solution. It's also a viable solution for people who aren't in your fortunate position of being okay with using only the global context.
Note: I'm still going to make the changes I described here/in #252.

FWIW I believe I need this outside of the Lambda use-case. Because I use async/await in scripts w/ a top-level native promise, if I want to attach breadcrumbs in those scripts I need something like a global context. As it is, I am seeing the same (non-attach) behavior as this issue.

@LewisJEllis thanks for the tip, maybe bluebird it can hold us over until #252 is ready. But bluebird isn't ideal as a long-term solution since we're trying to keep our Lambda code as light-weight and dependency-free as possible.

I've released v1.1.2 with the changes I described a few comments above; should help @adambiggs's case.
I've also realized that trying to automagically work with native promises used by any other dependencies you might have isn't going to be feasible, and it's the same deal with async/await. It'd be one thing if Promises were a node core library and we could piggyback on module caching, but since they're a global scope native the scope of what we can pull off via monkeypatching is much more limited. I don't think there's a good solution for native promises or async/await until async_hooks lands in node core and we can migrate as described in #264, so unfortunately I think we'll just have to point people to the bluebird workaround and hope for the best :/
@LegNeato I'm not sure exactly what the "bluebird with express middleware" case you're describing is, but it sounds like something we might be able to get working. Can you post an example snippet?

@LewisJEllis Even though I am using the express middleware (and thus have a sentry context), because I use async/await with native promises (?) breadcrumbs don't seem to be working. Even when I use bluebird in the module that creates the express app, I still don't get breadcrumbs. Interestingly, when I set the user context it does get sent up with the exception.
Breadcrumbs only appear to work when I use bluebird in every module used in the request...which is painful as I have hundred of modules. For production I can probably have webpack inject bluebird promises everywhere, but in development we use babel-node directly.
I'll try to distill my code into an example snippet.

If you're using async/await, it's not gonna be able to work - async/await runs on top of native promises, so even if you include bluebird I wouldn't expect it to help. I don't think there's an easy way to make async/await run on top of bluebird instead (maybe through some goofy babel/build setup to have it shim promises and then compile async/await in terms of that? not sure).

Ah, yeah. That makes sense why it worked for some scripts using async/await...I use a top-level promise and that is where I was testing the bluebird workaround.
Looks like I may be able to use https://github.com/chpio/babel-plugin-transform-promise-to-bluebird. Pretty big bummer to not be able to use native promises though.

Does this mean that the Express middleware does not work with async/await/native promises? i.e. will this work?
var app = require('express')();
var Raven = require('raven');

Raven.config('__DSN__').install();
app.use(Raven.requestHandler());

// async function now! So it catches the error and returns/rejects a promise.
app.get('/', async function mainHandler(req, res) {
    throw new Error('Broke!');
});

// Will this + `requestHandler` handle `mainHandler`'s return value's rejection, or no?
app.use(Raven.errorHandler());

app.listen(3000);

If it doesn't work, is Sentry aware of some other technique for adding a "global" rejection handler to log the error and return a 500?

It looks like mainHandler's rejection will not be handled by default. What do the Sentry folks think of this technique for patching express' Layer object to catch route handler rejections? From here.

Oops, didn't see your earlier comment. No, I wouldn't expect that to work unless express automatically routes that rejected promise's rejection "reason" to the errorHandler middleware, which doesn't seem to be the case.
That patching technique seems sound and viable at first glance/understanding, but I would have to experiment with it and better understand exactly how promises/async functions interact with express and its layer/middleware stuff. We'd also have to think about the right way to do it so as to affect behavior as little as possible and keep it easy for users to understand what to expect when they turn on this potential option.
Unfortunately that doesn't help much with the native promises situation, but I did recently see hapijs/lab#695 with an idea for binding the active domain by just patching Promise.prototype.then - I don't think it'll work 100%, but it would probably make a lot of common cases work like we'd want. Something worth revisiting once some higher-priority/lower-hanging things are finished.

Hey @LewisJEllis, thanks for your response.
Unfortunately I do not think that https://github.com/hapijs/lab/issues/695 is of much use because it only handles uncaught exceptions, not rejections:
const Raven = require('raven');

Raven.config('__DSN__').install();

const app = require('express')();

app.use(Raven.requestHandler());

/* Insert the patch from https://github.com/hapijs/lab/issues/695 here */

app.get('/reject', async () => {
  // This request will hang (unless you have this patch: https://gist.github.com/q42jaap/f2fb93d96fda6384d3e3fc51977dec90)
  throw new Error('boo');
});

app.get('/reject-in-promise', () => {
  // This request will hang.
  new Promise(() => {
    throw new Error('boo');
  });
});

app.get('/reject-in-then', () => {
  // This request will hang.
  Promise.resolve().then(() => {
    throw new Error('boo');
  });
});

app.get('/reject-in-then-async', () => {
  // Only this response will terminate.
  Promise.resolve().then(() => {
    setTimeout(() => {
      throw new Error('boo');
    });
  });
});

app.use(Raven.errorHandler());

app.use((err, req, res, next) => {
  res.status(500).send(err.message);
});

requestHandler's domain doesn't emit errors in any case except requests to /reject-in-then-async, presumably because all the other errors are caught—by Promise, internally, only then to be reported as unhandled rejections.
I say that https://github.com/hapijs/lab/issues/695 isn't of much use because it's not adding support for promises per se, it's adding support for mixing callbacks with promises—but I'm looking to move my code over to use promises pretty much entirely; and I don't really plan to chain promises using then because I'll be awaiting them.
So for me, I think the Express patch is pretty much what I need because I expect all uncaught exceptions to bubble up to the top layer via await. And then I'll probably shut down the process on 'unhandledRejection', like I currently do on 'uncaughtException', to make sure I'm not missing anything.
I understand and appreciate your concern to do what's right! At the same time, I hope you do investigate this situation more in the near term as I would imagine that more and more of the ecosystem will begin migrating towards the use of promises since Node 7.6.0 shipped with async/await support.

@wearhere yeah I wrote that Promise patch to support domains. My use case might be different then yours. I think it's better to rely on the Promise API for error-handling. Using domains (for servers) is IMO only for a last ditch effort to send a response.
If you don't have domains, then you pretty much have to shut down a server, upon an uncaughtException, otherwise leaks everywhere, because you will not be able to send a response for the request that caused the error.
Here is how I use domains in production servers:

const domain = require('domain');

app.use(function(req, res, next){

   const d = domain.create();
   d.once('error', makeHandleError(res));
   d.run(next);

});

Your handleError fn should look like
function makeHandleError(res){

  return function(err){

     // log the error here, and see your stack trace

     if(!res.headersSent){
        /// Send the error response how you like
      }

   }

}

IMO, domains are only a latch ditch effort to catch errors. I never saw them as reliable enough to catch all errors. Use the Promise API to do 99.99% percent of that.
The Promise.prototype.then patch will help you catch the weirdest of errors...but again domains should only be used as a last line of defense.
Other the other hand, for applications that are not servers, for example test frameworks, domains are more useful, TapJS, Suman, Lab, all use them, for good things.

Good news!  The Node.js 8.0.0 release notes say:

Native Promise instances are now Domain aware [84dabe8373] #12489.

I think that means you don't need to patch Promise.prototype or use Bluebird anymore

Yes, it does mean that; I was quite happy to see that note. The example I made in my first comment behaves differently on Node.js 8 - the key equality comparison now prints false instead of true! 8.x also brings lots of other exciting possibilities that will be coming up in raven-node 3.0 :)
I'm going to close this with a suggested resolution of:

If you can, use Node.js 8
If you can't use Node.js 8, use Bluebird promises
If you can do either of those, avoid using context() or wrap() and use only the global-scope context in Raven


Fix bugs of IbisDataFrame and IbisExecutionEngine

Codecov Report
Base: 100.00% // Head: 99.98% // Decreases project coverage by -0.01% :warning:

Coverage data is based on head (7fb6ad0) compared to base (5d887ba).
Patch coverage: 100.00% of modified lines in pull request are covered.

Additional details and impacted files
@@             Coverage Diff             @@
##            master     #368      +/-   ##
===========================================
- Coverage   100.00%   99.98%   -0.02%
===========================================
  Files          110      110
  Lines        10295    10297       +2
===========================================
  Hits         10295    10295
- Misses           0        2       +2

Impacted Files
Coverage Δ

fugue_ibis/dataframe.py
97.87% <100.00%> (-2.13%)
:arrow_down:

fugue_ibis/execution_engine.py
100.00% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


WHO IS YOUR DADDY AND WHAT DOES HE DO

If ArnoldC ever supports hash maps, I nominate this one-liner as the operator for reading the value for a given key.
Example:
Given the map { "choppa" : "now" }, the following code
WHO IS YOUR DADDY AND WHAT DOES HE DO "choppa"

would return "now"

I think "who is your daddy..." could be used for authentication!

I think it would be better match for extends


2.0.0 needs to be pushed to CocoaPods trunk

1.1.0 is still the latest version listed in CocoaPods.

Thanks, still cleaning up a few things first!

No problem, just a reminder then I guess.

Closed by 4d86fea8ee3a6c60e27a1426684b1682c54d9ba0.
Thanks for the reminder!


cant install upgrade-assistant

Problem encountered on https://dotnet.microsoft.com/platform/upgrade-assistant/tutorial/install-upgrade-assistant
Operating System: windows
Provide details about the problem you're experiencing. Include your operating system version, exact error message, code sample, and anything else that is relevant.
C:\Users\avistokar\Documents\OA>dotnet tool install -g upgrade-assistant
C:\Program Files\dotnet\sdk\6.0.100\NuGet.targets(130,5): warning : The plugin credential provider could not acquire credentials. Authentication may require manual action. Consider re-running the command with --interactive for dotnet, /p:NuGetInteractive="true" for MSBuild or removing the -NonInteractive switch for NuGet [C:\Users\avistokar\AppData\Local\Temp\gthvyyew.e44\restore.csproj]
C:\Program Files\dotnet\sdk\6.0.100\NuGet.targets(130,5): warning : The plugin credential provider could not acquire credentials. Authentication may require manual action. Consider re-running the command with --interactive for dotnet, /p:NuGetInteractive="true" for MSBuild or removing the -NonInteractive switch for NuGet [C:\Users\avistokar\AppData\Local\Temp\gthvyyew.e44\restore.csproj]
C:\Program Files\dotnet\sdk\6.0.100\NuGet.targets(130,5): error : Unable to load the service index for source https://pkgs.dev.azure.com/hc-ms/_packaging/HCMS/nuget/v3/index.json. [C:\Users\avistokar\AppData\Local\Temp\gthvyyew.e44\restore.csproj]
C:\Program Files\dotnet\sdk\6.0.100\NuGet.targets(130,5): error :   Response status code does not indicate success: 401 (Unauthorized). [C:\Users\avistokar\AppData\Local\Temp\gthvyyew.e44\restore.csproj]
The tool package could not be restored.
Tool 'upgrade-assistant' failed to install. This failure may have been caused by:

You are attempting to install a preview release and did not use the --version option to specify the version.
A package by this name was found, but it was not a .NET tool.
The required NuGet feed cannot be accessed, perhaps because of an Internet connection problem.
You mistyped the name of the tool.

For more reasons, including package naming enforcement, visit https://aka.ms/failure-installing-tool
I tried this :
C:\Users\avistokar\Documents\OA>dotnet tool install -g --interactive  upgrade-assistant
after a long pause I got this:
C:\Program Files\dotnet\sdk\6.0.100\NuGet.targets(130,5): error :     [CredentialProvider]Device flow authentication failed. User was presented with device flow, but didn't react within 90 seconds. [C:\Users\avistokar\AppData\Local\Temp\f3aem1rv.vsb\restore.csproj]
C:\Program Files\dotnet\sdk\6.0.100\NuGet.targets(130,5): error : Unable to load the service index for source https://pkgs.dev.azure.com/hc-ms/_packaging/HCMS/nuget/v3/index.json. [C:\Users\avistokar\AppData\Local\Temp\f3aem1rv.vsb\restore.csproj]
C:\Program Files\dotnet\sdk\6.0.100\NuGet.targets(130,5): error :   Response status code does not indicate success: 401 (Unauthorized). [C:\Users\avistokar\AppData\Local\Temp\f3aem1rv.vsb\restore.csproj]
The tool package could not be restored.
Tool 'upgrade-assistant' failed to install. This failure may have been caused by:

You are attempting to install a preview release and did not use the --version option to specify the version.
A package by this name was found, but it was not a .NET tool.
The required NuGet feed cannot be accessed, perhaps because of an Internet connection problem.
You mistyped the name of the tool.

For more reasons, including package naming enforcement, visit https://aka.ms/failure-installing-tool

@stokara Please try with ignore-failed-sources, this error should go away.

https://github.com/dotnet/sdk/issues/18402 to track executing by default with ignore-failed-sources


Add support to config module for on-call schedules (and some other fixes)

what

Update Github Actions
Adding support for schedules and schedule_rotations to config module
Updating config module README.md
Adding support for multiple rules to config module escalations
Resolves issue w/ team routing rules criteria
Resolves issue w/ team member sorting (or lack thereof)
Modifying for_each logic in config module to fix issue w/ mismatch between object schemas

why

We must have support for on-call schedules & rotations in the config module!

/test all

/test all

/test all

/test all


Documentation Site Down

Documentation appears to be down. @jagi have you released a new version that possibly had a glitch in it, or is Meteor hosting just down?
~Max

Because Jagi is awesome, you can just clone/download this repo and run the meteor project to access the doc's locally: https://github.com/jagi/meteor-astronomy-docs until the site is back online

@maxmatthews I haven't been doing anything to the documentation. It's the problem with free Meteor hosting which is not perfect. I'm gonna move entire documentation for 1.x and 2.0 to the different website. I haven't chosen what service I will use but maybe readme.io. I have to test a few different solutions.
And yes, you can use docs repo in cases where the docs website is down. Moving to another service will make it more reliable.

Awesome. I was happy when I found the github repo as a good offline backup. Thanks again for all the awesome work you do. Really excited about 2.0!

:)


URLEncodedUtils and NameValuePair shouldn't be here

There are only a few lines of code which use http-client's related APIs: URLEncodedUtils & NameValuePair.
List<NameValuePair> query = URLEncodedUtils.parse(parsedUri, "utf-8");

for (NameValuePair pair : query) {
	routerParams.openParams.put(pair.getName(), pair.getValue());
}

They shouldn't be here because:

It can be easily replaced
MultiDex 65k method count limits: http-client has 7521 methods in total. It doesn't seem to be a problem. But think of the limit is 65535, the cost is too much for a small router component.

They can be replaced by Uri. It handles urls such as xxx?key1=val+1&key2=val%202 pretty well.
Uri uri = Uri.parse(url);
Set<String> queryKeys = uri.getQueryParameterNames();
for (String key : queryKeys) {
	routerParams.openParams.put(key, uri.getQueryParameter(key));
}

Sure Uri.getQueryParameterNames() is only available since API 11 and if you want to support API 9, then you can also achive the goal by using Uri.getEncodedQuery(), it's available since API 1. Of course you need to parse the parameters set by your own, but that wouldn't be a problem.
This project uses http-client's API but there seems no jar file nor gradle dependence declared.
It's a nice project but I don't think anyone can run the testcases in this project without too much pain. I was going to make a pull request but I found this project doesn't practice gradle very well. I might need to refactor the whole structure if I want to start to work on this project, I guess.
A nice open source project should be friendly to other programmers, isn't it?

it's a great idea when someone not uses the httpClient anymore...


How to customize my own exception handler in token endpoint?

The exceptions thrown in each endpoint are caught by @ExceptionHandler methods in the same class. What i want to do is to catch these exceptions (such as OAuth2Exception, InvalidTokenException) in my @ExceptionHandler and then customize the output.
The raw exception output is something like {"error": "...", "error_description": "..."} which is not what i want. I expect the output to be {"code": 400, "message": "..."}.

This problem has been solved. Follow the workaround below:
1 extend OAuth2Exception to a new class, such as CustomOAuth2Exception. In the custom class, add some specific properties.
2 custom DefaultWebResponseExceptionTranslator and register the custom translator in AuthorizationServerConfiguration.
3 custom two jackson serializers annotated in OAuth2Exception and annotated your CustomOAuth2Exception with the two custom serializers.
4 use ObjectMapper to override initial serializers with custom serializers.

That's not really a workaround (WebResponseExceptionTranslator is provided for precisely this use case).

I'm sorry Dave, if that's not really workaround to solve this problem, what's the true. Because right now I am facing this problem and not solved yet.

I'm sorry Dave, if that's not really workaround to solve this problem, what's the true. Because right now I am facing this problem and not solved yet.

You don't need a workaround you need WebResponseExceptionTranslator (it's what it was designed for).

You don't need a workaround you need WebResponseExceptionTranslator (it's what it was designed for).

I used steps, which @jzbjyb specified and I can see that not all OAuth2 errors are caught in custom DefaultWebResponseExceptionTranslator.
So in case I send invalid refresh token the error is caught in my custom translator, but for instance, when I send invalid access token the error is NOT caught in custom translator, but only in DefaultWebResponseExceptionTranslator.
Are there any other places where I can register custom exception translator?
Maybe you have some thoughts why do I have that not all errors/exception are caught in custom exception translator?

Thanks to @dsyer 's suggestions! I came across the same problem just as @isaranchuk 's description. When  I send invalid access token, the error is not caught by my custom translator. What i know is that ExceptionHandler in oauth2's endpoint trigger translate() method of the translator class. But for InvalidTokenException, I don't know where it is handled.

@jzbjyb  as I can see AuthorizationEndpoint extends AbstractEndpoint where exception translator is initialized, see.
Seems there is an additional place where we need to define exception translator.
@dsyer what do you think?

How to set CustomOAuth2Exception in DefaultWebResponseExceptionTranslator  ?

In order to do this you need to extend DefaultWebResponseExceptionTranslator, so inheritance and polymorphism will be your friends.
Check @jzbjyb comment where you can find all steps to achieve appropriate result.

Can someone please elaborate on  @jzbjyb 's step 4?  on how to use OjectMapper ?

@jzbjyb How can i configure CustomOAuth2Exception,custom DefaultWebResponseExceptionTranslator in my security xml file.Can u elaborate your steps??

@dsyer  how can i set WebResponseExceptionTranslator in my xml configuration file

This is proving to be painful to implement, we have to add lots of code just to handle exceptions in a proper way, still not all of them are captured with this workaround, and yes, it is a workaround, because  the solution is not 'Spring-ish' look at the amount of code we have to do/test (and still it is not fully working) and compare it with exception handling in the MVC layer for example, there should be a solution to capture all exceptions in the authorisation process, and a proper way to add custom responses for these endpoints. I would love to see this issue re-opend.

Hi hassalih. Why is not springish in your opinion? When you have to customize a flow you add new handlers or translator or new service implementation. My needs were to customize only the response returned to the client compliant a specification.
I didn't need to change the flow, only change the response syntax.
If you have a better solution you are welcome.
Every algorithm is improvable infinitely for definition

Hi @lettierimarco
Look at the solution mentioned above, it is designed while looking at the problem from inside the code without looking at the outside world requirements, if I have a an exception and would like to send a response about it, then fine, I can send the response in a POJO and put in that POJO whatever properties/data I want, but here, you have to customise a hole lot of code to achieve a specific error design, I think the problem can be tracked down to this interface ( and many TokenEndpoint methods using it in the return statement)
`public interface WebResponseExceptionTranslator {
ResponseEntity<OAuth2Exception> translate(Exception e) throws Exception;

}`
The question is why the output is ResponseEntity<OAuth2Exception> ? We are sending a RESPONSE to the outside world from our web service, the structure of the response has to be flexible enough to be manipulated, say a POJO class which can be extended easily without thinking about serialisers. Something like ResponseEntity<POJOException> My point is that the way of calling the translator is correct, we need to call a translator, but the translator has to translate an inner exception to a POJO  facing the outside world, the problem here, the translator is translating from an exception to another exception !!

@hassalih you can rewrite the DefaultOAuth2ExceptionRenderer and change the output with your format. Just change the
OAuth2Exception exception = (OAuth2Exception) responseEntity.getBody();         CustomResponse<Object> response = new CustomResponse<Object>(exception.getOAuth2ErrorCode(),                 exception.getLocalizedMessage()); // You can define your own response format here.         if (response != null) {             writeWithMessageConverters(response, inputMessage, outputMessage);        }
After rewrite, you need to register it to

Then the output is pretty well as what you want.

Hi All ,
I am using Spring Security OAuth JWT token based implementation .
I want to handle Invalid Login Count if user hit oauth/token URI and enter invalid login credential ,At which place and How I can handle/check/catch invalid login exception to maintain login count for invalid attempts

Wow, this is probably one of the most difficult customization points I've ever experienced in Spring since it was called Interface21. There is no elegant way to handle this. Even inheritance can't be used as most methods are private.

Sorry for commenting on closed issue.
Probably easiest way is to make a custom OAuth2ExceptionJackson2Serializer, and then use it on your custom exception:
@JsonSerialize(using = MyOAuth2ExceptionJackson2Serializer.class)
public static class MyOAuth2Exception extends OAuth2Exception {
  // ...

I created new spring boot project (with oauht2 and zalando problems) which should be simpler. It should be possible run without any configuration.
Can be download here: https://bitbucket.org/ferinoc/zalando-web-oauth2.git
There is class IntegrationTest which contains "tests":
provideTokenCorrectCredentials - jwt token should be generated successfully to console
provideTokenWrongUserName - throw excetpion but exception is not catch in GeneralExceptionHandler
If you could check and try to make zalando problem works would be great :) Thank you.

@jzbjyb From your comments , I have learned much when I am facing the same question . Do you have a solution to handle all of the exceptions in OAuth2 to provide a uniform response format ? Thanks ahead .


support image filtering by minimum like to images

currently filtering support is only by max like.

Codecov Report

Merging #663 into master will increase coverage by 0.02%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #663      +/-   ##
==========================================
+ Coverage   37.75%   37.77%   +0.02%
==========================================
  Files          26       26
  Lines        2233     2234       +1
==========================================
+ Hits          843      844       +1
  Misses       1390     1390

Impacted Files
Coverage Δ

instabot/bot/bot_filter.py
46.39% <100%> (ø)
:arrow_up:

instabot/bot/bot.py
62.72% <100%> (+0.11%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 29fd0ff...71383a4. Read the comment docs.

Thanks!!!


Support using ReconfigurableOpenTelemetry instance through @Inject

Support using ReconfigurableOpenTelemetry instance through @Inject in addition to the traditional Jenkins hudson.ExtensionList#lookup().
Implemented introducing a com.google.inject.AbstractModule annotated with @Extension
Testing done

Submitter checklist

[x] Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
[x] Ensure that the pull request title represents the desired changelog entry
[x] Please describe what you did
[ ] Link to relevant issues in GitHub or Jira
[ ] Link to relevant pull requests, esp. upstream and downstream changes
[ ] Ensure you have provided tests - that demonstrates feature works or fixes the issue

@kuisathaverat please review


[sqlserver] run sqlserver test in non-utc timezone

What does this PR do?

Motivation

Review checklist (to be filled by reviewers)

[ ] Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
[ ] Add the qa/skip-qa label if the PR doesn't need to be tested during QA.
[ ] If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

Verified setting  env var TZ does change the sqlserver container timezone
docker exec -it compose-sqlserver-1 date
Tue Nov 26 10:42:23 EST 2024


how to use tch-rs/src/tensor/

Hello, i find something in the tch-rs/src/tensor/ directory, and the try_from trait is in the mod.rs file, could anyone tell me how to use the directory?

Closing as no update in the last few months, fwiw you can find a bunch of examples using this by grepping for try_from in the tests directory.


(MAINTENANCE) - VIP page with form for vips (edited)

Hello, @az I talked to Mark and he really would like to start with sending VIP invitations. He explained me steps and how we should work on it:

To run profile pages

[ ] button request a meeting should working
[ ] check the confirmation e-mail (request a meeting) if it's ok (do we have a draft somewhere?)

To run VIP invitations on the exhibitors' accounts:

[ ] we have to check the confirmation e-mail - do we have a draft somewhere?
[ ] if someone gets to the exhibitors profile with VIP invitation (special link), it would be good to remove "request a meeting" and usual form, because it can mislead the users

Adjust the E-commerce Berlin Expo profile for sending VIP invitations.

we don't need request a meeting button
we can remove usual form
we can add some info about benefits for VIPs

If adjustments on the E-commerce Berlin Expo profile won't be possible, Mark ask for running a temporary page:
https://ecommerceberlin.com/vip

@klaudia2209 how does it correlate to "account.ecommerceberlin.com" (this github project is devoted to account.ecommerceberlin.com)....i am moving it


More generic locker

Based on PR #56, merge it first.

Good commit. You can merge when you implemented my notes.

I'm now merging it into master (see #58).


Run async write in caller thread when queue is full for local cache

What changes are proposed in this pull request?
This PR will run the async write in call's thread when the queue is full.
Why are the changes needed?
Prevent the rejection of the async write during busy time
Does this PR introduce any user facing changes?
no

I will take a look at the test

alluxio-bot, merge this please.


is there any separate css file?

Subject of the issue
a set of .css is better.
is there any plan about it?

Hi @JadeTao !
Nope, the plan for the project is to continue with A Unified Styling Language, a CSS in JS implementation.
I wrote my reasons over here: CSS in JavaScript with JSS and React.
Thanks for asking!


refactor: move reconnect banner to a new component

Fixes Issue
**My PR closes #662 **
👨💻 Changes proposed(I had made another component inside component 📁 named as ReconnectBanner.jsx and import it in BuddyMatcher.jsx)
✔ Check List (Check all the applicable boxes)

[x] My code follows the code style of this project.
[x] This PR does not contain plagiarized content.
[x] The title and description of the PR is clear and explains the approach.

Note to reviewers

📷 Screenshots

@Dun-sin I think Now I had complete the Compilance/PR Checks .
I had setup the code locally before, but due to the issue I had to delete it. But my codes work fine.
You can check at your own to. Hope it get merged soon.

@Dun-sin I think Now I had complete the Compilance/PR Checks .
I had setup the code locally before, but due to the issue I had to delete it. But my codes work fine.
You can check at your own to. Hope it get merged soon.

If you can't test it locally, you can use the preview link from vercel right here in the PR and attach the screen recording

@LuseBiswas you also need to remove the package.json file from your PR, don't delete it, remove it. There's a difference please

@LuseBiswas looks like your build fails


Update Composer dependencies (2024-01-16-00-18)

Visual regression test passed!
View the visual regression test report


Update employee manager and delete functions.

delete functions include:
Delete a department.
Delete a role.
Delete a employee.

DONE added the functions.


TS error

Looks like the module declaration in client.d.ts conflicts with the default one from Vite here.
A simple fix would be to add a special loader query, like so:
declare module '*.svg?component' { ... }
But this would then force users of this plugin to write
import { ReactComponent as Foo } from 'path/to/foo.svg?component'
instead of
import { ReactComponent as Foo } from 'path/to/foo.svg'
Right?

Come to think of it, this might also be a good occasion to switch the default export to the component rather than the src, so instead of
import { ReactComponent as Foo } from 'path/to/foo.svg?component'
one could do
import Foo from 'path/to/foo.svg?component'
to get the React component, and doing
import Foo from 'path/to/foo.svg'
would "fallback" to the default Vite behavior which return the source, which I think makes sense.
But all these would definitely be breaking changes. But that might be fine for 0.x release, I guess ?

Looks like removing export default src would solve this issue?

Looks like removing export default src would solve this issue?

Yes, I can re-assign the value and it can be used! But it looks like the author of the library needs to modify it!

Should be fixed in the latest v0.4.0, by https://github.com/pd4d10/vite-plugin-svgr/pull/12


All background colors are incorrect

None of the background colors match the same foreground color, e.g., background color "011" is a different color from the foreground color "011". This makes it hard to find the right background color since they don't match the handy one-liner to display all colors. Note that the foreground colors I've tested do match the one-liner.
I have set all of the dir state background colors to "011" which is yellow.
POWERLEVEL9K_DIR_HOME_BACKGROUND="011"
POWERLEVEL9K_DIR_HOME_FOREGROUND="000"
POWERLEVEL9K_DIR_HOME_SUBFOLDER_BACKGROUND="011"
POWERLEVEL9K_DIR_HOME_SUBFOLDER_FOREGROUND="000"
POWERLEVEL9K_DIR_DEFAULT_BACKGROUND="011"
POWERLEVEL9K_DIR_DEFAULT_FOREGROUND="000"
POWERLEVEL9K_DIR_ETC_BACKGROUND="011"
POWERLEVEL9K_DIR_ETC_FOREGROUND="000"

The separator uses the correct color, but the background is a brownish orange that looks more like "003".

To make sure it's not the terminal messing with background colors, can I change the %F below to something else to have it change the background color instead of the foreground color? I tried %B (nope) and could not figure out the right Google query to find the codes. All I can find are ANSI escape codes.
for code ({000..255}) print -P -- "$code: %F{$code}This is how your text would look like%f"
                                          ^^

This may be related or the cause of the non-matching separators seen in #614.
System: Ubuntu 17.10
Terminal: GNOME Terminal 3.24.2
Font: DejaVuSansMono Nerd Font Mono Book

Here's a bit of history... The first 16 colors in a 256 color terminal are not actually colors. They are mappable pointers to any of the colors in from 16..255 (0 is the first). This was done so that you can set themes for your terminal.
If you want "true" colors, you should use the colors in the 16..255 range so that the color doesn't change when you change themes.

That makes sense, but wouldn't 011 as printed in the terminal using print match 011 as printed in the terminal using ZSH? If the first sixteen colors pointed to different colors for the foreground and background, that would explain the mismatched separator. Though, only the separators for sections I changed the background color of are mismatched.
In any case, I started with the shades of gray at the end (232 to 255) for the right side areas, and they were also different. However, that was when I first started so I'll try using some of the other colors in-between again.

Hey bro, take a look in this post, basically, if you are using Iterm It can help you.
https://github.com/bhilburn/powerlevel9k/wiki/Troubleshooting#segment-colors-are-wrong


lp

Bug reporting
A good bug report has some very specific qualities, so please read over our short document on reporting bugs before submitting a bug report.
To ask a question, go ahead and ignore this.

@Thomasnss ?


auto-detect if a resource has a new line at the end or not

Problem
See https://github.com/inlang/inlang/issues/875.
Proposal

When reading a resource, check if the JSON file contains an empty line at the end.
If a resource contains an empty line, store in resource.metadata that an empty line exists.
Upon serializing a resource, append a new line to the file if resource.metadata.emptyNewLine === true.

@samuelstroschein we should ship that asap, for jsonPlugin and i18next. When can we start with that? Are you still in development with the plugins?
I see this problem at every repository.

@NilsJacobsen yeah still doing the cal.com i18next update.
Feel free to move the plugin-json to the monorepo https://github.com/inlang/inlang/issues/936 and start implementing the feature there.

This is now fixed with the newest version of the plugin-json.
For those who come across this issue later on, here is the import for the latest version:
const { default: pluginJson } = await env.$import(
    "https://cdn.jsdelivr.net/npm/<EMAIL_ADDRESS>)


Node "Remove Mixlab Background" does not see u2net model

I have two models: u2net and u2netp, but node in my workflow dont see this models. Why?
This is node mix lab :

My files:

Info from cmd:
Failed to validate prompt for output 89:

RembgNode_Mix 85:

Value not in list: model_name: 'u2net' not in []
Output will be ignored
Failed to validate prompt for output 23:
Output will be ignored
Failed to validate prompt for output 98:
Output will be ignored
Failed to validate prompt for output 85:

(prompt):

Value not in list: model_name: 'u2net' not in []
Output will be ignored

I have two models: u2net and u2netp, but node in my workflow dont see this models. Why? This is node mix lab :  My files:  Info from cmd: Failed to validate prompt for output 89:

RembgNode_Mix 85:

Value not in list: model_name: 'u2net' not in []
Output will be ignored
Failed to validate prompt for output 23:
Output will be ignored
Failed to validate prompt for output 98:
Output will be ignored
Failed to validate prompt for output 85:

(prompt):

Value not in list: model_name: 'u2net' not in []
Output will be ignored

放到 models/rembg 文件夹下

I have two models: u2net and u2netp, but node in my workflow dont see this models. Why? This is node mix lab :  My files:  Info from cmd: Failed to validate prompt for output 89:

RembgNode_Mix 85:

Value not in list: model_name: 'u2net' not in []
Output will be ignored
Failed to validate prompt for output 23:
Output will be ignored
Failed to validate prompt for output 98:
Output will be ignored
Failed to validate prompt for output 85:

(prompt):

Value not in list: model_name: 'u2net' not in []
Output will be ignored

放到 models/rembg 文件夹下

Thank you. Now its ok.


[WIP] Temporarily disable test_resource_policy_auditor.py

WIP: Do not merge

Coverage decreased (-0.6%) to 71.415% when pulling ceab941f9e0af468b1aa170244a1980e98ad140a on cclauss:skip-blocking-tests into e17bb7fc9ac19919d1eaddb0a31ef4d9849e62fe on Netflix:develop.


Investigate Github Codespaces and how we can use them with Corteza

Investigate Github Codespaces and how we can use them with Corteza for frontend and backend development, writing Cypress tests and docs.

I took a look and I do not see this as an option for outside devs. We'll have to create dev-env on our pwn. Closing.


feat: .NET SDK - Host Functions, Async, and Cancellation Support

Continuing mhmd-azeez's .NET Host Functions Support with the following enhancements:

Implement asynchronous plugin calling
Support early cancellation using prerelease plugin-cancel branch
Add a bunch more unit tests including async tests
Fix missing handling of error case in creating plugin where a -1 pointer is returned

Still TODO:

Implement host functions registration into extism_plugin_new (currently passes an empty list of plugins)
Make an easy way to register host functions in a .NET friendly way
Add example host functions code to sample file
Test setting timeout and that it still works with unit tests

This PR depends on #270 .  Once that is merged, this should pass CI.

Closing PR due to attribution issue, will redo.


doc: demos: Function Plotter stylistic improvements

Refactored code to be more DRY and more generic.

Nice! Has the clip for the doc page to be redone?
Can you run isort to make the linter happy again (you can use make isort if you are on unix)

Giving up on making isort happy. This is what isort thinks the imports should look like when running on my computer using
>>> import isort
>>> isort.file('plotter_picocss.py')

Cannot run make isort since I'm on Windows.  Furthermore I think the semantic position of the additional NumberInput in the original input statement was totally fine.

@markusgritsch: No problem, thanks for trying. I ran make isort on your branch and pushed the resulting patch. If you are ok with this change, please squash all commits together

Codecov Report
Patch coverage has no change and project coverage change: -0.41 :warning:

Comparison is base (43213b5) 68.96% compared to head (6c6ae8b) 68.55%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #379      +/-   ##
==========================================
- Coverage   68.96%   68.55%   -0.41%
==========================================
  Files          68       68
  Lines        5168     5168
  Branches     1164     1156       -8
==========================================
- Hits         3564     3543      -21
- Misses       1326     1351      +25
+ Partials      278      274       -4

see 12 files with indirect coverage changes
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

@markusgritsch: I think that's close enough for now :) thanks for updating!

@markusgritsch: The docs are deployed to lona-web.org

schwere Geburt :)


Update README.md

This PR's title doesn't match our requirement.
Please make sure your title includes the corresponding Jira id.
@juntao, please update it.

This PR's title doesn't match our requirement.
Please make sure your title includes the corresponding Jira id.
@juntao, please update it.

This PR's title doesn't match our requirement.
Please make sure your title includes the corresponding Jira id.
@juntao, please update it.

Welcome

Thank you for raising a PR! We will review it shortly!


Add commands for gptj and gptneox

What does this PR do?

Adds commands for finetuning GPTJ-6b and GPTNEOX-20b on wikitext data set.
Here are the details :
GPTJ-6b was fine tuned for Language modelling with Wikitext data set on 8 Gaudi2 cards

GPTNEOX-20b was finetuned for Language modelling with Wikitext data set on 16  Gaudi2 cards

Before submitting

[ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
[ ] Did you make sure to update the documentation with your changes?
[ ] Did you write any new necessary tests?

@regisss  : Addressed review comments , kindly check and approve

@regisss  : Addressed review comments, hope I got your comments on the multi-node training for gpt-neox-20b correct.


cffi.error.VerificationError: LinkError: command 'gcc' failed with exit status 1

Python 2.7 Anaconda
Cuda-8.0

The nvcc path in make.sh file was wrong. Resolved.

@Bala93 ,hello i meet with the same problem,can you help me?

@Bala93 @Markovcom, hi i met same problem,can you tell me your solution?

@li10141110 please see this,https://github.com/jwyang/faster-rcnn.pytorch/issues/114#issuecomment-378286240

For me, I don't have root access. I can't make such symlinks to the system. So I just symlinked to the problem dir and it works now. Not recommending this, just for those in a similar situation to me with whom no environmental variable setting has helped.
ln -s ~/cuda/include/* ~/anaconda3/lib/python3.6/site-packages/torch/utils/ffi/../../lib/include/THC/

For me, I don't have root access. I can't make such symlinks to the system. So I just symlinked to the problem dir and it works now. Not recommending this, just for those in a similar situation to me with whom no environmental variable setting has helped.
ln -s ~/cuda/include/* ~/anaconda3/lib/python3.6/site-packages/torch/utils/ffi/../../lib/include/THC/

@Bala93 How did you resolve it, please? Same problem


[Feature][connector-v2][mongodb] mongodb support cdc sink

Purpose of this pull request
close #4834

Check list

[ ] Code changed are covered with tests, or it does not need tests for reason:
[ ] If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
[ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
[ ] If you are contributing the connector code, please check that the following files are updated:

Update change log that in connector document. For more details you can refer to connector-v2
Update plugin-mapping.properties and add new connector information in it
Update the pom file of seatunnel-dist

[ ] Update the release-note.

@hailin0 PTAL, thanks.

@EricJoy2048 PTAL, thanks.


clean up around the tty.

I think the tty area was probably a point that the developers were wondering about, so I commented on the organization and difficulties.

Thank you for this, had been wanting to ask you regarding the socket creation logic, this clean up makes it a lot better

Thanks for this! I liked that you included the reference to containerd. I had wondered about how it is handled from containerd side.

lgtm


Potentially add factory

One idea to avoid these many if statements link is to add a factory. An example is the ESS adapter and factory in RMG-Py's Arkane so at the bottom of each adapter we just add a line to register the adapter link then we could just do something like _registered_adapters[sampler_name] rather than a long chain of if, elif, elif, elif, elif, elif etc

This is a nice design pattern -- I will do it and open a PR.


Dashboard OperationalError fix

This resolves #160  by adding a function to check the connection is alive, and if not, creating a new one. This is called every time the connection is used, so the error should never occur again. Also some minor formatting changes.

Looks great, but I was wondering would it not be better to check the connection at their respective pages, rather than whenever you call a query?

Looks great, but I was wondering would it not be better to check the connection at their respective pages, rather than whenever you call a query?

That's actually a great idea, I'm going to implement that now

Cheerios!


!roulette % values

!roulette 50% to bet 50% of your points

The % character is a bit tricky. You can avoid it by using decimal percentage like 0.5 or if you don't want to use the dot use a comma like 0,2 for 20% of the points

Why is the % character tricky?
I could just, before I cast it to an int check if the last character is %
like this:
if msgparts[0].endswith('%'):
  percentage = max(0, min(100, int(msgparts[0][:-1])))
  value = user_points * (percentage / 100)
else
  value = int(msgparts[0])

I guess it is more of a protocol thing rather than trickiness. The % character is treated as NUL if I am not mistaken therefore it shouldn't be permitted in IRC based chats.


Use the shared OpenApi Serializer

To use the common gem to serialize an object as JSON we need :prefixes.
https://github.com/ManageIQ/manageiq-api-common/blob/a674fe29752de460977b445a482fcc2a8db728c7/lib/manageiq/api/common/open_api/serializer.rb#L10
What I found researching this issue
Since the Catalog and Approval API were missing :prefixes we could not use the shared common gem and had our own in the catalog and approval services
The Sources API and Topology API controllers have the ActionView::Rendering in their object hierarchy (brought in by the rail tie in the  jbuilder gem). The Catalog & Approval API don't have the ActionView::Rendering because they use the acts_as_tenant gem in the Gemfile which does a require and conflicts with the jbuilder railtie. This is a known issue in acts_as_tenant
https://github.com/ErwinM/acts_as_tenant/issues/201
The workaround suggested is to delay load the acts_as_tenant gem so it doesn't conflict with the jbuilder railtie by doing a require 'acts_as_tenant' some place else. In our case we don't need it in our Gemfile because the manageiq-api-common gem already has a it.
This PR attempts to figure out why Catalog service is missing the :prefixes.
The Sources and Topological API's use the Application controller as
class ApplicationController < ActionController::API

which is the same as how the Catalog API uses the Application controller
class ApplicationController < ActionController::API

The ApplicationController in Topo and Sources API have this hierarchy
[ActionController::Rendering, ActionView::Rendering, AbstractController::Rendering]
The Application Controller in Catalog and Approval have this hierarchy
[ActionController::Rendering, AbstractController::Rendering]
The ActionView::Rendering adds the prefixes method from the controller here
https://github.com/rails/rails/blob/5f3848a979e6be15a4d8bedcb9c68587094094fe/actionview/lib/action_view/rendering.rb#L163
The prefixes are needed so that we can use the shared gem.

@mkanoor can you move this WIP until we know for sure what direction we're going to take?

@bdunne Can you eyeball this? 👀

We are still trying to figure out who adds the ActionView::Rendering to topo and sources.

I think @jrafanie had a technique for figuring this out (if I recall it was something like patching ActionView::Rendering.included and dumping a call stack.)

I think @jrafanie had a technique for figuring this out (if I recall it was something like patching ActionView::Rendering.included and dumping a call stack.)

yeah, if you print the argument passed to included, I think that tells you who included it

On further researching this issue we found that ActionView::Rendering doesn't get included if acts_as_tenant & jbuilder gem are included together in an app
https://github.com/ErwinM/acts_as_tenant/issues/201
We are depending on :prefixes to be set by ActionView::Rendering which will only work if jbuilder is in the mix. If there is no jbuilder we won't see the :prefixes.
Wouldn't it be better that if we use our current_request to get the original URL rather than depend on :prefixes to be inserted magically by ActionView::Rendering. The catalog/approval/sources and topology are not using the jbuilder gem with the templates, our app would work without the jbuilder gem our input/output parameters are defined in the openapi spec and we wont need the JSON builders.

On further researching this issue we found that ActionView::Rendering doesn't get included if acts_as_tenant & jbuilder gem are included together in an app
ErwinM/acts_as_tenant#201

I opened a PR since acts_as_tenant completely busts other gems that want to "enhance" AC::Base or AR::Base in on_load hooks:
https://github.com/ErwinM/acts_as_tenant/pull/212

Wouldn't it be better that if we use our current_request to get the original URL rather than depend on :prefixes to be inserted magically by ActionView::Rendering. The catalog/approval/sources and topology are not using the jbuilder gem with the templates, our app would work without the jbuilder gem our input/output parameters are defined in the openapi spec and we wont need the JSON builders.

I originally noticed that jbuilder was involved because I tried to remove it from topology and the prefixes went away.  I wouldn't mind removing the dependency on jbuilder if we have a better solution for getting the request path for serialization.

@bdunne This is the code we use, which is a clone of your function with the subtle change to get the path from the original_url which is available from the current_request.
https://github.com/ManageIQ/catalog-api/blob/bf1a18797f18667f996d3c4e42218ff7aa81fd91/lib/open_api/serializer.rb#L18
This also gets us away from the black magic of Jbuilder and its conflicts with several other gems (acts_as_tenant)
If you are open to this please let us know we will close this PR and make a PR in manageiq-api-common with the original_url change.

If you are open to this please let us know we will close this PR and make a PR in manageiq-api-common with the original_url change.

I think we want to make the code in manageiq-api-common not depend on jbuilder and the way that it adds the prefixes.  But don't close this PR, once the changes are merged to manageiq-api-common, it would be great to share with catalog-api as well.

@mkanoor Does this just work now, or was something changed in manageiq-api-common?

@eclarizio The missing of the ActionView::Rendering was the reason why option[:prefixes] was missing. Jbuilder seems to bring in the ActionView::Rendering so the first attempt in this PR was to add the ActionView::Rendering.
Further research lead to a conflict between jbuilder rail tie and acts_as_tenant conflict. So now we don't include the ActionView::Rendering but delay the loading of the 'acts_as_tenant' gem which was done in your PR #382 which removed the acts_as_tenant gem from the Gemfile to use the one in manageiq-api-common and also have a require 'acts_as_tenant' in the application record.
This PR now just includes the changes to use the shared serializer. Since PR https://github.com/ManageIQ/catalog-api/pull/382 had the rest of the changes.

@bdunne it works because we delayed the loading of the acts_as_gem just like the sources api. so the jbuilder rail tie brings in the ActionView::Rendering and the options[:prefixes]. Not an ideal solution but for now it brings the Catalog Service in sync with the Sources and Topology Service and we can use the shared serializer. Ideally we would like to remove the dependency on the options[:prefixes]


Composer issue with laravel/prompts lock version

Description
When installing this to a relatively new and clean installation of Statamic, I ran into an issue with composer complaining about the laravel/prompts version already installed.
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - Root composer.json requires statamic-rad-pack/runway ^7.0 -> satisfiable by statamic-rad-pack/runway[v7.0.0, ..., 7.x-dev].
    - statamic-rad-pack/runway[v7.0.0, ..., 7.x-dev] require laravel/prompts ^0.1.17 -> found laravel/prompts[v0.1.17, ..., v0.1.25] but the package is fixed to v0.3.2 (lock file version) by a partial update and that version does not match. Make sure you list it as an argument for the update command.

Both laravel/framework (https://github.com/laravel/framework/blob/620f7176e6c452c3136829618f3ade2c3fa7798b/composer.json#L34) and statamic/cms (https://github.com/statamic/cms/blob/a600ab17d527cd7b67eede29f2209c8cf88af517/composer.json#L18) allow multiple versions of laravel/prompts to be used.
Unless runway requires this specific version of laravel/prompts. Can the composer.json be updated to be inline with both the Laravel Framework and Statamic CMS?
I have gotten around this by forcing the laravel/prompts version to use the one required for runway
composer require statamic-rad-pack/runway:^7.0 --with-all-dependencies
Steps to reproduce

composer create-project statamic/statamic test-project
cd test-project
composer require statamic-rad-pack/runway

Composer install error displayed.
Environment
Application Name: SL-Statamic-Starter-Base
Laravel Version: 11.36.1
PHP Version: 8.3.14
Composer Version: 2.8.4
Environment: local
Debug Mode: ENABLED
URL: statamic-starter-base.ddev.site
Maintenance Mode: OFF
Timezone: UTC
Locale: en
Cache
Config: NOT CACHED
Events: NOT CACHED
Routes: NOT CACHED
Views: NOT CACHED
Drivers
Broadcasting: log
Cache: file
Database: mysql
Logs: stack / single
Mail: smtp
Queue: sync
Session: file
Statamic
Addons: 2
Sites: 1
Stache Watcher: Enabled (auto)
Static Caching: Disabled
Version: 5.45.0 PRO
Statamic Addons
statamic-rad-pack/runway: 7.13.3
statamic/eloquent-driver: 4.19.1
Statamic Eloquent Driver
Asset Containers: file
Assets: eloquent
Blueprints: file
Collection Trees: eloquent
Collections: file
Entries: eloquent
Forms: eloquent
Global Sets: eloquent
Global Variables: eloquent
Navigation Trees: eloquent
Navigations: file
Revisions: file
Sites: file
Taxonomies: eloquent
Terms: eloquent
Tokens: file

We can probably remove the laravel/prompts version constraint from this package and rely on it coming from Statamic. I'm away for the next few weeks, so feel free to open a PR and I can merge it when I'm back.


[bazarr] Add bounded concurrency to episodes endpoint

Description of the change

This PR adds pagination to collection from the /api/episodes endpoint in the bazarr collector. This pagination uses bounded concurrency, sending no more than --series-batch-concurrency (default 10) batches to the bazarr server at a time, and asking for no more than --series-batch-size (default 50) seriesIds at a time.
fixes #237
A test image is available at ghcr.io/rtrox/exportarr:episodes-concurrency.
Benefits
Allows retrieval of series metrics from bazarr instances with a large number of serieses.
Possible drawbacks
More goroutines?
Applicable issues

fixes #237

Additional information

@rtrox ill run this alter tonight to get some benchmark numbers for comparision

@phyzical - that'd be great, I don't actually have a bazarr instance to test against >.< I'm reasonably confident from the unittests, but mind comparing the metric values too and just double checking that there aren't any logic bugs in the counting?

@phyzical - one more note -- in lidarr & sonarr we have a similar pattern, where a few specific metrics require us to send an API call per object. In both places, we hide these metrics behind the EnableAdditionalMetrics config value. I haven't made that change here yet, as it looks like I'd be gating the majority of the interesting metrics, but I'd like to think through what metrics are actually vital, and see if we can find ways to get them from the higher order APIs (or make some PRs to bazarr to do so).

i did have that if sitting there ready for heavy lifting, but as you've guessed it was kinda of all or nothing.
The other reason i didn't bother was that i thought i had a fairly large sample size (3k+ series) and i was able to get it all within 30 seconds, which was more than enough for my usecase (checking a dashboard once in a blue moon)
i personally think we should maybe be suggesting that these exporters are configured for longer intervals i.e once every 5-10 minutes anyway, that way if it is an expensive scrape it doesn't matter as much. the only outlier i can think of is alerts tied to disk space/health check, but even then id say finding out at 5-10 minutes for hobby stuff is still decent.
But ill give it a test still because  if its does speed things up why not!

Yea, that makes sense. The only thing to keep in mind is that grafana's prometheus integration's max_source_resolution is somewhat low (I think it's a minute, but don't quote me on that). So if we decided to give that advice, we'll need to document the process of adding max_source_resolution as a query parameter to the grafana prometheus integration, given most of our users are hobbyists.
That said, breaking this up was probably necessary anyway, in the linked issue the url alone was near 100K, which means the memory needed to deserialize the response was definitely going to be a bit crazy >.<.

yeah i saw the size of the log in the 20 mb range 😆
TBH ive been grafanaing for about 4 months so good to know the grafana limitation.
But i think another part of the issue here is also that we assume each request should perform a new scrape, most of the exporters ive used tend to work with a scrape interval internally and cache the result for the prometheus instance to just get served some cold hard data.
Not saying that should be done in this pr. But something that should be considered if it hasnt already.
(i assume itll be a "oh its on the list" and there just wasnt enough justification)

Believe it or not, caching is actually an anti-pattern in Prometheus exporters 🙃 From the exporter instrumentation docs:

Metrics should only be pulled from the application when Prometheus scrapes them, exporters should not perform scrapes based on their own timers. That is, all scrapes should be synchronous.

Accordingly, you should not set timestamps on the metrics you expose, let Prometheus take care of that. If you think you need timestamps, then you probably need the Pushgateway instead.

If a metric is particularly expensive to retrieve, i.e. takes more than a minute, it is acceptable to cache it. This should be noted in the HELP string.

The default scrape timeout for Prometheus is 10 seconds. If your exporter can be expected to exceed this, you should explicitly call this out in your user documentation.

We have a couple places that we do cache - in prowlarr, for example, as prowlarr exposes stats in an odd way which requires us to calculate and cache deltas to create accurate counts. Otherwise, we generally avoid caching and stick with synchronous calls.

yeah that makes sense. in most cases just adjusting interval and scrape times would suffice (has for me).
And its probably like anything prometheus keep the rules simple and locked down and it should just work out the box.
The caching is it a simple toggleable feature or is it fairly deep? it does feel like an edgecase at this point i would personally consider my library the large end but this other users has double what i do so 😆 (no saying it shouldnt be a feature but should depend on how complicated it becomes to support optional caching)
i did actually try asyncing the calls originally too but found it made things slower and i think i'm seeing the same during testing?
old:

# HELP bazarr_episode_subtitles_downloaded_total Total number of downloaded episode subtitles
# TYPE bazarr_episode_subtitles_downloaded_total gauge
bazarr_episode_subtitles_downloaded_total{url="http://bazarr:6767"} 64927
# HELP bazarr_episode_subtitles_filesize_total Total filesize of all episode subtitles
# TYPE bazarr_episode_subtitles_filesize_total gauge
bazarr_episode_subtitles_filesize_total{url="http://bazarr:6767"} 3.256260812e+09
# HELP bazarr_episode_subtitles_history_total Total number of history episode subtitles
# TYPE bazarr_episode_subtitles_history_total gauge
bazarr_episode_subtitles_history_total{url="http://bazarr:6767"} 17952
# HELP bazarr_episode_subtitles_missing_total Total number of missing episode subtitles
# TYPE bazarr_episode_subtitles_missing_total gauge
bazarr_episode_subtitles_missing_total{url="http://bazarr:6767"} 18619
# HELP bazarr_episode_subtitles_monitored_total Total number of monitored episode subtitles
# TYPE bazarr_episode_subtitles_monitored_total gauge
bazarr_episode_subtitles_monitored_total{url="http://bazarr:6767"} 26877
# HELP bazarr_episode_subtitles_unmonitored_total Total number of unmonitored episode subtitles
# TYPE bazarr_episode_subtitles_unmonitored_total gauge
bazarr_episode_subtitles_unmonitored_total{url="http://bazarr:6767"} 61021
# HELP bazarr_episode_subtitles_wanted_total Total number of wanted episode subtitles
# TYPE bazarr_episode_subtitles_wanted_total gauge
bazarr_episode_subtitles_wanted_total{url="http://bazarr:6767"} 18618
# HELP bazarr_movie_subtitles_downloaded_total Total number of downloaded movie subtitles
# TYPE bazarr_movie_subtitles_downloaded_total gauge
bazarr_movie_subtitles_downloaded_total{url="http://bazarr:6767"} 1973
# HELP bazarr_movie_subtitles_filesize_total Total filesize of all movie subtitles
# TYPE bazarr_movie_subtitles_filesize_total gauge
bazarr_movie_subtitles_filesize_total{url="http://bazarr:6767"} 2.42571861e+08
# HELP bazarr_movie_subtitles_history_total Total number of history movie subtitles
# TYPE bazarr_movie_subtitles_history_total gauge
bazarr_movie_subtitles_history_total{url="http://bazarr:6767"} 61
# HELP bazarr_movie_subtitles_missing_total Total number of missing movie subtitles
# TYPE bazarr_movie_subtitles_missing_total gauge
bazarr_movie_subtitles_missing_total{url="http://bazarr:6767"} 166
# HELP bazarr_movie_subtitles_monitored_total Total number of monitored movie subtitles
# TYPE bazarr_movie_subtitles_monitored_total gauge
bazarr_movie_subtitles_monitored_total{url="http://bazarr:6767"} 2126
# HELP bazarr_movie_subtitles_unmonitored_total Total number of unmonitored movie subtitles
# TYPE bazarr_movie_subtitles_unmonitored_total gauge
bazarr_movie_subtitles_unmonitored_total{url="http://bazarr:6767"} 5
# HELP bazarr_movie_subtitles_wanted_total Total number of wanted movie subtitles
# TYPE bazarr_movie_subtitles_wanted_total gauge
bazarr_movie_subtitles_wanted_total{url="http://bazarr:6767"} 166
# HELP bazarr_scrape_duration_seconds Duration of the last scrape of metrics from Exportarr.
# TYPE bazarr_scrape_duration_seconds gauge
bazarr_scrape_duration_seconds{url="http://bazarr:6767"} 28.936000262
# HELP bazarr_scrape_requests_total Total number of HTTP requests made.
# TYPE bazarr_scrape_requests_total counter
bazarr_scrape_requests_total{code="200",url="http://bazarr:6767"} 10
# HELP bazarr_subtitles_downloaded_total Total number of downloaded subtitles
# TYPE bazarr_subtitles_downloaded_total gauge
bazarr_subtitles_downloaded_total{url="http://bazarr:6767"} 66900
# HELP bazarr_subtitles_filesize_total Total filesize of all subtitles
# TYPE bazarr_subtitles_filesize_total gauge
bazarr_subtitles_filesize_total{url="http://bazarr:6767"} 3.498832673e+09
# HELP bazarr_subtitles_history_total Total number of history subtitles
# TYPE bazarr_subtitles_history_total gauge
bazarr_subtitles_history_total{url="http://bazarr:6767"} 18013
# HELP bazarr_subtitles_language_total Total number of downloaded subtitles by language
# TYPE bazarr_subtitles_language_total gauge
bazarr_subtitles_language_total{language="Arabic",url="http://bazarr:6767"} 51
bazarr_subtitles_language_total{language="Danish",url="http://bazarr:6767"} 41
bazarr_subtitles_language_total{language="Dutch",url="http://bazarr:6767"} 4
bazarr_subtitles_language_total{language="English",url="http://bazarr:6767"} 66628
bazarr_subtitles_language_total{language="Finnish",url="http://bazarr:6767"} 42
bazarr_subtitles_language_total{language="Hebrew",url="http://bazarr:6767"} 3
bazarr_subtitles_language_total{language="Italian",url="http://bazarr:6767"} 1
bazarr_subtitles_language_total{language="Norwegian",url="http://bazarr:6767"} 41
bazarr_subtitles_language_total{language="Polish",url="http://bazarr:6767"} 21
bazarr_subtitles_language_total{language="Portuguese",url="http://bazarr:6767"} 24
bazarr_subtitles_language_total{language="Romanian",url="http://bazarr:6767"} 1
bazarr_subtitles_language_total{language="Swedish",url="http://bazarr:6767"} 43
# HELP bazarr_subtitles_missing_total Total number of missing subtitles
# TYPE bazarr_subtitles_missing_total gauge
bazarr_subtitles_missing_total{url="http://bazarr:6767"} 18785
# HELP bazarr_subtitles_monitored_total Total number of monitored subtitles
# TYPE bazarr_subtitles_monitored_total gauge
bazarr_subtitles_monitored_total{url="http://bazarr:6767"} 29003
# HELP bazarr_subtitles_provider_total Total number of downloaded subtitles by provider
# TYPE bazarr_subtitles_provider_total gauge
bazarr_subtitles_provider_total{provider="betaseries",url="http://bazarr:6767"} 1486
bazarr_subtitles_provider_total{provider="bsplayer",url="http://bazarr:6767"} 52
bazarr_subtitles_provider_total{provider="manual",url="http://bazarr:6767"} 24
bazarr_subtitles_provider_total{provider="opensubtitles",url="http://bazarr:6767"} 6169
bazarr_subtitles_provider_total{provider="opensubtitlescom",url="http://bazarr:6767"} 763
bazarr_subtitles_provider_total{provider="podnapisi",url="http://bazarr:6767"} 3069
bazarr_subtitles_provider_total{provider="subscene",url="http://bazarr:6767"} 2
bazarr_subtitles_provider_total{provider="supersubtitles",url="http://bazarr:6767"} 1773
bazarr_subtitles_provider_total{provider="tusubtitulo",url="http://bazarr:6767"} 11
bazarr_subtitles_provider_total{provider="tvsubtitles",url="http://bazarr:6767"} 27
bazarr_subtitles_provider_total{provider="yifysubtitles",url="http://bazarr:6767"} 18
# HELP bazarr_subtitles_score_total Total number of downloaded subtitles by score
# TYPE bazarr_subtitles_score_total gauge
bazarr_subtitles_score_total{score="100.0%",url="http://bazarr:6767"} 1298
bazarr_subtitles_score_total{score="66.67%",url="http://bazarr:6767"} 9
bazarr_subtitles_score_total{score="66.94%",url="http://bazarr:6767"} 570
bazarr_subtitles_score_total{score="67.5%",url="http://bazarr:6767"} 58
bazarr_subtitles_score_total{score="67.78%",url="http://bazarr:6767"} 5
bazarr_subtitles_score_total{score="68.06%",url="http://bazarr:6767"} 1
bazarr_subtitles_score_total{score="68.33%",url="http://bazarr:6767"} 1
bazarr_subtitles_score_total{score="68.89%",url="http://bazarr:6767"} 29
bazarr_subtitles_score_total{score="69.44%",url="http://bazarr:6767"} 8
bazarr_subtitles_score_total{score="69.72%",url="http://bazarr:6767"} 5
bazarr_subtitles_score_total{score="70.28%",url="http://bazarr:6767"} 5
bazarr_subtitles_score_total{score="70.83%",url="http://bazarr:6767"} 1
bazarr_subtitles_score_total{score="71.11%",url="http://bazarr:6767"} 2
bazarr_subtitles_score_total{score="75.0%",url="http://bazarr:6767"} 4
bazarr_subtitles_score_total{score="75.83%",url="http://bazarr:6767"} 4
bazarr_subtitles_score_total{score="77.5%",url="http://bazarr:6767"} 4
bazarr_subtitles_score_total{score="78.33%",url="http://bazarr:6767"} 1
bazarr_subtitles_score_total{score="79.17%",url="http://bazarr:6767"} 5
bazarr_subtitles_score_total{score="80.0%",url="http://bazarr:6767"} 2
bazarr_subtitles_score_total{score="80.83%",url="http://bazarr:6767"} 1
bazarr_subtitles_score_total{score="81.67%",url="http://bazarr:6767"} 9
bazarr_subtitles_score_total{score="82.5%",url="http://bazarr:6767"} 2
bazarr_subtitles_score_total{score="83.33%",url="http://bazarr:6767"} 6
bazarr_subtitles_score_total{score="84.17%",url="http://bazarr:6767"} 1
bazarr_subtitles_score_total{score="85.0%",url="http://bazarr:6767"} 7
bazarr_subtitles_score_total{score="85.83%",url="http://bazarr:6767"} 7
bazarr_subtitles_score_total{score="87.5%",url="http://bazarr:6767"} 4
bazarr_subtitles_score_total{score="91.67%",url="http://bazarr:6767"} 115
bazarr_subtitles_score_total{score="91.94%",url="http://bazarr:6767"} 4723
bazarr_subtitles_score_total{score="92.22%",url="http://bazarr:6767"} 17
bazarr_subtitles_score_total{score="92.5%",url="http://bazarr:6767"} 991
bazarr_subtitles_score_total{score="92.78%",url="http://bazarr:6767"} 177
bazarr_subtitles_score_total{score="93.06%",url="http://bazarr:6767"} 84
bazarr_subtitles_score_total{score="93.33%",url="http://bazarr:6767"} 217
bazarr_subtitles_score_total{score="93.61%",url="http://bazarr:6767"} 27
bazarr_subtitles_score_total{score="93.89%",url="http://bazarr:6767"} 1382
bazarr_subtitles_score_total{score="94.17%",url="http://bazarr:6767"} 48
bazarr_subtitles_score_total{score="94.44%",url="http://bazarr:6767"} 1289
bazarr_subtitles_score_total{score="94.72%",url="http://bazarr:6767"} 77
bazarr_subtitles_score_total{score="95.0%",url="http://bazarr:6767"} 401
bazarr_subtitles_score_total{score="95.28%",url="http://bazarr:6767"} 132
bazarr_subtitles_score_total{score="95.83%",url="http://bazarr:6767"} 185
bazarr_subtitles_score_total{score="96.11%",url="http://bazarr:6767"} 24
bazarr_subtitles_score_total{score="96.67%",url="http://bazarr:6767"} 20
bazarr_subtitles_score_total{score="97.5%",url="http://bazarr:6767"} 2
bazarr_subtitles_score_total{score="98.06%",url="http://bazarr:6767"} 77
bazarr_subtitles_score_total{score="98.61%",url="http://bazarr:6767"} 889
bazarr_subtitles_score_total{score="98.89%",url="http://bazarr:6767"} 4
bazarr_subtitles_score_total{score="99.17%",url="http://bazarr:6767"} 360
bazarr_subtitles_score_total{score="99.44%",url="http://bazarr:6767"} 47
bazarr_subtitles_score_total{score="99.72%",url="http://bazarr:6767"} 57
# HELP bazarr_subtitles_unmonitored_total Total number of unmonitored subtitles
# TYPE bazarr_subtitles_unmonitored_total gauge
bazarr_subtitles_unmonitored_total{url="http://bazarr:6767"} 61026
# HELP bazarr_subtitles_wanted_total Total number of wanted subtitles
# TYPE bazarr_subtitles_wanted_total gauge
bazarr_subtitles_wanted_total{url="http://bazarr:6767"} 18784
# HELP bazarr_system_health_issues Total number of health issues by object and issue
# TYPE bazarr_system_health_issues gauge
bazarr_system_health_issues{issue="",object="",url="http://bazarr:6767"} 0
# HELP bazarr_system_status System Status
# TYPE bazarr_system_status gauge
bazarr_system_status{url="http://bazarr:6767"} 1
# HELP exportarr_app_info A metric with a constant '1' value labeled by app name, version, build time, and revision.
# TYPE exportarr_app_info gauge
exportarr_app_info{app_name="exportarr",build_time="2023-10-18T12:55:57.008Z",revision="294191ecae08120e407f1a8b2c5dda12ffb7b007",version="v1.6.0"} 1

new:

# HELP bazarr_episode_subtitles_downloaded_total Total number of downloaded episode subtitles
# TYPE bazarr_episode_subtitles_downloaded_total gauge
bazarr_episode_subtitles_downloaded_total{url="http://bazarr:6767"} 64927
# HELP bazarr_episode_subtitles_filesize_total Total filesize of all episode subtitles
# TYPE bazarr_episode_subtitles_filesize_total gauge
bazarr_episode_subtitles_filesize_total{url="http://bazarr:6767"} 3.256260812e+09
# HELP bazarr_episode_subtitles_history_total Total number of history episode subtitles
# TYPE bazarr_episode_subtitles_history_total gauge
bazarr_episode_subtitles_history_total{url="http://bazarr:6767"} 17952
# HELP bazarr_episode_subtitles_missing_total Total number of missing episode subtitles
# TYPE bazarr_episode_subtitles_missing_total gauge
bazarr_episode_subtitles_missing_total{url="http://bazarr:6767"} 18619
# HELP bazarr_episode_subtitles_monitored_total Total number of monitored episode subtitles
# TYPE bazarr_episode_subtitles_monitored_total gauge
bazarr_episode_subtitles_monitored_total{url="http://bazarr:6767"} 26877
# HELP bazarr_episode_subtitles_unmonitored_total Total number of unmonitored episode subtitles
# TYPE bazarr_episode_subtitles_unmonitored_total gauge
bazarr_episode_subtitles_unmonitored_total{url="http://bazarr:6767"} 61021
# HELP bazarr_episode_subtitles_wanted_total Total number of wanted episode subtitles
# TYPE bazarr_episode_subtitles_wanted_total gauge
bazarr_episode_subtitles_wanted_total{url="http://bazarr:6767"} 18618
# HELP bazarr_movie_subtitles_downloaded_total Total number of downloaded movie subtitles
# TYPE bazarr_movie_subtitles_downloaded_total gauge
bazarr_movie_subtitles_downloaded_total{url="http://bazarr:6767"} 1973
# HELP bazarr_movie_subtitles_filesize_total Total filesize of all movie subtitles
# TYPE bazarr_movie_subtitles_filesize_total gauge
bazarr_movie_subtitles_filesize_total{url="http://bazarr:6767"} 2.42571861e+08
# HELP bazarr_movie_subtitles_history_total Total number of history movie subtitles
# TYPE bazarr_movie_subtitles_history_total gauge
bazarr_movie_subtitles_history_total{url="http://bazarr:6767"} 61
# HELP bazarr_movie_subtitles_missing_total Total number of missing movie subtitles
# TYPE bazarr_movie_subtitles_missing_total gauge
bazarr_movie_subtitles_missing_total{url="http://bazarr:6767"} 166
# HELP bazarr_movie_subtitles_monitored_total Total number of monitored movie subtitles
# TYPE bazarr_movie_subtitles_monitored_total gauge
bazarr_movie_subtitles_monitored_total{url="http://bazarr:6767"} 2126
# HELP bazarr_movie_subtitles_unmonitored_total Total number of unmonitored movie subtitles
# TYPE bazarr_movie_subtitles_unmonitored_total gauge
bazarr_movie_subtitles_unmonitored_total{url="http://bazarr:6767"} 5
# HELP bazarr_movie_subtitles_wanted_total Total number of wanted movie subtitles
# TYPE bazarr_movie_subtitles_wanted_total gauge
bazarr_movie_subtitles_wanted_total{url="http://bazarr:6767"} 166
# HELP bazarr_scrape_duration_seconds Duration of the last scrape of metrics from Exportarr.
# TYPE bazarr_scrape_duration_seconds gauge
bazarr_scrape_duration_seconds{url="http://bazarr:6767"} 25.886018311
# HELP bazarr_scrape_requests_total Total number of HTTP requests made.
# TYPE bazarr_scrape_requests_total counter
bazarr_scrape_requests_total{code="200",url="http://bazarr:6767"} 6
# HELP bazarr_subtitles_downloaded_total Total number of downloaded subtitles
# TYPE bazarr_subtitles_downloaded_total gauge
bazarr_subtitles_downloaded_total{url="http://bazarr:6767"} 66900
# HELP bazarr_subtitles_filesize_total Total filesize of all subtitles
# TYPE bazarr_subtitles_filesize_total gauge
bazarr_subtitles_filesize_total{url="http://bazarr:6767"} 3.498832673e+09
# HELP bazarr_subtitles_history_total Total number of history subtitles
# TYPE bazarr_subtitles_history_total gauge
bazarr_subtitles_history_total{url="http://bazarr:6767"} 18013
# HELP bazarr_subtitles_language_total Total number of downloaded subtitles by language
# TYPE bazarr_subtitles_language_total gauge
bazarr_subtitles_language_total{language="Arabic",url="http://bazarr:6767"} 51
bazarr_subtitles_language_total{language="Danish",url="http://bazarr:6767"} 41
bazarr_subtitles_language_total{language="Dutch",url="http://bazarr:6767"} 4
bazarr_subtitles_language_total{language="English",url="http://bazarr:6767"} 66628
bazarr_subtitles_language_total{language="Finnish",url="http://bazarr:6767"} 42
bazarr_subtitles_language_total{language="Hebrew",url="http://bazarr:6767"} 3
bazarr_subtitles_language_total{language="Italian",url="http://bazarr:6767"} 1
bazarr_subtitles_language_total{language="Norwegian",url="http://bazarr:6767"} 41
bazarr_subtitles_language_total{language="Polish",url="http://bazarr:6767"} 21
bazarr_subtitles_language_total{language="Portuguese",url="http://bazarr:6767"} 24
bazarr_subtitles_language_total{language="Romanian",url="http://bazarr:6767"} 1
bazarr_subtitles_language_total{language="Swedish",url="http://bazarr:6767"} 43
# HELP bazarr_subtitles_missing_total Total number of missing subtitles
# TYPE bazarr_subtitles_missing_total gauge
bazarr_subtitles_missing_total{url="http://bazarr:6767"} 18785
# HELP bazarr_subtitles_monitored_total Total number of monitored subtitles
# TYPE bazarr_subtitles_monitored_total gauge
bazarr_subtitles_monitored_total{url="http://bazarr:6767"} 29003
# HELP bazarr_subtitles_provider_total Total number of downloaded subtitles by provider
# TYPE bazarr_subtitles_provider_total gauge
bazarr_subtitles_provider_total{provider="betaseries",url="http://bazarr:6767"} 1486
bazarr_subtitles_provider_total{provider="bsplayer",url="http://bazarr:6767"} 52
bazarr_subtitles_provider_total{provider="manual",url="http://bazarr:6767"} 24
bazarr_subtitles_provider_total{provider="opensubtitles",url="http://bazarr:6767"} 6169
bazarr_subtitles_provider_total{provider="opensubtitlescom",url="http://bazarr:6767"} 763
bazarr_subtitles_provider_total{provider="podnapisi",url="http://bazarr:6767"} 3069
bazarr_subtitles_provider_total{provider="subscene",url="http://bazarr:6767"} 2
bazarr_subtitles_provider_total{provider="supersubtitles",url="http://bazarr:6767"} 1773
bazarr_subtitles_provider_total{provider="tusubtitulo",url="http://bazarr:6767"} 11
bazarr_subtitles_provider_total{provider="tvsubtitles",url="http://bazarr:6767"} 27
bazarr_subtitles_provider_total{provider="yifysubtitles",url="http://bazarr:6767"} 18
# HELP bazarr_subtitles_score_total Total number of downloaded subtitles by score
# TYPE bazarr_subtitles_score_total gauge
bazarr_subtitles_score_total{score="100.0%",url="http://bazarr:6767"} 1298
bazarr_subtitles_score_total{score="66.67%",url="http://bazarr:6767"} 9
bazarr_subtitles_score_total{score="66.94%",url="http://bazarr:6767"} 570
bazarr_subtitles_score_total{score="67.5%",url="http://bazarr:6767"} 58
bazarr_subtitles_score_total{score="67.78%",url="http://bazarr:6767"} 5
bazarr_subtitles_score_total{score="68.06%",url="http://bazarr:6767"} 1
bazarr_subtitles_score_total{score="68.33%",url="http://bazarr:6767"} 1
bazarr_subtitles_score_total{score="68.89%",url="http://bazarr:6767"} 29
bazarr_subtitles_score_total{score="69.44%",url="http://bazarr:6767"} 8
bazarr_subtitles_score_total{score="69.72%",url="http://bazarr:6767"} 5
bazarr_subtitles_score_total{score="70.28%",url="http://bazarr:6767"} 5
bazarr_subtitles_score_total{score="70.83%",url="http://bazarr:6767"} 1
bazarr_subtitles_score_total{score="71.11%",url="http://bazarr:6767"} 2
bazarr_subtitles_score_total{score="75.0%",url="http://bazarr:6767"} 4
bazarr_subtitles_score_total{score="75.83%",url="http://bazarr:6767"} 4
bazarr_subtitles_score_total{score="77.5%",url="http://bazarr:6767"} 4
bazarr_subtitles_score_total{score="78.33%",url="http://bazarr:6767"} 1
bazarr_subtitles_score_total{score="79.17%",url="http://bazarr:6767"} 5
bazarr_subtitles_score_total{score="80.0%",url="http://bazarr:6767"} 2
bazarr_subtitles_score_total{score="80.83%",url="http://bazarr:6767"} 1
bazarr_subtitles_score_total{score="81.67%",url="http://bazarr:6767"} 9
bazarr_subtitles_score_total{score="82.5%",url="http://bazarr:6767"} 2
bazarr_subtitles_score_total{score="83.33%",url="http://bazarr:6767"} 6
bazarr_subtitles_score_total{score="84.17%",url="http://bazarr:6767"} 1
bazarr_subtitles_score_total{score="85.0%",url="http://bazarr:6767"} 7
bazarr_subtitles_score_total{score="85.83%",url="http://bazarr:6767"} 7
bazarr_subtitles_score_total{score="87.5%",url="http://bazarr:6767"} 4
bazarr_subtitles_score_total{score="91.67%",url="http://bazarr:6767"} 115
bazarr_subtitles_score_total{score="91.94%",url="http://bazarr:6767"} 4723
bazarr_subtitles_score_total{score="92.22%",url="http://bazarr:6767"} 17
bazarr_subtitles_score_total{score="92.5%",url="http://bazarr:6767"} 991
bazarr_subtitles_score_total{score="92.78%",url="http://bazarr:6767"} 177
bazarr_subtitles_score_total{score="93.06%",url="http://bazarr:6767"} 84
bazarr_subtitles_score_total{score="93.33%",url="http://bazarr:6767"} 217
bazarr_subtitles_score_total{score="93.61%",url="http://bazarr:6767"} 27
bazarr_subtitles_score_total{score="93.89%",url="http://bazarr:6767"} 1382
bazarr_subtitles_score_total{score="94.17%",url="http://bazarr:6767"} 48
bazarr_subtitles_score_total{score="94.44%",url="http://bazarr:6767"} 1289
bazarr_subtitles_score_total{score="94.72%",url="http://bazarr:6767"} 77
bazarr_subtitles_score_total{score="95.0%",url="http://bazarr:6767"} 401
bazarr_subtitles_score_total{score="95.28%",url="http://bazarr:6767"} 132
bazarr_subtitles_score_total{score="95.83%",url="http://bazarr:6767"} 185
bazarr_subtitles_score_total{score="96.11%",url="http://bazarr:6767"} 24
bazarr_subtitles_score_total{score="96.67%",url="http://bazarr:6767"} 20
bazarr_subtitles_score_total{score="97.5%",url="http://bazarr:6767"} 2
bazarr_subtitles_score_total{score="98.06%",url="http://bazarr:6767"} 77
bazarr_subtitles_score_total{score="98.61%",url="http://bazarr:6767"} 889
bazarr_subtitles_score_total{score="98.89%",url="http://bazarr:6767"} 4
bazarr_subtitles_score_total{score="99.17%",url="http://bazarr:6767"} 360
bazarr_subtitles_score_total{score="99.44%",url="http://bazarr:6767"} 47
bazarr_subtitles_score_total{score="99.72%",url="http://bazarr:6767"} 57
# HELP bazarr_subtitles_unmonitored_total Total number of unmonitored subtitles
# TYPE bazarr_subtitles_unmonitored_total gauge
bazarr_subtitles_unmonitored_total{url="http://bazarr:6767"} 61026
# HELP bazarr_subtitles_wanted_total Total number of wanted subtitles
# TYPE bazarr_subtitles_wanted_total gauge
bazarr_subtitles_wanted_total{url="http://bazarr:6767"} 18784
# HELP bazarr_system_health_issues Total number of health issues by object and issue
# TYPE bazarr_system_health_issues gauge
bazarr_system_health_issues{issue="",object="",url="http://bazarr:6767"} 0
# HELP bazarr_system_status System Status
# TYPE bazarr_system_status gauge
bazarr_system_status{url="http://bazarr:6767"} 1
# HELP exportarr_app_info A metric with a constant '1' value labeled by app name, version, build time, and revision.
# TYPE exportarr_app_info gauge
exportarr_app_info{app_name="exportarr",build_time="",revision="",version="development"} 1

so yeah its actually slower, which is something i noticed when i tried batching it originally, granted i didn't try hard to workout why

The caching is it a simple toggleable feature or is it fairly deep? it does feel like an edgecase at this point i would personally consider my library the large end but this other users has double what i do so 😆 (no saying it shouldnt be a feature but should depend on how complicated it becomes to support optional caching)

It's tied into the prowlarr logic, but it's also not terribly complicated. You can see the indexerStatCache here. That said, I only did caching there due to the shape of the data returned by prowlarr (their API accepts a time range, and gives aggregated stats by day. So we have to cache last seen values to calculate differentials).

so yeah its actually slower, which is something i noticed when i tried batching it originally, granted i didn't try hard to workout why

It's probably just the sheer number of calls. I set the default concurrency limit to 10 inflight requests. You can test bumping that up to 20 or 30 by setting --series-batch-concurrency. You can also bump up the batch size with --series-batch-size (default 50). If bumping either of those up makes a difference I'll just change the defaults. I should also add a collector stat for the number of calls being sent as well.
If it's still not faster, I think I'll still merge it, it looks like it's about a ~10% regression, which is likely worth it to reduce the size of the API requests. But if we can't actually get faster, I'll likely put the iterative call behind EnableAdditionalMetrics so folks can decide whether they want to see less metrics, or have a higher poll rate.
(@onedr0p feel free to veto me on that)

But if we can't actually get faster, I'll likely put the iterative call behind EnableAdditionalMetrics so folks can decide whether they want to see less metrics, or have a higher poll rate.

Sounds good to me :)

@rtrox ill give a few more tests around --series-batch-concurrency tonight after work 👍
But i think leveraging "EnableAdditionalMetrics" for sync v async i think is fair, though may be a bit misleading
But i think if it does actually allow users to use the implementation i think its a good idea.

@phyzical

But i think leveraging "EnableAdditionalMetrics" for sync v async i think is fair, though may be a bit misleading
That's not quite what I mean -- either way I'll want to switch to the threaded requests here, I'm not super comfortable leaving the episode history calls in place as is, the URLs are too large for me to be comfortable with on the wire, and the memory required for the responses is concerning. So I'm actually ok with a wall clock regression to move to the threaded model where request size is bounded.

What I mean is -- if we can't find defaults that let us reduce the wall clock time, I plan to put that call and the metrics it generates behind EnableAdditionalMetrics, I've been looking through the API code in bazarr, it looks like we can get a pretty reasonable number of metrics directly from the /series endpoint (episode files downloaded, episode files missing, series monitored.
So let's see if we can find some defaults that boost wall clock time, otherwise I'll start some refactoring to pull more metrics from the series api, and put the episode stats calls behind a flag.

Ah yep, sorry i get you now. i think your safe to just chuck both the history blocks behind the flag
I don't think you can get the data found in history in the main api call  (which is why i started using it) but i could be wrong.

ah sorry, life then work got in the way 😆
batch concurrency = 10 is what was being used above

~40mb
~24s
batch concurrency = 20

~42mb
~24-25s
batch concurrency = 30

~45-50mb
~24-25s
batch concurrency = 40

~50mb
~33s
batch concurrency = 50

~50mb
~33-34
its odd that things get slower the more concurrency i wonder if its more a limitation bazarr side? as in how well the python code can deal with concurrency vs a giant payload.
ill do a few more tests playing with the batch size now for some more numbers

batch-size 50 is the first one above
batch-size 100

~50 mb
~28s
batch-size 150

~50mb
~27s
batch-size 200

~28s
~50mb
batch-size 300

~21s
~70-100mb
batch-size 500

~21s
~100mb

So almost all the attempts floated around the 50mb range until i increased the batch size which makes sense.
But i think the "recommendation" should be increase batch to reduce ram usage at the cost of longer requests, increase batch size for faster responses at the cost of more ram usage.
personally i think ill just configure it to, have a batch size of 300-500 and use the default concurrency.

Thanks @phyzical, great stuff! Yea, I'd suspect the same. I'd guess either Bazarr, SQLite, or query performance as the issue, but this is probably good enough, 20-30 seconds at your library size seems reasonably solid.
So to land this, I think two updates:

I'll update the defaults to 300 batch size.
I'll put the history calls behind EnableAdditionalMetrics
[stretch] I may add some series level metrics, to ensure we can get some missing/monitored data for sonarr even without EnableAdditionalMetrics.

I should be be able to get to that over the long weekend so we can land this early next week.

sounds good to me 👍

Sorry, took a bit to get back to this PR, life got hectic >.< I think this should be ready for final review+merge.

was able to do a proper benchmark comparison diff wise for information it is exactly the same with the EnableAdditionalMetrics flag
👍

@onedr0p when you have time, i think this is good to be merged


missing values in netCDF data are ignored

values such as -1e30 used to flag missing data using the missing_value attribute are ignored by horizontal plots.

This seems not solved for side view images. Flightpath EDMO-Kiruna, Datafile for this case is in
/usr/nfs/software/icg1/exchange/j.-u.grooss/mss

Mmmh. The routine used for interpolating onto cross-sections does not support masks.
But maybe it properly works with NaN. I'll try something.

Is it better now and not misbehaving in different forms?

yes

Then I'll prepare a patch to develop.