tornado: Rewrite Django integration to duplicate less code.

Since essentially the first use of Tornado in Zulip, we've been
maintaining our Tornado+Django system, AsyncDjangoHandler, with
several hundred lines of Django code copied into it.
The goal for that code was simple: We wanted a way to use our Django
middleware (for code sharing reasons) inside a Tornado process (since
we wanted to use Tornado for our async events system).
As part of the Django 2.2.x upgrade, I looked at upgrading this
implementation to be based off modern Django, and it's definitely
possible to do that:

Continue forking load_middleware to save response middleware.
Continue manually running the Django response middleware.
Continue working out a hack involving copying all of _get_response
to change a couple lines allowing us our Tornado code to not
actually return the Django HttpResponse so we can long-poll.  The
previous hack of returning None stopped being viable with the Django 2.2
MiddlewareMixin.call implementation.

But I decided to take this opportunity to look at trying to avoid
copying material Django code, and there is a way to do it:

Replace RespondAsynchronously with a response.asynchronous attribute
on the HttpResponse; this causes Django to run its normal plumbing
happily in a way that should be stable over time, and then we
proceed to discard the response inside the Tornado get() method to
implement long-polling.  (Better yet might be raising an exception?).
This lets to eliminate patching of _get_response.

Calling the normal Django get_response method from zulip_finish,
rather than writing totally custom code to do that.  This resulted
in some mess with Monkey-patching MiddlewareMixin inside Tornado to
offer a way to skip running request middleware (to avoid running it
twice).  This lets us replace copying and monkey-patching
load_middleware to store response middleware.  I'm not yet sure
whether this is overall an improvement, but it does reduce the
Django code we're copying and patching to just a single 5-line function.

Adding detailed comments explaining how this is supposed to work,
what problems we encounter, and how we solve various problems, which
is critical to being able to modify this code in the future.

A key advantage of these changes is that the exact same code should
work on Django 1.11, Django 2.2, and Django 3.x, because we're no
longer copying large blocks of core Django code and thus should be
much less vulnerable to refactors.  There may be a modest performance
downside, in that we now run response middleware twice when
longpolling (once for the request we discard).  We could probably
avoid that with a more aggressive monkey-patching of MiddlewareMixin
to skip response middleware if response.asynchronous.
There's a bunch of further cleanups we may want to do with the new,
simplified system.
Extensively tested manually using print statements and asserts to confirm expected
behavior; test_tornado was also useful.

One property of our entire approach to asynchronous responses (with or without this PR) is that it can’t work correctly with Django “new”-style middleware (≥ 1.10) that doesn’t use the deprecated MiddlewareMixin. We maybe don’t care because we can reason about all the interactions between our limited use of asynchronicity and our limited use of middleware. But if that’s the case, maybe instead of monkey-patching MiddlewareMixin, we should rewrite our own middleware to inherit from our own new AsyncMiddlewareMixin? Then as far as the rest of Django and other third-party middleware is concerned, there are just two normal synchronous requests/responses instead of one weird asynchronous request/response.

That won't work, because even in Django 3.0, all of Django's built-in middleware also uses MiddlewareMixin (and the old-style thing was "deprecated" in Django 1.10).  I believe it's basically an error to call that mixin deprecated, in that there is no replacement API for having a process_response type function called.  I don't understand what's going on here fully, but I think it's basically a half-finished migration on Django's part; they got as far as the MIDDLEWARE/MIDDLEWARE_CLASSES point but didn't do the rest of what they had planned.

In what sense will that not work?
The story as I understand it is that DEP 0005 changed the basic type of middleware from
{process_request: (request: HttpRequest) -> HttpRequest, process_response: (response: HttpResponse) -> HttpResponse)
to the higher-order function
(get_response: (request: HttpRequest) -> HttpResponse) -> (request: HttpRequest) -> HttpResponse
which inherently guarantees that each request is paired with a response, and allows you to pass information from the request side to the response side using local variables instead of stuffing all your state into the HttpRequest object. MiddlewareMixin is just an adapter that converts the old style to the new style, and the choice to use it is now an implementation detail.
It isn’t possible to isolate the “process_response part” of new-style middleware generically. So now we’re breaking the abstraction barrier and assuming that middleware is implemented using the MiddlewareMixin adapter. My question is, why do we actually need to do that?

Perhaps this comment in django-upgrades.md should also be refreshed?

AsyncDjangoHandlerBase in zerver/tornado/handlers.py is a very
small patch on base.BaseHandler; see the comments with BY ZULIP for our changes (unfortunately now more different due to
style differences and mypy annotations).  This configurability
should also be contributed to Django upstream.

In what sense will that not work?

Effectively, yes, both for Django built-in middleware, and any third-party things like from the two_factor extension.  The "run the request middleware only once" work is only important to avoid running middleware that produces buggy behavior when run twice on a request, which is a problem some of our own middleware has (in particular, probably just LogRequests, which would clear state on the original request that we were hoping to preserve).  I suspect from some investigation and the structure of how request middleware generally works that Django's built-in middleware doesn't have that issue.  But it's definitely an abstraction barrier isolation.
If we want to avoid that sort of problem, we fundamentally need to break out of the Django _get_response without running middleware so that Django doesn't run the process_response piece during our original execution flow.  One idea for how to do that would be to raise an exception that inherits from BaseException but not Exception, which _get_response won't catch and turn into an HttpResponse and thus could be caught by self.get().
That might be a more robust hack?  I think it wouldn't require any monkey-patching.

Okay, let’s break even fewer abstraction barriers then: what if we make two entirely separate HttpRequest objects?

Tornado gets a request treq for /api/v1/events. It constructs a Django request dreq0 from treq, and sends it off to Django.
Django processes dreq0 as normal, including all middleware. It kicks off a Tornado task, and returns dres0 with an asynchronous flag.
Tornado sees the asynchronous flag and discards dreq0. (An exception would also work for this purpose, sure.)
Later, the Tornado task completes. Tornado constructs an entirely separate Django request dreq1 from treq, including some kind of asynchronous_result member indicating that the data is ready now, and sends it off to Django.
Django processes dreq1 as normal, including all middleware. It returns a response dres1 without an asynchronous flag.
Tornado writes dreq1 to the client.

One thing that goes wrong in this world is that the request gets logged twice. We fix that by making our logging middleware aware of asynchronous and/or asynchronous_result (perhaps using an AsyncMiddlewareMixin like I described). As far as other middleware is concerned, there are just two requests—is there anything wrong with that?

Well, the problem with that approach is how to construct the forged request.  If we construct it artificially, then it won't have the settings for the users' original request, so we'll send a response that e.g. ignores the locale settings the user specified and just returns things in English because that's what the forged request did.  But I guess the thought is to repeat the process we used to convert the Tornado request at self.request into a Django request, here?
    def get(self, *args: Any, **kwargs: Any) -> None:
        environ = WSGIContainer.environ(self.request)
        environ['PATH_INFO'] = urllib.parse.unquote(environ['PATH_INFO'])
        request = WSGIRequest(environ)
        request._tornado_handler = self

        set_script_prefix(get_script_name(environ))
        signals.request_started.send(sender=self.__class__)
        try:
            response = self.get_response(request)

            if response.asynchronous:
                # change: Before returning, we make a new request.
                return
        finally:
            signals.request_finished.send(sender=self.__class__)

That'd probably mostly work, at the cost of definitely doing the request and response middleware twice.  And we'd also need to do some hackery to take the fields that our code stores on the request object that we want to save and copy them over to the new Django HttpRequest object (probably mostly just request._log_data, which has timing data we want to use for the final request).
We can get rid of the response middleware twice issue by using the BaseException hack I described in my last message (which might still be worth doing), but the request middleware piece is probably fundamental to the concept of what we're doing here.

Yeah, that’s the thought. (I claim the second request is no more “forged” than the first one is.)

And we'd also need to do some hackery to take the fields that our code stores on the request object that we want to save and copy them over to the new Django HttpRequest object (probably mostly just request._log_data, which has timing data we want to use for the final request).

We could add a pointer from the second request to the first request that async-aware middleware knows how to follow (and normal middleware ignores).

I'm merging the prep commits through the new 986706c7e59a0002f27ff47ddc8b9b71e9d1a5b2, since they will help regardless of what solution we go with, and are required should we go with your proposal.  I'll see if I can finish prototyping that this afternoon to see if it works in practice.

I pushed back to this PR with a new commit that reworks things to use the approach @andersk suggested today.  It still needs manual testing, a few loose ends investigated, and probably some careful thought about, but if you look at the total diff for this PR, you should be able to see the new implementation pretty clearly.
It certainly feels like it lets us write much less in the way of monkey-patching of weird block comments; the main fear I'd have about it is that it creates performance issues by doing process_response work twice.  I think that's mostly resolvable with either the BaseException hack I mentioned above or just making our own middleware do nothing for responses doomed to not be returned.  Might not be a problem at all, but GET /events is our most important endpoint for scalability, so it certainly could be.

Yeah, I like how this is turning out, assuming it works. It sharpens the abstraction barrier between our code and Django internals. We’ve both alluded to ways we could further sharpen the abstraction barrier between parts of our own code, which I definitely support, but that’s less of a concern for following Django upgrades.

I just read the last commit for typos and things that seemed obviously confusing and didn't find anything.  It seems clearly more simple to me, and @andersk has weighed in, so maybe go ahead and squash before one final round of reviews?  For the record, I don't know enough about either Tornado or Django middleware to really find anything deep here, but it feels like I could read this code now in a pinch if I were debugging something.

OK I've pushed back here with a version that I believe to be mergable.  I haven't done the last round of manual testing for it, but folks should feel free to review as we're likely to merge approximately this.

The main names for things that we'll want to think about are these two:

response.asynchronous
request.saved_response

I think I'd prefer to do something more verbose and explicit like request.longpolling_completion_response and response.to_discard_for_longpolling.  But I'm not sure what good names are there.

I fixed a few small bugs and merged this.  I think we may want to iterate on variable naming a bit more, but fundamentally doing that will be an easy 6-line refactor, and I'd like to unblock the Django upgrade.
Thanks everyone for the feedback!  I'm pretty happy with how this ended up, in that it's a lot cleaner and also in a lot better position to further improve.


Pushgateway with basic auth

Is there a way to push metrics to a Prometheus Pushgateway that has basic auth enabled?
(Enabling basic auth in Pushgateway protects all HTTP endpoints including POST metrics)

Example: https://github.com/prometheus/client_java/blob/main/integration_tests/it_pushgateway/src/main/java/io/prometheus/client/it/pushgateway/ExampleBatchJob.java#L32C13-L32C91


SWLib Docs should show some usage plugins as well

Library Affected:
sw-lib
Browser & Platform:
All
Issue or Feature Request Description:
Doc enhancement
SWLib package brings a higher level framework kind of encapsulation for components from multiple packages(e.g. sw-runtime-caching). Although bringing all of the encapsulated package's documentation won't ever make sense, but I guess one feature that should be highlighted in the docs of SWLib is the way you add external plugins in handlers encapsulated in SWLib.
https://googlechrome.github.io/sw-helpers/reference-docs/stable/v0.0.7/module-sw-lib.html
^ URL does not currently highlight this thing. Doing so will ease the Dev's journey to plug and play responsive-image and background-sync queue plugins.

Would this just require surfacing the example from "RuntimeStrategyOptions" from this page more prominently?

yeah,
Just making sure that devs would know very clearly that this lib is extensible with plugins as well.

Calling out that we have an external plugin story seems like an important high-level topic for the documentation. @prateekbh Is there anything more beyond surfacing the RSO example that you would suggest adding here to help make it clear we work well with plugins?

I guess I am missing something,
https://github.com/prateekbh/Guess/blob/master/sw.js#L33
Here I am trying to add background-sync to my sw with sw-lib but it seems RequestWrapper is not exported from sw-lib.
Is there a way I add custom plugins into handlers of sw-lib ?

Pings on hangouts but will put here. See RuntimeStrategyOptions: https://googlechrome.github.io/sw-helpers/reference-docs/stable/v0.0.7/module-sw-lib.SWLib.html#.RuntimeStrategyOptions


Working with cordova?

This code works perfectly:
app.controller('MyController', ['$scope', 'usSpinnerService', function($scope, usSpinnerService){
    $scope.startSpin = function(){
        usSpinnerService.spin('spinner-1');
    }
    $scope.stopSpin = function(){
        usSpinnerService.stop('spinner-1');
    }
}]);

But this one:
app.controller('MyController', ['$scope', 'usSpinnerService', function($scope, usSpinnerService){
    $scope.myClick = function(){
        usSpinnerService.spin('spinner-1');
        cordova.exec(function(){
            $scope.data = "Done";
            usSpinnerService.stop('spinner-1');
        }, function(err) {
            callback('Nothing to echo.');
        }, "Echo", "echo", []);
    }
}]);

the spin start but never stop!
If I remove usSpinnerService.spin('spinner-1'); and usSpinnerService.stop('spinner-1');
Then I can see my $scope.data set to "Done"
Can you help me please?
Thank you

Hi @jycouet and thanks for opening this issue.
Why you're using cordova.exec? Cordova is wrapping what you see in browser into the mobile, so what works in browsers should work in mobile (except what related to cordova-webview and compatibility issues).
What you're trying to do?

Hi,
Sorry, I had this a long time ago.
My call to cordova was not placed well.
We can close this question.
Thank you

Thanks for your response


Update Docker test and fix Windows line endings in Dockerfile

clone notebooks in test script
also revert Pipfile to Python 3.7. I will change this to 3.8 later.

Actually, I was wrong. Just needed to clear the volume on OpenShift Data Science cluster. It works now with the PRELOAD check! Thank you @helena-intel


Figure out what's up with firstTime

Is this a custom property in this module? A standard property of pieces even though I can't find it in the A3 source? A misinterpretation of something that should be coming from the options object to insert but isn't being propagated? Figure this out.

It's a property on the event's data parameter: https://v3.docs.apostrophecms.org/reference/server-events.html


[Bug]: Postman collection not found

Contact Details
<EMAIL_ADDRESS>What happened?
For API examples when I tried to fork the postman collection, I am getting the following message in postman -
Collection doesn't exist
This collection seems to have been deleted or is no longer public

Source -

What browsers are you seeing the problem on?
No response
Relevant log output
No response

Thanks for taking the time to report the problem @likith-lt -- the fix should be live within the next 5 minutes.


[Impeller] vulkan: the back arrow isn't rendering in the gallery

engine version
commit 622751bdce53a8bc95ee933a13eccbabd23df913 (HEAD, upstream/master, upstream/main)
Author: Zachary Anderson<EMAIL_ADDRESS>Date:   Mon Jun 5 08:38:32 2023 -0700

    Check more optional accesses (#42528)

    Related https://github.com/flutter/flutter/issues/127701

screenshot

The back arrow is from an Icon font, so this is a dupe of https://github.com/flutter/flutter/issues/128283


"Problems" filter criteria find nothing when a new Java file is selected on the "Explorer" panel

Version: 1.42.1 (user setup)
Commit: c47d83b293181d9be64f27ff093689e8e7aed054
Date: 2020-02-11T14:45:59.656Z
Electron: 6.1.6
Chrome: 76.0.3809.146
Node.js: 12.4.0
V8: 7.6.303.31-electron.0
OS: Windows_NT x64 10.0.18362
Steps to Reproduce:

Install Language Support for Java: https://marketplace.visualstudio.com/items?itemName=redhat.java v0.58
Install SonarLint:https://marketplace.visualstudio.com/items?itemName=SonarSource.sonarlint-vscode v1.15
Open a Java project having some Java files with some mistakes (code smells, bugs, ...)
Open the "Problems" tab
Select a .java file having some code smells raised either by Language Support for Java or SonarLint
Enter into the filter field of the "Problems"tab  the word: "sonarlint" to see only issues raised by SonarLint
Select another .java file on the Editor panel : the "Problems" tab is showing: "No results found for the provided filter criteria"
Select the first file by clicking on its name on the open files
Select the second file the same way by clicking on its name (not on the Explorer panel, on the open files) => you see the issues matching filter criteria

Does this issue occur when all extensions are disabled?: No
Thanks
Alex

Are the problems shown without filter?

Yes, when there is no filter, I can see all the "problems/code issues" raised by the two extensions.

Maybe this screen recording will help to better understand the problem.

Can you please click on the funnel icon in the filter box and see if you have more filter enabled?

I have this in the funnel icon:

You have Show Active File Only filter on. Hence you are seeing problems only for the active editor. You have to disable this filter to see problems for all files.

I'm sorry but when I double-click on the file on the left panel to open it, I consider that this file is becoming the active one so I don't understand why you consider that doing this action should not display the "Problems" of this file?
Or maybe we have a different definition of what is the "active file". To me, that's the file that is currently opened / selected, the one I'm editing / about to edit.

Does the problems panel show errors when the above filter Show Active File Only is disabled?

When the Show Active File Only is disabled, I confirm I see ALL the errors of my project and the filtering based on the word "sonarlint" is working fine: I see only the errors reported by SonarLint.
The problem is that I expect, thanks to the Show Active File Only to see only the errors of the currently active file which is for me the purpose of this Show Active File Only option.
The behavior of the "Problems" tab is correct when I click on one of the "Opened Editors" tabs but it is not correct when I open a new file (ie: when a new tab is added to the list of "Opened Editors").
Did you manage to reproduce the problem on your side?

Thanks for the explanation and understood the issue.
Is it possible to provide a sample repo that can repro this issue?

You can reproduce the problem with this repo: https://github.com/agigleux/analyzers-playground

I tried to reproduce this using a sample extension and I do not see this issue.

Above repo is not helpful because there are no steps to repro other than just cloning.
I would recommend to help me in giving a small sample that I can use to repro.

The repo https://github.com/agigleux/analyzers-playground is the one I used to produce the small recorded video showing the problem.
You need first to open the file ValidationSocket.java, enter the filter "sonarlint" and then S4719_ExpectedIssue.java

It seems I have to install other tools too?


[CI] Integration tests are failing in firefox

Current Behavior

Currently the indexui_spec.js and settings_spec.js are failing in firefox but working fine in chrome.
Desired Behavior

Tests should work in all the intended browsers.
Please check this post for more info.

Contributor Guides and Handbook

🛠 Meshery Build & Release Strategy
📚 Instructions for contributing to documentation
Meshery documentation site and source
🎨 Wireframes and designs for Meshery UI in Figma
🙋🏾🙋🏼 Questions: Discussion Forum and Community Slack
🧪 Meshery Test Plan Spreadsheet

Thanks for opening this issue @vaibhav135, we'll get it prioritized in next Meshery Build & Release Meeting


jsx-indent wrong ternary indend

Hi again!
After updating from 6.3.0 to 6.4.0/6.4.1 this code generates error:

And this:

It wants one more indent before second ternary expression

Config the same: https://github.com/yannickcr/eslint-plugin-react/issues/897

It seems have a regression between v6.6.0 and v6.7.0. Test failed again on v6.7.0, but passed with v6.6.0.
Could we reopen this issue?


Great analog of this program

I found a more functional analog of this utility, with full text editor, balloon editor, image editor, auto OCR & translating text replacement, project saving and so on. Also, with a free version of deepl (deeplx) and a free version of google vision (google lens). I think you can take a lot from it so you don't have to program from scratch, and end up with a complete and user-friendly editor-translator.
https://github.com/dmMaze/BallonsTranslator

Just ran a few testruns.
BaloonsTranslator is better as a package, comic-translate seems to be better for auto translation and "set and forget".
Experiences with BaloonsTranslator;
First - on my ages old intel i5 4440 system with a GTX 970 I had frequent crashes using BallonsTranslator, when cuda was being used (GPU acceleration).
I was ultimately able to "fix" it, but it took some hefty trial and error. Basically - the default inpainting module (to be edited) used a complex ai model and instantly crashed with a cuda error, once I changed it to inpainter:AOT (also using cuda) those crashes went away.
Then I ran a test comic and got crashes after about a third of the comic being run, again cuda errors. I played around with the modules a little and ultimately figured out, that it was a race condition issue on my old system. Which would not occur, when I use cuda for OCR, CPU for inpainting, and CUDA for transating again. (time difference luckily is almost none in my case, because OCR takes the most time by far).
Setting up the program for the first time is more complicated than setting up comic-translate - but
in BaloonsTranslator

paint in quality is much better (higher) in difficut cases (yellowed comic pages because of time not being kind to paper)
false positive rate of "paint in" is lower (less faces painted over)
program already has a font outline color feature (pick which one you want)
program can be used without an API key (ocr is done on your machine, and google translate is free)
OCR always is free
The entire interface for moving text boxes, correcting text and changing font sizes is much more thought out

But, where comic-translate beats it is:

auto fontsize picking (have to double check)
better OCR using google vision (default ocr in BaloonsTranslator close, but google vision takes it)
comic-translate is much smaller in install size (400mb vs 1.7gb))
the "from/to" font-size limit picking option in comic-translate is a better concept, than the settings concept in BaloonsTranslator
text line splitting is better in comic-translate

also comic-translate doesnt require you to have a cuda compliant GPU in your system, because you can hand over OCR to google vision.
That said, if you want to work on bubble text placement or text editing, the interface in BaloonsTranslator is far superior.
All things considered, the full auto mode in comic-translate still takes it -- but I have to do a second test run to be sure (to be edited).
Also - I picked the free Deepl translation module in BaloonsTranslator once, and my IP was instantly temporarily banned (the program informed me), because I didnt set a delay. If you find out before me, what this delay has to be - please share... ;)

OH MY GOD... THIS IS THE FEATURE I'VE BEEN TRYING TO EXPLAIN FOR DAYS!!! The first image in BallonsTranslator explains everything, actually.
@notimp  Can you add me on Discord? I couldn't manage to set up BallonsTranslator. I'm getting an error; could you help me if I share my screen?
Username: Barbunay

OH MY GOD... THIS IS THE FEATURE I'VE BEEN TRYING TO EXPLAIN FOR DAYS!!! The first image in BallonsTranslator explains everything, actually.
@notimp Can you add me on Discord? I couldn't manage to set up BallonsTranslator. I'm getting an error; could you help me if I share my screen? Username: Barbunay

Argl....
(I'm on current Windows 11.)
Use this as a guideline:
(because thats what I already had installed)
https://github.com/ogkalu2/comic-translate/issues/113 (when you get to the virtualization steps in the comic-translate installation guide, skip them.)
Then I used the BallonsTranslator package you can donwload from mega or google drive (linked in it installation tutorial) (beware that the final installation will be huge in size (7 gigs or so, because dev version, or because of cuda drivers.)), and doubleclicked the .bat the tutorial told me to launch. During the install process, I got several errors, that the scripts folder of the project wasnt added to PATH, so I copied the path that popped up, and added it to PATH (you can use the tutorial in the comic-translate readme on how to do it).
My path was C:\Users\USERNAME\Documents\ballons-translator\ballontrans_pylibs_win\Scripts
but yours will probably be different.
After that everything installed fine (likely even into folders within the ballons-translator folder, because it balooned like crazy in size - but i'm unsure (if they virtualized pip installs or not...) ) and the program started. Installs took a while, but eventually it started.
After that I instantly ran into the first CUDA errors and silent program crashes out of the gate (upon running it), then the update .bat the tutorial told me to run complained that there was a git version conflict, and it couldnt run.
So I did the sensible thing, deleted the entire folder (downloaded from mega, - not the git one, didnt download stuff from ballons github yet).
Then downloaded the entire ballons github zipfolder with a browser (did the same for comic-translate btw, so if you havent so far, you can try that as well) - so for ballons, thats: https://github.com/dmMaze/BallonsTranslator/archive/refs/heads/dev.zip (for comic-translate, it would be: https://github.com/ogkalu2/comic-translate/archive/refs/heads/main.zip), I unzipped the ballons zip into the previous project location (old folder deleted entirely, but recreated one with the same name).
I then clicked on the lauch_win.bat file.
Stuff got installed, but not much stuff, as most of the dependencies had been installed before (comic-translate, or first balloon install try run).
Program started.
This time on my first translation run, it didnt crash silently anymore - but it threw those cuda errors, I talk about above, and crashed on those.
Then in ballon-translates settings, I set the paint in module to inpainter:AOT.
I also set Translator to use google translation and languages to Netherlands > German (because my testcomic was in dutch).
Then I set all the modules in ballon-tranlates settings from CUDA to cpu. Then I had no crashes and the test comic converted.
Then I set OCR, and detect text to cuda again. (But left the paintin module to use CPU, because of said race condition issue I encountered (see first posting (program would crash with a CUDA dindt responde fast enough error one third into the comic.)).)
And then everything worked, I had no crashes, and the comic converted in about 20 minutes.
The settings in ballon-translate look like information overload, but they arent very many, and the make sense, once you understand what, what module is doing.

Also to be able to use CUDA at all (gpu acceleration) you need to have an NVidia GPU in your system - and the correct versions of torch and torchvision installed.
comic-translate has a section on how to install those on an NVidia system in its installation tutorial. ballon-translate needs them as well (afair it tries to auto install them though).
those are the versions I have installed:

for a try run in ballon-translator with every module set to use CPU, those wouldnt be needed, but they likely would be needed for you to get GPU support in ballon-translate as well. you can follow the guide on the comic-translate install tutorial page to install those.
the versions of your cuda drivers in windows you can check by typing in:
nvidia-smi
followed by enter into a cmd command line window.
CUDA version will show in the second line on the right:

@notimp pls add me on discord :(
this is hard for me
Username: Barbunay

@notimp pls add me on discord :( this is hard for me Username: Barbunay

I will try to answer your questions in here, but as you see, my installation also wasnt very "straight forward".

I had some dependencies already installed from comic-translate
I installed the "ready to download package" the ballon-translate tutorial told me I could use, first. But then I uninstalled it, since I had instant crashes, and the package couldnt update like the tutorial indicated it should be able to.
Then I downloaded the entire ballon-translate .zip package from github (link in this posting), and started over with it.

So my installation wasnt clean.
And me achieving a working installation wasnt straight forward.
But I still wrote down all my steps.
Make sure you are on python-3.9.13 and no newer version, to be able to get both comic-translate and ballon-translate to work.
Start fresh with uninstalling python (see: https://github.com/ogkalu2/comic-translate/issues/113) and deleting the pip temp folder (also see: https://github.com/ogkalu2/comic-translate/issues/113) then follow my steps.
Or post the error messages you are getting. :)
On a general not, what you are doing here - python is a programming language, which doesnt need compiled files to run. Just the product files you code. This is why installation is so horribly user unfriendly. :)
You have to use command line (with ballon-translate, maybe you dont, actually... :) ) you have to read what that command line window tells you, while the installation is going on.
What both comic-translate and ballon-translate do is to depend on you having python-3.9.13 installed, then both their installers tell python to install other modules. Those modules are fetched from the internet using the requirements.txt inside the respective projects folders. Those are then handed over to pip (the python module installer) and pip installs the modules from the internet.
If your python version is to new, some of those might fail. Leading you to get to the programs not launching in the end.
If you updated your pip installer, because it constantly says, that a never one is available, and you really should, the newer one might install newer modules, and as a result the program in the end might then fail to launch. So dont update the pip installer even though it constantly negs you to do so.
Getting something working in python is basically a game of having versions of the modules installed that are not so different from the ones that the developers used, that the program breaks.
Which is why deleting the pip cache folder, unsintalling python, by using its uninstaller (say yes to all questions about deleting settings as well), and then starting with the specific python version named in here (python-3.9.13) is so important.
After that its really just following installation tutorials.
Or in the ballon-translate case, having python-3.9.13 installed, downloading the .zip file from https://github.com/dmMaze/BallonsTranslator/archive/refs/heads/dev.zip , unzipping it into a folder in your Documents folder on windows. Going into that folder, and doubleclicking the launch_win.bat . That should be all thats sneeded for ballon-translate.
I still documented, my entire installation process - because I had to do it over - and in that case, some modules from pip might have already been installed....
As for the rest, the cmd interface in windows should tell you error messages that you can post. I'll try to help you with those, but I'm no dev.
And even though I have supported thousands of people before, I always drew the line at not supporting them in private messages on discord. Sorry... :/

@notimp i did it !!!
and this program is amazing
I couldn't find a way to manually add text in the program. Otherwise, it actually offers everything I want.

Google Translate doesn't require an API Key. I've just fixed that.

@notimp i did it !!! and this program is amazing I couldn't find a way to manually add text in the program. Otherwise, it actually offers everything I want.

W to activate text block creating mode, then drag the mouse on the canvas with the right button clicked to add a new text block. (see the text editing gif)
Read the manual, sometimes you have to use keyboard (for example ctrl+s to save picture), right mouse button (like add a new text block) and so on...

I just troubleshooted with the creator of Ballon Translators google lens plugin for one and a half days, and for the first time got the google lens plugin to work (VPN was needed, in Europe, google lens API seemingly just doesnt respond).
With google lens working as OCR in Ballon translator - this shifts my previous statement around again.
Google lens for OCR and Google translator for translation can be used with Ballon translator, both are free. Google lens is better than the default OCR running on Ballon translator.
Translation with both of those is very fast - as in 14 Minutes per comic. - Auto text quality now is "as good as with comic-translate", aside from text formating. Auto text formating in Ballon Translator is still far worse. (Auto paint in is better.)
I then did a quality control pass on Ballon translator for an entire comic, and BOY is this tiresome... Used mouse wheel for font size switching and still -- having to manually reformat the text to change its flow (linebreaks) is so bothersome...
Takes about twice the time that formating text in comic-translate does. But the end result is better (wysiwyg), and Ballon Translator can be used by a monkey (quality control steps are very easy)...
So comic-translate still wins outright. But...

Ballon Translator can be Free OCR/Free Translation (google lens/google translate)

Ballon Translator can be twice as fast as Comic translate thate way (15 instead of 30 minutes)

Ballon Translators full auto results are very servicable.

Ballon Translator give you the best quality (wysiwyg) on a quick quality control pass but... It takes ages.

comic-translates full auto mode is better.

comic translate is better (buch faster) for a quick quality control pass.

end quality will not be as perfect as with Ballon translator, if you put in the 1h30 of editing a comic/manga in post.

All in all comic-translator still takes it for my purposes. Even though its not free. (20-60 Euro cent per comic, for paid ocr and paid translation).
Keep up the good work, I'll tick to comic-translate as my go to. :)

@notimp

Ballon Translator can be Free OCR/Free Translation (google lens/google translate)
In comics translator you can also translate with google-translator for free, and OCR is also free by default. But for me personally, the Google translation was disgusting, I'd rather pay 1 cent per page than rack my brain trying to figure out what he meant and how to fix it, and spend half a day translating most of the text into human language. There is no such problem in GPT, you only need to edit a little bit.
And the free default OCR suits me, there are few errors) And GPT4o translates perfectly even with minor errors, you don’t need to correct it

If I'm not mistaken, google vision is 20cents per comic (although google gifts you 200 USD worth of credits on your first use). It very decent OCR quality - so its worth it. But. :)
Google vison give you about the same OCR quality - at no cost. I understand why it might not be implemented in comic-translate, because you are piggybacking on a service that is not intended for mass use, and isnt available in API form, while google vision is the paid analog to it - and doesnt cost much, so thats totally fine.
But - I have to say - after I had my 2 Issues with Balloon translator attended to:
1.: https://github.com/dmMaze/BallonsTranslator/issues/554
2.: https://github.com/dmMaze/BallonsTranslator/issues/558
And ofter some HEFTY configuration work to get the best results out of their automated text placement - it will become my go to for one type of comic.
Here is the breakdown:
Google lens is free and good. (Ballon Translator)
Google Vision is low cost and good (comic-translate)
Translation modules you offer many, and they offer many - although thei havent implemented GPT4o-mini yet.
Thats the baseline. (Their (Ballon Translators) on-machine OCR is worse, although not by much - and alos free.)
In projects where I know, that 90% of the comic will use one fontsize, and bubbles are big, and there is ample space for positioning the translated text in a decent fontsize --
Ballon Translator will now become my go to. OCR and translation takes half the time. And if you do a manual pass over textbubbles, simply resizing the textboxes, and in few cases, changing fonts, Ballon translator will give you better results. Largely to the better Paint in module -- if your comics need it. Also its text editing interface is better.
In projects where auto recognition of varying font sizes is more important, of where text placement is more essential (textboxes without ample space to place translated text), comic-translate wins every time, because its auto text fitting (size, linebreaks) is way better. Also in "full auto - set and forget" cases, comic-translate wins outright.
But as soon as you do a manual quality control pass - with the correct starting settings (freeze font, freeze font color, freeze font contouring color, freeze contouring size, freeze font size, and disable the "try to position text like the buble" option in Ballon translator), quality control in that program becomes bareable (Although for some reason its still more fun in comic-translate), and if the comic allows for it (so large speech bubbles, almost always adressable with text in one font size (you can edit that selectively later of course, but if you have to frequently it quickly takes up time)), and the end result will look better than with comic translate.
Comic-translate is easier to use, more fun to use, gives you better results out of the box and in full auto mode, but paint in and the text editing capabilities are further developed in Ballon Translator. Thats essentially it.
As for Google translate --
it depends on the language pair you are translating. Italian > German gave be horrible results. Dutch > German (which are very similar languages, so if I voice out written Dutch text phonetically there is a 40% chance, that I will understand it by just knowing german -- reading Dutch without phonetically voicing out the words, is not possible for a German speaker though. :) ) have usable results.
Aside from the frequent ocurence of "helping!!!!" instead of "help!!!" in speach bubles es well as "waiting!!" instead of "please wait!!" the translation of that language pair is fine. GPT4o-mini translation is better - but not by that much, that it beats free.. ;) With Italian this is entirely different. There I need ChatGPT4o-mini (economical and decent ;) ), because google translate results are not usable.

how much text have you done here. horror...
I am quietly developing free technologies of companies that are presented in CT through official APIs. and by the way, that google lens api and BT lens OCR are the same thing. only I just use Google's open methods of obtaining information.

and gentlemen, I advise you to start a discussion for this, not an issue. this will make it easier for developers.

@notimp
What inpainting model are you using for BallonsTranslator when you say it's much better ? The default (lama large) or AOT ?
Also i just implemented that kind of rendering in comic-translate.

@notimp What inpainting model are you using for BallonsTranslator when you say it's much better ? The default (lama large) or AOT ? Also i just implemented that kind of rendering in comic-translate.

AOT. I have to say, that those are experiences with one kind of comic, with old yellowed pages. When using your inpaint module on a black and white comic, the results of the inpaint module comic-translate was using were better than in my most "common" case. (Again most of my comics have yellowed pages, and I kind of like it that way).
That said, AOT for my usecase was better all around. Better cleaning, less false positives on faces.
ALso I couldt test lama large, because by GPU coughs and throws errors.
So my data points are screwed. Try to collect more data, before making decisions. :)

@notimp

@notimp What inpainting model are you using for BallonsTranslator when you say it's much better ? The default (lama large) or AOT ? Also i just implemented that kind of rendering in comic-translate.

AOT. I have to say, that those are experiences with one kind of comic, with old yellowed pages. When using your inpaint module on a black and white comic, the results of the inpaint module comic-translate was using were better than in my most "common" case. (Again most of my comics have yellowed pages, and I kind of like it that way (dont want to remove that in a first step))).
That said, AOT for my usecase was better all around. Better cleaning, less false positives on faces.
Also I couldnt test lama large, because my GPU coughs and throws errors.
So my data points are skewed. Try to collect more data, before making decisions. :)

Okay. I tried the comic you posted previously and segmentation seemed to be the big issue so i made a few changes and inpainting should work much better on those kinds of images

@notimp What inpainting model are you using for BallonsTranslator when you say it's much better ? The default (lama large) or AOT ? Also i just implemented that kind of rendering in comic-translate.

AOT. I have to say, that those are experiences with one kind of comic, with old yellowed pages. When using your inpaint module on a black and white comic, the results of the inpaint module comic-translate was using were better than in my most "common" case. (Again most of my comics have yellowed pages, and I kind of like it that way (dont want to remove that in a first step))).
That said, AOT for my usecase was better all around. Better cleaning, less false positives on faces.
Also I couldnt test lama large, because my GPU coughs and throws errors.
So my data points are skewed. Try to collect more data, before making decisions. :)

Okay. I tried the comic you posted previously and segmentation seemed to be the big issue so i made a few changes and inpainting should work much better on those kinds of images

https://github.com/bropines/chrome-lens-py/tree/main
Can you add support? I mostly use BT, but it would be nice for people to have another OCR method. It seems like I even tested it

Did some additional testing with lama large 512px on a second machine, here are the results:

lama large 512px is better than AOT, in terms of fill in quality (less white added to the mix if the model is unsure)
both lama large and AOT have less false positives on faces (as in almost no false positives at all.)
text detection seems to be better in BallonTranslator as well. Not by much, but there are fewer textboxes missed overall. (the difference isnt huge because the false negative rate on comic-translate isnt high to beginn with)

Tested all of that using the same Comics, with the same yellowed pages you saw me using here all along. :) So no input on the performance on other comics from me yet.
Will test the new inpaint module in comic-translate in the near future, thanks for trying to adress my specific issues. :) (No ETA.)
In case you run into GPU errors using lama large 512 and cuda, here are the versions of torch, torchvision and torchaudio, I have installed:
pip install torch==2.2.2 torchvision==0.17.2 torchaudio==2.2.2 --index-url https://download.pytorch.org/whl/cu121


Permissions problems

3.2.4
Players don't have access to quests' journal even having quests.journal permission node
PermissionsEx 1.23.4

Doesn't seem to be an issue on my test server, which uses PEX. Permissions are handled through Vault, so you try seeking help from them, but its more likely an issue with your PEX setup. Regardless, without an error or some kind of lead to go on, this is beyond my ability to assist you.


1.4.1 crashes on startup

title

Can you post your playnite.log?

Can you post your playnite.log?

Apparently after reinstalling it multiple times, it was fixed? It's odd. I guess go ahead and close this because the log file was already overwritten.

Apparently after reinstalling it multiple times, it was fixed? It's odd. I guess go ahead and close this because the log file was already overwritten.


When terminal is opened it gives uncaught exception

Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behavior:

Go to '...'
main page
Click on '....'
open terminal
Scroll down to '....'
See error

Expected behavior
It should open the terminal to run the commands , but when we ran the command it gives the following error.
kubectl get ns
Access is denied
At line:1 char:1

kubectl get ns

  + CategoryInfo          : OperationStopped: (:) [], UnauthorizedAccessException
  + FullyQualifiedErrorId : System.UnauthorizedAccessException

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

Lens Version:3.4.0
OS: [e.g. OSX]Windows10
Installation method (e.g. snap or AppImage in Linux): exe

Logs:
When you run the application executable from command line you will see some logging output. Please paste them here:
Your logs go here...

Kubeconfig:
Quite often the problems are caused by malformed kubeconfig which the application tries to load. Please share your kubeconfig, remember to remove any secret and sensitive information.
your kubeconfig here

Additional context
Add any other context about the problem here.

Hello, I tried Lens for the first time in my system. Looks really good! But when I want to change some values in the Terminal and Vave .... it shows the same issue like satya67 mentioned. Is there a solution for that? Thanks for your support. Cheers!

I think this is related to : https://github.com/lensapp/lens/issues/489 ? Because the kubectl.exe file is 0kb and we dont have access to the internet on that server where we run Lens.

Is this still a problem in 4.1.4?


Where is the apk ?

How to get the folioreader apk

@netroby FolioReader is an EPUB SDK and not just an APK based app.

Sorry for that. thanks for your help


generate API to return list of monitoring elasticsearch for the service

The api will be:
/monitoring/elastic GET
and will return a json with the data from the table of monitoringinfoelastic

depends on #151

API will be /monitoring/elastic/list/{customizationid}
since we need to get the monitoring info of that customization instance

Description
Values

Description:
we need to read the list of monitoring info in the new table from #151 filtered by customizationid

Screen that needs the endpoint:
monitoring page

Endpoint required:
/api/monitoring/elastic/list/{customizationid}

Call Type:
GET

Done. The API is /api/monitoring/elastic/info/list/{customizationId}


New version: twpayne.chezmoi 2.55.0

Checklist for Pull Requests

[ ] Have you signed the Contributor License Agreement?
[ ] Is there a linked Issue?

Manifests

[ ] Have you checked that there aren't other open pull requests for the same manifest update/change?
[ ] This PR only modifies one (1) manifest
[ ] Have you validated your manifest locally with winget validate --manifest <path>?
[ ] Have you tested your manifest locally with winget install --manifest <path>?
[ ] Does your manifest conform to the 1.6 schema?

Note: <path> is the directory's name containing the manifest you're submitting.

Automated with GoReleaser
Microsoft Reviewers: Open in CodeFlow

The error is:

##[error]1ES PT Error: Job 'InstallerValidation' uses the release tasks ["AzureFunction@1","AzureFunction@1"] but is not marked as a release job. Release tasks will soon be disallowed outside of release jobs. See https://aka.***/1espt/custom-release-job

This looks like a problem in the winget-pkgs automation, as this PR contains only YAML files which are extremely similar to https://github.com/microsoft/winget-pkgs/pull/190156.

@wingetbot run


Address clippy 1.66 linting suggestions

I'd like to resolve and merge https://github.com/dburgener/cascade/pull/101 before taking this, since having this issue helps for testing #101.  Once #101 merges, these sorts of issues become less urgent and can be addressed as convenient.


Add support for audio fields

Want to allow people to access their audio files from this API client.

Fixed by #5!


mautrix-telegram: 0.4.0.post1 -> 0.5.0

Motivation for this change
The available version is still a release candidate so this PR will be a draft until the stable realease
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Assured whether relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

Final version is now out and this PR was updated

@GrahamcOfBorg build mautrix-telegram

@GrahamcOfBorg build mautrix-telegram

Btw we should definetely backport this as mautrix-telegram 0.4.0.post1 fails when running mautrix-telegram -g with the following error:
→ ./result/bin/mautrix-telegram ../mautrix-telegram/config.yaml
Traceback (most recent call last):
  File "/nix/store/k0lgh8g637r35bb6ywl90b7xfyhdak4l-python3.7-mautrix-telegram-0.4.0.post1/bin/.mautrix-telegram-wrapped", line 7, in <module>
    from mautrix_telegram.__main__ import main
  File "/nix/store/k0lgh8g637r35bb6ywl90b7xfyhdak4l-python3.7-mautrix-telegram-0.4.0.post1/lib/python3.7/site-packages/mautrix_telegram/__main__.py", line 31, in <module>
    from .web.provisioning import ProvisioningAPI
  File "/nix/store/k0lgh8g637r35bb6ywl90b7xfyhdak4l-python3.7-mautrix-telegram-0.4.0.post1/lib/python3.7/site-packages/mautrix_telegram/web/__init__.py", line 1, in <module>
    from .provisioning import ProvisioningAPI
  File "/nix/store/k0lgh8g637r35bb6ywl90b7xfyhdak4l-python3.7-mautrix-telegram-0.4.0.post1/lib/python3.7/site-packages/mautrix_telegram/web/provisioning/__init__.py", line 28, in <module>
    from ...user import User
  File "/nix/store/k0lgh8g637r35bb6ywl90b7xfyhdak4l-python3.7-mautrix-telegram-0.4.0.post1/lib/python3.7/site-packages/mautrix_telegram/user.py", line 33, in <module>
    from .abstract_user import AbstractUser
  File "/nix/store/k0lgh8g637r35bb6ywl90b7xfyhdak4l-python3.7-mautrix-telegram-0.4.0.post1/lib/python3.7/site-packages/mautrix_telegram/abstract_user.py", line 25, in <module>
    from telethon.tl.types import (
ImportError: cannot import name 'UpdateChatAdmins' from 'telethon.tl.types' (/nix/store/06amhafx1ahwdcgw6kjkq0qjggim5dyp-python3.7-telethon-1.5.5/lib/python3.7/site-packages/telethon/tl/types/__init__.py)

backported in 622bb133c2d4b15253a9cd358721cf5b85ec832f, 199a8e7ac824a83226a122d94ae3118d5cfedc8c, and 19ad18695ed08d01199cd868d16da0077e70f2bf


Add Requestor to kubectl output, moves Issuer name from wide to default outpt

This PR changes the behaviour of kubectl get cr to now include the username of the requestor, and outputs the issuer name by default without -o wide. This makes the output more inline with upstream certificates.k8s.io, and looks better to me :)
$ kubectl get csr
NAME   AGE   SIGNERNAME                            REQUESTOR          CONDITION
john   24m   kubernetes.io/kube-apiserver-client   kubernetes-admin   Pending

$ kubectl get cr
NAME           READY   ISSUER              REQUESTOR                                         AGE
osm-ca-2fvdl   True    selfsigned-issuer   system:serviceaccount:cert-manager:cert-manager   13m

$ kubectl get cr -o wide
NAME           READY   ISSUER              REQUESTOR                                         STATUS                                         AGE
osm-ca-2fvdl   True    selfsigned-issuer   system:serviceaccount:cert-manager:cert-manager   Certificate fetched from issuer successfully   14m

`kubectl get certificaterequest` now outputs the Issuer name and the username of the requestor by default

/assign @jakexks @irbekrm

I like it but it is pretty wide with a service account name. Anyway, I'm sure people use wide enough terminals nowadays.
/lgtm

/kind feature


Negative throttle

Hi,
I love your mod. Its an awesome project. Just one issue. when using xbox controller, by pulling left analog stick down i get negative acceleration. Like props can spin backwards. Is there a seting i am missing to make throttle from 0-1. 0 can be either on left stick in middle or down. I could play either way. Maybe because xbox analog is springed its better to place 0 in middle. Like it is now. But how to set it up so when i pull down throtle stays 0 and doesnt got to -1?
If this is a missing feature/bug, I am willing to contribute to your code

There is a "3D Mode" checkbox in the Input tab that you can uncheck to make the motors spin only one way. I intentionaly made 3D mode to be on by default because most of the people will be using controllers with centering springs


[rom_ext] Startup from ROM handoff

[ ] Verify configuration expectations from ROM
[ ] Setup fault handling (ISRs, shutdown handling)
[ ] Setup any additional alert handling
[ ] #18649

Test Plan:

Verify Expectations: Would need to corrupt a register between ROM and ROM_EXT.  Perhaps we can do this with a breakpoint and JTAG?
Fault handling: owner-stage test program that does not set its own interrupt handlers.  Provoke faults and observe a proper shutdown message from ROM_EXT handlers.


Likert scale font size

We have received feedback that the font size for the markers on the likert scale are too small.
Increase the font size but be aware of likert scales with very long marker text.

 General refinement


Rehabilitate lazy_sort; use static dispatch for lazy_sort benchmarks

Problem: lazy sort's .sorted() uses a function pointer for the
comparison. Calling through a function pointer is known as virtual
dispatch, and if the compiler can't translate that back to static
dispatch through devirutalization, then we have virtual call overhead.
This makes the comparison with .sort() and BinaryHeap unfair.
Proposed Solution: Fix .sorted() to use static dispatch in the next major version.
local results (show a big improvement):
test benches::a_heap_bench     ... bench:     354,054 ns/iter (+/- 5,278)
test benches::a_lazy_bench     ... bench:     376,453 ns/iter (+/- 22,853)
test benches::a_standard_bench ... bench:   3,202,741 ns/iter (+/- 101,769)
test benches::b_heap_bench     ... bench:   1,075,371 ns/iter (+/- 45,413)
test benches::b_lazy_bench     ... bench:   1,056,641 ns/iter (+/- 49,884)
test benches::b_standard_bench ... bench:   3,313,681 ns/iter (+/- 294,452)
test benches::c_heap_bench     ... bench:   3,713,321 ns/iter (+/- 236,979)
test benches::c_lazy_bench     ... bench:   3,960,629 ns/iter (+/- 156,475)
test benches::c_standard_bench ... bench:   3,372,774 ns/iter (+/- 161,264)

cheeky ping to @dtolnay

Thanks for this, that's quite a dramatic difference 😃
I'm going to leave this PR open for the time being, as a reminder until the necessary refactoring can be done.

I'm going to close this now that the recommended work has been done as part of #7


Organize base/styles like the other features

Now that the other features have a way to switch between Native and Web
and use an import-export logic based on features rather than on files,
make base/styles consistent with the other features.

+1


Add support for importing JSON files

This change adds support for importing JSON files using Langchain's JSONLoader.  Here are the major changes:

Created new routes and types in the backend-server
Added appropriate frontend views
Added HandleJsonDataSource.php that handles the uploaded files.   Along with json files, I also allow zip files to be uploaded.  Zip support required adding the "zip" php extension to the docker file.
Added jsonHandler.ts to the llm-server that imports the json files.  It uses Langchain's JSONLoader to import the files as documents.

How is this coming?
I would love to load products list.


zsh-syntax-hightling + zsh-autocorrect breaks history-incremental-pattern-search-backward

In my zsh configuration if I load both zsh-syntax-highlighting and zsh-autocorrect then history-incremental-pattern-search-backward is broken (I have it bound to ^R.
If I disable one or the other, then history-incremental-pattern-search-backward.
If I leave ^R bound to its default, history-incremental-search-backward then that works as well.
The load order of zsh-syntax-highlighting and zsh-autocorrect doesn't seem to make any difference.
I created a minimal test case for you: https://gist.github.com/docwhat/0cd192c066a5f9da69244f1631c5d481
My files:

all my dotfiles -- The zshrc files are loaded from zsh/startup/*.zsh.
my plugins being loaded
where I do bindkeys

In my zsh configuration if I load both zsh-syntax-highlighting and zsh-autocorrect then history-incremental-pattern-search-backward is broken (I have it bound to ^R.
If I disable one or the other, then history-incremental-pattern-search-backward.
If I leave ^R bound to its default, history-incremental-search-backward then that works as well.
The load order of zsh-syntax-highlighting and zsh-autocorrect doesn't seem to make any difference.
I created a minimal test case for you: https://gist.github.com/docwhat/0cd192c066a5f9da69244f1631c5d481

Thanks for the minimal example.  Haven't looked yet, but two quick comments:

This will probably fixed by #245 (the fix for which is already written, and is only waiting for zsh 5.3 to be released).

I wonder if the patch at https://github.com/zsh-users/zsh-autosuggestions/issues/126#issuecomment-217041261 fixes this issue as well.  (Need to run make after applying it to zsh-autosuggestions.)

I'm looking forward to 5.3, then. :-)
The mini-patch you provided did not fix my test-case (posted above) nor my normal shell environment. :-(

Thanks for testing the patch and for the very detailed followup.
Are you sure z-asug is involved here?  If I do just:
% bindkey '^R' history-incremental-pattern-search-backward
% echo foo
% echo bar
% <press Ctrl+R>echo<press Ctrl+R>

then the result is different if I source z-sy-h after the bindkey,
compared to if I don't source z-sy-h at all.
That's with zsh master and z-sy-h master (which still use the
5.2-and-earlier codepath), in zsh -f.

... and #245 does not solve this issue.

Milestone 0.6.0 — I'd like to understand the root cause before 0.6.0.

After much digging through code and reading up on what zle and widgets are, etc...
This fixes the problem:
source ~/zsh-syntax-highlighting/zsh-syntax-highlighting.zsh
source ~/zsh-autosuggestions/zsh-autosuggestions.zsh

typeset -ar ZSH_AUTOSUGGEST_IGNORE_WIDGETS=(
  "${ZSH_AUTOSUGGEST_IGNORE_WIDGETS[@]}"
  zle-isearch-update
)

So it looks like whatever zsh-autosuggest is doing to zle-isearch-update is breaking backwards search, but only if I enabled zsh-syntax-highlighting too.

After much digging through code and reading up on what zle and widgets are, etc...

Feel free to join #zsh or #zsh-syntax-highlighting on freenode with any questions.

This fixes the problem:
source ~/zsh-syntax-highlighting/zsh-syntax-highlighting.zsh
source ~/zsh-autosuggestions/zsh-autosuggestions.zsh

typeset -ar ZSH_AUTOSUGGEST_IGNORE_WIDGETS=(
  "${ZSH_AUTOSUGGEST_IGNORE_WIDGETS[@]}"
  zle-isearch-update
)

So it looks like whatever zsh-autosuggest is doing to zle-isearch-update is breaking backwards search, but only if I enabled zsh-syntax-highlighting too.

That's interesting.
This explains why the redrawhook branch is also affected: it wraps
zle-isearch-update in the pre-5.2 way.
Could you try whether the problem reproduces with zle-isearch-update() { zle .zle-isearch-update  -- "$@" ; false } and zsh-autosuggestions, but without z-sy-h?  I don't see anything obviously wrong in either codebase, but it's late here and I could easily be missing something.
Thanks for narrowing this down.

Could you try whether the problem reproduces with zle-isearch-update() { zle .zle-isearch-update  -- "$@" ; false } and zsh-autosuggestions, but without z-sy-h?

I took it for a spin in my test-gist and everything worked correctly. The problem didn't appear.
# Lines from .zshrc:
zle-isearch-update() { zle .zle-isearch-update  -- "$@" ; false }
source ~/zsh-autosuggestions/zsh-autosuggestions.zsh

zle -la | grep isearch returned nothing, so I'm not sure the test did what you wanted.

Could you try whether the problem reproduces with zle-isearch-update() { zle .zle-isearch-update  -- "$@" ; false } and zsh-autosuggestions, but without z-sy-h?

I took it for a spin in my test-gist and everything worked correctly. The problem didn't appear.
# Lines from .zshrc:
zle-isearch-update() { zle .zle-isearch-update  -- "$@" ; false }
source ~/zsh-autosuggestions/zsh-autosuggestions.zsh

zle -la | grep isearch returned nothing, so I'm not sure the test did what you wanted.

Indeed it didn't, sorry about that.  What I had in mind was zle-isearch-update() {}; zle -N zle-isearch-update.

Yup; this causes the exact same issue in zsh-autosuggest:
# Lines from .zshrc
zle-isearch-update() {}; zle -N zle-isearch-update
source ~/zsh-autosuggestions/zsh-autosuggestions.zsh

Yup; this causes the exact same issue in zsh-autosuggest:
# Lines from .zshrc
zle-isearch-update() {}; zle -N zle-isearch-update
source ~/zsh-autosuggestions/zsh-autosuggestions.zsh

In this case, I suggest you report this as a bug against zsh-autosuggestions, since you're about to reproduce the bug without zsh-syntax-highlighting.
Thanks for your help diagnosing the issue!

Done!


refactor(docs): reduce visibility of from_center_cartesian in our documentation

Overview
We want to encourage users to use the center() + offset method when they are writing their own custom locations based on labware in protocols rather than using arbitrary percentages on a well. We should no longer call out from_center_cartesian in the Labware section of the docs, further explain how best to use the function in the docstring and point users interested in this function to read the API reference for more details.
Changelog

Remove reference to from_center_cartesian in new_labware.rst
Further explain from_center_cartesian in the docstring
Add a pointer to the API reference from versioning.rst

Review requests
Check out everything and see that it makes sense to you.
Risk assessment
Low. Changing around some docs.

Agreed with these changes. This is also under discussion in #7579.


Popover not opening on click when content loaded from json file (Wenzhixin bootstrap-table)

Sandy thanks for the great work!  I have been using your webui-popover and it is great.
Recently I have been trying to use it along with Wenzhixin bootstrap-table for it's sort capabilities. When I load my content into the bootstap-table via a .json file nothing happen when I click the link that should open a popover.  Best I can tell everything loads correctly but the popover doesn't open when clicked.
Any suggestions ?
Thanks

Hi, I've tried the Wenzhixin bootstrap-table work with webui-popover and found it just working,  I can't find the problem which you have. Here is my testing code:
   $('#table').bootstrapTable({
  		    columns: [{
  		        field: 'id',
  		        title: 'Item ID'
  		    }, {
  		        field: 'name',
  		        title: 'Item Name'
  		    }, {
  		        field: 'price',
  		        title: 'Item Price'
  		    }],
  		    data: [{
  		        id: 1,
  		        name: 'Item 1',
  		        price: '$1'
  		    }, {
  		        id: 2,
  		        name: 'Item 2',
  		        price: '$2'
  		    }]
  		});

  		$('.abc').webuiPopover({
  			title:'ddd',
  			content:'dxxxxx'
  		});

Thank you for the reply.  I believe my problem stems from having the table populating from a .json file.
<!DOCTYPE html>

    <link type="text/css" rel="stylesheet" href="/css/base_style.css"/>
	<link href="/css/bootstrap.css" rel="stylesheet">
    <link href="/css/bootstrap-table.css" type="text/css" rel="stylesheet" >

2014-2015

2015-2016

        Date
        Time
        Opponent
        Location
        Results

        Date
        Time
        Opponent
        Location
        Results

    <link rel="stylesheet" href="/jquery.webui-popover.css">

JSON
[
    {
        "date": "11/3/15",
        "time": "12:00 pm",
        "opponent": "Test 1",
		"location": "<a href=\"#\" class=\"abc\" tabindex=\"0\" role=\"button\" data-toggle=\"popover\" title=\"Test 1<\/center>\" data-trigger=\"click\" data-placement=\"auto-bottom\">home<\/a>",
		"results": ""
    },
     {
        "date": "11/4/15",
        "time": "2:00pm",
        "opponent": "Test 2",
		"location": "Away",
		"results": ""
    },
	    {
        "date": "11/5/15",
        "time": "1:00 pm",
        "opponent": "Test 3",
		"location": "Home",
		"results": ""
    }
]

Everything looks fine on the resulting page but when you click what should be the popover nothing happens.

Just checking back to see if you have had an opportunity to look at this issue again ?

Sorry for so long delay, I tried your code and found the reason.
The problem is in your code,when  you inited the popover, but the data of the bootstrap-table was not loaded,so your jQuery selector $('.abc') will return empty array, of course the popover can't be initialised.
To fix the problem, I suggest you init the bootstrap-table by javascript code instead of using data attribute, so you can init the popover in onLoadSuccess of bootstrap-table.
Here is my code, I've tested it, the popover worked fine with bootstrap-table.

        Date
        Time
        Opponent
        Location
        Results

$('#jsontable').bootstrapTable({
        onLoadSuccess:function(data){
            $('.abc').webuiPopover({
                  title:'ddd',
                  content:'dxxxxx'
            });
        }
      });

Thank you for the help.  That solved my problem!  Incase anyone else looks at this for a similar issue...
The above solution worked perfectly on the original page load but if a user used the sort arrows to reorder the table it was again not working. (returning an empty array) To fix this I made one minor change to the javascript code such that the popover works on both original page load and after sorts. I changed onLoadSuccess to onPostBody
$('#jsontable').bootstrapTable({
        onPostBody:function(data){
            $('.abc').webuiPopover({
                  title:'ddd',
                  content:'dxxxxx'
            });
        }
      });

Hi,
I'm re-opening the thread on this matter because I'm having the same problem as Same0717 which is that on column reorder or page change the popover doesn't work anymore. And the onPostBody solution doesn't work for me as it doesn't return the data array.
Does anyone got this right ?
Here is my code which works for me only on first table load.
`onLoadSuccess:function(data){
	$.each(data, function (key, array) {
		var row_designation = array['designation'];
		var row_content = array['details'];

		$('.popover_inbox_' + array['id']).webuiPopover({

			title: row_designation,
			content: row_content,
			placement:'auto',
			container: document.body, // The container in which the popover will be added (i.e. The parent scrolling area)
		    width:'800',//can be set with  number
		    height:'350', //can be set with  number
		    trigger:'hover',
		    animation: 'fade', //pop with animation,values: pop,fade (only take effect in the browser which support css3 transition)
		    delay: {//show and hide delay time of the popover, works only when trigger is 'hover',the value can be number or object
		        show: null,
		        hide: 300
		    },
		    padding:true,
		    type:'html',
		});
	});
}`


Allow Freeform Component Metdata

In what area(s)?
/area runtime
Describe the proposal
Currently component.dapr.io/v1alpha1 spec.metadata must be structured as an array of { key: string, value: string } entries. This constrains what information may be provided via metadata and arguable makes component definitions more verbose to read.
I propose v1alpha2 that supports a map[string]interface{}-like freeform dictionary for the spec.metadata field. This would allow a cleaner component definition as well as rich values beyond strings.

It looks like this was done in part to support allowing secret values. I wonder if there's not a better way to enable that?

Mostly due to this: https://github.com/kubernetes/gengo/issues/138

@yaron2 would a JSON field with https://github.com/kubernetes-sigs/controller-tools/issues/281 get close? That seems the be "recommendation" for arbitrary objects in a given field.

Apparently there's support for custom marshalling and unmarshalling: https://github.com/mongodb/mongodb-kubernetes-operator/pull/134/files#diff-49c488ba1252fd36602f98c8df647f9eR74
Not sure if that'd meet that same need.

Another option would be to have Dapr dynamically managed CRDs for components: https://github.com/open-policy-agent/gatekeeper/blob/master/pkg/controller/constrainttemplate/constrainttemplate_controller.go
But that seems like a massive amount of work 😅.

Per component CRDs would result in 10s or 100s of CRDs, and user feedback has been that the single component CRD is a low maintenance preferred solution.
There is a way to do this, basically patch up the generated files manually and write the correct deep copying logic. This is possible because k8s-codegen will still autogenerate the files, they just will not compile due to errors in the DeepCopy function for map[string]interface{} fields.
So essentially write a tool that generates and files and then patches up required places in the code.

Closed with https://github.com/dapr/dapr/pull/2228


Support model selection based on materialization type

Describe the feature
Current model selection syntax works with selecting based on model names, sources, tags, packages, and paths. These solve the majority of potential use cases but adding the ability to select models based on a materialization type would be very handy.
A specific example where that can be used: my team's dbt setup currently auto-runs changed models on production when an MR gets merged to master, however we realistically only want to deploy changed views and not to cause incremental materializations for example as those are typically based on a schedule.
I imagine as an actual CLI command this would look something like when implemented:
dbt run --models materialization:view

Describe alternatives you've considered
This same behaviour can be achieved if a user tags all view (or non-view) models but this is more cumbersome so not ideal.
Additional context
This feature should be supported across databases.
Who will this benefit?
Any dbt user who wants to trigger runs based on a materialization type.

hey @azhard - thanks for opening this issue. This is a dupe of https://github.com/fishtown-analytics/dbt/issues/2425, prioritized for our next feature release. Closing this!

Awesome! Thanks @drewbanin


how to override iam_role in child terragrunt.hcl?

Hi,
I've set the iam_role variable in the root terragrunt.hcl:
iam_role = "arn:aws:iam::${local.common.account_id}:role/terragrunt"

I'm trying to clear the iam_role by overriding it in the child terragrunt.hcl, I've tried both iam_role = null and iam_role = "" without success.
Is there a way to clear the iam_role within the child terragrunt.hcl?

Hi,
with current available functionality not sure if iam_role can be cleared in HCL, however, definition can be extracted to a separated HCL file and included if needed
include "iam_role" {
  path = find_in_parent_folders("iam_role.hcl")
}

Thanks @denis256, i'm already using this exact same workaround.


Reduce default privileged plugins

We should not allow privileged to plugins we don't control ourselves.
Also, plugins/docker and plugins/ecr should be replaced by docker-buildx.
Does somebody know whether our buildx plugin has support for GCR?

This is a breaking change.

We should consider to not escate anything by default. Users should be aware of contsiners running in privilrged mode and should only enable it it required and knowing the impact.


MSOutput - unhandled exception in _getDatasetSize()

Impact of the bug
MSOutput
Describe the bug
While trying to determine dataset size if the service for any reason fails to do so (e.g. fails to get DID info from Rucios etc.) we do not handle the exception [1] but just let it reach the upper level pipeline exception handler, while proper actions  should be taken at place. In result we pollute the logs with exceptions, which in many cases could be just predictable/expected behavior.
[1]
https://github.com/dmwm/WMCore/blob/e45a4425c5437d194d0d02814b7011c752c33548/src/python/WMCore/MicroService/MSOutput/MSOutput.py#L696
[2]
2021-03-23 01:14:48,521:ERROR:MSOutput: MSOutputProducer Pipeline Possibly broken read from ReqMgr2 API or other. Err: 'bytes'. Continue to the next document.
Traceback (most recent call last):
  File "/data/srv/HG2103g/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.4.6.pre9/lib/python2.7/site-packages/WMCore/MicroService/Unified/MSOutput.py", line 545, in msOutputProducer
    msPipeline.run(request)
  File "/data/srv/HG2103g/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.4.6.pre9/lib/python2.7/site-packages/Utils/Pipeline.py", line 140, in run
    return reduce(lambda obj, functor: functor(obj), self.funcLine, obj)
  File "/data/srv/HG2103g/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.4.6.pre9/lib/python2.7/site-packages/Utils/Pipeline.py", line 140, in <lambda>
    return reduce(lambda obj, functor: functor(obj), self.funcLine, obj)
  File "/data/srv/HG2103g/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.4.6.pre9/lib/python2.7/site-packages/Utils/Pipeline.py", line 72, in __call__
    return self.run(obj)
  File "/data/srv/HG2103g/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.4.6.pre9/lib/python2.7/site-packages/Utils/Pipeline.py", line 75, in run
    return self.func(obj, *self.args, **self.kwargs)
  File "/data/srv/HG2103g/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.4.6.pre9/lib/python2.7/site-packages/WMCore/MicroService/Unified/MSOutput.py", line 622, in docInfoUpdate
    bytesSize = self._getDatasetSize(dataItem['Dataset'])
  File "/data/srv/HG2103g/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.4.6.pre9/lib/python2.7/site-packages/WMCore/MicroService/Unified/MSOutput.py", line 695, in _getDatasetSize
    return didInfo["bytes"]
KeyError: 'bytes'

How to reproduce it
Just look at the MSOutput  logs
Expected behavior
In case the situation is expected, we should handle the exception at the place where it happens (even if we need to create a new class of exceptions), and propagate only the unexpected ones upstream.
Additional context and error message
N/A

For the record, I just spotted this problem in the MSOutput logs, and I believe it's actually a blocker to moving the workflow forward and eventually archiving it. Fresh logs are:
2021-06-15 10:43:19,828:INFO:MSOutput: Producing MongoDB record for workflow: pdmvserv_RVCMSSW_11_3_0_pre6H125GGgluonfusion_14__2026D76PU200_RECOonly_210419_202554_1748
2021-06-15 10:43:19,841:DEBUG:connectionpool: http://cms-rucio.cern.ch:80 "GET /dids/cms/%2FRelValH125GGgluonfusion_14%2FCMSSW_11_3_0_pre6-PU_113X_mcRun4_realistic_v6_2026D76PU200_RECOonly-v1%2FGEN-SIM-RECO?dynamic=True HTTP/1.1" 404 None
2021-06-15 10:43:19,842:ERROR:Rucio: Data identifier not found in Rucio: /RelValH125GGgluonfusion_14/CMSSW_11_3_0_pre6-PU_113X_mcRun4_realistic_v6_2026D76PU200_RECOonly-v1/GEN-SIM-RECO. Error: Data identifier not found.
Details: Data identifier 'cms:/RelValH125GGgluonfusion_14/CMSSW_11_3_0_pre6-PU_113X_mcRun4_realistic_v6_2026D76PU200_RECOonly-v1/GEN-SIM-RECO' not found
2021-06-15 10:43:19,842:ERROR:MSOutput: MSOutputProducer Pipeline Possibly broken read from ReqMgr2 API or other. Err: 'bytes'. Continue to the next document.
Traceback (most recent call last):
  File "/data/srv/HG2106g/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.4.9.pre8/lib/python2.7/site-packages/WMCore/MicroService/MSOutput/MSOutput.py", line 547, in msOutputProducer
    msPipeline.run(request)
  File "/data/srv/HG2106g/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.4.9.pre8/lib/python2.7/site-packages/Utils/Pipeline.py", line 140, in run
    return reduce(lambda obj, functor: functor(obj), self.funcLine, obj)
  File "/data/srv/HG2106g/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.4.9.pre8/lib/python2.7/site-packages/Utils/Pipeline.py", line 140, in <lambda>
    return reduce(lambda obj, functor: functor(obj), self.funcLine, obj)
  File "/data/srv/HG2106g/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.4.9.pre8/lib/python2.7/site-packages/Utils/Pipeline.py", line 72, in __call__
    return self.run(obj)
  File "/data/srv/HG2106g/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.4.9.pre8/lib/python2.7/site-packages/Utils/Pipeline.py", line 75, in run
    return self.func(obj, *self.args, **self.kwargs)
  File "/data/srv/HG2106g/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.4.9.pre8/lib/python2.7/site-packages/WMCore/MicroService/MSOutput/MSOutput.py", line 624, in docInfoUpdate
    bytesSize = self._getDatasetSize(dataItem['Dataset'])
  File "/data/srv/HG2106g/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.4.9.pre8/lib/python2.7/site-packages/WMCore/MicroService/MSOutput/MSOutput.py", line 697, in _getDatasetSize
    return didInfo["bytes"]
KeyError: 'bytes'

Without giving too much thought, I see no reason to block this workflow because we failed to fetch its size in Rucio. It could actually well be that the workflow is getting announced, even though an output dataset didn't manage to be produced.
IMO, if a workflow is in one of the MSOutput expected statuses ['closed-out', 'announced'], it's then ready to be archived.
PS.: The workflow above is stuck in the system for almost 2 months now, which means, it never gets archived and just contribute to the overall system load.
Bumping the priority of this ticket.

Hi @amaltaro,
This is a strange behavior to me. Why do we not take any action at a point with a well known exception at [1], which is caught in [2] and an empty dict returned. But instead we try to fetch a non existing key from and empty dictionary and let (now a different  and confusing) Exception propagate to the top level general exception handler of the pipeline? This is in my opinion is:

Misleading - because of the Exception metamorphosis that is happening halfway down the road
It is a blocker, because at that  point it is definite the DID in question is missing from Rucio [3] (for an unknown reason), and the record about this workflow will never be created in MongoDB. This one is doomed to never move forward.

[1]
https://github.com/dmwm/WMCore/blob/4d72c731ed444d9c84b8417d2c6a911fa2535b2c/src/python/WMCore/MicroService/MSOutput/MSOutput.py#L696
[2]
https://github.com/dmwm/WMCore/blob/389f54600a739448d2809048a8e7fd20807fa70a/src/python/WMCore/Services/Rucio/Rucio.py#L828-L833
[3]
$ rucio get-metadata cms:/RelValH125GGgluonfusion_14/CMSSW_11_3_0_pre6-PU_113X_mcRun4_realistic_v6_2026D76PU200_RECOonly-v1/GEN-SIM-RECO
2021-06-15 22:03:04,529	ERROR	Data identifier not found.
Details: Data identifier 'cms:/RelValH125GGgluonfusion_14/CMSSW_11_3_0_pre6-PU_113X_mcRun4_realistic_v6_2026D76PU200_RECOonly-v1/GEN-SIM-RECO' not found

Todor, just to make sure I understood you. Are you saying that we should enclose the _getDatasetSize call in a try/except block and handle the KeyError properly, such that that workflow can move forward? If so, then we are on the same page and this is exactly what I meant in my comment above.

Or a variation of that, one could check if didInfo is an empty dictionary, which would - practically speaking - have the same result as in the comment above.

Hi @amaltaro, Thanks!
Indeed, we are on the same page here. Both ways would do in my opinion. How we decide to identify the error, either by the empty dictionary or by handling the already metamorphosed new KeyError exception, it doesn't matter to me. The discussion that we need to have is more on the type of action we decide to take in that situation. So I would suggest, in case we decide to block the workflow until a manual interaction takes place, then we should signal that to someone, instead of letting it stale in the queue until we find it by accident months later. In this case the AlertManager is not a bad choice. The other option should be to do some additional checks (if there are any possible) to see how bad the situation is before we let it pass through or alert.
Of course it all depends also on the frequency at which we get those errors -  the more we have them the more automated action we'd need.

@haozturk Hasan, I think your input here would be quite important.
I assume that, if a workflow is in closed-out or announced, the Unified system has already verified the workflow completion, the output dataset stats, etc etc and things are good enough for the workflow announcement (including relvals?).
So, my question is, do you think we need any sort of alerts for such cases where one or more of the output datasets cannot be found in Rucio? My plan is to just bypass output datasets not inserted into Rucio (Copies=0), make a record in the log, and proceed with standard MSOutput data placement. If such cases are quite rare, we could extend it to also make an alert in the usual dmwm slack channel?
Please let us know what your thoughts are on this. Thanks

@amaltaro

I assume that, if a workflow is in closed-out or announced, the Unified system has already verified the workflow completion, the output dataset stats, etc etc and things are good enough for the workflow announcement (including relvals?).

This is true

So, my question is, do you think we need any sort of alerts for such cases where one or more of the output datasets cannot be found in Rucio? My plan is to just bypass output datasets not inserted into Rucio (Copies=0), make a record in the log, and proceed with standard MSOutput data placement. If such cases are quite rare, we could extend it to also make an alert in the usual dmwm slack channel?

I actually had hard time understanding the issue here. Do you mean an alert for cases where MSOutput fails to make an output data placement for a given dataset?

I actually had hard time understanding the issue here. Do you mean an alert for cases where MSOutput fails to make an output data placement for a given dataset?

MSOutput runs in two modes:

producing workflow entries, thus fetching information from Rucio and deciding how the data placement will be done (where this error happens)
consuming workflow entries, based on the information produced in the thread above.

So, my question is, should we create any alerts for workflows going through the MSOutput service, and failing to find a given output dataset in Rucio? Imagine that a RelVal workflow has 3 datasets to be announced:

blah/GEN-SIM-RECO
blah/DQMIO
blah/AODSIM

supposing that the DQMIO merge job failed, meaning that no DQMIO dataset has been produced - thus making it unknown to Rucio - should we create a notification if the Rucio query get the DQMIO container size is empty (exception raised for "Data identifier not found")?
If it's still not clear, please ping me on slack.

Thanks a lot for the explanation Alan, now it's clear. If a workflow is moved to closed-out, this means that satisfactory statistics have been produced for all outputs. Sometimes, we bypass some workflows if the output is partially produced and we record these cases in JIRA. But, I don't see any possibility to see a workflow in closed-out with its output completely missing from Rucio. So, if the output is completely missing from Rucio, yes, we should create an alert for it. I actually don't remember even one single case, so, I expect this alert should be very rare. I'd prefer email alerts if possible, it's easier to organize them. If not possible, Slack alert is also fine.
Depending on the alerts that we take, I believe we can decide how to proceed in the future (Whether to suppress them or keep them)

That approach looks good to me.
Okay, I am going to make email alerts in case an output dataset cannot be found in Rucio, and let the usual workflow output data placement proceed with the rest of the datasets.
I believe we are going to get tens of emails once we upgrade CMSWEB services in the next Tuesday (likely with workflows that are months old in MSOutput...)

Thanks Alan. Is one alert going to be generated for one missing output? Repetitive alerts really makes things harder to track.

It turns out the fix provided was incomplete! The service still tries to place that output dataset at one tape destination(!)
Fresh logs post CMSWEB upgrade report:
2021-08-03 09:55:12,446:INFO:MSOutput: Creating Rucio TAPE rule for container: /MinBias_TuneCP2_13TeV-pythia8/RunIISummer20UL18MiniAOD-FSUL18_pilot_106X_upgrade2018_realistic_v11_L1v1-v1/MINIAODSIM and RSE: T1_RU_JINR_Tape
2021-08-03 09:55:13,318:DEBUG:connectionpool: http://cms-rucio.cern.ch:80 "POST /rules/ HTTP/1.1" 201 None
2021-08-03 09:55:13,319:INFO:MSOutput: Creating Rucio TAPE rule for container: /MinBias_TuneCP2_13TeV-pythia8/RunIISummer20UL18MiniAODv2-FSUL18_pilot_106X_upgrade2018_realistic_v16_L1v1-v1/MINIAODSIM and RSE: T1_RU_JINR_Tape
2021-08-03 09:55:13,362:DEBUG:connectionpool: http://cms-rucio.cern.ch:80 "POST /rules/ HTTP/1.1" 404 None
2021-08-03 09:55:13,363:ERROR:Rucio: Exception creating rule replica for data: ['/MinBias_TuneCP2_13TeV-pythia8/RunIISummer20UL18MiniAODv2-FSUL18_pilot_106X_upgrade2018_realistic_v16_L1v1-v1/MINIAODSIM']. Error: Data identifier not found.
Details: Data identifier cms:/MinBias_TuneCP2_13TeV-pythia8/RunIISummer20UL18MiniAODv2-FSUL18_pilot_106X_upgrade2018_realistic_v16_L1v1-v1/MINIAODSIM is not valid.
2021-08-03 09:55:13,363:INFO:MSOutput: Tape requests partially successful for: pdmvserv_task_PPD-RunIISpring21UL18FSGSPremix-00002__v1_T_210504_144226_8134. Marking it as 'pending'

My bad, since I only test the producer thread. The consumer thread needs to be tested in a private VM pointing to Rucio Prod... reopening.


LineNumber property

Hi, i'm trying to "bind" two editors together, this just means that the last "line number" of the first editor will become the first linenumber of the second editor, just to allow me to have a "read only" area on the "editor"

should this work? knowing that most of this will run as "interop"

a code very "similiar" will run on the "playground" without issues.
looking at the "create" function used in the JsInterop File, maybe that the function is being set as a "string".
am i doing something wrong?

Currently, C# LineNumbers property is just a string, but it can be a function as well in JavaScript. I'll figure out a way to achieve the same thing in C# and publish a new version of BlazorMonaco this weekend.

Hi @serdarciplak
Any news on this subject?

Sorry. I couldn't find time for this yet. But I see this as a priority item for this repo. I'll solve it asap.

I've added support for this in v2.1.0. Just set the LineNumbersLambda property of the construction options to a lambda and it will be called by the editor when needed.

Hi and thanks for the attention.
I was testing, and when using the the new property LineNumbersLambda when the page loads the components, it throws the following:

My project runs on blazor-side, this functionality should be supported in this model?

I haven't tested it on a server-side app. I'll check it out.

Hi @serdarciplak
Did you manage to take the test?

Any news about this? I'm still seeing it running in server-side mode

I finally had time to have a look at this and I think I have a fix. Here is an explanation of the new flow;

A new JS-side cache is added for the line numbers.
When MonacoEditor requests a line number for the first time, a dotnet call is initiated and an empty string is returned as the line number.
When an async dotnet call completes, it adds the returned value to the JS-side cache.
When all active dotnet calls complete, the line numbers are redrawn. In this redraw, the lineNumbers JS function returns the cached values and Monaco Editor displays these values as the line numbers.
Also, a new editor instance method named ReloadLineNumbers is added. It can be used if the dotnet side needs to reset the JS-side cache and provide new line numbers.

@psantosl It would be great if you can review the PR #147 as you already know the context here.