Add Apache Airflow as supported Pipeline runtime

Items left:

Update instructions to include object storage requirements
Add screenshots for airflow runtime metadata instructions
Update drop down for exports

Items left:

Update instructions to include object storage requirements
Add screenshots for airflow runtime metadata instructions
Update drop down for exports


Begriddled puzzles

Think about adding some Begriddled puzzles, or design something similar.

Would they be more fun if there were more clues to assemble? Sometimes I just get stuck after finding the letters.
Ideas:

Include the total number of lines to draw, so you can check whether you missed any.
Use a grid of letters instead of numbers, then include all pairs of letters from the answer somewhere in the grid.


Starred messages view shows messages in inaccessible streams

If you star a message in a stream and either:

Unsub from the stream.
Deactivate the stream.

The message will stay visible in the Starred messages view even though it shouldn't. E.g. in the screenshot below you can see messages I starred - first one in a deactivated stream, second one in a private stream I unsubscribed from.

@sahil839 Would you be interested in looking into this?

Sure, I will fix this.
@zulipbot claim.

@sahil839 , Are you still working on this? I would like to contribute too.

@sahil839 , Are you still working on this? I would like to contribute too.

Yes I am working on this.

Yes I am working on this.

I am not getting time to work on this so I am abandoning it for now. WIll get back to it when I get time (if this will not be solved till then). Anyone who is taking this can have a look on the discussion here.
@Pranav2612000 you can take this if you want.

I am not getting time to work on this so I am abandoning it for now. WIll get back to it when I get time (if this will not be solved till then). Anyone who is taking this can have a look on the discussion here.
@Pranav2612000 you can take this if you want.

I have gone through the discussions on  Zulip Community , i think i can fix this
@zulipbot claim.

I have gone through the discussions on  Zulip Community , i think i can fix this
@zulipbot claim.

@deto-5420 are you working on this issue.

@deto-5420 are you working on this issue.

@zulipbot claim

@zulipbot claim

Looks like there is some misconversation @m-e-l-u-h-a-n .I am working on it which I mentioned in Czo. Thanks in advance.

Looks like there is some misconversation @m-e-l-u-h-a-n .I am working on it which I mentioned in Czo. Thanks in advance.

Actually I just looked into the thread mentioned by sahil above but couldn't find your message there, sorry if you have started over it(please confirm) I will unclaim it.

Actually I just looked into the thread mentioned by sahil above but couldn't find your message there, sorry if you have started over it(please confirm) I will unclaim it.

Hi @aryanshridhar , I hope everything is going well. I suggest claiming the issue and assigning it to you so it would be clear someone is working on it.

@aryanshridhar are you still working on this, Can I take this up if you are busy?


remove support for overriding installer image

The support for overriding the installer image has been broken for a while. There is little use for it as one can create a custom release image with whatever installer image is desired.

/hold
This build on #738.

/lgtm

/hold cancel
#738 has merged.


Update pry-byebug and awesome-print

Updates pry-byebug and awesome print to the latest versions.
Current version fails on Ruby 2.2.0:

Tested on Ruby 2.2.0

Hey @nanoxd ! Thank you for submitting this pr and sorry for the delay.
I guess this change would make it impossible to use this code with ruby 2.0.0 and 2.1.1, right?

@nanoxd I just checked at rubygems.org and confirmed that both would work with any ruby version >= 2.0.0
Thank you!

@nanoxd Just did a new release with your changes. Try it out please. Version is 0.5.5.

@plribeiro3000 Sorry about the lack of communication. Thanks for merging!

Sure. No problem. =)


Replace #570. VQE factories.

Summary
This pull request replaces #570, which in turn was trying to fix issue #384.
Details and Comments
I have modified the getters and setters from vqe factories in order for them to access the vqe properties instead of modifying their own attributes. The reason is that if we want to use the deprecated setters they need to modify the attributes from vqe. With the current implementation we were able to set different values to vqe_ucc_factory.ATTRIBUTE and vqe_ucc_factory._vqe.ATTRIBUTE.
I have also implemented some unittests to verify that the setters and getters behave correctly.
I am not sure what would be the best course of action for the ansatz, since it is still an attribute from the factory and the vqe. I don't see a reason why we wouldn't depracate it as well, but for now its setter sets both attributes to the same value.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.2 out of 3 committers have signed the CLA.:white_check_mark: iuliazidaru:white_check_mark: manoelmarques:x: MarcDrudisYou have signed the CLA already but the status is still pending? Let us recheck it.

I just wanted to add a comment to say that I have changed all the unittests that were using a UVCC Factory because they were using quantum_instance positionally, and now it can only be used as a keyword argument.
This change was necessary and from now on all uses of that factory will have to take this into account.

using quantum_instance positionally, and now it can only be used as a keyword argument.

Can we not keep these arguments but deprecate them so that existing code continues to work. This is what we would normally do such that a user has time to change their code over to whatever the new way would be. We never enforced the arguments to be used only as keywords.

Can we not keep these arguments but deprecate them so that existing code continues to work

We actually outright remove these arguments in this PR. You can still provide them via the factories __init__ method but they will all be gathered inside of kwargs. In that way all VQE.__init__ arguments (which are not explicitly handled by the factory, like ansatz, initial_state and initial_point) are now "automagically" in-sync with the VQE.

The thing is that the removal of these - even though they can still be supplied via kwargs, breaks existing code that is using them positionally - as witnessed by changes to notebooks, unit tests etc. Maybe we can make quantum_instance Optional and default it to None, same for include_custom i.e change to None from False (False is VQE's default for that anyway). Can we not find a transition path where existing code, that may have used these positionally as we ourselves did, continues to work albeit that is raises deprecation warnings when it detects them being used (i.e. non-None values). We would want to change the notebook to avoid the warnings, but the notebook (and other code users may have done) ought not be broken.

The thing is that the removal of these - even though they can still be supplied via kwargs, breaks existing code that is using them positionally - as witnessed by changes to notebooks, unit tests etc. Maybe we can make quantum_instance Optional and default it to None, same for include_custom i.e change to None from False (False is VQE's default for that anyway). Can we not find a transition path where existing code, that may have used these positionally as we ourselves did, continues to work albeit that is raises deprecation warnings when it detects them being used (i.e. non-None values). We would want to change the notebook to avoid the warnings, but the notebook (and other code users may have done) ought not be broken.

We could keep all of the arguments we had previously and raise deprecation warnings, or we could check for the type of ansatz (and if it is a QuantumInstance pass it to the VQE as a QuantumInstance and raise a warning).

I tried using type checks to find an alternative that fulfills both requirements (would be usable by keyword arguments or positionally). However, one of the many issues this aproach has is that inputs such as VQEFactory(quantum_instance, ansatz=ansatz) wouldn't work (assuming that ansatz is the first positional argument in the constructor).
The only reasonable solution I can come up with is to leave the arguments of the constructor exactly as they were, but if some of them has to be deprecated, to raise a deprecation error.
However, as @mrossinek pointed out, until we finally remove it from the code, every time we make a change to VQE, we will need to modify this class as well.

The only reasonable solution I can come up with is to leave the arguments of the constructor exactly as they were, but if some of them has to be deprecated, to raise a deprecation error.

Yes, in order to allow existing code to work then this should be the route we take unless its impossible and a breaking change has to be done.

until we finally remove it from the code, every time we make a change to VQE, we will need to modify this class as well.

VQE itself should only be making changes the same way, i.e. deprecating as necessary. We can choose to only support any new function via the new way. So hopefully this does not happen in practice. But yes, the main idea of discontinuing the way things were done was to avoid this going forwards, especially as we expect changes around the algorithms as they are adapted over to use primitives.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

137 of 156 (87.82%)  changed or added relevant lines in 3 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.07%) to 83.513%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

qiskit_nature/algorithms/ground_state_solvers/minimum_eigensolver_factories/numpy_minimum_eigensolver_factory.py
14
15
93.33%

qiskit_nature/algorithms/ground_state_solvers/minimum_eigensolver_factories/vqe_ucc_factory.py
61
69
88.41%

qiskit_nature/algorithms/ground_state_solvers/minimum_eigensolver_factories/vqe_uvcc_factory.py
62
72
86.11%

Totals

Change from base Build<PHONE_NUMBER>:
-0.07%

Covered Lines:
9234

Relevant Lines:
11057

💛  - Coveralls

Do not merge the branch yet, since I still need to write the deprecation warnings for the case where we use the arguments positionally.

The way I dealt with the constructor is that  I kept the same order of arguments but I have changed the name of the positional arguments we want to deprecate to **_deprecate. This way if the user was using them as keyword arguments, they will still be passed to vqe as kwargs. Otherwise, if used positionally, they will raise a deprecation warning and update kwargs[] = **_deprecated.
I believe this makes it clear for the user that he should not be using them positionally, we won't have to make any changes to the factory even if we make changes to vqe, and the old code will keep working.

Okay, what I ended up doing is to create a decorator in deprecate.py that takes care of the particular problem we have.
If you try a script that looks something like this you should see that it works but raises deprecation warnings:
from qiskit_nature.algorithms.ground_state_solvers.minimum_eigensolver_factories import VQEUCCFactory
from qiskit.utils import QuantumInstance
from qiskit import Aer
import numpy as np

quantum_instance = QuantumInstance(backend=Aer.get_backend("aer_simulator_statevector"))

VQEUCCFactory(quantum_instance, None, np.array([1,2,3]), include_custom=True)
#This raises warnings for quantum_instance, optimizer (which is the second postitional argument) but not for initial_point (since it is not being deprecated), nor for include_custom (because it has been passed as a keyword argument).

I believe this implementation is much more clean and easy to modify.

@woodsp-ibm The CI is failing with a rather cryptic error message: https://github.com/Qiskit/qiskit-nature/runs/6402562094?check_suite_focus=true#step:8:802
Do you know how to resolve this? I have not seen this one before.

The CI is failing with a rather cryptic error message:
Do you know how to resolve this? I have not seen this one before.

It seems that a print() statement was left in the factory. This generated an empty output cell (just a new line in it) that seems to end up causing the error that was seem around raw. Removing it and it works fine - the notebooks as checked in built fine it was only when the html is built from the re-run version that the issue arose. I pushed a changed to remove the print so it should go through now ok.
In regards of the two notebooks, 03 and 04, they should be updated as they emit deprecation warnings. They still work which is of course the intent of the deprecation, but they should be switched over to the new way i.e. keyword versus positional argument, so they show the right way of doing things now.

The CI is failing with a rather cryptic error message:
Do you know how to resolve this? I have not seen this one before.

It seems that a print() statement was left in the factory. This generated an empty output cell (just a new line in it) that seems to end up causing the error that was seem around raw. Removing it and it works fine - the notebooks as checked in built fine it was only when the html is built from the re-run version that the issue arose. I pushed a changed to remove the print so it should go through now ok.
In regards of the two notebooks, 03 and 04, they should be updated as they emit deprecation warnings. They still work which is of course the intent of the deprecation, but they should be switched over to the new way i.e. keyword versus positional argument, so they show the right way of doing things now.

Thank you for fixing the CI! I fixed the tutorials. However, I am wondering, is there a good way of seeing that the tutorials are throwing deprecation warnings? I have to check manually but I am sure there is an easier way.

However, I am wondering, is there a good way of seeing that the tutorials are throwing deprecation warnings? I have to check manually but I am sure there is an easier way.

I do not believe we gather these in CI at all - @manoelmarques rather they just end up in the notebook output when run which ends up in the html. Now we do capture deprecations warnings from the logs when running tests and CI gathers these up and  you can see this in build details section. For any tests we run, where we know we are testing deprecated code, its our policy to suppress the warning for that test knowing it will occur. The intent is to only have messages come out from places we cannot avoid, or places that we have to change in the code. At present the factory testing seems to be emitting a large number of these that need to be supressed - seems that a number of warnings from various places are in the output. Some it maybe code that needs to be updated, some may be from tests that we know will do this and we can therefore ignore them in the test. That deprecations file ought to be checked over before the next release and appropriate remedies enacted in the code.

Hello, I have been taking care of deprecation warnings. One of them brought me an interesting problem.
The warning in question happened in GroundStateEigensolver.evaluate_operators(). Here we were trying to get some attributes of the solver:
` # try to get a QuantumInstance from the solver
quantum_instance = getattr(self._solver.minimum_eigensolver, "quantum_instance", None)
and try to get an Expectation from the solver
expectation = getattr(self._solver.minimum_eigensolver, "expectation", None)Note that self._solver isUnion[MinimumEigensolver, MinimumEigensolverFactory]. Before, this was okay since every subclass of MinimumEigensolverFactory happenned to have quantum_instance and expectation (although we could have happened to create some new factory that didn't need expectationand we would have run into an error here). In that sense, we could use MinimumEigensolver instances or their factories interchangebly and we were also avoiding dealing with mypy. However, now we need to make the difference between these two cases, i.e.:if isinstance(self._solver, MinimumEigensolverFactory):
try to get a QuantumInstance from the solver
quantum_instance = getattr(self._solver.minimum_eigensolver, "quantum_instance", None)
and try to get an Expectation from the solver
expectation = getattr(self._solver.minimum_eigensolver, "expectation", None)
else:
quantum_instance = getattr(self._solver, "quantum_instance", None)
expectation = getattr(self._solver, "expectation", None)`
This code would work every time since all three instances of MinimumEigensolverFactory {NumpyMinimumEigensolverFactory,VQEUCCFactory,VQEUVCCFactory} happen to have a minimum_eigensolver() method with some subclass of MiniumEigensolver as an output.
However, this creates a new problem since now we are calling MinmumEigensolverFactory.minimum_eigensolver(), which doesn't exist a priori.
Is there a reason against adding a getter for minimum_eigensolver() to the base class MinimumEigensolverFactory? It would look like this:
@abstractproperty    @property def minimum_eigensolver(self) -> MinimumEigensolver:     """Returns the solver instance."""     raise NotImplementedError
Thank you for your time.

The base-class MinimumEigensolverFactory should indeed already specify the minimum_eigensolver attribute. You can go ahead and add that :+1:

@mrossinek Comment on linking this PR to close that issue https://github.com/Qiskit/qiskit-nature/issues/426#issuecomment-1143802032

I wanted to leave a comment saying that since the two classes are almost a copy of one another it can get hard to make changes in both of them. If the classes need to be changed again, specially if that is done before finally removing the deprecated properties, it might be worth it to look into making a superclass that these two can inherit from. In the end the only difference they have is in the get_solver() method.


[BUG] Network setup error in 0.3.0-preview1

Describe the bug
The network interface chosen during the installer is not present when booting into the system after the installation.
To Reproduce
Steps to reproduce the behavior:

Go to through the ISO installer
When asked to choose the management NIC (in a 4 NIC system), only 1 shows as active. In my case eth2 on Del R620's, while the first SFP+ port is connected.
Choose the active NIC so you can continue
Reboot after finishing the installer
On reboot, no interface is up, and interface are name eno* rather than eth*

Expected behavior
The network interface chosen in the installer should be named and numbered the same as when the system is installed.
Environment:

Harvester ISO version: 0.3.0-preview1
Underlying Infrastructure: Baremetal with Dell PowerEdge R620

This might be related to https://github.com/harvester/harvester/issues/1200#issuecomment-917160616, although I tried installing both using the ISO written to a USB as well as the ISO directly.

Just in case anybody else runs into this, when starting the installer ISO, press Esc, then e to edit the kernel parameters and add net.ifnames=1 to the end of the line. Then press enter to continue the boot process. Given that net.ifnames=1 is enabled after the installation, the installer should probably also have this kernel parameter set.

Thank you so much for posting this. I was encountering the same issue and your comment helped me fix it.


(Fixes #236) Add support for exalead

Fixes #236
Checklist

[x] I have read the Contribution & Best practices Guide and my PR follows them.
[x] My branch is up-to-date with the Upstream master branch.
[ ] I have added necessary documentation (if appropriate)

Changes proposed in this pull request:

Added support for Exalead

Reading code in the GitHub web UI is a common practice.  Code wider than 127 characters requires scrolling left and right in that UI which is highly sub-optimal for the reader.
I would rather see you wrap these lines or that you tell flake8 to excludes the file app/templates/index.html or the directory app/templates/  See the flake8 docs on how to do this.

tell flake8 to excludes the file app/templates/index.html or the directory app/templates/ See the flake8 docs on how to do this.

Flake8 is used for python files only and not for HTML files or other files and this build is failing due to this and discussion is also for the mentioned line here
Thanks for review.

Further rebase will make the previous 2 upstream commits into mine. Hence no need to rebase further.
@cclauss @gabru-md please review.

Further rebase will make the previous 2 upstream commits into mine. Hence no need to rebase further.
@cclauss @gabru-md please review. Example video at Youtube

And please don't mix so many into a single PR/issue. Ideally, each of the search engine implementation should be a separate issue-PR pair so that implementation/requirements can be discussed.

I will do this from next time.
@niranjan94 please review :)

@bhaveshAn can we have a test deployment please ? :)

@niranjan94 can i use ngrok ?

@bhaveshAn ngrok would require you to have a local instance up and running. Hence it isn't always reliable as many times we'll end up with dead links and late reviews. Please use heroku. Easy and free.

Did but can you tell me what to do for bower ?

@niranjan94 For now it looks https://ancient-wave-72336.herokuapp.com/
can you help me? please

@bhaveshAn how did you deploy to heroku ? Bower should automatically get downloaded as long as the nodejs build pack is present.

actually i ran ``heroku buildpacks:set heroku/python``` and then push

https://github.com/fossasia/query-server just use the one-click deployment button on README

@bhaveshAn how did you deploy to heroku ? Bower should automatically get downloaded as long as the nodejs build pack is present.

After git push heroku bhavesh-yandex:master it didn't installed bower

Hey @bhaveshAn, You also need to have the node buildpack.

Did that alongwith python buildpack. Didn' succeed.
@AnshulMalik Can you please review the PR.

@gabru-md @S2606 please review.
Youtube Video

@bhaveshAn please provide a deployment here

@niranjan94
https://shrouded-reef-37326.herokuapp.com/ This heroku deployment is working for Exalead and Teoma and not for Lycos.
While all the three running locally and at http://0ebf3762.ngrok.io/

Please debug the heroku deployment and fix lycos. We need everything to
work on heroku..
On Sun, Oct 15, 2017, 8:47 PM Bhavesh Anand<EMAIL_ADDRESS>wrote:

@niranjan94 https://github.com/niranjan94
https://shrouded-reef-37326.herokuapp.com/ This heroku deployment is
working for Exalead and Teoma and not for Lycos.
While all the three running locally and at http://0ebf3762.ngrok.io/
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/fossasia/query-server/pull/240#issuecomment-336773396,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACSwFO6nHAlnYOSY31fVQwydpL-1OCJzks5sstHYgaJpZM4PxpyM
.

@niranjan94 Sure. I will let you know after debugging/fixing it.
Apologies, Did down thumbs emoji by mistake in previous comment. Hope you don't mind :disappointed:

As Lycos wasn't able to run on heroku, so removing it. Will create separate issue and PR.
Heroku deployment for Exalead and Teoma is at https://limitless-mesa-39171.herokuapp.com/
@niranjan94 @mariobehling Please review. Thanks !

Hey @bhaveshAn , I tried to list 40 results of teoma, but looks like they were duplicates, can you look into it please?

Please take out Teoma as its results come from Ask.com. Also, resolve merge conflicts. Cheers.

Please take out your Teoma references and rebase. Cheers.

@mariobehling Please review. Removed Teoma from the files and squashed the commits. Thanks !

@mariobehling Please review. Removed Teoma from the files and squashed the commits. Thanks !
@niranjan94 please review . Providing the heroku deployment at https://powerful-shelf-77291.herokuapp.com/

Exalead showing all the 100 results.
@vaibhavsingh97 @cclauss @shashank-sharma please review. Providing the heroku deployment at https://powerful-shelf-77291.herokuapp.com/


Monitoring for Lives without getting blocked

Hi there,
Just wondering if anyone has figured out a way to monitor for Lives without getting the 'suspicious activity' and subsequent 'change your password' prompts from Instagram?
If not, I've thought of a potential workaround, but I'm not sure how to implement it exactly. Basically, you have a spare Android phone (or PC running Bluestacks) with the Instagram app installed, then when a live notification comes in from the Instagram app, an app forwards that notification on to the PC where it'll trigger PyInstaLive to run. Anyone have any ideas how to make this work?

@zylopr
On the android phone you can install "Macdroid" free version (alternatively, "Automate" free version also works, as does "Tasker," but that one costs money). Any of those apps can be configured to detect Instagram notifications and, in response, trigger custom HTTP get requests to a web server running on your PC -- passing along the notification text. Your server can parse the notification text to see if it is a live video, and who the user is, and make the appropriate call to PyInstaLive. This would of course require you to install a web server on your computer. There might be easier ways but they don't come immediately to mind. A good old fashioned apache/php setup would work. For Windows there are bundled softwares like XAMPP or MAMP that can set this up for you.
For example:
In the Macdroid app, on your android, you would create a new macro (consisting of a trigger and an action). Set the trigger to "Notification received" from Instagram App (you can optionally set a text filter in this step, but it might be easier for your web server to do it.
For the action choose "HTTP GET" and set the URL to https://your.server.ip.address:port/whatever.php?title=[not_title]&text=[notification]&textlines=[not_text_lines]&tickertext=[not_ticker]
"not_title" is short for notification title, etc.. You might not need all those variables, but Instagram might change which ones serve the "Bob is starting a live video" or "Bob is going live now with Sarah" texts. Of course you would replace parts of the address to match your server address, but the variables should be typed just like they appear, with brackets.
Make sure to check the boxes "url encode parameters" and "http get (no web browser)". You can optionally have it check for an http response, but you don't have to worry about that if you don't want to.
Then, in your server's PHP page (or whatever other server language is most familiar to you) you would regex check the $_GET["text"], $_GET["textlines"], etc, variables, for a pattern matching the usual live notification, for example on a windows server you could do something like this:
$text = urldecode($_GET["text"]);

// i.e. "Zylopr: Bob started a live video"
// i.e. "Zylopr: Bob is going live with...."
if(preg_match('/\((.+)\): (.+) started a live video/U', $text, $matches) ||
   preg_match('/\((.+)\): (.+) is going live/U', $text, $matches))
{
  $yourAccountName = $matches[1]; // if you have multiple accounts
  $yourPassword = "yourpassforthataccount";
  $theirAccountName = $matches[2];

  // starts pyinstalive in a visible window on your PC and immediately
  // returns so we don't leave macdroid hanging
  pclose(popen("start \"customwindowtitle\" cmd /C pyinstalive -u $yourAccountName -p $yourPassword -d $theirAccountName" ,"r"));
}

This should get you started. Once you achieve basic functionality you can flesh out your program to do more, like deal with conflicting lives (I don't know if pyinstalive can run multiple instances, and even if it could it might look suspicious if you ran simultaneous downloads. You can create a list of users sorted by priority, and in the case of a conflict kill an existing download for a lower priority user.
Let me know how it goes. Good luck.

Thanks for that very detailed response, coder39248583. I had actually just managed to get it working earlier using a combination of Tasker, AutoRemote and EventGhost, which together, achieves the task without needing any coding. Tasker also lets me filter the notifications that get sent to the PC based on the notification's text.
PyInstaLive can indeed handle simultaneous downloads. From what I've seen, the actual downloading of the data doesn't need authentication (only to get the URL in the first place). The Python script that was posted in this forum recently that finds the segments of the Live prior to starting PyInstaLive doesn't need any user credentials at all to work, so it'd probably be OK to have several downloads going. Now the only thing that's missing is for someone to integrate that Python script into PyInstaLive.
Thanks again.

@zylopr
Glad you got it working!
I took a look at the script to download previous (missed) live segments and it has the potential to make many thousands of bad requests to Instagram's servers, and miss portions of video, so I would keep that in mind.
Individual live segments are numbered (18700.m4v, 18902.m4v, 19103.m4v...), usually incrementing 2001 or 2002 each segment. The script in question guesses the URLs of the missing segments, and when it finds one, it jumps ahead 2000 and increments its guesses by 1 until it finds the next. So in the best case scenario it would only make a few bad requests to Instagram's servers for each segment downloaded. The problem is that sometimes Instagram increments the segments slightly less than 2000, so jumping ahead 2000 might move you slightly past the correct segment, not only missing the download of that segment, but you are now incrementing by 1 and making ~2000 bad requests until the next find.

@zylopr
Also, I just discovered that pyinstalive often cuts recordings short by an indeterminate number of segments. (Some of?) those missed segments can be found in the dash manifest, the url of which is located in the the downloads json.
Given that this repository is inactive, relies on an inactive Instagram api, and misses available portions of video (both at the beginning and end of a live), it might be a good idea to fork it and revamp. I nominate you, sir.

Thanks for the replies, coder39248583.
Perhaps a solution to the script skipping a segment when it's just under a 2000 increment is to change it to something like "i += 1998"? It'd be a few more requests overall, but with a lower chance of missing a segment and also avoiding ~2000 bad requests when the increment is actually under 2000.
I think I have had instances where it's missed segments, because it sits there for a long period of time without any new segments being found. Despite this, it doesn't seem like it's caused any issues with Instagram blocking my IP address.
As for forking and revamping it, as much as I'd like to, unfortunately my programming skills amount to around about 0.

@zylopr
The jumps for that given live were 1968, 1969, 2001, and 2002. What an awkward music collection. Jump sizes in other lives I checked ranged from 33 to 37367! Marty!!
So bad requests would be impossible to avoid using that method, to say the least.
However! I just realized something. These jump sizes refer to video segment lengths (in milliseconds). So checking the video length of the last segment downloaded will tell us how much to add to the number to receive the next video.
We know the first video is "...-init.m4v. I don't know if it has a length, or how to find the first URL with a number. I can do some experimenting. But this method has the potential of 0 bad requests and 0 missed segments.
I'll need the JSON or the included dash manifest URL of a recent live where the segment links are still active to investigate further.

Here's a dash manifest URL: https://scontent.cdninstagram.com/hvideo-vll3/_nc_cat-100/v/rQBp4OuZBxyzlKzJc5oLn/_nc_ohc-U3FctwEyN7YAX_R1B-4/live-dash/dash-hd/17930152723586011.mpd?_nc_rl=AfBRlvciQ9Cy1GqoiqE&ccb=2-4&ms=m_CN&oh=52ccd49dc2aceee5a78f36d05fc7debf&oe=60C5A24B
Is that what you're after?

@zylopr
Glad you got it working!
I took a look at the script to download previous (missed) live segments and it has the potential to make many thousands of bad requests to Instagram's servers, and miss portions of video, so I would keep that in mind.
Individual live segments are numbered (18700.m4v, 18902.m4v, 19103.m4v...), usually incrementing 2001 or 2002 each segment. The script in question guesses the URLs of the missing segments, and when it finds one, it jumps ahead 2000 and increments its guesses by 1 until it finds the next. So in the best case scenario it would only make a few bad requests to Instagram's servers for each segment downloaded. The problem is that sometimes Instagram increments the segments slightly less than 2000, so jumping ahead 2000 might move you slightly past the correct segment, not only missing the download of that segment, but you are now incrementing by 1 and making ~2000 bad requests until the next find.

Well I have been written another using aiohttp to find all segements by brute force. But it would be terrible if you have missed thousands of segements.

@zylopr @cykac04
I have taken this discussion to the relevant issue..
Thanks for the manifest @zylopr


Fixed validation for non-lowercase header parameters

Names of the parameters were normalized, but names expected by
the validator were not. This lead to failed validation, when OAS was
specifying name of the header parameter not in lowercase.

Nice! Thanks :)


fix: update package.json to fix Typescript error TS7016

This update fixes #110.

Fixed in https://github.com/single-spa/single-spa-vue/releases/tag/v3.0.0


flyTo with Rectangle doesn't play well with terrain

The Camera.flyTo({ destination: Rectangle }) function flies the camera so that the specified Rectangle is in view.
Rectangles are lat/long based objects without height information, and the flyTo function always seems to use a height of 0. If, when using terrain, you call flyTo with a small rectangle in a high-altitude area, the camera will be placed beneath the terrain, and the user sees astronomy instead of the intended area.
The other fly functions aren't able to handle this scenario well either:

Camera.flyTo({ destination: Cartesian3 }) flies the camera itself to a specific location. Using this to fly to a rectangle would require a lot of precomputation, duplicating logic already in Cesium.
Camera.flyToBoundingSphere with BoundingSphere.fromCornerPoints. This isn't convenient if you want to view a rectangular area on a rectangular viewport. The area will indeed appear in view, but it won't fill the screen.
Viewer.flyTo. This assumes you have an Entity, DataSource etc that you want to zoom to (maybe I want to zoom to an arbitrary area), and also that you are using Viewer (maybe I am using CesiumWidget).

I can think of three potential solutions to this problem, from most to least general:

Extend Camera.flyTo to fly to OrientedBoundingBox objects.
Add a height parameter to Camera.flyTo, allowing the Rectangle to be given a single flat height. This would require minimal code change, mainly to the 0.0 values here.
Add a toTerrain boolean option to Camera.flyTo, where Cesium samples appropriate terrain heights and zooms to the given Rectangle on the terrain. This is admittedly a bit less general, but would still likely be useful to many consumers.

I think my preferred approach here would be to add a height parameter to Camera.flyTo since it's the simplest but also gives the developer the most flexibility. For example, this would allow you to easily do the sample terrain heights before flying (do you sample the center, the 4 corners of the rectangle? Up to you) and you'd probably want the height to be not directly at that terrain height, but some offset above that.
Curious to hear what others think.

We already have a private function computeFlyToLocationForRectangle which takes a rectangle and scene and samples the terrain and produces a final Cartographic position that can then be passed to the camera. It's used by both the Viewer for zooming to imagery layers and the Geocoder for zooming to rectangular results.
The easy way out would just be to expose this function, since it does exactly what you want for rectangles.
Longer term, I think we definitely need to revisit the Camera API, but I think there are lots of things we need to think about in this regard.


IdentityManageUI is creating two tables in the DB that perform the same thing

### THANK YOU FOR CREATING THIS TEMPLATE!!! It has been a BIG HELP!!!
I am hoping you can help me solve this issue. I am trying to determine where IdentityManagerUI is causing this issue. I believe the problem is in the ApplicationRoles.cs definition within the ApplicationRole identity with the public virtual. But, I can't be positive.
I am using the ApplicationDbContext.cs to change the names of the Tables in the Database. One of the tables, "ASPNETRoles" is supposed to be named "Roles".
When I run the Package Manager Commands (add-migration, update-database), the commands are creating both Tables.
Once I run my application, ASPNETRoles has the actual Role Data in it. The other table (Roles) only has "Id" column in it.
I am including screenshots of the tables, the migration file, ApplicationDbContext.cs, Program.cs, and ApplicationRole.cs.
------ Files are Listed below ------

20220718032147_after IdentityManagerUI Fixes.cs.txt
ApplicationDbContext.cs.txt
Program.cs.txt
ApplicationRole.cs.txt

Could you send me a redacted version of the project, or a new minimum
project showing the issue so I can replicate and test it on my end?
@.***
On Mon, Jul 18, 2022, 1:55 PM Richard Cross @.***>
wrote:

I am hoping you can help me solve this issue. I am trying to determine
where IdentityManagerUI is causing this issue. I believe the problem is in
the ApplicationRoles.cs definition within the ApplicationRole identity with
the public virtual. But, I can't be positive.
I am using the ApplicationDbContext.cs to change the names of the Tables
in the Database. One of the tables, "ASPNETRoles" is supposed to be named
"Roles".
When I run the Package Manager Commands (add-migration, update-database),
the commands are creating both Tables.
Once I run my application, ASPNETRoles has the actual Role Data in it. The
other table (Roles) only has "Id" column in it.
I am including screenshots of the tables, the migration file,
ApplicationDbContext.cs, Program.cs, and ApplicationRole.cs.
------ Files are Listed below ------
[image: Identity-ASPNETRoles-screenshot]
https://user-images.githubusercontent.com/4992309/179442321-1d6a1847-a40d-4c41-9053-4678b5b308f2.png
[image: Identity-Roles-screenshot]
https://user-images.githubusercontent.com/4992309/179442360-d432f898-ef6a-434a-a7b2-3e8b77914e2d.png
20220718032147_after IdentityManagerUI Fixes.cs.txt
https://github.com/GSGBen/IdentityManagerUI-NetCore6-Template/files/9129350/20220718032147_after.IdentityManagerUI.Fixes.cs.txt
ApplicationDbContext.cs.txt
https://github.com/GSGBen/IdentityManagerUI-NetCore6-Template/files/9129351/ApplicationDbContext.cs.txt
Program.cs.txt
https://github.com/GSGBen/IdentityManagerUI-NetCore6-Template/files/9129353/Program.cs.txt
ApplicationRole.cs.txt
https://github.com/GSGBen/IdentityManagerUI-NetCore6-Template/files/9129354/ApplicationRole.cs.txt
—
Reply to this email directly, view it on GitHub
https://github.com/GSGBen/IdentityManagerUI-NetCore6-Template/issues/1,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADX4AQOHEXIMBUNFH2G4AQTVUTITTANCNFSM5323KAIA
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

Whoops, didn't see the redaction. ben [at] goldensyrupgames.com


Add Azure asr provider

Add the api_call logic for Azure asr provider

Doesn't seem to be straightforward: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/rest-speech-to-text
This service requires one to spin up an instance with the STT service...


Support conditionals and loops in dynamic views

Nowadays dynamic views support parallel sequences.
It would be great if they also support conditionals and loops, so we can communicate those type of dynamic behavior accordingly.
Given this example:
workspace {

    model {
        a = softwareSystem "A"
        b = softwareSystem "B"
        c = softwareSystem "C"
        d = softwareSystem "D"
        e = softwareSystem "E"
    }

    views {

        dynamic * {
            a -> b "Get data (id)"

            b -> c "Get Core data (id)"
            c -> b "Core data"

            b -> d "Get Additional data (id)"
            d -> b "Additional data"

            b -> e "Get Part data (part id)"
            e -> b "Part data"

            b -> b "Aggregate Data"
            b -> a "Data"
            autoLayout
        }
    }

}

Let’s suppose getting Additional Data is only required under some condition in Core data (conditional), and Core data include a list of Parts, which requires getting Part data for each part separately (Looping)
Nowadays there is no way to communicate that behavior as part of the dynamic view.
I believe a Communication Diagram generated from that dynamic view could support representing it based on Guards on messages. On the other hand, PlantUML (and probably others) sequence diagrams support those concepts by “opt”, “alt/else” and “loop” messaging blocks.

The C4 model and Structurizr tooling are designed to be simple, and cater for "a majority" audience. What you're asking for takes you across that line, and into the realms of UML. If you need such features, I think you may have outgrown the tooling, and you're likely better off using UML directly given its much richer semantics.

Thanks Simon.
I do see the benefits of modeling that behavior with Structurizr, as it would let me separate it from the diagramming tool used to represent it. But I also fully understand you reject those features if they are out of scope of C4 model and Structurizr.
I think I got confused because I found parallel sequences in scope and, then, I didn't think conditions/loops are out of scope. What are the key differences between them to let first be accepted while second are not?

The parallel sequence support really does nothing more than reuse the same sequence numbers, and I added it to provide a simple way to show parallel activities with the Structurizr cloud service/on-premises installation/Lite. I did look at making this more sophisticated (e.g. supporting nested hierarchical numbering, like UML does), but the UML specification for this is quite long and not particularly straightforward. So what we have at the moment is a "it's enough for most people" approach.
Translating dynamic views to PlantUML/WebSequenceDiagrams/etc diagram requires a stricter set of semantics. It also requires more information than what's captured in the current version of the Structurizr workspace to do this, because the workspace doesn't include any information about these parallel sequences. If you look at the JSON for the dynamic views, you'll see that each relationship has a sequence number (named order), but there's nothing in there that describes the grouping of relationships into parallel sequences. All of this is possible, of course, but it'll require a significant amount of work, and the end-result will essentially be a re-implementation of the semantics that URL already has. Add on conditionals, guards, etc and you're essentially into UML territory. If this level of precise semantics are required by people, I'd recommend dropping Structurizr and adopting a fully fledged UML tool.

I'm not dropping Structurizr to adopt UML, but the other way around. No intention at all to move back to UML.
But there are some basic needs on dynamic behavior I need to reflect on architecture diagrams to ensure architecture is communicated properly. If they are not supported on C4 Model / Structurizr then I will explore the option to use directly PlantUML for dynamic views. My first thought is I will miss some benefits of using Structurizr:

Double check relationships in the sequence diagram are already defined in the model.
Generating headers with elements automatically based on the elements interacting on that sequence
Using element ids when defining the relationships

If you have any advice, I would appreciate it :) . Thanks!

Thank you!

Hi Simon. Question about approach 2:
As you pointed to, and also documented in the DSL language reference, we can only use the identifiers in the DSL-based workspace or in another DSL-based workspace extending the original one. Once a DSL-based workspace is parsed, those identifiers are gone and cannot be used anymore to reference to the elements.
Is it a decision part of the design? Or maybe a current limitation that could be fixed?

Correct, the DSL identifiers are only used while the DSL is being parsed, and they're not retained in the JSON version of the workspace. I have no plans to change this.

Is there any possible workaround? Can I create a script or plugin to, for example, add a custom property on the element to keep track on the identifier used in DSL?

Unfortunately not. And things like implied relationships are not assigned identifiers during DSL processing, so there would likely be some identifiers missing too.

What if StructurizrDslParser allows getting access to the identifier register after parsing?
https://github.com/structurizr/dsl/blob/master/src/main/java/com/structurizr/dsl/StructurizrDslParser.java#L34
I believe it would let me use same DSL identifiers in PlantUML code, as I would be able to get elements in the model with those identifiers to peform required actions to validate against the model and properly render elements.
What do you think?

Sure, I can add a public getter for the identifiers register.

Great! Thank you!


Invariant Violation: Callback with id 55: PhotoManager.getNetImageBase64() not found

i write a native method for get the network image's base64 value
in program, i need execute much times
native method is:
@objc func getNetImageBase64(param: String,callback: RCTResponseSenderBlock) {     responseCallback = callback;     Just.get(param) { (r) in       if (r.ok && r.statusCode == 200) {         if let _ = r.content {           self.responseCallback([AppUtil.createSuccessResponseCallback(r.content!.base64EncodedStringWithOptions(.Encoding64CharacterLineLength))]);         } else {           self.responseCallback([AppUtil.createFailResponseCallback("content is null")]);         }       } else {         self.responseCallback([AppUtil.createFailResponseCallback("fail, statusCode=\(r.statusCode), message=\(r.error)")]);       }     }   }
javascript code is:
PhotoManager.getNetImageBase64('http://image.com/image', ((rr) => {console.log(rr);}));                     PhotoManager.getNetImageBase64('http://image.com/image', ((rr) => {console.log(rr);}));                     PhotoManager.getNetImageBase64('http://image.com/image', ((rr) => {console.log(rr);}));
in chrome debug console, it tell me error:
Invariant Violation: Callback with id 55: PhotoManager.getNetImageBase64() not found
i have execute the method three times, but it only execute one time, and has the error
for test, in native code, i callback direct, not in the image request callback, it's ok
why?

@facebook-github-bot stack-overflow


Adding Analytics permissions to lambda trigger function fails

Note: If your issue/bug is regarding the AWS Amplify Console service, please log it in the
Amplify Console GitHub Issue Tracker

Describe the bug
A clear and concise description of what the bug is.
Adding permissions to access analytics from a lambda trigger function results in:
CREATE_FAILED               AmplifyResourcesPolicy                                                 AWS::IAM::Policy           Tue Mar 24 2020 16:02:22 GMT-0700 (Pacific Daylight Time) The policy failed legacy parsing (Service: AmazonIdentityManagement; Status Code: 400; Error Code: MalformedPolicyDocument; Request ID: bc7c5f42-a965-46d0-aa48-cfa5b3101575)
Amplify CLI Version
You can use amplify -v to check the amplify cli version on your system
4.17.1
To Reproduce
Steps to reproduce the behavior or terminal output if applicable
box:dir user$ amplify add function
Using service: Lambda, provided by: awscloudformation
? Provide a friendly name for your resource to be used as a label for this category in the project: notifyUsersOnNewContent
? Provide the AWS Lambda function name: notifyUsersOnNewContent
NodeJS found for selected function configuration.
? Choose the function template that you want to use: Lambda trigger
? What event source do you want to associate with Lambda trigger? Amazon DynamoDB Stream
? Choose a DynamoDB event source option Use API category graphql @model backed DynamoDB table(s) in the current Amplify project
Selected resource story
? Choose the graphql @model(s) Story, Take, StoryVote, TakeVote
? Do you want to access other resources created in this project from your Lambda function? No
? Do you want to edit the local lambda function now? No
Successfully added resource notifyUsersOnNewContent locally.

box:dir user$ amplify push
*** This succeeds

box:dir user$ amplify update function
Using service: Lambda, provided by: awscloudformation
? Please select the Lambda Function you would want to update notifyUsersOnNewContent
? Do you want to update permissions granted to this Lambda function to perform on other resources in your project? Yes
? Select the category analytics
Analytics category has a resource called story
? Select the operations you want to permit for story create, read, update, delete

You can access the following resource attributes as environment variables from your Lambda function
var environment = process.env.ENV
var region = process.env.REGION
var analyticsStoryId = process.env.ANALYTICS_STORY_ID
var analyticsStoryRegion = process.env.ANALYTICS_STORY_REGION

? Do you want to edit the local lambda function now? No
Successfully updated resource

box:dir user$ amplify push

Current Environment: dev

| Category      | Resource name           | Operation | Provider plugin   |
| ------------- | ----------------------- | --------- | ----------------- |
| Function      | notifyUsersOnNewContent | Update    | awscloudformation |
| Auth          | storyAuth               | No Change | awscloudformation |
| Storage       | storyMedia              | No Change | awscloudformation |
| Api           | story                   | No Change | awscloudformation |
| Analytics     | story                   | No Change | awscloudformation |
| Notifications | story                   | No Change |                   |
? Are you sure you want to continue? Yes
⠋ Updating resources in the cloud. This may take a few minutes...

 This may take a few minutes...

UPDATE_IN_PROGRESS apistory AWS::CloudFormation::Stack Tue Mar 24 2020 16:01:52 GMT-0700 (Pacific Daylight Time)
⠸ Updating resources in the cloud. This may take a few minutes...

UPDATE_IN_PROGRESS amplify-story-dev-174558-apistory-H5AFAQK3RFB2 AWS::CloudFormation::Stack Tue Mar 24 2020 16:01:52 GMT-0700 (Pacific Daylight Time) User Initiated
⠙ Updating resources in the cloud. This may take a few minutes...

 This may take a few minutes...

UPDATE_COMPLETE    ConnectionStack AWS::CloudFormation::Stack Tue Mar 24 2020 16:02:04 GMT-0700 (Pacific Daylight Time)
UPDATE_IN_PROGRESS ConnectionStack AWS::CloudFormation::Stack Tue Mar 24 2020 16:02:03 GMT-0700 (Pacific Daylight Time)
⠴ Updating resources in the cloud. This may take a few minutes...

UPDATE_IN_PROGRESS CustomResourcesjson AWS::CloudFormation::Stack Tue Mar 24 2020 16:02:07 GMT-0700 (Pacific Daylight Time)
⠹ Updating resources in the cloud. This may take a few minutes...

UPDATE_COMPLETE_CLEANUP_IN_PROGRESS amplify-story-dev-174558-apistory-H5AFAQK3RFB2 AWS::CloudFormation::Stack Tue Mar 24 2020 16:02:09 GMT-0700 (Pacific Daylight Time)
UPDATE_COMPLETE                     CustomResourcesjson                            AWS::CloudFormation::Stack Tue Mar 24 2020 16:02:07 GMT-0700 (Pacific Daylight Time)
⠼ Updating resources in the cloud. This may take a few minutes...

UPDATE_COMPLETE apistory AWS::CloudFormation::Stack Tue Mar 24 2020 16:02:14 GMT-0700 (Pacific Daylight Time)
⠼ Updating resources in the cloud. This may take a few minutes...

UPDATE_IN_PROGRESS functionnotifyUsersOnNewContent AWS::CloudFormation::Stack Tue Mar 24 2020 16:02:17 GMT-0700 (Pacific Daylight Time)
⠹ Updating resources in the cloud. This may take a few minutes...

UPDATE_IN_PROGRESS amplify-story-dev-174558-functionnotifyUsersOnNewContent-1NW8W3ZS47KKY AWS::CloudFormation::Stack Tue Mar 24 2020 16:02:17 GMT-0700 (Pacific Daylight Time) User Initiated
⠦ Updating resources in the cloud. This may take a few minutes...

UPDATE_ROLLBACK_IN_PROGRESS amplify-story-dev-174558-functionnotifyUsersOnNewContent-1NW8W3ZS47KKY AWS::CloudFormation::Stack Tue Mar 24 2020 16:02:23 GMT-0700 (Pacific Daylight Time) The following resource(s) failed to create: [AmplifyResourcesPolicy].
CREATE_FAILED               AmplifyResourcesPolicy                                                 AWS::IAM::Policy           Tue Mar 24 2020 16:02:22 GMT-0700 (Pacific Daylight Time) The policy failed legacy parsing (Service: AmazonIdentityManagement; Status Code: 400; Error Code: MalformedPolicyDocument; Request ID: bc7c5f42-a965-46d0-aa48-cfa5b3101575)
UPDATE_COMPLETE             LambdaFunction                                                         AWS::Lambda::Function      Tue Mar 24 2020 16:02:22 GMT-0700 (Pacific Daylight Time)

*** more rollback

Expected behavior
A clear and concise description of what you expected to happen.
amplify push should succeed.
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):

OS: [e.g. Mac/Windows/Ubuntu]
Mac 10.15.3
Node Version. You can use node -v to check the node version on your system
v13.8.0

Additional context
Add any other context about the problem here.

I believe the problem is in the cloudformation-template.json, specifically where it defines AmplifyResourcesPolicy.  It's trying to join together the pieces of the arn, but the two refs aren't getting populated as parameters.  If I replace the refs with hardcoded value, the push works...
So, somehow the params aren't getting populated...
"Fn::Join": [
										"",
										[
											"arn:aws:mobiletargeting:",
											{
												"Ref": "analyticsstoryRegion"
											},
											":",
											{
												"Ref": "AWS::AccountId"
											},
											":apps/",
											{
												"Ref": "analyticsstoryId"
											}
										]
									]

Unfortunately, although hardcoding the values into the policy allowed the service to deploy, the environment variables on the lambda config are not correctly set.  Instead of the real values, they just end up with the parameter names, analyticsstoryRegion and analyticsstoryId...

Another update on this.  In addition to the failure to push and the lack of usable environment variables in the function itself, it appears that the permissions are not setup properly for the IAM role to access the pinpoint resource.  The following error results from calling SendUsersMessages:
AccessDeniedException: User: arn:aws:sts::*****:assumed-role/storyLambdaRoled***-dev/notifyUsersOnNewContent-dev is not authorized to perform: mobiletargeting:SendUsersMessages on resource: arn:aws:mobiletargeting:us-east-1:********:apps/*******/messages

any progress on this?  It would be nice to not have to hard code environment specific data in the cloud formation files...

Sorry for not responding earlier, we were going through some major updates in the function category.
I reproduced the bug, will send a fix for it soon.

Any update on this?

@ajpaulingalls
The Amplify CLI function category has gone through some major updates and refactoring.
I just tested with the CLI version 4.21.1 and the bug is not there anymore.
Please try and confirm.


AWS Lambda function integration

Ability to connect to aws, using aws credentials.
Which allows users to run lambda functions locally. Just like sublime text which already has an plugin to do this.

Custom integrations are most likely should be addressed with extensions (similar to plusing for Sublime)
Have you tried to search for extensions? and check AWS docs?
Looks like it already exists https://aws.amazon.com/visualstudiocode/
/question


cri: improve CRI log redirection efficiency

currently, containerd CRI log framework work as following (take STDOUT as example)
{log file} <----- {redirect goroutine(decorate CRI log) } <-------{ stdoutGroup }<-------{goroutine (do io.Copy)} <------- { container }
we can replace io.Copy to Copy( sync.Pool based implementation) for improve effectiency :-)
Signed-off-by: zhang he<EMAIL_ADDRESS>

Is this normally taking advantage of WriterTo or ReaderFrom already in these cases?


gnat{13,14}: bootstrap with gnat-bootstrap{13,14}

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
25.05 Release Notes (or backporting 24.11 and 25.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Merge this together https://github.com/NixOS/nixpkgs/pull/361717


Chains can be configured to only allow whitelisted applications.

Blocks with operations and messages that are not from whitelisted applications are considered invalid.
Part of https://github.com/linera-io/linera-protocol/issues/305.

As discussed, we will just add one Option<ChainId> to OpenChain that sets the application managing the chain for now.
If it is Some, only operations from that application are allowed. Operations from other applications, as well as system operations, are excluded.
All incoming messages are allowed, though.


update tardis recipe

Description
We recently changed TARDIS versioning to CalVer. Then I nticed this pipeline was "stuck" running the old v3.0.dev4019 release. This happens because when running git tag the semantic versions appear at the bottom, while the new release-{date} are close to the top:
0.9.1
0.9.2
1.0.dev1419
1.0rc3
3.0.dev3383
3.0.dev3384
3.0.dev3388
3.0.dev3396
master_bkp
release-2021.07.01
release-2<IP_ADDRESS>
release-2<IP_ADDRESS>
release-2<IP_ADDRESS>
release-2<IP_ADDRESS>
release-2021.7.2.dev1+g58e4e79d
release-2021.7.5
v0.9a
v1.0
v1.0.1
v1.0rc3
v2.0
v2.0.1
v2.0.2
v2_test1
v2_test2
v2_test3
v3.0-alpha.2
v3.0-alpha.3
v3.0.a1
v3.0.dev3403
(...)
v3.0.dev3998
v3.0.dev4010
v3.0.dev4019

Since this pipeline is running git 2.1.4 we couldn't use git describe or git tag --sort to get the latest tag gracefully. Then, I needed to use the GitHub API to get it.
Also, I made some changes to force the installation to use the tardis_env3.yml environment file, so we don't need to update dependencies here.
Finally, I overloaded the test function instead of install.

@epassaro thank you for looking into this. I like your unique approach! This kind of integration testing is quite innovative and much of the landscape has not been explored yet so it's always good to explore alternative ideas. Do you want me to start reviewing already or are there more changes you would like to make?

Yes, sure. Please take a look @esc

@epassaro thank you for the patch!


chore: move from sanitize-html to js-xss

I realized that our dependency to sanitize rendered markdown causes a huge chunk of runtime warnings:

This is due to a transitive postcss dependency, see this issue.
There are two options:

use an isomorphic library (this PR)
render & sanatize markdown on server (preferred but larger architectural change)

Impact:

tests pass & visual inspection looks good ✅
14% reduction in gzipped bundle size:
441.4 kB -> 381.12 kB

Can you please add tests for the h elements? It was added to be allowed recently but we don't have tests. And also a specific tests for removed some html tag element, like code? We don't have tests for that.
Besides that, the changes looks fine!

Can you please add tests for the h elements? It was added to be allowed recently but we don't have tests. And also a specific tests for removed some html tag element, like code? We don't have tests for that.
Besides that, the changes looks fine!

Good idea - I might do this before & outside of this PR

nice one, it seems the new package is trust worthy https://socket.dev/npm/package/xss, can we add ADR for this? 🎉


Clone of SSCS-10865 to test azure pipelines

Change description
Clone of SSCS-10865 to test azure pipelines
Does this PR introduce a breaking change? (check one with "x")
[ ] Yes
[x] No

not needed anymore


CreateThread/QueueAPC block on Windows 10

Hi DarthTon,
this is not really an issue with Blackbone itself but I hope you can help me nevertheless.
I am using a variant of the Blackbone driver for Manual-Mapping my stuff but recently on Windows 10 (even before Win10FC) the injection was "blocked" by a ring-0 ac for both APC based and CreateThread based execution (BBCallRoutine I guess). The same code still works on Win 7 and works flawlessly on regular processes. Of course I remove the CreateThread notify routines so my calls should go through.
I already did the following to debug the issue:

I traced every single instruction a new thread/apc executes and could verify that the execution stops silently as soon as the entry-point (first instruction) of the payload is invoked. No hooks on these routines.
Due to this I tried both methods of allocating memory (mapping kernel memory to user memory and direct allocation as done by Blackbone) for the thread code but that did not help.
I checked the SE_SIGN level, the flag2 entry in EPROCESS and all other process protection/cfg relevant flags but everything is off. Unfortunately DEP cannot be disabled for native x64 processes as indicated in a comment in that routine.
I disabled SGX in bios.

At this point I am highly confused to why the exeuction does not work (I assume somehow the first instruction of the routine triggers an exception which is then caught). Can you provide any ideas of which mechanisms could further block injection or hint me to anything I can still try?

Hard to tell. Does the thread simply die after first instruction? What's the memory protection of payload region before and after execution?

I am quite sure that the thread simply dies (at least the execution does not hang even though these BBCallRoutine calls use the parm to wait for the thread to finish - the functions immediately return). I will look up the memory protection of the payload in the evening and report back.

@DarthTon Unfortunately this did not bring up anything spectacular I think:
BlackBone: BBQueryMemoryProtection: ZwQueryVirtualMemory(0x1C821470000): AllocationBase=0x1C821470000 AllocationProt=PAGE_EXECUTE_READWRITE RegionSize=0x2000 State=MEM_COMMIT Protect=PAGE_EXECUTE_READWRITE Type=MEM_PRIVATE
BlackBone: DummyCreateThreadCallback:<PHONE_NUMBER>001088,<PHONE_NUMBER>001070, 1) <--- Dummy PsSetThreadCreateNotify callback is triggered (creation).
BlackBone: BBQueryMemoryProtection: ZwQueryVirtualMemory(0x1C821470000): AllocationBase=0x1C821470000 AllocationProt=PAGE_EXECUTE_READWRITE RegionSize=0x2000 State=MEM_COMMIT Protect=PAGE_EXECUTE_READWRITE Type=MEM_PRIVATE
BlackBone: DummyCreateThreadCallback:<PHONE_NUMBER>001088,<PHONE_NUMBER>001070, 0) <--- Dummy PsSetThreadCreateNotify callback is triggered (creation, thread end).

According to ZwQueryVirtualMemory the memory region of the thread entry point is perfectly executable, the thread immediately returns with exit code of 0. ZwCreateThreadEx does not fail... It is a mystery to me how they prevent the memory access to that page.
Interestingly if I change the entry point of the thread to something like 0x1234 (inaccessible) the game crashes. If i change it to my memory region with the first instruction being something like 0xCC the game does not crash! (Instruction is not executed). I cannot really find an explaination how they prevent that code to trigger without setting some kind of guard to it or using a hardware breakpoint which is impossible as they do not know about the fresh thread (hwbps need to be set per thread base afaik).

I also tried a third method of allocating the memory with ZwCreateSection and ZwMapViewOfSection to make it look more legitimate. Unfortunately also without success.

Can I test it myself somehow?

Yes, the protected game is DeadByDaylight (AC=EAC). Unfortunately you would need
a way to load your driver without enabling test-signing (i.e. using some kernel exploit or
having a certificate).
Besides you need to change the pool-tag and disable
ThreadCreation notify routines (otherwise the thread gets directly blocked) and preferably
also unregister ObsCallbacks (Creating threads opens up a handle afaik this could be directly
used to stop the thread). Also you should use Thread flags that skip out on dll thread-attach calls (not sure if they use this). Still then you would not be able to attach a debugger.
I can provide you with the game and give further help if needed. On the other hand I found a
workaround in the meantime. When I (instead of allocating the memory with BlackBone) simply
"hijack" some memory of an existing module, changing the protection and wiping it all my calls go through. Of course this can induce crashes if you choose a bad spot for this memory.
Still I would be very curious about how they achieve this protection.

好人，哥哥


How to configure user password authentication login?

How to configure user password authentication login? At present, I have tried all the methods according to the online method, such as adding in server.conf.
Auth-user-pass-verify /usr/local/open***/etc/checkpsw.sh via-env
Client-cert-not-required
Username-as-common-name
Both checkpsw.sh and psw-file have been added.
Add in ovpn file
Auth-user-pass

This is not an issue with the script.
I'd suggest to read the OpenVPN manual and understand what you are doing instead of copy-pasting information from unreliable sources.


No FileList is returned from Agora Server after stop the Cloud Recording

Hi,
I have successfully done Agora Cloud Recording implementation in my React Native App with AWS. Now i need to store some screenshots of recording so that in future i can show those screenshots of records as thumbnail of the video.
I have used bellow configs for uploading the snapshots to AWS:

From the above 👆 config you understand that i have used snapshotConfig. From the doc (link: https://docs.agora.io/en/cloud-recording/reference/rest-api/stop#http-response#http-response) i have seen for snapshotConfig no fileListMode will be returned. Okay, this is good. That means Agora server will also not return fileList in the response of STOP recording API.
My question is, if no fileList is returned then how we will be able to store the file name & file url to database? Is there any other way to get the file url? File means "jpg" images.
FYI, this is response from Agora when i stop the recording. No  fileList (means .jpg files name) is inside serverResponse 👇 but the snapshots were uploaded in AWS successfully ✅

Please help me, please 🙏

@plutoless Could you pls take a look?

@anisurrahman072
you may be use following pattern to find result jpg files in your OSS
__uid_s_uid_e_video.jpg


Fix incorrect actions ref_name (#25357)

Just a simple fix the result of ${{ gitea.ref_name }} to show the shortened name rather than the full ref.

I fixed the column that the comment on that line starts at so lint-backend won't complain and it lines up with the other comments.  If you want me to redo this so it's just one commit, I can. Or you can just redo it yourselves since it's so trivial. Not sure whether it's actually helpful to even submit a PR for something this simple. :/


Support for Viomi S9 UV (viomi.vacuum.v18)

I have just got a Viomi S9 UV (viomi.vacuum.v18).
How do I help to get this model supported?

Should be no problem! Will add support for that vacuum in the next update!

Support is now added as of version 0.9.17


Unhandled error detected

Hello,
I encountered the following error while setting up after installing the Shelly plugin in Matterbridge:
`
[07:37:16.715] [shellyswitch25-E8DB84A20980] - model: SHSW-25
[07:37:16.715] [shellyswitch25-E8DB84A20980] - firmware: v1.14.0-gcb84623
[07:37:16.716] [shellyswitch25-E8DB84A20980] - profile: switch
[07:37:16.716] [shellyswitch25-E8DB84A20980] - components:
[07:37:16.717] [shellyswitch25-E8DB84A20980]   - wifi_ap (WiFi)
[07:37:16.717] [shellyswitch25-E8DB84A20980]   - wifi_sta (WiFi)
[07:37:16.718] [shellyswitch25-E8DB84A20980]   - wifi_sta1 (WiFi)
[07:37:16.718] [shellyswitch25-E8DB84A20980]   - mqtt (MQTT)
[07:37:16.719] [shellyswitch25-E8DB84A20980]   - coiot (CoIoT)
[07:37:16.719] [shellyswitch25-E8DB84A20980]   - sntp (Sntp)
[07:37:16.720] [shellyswitch25-E8DB84A20980]   - cloud (Cloud)
[07:37:16.720] [shellyswitch25-E8DB84A20980]   - relay:0 (Relay)
[07:37:16.720] [shellyswitch25-E8DB84A20980]   - relay:1 (Relay)
[07:37:16.721] [shellyswitch25-E8DB84A20980]   - meter:0 (PowerMeter)
[07:37:16.721] [shellyswitch25-E8DB84A20980]   - meter:1 (PowerMeter)
[07:37:16.721] [shellyswitch25-E8DB84A20980]   - input:0 (Input)
[07:37:16.722] [shellyswitch25-E8DB84A20980]   - input:1 (Input)
[07:37:16.723] [shellyswitch25-E8DB84A20980]   - sys (Sys)
[07:37:16.780] [Matterbridge] Added and registered bridged device (1/1) shellyswitch25-E8DB84A20980 (MA-bridgedNode) for plugin matterbridge-shelly
[07:37:16.782] [Matterbridge shelly plugin] Loaded from cache Shelly device shellyswitch25-3C6105E4F0F1 host <IP_ADDRESS>
[07:37:16.783] [shellyswitch25-3C6105E4F0F1] Shelly added shellyswitch25-3C6105E4F0F1 device id shellyswitch25-3C6105E4F0F1 host <IP_ADDRESS>
[07:37:16.783] [shellyswitch25-3C6105E4F0F1] - gen: 1
[07:37:16.783] [shellyswitch25-3C6105E4F0F1] - mac: 3C6105E4F0F1
[07:37:16.784] [shellyswitch25-3C6105E4F0F1] - model: SHSW-25
[07:37:16.784] [shellyswitch25-3C6105E4F0F1] - firmware: v1.14.0-gcb84623
[07:37:16.784] [shellyswitch25-3C6105E4F0F1] - profile: cover
[07:37:16.784] [shellyswitch25-3C6105E4F0F1] - components:
[07:37:16.785] [shellyswitch25-3C6105E4F0F1]   - wifi_ap (WiFi)
[07:37:16.785] [shellyswitch25-3C6105E4F0F1]   - wifi_sta (WiFi)
[07:37:16.785] [shellyswitch25-3C6105E4F0F1]   - wifi_sta1 (WiFi)
[07:37:16.786] [shellyswitch25-3C6105E4F0F1]   - mqtt (MQTT)
[07:37:16.786] [shellyswitch25-3C6105E4F0F1]   - coiot (CoIoT)
[07:37:16.786] [shellyswitch25-3C6105E4F0F1]   - sntp (Sntp)
[07:37:16.787] [shellyswitch25-3C6105E4F0F1]   - cloud (Cloud)
[07:37:16.787] [shellyswitch25-3C6105E4F0F1]   - roller:0 (Roller)
[07:37:16.787] [shellyswitch25-3C6105E4F0F1]   - meter:0 (PowerMeter)
[07:37:16.787] [shellyswitch25-3C6105E4F0F1]   - input:0 (Input)
[07:37:16.788] [shellyswitch25-3C6105E4F0F1]   - input:1 (Input)
[07:37:16.788] [shellyswitch25-3C6105E4F0F1]   - sys (Sys)
[07:37:16.815] [Matterbridge] Added and registered bridged device (2/2) shellyswitch25-3C6105E4F0F1 (MA-bridgedNode) for plugin matterbridge-shelly
[07:37:16.816] [Matterbridge shelly plugin] Loaded from cache Shelly device shellyswitch25-3C6105E4E892 host <IP_ADDRESS>
[07:37:16.817] [shellyswitch25-3C6105E4E892] Shelly added shellyswitch25-3C6105E4E892 device id shellyswitch25-3C6105E4E892 host <IP_ADDRESS>
[07:37:16.817] [shellyswitch25-3C6105E4E892] - gen: 1
[07:37:16.817] [shellyswitch25-3C6105E4E892] - mac: 3C6105E4E892
[07:37:16.818] [shellyswitch25-3C6105E4E892] - model: SHSW-25
[07:37:16.818] [shellyswitch25-3C6105E4E892] - firmware: v1.14.0-gcb84623
[07:37:16.818] [shellyswitch25-3C6105E4E892] - profile: cover
[07:37:16.818] [shellyswitch25-3C6105E4E892] - components:
[07:37:16.819] [shellyswitch25-3C6105E4E892]   - wifi_ap (WiFi)
[07:37:16.819] [shellyswitch25-3C6105E4E892]   - wifi_sta (WiFi)
[07:37:16.819] [shellyswitch25-3C6105E4E892]   - wifi_sta1 (WiFi)
[07:37:16.819] [shellyswitch25-3C6105E4E892]   - mqtt (MQTT)
[07:37:16.820] [shellyswitch25-3C6105E4E892]   - coiot (CoIoT)
[07:37:16.820] [shellyswitch25-3C6105E4E892]   - sntp (Sntp)
[07:37:16.820] [shellyswitch25-3C6105E4E892]   - cloud (Cloud)
[07:37:16.820] [shellyswitch25-3C6105E4E892]   - roller:0 (Roller)
[07:37:16.820] [shellyswitch25-3C6105E4E892]   - meter:0 (PowerMeter)
[07:37:16.821] [shellyswitch25-3C6105E4E892]   - input:0 (Input)
[07:37:16.821] [shellyswitch25-3C6105E4E892]   - input:1 (Input)
[07:37:16.821] [shellyswitch25-3C6105E4E892]   - sys (Sys)
[07:37:16.845] [Matterbridge] Added and registered bridged device (3/3) shellyswitch25-3C6105E4E892 (MA-bridgedNode) for plugin matterbridge-shelly
[07:37:16.847] [Matterbridge shelly plugin] Loaded from cache Shelly device shellyswitch25-3C6105E534F6 host <IP_ADDRESS>
[07:37:16.848] [ShellyKüche1] Shelly added ShellyKüche1 device id shellyswitch25-3C6105E534F6 host <IP_ADDRESS>
[07:37:16.848] [ShellyKüche1] - gen: 1
[07:37:16.848] [ShellyKüche1] - mac: 3C6105E534F6
[07:37:16.849] [ShellyKüche1] - model: SHSW-25
[07:37:16.849] [ShellyKüche1] - firmware: v1.14.0-gcb84623
[07:37:16.849] [ShellyKüche1] - profile: switch
[07:37:16.850] [ShellyKüche1] - components:
[07:37:16.850] [ShellyKüche1]   - wifi_ap (WiFi)
[07:37:16.850] [ShellyKüche1]   - wifi_sta (WiFi)
[07:37:16.851] [ShellyKüche1]   - wifi_sta1 (WiFi)
[07:37:16.851] [ShellyKüche1]   - mqtt (MQTT)
[07:37:16.851] [ShellyKüche1]   - coiot (CoIoT)
[07:37:16.851] [ShellyKüche1]   - sntp (Sntp)
[07:37:16.852] [ShellyKüche1]   - cloud (Cloud)
[07:37:16.852] [ShellyKüche1]   - relay:0 (Relay)
[07:37:16.852] [ShellyKüche1]   - relay:1 (Relay)
[07:37:16.853] [ShellyKüche1]   - meter:0 (PowerMeter)
[07:37:16.853] [ShellyKüche1]   - meter:1 (PowerMeter)
[07:37:16.853] [ShellyKüche1]   - input:0 (Input)
[07:37:16.854] [ShellyKüche1]   - input:1 (Input)
[07:37:16.854] [ShellyKüche1]   - sys (Sys)
[07:37:16.882] [Matterbridge] Added and registered bridged device (4/4) ShellyKüche1 (MA-bridgedNode) for plugin matterbridge-shelly
[07:37:17.047] [Logger] Unhandled error detected: Cannot read properties of undefined (reading 'cfg_rev')
at ShellyDevice.create (file:///usr/local/lib/node_modules/matterbridge-shelly/dist/shellyDevice.js:665:88)
at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
at async ShellyPlatform.addDevice (file:///usr/local/lib/node_modules/matterbridge-shelly/dist/platform.js:1298:22)
at async Shelly. (file:///usr/local/lib/node_modules/matterbridge-shelly/dist/platform.js:118:17)
file:///usr/local/lib/node_modules/matterbridge-shelly/dist/shellyDevice.js:665
componentsPayload = { components: btHomeComponents, cfg_rev: btHomePayload.cfg_rev, offset: 0, total: btHomeComponents.length };
^
TypeError: Cannot read properties of undefined (reading 'cfg_rev')
at ShellyDevice.create (file:///usr/local/lib/node_modules/matterbridge-shelly/dist/shellyDevice.js:665:88)
at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
at async ShellyPlatform.addDevice (file:///usr/local/lib/node_modules/matterbridge-shelly/dist/platform.js:1298:22)
at async Shelly. (file:///usr/local/lib/node_modules/matterbridge-shelly/dist/platform.js:118:17)
Node.js v22.9.0
`

Hi,
thanks for reporting the issue.
Can you tell me the version matterbridge and matterbridge-shelly plugin?
Can you provide me the full log please?
https://github.com/Luligu/matterbridge?tab=readme-ov-file#how-to-send-the-debug-log-files

Fixed.
[1.0.2] - 2024-10-13
Added

[shelly]: Added scheduled_restart event from Sys to set the devices that will not be loaded from cache at restart.
[shelly]: Added component Devicepower.
[shellyplussmoke]: Added battery readings (battery level and voltage) to shellyplussmoke and shellyhtg3 (all devices with Devicepower component).

Fixed

[BTHome]: Fixed issue to BTHome components discovery.


LineChart data is being truncated

Hi, it seems data is being truncated from my line chart. I was using 0.0.9 but updated to 0.1.1 and still have the same problem. I also switched to a historicalBarChart with the same result. Data after the 10th index is just being dropped.

Chart object:
var months = ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec'];
vm.yoyEventsByMonth = {
options:{
chart:{
type: 'lineChart',
height: 400,
legend: {
updateState: false
},
useInteractiveGuideline: true,
x: function(d){return d.x;},
y: function(d){return d.y;},
xAxis: {
tickFormat: function (d){
return months[d];
}
}
},
title:{
enable: true,
text: 'Event Totals by Year',
class: 'h3'
}
},
data: []
};
Data:
[{"key":2014,"values":[{"x":"0","y":0,"series":0},{"x":"1","y":1,"series":0},{"x":"2","y":0,"series":0},{"x":"3","y":0,"series":0},{"x":"4","y":0,"series":0},{"x":"5","y":0,"series":0},{"x":"6","y":0,"series":0},{"x":"7","y":0,"series":0},{"x":"8","y":0,"series":0},{"x":"9","y":1,"series":0},{"x":"10","y":0,"series":0},{"x":"11","y":1,"series":0}]},{"key":2015,"values":[{"x":"0","y":0,"series":1},{"x":"1","y":38,"series":1},{"x":"2","y":8,"series":1},{"x":"3","y":0,"series":1},{"x":"4","y":0,"series":1},{"x":"5","y":0,"series":1},{"x":"6","y":0,"series":1},{"x":"7","y":0,"series":1},{"x":"8","y":0,"series":1},{"x":"9","y":0,"series":1},{"x":"10","y":0,"series":1},{"x":"11","y":0,"series":1}]}]
I was expecting it to go out to December regardless of if y had a value. In the first series, y does have 1 but still no luck. I also pasted this data into the plunker for this chart of it was truncated at October. Thanks for any light you can shed on this.

Try to convert x values to numbers: x: function(d){return +d.x;}
If don't help, try to extend domain on x axis:
chart: {
  xDomain: [0, 11]
}

@krispo The first solution worked. Thank you!


AdsLoader error: AdError 1009

Hi,
As part of the sign-up process to AdSense AFV, Google requests a test page that demonstrates a player running a test AdSense AFV tag. However, when i run the test tag in Videojs-ima, i'm getting this error:
videojs.ima.js?ver=4.9.6:1368 AdsLoader error: AdError 1009: The VAST response document is empty. SdkImpl.onAdsLoaderError @ videojs.ima.js?ver=4.9.6:1368
The AdSense AFV tag works fine in the Videojs-ima and the VAST Inspector => https://developers.google.com/interactive-media-ads/docs/sdks/html5/vastinspector
Any tips for me when trying to get AFV to load on our site?

Hi,
You might need to reach out to the Ad Manager Support Team to solve this issue. Some of your ad tag url parameters might be preventing the ad server from returning a complete VAST XML. Can you share the ad tag you were using when you encountered this issue?

Is it related https://github.com/googleads/videojs-ima/issues/683 ?

Hi,
The issue described above is ad serving related, it is not caused by the VideoJS-IMA plugin, reaching out to your ad provider would be the only way to fix it.

Hi,
I’m closing this issue because it has been inactive for more than a month. Please reopen if you still encounter this issue with the latest stable version
Thank you!

i'm getting this issue as well now, added ads.txt and nothing. The weird thing is , on a test page it works sparadically, when I put on the site template, where it has to be for the G rep to look, not a thing, zip, nada. no errors or anything and exactly the same code is the outside site test page


Add hashCode to RenderObject.toString()

When dumping the render tree, it allows you to search the
output in a text editor for specific render objects that you
printed for debug purposes, thus greatly speeding up your
debugging.

@abarth

This will make the trees WAY more noisy and ugly. Is there another way we could address this?

Can you paste the output of a debugDumpRenderTree with this patch?

This will make the trees WAY more noisy and ugly

Actually, I think it's hardly noisier at all, and the ability to instantly jump to a desired point in the dumped tree is highly useful.
https://gist.github.com/tvolkert/4ce1b1bc7f7f6c6719146d4a55abb267

Yeah, that's not as bad as I feared.
We have several toString functions that include the hashCode. We should make sure we are consistent in how we present that. InkFeature uses runtimeType@hashCode. _SemanticsFragment uses runtimeType(hashCode). SemanticsOwner uses runtimeType@hashCode. AssetBundle uses runtimeType@hashCode(data). ObjectKey uses [runtimeType1 runtimeType2@hashCode]. UniqueKey uses [hashCode]. State uses runtimeType(hashCode, data). OverlayEntry uses runtimeType@hashCode(data).
BoxHitTestEntry uses @ for something else, runtimeType@position. Keys in general use [] to represent that they're a key. I like that, it reads well.
How about using # for hashCode? The full pattern would be runtimeType#hashCode(data). UniqueKey would be [#hashCode]. Classes in hiearchies without data would just omit the () part.

We'll have to update the debugging page on the Web site, which includes samples dumps.

I like #$hashCode.  I was gonna use @ since that's fairly common, but @abarth said that has existing semantic meaning since we use it for coordinates.

I'm not sure we're consistent about how we use @.  Using # makes good sense since it's a hashtag. :)

This is great. Thanks!


la1ere: french overseas (9 regions) public and free

CONTEXT
French overseas public channels are available without authentication from the portal : https://la1ere.francetvinfo.fr/
Apparently, all formats/manifests are the same as those from france.tv (extractor francetv.py):
indeed, a few videos are even shared by both sites.
For its developers, it should be straightforward to derive a new extractor la1ere.py from francetv.py (only two URL templates/pages embed videos) - i really hope my feeling is right, and would gladly help if i had some knowledge in video streams and python.
The video offer is free, very large and covers documentaries, news, fictions of many cultures. It would be a nice gift for those interested in those, as internet speed may be slow overseas, preventing streaming :)
9 regions are covered (below,  refers to one of)

guadeloupe
guyane
martinique
mayotte
nouvellecaledonie
polynesie
reunion
saintpierremiquelon
wallisfutuna

Here is the mosaic of videos available for the chosen region:
https://la1ere.francetvinfo.fr/REGION/programme-video/
(as said before, a few of them are also available at: https://www.france.tv/la1ere/REGION/toutes-les-videos/)
TEMPLATES OF EMBEDDING URLs
In the end, final pages embedding videos may have 2 paths only:

https://la1ere.francetvinfo.fr/REGION/programme-video/diffusion/PAGE
https://la1ere.francetvinfo.fr/REGION/programme-video/SHOW/diffusion/PAGE

The latter is used for series, news ...
The extractor may be flexible and even not consider/whitelist the region list, letting it be free (prophylaxis/evolutions), as the substring "/diffusion/" already filters templates.
EXAMPLES:
Documentary:
https://la1ere.francetvinfo.fr/martinique/programme-video/diffusion/4774522-origine-kongo.html
News:
https://la1ere.francetvinfo.fr/guadeloupe/programme-video/la1ere_guadeloupe_le-13h-en-guadeloupe/diffusion/5643549-emission-du-lundi-29-janvier-2024.html

I used a VPN to set my location as France so I could access.
What I saw is that the video is streamed by the player a chunk at a time using URLs like this:
(first chunk)
https://cloudingest.ftven.fr/ZXhwPTE3MjIyODYxNDN+YWNsPSUyZip+aG1hYz1mMjE3ZWE2NzRhNmRhZjhkZmUwMjA0MmFlNWEyMmM0ZDM5NmY3MmJmZjFhNTE2MWNhMWU2MWZkYjdkMmMxMmRk/462436a625346/97790f3a-b23a-4004-bd9c-b89fde6f95bf-1681205990_france-domtom_TA.ism/dash/97790f3a-b23a-4004-bd9c-b89fde6f95bf-1681205990_france-domtom_TA-video=2000000.dash
(second chunk)
https://cloudingest.ftven.fr/ZXhwPTE3MjIyODYxNDN+YWNsPSUyZip+aG1hYz1mMjE3ZWE2NzRhNmRhZjhkZmUwMjA0MmFlNWEyMmM0ZDM5NmY3MmJmZjFhNTE2MWNhMWU2MWZkYjdkMmMxMmRk/462436a625346/97790f3a-b23a-4004-bd9c-b89fde6f95bf-1681205990_france-domtom_TA.ism/dash/97790f3a-b23a-4004-bd9c-b89fde6f95bf-1681205990_france-domtom_TA-audio_fre=96000.dash
I don't know if youtube-dl can download a video without a link to a single resource, especially I'm not sure whether we can reconstruct how to query these .dash files from the source page.

yt-dl knows how to deal with fragment manifests, whether HLS (m3u8) or DASH (mpd). If, as with On:ORF, eg, a site may serve a single show as multiple parts, the site extractor has to return a multi_video playlist, which has to be concatenated manually for now.
For this site we have to reverse-engineer the site's behaviour because there are no media links in the non-JS page that yt-dl receives. In the browser tools network tab, look for a request by XHR for a DASH manifest, before the video starts to stream. If that URL can be reconstructed from information available in the page URL or the page itself, then it should be possible to create an extractor, provided that the video is not DRM-encrypted (does it play in a browser with DRM aka EME disabled?).
My initial impression from the page source is that the site geo-blocks outside FR and its remnant empire. The .dash URLs are giving me 403 but may just have expired.

@dirkf ah, ok that's good to know. I'm definitely new to the project so I'm glad you're here with such info.
SO,
There is a request to a .mpd file, but the issue is it has some pretty cooked query params and it's not evident how to derive them from the page. I tried looking at the initiatior of the xhr but it's all minified garbage js.
For the page https://la1ere.francetvinfo.fr/martinique/programme-video/diffusion/4774522-origine-kongo.html, the .mpd request is:
https://cloudingest.ftven.fr/ZXhwPTE3MjIyOTI1MjR+YWNsPSUyZip+aG1hYz1jYzM5Mzg0YzRmNjU5YjNkMDYzM2NhN2Q4YzJjMDQwYjc2NTI4MTNmOWMyNzk5MzE4MDkzNmU4N2I3OWJkNGRm/462436a625346/97790f3a-b23a-4004-bd9c-b89fde6f95bf-1681205990_france-domtom_TA.ism/manifest.mpd?hdnea=exp=1722271524~acl=%2f*~hmac=c294fe8aaae7faaa23735f371a367deb6a6365d6638b6fd8201c9bd83a2cb414
Let's break this up into parts:

https://cloudingest.ftven.fr - this is fine

ZXhwPTE3MjIyOTI1MjR+YWNsPSUyZip+aG1hYz1jYzM5Mzg0YzRmNjU5YjNkMDYzM2NhN2Q4YzJjMDQwYjc2NTI4MTNmOWMyNzk5MzE4MDkzNmU4N2I3OWJkNGRm - does not appear in the webpage or the script bundle - must be generated

462436a625346 does not appear in the webpage or the script bundle - must be generated

97790f3a-b23a-4004-bd9c-b89fde6f95bf appears in the webpage as the data-id attribute of the #mainContent element.

_france-domtom_TA.ism/manifest.mpd - this is fine

`?hdnea=exp=1722271524~acl=%2f*~hmac=c294fe8aaae7faaa23735f371a367deb6a6365d6638b6fd8201c9bd83a2cb414`` - these query params include an HMAC which is a bit scary as we'd have to figure out how this is generated... and it seems to be happening inside the magnetoscope player code.

tl;dr: requesting the .mpd file is complicated and involves some crypto that seems unlikely to be cracked.

Hmm, I just found something else.
(reference URL is https://la1ere.francetvinfo.fr/martinique/programme-video/diffusion/4774522-origine-kongo.html)
The entire .mpd URL including the HMAC query params can be retrieved by
https://hdfauth.ftven.fr/esi/TA?format=json&url=https%3A%2F%2Fcloudingest.ftven.fr%2F462436a625346%2F97790f3a-b23a-4004-bd9c-b89fde6f95bf-1681205990_france-domtom_TA.ism%2Fmanifest.mpd
This returns:
{
    "url": "https://cloudingest.ftven.fr/ZXhwPTE3MjIyOTI1MjR+YWNsPSUyZip+aG1hYz1jYzM5Mzg0YzRmNjU5YjNkMDYzM2NhN2Q4YzJjMDQwYjc2NTI4MTNmOWMyNzk5MzE4MDkzNmU4N2I3OWJkNGRm/462436a625346/97790f3a-b23a-4004-bd9c-b89fde6f95bf-1681205990_france-domtom_TA.ism/manifest.mpd?hdnea=exp=1722271524~acl=%2f*~hmac=c294fe8aaae7faaa23735f371a367deb6a6365d6638b6fd8201c9bd83a2cb414"
}

Exactly what we need.
So, if we can chef up this URL we can get the .mpd.
We need therefore to be able to find/generate:

462436a625346 (not in webpage or bundle)
97790f3a-b23a-4004-bd9c-b89fde6f95bf (in webpage)
1681205990 (not in webpage or bundle)

@dirkf even if I did write an exporter for this, the tests would always fail without a France VPN.  Or could we publish it without tests - but then, how would we know when it breaks...

Normally someone in the relevant region, or who has a VPN so as to appear so, is both sufficiently interested and sufficiently skilled to work on the extractor. If any extractor tests can't be run outside the region, the author should post a transcript showing the tests working and then disable them.
OP suggested that this site is like FranceTV, so it may be possible to use the yt-dlp FranceTV extractor (I suspect that ours is broken) as a basis, or just to add a new extractor class in the FranceTV extractor.
The player_version thing may just be instrumentation, but otherwise might be found in one of the JS modules.
The JSON seems to have a good set of metadata. Maybe some specific headers are needed to get the DASH manifest without Akamai "blocage", or maybe just an acceptable IP address.

@dirkf ah, i thought the tests were run on every extractor before every release - i guess they're just run when merging the pull request? I've got a free france VPN set up for this so I can try to work on it

There's no point scheduling a test that will always fail so the test-case will typically include 'skip': 'only available in FR', or similar. A maintainer who is working on the extractor would comment those out and then restore all those that are found still to be applicable.

I wrote an extractor for both URL formats in the original request. Tested on my end using a france VPN and both work to download the video.
@dirkf could you confirm that nothing is needed to download an mpd/m3u8 resource other than returning "formats" in the _real_extract object? This seemed suspiciously simple.

@dirkf could you expand no the http_headers part of that? i don't see any example of that in the francetv extractor, it just extracts and sorts the formats. Thanks in advance, I know i've been blowing u up lately


Error: Error [ERR_FR_MAX_BODY_LENGTH_EXCEEDED]: Request body larger than maxBodyLength limit

Hello. Could you please explain where is this error shown? 🤔

path add 8M png image

Could you provide more details and examples or such? I'm not sure I understand. 😅

steps:
  - name: Upload a picture
    uses<EMAIL_ADDRESS># Use this for the most stable version
    with:
      path: ./img.png  # required
      client_id: ${{secrets.IMGUR_CLIENT_ID}}  # required
      description: My picture  # optional

img.png is 8M image
then, get error: Request body larger than maxBodyLength limi

Strange. CC: @lunatic-fox, @Snailedlt.

@bingryan do you have a link to the repo?

@bingryan Do you still have this issue?


Provide information about where the text comes from for the text representation of the results

We need a greater level of detail about how the paragraph about the data summary was generated.

There are 4 components to a projection statement that always: magnitude of change, for when, relative to when, and what scenario.

Also note that there's some overlap between this description and the table intros.


Better way to differentiate between 400 errors.

When using GetProfileAndToken, there are a few cases where we might have an error. Namely:

The Client ID was invalid.
The Secret key was invalid or expired.
The code itself was invalid.

We want to handle these cases differently (e.g., have our callback handler return a 500 in the first two cases so alarm bells go off, but a 400 in the latter case since not our problem). However, it's difficult to do this because of the way that the error is returned in the SDK. The JSON bodies of these responses looks like this:
{"error":"invalid_client","error_description":"Invalid client_id."}
{"error":"invalid_client","error_description":"Invalid client secret."}
{"error":"invalid_grant","error_description":"The code 'XXXYYYZZZ' has expired or is invalid."}

...but the workos_error.HTTPError only returns the following:
Code: 400
Status: 400 Bad Request
RequestID: A UUID
ErrorCode: ""
Errors: []
FieldErrors: []
IsRequestError: true
Message (see below)

The message ends up using this logic to mash up the strings, so the three errors I talked about above would result in these Message values:
"invalid_client Invalid client_id"
"invalid_client Invalid client secret."
"invalid_grant The code 'XXXYYYZZZ' has expired or is invalid."

So in the end, while in theory the SDK gives us an error code so that a developer doesn't need to parse a string, I still end up needing to parse a string.
Ideally, the code (e.g., "invalid_client" and "invalid_grant") would be in a field like "ErrorCode.

That is still very relevant as it also applies to handling all the authentication errors which is basically impossible in the current state of the SDK if I'm correct. Why was #203 closed ?