Spyder 4.0 fails to connect to kernel

Issue Report Checklist
[* ] Searched the issues page for similar reports
[* ] Read the relevant sections of the Spyder Troubleshooting Guide and followed its advice
[* ] Reproduced the issue after updating with conda update spyder (or pip, if not using Anaconda)

[ ] Could not reproduce inside jupyter qtconsole (if console-related)
[* ] Tried basic troubleshooting (if a bug/error)
[* ] Restarted Spyder
[* ] Reset preferences with spyder --reset
[* ] Reinstalled the latest version of Anaconda
[* ] Tried the other applicable steps from the Troubleshooting Guide
[* ] Completed the Problem Description, Steps to Reproduce and Version sections below

Problem Description
I had anaconda 2.7 and older spyder version and everything used to be working fine. I just installed the anaconda 3.7 and updated the spyder to v 4.0.1 (I still have my anaconda 2.7 installation and plan to keep it). when opening spyder , the spedyer cannot connect to kernel , following is the materials showed up on spyder consul.
Pyhton 3.7.6
Spyder 4.0.1
Ubuntu 18.04
there seems to be the issue for other versions of spyder but solution were not the same on different version and instances
An error ocurred while starting the kernel
Traceback (most recent call last):
File "/home/rahikalantari/anaconda3/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/home/rahikalantari/anaconda3/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/rahikalantari/anaconda3/lib/python3.7/site‑packages/spyder_kernels/console/main.py", line 11, in
start.main()
File "/home/rahikalantari/anaconda3/lib/python3.7/site‑packages/spyder_kernels/console/start.py", line 287, in main
import_spydercustomize()
File "/home/rahikalantari/anaconda3/lib/python3.7/site‑packages/spyder_kernels/console/start.py", line 39, in import_spydercustomize
import spydercustomize
File "/home/rahikalantari/anaconda3/lib/python3.7/site‑packages/spyder_kernels/customize/spydercustomize.py", line 17, in
import pdb
File "/home/rahikalantari/anaconda3/lib/python3.7/pdb.py", line 76, in
import code
File "/home/rahikalantari/code.py", line 36
parser.add_argument('‑‑log‑interval', type=int, default=10, metavar='N',
^
SyntaxError: invalid syntax
What steps reproduce the problem?

call spyder on the consul
as soon as it opens up, it cannot find the kernel

What is the expected output? What do you see instead?
I expect to see python version and be able to write python command
i see following
"An error ocurred while starting the kernel"
Paste Traceback/Error Below (if applicable)

PASTE TRACEBACK HERE

```An error ocurred while starting the kernel
Traceback (most recent call last):
File "/home/rahikalantari/anaconda3/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/home/rahikalantari/anaconda3/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/rahikalantari/anaconda3/lib/python3.7/site‑packages/spyder_kernels/console/__main__.py", line 11, in
start.main()
File "/home/rahikalantari/anaconda3/lib/python3.7/site‑packages/spyder_kernels/console/start.py", line 287, in main
import_spydercustomize()
File "/home/rahikalantari/anaconda3/lib/python3.7/site‑packages/spyder_kernels/console/start.py", line 39, in import_spydercustomize
import spydercustomize
File "/home/rahikalantari/anaconda3/lib/python3.7/site‑packages/spyder_kernels/customize/spydercustomize.py", line 17, in
import pdb
File "/home/rahikalantari/anaconda3/lib/python3.7/pdb.py", line 76, in
import code
File "/home/rahikalantari/code.py", line 36
parser.add_argument('‑‑log‑interval', type=int, default=10, metavar='N',
^
SyntaxError: invalid syntax

## Versions
<!--- You can get this information from Help > About Spyder...
or (if Spyder won't launch) the "conda list" command
from the Anaconda Prompt/Terminal/command line. --->

* Spyder version: 4.0.1
* Python version: 3.7.6
* Qt version: 5.9.7
* PyQt version: 5.0.2
* Operating System name/version: Ubuntu 18.04

### Dependencies
<!--- Please go to the menu entry Help > Dependencies,
press the Copy to clipboard button and paste below --->

PASTE DEPENDENCIES HERE

Duplicate of #11402


Exception thrown when downloading a video from facebook

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

[X] I understand that I will be blocked if I intentionally remove or skip any mandatory* field

Checklist

[X] I'm reporting a bug unrelated to a specific site
[X] I've verified that I have updated yt-dlp to nightly or master (update instructions)
[X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
[X] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[X] I've searched known issues and the bugtracker for similar issues including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue

Provide a description that is worded well enough to be understood
Here is the logs for the error that it is thrown while trying to download videos from facebook
FATAL EXCEPTION: main
Process: com.random.facebookviddownloader, PID: 7725
Provide verbose output that clearly demonstrates the problem

[X] Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
[X] If using API, add 'verbose': True to YoutubeDL params instead
[X] Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output
Here is the logs for the error that it is thrown while trying to download videos from facebook

FATAL EXCEPTION: main
Process: com.random.facebookviddownloader, PID: 7725

java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:503)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:860)
Caused by: java.lang.reflect.InvocationTargetException
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:493)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:860)
Caused by: com.yausername.youtubedl_android.YoutubeDLException: ERROR: [facebook]<PHONE_NUMBER>86804: Cannot parse data; please report this issue on https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using yt-dlp -U

That's not a yt-dlp verbose log
Anyways this is a dupe of #4311
Don't use a VPN if you're using one, FB blocks those


CASMINST-5241: Update NCN CFS configurations in stage 0; remove now-redundant stage 5

Description
NOTE: I still need to do some testing of this procedure. I am opening this as a draft PR for the moment while I do testing.
This PR makes the following changes:
apply_csm_configuration.sh script

Adds the --no-enable flag. Normally the script creates an updated CSM configuration for all of the NCNs, disables the NCNs in CFS, re-enables them with the new configuration, and waits until they configure successfully. This flag has the script instead not re-enable them (but instead just updates them to use the new configuration, when they are later enabled).
Also change where the script keeps its files, so that they can be located and backed up during a CSM upgrade.

CSM upgrade
Stage 0
Direct the upgrader to follow the Getting Started Guide in most cases, except in the case where CSM is the only product installed on the system. This is only expected to be the case for internal installs. In that case, we instead direct the user to use the apply_csm_configuration.sh script (with the new flag) so that the NCNs will use the csm-1.3 configuration when they run CFS. This will prevent the errors reported in CASMINST-5241.
Stage 2
Modify the commands where the artifacts are copied from ncn-m001 to ncn-m002. Instead of copying the log files, now it creates a tar archive of them, and also pulls in the files from the apply_csm_configuration.sh script (if any -- it's fine if there are none). This archive is both uploaded into S3 and is copied to ncn-m002 (in case it is needed for more immediate reference).
Stage 5
This stage (NCN personalization) is now obsolete, between the GSG changes and the stage 0 changes made with this PR. This PR removes the stage. Stage 4 now proceeds to CSM health validation.
Checklist Before Merging

[X] If I added any command snippets, the steps they belong to follow the prompt conventions (see example).
[X] If I added a new directory, I also updated .github/CODEOWNERS with the corresponding team in Cray-HPE.
[X] My commits or Pull-Request Title contain my JIRA information, or I do not have a JIRA.

I'm marking this ready for review, but I am also going to add the do not merge tag on it, because I still need to do testing. I just want to make it available for review in the meantime.

/backport main

I tested the stage 0 changes (which included the updated scripts) on mug and verified that they work.

This PR is ready for merge. The main branch backport is still undergoing some edits.


Something is up with Aria2

Trying to download episodes using aria2 seems to yield alot of errors these days. Trying to download Magia Record from twist.moe yields me the error 403 message yet when I managed to remove Aria2 from the config.json, allowing me to use the slow internal downloader, I'm able to download anime episodes, albeit at a reduced rate.
I think this issue describes whats going on perfectly -> https://github.com/vn-ki/anime-downloader/issues/493

Known issue and the reason for https://github.com/vn-ki/anime-downloader/pull/498 however the reason for #498 not being merged yet is because of https://github.com/vn-ki/anime-downloader/pull/504, we have yet to decide on which the replacement is going to be.
As for aria2, I have yet to find a good reason for the sudden 403 errors across providers and devices. My best guess is that cloudflare pushed some new tech which detects aria2. Editing command line arguments doesn't seem to help hence I think it's some core issue unfixable by us.

Known issue and the reason for #498 however the reason for #498 not being merged yet is because of #504, we have yet to decide on which the replacement is going to be.
As for aria2, I have yet to find a good reason for the sudden 403 errors across providers and devices. My best guess is that cloudflare pushed some new tech which detects aria2. Editing command line arguments doesn't seem to help hence I think it's some core issue unfixable by us.

Roger that. Thank you for your hard work anyway. I'll close this


Use LabIcon in code-snippet package

Update the icon usage in code snippets to use LabIcons
This includes:

Moving the code-snippet logo into application
tweaking the svg viewBox to remove excess space around the icon
which make it properly fill the width off the icon in the css
replace all icons set with css with LabIcon instances
some minor icon css size tweaking

Fixes #443
Fixes #430
Developer's Certificate of Origin 1.1
   By making a contribution to this project, I certify that:

   (a) The contribution was created in whole or in part by me and I
       have the right to submit it under the Apache License 2.0; or

   (b) The contribution is based upon previous work that, to the best
       of my knowledge, is covered under an appropriate open source
       license and I have the right under that license to submit that
       work with modifications, whether created in whole or in part
       by me, under the same open source license (unless I am
       permitted to submit under a different license), as indicated
       in the file; or

   (c) The contribution was provided directly to me by some other
       person who certified (a), (b) or (c) and I have not modified
       it.

   (d) I understand and agree that this project and the contribution
       are public and that a record of the contribution (including all
       personal information I submit with it, including my sign-off) is
       maintained indefinitely and may be redistributed consistent with
       this project or the open source license(s) involved.

The tests actually consistently fail locally as well, I'm taking a look to figure out why

I've tracked down the cause: It's the same issue as has been raised in the git extension https://github.com/jupyterlab/jupyterlab-git/issues/604
I'm testing a work around but given this is now hitting us twice I'm going to open a PR on lab with my idea I suggested in the git pr

I've pushed a workaround fix for the tests and will be opening a PR on lab to fix it there since it's also happening to jupyterlab-git


Operator bug

This was originally part of #424 but I pulled it out because it seems to trigger an unrelated bug.
This PR passes all tests without OpenMP, but fails to compile with OpenMP.
~I believe this might be related to this bug~

I'm not sure I understand what's going on here and why is this related to openmp
also, was it on purpose to have both const and constant in the test?


AnimatedContainer does not work after build and deploy

This is the simplest implementation of AnimatedContainer. https://gist.github.com/casetofon/5bb75ad2aa54e93a5d9c171ffbca87ba
It works fine when running it in the IDE and it works fine on dartpat.dev.
Upon building the web project and deploying it, the screen turns grey for the duration of the animation. I tried it from multiple browsers and computers.
You can observe the bug live here: https://autonet-275416.firebaseapp.com/#/
To reproduce, build and deploy the code in the gist as web app.

I just upgraded Flutter and it works.


Add learn more footer section.

Make cards fully clickable on homepages
Remove unused classes from homepage cards
Add cta-learnmore section
Add Stack Overflow icon

The only concern I have here is "Talk to an expert" button label. This sends people to the same typeform found on chain.link, which is for businesses requesting help integrating and such. Coming from the docs though, developers may think this is a way to get 1-0n-1 technical help, instead of first heading to StackOverflow or Discord, and fill the type form with technical support questions.
How do we feel about labeling this button with something else, such as "Need integration support? Talk to an expert"?
@dwightjl @StephenFluin

How do we feel about labeling this button with something else, such as "Need integration support? Talk to an expert"?

Done! Great suggestion. Confirmed the change with the design and growth team.


[WIP] [BOW-113] Show/rundown metadata

Codecov Report
All modified lines are covered by tests :white_check_mark:

Comparison is base (1587ff4) 15.88% compared to head (2f07972) 15.88%.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #119   +/-   ##
=======================================
  Coverage   15.88%   15.88%
=======================================
  Files           5        5
  Lines         957      957
  Branches        2        2
=======================================
  Hits          152      152
  Misses        805      805

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Extract missing courses for course catalog vendor with no pagination (Clean Catalog)

I'm testing a new course catalog vendor - Clean Catalog.
Their pages have a consistent look with a "load more" button. Configuration + extraction was successful.
See examples below.
Bristol Community College - https://catalog.bristolcc.edu/classes

Configure + extraction successful: https://master.ai-course-crawler.development.c66.me/catalogues/43
Link to Google extract: https://docs.google.com/spreadsheets/d/1OY7oNMQ3DBgTyVqJOUF13MSvp9wR-hBYICFvqE7CEWo/edit?usp=sharing
Only the first page (up to the "load" more button was extracted. Extracted data look good!
Missouri Valley College - https://moval.cleancatalog.net/classes

Configure + extraction successful: https://master.ai-course-crawler.development.c66.me/extractions/53
No courses extracted.
Villinova University - https://live-villanova-catalog.cleancatalog.io/classes

Configure + extraction successful: https://master.ai-course-crawler.development.c66.me/extractions/55
No courses extracted.

Thanks for testing this.
There's #22 about this topic ('load more' pagination), but I'm not sure there will be an approach that will work generically for other catalogues with this kind of UI element.
That said, custom support for this catalogue should be simple. The "load more" buttons link to paginated URLs which we can extract in a loop until we run out of courses.


Enable coverage for subprocess and shutdown sever gently

Test (don't merge yet) to see if we get a better code coverage. Currently some part of the code is reported as uncover because it is run within a sub process.

Those modification allow to have the right coverage from coverage.py, even for luigi/mrrunnner.py that is run on the minicluster. However, there seems to be some using with the way coveralls-python export the coverage.

With this PR the coverage should now be properly reported in coveralls.

@erikbern it might interest you:
The way the sever was shutdown was an issue to get a proper coverage. But I think it was also the source of the false positive in Travis. The shutdown_handler was registered with the atexit and shutdown_handler was calling os._exit(0). So nose was always exiting with 0.

Oh that's dangerous. Makes sense.
Is this ready to be merged?

The thread shouldn't shutdown the entire process because shutdown_handler will not be called within the thread. However, shutdown_handler was call at the end of the process during the cleanup part. By calling os._exit(0), it prevents any further clean up (close socket, flush buffer...) and force the return code to be 0 even if there was an error.
This PR is ready to be merged.

84% coverage now, yeah!


App crash against Kubernetes 1.20

Describe the bug
Lens displays App crash after connecting to a Kubeadm based 1.20.0 cluster for a minute or so.
To Reproduce
Steps to reproduce the behavior:

Upgrade a Kubernetes cluster from 1.19.4 to 1.20.0
Open Lens.
Do nothing, or click a few things, or wait for about 1 min.
See App Crash

Environment (please complete the following information):

Lens Version: 4.0.0
OS: Debian 10.7 Buster
Installation method AppImage

Logs:
When you run the application executable from command line you will see some logging output. Please paste them here:
Error stack:

App crash at http://02bceedc-95b4-4c44-89dd-435741bb4e51.localhost:34031/cluster
 TypeError: Cannot read property 'kind' of undefined
    at r.value (http://02bceedc-95b4-4c44-89dd-435741bb4e51.localhost:34031/build/Lens.js:2:3776353)
    at http://02bceedc-95b4-4c44-89dd-435741bb4e51.localhost:34031/build/Lens.js:2:8680755
    at Array.map (<anonymous>)
    at http://02bceedc-95b4-4c44-89dd-435741bb4e51.localhost:34031/build/Lens.js:2:8680723
    at Array.map (<anonymous>)
    at r.value (http://02bceedc-95b4-4c44-89dd-435741bb4e51.localhost:34031/build/Lens.js:2:8680667)
    at r.value (http://02bceedc-95b4-4c44-89dd-435741bb4e51.localhost:34031/build/Lens.js:2:8686015)
    at ve (http://02bceedc-95b4-4c44-89dd-435741bb4e51.localhost:34031/build/Lens.js:2:18448)
    at http://02bceedc-95b4-4c44-89dd-435741bb4e51.localhost:34031/build/Lens.js:2:70625
    at ce (http://02bceedc-95b4-4c44-89dd-435741bb4e51.localhost:34031/build/Lens.js:2:16554)

Kubeconfig:
Hasn't changed since before the app crashes started
Additional context
Just upgraded the "bare metal" cluster with kubeadm to 1.20.0.  Cluster came up healthy, opened Lens 4.0.0 everything seemed fine, all things look normal. After about 60 seconds, App crash

Lens just updated to 4.0.1 (Nice work with the AppImage updates on Linux).
Still crashes anywhere between 10-60 seconds of use.
ctrl-r (reload) is a good workaround.

Can confirm for Windows, Lens 4.0.1

Same here

same issue here.

same issue here.

+1 Lens 4.0.3, k8s 1.20

same issue here.
+1 Lens 4.0.3, k8s 1.20

Fixed in #1804


Prioritise UITapGestureRecognizer on interop views with Cooperative interop mode. Prioritise UIScreenEdgePanGestureRecognizer on ascendant views.

Before that change taps on interop views with cooperative mode would be discarded by compose, which expects non-discrete gesture holding on a point before failing and redirecting touches sequence to the interop view.
The requirement of holding ultimately made the tap gesture impossible due to its requirement of being a short press/release sequence, which contradicts the compose heuristics.
This scenario is fixed in this PR by allowing precedence of UITapGestureRecognizer over other recognisers.
Since most of the native tap/double tap interactions are built on top of UITapGestureRecognizer it should improve the UX for default interop properties with stable constructor setting the interaction mode to be Cooperative.
Coincidentally this change inspired a cheap fix for interactive pop bugs.
Resolves https://youtrack.jetbrains.com/issue/CMP-6683
Fixes https://youtrack.jetbrains.com/issue/CMP-6622/Gestures-dont-work-on-native-ios-view-embedded-with-UIKitView-for-compose-1.7.0-beta01
Demo:
https://github.com/user-attachments/assets/8b679ff0-6edd-42c3-9839-fe4e14cc4dff
Testing

Taps on interop views with cooperative interaction mode are properly registered now and processed by native UITapGesture recogniser attached to that view.
Interactive pop on UINavigationController should not get stuck in an awkward state preventing the gesture.

This should be tested by QA
Release Notes
Fixes - iOS

Taps should be properly registered on interop views with UIKitInteropInteractionMode.Cooperative interaction mode.
Interactive pop on UINavigationController should recognize correctly.

Unfortunately, it fixes the swipe back only partially. STRs:

Create UINavigationController with some rootViewController
Push Compose View Controller
Slowly swipe back, and cancel gesture, by returning to the the current Compose View Controller
Perform some interaction with Compose View Controller
Try to swipe back again
Result: Swipe back does not work.

Because the issue considered as critical, and the fix make the overall experience better, I think it worth adding it to the 1.7.2 release. Also, due to complexity of the interop gestures, I would not go with dramatical changes in the frature's code.
The issue should be fixed in 1.8.0.


Update tidis-introduction.md

Some minor changes

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Add post selection gates and update simulators to support these gates

As a sort of post-selection. The recent change to simulator broke this test. Not sure how to fix it given the new interface. @dabacon ideas?
        result = simulator.simulate(
            circuit,
            qubit_order=[ancilla_qubit] + list(system_qubits),
            initial_state=current_state,
            repetitions=reps)

        # Determine the bit by majority vote
        num_ones = numpy.count_nonzero(result.measurements['ancilla_qubit'])
        num_zeros = reps - num_ones
        last_measured_bit = num_ones > num_zeros

        # Get one of the resulting states with the correct bit measured
        current_state = result.final_states[
                numpy.where(result.measurements['ancilla_qubit'] ==
                            last_measured_bit)[0][0]]

The elegant solution is a gate which the simulators understand.  Let's make this the issue to track that.

It's now possible to specify these gates using the _act_on_ protocol.

@kevinsung Is the idea here that we want to specify ahead of time that you only care about when measurement m1 is 1, so retry until that happens?
If that's the case we have the ability to do that very generically now. All simulators support act_on, using their specified subclass of ActOnArgs as the simulator state. (i.e. density matrix simulator uses DensityMatrixActOnArgs, which contains a density matrix and various metadata).
Any class implementing ActOnArgs is required to implement the copy method, which is a deep copy (except for the RNG), so you can think of it as a snapshot of the state that can be used for repetitions etc. All ActOnArgs classes also have to implement the perform_measurement method.
So we could augment MeasurementGate with a new parameter required_value or something, and update MeasurementGate._act_on_ to call c = args.copy(); c.measure() until it gets the desired measurement. We'd then just need one new abstract function on ActOnArgs to read the resulting state from c back into the original. Should be pretty straightforward. This would then light up this feature for all simulators.
(Assuming that's what's wanted here, and if this feature is still desired.)
cc @95-martin-orion @smitsanghavi @balopat

In my opinion, the "correct" way to implement post-selection in simulation is with a new gate whose _act_on_ method simply applies the |0)(0| or |1)(1| operator without checking it against the existing state. This requires no trial-and-error, and (assuming we don't normalize the state after) allows us to determine what percentage of runs we "discarded" to get that result. I'd prefer not to use MeasurementGate for this, as it's expected to function on both simulators and hardware.
As Craig notes, it's possible to manually construct these gate types, but I think having built-in versions in Cirq could be worthwhile. For extensibility, I'd suggest having a base type that accepts any set of non-unitary operators as well as an implementation of that base type which performs post-selection in the Z basis (i.e. with |0)(0| or |1)(1|).
(As an aside: the "correct" way to do post-selection in hardware would be a while-loop around the entirety of a circuit up to the post-selected measurement, with the loop conditioned on the result of that measurement. While-loops are part of #3234.)

Makes sense. As soon as I posted that I realized there's probably a deterministic operation.

Bumping this to say this would still be really cool to have for simulator-optimization, but in lieu of that the repeat-until introduced in #5018 can achieve this (albeit at a much slower pace).


perf: stick with SFC as long as possible

@neoziro: this is still work in progress but I wanted to share it to have some initial feedback.

I benchmarked this "new" mapProps against the current mapProps and it looks like it doesn't bring any performance benefits so I'm closing this issue.

@matthieuprat good approach ;)


When youre using your derivative methods, you can replace the diagonal arrays across the loops! this way you go from a O(n^2) method each time to a O(n) on average, with a cold start or a worse case of O(n^2)

change your softmax derivative function from
    def derivative(self, dout: Tensor) -> Tensor:
        if dout and not dout.order == 2:
            raise ValueError('Softmax derivative only for tensors of order 2')

        num_samples, num_classes = self.x.shape
        grad = Tensor([], (num_samples, num_samples), init='zeros')
        for n in range(num_samples):
            sm = Tensor.diagflat(self.x.tensor[n])
            outer_prod = Tensor(self.x.tensor[n], (num_classes, 1))
            grad = grad + (sm - (outer_prod @ outer_prod.T))
        return grad if not dout else dout.T @ grad

to
    def derivative(self, dout: Tensor) -> Tensor:
        if dout and not dout.order == 2:
            raise ValueError('Softmax derivative only for tensors of order 2')

        num_samples, num_classes = self.x.shape
        grad = Tensor([], (num_samples, num_samples), init='zeros')
        sm = Tensor.zeroes(n, n) # idk if this exists
        for n in range(num_samples):
            sm = Tensor.diagflat(self.x.tensor[n], zeroes=sm) # once you alter this method
            outer_prod = Tensor(self.x.tensor[n], (num_classes, 1))
            grad = grad + (sm - (outer_prod @ outer_prod.T))
        return grad if not dout else dout.T @ grad

closed with ea838c65c389e1604110cb720222a5c14ba97495


Typo on README.md

can take from seconds got maybe even hours

to

can take from seconds to hours

thank you @joshmiller83 ! :)


Update gitleaks.toml to support gitleaks 4

What does this PR do?
This updates .gitleaks.toml to support the new config changes in gitleaks v4.  It is done in a backwards compatible way with v3 and v2 until everyone has upgraded to v4.  This will require whitelisted values or files to be update in two locations, but is the only way to support both versions.

@ismarc Do you mind rebasing this branch - seems like a good addition but it went stale vs master.


Fix Smallest Function

There is currently a method called smallest that lives in /lib/smallest.js.
It is incomplete and needs to be fixed!
The smallest functions takes in an array and an ordering function as parameters, and returns the smallest item in the array.
For example:
 expect(largest([5, 4, 1, 3, 8], (a, b) => a < b)).toBe(1);
Acceptance Criteria:

Running yarn test <methodName> results in tests passing.
You have written a skeleton method for someone else to work on.
You have written tests surrounding your skeleton method.
Running yarn lint does not print any errors to the console!
Optional: write one or two more tests surrounding the method you are solving to account for potential edge cases.

Please include the skeleton of a new method + an accompanying test for someone else to work on at the time of creating a pull request! A pull request will most likely be denied if it does not contain a skeleton method for someone else to work on! For more information, please read the Contributing Guide.
Thank you so much for your contribution!

I have started on this issue.


update permissions

Description
Fix error on aws permissions
Issues Resolved
[List any existing issues this PR resolves]
Contribution Check List

[ ] All tests pass.
[ ] New functionality includes testing.
[ ] New functionality has been documented in the README if applicable
[ ] New functionality has been documented in CHANGELOG.MD

          1 Error

:no_entry_sign:
Please add labels to this Pull Request

  Generated by :no_entry_sign: Danger


Resend poller not differentiating between URL's with trailing slash (causing infinite 'resend' requests)

The resend poller isn't differentiating between "http://localhost:9001" and "http://localhost:9001/" (with the trailing slash that was added for constellation).
In the event that auto-discovery is disabled, then this has the impact of causing a "resend" request every 2 seconds (ad-infinitum).
To reproduce:

Set up two nodes, with both configured as peers:

    "peer": [ {"url": "http://localhost:9001" }, {"url": "http://localhost:9002"} ],

On node 1, disable auto-discovery:

    "disablePeerDiscovery": true,

Start up both nodes.
On examination of the logs on node 2, you will observe the following every 2 seconds: "Enter Request : POST : /resend"

Temporary workaround:

Add a backslash to the peer url's in the config, e.g.:

    "peer": [ {"url": "http://localhost:9001/" }, {"url": "http://localhost:9002/"} ],

Although the resend poller doesn't differentiate, it no longer causes an infinite loop, as as such the main issue is fixed.


Build break: no-engine

Configure no-engine tries to build several engines:
From ntdll.mak
E_SHLIB=$(ENG_D)\padlock.dll
$(ENG_D)\capi.dll $(ENG_D)\dasync.dll
The build will then fail with this error:
e_padlock.c
openssl\include\openssl/engine.h(70): fatal error C1189: #error:  ENGINE is disabled.
On a related note, no-capieng does not seem to have any effect.

Hopefully, we'll get unified builds for native windows real soon (in review for the moment).  I've already lined up the required fixes to support those config options with the unified build scheme.  Doing the same in util/mk1mf.pl is...  painful.

No longer relevant.


[BEAM-9402] Remove options overwrite

The option does not allow addition of user provided option like in case of validates runner tests.
This means that the test always tries to run against DirectRunner.
Further the streaming flag can be provided by the target which is executing this test.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

R: @HuangLED @rohdesamuel

Friendly ping @HuangLED @rohdesamuel

The option was there to make sure the test case runs through streaming mode. I don't understand why this causes running against direct runner always.
otherwise LGTM though.

The issue was that the options are either interpreted from the command line or from the test if passed explicitly.
My marking the pipeline streaming in the test also means that the pipeline will run as streaming even when running batch validates runner so I think its better to remove the string flag from the test.
Thanks!

Run Python2_PVR_Flink PreCommit

Run Python PreCommit

Run Python PreCommit

Thanks for the fix.  LGTM

Thanks!


AWSRioEnv init throws error when credentials are not returned from botocore

There is no guarantee that botocore will return a credentials object when boto3.Session.get_credentials() is called.
see: https://github.com/boto/botocore/blob/ddc8d9acdb67598c5a15b3e56eec8ff023993e30/botocore/credentials.py#L1655
This can cause errors when AWSRioEnv is constructed with None as the credentials object.

@roarmstrong I have pushed an attempt at a fix to stable, basic idea is to retry up to 10 times with some sleep in between. Also added assert in AWSRioEnv, setup_local_env will raise IOError if credentials can not be loaded after 10 attempts.


Update to redis 0.26

Changed from Value::Data to Value::BulkString, due to the change in redis 0.26.
However, this is not compatible with the released Deadpool redis, so this version is pointing to master with the commit:
https://github.com/bikeshedder/deadpool/pull/342. Should be changed when released.

Thanks for looking out for the library, and making it compatible with the newest redis-rs release. It is a big help for me.
Everything looks good on your PR, but I'd be waiting on the deadpool-redis release (thank you for linking the corresponding issue). Until then, if you don't mind, can you please revert the version bump and the Changelog changes? I like to do releases from the master branch. I'll add your message and name to the Changelog when I release it. Thanks a lot!

deadpool-redis 0.16.0 just released! (cf. changelog)

Thanks for the notification, I'll merge this and put out a release.


Issues compiling hello world

After running the install steps with cmake from the README's building steps, i am met with the below errors:
/usr/bin/ld: /tmp/ccb5Bytw.o: in function `main::{lambda(served::response&, served::request const&)#1}::operator()(served::response&, served::request const&) const':
main.cpp:(.text+0x76): undefined reference to `served::response::operator<<(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
/usr/bin/ld: /tmp/ccb5Bytw.o: in function `main':
main.cpp:(.text+0x11a): undefined reference to `served::multiplexer::multiplexer()'
/usr/bin/ld: main.cpp:(.text+0x192): undefined reference to `served::multiplexer::handle(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)'
/usr/bin/ld: main.cpp:(.text+0x1b6): undefined reference to `served::methods_handler::get(std::function<void (served::response&, served::request const&)>)'
/usr/bin/ld: main.cpp:(.text+0x2d3): undefined reference to `served::net::server::server(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, served::multiplexer&, bool)'
/usr/bin/ld: main.cpp:(.text+0x328): undefined reference to `served::net::server::run(int, bool)'
/usr/bin/ld: /tmp/ccb5Bytw.o: in function `served::net::server::~server()':
main.cpp:(.text._ZN6served3net6serverD2Ev[_ZN6served3net6serverD5Ev]+0x2e): undefined reference to `served::net::connection_manager::~connection_manager()'
collect2: error: ld returned 1 exit status

seems like it is a linker error? i am compiling with g++ -std=c++11 main.cpp -Iinclude -Iserved/src/served -lboost_system -lpthread from the directory i cloned the served project into.
i will try building with blaze tomorrow to see if this resolves it

Dont forget to link against libserved. To have it you must build and install served. Also for some reason my application did not start so I had to do this as well:
ln /usr/local/lib/libserved.so.1.4 /usr/lib64/libserved.so.1.4

I encountered the same problem when compiling. Can you provide us with a standard compilation format。
i am compiling with clang++-9 a.cpp -std=c++11  -lboost_system

Dont forget that you must build and install served. Once you have done that you must link against libserved.so.


avro-extensions -- feature to specify avro reader schema inline

in the task json for all events.
See https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/druid-development/tg2q3VHN5rQ/5kUuiTouEQAJ

This is something we need in our set up. Thanks Himanshu. Which release is this expected to be a part of ?

This should be reasonable to be in 0.9.2, and if there's a community need even more so

@skoppar do you have single schema for all the events in the kafka topic or multiple ? is it possible for you to test this patch?
are you already trying to use avro extension with schema_repo and facing some error?

LGTM, 👍

updated the docs.

Hi Himanshu,
I havent tried Schema repo and was about to extend the AvroBytesDecoder myself and thats when I saw your comment on another google thread. https://groups.google.com/forum/#!searchin/druid-user/avro$20druid|sort:relevance/druid-user/72NKnm6eBRY/VCgyaTKEAwAJ

Hi Himanshu,
I am not able to get the avro stream to work with the new jar. I have
packaged your changes which produced
"druid-avro-extensions-0.9.2-SNAPSHOT.jar".
I moved the original jar into a subdirectory so that there are no conflicts
and copied the snapshot jar in current directory (imply-1.3.0/dist/druid/
extensions/druid-avro-extensions). Also changed imply-1.3.0/conf-
quickstart/druid/_common/common.runtime.properties to include avro library:
druid.extensions.loadList=["druid-avro-extensions", "druid-histogram",
"druid-datasketches", "druid-kafka-indexing-service"]
I've modified kafka.json in imply-1.3.0/conf-quickstart/tranquility to the
one attached.
Upon starting imply:
bin/supervise -c conf/supervise/quickstart.conf
I do not see any topics being picked in the tranquility logs
[imply-1.3.0/var/sv/tranquility-kafka.log]:
2016-08-08 17:27:56,913 [main] INFO  k.c.ZookeeperConsumerConnector -
[cust1-tranquility_skoppar-mac15.com-1470677276325-37ef097f], Creating
topic event watcher for topics ((?!))
2016-08-08 17:27:56,929 [main] INFO  k.c.ZookeeperConsumerConnector -
[cust1-tranquility_skoppar-mac15.com-1470677276325-37ef097f], Topics to
consume = List()
Doesn't it mean that no topics have been registered?
Only relevant entries in the log are:
2016-08-08 17:27:56,151 [main] INFO  c.m.t.kafka.writer.WriterController -
Ready: [topicPattern] -> dataSource mappings:
2016-08-08 17:27:56,152 [main] INFO  c.m.t.kafka.writer.WriterController -
[(?!)] -> AvroSource (priority: 1)
I am new to Druid, so please pardon my ignorance. Is the
publicAvrokafka.json file right? At this point I havent added any business
functionality. The derived fields are coming from an ETL process outside of
Druid. I just want to be able to ingest avro data through kafka and that
doesnt seem to work. To be sure its not a kafka issue, I have tried using
kafka-console-consumer and I can receive events there.
regards
Sunita
On Fri, Aug 5, 2016 at 7:35 AM, Himanshu<EMAIL_ADDRESS>wrote:

updated the docs.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/druid-io/druid/pull/3249#issuecomment-237867062, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ARAoyRcBJ9n4QvVBCCUoH05HJTDWJh38ks5qc0olgaJpZM4JNFH8
.

@skoppar i haven't really used kafka with tranquility and information provided does not look like avro parsing related failures.
i don't see any kafka topic etc input information in your spec , not sure how you provide it to tranquility... @gianm can you spot something?

Thanks for quick response Himanshu.
From what I understand,
"properties": {
"topicPattern": "raw-cust1"
}
Provides the topic which is per datasource level
and below properties provide overall cluster connection. Adding
group_id to data source specs might
be more advisable, but I haven't tried that yet. I am using all default ports.
"kafka.group.id":"cust1-tranquility",
"zookeeper.connect":"localhost",
"kafka.zookeeper.connect":"localhost"
regards
Sunita
On Mon, Aug 8, 2016 at 11:47 AM, Himanshu<EMAIL_ADDRESS>wrote:

@skoppar https://github.com/skoppar i haven't really used kafka with
tranquility and information provided does not look like avro parsing
related failures.
i don't see any kafka topic etc input information in your spec , not sure
how you provide it to tranquility... @gianm https://github.com/gianm
can you spot something?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/druid-io/druid/pull/3249#issuecomment-238337370, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ARAoycaGfewz_18wXsiKchgnJC_MDnRYks5qd3mygaJpZM4JNFH8
.

Another fact is, Zookeeper is able to recognize the consumer group and the
group is visible upon executing zookeeper shell. Also from tranquility
side, zookeeper connection seems to be present. The Kafka broker connection
is what seems to be missing and the topic list is empty as mentioned in the
previous comment. Appreciate any help  @gianm https://github.com/gianm
On Mon, Aug 8, 2016 at 12:14 PM, Sunita Koppar <
<EMAIL_ADDRESS>wrote:

Thanks for quick response Himanshu.
From what I understand,
"properties": {
"topicPattern": "raw-cust1"
}
Provides the topic which is per datasource level
and below properties provide overall cluster connection. Adding group_id to data source specs might
be more advisable, but I haven't tried that yet. I am using all default ports.
"kafka.group.id":"cust1-tranquility",
"zookeeper.connect":"localhost",
"kafka.zookeeper.connect":"localhost"
regards
Sunita
On Mon, Aug 8, 2016 at 11:47 AM, Himanshu<EMAIL_ADDRESS>wrote:

@skoppar https://github.com/skoppar i haven't really used kafka with
tranquility and information provided does not look like avro parsing
related failures.
i don't see any kafka topic etc input information in your spec , not sure
how you provide it to tranquility... @gianm https://github.com/gianm
can you spot something?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/druid-io/druid/pull/3249#issuecomment-238337370, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ARAoycaGfewz_18wXsiKchgnJC_MDnRYks5qd3mygaJpZM4JNFH8
.

@skoppar seems like you have specified the properties at wrong place in json spec file,
the properties needs to be defined outside the spec file as done here -
https://github.com/druid-io/tranquility/blob/master/distribution/src/universal/conf/kafka.json.example

Thanks Nishant for bringing it to my notice.
Yea I see there is some discrepancy in the spec now. Will check that and
update this space
On Tue, Aug 9, 2016 at 10:24 AM, Nishant<EMAIL_ADDRESS>wrote:

@skoppar https://github.com/skoppar seems like you have specified the
properties at wrong place in json spec file,
the properties needs to be defined outside the spec file as done here -
https://github.com/druid-io/tranquility/blob/master/
distribution/src/universal/conf/kafka.json.example
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/druid-io/druid/pull/3249#issuecomment-238626096, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ARAoyR_oyh5xb9GKE0Aimiqdbj8DTUhYks5qeLfWgaJpZM4JNFH8
.

Hi Himanshu,
Looks like there is a dependency on io.druid.druid-api-0.9.1.jar based on the errors I see. imply-1.3.0/dist/tranquility/lib has a huge list of jars. I hope I replace fewer jars. Are there any documented dependencies? Attaching the exception I get now.
WithPR.txt

@skoppar what is the tranquility config you are using now?

@gianm attaching the current tranquility-kafka json file.
publicAvroKafka.txt

@skoppar @gianm given that exception came from StringInputRowParser , I believe avro_stream parser did not get recognized and that probably means avro-extensions module wasn't loaded.  did you see any avro-extension loading failures or a message regarding loading of avro-extension in the logs?

@himanshug doesnt look like loading the jar is the issue. I could be wrong though. Attaching the indexing-log in imply-1.3.0/var/druid/indexing-logs which shows the druid-avro-extensions-0.9.2-SNAPSHOT.jar being added. Attaching the indexing log. Please let me know if I should be looking at a wrong place.
index_realtime_metrics-kafka.txt

@skoppar one of the problems in your json is that "parseSpec" is inside "avroBytesDecoder" while it should be a level up inside "parser" .

oops ! let me relook at the complete json
On Tue, Aug 9, 2016 at 1:14 PM, Himanshu<EMAIL_ADDRESS>wrote:

@skoppar https://github.com/skoppar one of the problems in your json is
that "parseSpec" is inside "avroBytesDecoder" while it should be a level up
inside "parser" .
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/druid-io/druid/pull/3249#issuecomment-238676376, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ARAoyXAMybK3Ysq-K-Ou47LgWSrF7UY4ks5qeN-YgaJpZM4JNFH8
.

Its working now. 2 more changes I had to do was:

Specify the format in parseSpec
Add the path to druid extensions manually in quickStart.conf

With that done to the attached conf, I am able to process records with schema inline.

@skoppar glad to know that it worked out for you and thanks for testing the patch.

👍

Well not completely. Trying to figure out why segments are not getting
created. But the patch did work well. This must be something else I am
missing. I have 800 messages received and unparseableCount=800. I tried
changing timestamp to be current, but still get the same results.
Appreciate any inputs you have on this, however, it does not seem to be
related to the changes in the PR.
On Aug 10, 2016, at 10:47 AM, Himanshu<EMAIL_ADDRESS>wrote:
@skoppar https://github.com/skoppar glad to know that it worked out for
you and thanks for testing the patch.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/druid-io/druid/pull/3249#issuecomment-238946405, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ARAoyTP1qdVFbw4wClZT9FgayJfKcbtOks5qeg7BgaJpZM4JNFH8
.

Well not completely. Trying to figure out why segments are not getting
created. But the patch did work well in terms of pulling the schema and
getting right count of events. This must be something else I am missing. I
have 800 messages received and unparseableCount=800. I tried changing
timestamp to be current, but still get the same results. Appreciate any
inputs you have on this, however, it probably is not related to the changes
in the PR.
On Aug 10, 2016, at 10:47 AM, Himanshu<EMAIL_ADDRESS>wrote:
@skoppar https://github.com/skoppar glad to know that it worked out for
you and thanks for testing the patch.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/druid-io/druid/pull/3249#issuecomment-238946405, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ARAoyTP1qdVFbw4wClZT9FgayJfKcbtOks5qeg7BgaJpZM4JNFH8
.

@skoppar enable debug logging and that would show you more info.

Figured it out with the help of a kafka consumer class. The console consumer showed data just fine, so couldn't spot the error. The rootcause is, I was sending the avro message using StringEncoder. Using DefaultEncoder on producer side fixed the issue. Thanks @himanshug and @gianm. Awaiting 9.2 release :) 👍


Release 2.2.0 seems to contain <IP_ADDRESS>

elasticsearch-jdbc release 2.2.0 seems to contain <IP_ADDRESS> not 2.2.*?
Where as master seems to contain the 2.2.* version for Elasticsearch 2.2.
I would prefer to use a release rather than master.
When will a 2.2.* release be created for elasticsearch-jdbc?

:+1:  I have exactly the same issue.

yeah I am having this problem too

Thanks for the notice, yes, I have been aware of messing things up. Sorry!
I pushed some more commits and released a fix version <IP_ADDRESS>, nothing fancy added, only a new build. I'm trying to move to a pure gradle build.
Quick download link:
http://xbib.org/repository/org/xbib/elasticsearch/importer/elasticsearch-jdbc/<IP_ADDRESS>/elasticsearch-jdbc-<IP_ADDRESS>-dist.zip

Ok thanks, will take a look

Thanks. Looks good! :+1:


Add Cryptocurrency Convertor

Is your feature request related to a problem? Please describe.
Users often need to convert between cryptocurrencies and fiat currencies for various transactions and financial decisions. Currently, they have to use external sites or apps to get these conversion rates, which is time-consuming and inconvenient. Additionally, relying on third-party sources may lead to inconsistent or outdated exchange rates, causing potential inaccuracies in transactions.
Describe the solution you'd like
Integrate a cryptocurrency converter into the platform to provide real-time exchange rates and conversion capabilities directly within the interface. This feature will streamline the process, ensuring users have quick, accurate, and convenient access to the latest conversion rates.

Kindly assign to me ASAP.

Closing this as exists #156 #1680


Homebridge discovering device then crashing while trying to add

I own a Kogan SmarterHome Air Conditioner that can both heat and cool. I have trued adding it to HomeKit through this plugin but then when it discovers the device my local area network it proceeds to crash homebridge. Here is the link to the air conditioner: https://www.kogan.com/au/buy/kogan-smarterhome-41kw-portable-air-conditioner-14000-btu-reverse-cycle/?openModal=2
Here is the log from homebridge:
`[5/12/2019, 8:32:13 PM] [TuyaLan] Discovered AC (71645600b4e62d215921)
C:\Users\xxxxx\AppData\Roaming\npm\node_modules\homebridge-tuya-lan\node_modules\node-forge\lib\aes.js:203
var len = tmp.length();
^
TypeError: tmp.length is not a function
at forge.aes.Algorithm.initialize (C:\Users\xxxxx\AppData\Roaming\npm\node_modules\homebridge-tuya-lan\node_modules\node-forge\lib\aes.js:203:19)
at new forge.cipher.BlockCipher (C:\Users\xxxxx\AppData\Roaming\npm\node_modules\homebridge-tuya-lan\node_modules\node-forge\lib\cipher.js:118:18)
at Object.forge.cipher.createCipher (C:\Users\xxxxx\AppData\Roaming\npm\node_modules\homebridge-tuya-lan\node_modules\node-forge\lib\cipher.js:42:10)
at new TuyaAccessory (C:\Users\xxxxx\AppData\Roaming\npm\node_modules\homebridge-tuya-lan\lib\TuyaAccessory.js:28:36)
at EventEmitter.TuyaAccessory.discover.on.config (C:\Users\xxxxx\AppData\Roaming\npm\node_modules\homebridge-tuya-lan\index.js:77:32)
at EventEmitter.emit (events.js:182:13)
at Socket.server.on.msg (C:\Users\xxxxx\AppData\Roaming\npm\node_modules\homebridge-tuya-lan\lib\TuyaAccessory.js:343:24)
at Socket.emit (events.js:182:13)
at UDP.onMessage [as onmessage] (dgram.js:628:8)`
here is my config.json file:
`{
"bridge": {
"name": "Homebridge",
"username": "xx:xx:xx:xx:xx",
"port": 51826,
"pin": "xxx-xx-xxx"
},
"description": "This is an example configuration file.",
"platforms": [{
"platform": "TplinkSmarthome",
"name": "TplinkSmarthome"
},
{
"platform": "TuyaLan",
"devices": [{
		"name": "AC",
		"type": "AirConditioner",
		"manufacturer": "Kogan",
		"model": "SmarterHome Air Conditioner",
		"id": "71645600b4e62d215921",
		"key": "--------key---------"
	}
	]
}

],
"accessories": [{
"accessory": "Xbox",
"name": "Xbox",
"ipAddress": "xx.x.x.x",
"liveId": "xxxxxxxxxxxx"
}]
}`

#2

@AMoo-Miki can you have a look into this and try and see why it’s not working

I looked at node-forge code and it looks like it is not liking your key.

Update your plugin to the latest version with npm i -g AMoo-Miki/homebridge-tuya-lan as I have updated the plugin's dependencies to use a newer version of node-forge.
Count the number of characters in your key; the valid lengths are 16, 24, and 32. If yours is none, please go through the process of obtaining the id and key again.

Lemme know how it goes.

Please give the latest rc release a shot by doing npm i -g homebridge-tuya-lan@rc. It includes logic to understand this newer API.
A known issue is that the current state is not reported by some devices; they however work fine as soon as you interact with them. I have reached out to Tuya for a solution.
The Setup Instructions have changed in case you need to obtain a fresh id and key.

Ok, thanks. I will have a look at this know.

Just a quick update. I have just used the command that you provided and it has discovered the device however homebridge crashes not long after discovering the device and is giving an Key Error saying that my key is too long. I will try and get and updated key and id now and report back.

@TheHomekitWayFinder, regarding Tuya and Homekit, that is what they told me. Let's hope it is not limited to Siri Shortcuts. Their implementation would save us from all these hoops.

When I try to get the key for my air conditioner, tuya-lan-find starts to print out somethings and then crashes

If it shows garbage, it is an indicator of a bad certificate. Delete  any previous NodeMITMProxyCA certificates you might have on the phone before you start tuya-lan-find. I haven't had anyone else report using tuya-lan-find yet but I am sure there would be edge cases that need handling.
Regarding Homebridge crashing, can you get your hands on some logs to see what's causing it?

I ran it on Windows and it didn't give me any trouble. It shows many "socket hang up" errors but those can be ignored. I will suppress them soon.

Unfortunately, it never seems to produce any log files and all I have is a picture
of it crashing and saying that it has an invalid key length. I will try and run tuya-lan-find again after deleting the certificates again.
Here is the photo of the error:

Is it possible that you are adding the key and id inverted?

No, as in the app id now shows the Virtual ID and i have compared the two and they are both the same so im presuming it's just that the key is wrong.

Can you please update with npm i -g homebridge-tuya-lan@rc and give it shot?

I have just run tuya-lan-find again and it gives lots of socket hang up errors but it doesnt give the id or key of my air conditioner. I have also just today purchased a Arlec 4 outlet power strip and that uses the tuya protocol and therefore i need to get this working so that i can add that to homekit.
I will run the update and give it another shot in a sec.

The update still isnt working. Unfortunately

it crashed? How long is your key?

yep it crashed again. in my config file my key is 16 characters long however it was 17 before so I believe that my key may be wrong.
Version of
tuya-lan-find is 1.5.0-rc.4
node is 10.15.0
homebridge is 0.4.50

netsh advfirewall firewall add rule name="MITM Proxy" dir=in action=allow protocol=TCP localport=8080

im still getting the socket hang up errors and nothing else so i dont know whats happening. I used that command to change the firewall but its still giving me the same result

While these socket timeouts are shown, can you please kill the Tuya app on your phone and open it again? Please make sure you kill it and not just minimize it.
PS, I just pushed v1.5.0-rc.5 to NPM to suppress the socket hang up messages.

No, i am using the kogan smarterhome app. i am making sure that i kill the app by double clicking the home button and swiping up on the app

This will only work with the Tuya Smart app to fetch the id and key.

ok i will try that now

Oops, and I guess if you add it to Kogan, you will have to reset it again which most probably changes the id and key.

there are 2 instances of the tuya.m.my.group.device.list do i change both of them

yes please.

it still isnt pulling the key and id

I see, give me a second. Lemme cook up something.

yes it did, all of the same socket hang up errors.

Download this file, unzip, and place kogan.js next to the cli.js file. To run it, open cmd in this folder and type node kogan.js.
kogan.zip

it doesnt have any of the socket hang up errors like before but it does have a connect ENOBUFS and connect EADDRINUSE but its still not giving me the key and id of the air conditioner

Where was EADDRINUSE ?

Can you delete the cert from your phone and scan the QR code from the Kogan.js script?

I deleted the certificate from the cli.js script before I scanned the QR code from the Kogan.js script. But I can’t do anything at the moment as I have just gone out
Sent with GitHawk

I understand. I will think a bit to see what we can do about this and post back.

Do you want me to see if I can source the signature some how?
Sent with GitHawk

The key and signature, yeah.

What about MITMproxy that has a web interface like anyproxy did
Sent with GitHawk

You mean this? I just tested it on Windows and it is great.
After you install it, change the proxy settings on your phone to your Windows IP and 8080. Then open mitm.it on your phone to install the certificate; the process is identical to this plugins.

I have just tried to reset the device and add it to the TuyaSmart app, however the device keeps timing out before the app has finished pairing. But when I go back into the Kogan SmarterHome app it pairs very quickly
Sent with GitHawk

I just tried using Mitmproxy after adding the air conditioner back to the Kogan SmarterHome app and it would pull up any details to do with it. It pulls up lots for websites but nothing for apps. The same happens when I try to get the id and key of my Arlec Grid Connect Power Strip.
Sent with GitHawk

Do u have a Mac?

Yes I do
Sent with GitHawk

I guessing this is out of date but, https://gist.github.com/aceslick911/0cac96ab122fa9c6cd9ca18a696ea961

Thanks. Sadly it didn't provide much help. Lemme copy-paste what I wrote someone else for you

Ok will try that in a bit. Is there anyway of maybe asking if Tuya could present the local key in the app when connected to the same network as the devices as it would make it much easier for external integrations such as homebridge. The only thing would be security.
Sent with GitHawk

Here is the signature that I acquired from command prompt when running homebridge with the old key and that isn’t working. It shows up but then when controlling through HomeKit the device doesn’t respond to the commands but still appears online. I guessing this is because it has the wrong key.
Ready to handle AC (AirConditoner:3.1) with signature {“1”:false, “2”:24, “3”:17, “4”:“HEAT”, “5”:“1”, “19”:”C”, “102”:false, “103”:0, “104”:false, “105”:0, “106”:0, “110”:0}
Sent with GitHawk

The 3.1 devices don't use encryption while fetching the signature.
One way to check if the key is really bad is to watch the logs and trigger a change on the device using the controls on it, maybe just turn it on or off, or change the temperature. If you see an error about Odd message ... with command 8 it means the key is bad. If no error is shown, it is probably good.

I just tried the pcap method but it was giving errors I don’t know whether I was doing it wrong orwhat
Sent with GitHawk

Do you know what error it gave you? Also can you check if the key is really good using my previous message?

Also did you mean the UDID or the ECID

@AMoo-Miki i have just sent through the 2 files that you requested from wireshark through to your email. I dont know my correct key so therefore i didnt provide it. If there is anything else you need just let me know. I hope that you are able to distinguish a key from that.

I just ran thing that you asked me to do previously to check the key and it kept coming back with odd message from AC with command 8
Sent with GitHawk

There wasn’t any error it was just that I was using the wrong device number. I was using the ECID rather than the UDID therefore it wasn’t working.
Sent with GitHawk

Also did you mean the UDID or the ECID

Silly me. I fixed the instructions. Thanks for pointing it out.

@AMoo-Miki sorry I have been so busy recently and when I have time I will try it and let you know

Of course. Ping me whenever you get a chance.

@AMoo-Miki I have just gotten round to trying some stuff out, and I bought a new powerboard the other week and i wanted to try and get that into home bridge so when I got it I loaded it into the tuya smart app and have just today been able to successfully retrieve the key from the device after following the instructions I realised that when I had been previously attempting to retrieve the key I hadn't been granting the certificate full trust settings, I have also been able to load the air conditioner into homebridge fully and I am now able to control it through HomeKit. However, do you know whether there is any way to separate the outlets on the powerboard as I am using the board in many different rooms however HomeKit won't allow me to separate them based on places and add each outlet to favourites instead of just the master board.

@TheHomekitWayFinder that is good to know.
I too found the grouping annoying and good news is that with the latest iOS, they have an option to separate the icons. I just saw it in my 13.3.1; give it a shot.
Lemme know if I can help in any other way.


google colab example is not complete

the google colab example does not look correct. Needs probably to be checked

Checked and updated the develop branch


Process 3SAT problems in CNF form

Hello
Is there any interface where a 3SAT problem in a DIMACS CNF file format can be read from a file and automatically be parsed in PCBO().add_constraint ?
for example, can a CNF formula (x0 or x1 or  ~x2)(x1 or x2 or x3) be represented as [0,1,-2][1,2,3] ?

Sorry for the late reply! That is a great suggestion, and it is now on my to do list.


Var with default value in order to prevent the error

Error applying plan:
1 error(s) occurred:

module.database.module.db_instance.aws_db_instance.this (destroy): 1 error(s) occurred:

aws_db_instance.this: InvalidParameterValue: The parameter FinalDBSnapshotIdentifier is not a valid identifier. Identifiers must begin with a letter; must contain only ASCII letters, digits, and hyphens; and must not end with a hyphen or contain two consecutive hyphens.
status code: 400, request id: xxxxx

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

duplicated


fixes for spark integration

fix netty and guava version for incompatible with spark (shaded)
fix a select call broken since last change
fix readme according to new interface
fix TiRange generic return type for scala
add more filter for gitignore
make TiCluster autoclosable

LGTM.


Update defender baseline to include missing license requirement for section 1

🗣 Description
Upon review of the Defender Rego code I discovered that a few of the policies 1.1, 1.3 and 1.5 rely on the Get-ATPProtectionPolicyRule cmdlet which requires the Defender for Office Plan 1 license. This needs to be reflected in the baseline license requirements section.
💭 Motivation and context
Change required to reflect the dependency of the Get-ATPProtectionPolicyRule cmdlet required for section one policies in the baseline.
Closes #598
🧪 Testing
N/A
✅ Pre-approval checklist

[x] This PR has an informative and human-readable title.
[x] Changes are limited to a single goal - eschew scope creep!
[x] All future TODOs are captured in issues, which are referenced
in code comments.
[x] All relevant type-of-change labels have been added.
[x] I have read the CONTRIBUTING document.
[x] These code changes follow cisagov code standards.
[x] All relevant repo and/or project documentation has been updated
to reflect the changes in this PR.
[x] Tests have been added and/or modified to cover the changes in this PR.
[x] All new and existing tests pass.

✅ Pre-merge checklist

[ ] Revert dependencies to default branches.
[ ] Finalize version.

✅ Post-merge checklist

[ ] Create a release.

Sections 1, 2 and 3 have licensing requirements for Plan 1 but they all are phrased differently. I recommend standardizing them so that they look alike for consistency.

@tkol2022 - Check the updated licensing info for policy section 3 to make it consistent with licensing info in sections 1-2.


Non necessary import of I2C if using an UART sensor

When using the UART sensor, you don't need the adafruit_bus_device library.
However, the following line in the library import what is required for the I2C version:
from adafruit_bus_device.i2c_device import I2CDevice
No big deal, and I have no clue how to do things better because all my idea seems to be worse that current situation (like putting the import on the I2C side or separating the two).
Also I don't have the I2C version to test, so I could not really test a change on that side.
But it could be an improvement for user of UART sensor.

@dglaude I know that we have done something like that (putting the i2c import in the i2c class) a few times before and just put a pylint ignore on that line.

I believe @kattni has the hardware to test this, so it is best that she test this option. I prefer not to blindly propose a change.

@dglaude You may be able to move from adafruit_bus_device.i2c_device import I2CDevice into the PM25_I2C class (https://github.com/adafruit/Adafruit_CircuitPython_PM25/blob/f70830bcf712e3f74c4756ed663b4fd17029c9a3/adafruit_pm25.py#L122). Then it would only be imported if using I2C. It may require a pylint: disable= for an import in the "wrong" place. I can test the changes if you'd like to put in a PR.

I have been trying that in #7 but black is not happy yet and is not telling me exactly why.
As I don't have a local black install, I think I will give up for now.

@dglaude if you have pip3, you can install black and pylint just with:
pip3 install --user black pylint. You may not need the --user, and if you are on Windows, it may be pip instead of pip3.

Thanks, I will read this guide: https://learn.adafruit.com/improve-your-code-with-pylint/overview
I have already the right pylint exclude, now I need to figure out what black want from me.

Should be solved with #7 .
This require testing with I2C air quality sensor, but it works in UART and I2C without sensor.

The Pull Request #8 might solve this a better way than #7
However, as a package, the way to import and use is a bit different.
I am no expert in "import ... as ..." that could limit this issue.
Please consider the best option, also @kattni if some learn guide use this, you may want to sync.
Finally @gadgetoid has plan to see if he can bring improvement from Pimoroni UART only library. He might want to do that after the merge.

#8 is now accepted, this solve the problem.


Inconsistant results between CPU and GPU

Hi,
I ran single_image_inference.py on both cpu and gpu but got difference results.
CPU result is 0.6065 whereas GPU result is 0.6265.
The difference is in 10-2 magnitute and is acceptable but still curious about the inconsistance. Any idea to solve this?
CPU: Mac M1, torch:2.3.0
GPU: a10, torch:2.0.0

Hi @qiuzhewei
I'm not entirely sure, as I always use GPU to reduce the time needed for inference. Have you tried using the same Python environment for CPU and GPU? The difference could be due to different versions of some Python packages.

Update, tried following steps

I downgraded CPU pytorch version to 2.0.0 but the difference gap still persists.
Using CPU in A10 machine and the result is 0.6272, which is more reasonable compared to M1 CPU result.

I see. Then the difference must stem from the different hardware/OS. Unfortunately, I don't think you can do anything about it, but as long as you use the same machine to calculate the results, they are consistent, so it shouldn't be a big deal.

I see. Then the difference must stem from the different hardware/OS. Unfortunately, I don't think you can do anything about it, but as long as you use the same machine to calculate the results, they are consistent, so it shouldn't be a big deal.

Got you. And in what purpose do files, dither.cpp and dither.so serve?
I comment out two lines in distortions.py：
dither_cpp = ctypes.CDLL(PROJECT_ROOT / "utils" / "dither_extension/dither.so").dither dither_cpp.argtypes = [     ctypes.POINTER(ctypes.c_float),     ctypes.POINTER(ctypes.c_float),     ctypes.c_int,     ctypes.c_int,     ctypes.c_int, ]
And then delete dither_extension directory, it seems everything works fine.

Those files are needed for some synthetic degradations that we employ during training. If you want to use the model just for inference, you can delete them.

One more question, for every single image, the preprocess procedure converts it into two tensors with the shape 5x3224x224 and then feed them into pth model. From my experiment, the preprocess time is way much more than plain model inference on cuda, 100ms+ vs 15ms, so the preprocess is computation bottleneck, is there any way to speed up the preprocess procedure?

By default, the code takes the four corners and center crops and averages their quality scores. This strategy sometimes makes the results more robust but is more computationally expensive. If you simply feed the tensor of the full input image to the input image you might reduce the preprocessing time (depending on the size of the image). If you want to try this strategy, just copy the minimal working example.
Also, if you find our model useful for your work please leave a star to this repo.

Thanks for your reply. Sure it is, starred!
BTW: Quickest reply ever received, thank you. Thumb up!!

Thank you very much! I always try my best to address the issues ASAP 💪

By default, the code takes the four corners and center crops and averages their quality scores. This strategy sometimes makes the results more robust but is more computationally expensive. If you simply feed the tensor of the full input image to the input image you might reduce the preprocessing time (depending on the size of the image). If you want to try this strategy, just copy the minimal working example.
Also, if you find our model useful for your work please leave a star to this repo.

For your training procedure, you randomly crop 224x224 (padding with zero of out of bound), but in the minimal working example, there is no Resize operation in the preprocess, not sure why the inconsistance between training and inference. From my knowledge, it is better to make the preprcess the same in both training and inference for the sake of getting a better final result.

This strategy is quite common in IQA. Note that resizing an image modifies its quality based on the interpolation algorithm, so downscaling every input image to 224x224 is not recommended. Computing the average quality score between the corners and the center crop addresses the mismatch between training and testing that you mentioned, but in my experience simply using the whole input image leads to very similar results (sometimes even better ones). Therefore, using a different preprocessing for training and testing does not seem to be a problem for our model (and similar ones).

I see, if my purpose is to extrat the best frame from a video, it is ok to downscale all frames to 224x224 and compute scores and then get the argmax one?

Since the preprocessing would be the same for every frame I think that it would be ok for your purpose. And yes, the inference time will be a lot faster than using the whole input frame.


Swift 2.0 support?

Do you want to support Swift 2.0 and the RealmSwift bridge?

RealmResultsController is built using Swift2.0
In order to use it you need to install Realm and RealmSwift frameworks manually, apart from RealmResultsController.
You have more information in the README file

Okay, fine. I integrate Realm and RealmSwift via cocoapods and use_frameworks! parameter. Is that the problem? I don't want to integrate Realm and RealmSwift manually. The dependency manager cocoa pods is great. I think, your dependency links are wrong.

Hi @AFcgi, I just created a test application using RealmResultsController and installing Realm via Cocoapods and it worked without any issue.

If you are using the Demo app included in the Repo, it is prepared to use the Realm frameworks directly (it expects to find them in the Carthage folder, although you would have to build them manually until Realm adds support for Carthage in their Swift 2.0 branch)

The library doesn't support Cocoapods yet, not until Realm merges the swift-2.0 branch, because we can't set the dependency in the podspec to a specific branch.

My recommendation to use the library in this moment (until we can add compatibility with Cocoapods and Carthage) is to either:

Build the RealmResultsController.framework yourself (clone the repo, execute carthage build --no-skip-currrent, then copy the .framework to your app). After that, you can install Realm via whatever way you want, as long as it is the swift 2.0 version

Copy the source code of the RealmResultsController to your app, and install the swift2.0 Realm.

In any case, I recomend to build Realm frameworks manually as they specify in their documentation
If you still have problems to make this library work in your app, please let me know and i'll try yo help!


Update Identity README/MSDocs

Update Docs with details of:

[ ] PowerShell Credential - Add it to the DAC chain diagram and the Development Platform Credentials matrix section in the README. Update MS Docs with this credential as well.
[ ] Authenticating with Sovereign Clouds - Add a brief description of the feature with code samples/link to code samples in README/MS-Docs.

I noticed these additional docs issues recently

Update this DefaultAzureCredential doc to include PowerShellCredential
VSCode is missing from the diagram both above and in the main readme

@g2vinay We should validate whether the above items also need to be addressed in the other langs.


Token program doing custom deserialization

Problem
Hand-written serializers/deserializers:
https://github.com/solana-labs/solana-program-library/blob/master/token/src/state.rs#L524
Proposed Solution
Pick a serialization library and switch to it before we publish the Token program to Mainnet Beta. There's plenty of precedent for Serde/Bincode, but I'm open to something more language agnostic.

cc: @jackcmay @mvines

@garious Haven't found one yet that doesn't suck.  What we have today is essentially Rust's C FFI.  Bincode in my book is a non-starter, the overhead for even simple ser/de operations was astronomical.  I also tried Near's Borsh but that was also very expensive.


Prevent godot crash from importing a certain kind of invalid gltf

Invalid type: mistach interpolation CUBICSPLINE and value size. Might need to change the description and maybe more protections are needed.
Fixes #83617

Thanks!


PyAAF documentation requirement needs update

When #383 was completed the documentation wasn't updated to reference that the requirement is now pyaaf2

Should be addressed by #554


Reorganized catalog, balancing catagories and split off floating point

This makes a lot of sense, thanks!


[CHORE] change migrations class name to uppercase

CHORE
Objective:

change this to remove the warning from sonar scan

I would actually just ignore migrations in sonar


#8540: Upgrade eltwise binary ops to support queue_id /output_tensor / uint output dtype

Eltwise binary EQ should produce a boolean or integer tensor as output #8540
Add Q_ID to ElemwiseBinary/ELEMWISE_BINARY_EQ #9109

What changed
After this change ttnn's eltwise binary ops like eq

respect output_dtype or output_tensor's dtype. In case uint32 return type requested, kernel calls sfpu typecast.
supports ouput_tensor
supports queue_id

Example 1: uint32 dtype
Eq returns bfloat by default, but if you pass a dtype the result will be correctly formed with 0 and 1.
result = ttnn.eq(input_tensor_a, input_tensor_b, dtype=ttnn.DataType.UINT32)

Same works if you call ttnn::eq or any other eltwise binary from C++
Example 2: output_tensor
output_tensor_preallocated = ttnn.ones([h, w], output_dtype, ttnn.TILE_LAYOUT, device, ttnn.L1_MEMORY_CONFIG)
ttnn.eq(input_tensor_a, input_tensor_b, dtype=output_dtype, output_tensor=output_tensor_preallocated)

queue_id example
result = ttnn.eq(1, input_tensor_a, input_tensor_b)

Why
There is a request to support integer or boolean return type in eq.
Caution
This does not work on Greyskull
Most operations do not support integer inputs for now.
Typecast from int to bfloat is not implemented yet either.
It means that the use of the int tensor is limited.

https://github.com/tenstorrent/tt-metal/actions/runs/9377463343 😃


[AutoRelease] t2-datafactory-2022-10-09-06495(Do not merge)

https://github.com/Azure/sdk-release-request/issues/3250
Live test success
https://dev.azure.com/azure-sdk/internal/_build?definitionId=987

issue link:https://github.com/Azure/sdk-release-request/issues/3250

/check-enoforcer override


SyntaxError Unexpected end of script at t.getComponentName(airbnb-prop-types/src/helpers/getComponentName)

We got this error from airbnb proptypes... haven't faced such issue before.
it throws an error on line
export default function getComponentName(Component) {

For more detail can check here for issue that we catch on our sentry instance: https://umai.sentry.io/share/issue/23296973e27047f28f20447e8519fcc4/
Versions:
├─┬<EMAIL_ADDRESS>│ │ ├─┬<EMAIL_ADDRESS>│ │ │ └──<EMAIL_ADDRESS>

Seems like an issue with your bundler or something - the actual file is complete, as you can see here<EMAIL_ADDRESS>


Reformating docs

Description
Updating Documentation
Type of change
Please delete options that are not relevant.

This change is a documentation update

Change Status

Complete

Checklist

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[ ] My code has been reviewed by at least one peer
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[x] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes
[x] There are no merge conflicts

Looks good! Merging


esp_lcd RGB panel buffer issue.  (IDFGH-12838)

Answers checklist.

[X] I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
[X] I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
[X] I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.
5.0.6
Espressif SoC revision.
ESP32-S3 N16R8
Operating System used.
Linux
How did you build your project?
Command line with CMake
If you are using Windows, please specify command line type.
None
Development Kit.
ESP32-S3 Devkit
Power Supply used.
USB
What is the expected behavior?
The display to render properly
What is the actual behavior?
I am not sure what you would call it, It resembles tearing but I do not think that is what is happening.
Here is a small video that shows the issue.

I am calling lv_display_flush_ready from inside of the vsync callback. I have tried using the semaphores like what is done in the example and there was no change at all. It is almost like the buffer is being written to while it is sending.
This is as much a question as it is a possible bug. I was looking at the source code in the ESP-IDF and I am not sure exactly when and how the buffers get flip flopped. Is the swapping of the buffers the responsibility of the user? Or is it handled by the driver? Is it the responsibility of the user to keep things in sync and only call esp_lcd_panel_draw_bitmap during a vsync event?.
I am using octal SPIRAM and octal FLASH. should I push the transfer speed up to 120mhz? and if so how do I got about doing that?
Steps to reproduce.
RGB bus display
Lanes: 16
Width: 800
Height: 480
Color format: RGB565
GUI Framework: LVGL 9.1
Number of frame buffers: 2
Bounce buffer: no
On-demand: no
Set up the display
Create 2 sliders,
resize them so they are large,
align the sliders to the side of the display
loop setting the value to move the slider position
Debug Logs.
No response
More Information.
No response

Hi @kdschlosser, the flickering of the right-side widget in the video is indeed abnormal. This phenomenon is likely not caused by screen tearing. Could you please tell me which model of LCD IC you are using? And could you also upload your project?

Ohh.. the project is quite involved. The project binds LVGL to MicroPython. You can view the binding code here...
https://github.com/kdschlosser/lvgl_micropython/blob/main/ext_mod/lcd_bus/esp32_src/rgb_bus.c
The IC that is being used is the ST7262.
I believe the display being used is a 5" Sunton display board. with the ESP32-S3 onboard.
What it almost looks like to me is a synchronization issue between LVGL and the RGBPanel driver in the esp-idf. When does the panel driver start sending the frame buffer? Does it do it right after initialization? or so it sit there idle until it is told to transfer the buffer?
It looks like the buffer is being written to while it is transferring.....
LVGL has mechanics in it that causes the program to stall while a framebuffer is being transmitted. The issue with RGB is that when the buffer is passed to be transmitted it is always going to be in a state where it is being transmitted. It is like this until the panel driver is told to use a different buffer. When does the panel driver change the buffers? does it wait until a vsync interrupt occurs? or does it change them immediately?
There is no way to really know if the buffer is able to be swapped other than using the vsync event and we don't want to cause a stall in an ISR for obvious reasons. And no memory allocation can occur inside the ISR either. The only thing that is able to be done is we can tell LVGL that the buffer has finished transferring. so what happens when that is done is LVGL will then go about copying the data from the buffer that was just transferred to the second buffer and this will happen at the same time the transfer to the display is occurring. Using the vsync is not a fantastic way to let LVGL know that the buffer has been transferred because LVGL is expecting the buffer to no longer be in use anymore. This is not the case when using the RGB driver. The buffer will always be in a state of being transferred until the panel driver is passed a new buffer.
What throws things out of sync is when we tell the panel driver to write a new buffer and as soon as the next vsync occurs that flag gets set to tell LVGL the buffer is good to go when it actually has not been written yet. LVGL then swaps the buffers around and it now starts writing to the buffer that is in the process of being sent.
I am almost 100% certain that this is being caused by a synchronization issue between the panel driver and LVGL. I am not sure how to fix it without using some kind of an intermediate buffer and it's contents needing to be copied. That is what I am trying to avoid because it's a performance killer when rendering an 800 x 480 x 16bit display.
I did want to note that only a single core is being used, so it's not an issue being caused by external code to the IDF trying to write to the buffers at the same time. DMA memory is being used for the frame buffers which would allow for a bumping of heads between the panel driver and LVGL a problem cannot exist where portions of LVGL code are writing to the frame buffer at the same time.
I have spoken to @kisvegabor on a video chat about this issue and he might be better to explain the mechanics of LVGL.

OK... BTW, we have an example rgb_lcd_8bit  that achieves synchronization between LVGL rendering and RGB refreshing through multiple buffers, preventing screen tearing.

@Lzw655
While that is a great example it is self defeating. The design of LVGL and using double buffering with DMA memory is the ability to do other things while the buffer is being transferred. What your code does is it negates that from happening. you are stalling the flush callback until the vsync even occurs which is only going to occur once the buffer has transferred. While it would work to keep things in sync it is not going to provide the best in terms of performance.
We want to be able to do other things while that buffer is transferring and not block the flush callback.
That's where the issue is. how do we do that while keeping things in sync? What I think is happening is in the vsync even which can happen at any time is when we need to let lvgl know the buffer has been transferred but the issue is not knowing what buffer has actually been transferred. if say buffer 1 is in the middle of sending and then LVGL goes and renders new data to the second buffer and then calls the flush function to send the buffer to the display soon after the first buffer finishes and the vsync event gets called. But the second buffer has not actually finished transferring and LVGL has just been notified that it has. See where the problem is? Then LVGL goes to copy information from one buffer to the next in order to sync the buffers and you end up with the problem seen in the example above.
Maybe this issue needs to get corrected in LVGL by adding a flag for each buffer.
I am going to talk to @kisvegabor about this and see what he says.

I found a solution. It's not exactly what I want but at least it is going to allow more things to get processed and done while the framebuffer is transmitting.
Thank you @Lzw655 's your example is what inspired what I did.
The problem I had with your example is the program stalling right after calling the tx_color function. Doing that completely negates any performance gains from using double buffering and DMA memory. While I know that double buffering is used to stop issues where rendering to the buffer is being done at the same time the buffer is transmitting it also carries with it a performance gain due to the use of DMA memory and being able to do other things while the transmitting is happening.
So what I did was instead of having the stall right after the tx_color is called in the flush callback I registered another callback with LVGL that gets called at the start of a refresh before anything gets written to the frame buffer.
void lv_display_set_flush_wait_cb(lv_display_t * disp, lv_display_flush_wait_cb_t wait_cb)

So now the flush function will exit and LVGL and do it's thing with timers or whatever else it needs to do and then the user code will run and make any changes that need to be made to LVGL and once lv_task_handler gets called and the timer has expired for the refresh is when that callback will get called. In that callback is where I have the code to collect the notification from FreeRTOS and then I call flush_ready in lvgl which will skip the spinning wheels that is done to wait for that function to get called.
This is going to provide more time to do other things.
Now.. I could shrink the time it spends waiting down even more if there was a function in the ESP-IDF that would allow me to get the frame buffer number by passing the buffer to that function. There is the possibility of a vsync occurring before the code comes full circle. Because there is no way to identify what buffer just flushed in the vsync callback I am not able to notify the task if it needs to wait or if it doesn't.
I need to be able to pull the index number of the buffer that the vsync callback was called for.

another note on this. I submitted a PR to LVGL that reworks the refreshing of the display. If it gets merged there is going to be an option that can be turned on that will kick out of the refresh if there is a buffer that is still being transmitted. This will allow things to continue running if DMA memory is being used instead of sitting there spinning it's wheels. I believe this is going to be a better design as the user can use a non blocking call to check and see if a vsync has occurred and if it has then it can call the flush ready function in the task loop.


help

hi, for some reason i cant open your bot

You've replaced the variable instead of it's value.
It should be,
configs["token"] = "NDI......WA8"

P.S.  I suggest you change your password and get a new key.


added attributions to the code and added the explanation of backpropagation algorithm in simple terms

Explained the backpropagation in simple terms

[ ] I have read freeCodeCamp's contribution guidelines.
[ ] My pull request has a descriptive title (not a vague title like Update index.md)
[ ] My pull request targets the master branch of freeCodeCamp.
[ ] None of my changes are plagiarized from another source without proper attribution.
[ ] My article does not contain shortened URLs or affiliate links.

If your pull request closes a GitHub issue, replace the XXXXX below with the issue number.
Closes #XXXXX

Thank you for your contribution to the page! 👍
We're happy to accept these changes, and look forward to future contributions. 🎉


improve ford performance

results of dynamically executed code now cached(renderers) like it used to be (hook files)
renderers also no longer parsed per resource path, with /: %/foo syntax handled by passing through "at current path, /foo" as a literal to be bound to the resource-specific path at compilation.

this in turn means that their compilation is largely cached, as stack trace information is no longer (mistakenly) the resource path. Fixing urbit/urbit#697.
tree comments have been moved to a separate renderer, isolating their /: usage

Additionally,

jet hints throughout ford for aid in profiling
various tweaks to ++pi-tell to make the profiler output format more readable

Not separating this into like 5 feature branches for each of the bullet points signed off by @galenwp; as is breaching this in tomorrow if necessary, though preliminary testing against a master pill says 2d0dd16 should take care of that.

I don't quite get it - is it done? Is it a breach? Is part of it a breach?

I think this is mergeable, but it's a little risky.  I say merge it and deploy it, and we'll figure it out later.  @ohAitch?

Yeah basically. I would run basic tests wrt deploying on a fake master-network, checking that http://localhost:8443/static still works, etc. but it should be fine.

This should (hopefully) fix the talk reloading issue.

It's really much better overall to have the breaching parts be a separate branch and merge that at breach time, rather than stubbing it in as a dead code path and making it undead at breach time. Just for future consideration...

There are no longer any breaching parts. The dead code path isn't even /stubbed/, and there might in fact blocked requests using the horn, which this until-breach version of %ford will execute correctly. (To the extent I've tested it, which is not comprehensive)
Type-safe state transitions etc.

temporary rename of %see to %sei
REVERT on breach

Yes
After the next breach, the backwards-compatible type will not be necessary, and the old behavior should be removed(and the %see name reclaimed). Until then, the change works fine, it just has a slightly ugly $% tag.

should perhaps have named the commit "NO LONGER BREACHING rename %see to %sei"

OK, if it does nothing but make the types nest, sure.

You have to do ~something~ to old-typed data, and "recognize it as perfectly valid new-typed data" is the simplest option.
To expand:
Breaches are philosophically wrong. An urbit should never lose its mind, even when acquiring new/better functionality. If we were on a proper network, this would have been a %zuse kelvin bump, with ford getting a state-adapter that walked through all of the horns it held and converted them to the new type; as stands, I'm fine with a simpler-but-less elegant approach which still perfectly preserves type safety.

I am aware that breaches are a terrible hack, which is why I only care about functional changes. If the only difference between this tree and a breached version of it is the names in the type, that's cool. If functionality would be added on the breach, that would be less cool.

(consider e.g. a simple bisect leading you to a commit that says "uncomment that stuff that was commented out")

The only post-breach action this requires is a revert of the last commit, which purely renamed the new type and added the old one(/handling) back in. The pull as stands already has all the new functionality, promise :3

Sure. I'm just on guard against "this is something, therefore we must merge it". :)

Well, specifically it is performance fixes to ford, without which e.g. webtalk is unusable. There is in fact a question of stability, but the "therefore we must merge it" is on fairly firm ground.


Create "feature/head" branch based on "develop" branch

Create "feature/head" branch based on "develop" branch to separate dev processes

executed commands
C: ..\homepage>git checkout -b feature/head
Switched to a new branch 'feature/head'
C: ..\homepage>git branch -a
develop

feature/head
main
patch/README.md
remotes/origin/HEAD -> origin/main
remotes/origin/develop
remotes/origin/main
remotes/origin/patch/README.md

C: ..\homepage>git log --all --decorate --oneline --graph

4cce311 (HEAD -> feature/head, origin/develop, develop) Added index.html template
| * 5b771d3 (origin/patch/README.md, patch/README.md) Revert "hw: clean readme"
| * e29c581 hw: clean readme
| * 6d23c22 hw: update readme
|/
645289c (origin/main, origin/HEAD, main) Initial commit

branch created locally