22.2.1: issue with building documentation

Just tested that PR and I'm still not able to build documentation
[tkloczko@barrel SPECS]$ rpmbuild -ba --with check --with failing_tests python-zmq.spec
warning: Downloading https://github.com/zeromq/pyzmq//archive/v22.2.1/python-zmq-22.2.1.tar.gz to /home/tkloczko/rpmbuild/SOURCES/python-zmq-22.2.1.tar.gz
warning: Downloading https://github.com/zeromq/pyzmq//pull/1619.patch#/python-zmq-Small-update-clarification-to-auth-docs.patch to /home/tkloczko/rpmbuild/SOURCES/python-zmq-Small-update-clarification-to-auth-docs.patch
Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.a3TFSQ
+ umask 022
+ cd /home/tkloczko/rpmbuild/BUILD
+ cd /home/tkloczko/rpmbuild/BUILD
+ rm -rf pyzmq-22.2.1
+ /usr/bin/gzip -dc /home/tkloczko/rpmbuild/SOURCES/python-zmq-22.2.1.tar.gz
+ /usr/bin/tar -xof -
+ STATUS=0
+ '[' 0 -ne 0 ']'
+ cd pyzmq-22.2.1
+ /usr/bin/chmod -Rf a+rX,u+w,g-w,o-w .
+ /usr/bin/cat /home/tkloczko/rpmbuild/SOURCES/python-zmq-shebangs.patch
+ /usr/bin/patch -p1 -s --fuzz=0 --no-backup-if-mismatch -f
+ /usr/bin/cat /home/tkloczko/rpmbuild/SOURCES/python-zmq-man3_level.patch
+ /usr/bin/patch -p1 -s --fuzz=0 --no-backup-if-mismatch -f
+ /usr/bin/cat /home/tkloczko/rpmbuild/SOURCES/python-zmq-Small-update-clarification-to-auth-docs.patch
+ /usr/bin/patch -p1 -s --fuzz=0 --no-backup-if-mismatch -f
+ /usr/bin/rm -rf bundled
+ /usr/bin/chmod -x examples/pubsub/topics_pub.py examples/pubsub/topics_sub.py

[..]

running egg_info
creating pyzmq.egg-info
writing pyzmq.egg-info/PKG-INFO
writing dependency_links to pyzmq.egg-info/dependency_links.txt
writing requirements to pyzmq.egg-info/requires.txt
writing top-level names to pyzmq.egg-info/top_level.txt
writing manifest file 'pyzmq.egg-info/SOURCES.txt'
listing git files failed - pretending there aren't any
reading manifest file 'pyzmq.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'setupegg.py'
warning: no files found matching 'tox.ini'
warning: no files found matching '.travis.yml'
no previously-included directories found matching 'docs/build'
no previously-included directories found matching 'docs/gh-pages'
warning: no files found matching 'bundled/zeromq/COPYING'
warning: no directories found matching 'bundled/zeromq/include'
warning: no directories found matching 'bundled/zeromq/src'
warning: no directories found matching 'bundled/zeromq/tweetnacl'
warning: no directories found matching 'bundled/zeromq/external/wepoll'
warning: no previously-included files found matching 'bundled/zeromq/src/Makefile*'
warning: no previously-included files found matching 'bundled/zeromq/src/platform.hpp'
warning: no previously-included files found matching 'setup.cfg'
warning: no previously-included files found matching 'zmq/libzmq*'
warning: no previously-included files matching '.deps/*' found anywhere in distribution
warning: no previously-included files matching '*.so' found anywhere in distribution
warning: no previously-included files matching '*.pyd' found anywhere in distribution
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files matching '.DS_Store' found anywhere in distribution
warning: no previously-included files matching '.mailmap' found anywhere in distribution
warning: no previously-included files matching 'Makefile.am' found anywhere in distribution
warning: no previously-included files matching 'Makefile.in' found anywhere in distribution
adding license file 'COPYING.BSD'
adding license file 'COPYING.LESSER'
adding license file 'AUTHORS.md'
writing manifest file 'pyzmq.egg-info/SOURCES.txt'
+ PBR_VERSION=22.2.1
+ SETUPTOOLS_SCM_PRETEND_VERSION=22.2.1
+ SPHINXOPTS=-j48
+ /usr/bin/python3 setup.py build_sphinx -b man --build-dir build/sphinx
running build_sphinx
Running Sphinx v4.2.0

Configuration error:
There is a programmable error in your configuration file:

Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/sphinx/config.py", line 328, in eval_config_file
    exec(code, namespace)
  File "/home/tkloczko/rpmbuild/BUILD/pyzmq-22.2.1/docs/source/conf.py", line 65, in <module>
    import zmq
  File "/home/tkloczko/rpmbuild/BUILD/pyzmq-22.2.1/zmq/__init__.py", line 103, in <module>
    from zmq import backend
  File "/home/tkloczko/rpmbuild/BUILD/pyzmq-22.2.1/zmq/backend/__init__.py", line 32, in <module>
    raise original_error from None
  File "/home/tkloczko/rpmbuild/BUILD/pyzmq-22.2.1/zmq/backend/__init__.py", line 27, in <module>
    _ns = select_backend(first)
  File "/home/tkloczko/rpmbuild/BUILD/pyzmq-22.2.1/zmq/backend/select.py", line 32, in select_backend
    mod = import_module(name)
  File "/usr/lib64/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/home/tkloczko/rpmbuild/BUILD/pyzmq-22.2.1/zmq/backend/cython/__init__.py", line 6, in <module>
    from . import (
ImportError: cannot import name 'constants' from partially initialized module 'zmq.backend.cython' (most likely due to a circular import) (/home/tkloczko/rpmbuild/BUILD/pyzmq-22.2.1/zmq/backend/cython/__init__.py)

The docs assume an in-place build of pyzmq (python3 setup.py build_ext --inplace) by adding the repo root to sys.path.
If you're not doing that, you could patch that out, or make it opt-out with an env variable or some such.

The docs assume an in-place build of pyzmq (python3 setup.py build_ext --inplace) by adding the repo root to sys.path.
If you're not doing that, you could patch that out, or make it opt-out with an env variable or some such.

Using python3 setup.py build_ext is ALREADY deprecated from more than 3 years.
Please update your build procedures.
Currently used build procedure is described in PEP517.

To be clear, all build procedures are fully PEP517 and have been for years. I should have typed pip install -e ., which is the same. But anyway, this issue is closed because none of that is relevant anymore.

o be clear, all build procedures are fully PEP517 and have been for years. I should have typed pip install -e ., which is the same. But anyway, this issue is closed because none of that is relevant anymore.

So I completely do not understand why you recommend me to use deprecated procedure.

I'm not doing that. You are responding to a comment from years ago that is no longer accurate. It worked then, it doesn't now.


Explore writing basic tests for UI components by mocking data

solved this issues by adding dummy data -> UI works fine


Ace binding in editor: Ctrl-F broken

From Piazza: https://piazza.com/class/jg1gbtzfzh816w?cid=750
The editor is the Ace editor, you can find keyboard shortcuts here: https://github.com/ajaxorg/ace/wiki/Default-Keyboard-Shortcuts
Some useful shortcuts:
Jump to line number is Ctrl+L (Cmd+L on Mac)
Find next - highlight the token you want to find the next occurrence of and press Ctrl+K (Cmd+G on Mac).
I am not sure why Ctrl+F is broken.

From a previous issue:
Notably (and sorely) missing:
-p: go to the previous line


Q: duplicate links?

Hi again, just a question: if this behaviour is currently to be expected?

In the graph.Graph dictionary I saw that some files have the same file listed multiple times as a dependency.
I assume this isn't really noticeable in the grand scheme of things but I was wondering if it is known behaviour and can be avoided?

I think this has been addressed.


Fix RenderThread's notify mechanism

Summary of the Pull Request
Fix a bug where the Renderer::PaintFrame method:

is not called until the next RenderThread::NotifyThread call but needs to be called because there the terminal was updated (theoretical bug)
is called twice but needs to be called only once (verified bug)

References
The bug was introduced by #3511.
PR Checklist

[x] CLA signed. If not, go over here and sign the CLA

Detailed Description of the Pull Request / Additional comments
Before
First bug
In the original code, _fNextFrameRequested is set to true in render thread because std::atomic_flag::test_and_set is called.
This is wrong because it means that the render thread will render the terminal again even if there is no change after the last render.
I think the the goal was to load the boolean value for _fNextFrameRequested to check whether the thread should sleep or not.
The problem is that there is no method on std::atomic_flag to load its boolean value. I guess what happened was that the "solution" that was found was to use std::atomic_flag::test_and_set, followed by std::atomic_flag::clear if the value was false originally (if std::atomic_flag::test_and_set returned false) to restore the original value. I guess that this was believed to be equivalent to just a simple load, without doing any change to the value because it restores it at the end.
But it's not: this is dangerous because if the value is changed to true between the call to std::atomic_flag::test_and_set and the call to std::atomic_flag::clear, then the value ends up being false at the end which is wrong because we don't want to change it! And if that value ends up being false, it means that we miss a render because we will wait on _hEvent during the next iteration on the render thread.
Well actually, here, this not even a problem because when that code is ran, _fPainting is false which means that the other thread that modifies the _fNextFrameRequested value through RenderThread::NotifyPaint will not actually modify _fNextFrameRequested but rather call SetEvent (see the method's body).
But wait! There is a problem there too! std::atomic_flag::test_and_set is called for _fPainting which sets its value to true. It was probably unintended. So actually, the next call to RenderThread::NotifyPaint will end up modifying _fNextFrameRequested which means that the data race I was talking about might happen!
Second bug
Let's go back a little bit in my explanation. I was talking about the fact that:

I guess what happened was that the "solution" that was found was to use std::atomic_flag::test_and_set, followed by std::atomic_flag::clear if the value was false originally (if std::atomic_flag::test_and_set returned false) to restore the original value.

The problem is that the reverse was done in the implementation: std::atomic_flag::clear is called if the value was true originally!
So at this point, if the value of _fNextFrameRequested was false, then std::atomic_flag::test_and_set sets its is set to true and returns false. So for the next iteration, _fNextFrameRequested is true and the render thread will re-render but that was not needed.
After
I used std::atomic<bool> instead of std::atomic_flag for _fNextFrameRequested and the other atomic field because it has a load and a store method so we can actually load the value without changing it.
I also replaced _fPainting by _fWaiting, which is basically the opposite of _fPainting but stays true for a little shorter than _fPainting would stay false. Indeed, I think that it makes more sense to directly wrap/scope just the call to WaitForSingleObject by setting my atomic variable to true just before and to false just after because:

It makes more sense while you're reading the code: it's easier IMO to understand what the purpose of _fWaiting is (that is, to call SetEvent from RenderThread::NotifyPaint if it's true).
It's probably a tiny bit better for performance because it will become true for a little shorter which means less calls to SetEvent.

Warning
I don't really understand std::memory_orders.
So I used the default one (std::memory_order_seq_cst) which is the safest.
I believe that if no read or write are reordered in the two threads (RenderThread::NotifyPaint and RenderThread::_ThreadProc), then the code I wrote will behave correctly.
I think that std::memory_order_seq_cst enforces that so it should be fine, but I'm not sure.
Validation Steps Performed
I tried to reproduce the second bug that I described in the first section of this PR.
I put a breakpoint on RenderThread::NotifyPaint and on Renderer::PaintFrame. Initially they are disabled. Then I ran the terminal in Release mode, waited a bit for the prompt to display and the cursor to start blinking. Then I enabled the breakpoints.
Before
Each RenderThread::NotifyPaint is followed by 2 Renderer::PaintFrame calls. ❌
After
Each RenderThread::NotifyPaint is followed by 1 Renderer::PaintFrame call. ✔

I wonder if the CI failed because it's bugged or if it's because my code is bad. It failed on a test that uses the RenderThread that I modified. Could somebody force it to run again, please?

Reran failed x86 leg. That test has been a pain in our side. It may or may not be your fault.

@msftbot make sure @miniksa signs off on this

If NotifyPaint is called twice very quickly while the render thread is waiting, it will end up calling SetEvent twice which will make the render thread not wait for the next time when it is supposed to wait which results in two renders while only one was needed. So I added a ResetEvent to make sure that this doesn't happen (see last commit 1664025)

@msftbot, make sure @DHowett-MSFT signs off

(It looked good to me; GH didn't show that @zadjii-msft had signed off, but I would have been the third.)

:tada: Once again, thanks for the contribution!
This pull request was included in a set of conhost changes that was just
released with Windows Insider Build 19603.


Dart analyzer keeps crashing

Dart analyzer keeps crashing in flutter 1.12.13+hotfix.5(latest stable release).
i tried everything

Deleted flutter/bin/cache(run flutter docter).
2.Uninstalled flutter and android studio(every single file).
but the error is still not fixed.

This is happening to several people, see this:
#47538
#46616
#47821
On Dart repo there are a few issues related too.
I had to do downgrade to 1.9.1 becouse analizer crashing is anoying.

Dart analyzer keeps crashing in flutter 1.12.13+hotfix.5(latest stable release).
i tried everything

Deleted flutter/bin/cache(run flutter docter).
2.Uninstalled flutter and android studio(every single file).
but the error is still not fixed.

It would be good if you shared your error log and flutter doctor's output, also that you activate the intrumentation log to see if it throws something interesting at you.

Hi @saurabhguptarock
I see there's an open issue  addressing the case you described.
Please follow up on that issue,
I'm closing the current one as duplicate.
If you disagree please write in the comments
and I will reopen it.
Thank you


Getting the native docker image back

We had to go back to jvm version, let's try to get native image back again.

https://github.com/KaotoIO/kaoto-backend/issues/111

Codecov Report

Merging #134 (654e2a5) into main (a3c7f6c) will increase coverage by 0.38%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##               main     #134      +/-   ##
============================================
+ Coverage     80.85%   81.23%   +0.38%
- Complexity      144      148       +4
============================================
  Files            43       43
  Lines          2047     2057      +10
  Branches        313      314       +1
============================================
+ Hits           1655     1671      +16
+ Misses          281      278       -3
+ Partials        111      108       -3

Impacted Files
Coverage Δ

...kaoto/backend/metadata/parser/JarParseCatalog.java
81.81% <0.00%> (-2.33%)
:arrow_down:

...o/backend/metadata/parser/ClusterParseCatalog.java
85.29% <0.00%> (-2.21%)
:arrow_down:

.../backend/api/resource/v1/IntegrationsResource.java
51.72% <0.00%> (+5.17%)
:arrow_up:

...o/backend/api/resource/v0/IntegrationResource.java
72.22% <0.00%> (+5.55%)
:arrow_up:

...kend/api/service/deployment/DeploymentService.java
67.64% <0.00%> (+8.82%)
:arrow_up:

...ackend/api/resource/v0/ViewDefinitionResource.java
67.74% <0.00%> (+9.67%)
:arrow_up:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


[release-v0.12.1] Removed arm/v7 to make the jobs working again for now

Cherry-pick of #391

/lgtm


The "canonical" header files copied to Platform/include cause problems

In the build steps on each platform there is a script which copies a large number of include files from their original locations to Platform/include.  It is not clear why this step is needed.  The header files are not modified in any way, only their location is changed.
There are two problems with this approach:

One can easily forget and end up editing the copied file(s) and then in the next build your changes are wiped out when the copy happens again.

This process gives Visual Studio on Windows fits.  It watches all the header files in the whole project and the fact that the files that have been copied are newer than the originals and this apparently throws VS into a tizzy such that it always complains that the project is out of date and insists on "rebuilding" the EPUB project  - which consists only of re-copying the files again.  Very annoying.

I believe we should skip this step and point the builds at the original header files.  I would be happy to hear why this copying is valuable but I simply don't see it.

Note that the move from Eclipse to Android Studio means that the C++ headers are now handled by a cross-platform Gradle script, rather than a platform-dependent shell script (zsh MakeHeaders.sh): https://github.com/readium/readium-sdk/blob/develop/Platform/Android/epub3/build.gradle#L9


Create CODE_OF_CONDUCT.md

Signed-off-by: K Rain Leander<EMAIL_ADDRESS>Description

Why is this needed

Fixes: #
How Has This Been Tested?

How are existing users impacted? What migration steps/scripts do we need?

Checklist:
I have:

[ ] updated the documentation and/or roadmap (if required)
[ ] added unit or e2e tests
[ ] provided instructions on how to upgrade

Should we follow the same path?
https://github.com/tinkerbell/tink/blob/master/CODE_OF_CONDUCT.md
A centralized code of conduct? in this way we do not have to change it in all the places

Definitely! We could simply have a short statement of TL;DR Be Kind, please see the full CoC. Keen?

That's what we have to tink, I think we can copy/paste that

Love it!


Bump api-go version to fix Payload marshaling

What was changed
I upgraded our api-go dependency
Why?
Marshaling for binary/null payloads was completely broken, and while there Chad and I decided that shorthand marshaling should be opt-in rather than opt-out.

You can see from https://github.com/temporalio/features/pull/375 that the feature tests pass when that repo has been updated

API repo is tagged, I'll set this to merge once tests pass


Abacus cli tools should be written in es6

I was looking through some of the code for the abacus cli tools and it seems that it was all written in es5.  There's even a comment in each of the files saying "Implemented in ES5 for now".
I was just wondering what was the reasoning behind this?  Is there any reason that the cli tools can't be written in es6 and transpiled back to es5 like the rest of the abacus modules?

One of the reasons was to minimize the amount of dependencies required to bootstrap the whole dev environment. If you feel like converting them to ES6 and are able to do so without introducing any complications in the bootstrapping of the dev env, please feel free to submit a pull request. If you're not planning to submit a pull request soon, then please let me know and I'll close the issue (as leaving an issue that nobody is planning to fix anytime soon is not really useful). Thanks!

Thanks for the response.  I was more curious than anything else as I'm currently writing a few node command line tools.


Torque on analysis doesn't work

STR:

Create a map with a polygon dataset.
Add centroid analysis.
Set animated style. Torque tiles fails with an error: "TorqueRenderer: Argument to ST_X() must be a point"

cc @javisantana

Checked by @javisantana and @donflopez, working now.


How is multiple jobs processed ?

Hi,
If there is 10 jobs coming (assume each job requires 50% CPU running for 2 min), how many will the worker server takes and processes ?
As I used Resque before, I can set worker number to 2 per server, so the server will take only 2 jobs at one time and using 10 mins to finish all the job. And I can view the process detail via a web UI.
So I want to know if I could set a worker number for the AEJ like resque? And also can I view the current processing jobs like Resque web UI?
Thanks, the AEJ is a great tool to enjoy aws eb with no extra pain = ]

Emm... based my test, if I flood the server with background jobs, the worker server will just try to process as much as possible. And result few jobs timeout due to, most likely, too many jobs fighting for resources.
So not sure if it's possible to process job in a orderly fashion?  Really appreciate if @tawan could give a bit guide.

Hi @lovesmall
Thanks for you submission!
One major advantage of Elastic Beanstalk Worker environments is the autoscaling feature. If enabled, it launches additional EC2 instances when load increases. Given your example, autoscaling would start additional woker servers, because the original server's load increased. You can enable the autoscaling feature when you launch the worker environment. See the screenshot below!

Given your example, I'd also reduce the number of concurrent requests to 2. This will prevent a worker instance from fetching 10 jobs at once. As you have mentioned, a job requires 50% CPU time, it makes sense, to limit a server to two concurrent jobs, which you can achieve by reducing the number of concurrent HTTP connections.

Hope the answers your question!
If so, feel free to update the FAQ section of the README, and issuing a pull request. Only if you want of course ;)
Cheers,
Tawan

Hi Tawan,
Thank you SO MUCH for tell me the HTTP connections setting, just when I was about to switch back to resque. It works good now.
I'm using auto-scaling but this issue is not related to that. Because even with auto-scaling, i still need handle jobs in an orderly way, as I could have large number of jobs comes in few seconds, and I need "NOT" allow worker takes in too many jobs simultaneously.
For the HTTP connection, I've set my worker server to 2. When there are 200+ message in the queue, I could see  3 or 4 messages "In Flight" at any time now(which is fine). Shouldn't it be "2" at most? Is it like this:

Send a message to worker via HTTP request, a HTTP connection is established.
The HTTP connection is kept open while worker process the job.
Worker finish job and delete the message, HTTP connection is closed.

Thank you if you could clarify a bit on this.
Best,
Yiheng

Hi @lovesmall
Basically you have correctly described the message's lifecycle when consumed by EB worker environments.

Message is in queue (visible)
EB worker SQS daemon fetches message from queue (in flight)
SQS Daemon tries to establish an HTTP connection to the worker application but blocks as long as max other HTTP connections are already open. When the max HTTP connection setting is set to 2, this means that in total 3 messages can be in flight at this step. The third message in flight, is the one that the daemon already fetched from the queue, but it has to wait until it can open an HTTP connection.
The HTTP is established and the message is passed as an HTTP request to the worker application.
The worker application responds with an 200 OK status
The SQS daemon sends a DELETE message call to the queue.

Please, also have a look at this SO discussion.
Cheers,
Tawan

Hi @lovesmall
Thanks for you submission!
One major advantage of Elastic Beanstalk Worker environments is the autoscaling feature. If enabled, it launches additional EC2 instances when load increases. Given your example, autoscaling would start additional woker servers, because the original server's load increased. You can enable the autoscaling feature when you launch the worker environment. See the screenshot below!

Given your example, I'd also reduce the number of concurrent requests to 2. This will prevent a worker instance from fetching 10 jobs at once. As you have mentioned, a job requires 50% CPU time, it makes sense, to limit a server to two concurrent jobs, which you can achieve by reducing the number of concurrent HTTP connections.

Hope the answers your question!
If so, feel free to update the FAQ section of the README, and issuing a pull request. Only if you want of course ;)
Cheers,
Tawan
@tawan
I saw your comment from four years ago. When enabling autoscaling on a worker environment, a load balancer will have to be created manually, correct? My autoscaling worker environment is stuck in a Severe Health State. I have tried relaunching the environment but I still cannot sustain a healthy environment. Environment health has transitioned from No Data to Severe. Do you have any additional instructions? Thank you!


Detect policies

Coverage decreased (-0.4%) to 98.712% when pulling 0cd88df5266216c1496e620cadc895c0aad8a689 on sebdesign:master into 40ab9b559f8290ad21aa52e84f1276b49d4920f0 on stefanzweifel:master.

Thanks for the contribution! 👍
Looks great so far and the PR does exactly what it should do. Will review and merge in the upcoming days.
Sidenote: I will write a bigger refactor PR in the upcoming days. I would like to have all component configurations in one single place ...

Thanks! The component configuration should be refactored, because we'll need sophisticated detection for some components, like policies or middleware. If you want any help let me know you!

Coverage increased (+0.05%) to 98.696% when pulling e209acfa541ad8f8efadc7f5f9a38edfc7e7cf21 on sebdesign:master into df1b7fdfe31c7d6ac98ce90d660b7206f43080fb on stefanzweifel:master.


Add machine "discovery"

Discover unrecognized machines which attempt to network boot from bootcfg. Collect basic instance data as a bootcfg Instance or Machine resource.
This is valuable to expose to gRPC clients (UIs or CLIs) so a user can create groups which match discovered machine instances. For example, a UI might allow a discovered instance to be assigned to a Profile and create the bootcfg match Group via gRPC call.
Depends on #97

Include state about whether a node has successfully been provisioned by bootcfg before, where success if defined by receiving a checkin from a unit on host, which can perform some checks.

This sounds very familiar to Foreman discovery. Essentially it dynamically creates the ipxe boot file (or defaults to discovery) based on if the system already exists in the db. If not, it gets a discovery image which is a stripped down fedora image with puppet for reporting hardware facts. The system stays in that state until set to provision from the ui/cli when it reboots and boots to the proper provisioning ipxe file.

Any updates on this functions? I think its related to this https://github.com/coreos/matchbox/blob/master/matchbox/cli/instance_list.go
It would be very usefull to have it.


tbpore, version 0.1.0

Hello,
this draft PR is just for a WIP review, this is not getting merged now. Please check if you agree with the current changes. This is a small amount of work, just from a single day, but if you spot issues here the best is to solve/change ASAP. Main points so far:

H37RV genome and its faidx is explictly added to repo. Do you want to change this approach?
Due to python path module issues, I copied what was in tbpore/tbpore.py into tbpore/cli.py. In summary, tbpore/tbpore was first in sys.path than tbpore, so it would search all the modules (e.g. from tbpore.constants in tbpore/tbpore.py instead of tbpore). By removing tbpore/tbpore.py and moving its contents into tbpore/cli.py, everything worked in my env. There is probably a better solution than this, allowing us to keep tbpore/tbpore.py, but I did not dive too much into it. Do you also have this issue? Should we keep as is described in this PR, or should we roll back into having tbpore/tbpore.py?
--threads and --mem_mb parameters added;
External tool execution done with class ExternalTool;

TODOs:

Let user specify log folder for ExternalTools execution;
Implement decontamination step (Zam found it better to do all steps, and leave decontamination for later, as it is not that important);
Implement the remaining steps;
Refactor;
Add unit tests;

I wonder if the reason the tbpore file didn't work is I accidently forgot to change this after copying it from snakefmt
https://github.com/mbhall88/tbpore/blob/603ae00020b797635ca340eae0e0172a5601db2b/pyproject.toml#L13
It should work, as it was working for me fine and this is the exact same file structure I use in snakefmt. Are you running tbpore in a poetry env? i.e. poetry run tbpore -h

That is probably my fault as I am not experienced with poetry. I tried to install the packages with poetry install (as like in pip install) and was trying to run as PYTHONPATH="." python tbpore/__main__.py. In the end, I made the aforementioned changes, and extracted a requirements.txt from poetry and pip installed them, and it is working now. Next time will try poetry run tbpore -h, but will leave tooling as last step now. BTW do you know how the end user will install tbpore actually? I did not search much, but I haven't had much success on seamlessly combining conda+poetry or conda+pip. I am thinking about a 2-step approach where the user creates the conda env, and then installs tbpore inside the conda env. I am guessing tbpore will be deployed to Pypi, giving how easy it is to publish to Pypi if poetry is set up?

Summary of updates from today's commits, excluding those addressing previous comments:

Binary skeletons are stored in a .cache dir in the repo root
H37RV mask added
All remaining steps finished
External tools parameters that might (rarely) change, but that are not changeable through tbpore args are now listed in a .config.yaml file, making them more configurable: https://github.com/mbhall88/tbpore/pull/8/commits/6703cfb0ef007b52dd37a69745adf9a8d3a1ca42 and https://github.com/mbhall88/tbpore/pull/8/commits/b7aa4cfcac2c426a986f05077e9e323f8a0695be
--logdir parameter added
Mocking the execution of external tools, making tests better
Added a test to check all tools are called in the correct order and with the correct arguments
General refactoring

TODO:

Add more unit tests;
Refactor tooling;
Lint/format;

That is probably my fault as I am not experienced with poetry. I tried to install the packages with poetry install (as like in pip install) and was trying to run as PYTHONPATH="." python tbpore/main.py. In the end, I made the aforementioned changes, and extracted a requirements.txt from poetry and pip installed them, and it is working now. Next time will try poetry run tbpore -h, but will leave tooling as last step now. BTW do you know how the end user will install tbpore actually? I did not search much, but I haven't had much success on seamlessly combining conda+poetry or conda+pip. I am thinking about a 2-step approach where the user creates the conda env, and then installs tbpore inside the conda env. I am guessing tbpore will be deployed to Pypi, giving how easy it is to publish to Pypi if poetry is set up?

The tooling was setup already though. It is always easier to set that up at the start. The way I do it is install poetry inside the conda env. Then whenever I want to run tbpore use poetry run tbpore .... You will need to change the leftover snakefmt in pyproject.toml for this to work (just checked locally and that was why it wasn't working for you before with the main function in tbpore.py).
I was planning to distribute tbpore on pypi (using poetry) and bioconda. then the installation process is just conda install tbpore and you get all the external dependencies.

I was planning to distribute tbpore on pypi (using poetry) and bioconda. then the installation process is just conda install tbpore and you get all the external dependencies.

Perfect for me.
Re: tooling, poetry and tbpore/tbpore.py: sorry, old c++ grandpa can't keep up with these amazing modern tooling. poetry running just fine, and I updated it and tbpore/tbpore.py is back, so all fine I think tooling-wise

Codecov Report

Merging #8 (c5e2f38) into main (6b99c4b) will increase coverage by 15.45%.
The diff coverage is 100.00%.

@@             Coverage Diff             @@
##             main       #8       +/-   ##
===========================================
+ Coverage   75.00%   90.45%   +15.45%
===========================================
  Files           6        7        +1
  Lines         112      199       +87
  Branches       21       26        +5
===========================================
+ Hits           84      180       +96
+ Misses         18       12        -6
+ Partials       10        7        -3

Flag
Coverage Δ

unittests
90.45% <100.00%> (+15.45%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

tbpore/constants.py
100.00% <100.00%> (ø)

tbpore/external_tools.py
100.00% <100.00%> (ø)

tbpore/tbpore.py
92.52% <100.00%> (+18.67%)
:arrow_up:

tbpore/utils.py
96.42% <100.00%> (+1.19%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 603ae00...c5e2f38. Read the comment docs.

Summary of updates from today's commits, excluding those addressing previous comments or issues:

Removed --logdir parameter - I favoured to create a logs dir inside outdir so that the number of parameters is reduced;
Defaulting --outdir from . to tbpore_out - as such is easier to see which output files were created;
Finished unit tests (see above for coverage);
Sorted out tooling (was fine since the start, I was failing to use it well);
Formatting done;

I think this PR is ready for review. Closes #2, #3, #4, #5, #6, #7, #9, #10. Still open: #11, #12, #13, #14

I had to create this dev branch and change the base branch of this PR from main to dev because the latest commit to the main branch creates a conflict with this PR, which causes github actions to not trigger CI

All builds now work, see https://github.com/mbhall88/tbpore/issues/12#issuecomment-1109568596

See https://github.com/mbhall88/tbpore/issues/16#issuecomment-1115986893 for a detailed comparison between H2H and tbpore (at the tip of this PR) results. If they look ok for you, this PR can then be merged


修改文件名，seata.io 查看 blog，带空格 url 无法访问

https://github.com/seata/seata.github.io/blob/develop/site_config/blog.js

https://github.com/seata/seata.github.io/blob/develop/site_config/blog.js

https://github.com/seata/seata.github.io/blob/develop/site_config/blog.js

改了

https://github.com/seata/seata.github.io/blob/develop/site_config/blog.js

改了


Not an issue.

Hi guys. Just noticed this script. Looks good.
Started writing myself something but only just seen this!
Might be worth having a read through my code and pulling in some of my lines and I’ll kill my repo?
https://github.com/phillips321/adaudit
FYI mines completely online.
Cheers and good work!

Thanks, I'll check it out!


Suport to Avalonia11

Now Avalonia 11 is officially released, and Avalonia-QRCode 1.0.0 is not supported. Maybe you can do some updates

The lovely @rabbitism submitted a PR and I've just merged :)


feat: drop poetry2nix

Constant build failures due to conflicts between nixpkgs and pypi shouldn't be getting in the way of me updating this flake, #71. I need to move this flake from poetry2nix to a derivation.

2af20dd66687ac63e7fd76cafd1be3d5122ac978 is in master? dunno why this didnt close


Button#getLabel() is marked as @Nonnull, but is an optional field in API documentation

General Troubleshooting

[X] I have checked for similar issues on the Issue-tracker.
[X] I have checked for PRs that might already address this issue.

Version of JDA
5.0.0-beta.19
Expected Behaviour
Change Nonnull to Nullable
Code Example for Reproduction Steps
.

Code for JDABuilder or DefaultShardManagerBuilder used
.

Exception or Error
No response

Button has to have either label or emoji, but both fields are optional. Label-less button example:

Have you checked if the method throws?
If not I'll make a PR about this shortly.

It cannot be null in jda.


bigquery: stream older data into partitioned tables

Client
BigQuery
Describe Your Environment
golang:1.9 docker image
# vendor
name = "cloud.google.com/go"
revision = "dfffe386c33fb24c34ee501e5723df5b97b98514"
version = "v0.30.0"

Expected Behavior
Stream data older than 7 days into time partitioned tables using (u *Uploader) Put. The documentation says it is possible to stream data not older than 1 year in the past.
Actual Behavior
I tried adding the $yyyymmdd suffix (#427) for a 35 days old entry but I got this error message:

The destination table's partition download_2018$20180918 is outside the allowed bounds. You can only stream to partitions within 31 days in the past and 16 days in the future relative to the current date.`

Then I tried adding an entry not older than 31 days and I got this error message:

Value<PHONE_NUMBER>000000 for field hit_time of the destination table palcomp3-hotbkp:hits.download_2018$20180923 is outside the allowed bounds. You can only stream to date range within 7 days in the past and 3 days in the future relative to the current date.

Table schema and partitioning config:
bigquery.TableMetadata{
	Schema: bigquery.Schema{
		{Name: "music_id", Required: true, Type: bigquery.IntegerFieldType},
		{Name: "user_id", Required: false, Type: bigquery.IntegerFieldType},
		{Name: "hit_time", Required: true, Type: bigquery.TimestampFieldType},
	},
	TimePartitioning: &bigquery.TimePartitioning{Field: "hit_time", RequirePartitionFilter: true},
	Clustering:       &bigquery.Clustering{Fields: []string{"music_id"}},
}

Sorry for the not-great experience. :/ This seems to be an error with the underlying service, though, as opposed to the client libraries. Would you mind re-filing this bug there? This feedback would be very valuable for them. See the list of backend issue trackers here: https://cloud.google.com/support/docs/issue-trackers

@FdeFabricio As far as I understand, passing Field in TimePartitioning assumes partitioning column. At the same time, suffix in table name is used for ingestion-time partitioned tables. As far as I understand, suffix should not be used in case of column-based partitioning.

@chemikadze Yeah, you are right. There was a divergence with the API and its documentation, since this is a feature that has been rolled out very recently. But it is working ok now.
ref: https://issuetracker.google.com/issues/118122471


Discord link not working

I'm trying to join the Discord as linked from the repo's readme, but I get an error message saying "this invite may be invalid, or you may not have permission to join".
Also, the name of the Discord server the invite links to is "Banshee 3D" and not "BS::Framework".

The Discord lists the invite link as valid. Any chance you have perhaps reached the 100 server limit on your account?
Banshee 3D is the shared discord we use for both projects.

It seems to be a Discord issue. I've logged in on the Discord website and I was able to join, but when I wasn't logged in clicking "join" resulted in an error rather than showing the login screen.
In any case, the link is indeed valid.


[SPARK-24499][DOC][FOLLOW-UP] Split the page of sql-programming-guide.html to multiple separate pages

What changes were proposed in this pull request?
Forgot to clean remove the link for Upgrading From Spark SQL 2.4 to 3.0 when merging to 2.4
How was this patch tested?
N/A

cc @dongjoon-hyun @xuanyuanking

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4100/
Test PASSed.

Test build #97555 has started for PR 22769 at commit 4027ba1.

Test build #97555 has finished for PR 22769 at commit 4027ba1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97555/
Test PASSed.

Merged to branch-2.4.


Vue3 few props not working

I was using vue3.
vue-tel-input: ^6.0.0-beta.7
few props are not working.
import VueTelInput from 'vue3-tel-input';
import 'vue3-tel-input/dist/vue3-tel-input.css';

import App from './App.vue';

const VueTelInputOptions = {
  mode: 'international', // working
  defaultCountry: 'TW', // working
  onlyCountries: ['TW', 'GH', 'GB', 'US', 'CA'], // working
  placeholder: 'Enter2 a phone number', // not working
  dynamicPlaceholder: true, // not working
  enabledCountryCode: true, //not working
};

const app = createApp(App);
app.use(VueTelInput, VueTelInputOptions);

@crazwade Please refer to the list of available props in https://iamstevendao.github.io/vue-tel-input/documentation/props.html. Some of the props you listed are outdated/removed.


grype: 0.34.2 -> 0.34.3

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/anchore/grype/releases.
meta.description for grype is: "Vulnerability scanner for container images and filesystems"
meta.homepage for grype is: "https://github.com/anchore/grype"
meta.changelog for grype is: "https://github.com/anchore/grype/releases/tag/v0.34.3"
Updates performed

Golang update

To inspect upstream changes
Impact

Checks done (click to expand)

built on NixOS
The tests defined in passthru.tests, if any, passed
Warning: no invocation of /nix/store/w5qqpf6mkb3vf6a8bnlk7b7x1lsl6f7v-grype-0.34.3/bin/grype had a zero exit code or showed the expected version
0 of 1 passed binary check by having a zero exit code.
0 of 1 passed binary check by having the new version present in output.
found 0.34.3 with grep in /nix/store/w5qqpf6mkb3vf6a8bnlk7b7x1lsl6f7v-grype-0.34.3
found 0.34.3 in filename of file in /nix/store/w5qqpf6mkb3vf6a8bnlk7b7x1lsl6f7v-grype-0.34.3

Rebuild report (if merged into master) (click to expand)

2 total rebuild path(s)

1 package rebuild(s)

1 x86_64-linux rebuild(s)
1 x86_64-darwin rebuild(s)

First fifty rebuilds by attrpath
grype

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/w5qqpf6mkb3vf6a8bnlk7b7x1lsl6f7v-grype-0.34.3 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A grype https://github.com/r-ryantm/nixpkgs/archive/2c2ffb21fc9cf10e6fb94c48913ed3e764b2dd9b.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/w5qqpf6mkb3vf6a8bnlk7b7x1lsl6f7v-grype-0.34.3
ls -la /nix/store/w5qqpf6mkb3vf6a8bnlk7b7x1lsl6f7v-grype-0.34.3/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

1 package built:

grype

Maintainer pings
cc @fabaff @06kellyjac @kalbasit for testing.

Actually I'm going to update this myself as I want to fix something


UnicodeDecodeError: gbk - Illegal Multibyte Sequence

Traceback (most recent call last):
  File "C:\Users\chenxianrui\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\chenxianrui\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\chenxianrui\AppData\Local\Programs\Python\Python39\Scripts\mkdocs.exe\__main__.py", line 7, in <module>
  File "C:\Users\chenxianrui\AppData\Local\Programs\Python\Python39\lib\site-packages\click\core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\chenxianrui\AppData\Local\Programs\Python\Python39\lib\site-packages\click\core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "C:\Users\chenxianrui\AppData\Local\Programs\Python\Python39\lib\site-packages\click\core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\chenxianrui\AppData\Local\Programs\Python\Python39\lib\site-packages\click\core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\chenxianrui\AppData\Local\Programs\Python\Python39\lib\site-packages\click\core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "C:\Users\chenxianrui\AppData\Local\Programs\Python\Python39\lib\site-packages\mkdocs\__main__.py", line 234, in serve_command
    serve.serve(dev_addr=dev_addr, livereload=livereload, watch=watch, **kwargs)
  File "C:\Users\chenxianrui\AppData\Local\Programs\Python\Python39\lib\site-packages\mkdocs\commands\serve.py", line 83, in serve
    builder(config)
  File "C:\Users\chenxianrui\AppData\Local\Programs\Python\Python39\lib\site-packages\mkdocs\commands\serve.py", line 76, in builder
    build(config, live_server=live_server, dirty=dirty)
  File "C:\Users\chenxianrui\AppData\Local\Programs\Python\Python39\lib\site-packages\mkdocs\commands\build.py", line 332, in build
    config.plugins.run_event('post_build', config=config)
  File "C:\Users\chenxianrui\AppData\Local\Programs\Python\Python39\lib\site-packages\mkdocs\plugins.py", line 522, in run_event
    result = method(**kwargs)
  File "C:\Users\chenxianrui\AppData\Local\Programs\Python\Python39\lib\site-packages\mkdocs_exporter\plugins\pdf\plugin.py", line 151, in on_post_build
    self.loop.run_until_complete(asyncio.gather(*concurrently(self.tasks, max(1, self.config.concurrency or 1))))
  File "C:\Users\chenxianrui\AppData\Local\Programs\Python\Python39\lib\site-packages\nest_asyncio.py", line 90, in run_until_complete
    return f.result()
  File "C:\Users\chenxianrui\AppData\Local\Programs\Python\Python39\lib\asyncio\tasks.py", line 258, in __step
    result = coro.throw(exc)
  File "C:\Users\chenxianrui\AppData\Local\Programs\Python\Python39\lib\site-packages\mkdocs_exporter\plugins\pdf\plugin.py", line 147, in limit
    return await asyncio.create_task(coroutine)
  File "C:\Users\chenxianrui\AppData\Local\Programs\Python\Python39\lib\asyncio\futures.py", line 284, in __await__
    yield self  # This tells Task to wait for completion.
  File "C:\Users\chenxianrui\AppData\Local\Programs\Python\Python39\lib\asyncio\tasks.py", line 328, in __wakeup
    future.result()
  File "C:\Users\chenxianrui\AppData\Local\Programs\Python\Python39\lib\asyncio\futures.py", line 201, in result
    raise self._exception
  File "C:\Users\chenxianrui\AppData\Local\Programs\Python\Python39\lib\asyncio\tasks.py", line 256, in __step
    result = coro.send(None)
  File "C:\Users\chenxianrui\AppData\Local\Programs\Python\Python39\lib\site-packages\mkdocs_exporter\plugins\pdf\plugin.py", line 122, in render
    html = self.renderer.preprocess(page)
  File "C:\Users\chenxianrui\AppData\Local\Programs\Python\Python39\lib\site-packages\mkdocs_exporter\plugins\pdf\renderer.py", line 69, in preprocess
    preprocessor.script(importlib_resources.files(js).joinpath('pagedjs.min.js').read_text())
  File "C:\Users\chenxianrui\AppData\Local\Programs\Python\Python39\lib\pathlib.py", line 1267, in read_text
    return f.read()
UnicodeDecodeError: 'gbk' codec can't decode byte 0xbd in position 6267: illegal multibyte sequence

This error typically occurs when there is an issue with decoding a byte sequence using the 'gbk' codec, which is a character encoding commonly used for simplified Chinese text. It suggests that there might be a problem with the content of the file being processed, possibly due to an encoding mismatch or an invalid multibyte sequence.
I have followed the instruction and installed the prerequisite and when I run the "mkdocs serve", the error msg is above.

Hey @yanedie ,
Have you tried running it in Python virtual environments instead of directly on your system?
I am not an expert but by means of your logs, i would claim that this isn't a problem of this mkdocs-exporter. I had some issues too with this plugin, but most of the times it was a problem relating to my System or my CI/CD environment, and after i fixed it,  the plugin ran without adjustments.

Hello,
I've just released the v4.0.1 which should solve the issue you are currently encountering @yanedie.


Add driver for SinoWealth-based mice

Continuing on #24. This does not yet work at all. I managed to get it working locally, but it involved some terrible hacky work-arounds and I want to make it sane before staging. I will give a heads-up once this is complete.

Rebased on master on squashed a few commits. New ones go after sinowealth: rename a bunch of fields.
@whot I tried to add firmware version field into the ratbag.Device, but the code structure I got was kind of ugly: I moved ratbag.Device creation into start method of SinowealthDevice, and made it return the Device. Maybe you could guide me into a better way to do it?

fwiw, I purposely didn't put much of an API around the device initialization. There's the driver entry point new_with_devicelist but since we're using the HidrawDriver here for some of the boilerplate the only API we need is: at some point probe() is called by ratbag. And following that, you should either throw an exception if the device is not supported/missing something. Or (eventually) emit the device-added signal (one per device, you could have multiple per hidraw device). Beyond that, anything else is left to the driver to do as they see fit, even the signal doesn't need to be emitted in immediate response to start, you could schedule some work for the future which then emits the signal.
So this is the long way to say "whatever works for you" :)
The other two drivers already have the ratbag.Device creation inside start() so that's definitely a good option. Unless you come up with something better in which case I'll be happy to copy that idea from you.

So, depressingly - I think I never saw the above changes. Apologies for letting this slip for that long but I think this looks good. Does it work? :)

@whot, sorry, I haven't taken a look a this in a very long time.
To be honest, I don't really have any motivation to work on this.
Also, since then the sinowealth driver in libratbag has changed a lot.
Still, I'll take a look at it now, although I don't think I will get it into an acceptable state.
Right now, reading from the mouse does seem to work, but the max DPI is incorrect (because the sensor type is not read from the mouse) and every resolution profile is set as enabled even when they aren't.

@whot, sorry, I haven't taken a look a this in a very long time.
To be honest, I don't really have any motivation to work on this.

yeah, I understand and I too struggled to find time for this recently. There's a bit of a chicken and egg situation with this rewrite - until we have driver support there's no motivation to work on it, but without others working on it we'll never get there :) Thanks for your efforts though.

Regarding the failing tests: I'm not sure how to fix that, I understand Reply is a strongly typed object returned by Query.run so we have to overload Query somehow to return a different type of Reply, but don't have an idea how to actually implement that.

Reply is just a class that gets dynamically added attributes depending on the type of query. e.g. the reply_spec for QueryFWVersion has a report_id, cmd and version, so the Reply returned by this query will have those three properties. This way we can have the code using the Reply object look sane. The type of Reply is always reply though, it just uses the python ability to do this:
class Foo(object):
    pass

f = Foo()
f.bar = "bar"

In the hidpp20 code I started separating this a bit, so the replies are actually strongly typed but this was an idea I had after this driver - and tbh I think this driver isn't as complicated as HID++ so it doesn't need it as much.
This obviously doesn't play well with mypy but at least here the best solution is to just use #  type: ignore in those lines.


Query Frequency Indexing Error

We have encounterd a TypeError when executing the code coming from this line on the gui.py code.  It seems the databases is not being indexed properly without this additional call.
freq =  self._gui.freq_db[self.laser]["freq"] -> freq =  self._gui.freq_db[self.laser][0]["freq"]

Thanks for the bug report. Can you link to the original line in the source code via git hub to make it easier to locate please?

Also please can you post all the info we’d need to reproduce this issue like

output of ‘conda list’ and more generally how you installed /set this up
how you launched the code
logs (in verbose mode) from server and master
hardware set up
config files your using etc

As we don’t hit this bug my guess is it’s something to do with simulation mode being broken, a problem with some of the Oxford specific code like config file loading, or the config files being out of date. I’ll look at it next week, but it would help a lot to have enough info to be able to reproduce the issue rather

Thank you for the feedback!  The problematic line is 375 of gui.py, though every line where the code tries to access elements of freq_db runs into the same issues.
https://github.com/OxfordIonTrapGroup/wand/blob/8757c942e74de272ff4b9ea7bf9a10194b543871/wand/gui.py#L373-L376
We ran wand_server through the Anaconda prompt in an environment containing WAnD and get the following output:
$ wand_server -n example -v

INFO:wand.drivers.high_finesse:Connected to WLM server
WARNING:wand.drivers.high_finesse:No Switch Detected
INFO:wand.drivers.high_finesse:Connected to WLM 8 rev 4086, firmware 6491.3
WARNING:wand.tools:Unable to backup configuration file
INFO:wand.frontend.wand_server:server started
INFO:wand.frontend.wand_server:Taking new frequency measurement for example_854
INFO:wand.frontend.wand_server:task 0 complete
INFO:wand.frontend.wand_server:Taking new frequency measurement for example_850
INFO:wand.frontend.wand_server:task 1 complete
INFO:wand.frontend.wand_server:Taking new frequency measurement for example_866
INFO:wand.frontend.wand_server:example_866 fast mode timeout
INFO:wand.frontend.wand_server:task 2 complete

We then ran wand_gui through another Anaconda prompt and get this output:
$ wand_gui -n example -v

INFO:wand.frontend.wand_gui:No connection to server 'example'
INFO:wand.frontend.wand_gui:No connection to server 'example'
INFO:wand.frontend.wand_gui:Reconnected to server 'example'
INFO:wand.frontend.wand_gui:Reconnected to server 'example'
ERROR:wand.gui:Error connecting to server 'example' list indices must be integers or slices, not str here
<class 'TypeError'> gui.py 232
. . . .
WARNING:quamash.QEventLoop:Event callback failed: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request
Traceback (most recent call last):
  File "c:\users\laserstation\anaconda3\envs\wand\lib\site-packages\Quamash-0.5.5-py3.6.egg\quamash\_windows.py", line 41, in _process_events
  File "c:\users\laserstation\anaconda3\envs\wand\lib\asyncio\windows_events.py", line 437, in finish_recv
    return ov.getresult()
OSError: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\users\laserstation\anaconda3\envs\wand\lib\site-packages\Quamash-0.5.5-py3.6.egg\quamash\_windows.py", line 44, in _process_events
  File "c:\users\laserstation\anaconda3\envs\wand\lib\asyncio\windows_events.py", line 79, in set_exception
    super().set_exception(exception)
  File "c:\users\laserstation\anaconda3\envs\wand\lib\asyncio\futures.py", line 361, in set_exception
    raise InvalidStateError('{}: {!r}'.format(self._state, self))
asyncio.futures.InvalidStateError: CANCELLED: <_OverlappedFuture cancelled>

where . . . . indicates that the TypeError was repeatedly printed until the program terminated.  While the master was running, it indicated that there was no connection between the master and the server:

We were able to correct this issue by changing every instance of data being called from freq_db (in both gui.py and  ControlInterface.get_freq in server.py) from, e.g.,
freq = self._gui.freq_db[self.laser]["freq"]
to
freq = self._gui.freq_db[self.laser][0]["freq"]
When we make these changes, the logging for the server and master respectively are as follows:
$ wand_server -n example -v

INFO:wand.drivers.high_finesse:Connected to WLM server
WARNING:wand.drivers.high_finesse:No Switch Detected
INFO:wand.drivers.high_finesse:Connected to WLM 8 rev 4086, firmware 6491.3
WARNING:wand.tools:Unable to backup configuration file
INFO:wand.frontend.wand_server:server started
INFO:wand.frontend.wand_server:Taking new frequency measurement for example_850
INFO:wand.frontend.wand_server:task 0 complete
INFO:wand.frontend.wand_server:Taking new frequency measurement for example_854
INFO:wand.frontend.wand_server:task 1 complete
INFO:wand.frontend.wand_server:Taking new frequency measurement for example_866
INFO:wand.frontend.wand_server:task 2 complete

$ wand_gui -n example

INFO:wand.frontend.wand_gui:No connection to server 'example'
INFO:wand.frontend.wand_gui:No connection to server 'example'
INFO:wand.frontend.wand_gui:Reconnected to server 'example'
INFO:wand.frontend.wand_gui:Reconnected to server 'example'

and the GUI indicates that it is able to communicate with the server.
Lastly, if we add a line to the very start of update_freq in gui.py:
print(self._gui.freq_db[self.laser][0])
we get the following output from the master:
$ wand_gui -n example

INFO:wand.frontend.wand_gui:No connection to server 'example'
INFO:wand.frontend.wand_gui:No connection to server 'example'
INFO:wand.frontend.wand_gui:Reconnected to server 'example'
INFO:wand.frontend.wand_gui:Reconnected to server 'example'
{'status': 1, 'timestamp':<PHONE_NUMBER>.642318, 'freq': 0}
ERROR:wand.gui:Error connecting to server 'example' list indices must be integers or slices, not str here
<class 'TypeError'> gui.py 232
. . . .
ERROR:wand.gui:Error connecting to server 'example' list indices must be integers or slices, not str here
<class 'TypeError'> gui.py 232
WARNING:quamash.QEventLoop:Event callback failed: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request
Traceback (most recent call last):
  File "c:\users\laserstation\anaconda3\envs\wand\lib\site-packages\Quamash-0.5.5-py3.6.egg\quamash\_windows.py", line 41, in _process_events
  File "c:\users\laserstation\anaconda3\envs\wand\lib\asyncio\windows_events.py", line 437, in finish_recv
    return ov.getresult()
OSError: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\users\laserstation\anaconda3\envs\wand\lib\site-packages\Quamash-0.5.5-py3.6.egg\quamash\_windows.py", line 44, in _process_events
  File "c:\users\laserstation\anaconda3\envs\wand\lib\asyncio\windows_events.py", line 79, in set_exception
    super().set_exception(exception)
  File "c:\users\laserstation\anaconda3\envs\wand\lib\asyncio\futures.py", line 361, in set_exception
    raise InvalidStateError('{}: {!r}'.format(self._state, self))
asyncio.futures.InvalidStateError: CANCELLED: <_OverlappedFuture cancelled>

From this output, it looks like freq_db is being initialized as a dictionary of lists of dictionaries, with each list being of length 1, instead of just a dictionary of dictionaries.
It's very much worth noting that our version of the code is heavily modified.  We removed all material related solely to OSA, and it's entirely possible we changed something important in the process.  However, freq_db appears to be initialized in more or less the same way as laser_db is (in the below code snippet?), so we don't see why they would be initializing with different structures.
https://github.com/OxfordIonTrapGroup/wand/blob/8757c942e74de272ff4b9ea7bf9a10194b543871/wand/frontend/wand_gui.py#L219-L260
On the software environment side, we're running this code on a Windows 10 machine in an environment with the following packages:
$ conda list

# Name                    Version                   Build  Channel
aiohttp                   3.4.2            py35hfa6e2cd_0
alabaster                 0.7.11                   py35_0
artiq                     5.6911.e9b78b62.beta               0    https://conda.m-labs.hk/artiq-beta
artiq-board-kasli-mitll   5.6911.e9b78b62.beta               0    https://conda.m-labs.hk/artiq-beta
artiq-board-kc705-nist_clock 5.6911.e9b78b62.beta               0    https://conda.m-labs.hk/artiq-beta
astroid                   1.6.5                    py35_0
async-timeout             3.0.0                    py35_0
attrs                     19.1.0                     py_0
babel                     2.7.0                      py_0
backcall                  0.1.0                    py35_0
binutils-or1k-linux       2.27                 h93a10e1_6    m-labs
blas                      1.0                         mkl
bleach                    2.1.4                    py35_0
bscan-spi-bitstreams      0.10.0                        2    m-labs
ca-certificates           2019.5.15                     1
certifi                   2018.8.24                py35_1
cffi                      1.11.5           py35h74b6da3_1
chardet                   3.0.4                    py35_1
cloudpickle               1.2.1                      py_0
colorama                  0.3.9            py35h32a752f_0
curl                      7.65.2               h2a8f88b_0
decorator                 4.4.0                      py_0
defusedxml                0.6.0                      py_0
docutils                  0.14             py35h8ccb97f_0
entrypoints               0.2.3                    py35_2
git                       2.22.0                        0    conda-forge
h5py                      2.8.0            py35h3bdd7fb_2
hdf5                      1.10.2               hac2f561_1
html5lib                  1.0.1                    py35_0
icc_rt                    2019.0.0             h0cc432a_1
icu                       58.2                 ha66f8fd_1
idna                      2.7                      py35_0
idna_ssl                  1.1.0                    py35_0
imagesize                 1.1.0                    py35_0
intel-openmp              2019.4                      245
ipykernel                 4.10.0                   py35_0
ipython                   6.5.0                    py35_0
ipython_genutils          0.2.0            py35ha709e79_0
isort                     4.3.4                    py35_0
jedi                      0.12.1                   py35_0
jinja2                    2.10                     py35_0
jpeg                      9b                   hb83a4c4_2
jsonschema                2.6.0            py35h27d56d3_0
jupyter_client            5.3.1                      py_0
jupyter_core              4.5.0                      py_0
keyring                   13.2.1                   py35_0
krb5                      1.16.1               hc04afaa_7
lazy-object-proxy         1.3.1            py35hfa6e2cd_2
libcurl                   7.65.2               h2a8f88b_0
libgit2                   0.27.8               hfac1375_0    conda-forge
libiconv                  1.15                 h1df5818_7
libpng                    1.6.37               h2a8f88b_0
libsodium                 1.0.16               h9d3ae62_0
libssh2                   1.8.2                h7a1dbc1_0
libusb                    1.0.20                        0    m-labs
llvmlite-artiq            0.23.0.dev               py35_5    m-labs
markupsafe                1.0              py35hfa6e2cd_1
mccabe                    0.6.1                    py35_1
mistune                   0.8.3            py35hfa6e2cd_1
mkl                       2018.0.3                      1
mkl_fft                   1.0.6            py35hdbbee80_0
mkl_random                1.0.1            py35h77b88f5_1
multidict                 4.4.0            py35hfa6e2cd_0
nbconvert                 5.5.0                      py_0
nbformat                  4.4.0            py35h908c9d9_0
nidaqmx                   0.5.7                    pypi_0    pypi
numpy                     1.15.2           py35ha559c80_0
numpy-base                1.15.2           py35h8128ebf_0
numpydoc                  0.9.1                      py_0
openocd                   0.10.0                        6    m-labs
openssl                   1.1.1c               he774522_1
packaging                 17.1                     py35_0
pandoc                    <IP_ADDRESS>                       0
pandocfilters             1.4.2                    py35_1
parso                     0.5.1                      py_0
pickleshare               0.7.4            py35h2f9f535_0
pip                       18.0                  py35_1001    conda-forge
prettytable               0.7.2                      py_3    conda-forge
prompt_toolkit            1.0.15           py35h89c7cb4_0
psutil                    5.4.7            py35hfa6e2cd_0
pycodestyle               2.4.0                    py35_0
pycparser                 2.19                     py35_0
pydaqmx                   1.4.2                    pypi_0    pypi
pyflakes                  2.0.0                    py35_0
pygit2                    0.27.1           py35hfa6e2cd_0    conda-forge
pygments                  2.4.2                      py_0
pylint                    1.9.2                    py35_0
pyparsing                 2.4.2                      py_0
pyqt                      5.9.2            py35h6538335_2
pyqt5                     5.13.0                   pypi_0    pypi
pyqt5-sip                 4.19.18                  pypi_0    pypi
pyqtgraph                 0.10.0           py35h28b3542_3
python                    3.5.6                he025d50_0
python-dateutil           2.7.3                    py35_0
python-levenshtein        0.12.0                   py35_0    conda-forge
pythonparser              1.1                        py_8    m-labs
pytz                      2019.2                     py_0
pywin32                   223              py35hfa6e2cd_1
pyzmq                     17.1.2           py35hfa6e2cd_0
qt                        5.9.7            vc14h73c81de_0
qtawesome                 0.5.7                      py_0
qtconsole                 4.5.3                      py_0
qtpy                      1.9.0                      py_0
quamash                   0.5.5                      py_4    m-labs
regex                     2018.08.29       py35hfa6e2cd_0
requests                  2.13.0                   py35_0    conda-forge
rope                      0.14.0                     py_0
scipy                     1.1.0            py35h4f6bf74_1
setuptools                40.2.0                   py35_0
simplegeneric             0.8.1                    py35_2
sip                       4.19.8           py35h6538335_0
six                       1.11.0                   py35_1
snowballstemmer           1.9.0                      py_0
sphinx                    2.1.2                      py_0
sphinxcontrib-applehelp   1.0.1                      py_0
sphinxcontrib-devhelp     1.0.1                      py_0
sphinxcontrib-htmlhelp    1.0.2                      py_0
sphinxcontrib-jsmath      1.0.1                      py_0
sphinxcontrib-qthelp      1.0.2                      py_0
sphinxcontrib-serializinghtml 1.1.3                      py_0
spyder                    3.3.1                    py35_1
spyder-kernels            0.2.6                    py35_0
sqlite                    3.29.0               he774522_0
testpath                  0.3.1            py35h06cf69e_0
tk                        8.6.8                hfa6e2cd_0
tornado                   5.1.1            py35hfa6e2cd_0
traitlets                 4.3.2            py35h09b975b_0
vc                        14.1                 h0510ff6_4
vs2015_runtime            14.15.26706          h3a45250_4
wand                      0.1                      pypi_0    pypi
wcwidth                   0.1.7            py35h6e80d8a_0
webencodings              0.5.1                    py35_1
wheel                     0.31.1                   py35_0
win_unicode_console       0.5              py35h56988b5_0
wincertstore              0.2              py35hfebbdb8_0
wrapt                     1.10.11          py35hfa6e2cd_2
yarl                      1.2.6            py35hfa6e2cd_0
zeromq                    4.2.5                he025d50_1
zlib                      1.2.11               h62dcd97_3

and our configuration files (example_server_config.pyon and example_gui_config.pyon respectively) are shown below:
{"lasers": {
    "example_850": {
        "auto_exposure": true,
        "channel": 1,
        "exposure": [12, 2],
        "f_ref":<PHONE_NUMBER>00000.0,
        "fast_enabled_at": 0,
        "fast_mode": false,
        "fast_mode_set_at":<PHONE_NUMBER>.9685378,
        "host": "<IP_ADDRESS>",
        "lock_capture_range":<PHONE_NUMBER>.0,
        "lock_gain": 1e-10,
        "lock_owner": "",
        "lock_poll_time": 0.1,
        "lock_ready": true,
        "locked": false,
        "locked_at": 0
    },
    "example_854": {
        "auto_exposure": true,
        "channel": 2,
        "exposure": [8, 4],
        "f_ref":<PHONE_NUMBER>00000.0,
        "fast_enabled_at": 0,
        "fast_mode": false,
        "fast_mode_set_at":<PHONE_NUMBER>.0285378,
        "host": "<IP_ADDRESS>",
        "lock_capture_range":<PHONE_NUMBER>.0,
        "lock_gain": 2e-09,
        "lock_owner": "me",
        "lock_poll_time": 0.1,
        "lock_ready": true,
        "locked": false,
        "locked_at":<PHONE_NUMBER>.006538
    },
    "example_866": {
        "auto_exposure": false,
        "channel": 3,
        "exposure": [5, 3],
        "f_ref":<PHONE_NUMBER>00000.0,
        "fast_enabled_at": 0,
        "fast_mode": false,
        "fast_mode_set_at":<PHONE_NUMBER>.3268256,
        "host": "<IP_ADDRESS>",
        "lock_capture_range":<PHONE_NUMBER>.0,
        "lock_gain": 1e-10,
        "lock_owner": "",
        "lock_poll_time": 0.1,
        "lock_ready": true,
        "locked": false,
        "locked_at":<PHONE_NUMBER>.6695378
    }
}}

{
    "servers": {
        "example": {
                "host": "<IP_ADDRESS>",
                "notify": 3250,
                "control": 3251
        }
    },
    "display_names": {

        "866":"example_866",
        "850":"example_850",
        "854":"example_854",

    },
    "layout": [
        ["850", "854", "866", ]

    ]
}

Thank you again for your time and energy.  I don't know how reproducible this error will be with the code in this repository, since again, our code is modified.  We can send the code we are using if it would help.

Hmmm...the change you’ve made to the code doesn’t look right to me so I think something is very broken here.
Please try running an unmodified version of the code in simulation mode (no hardware) and see what happens.

to elaborate: if you look at where the freq dB is defined it’s just a dictionary keyed by the laser names https://github.com/OxfordIonTrapGroup/wand/blob/8757c942e74de272ff4b9ea7bf9a10194b543871/wand/frontend/wand_server.py#L123 so I don’t see why the extra index would be needed

Glad you were able to sort it out. Do you have everything up and running now?
BTW I've pushed a few commits recently that may make things easier for you.


Add DrawAndEditGraphics3D.

@anttikajanus - Just adding DrawAndEditGraphics for 3D. Should be same as 2D. Can you take a quick look.
K

Looks good after small cleanup. Please merge after that.

Remove Scaling options and other that aren't supported in 3D.

Continuing my previous comment. We have removed Scale/Rotate options in 3d.

Thx for catching that; removing/re-checking options now.


Fix "Unread" count on notification page

Issue
Unread notification count doesn't include local notifications
Fix
Add both the count and display it in the unread notification count.
Screenshot of the Fix

Note
Right now I don't have any unread github notification. So not able to test it.
Please check that too :)
Fixes #427

👍 works as expected. Thanks for another PR 😊

Great :)


fixing single attribute selectors

This is a WIP to fix the single attribute selectors when saved. They previously weren't appearing as grayed out because the disabled prop wasn't getting passed through (this was happening in a few other places too which I've changed).
However now there's the behavior of attribute selectors not holding their saveData value when they are used in a saved transformation. I logged saveData and then attributeName from inside the body of the TransformColumn component during the creation of a saved transformation and saw this:

Which is suspicious, given it's happening twice and the second time attributeName is null though it is present in the saveData.
Let me know if you have ideas about a fix for this / feel free to commit anything

Basically, to keep the attribute selection updated (i.e., if the chosen attribute gets deleted, we reset the dropdown), we use a useEffect to check if the chosen attribute is still an attribute of the selected data context. But in a saved transformation, the attribute needs to stay the same even when there's no data context, so I added a change that ignores the result of useAttribute if the selector is disabled.


⚠ AZ-Launcher has degraded performance

In c553b28, AZ-Launcher (https://az-launcher.nz/) experienced degraded performance:

HTTP code: 200
Response time: 2838 ms

Resolved: AZ-Launcher performance has improved in 38738a9 after 11 minutes.


[turborepo] cache doesn't invalidate on change in dependent code.

What version of Turborepo are you using?
1.9.1
What package manager are you using / does the bug impact?
npm
What operating system are you using?
Mac
Describe the Bug
I am running a turbo monorepo that has a single workspace right now.  web
The web is a react ssr app, built using webpack and has multiple subparts like dashboard etc.
My turbo.json is defined as below, which specifies few tasks to build each of them.
{
  "$schema": "https://turbo.build/schema.json",
  "pipeline": {
    "build": {
      "outputs": []
    },
    "build:client": {
      "inputs": ["!**/views/dashboard/**", "!**/views/supply/**", "!**/views/partners/**", "!src/**/server.js", "src/views/**/*.*"],
      "outputs": ["build/**/!(*dashboard*|*supply*|*partners*|*server*).*"],
      "dependsOn": ["^build:client"]
    },
    "build:dashboard": {
      "inputs": ["src/views/dashboard/**/*"],
      "outputs": ["build/**/*dashboard*.js"],
      "dependsOn": ["^build:dashboard"]
    },
    "build:supplier": {
      "inputs": ["src/views/supply/**"],
      "outputs" : ["build/**/*supply*.js"],
      "dependsOn": ["^build:supplier"]
    },
    "build:all": {
      "outputs": []
    }
  }
}

There is also a build:all command, which essentially runs build:client build:supplier build:dashboard commands.
The inputs for each commands are clearly defined based on the entry points defined in part of the webpack config shared below
const env = config.global.env
  const definePluginAdditionalVariables = {}

  if (env === 'development') {
    require('dotenv').config()
    environmentVariablesCheck(developmentRequiredEnvironmentVariables)
    developmentRequiredEnvironmentVariables.forEach(
      entry => (definePluginAdditionalVariables[`process.env.${entry}`] = JSON.stringify(process.env[entry])),
    )
  } else if (env === 'production') {
    environmentVariablesCheck(requiredEnvironmentVariables)
    requiredEnvironmentVariables.forEach(
      entry => (definePluginAdditionalVariables[`process.env.${entry}`] = JSON.stringify(process.env[entry])),
    )
  }

  const clientConfig = {}
  clientConfig.name = 'app'
  clientConfig.target = 'web'
  clientConfig.entry = {
    dashboardIndex: path.join(config.dirs.path_src, 'views', 'dashboard', 'index.js'),
    dashboardIndex: path.join(config.dirs.path_src, 'views', 'dashboard', 'index.js'),
    dashboardLeadsTemplates: path.join(config.dirs.path_src, 'views', 'dashboard', 'leads', 'templates.js'),
    dashboardInventories: path.join(config.dirs.path_src, 'views', 'dashboard', 'inventories', 'index.js')
.
.
.
  }
  clientConfig.output = {
    path: config.dirs.path_build_public,
    filename: '[name].js',
  }

  clientConfig.bail = false //Dont continue if errors
  clientConfig.devtool = false
  clientConfig.module = {
    rules: [
      {
        test: [/\.jsx?$/, /\.tsx?$/],
        exclude: /node_modules/,
        loader: 'babel-loader',
        options: {
          presets: [
            [
              '@babel/preset-env',
              {
                modules: false,
                loose: true,
                useBuiltIns: 'usage',
                corejs: 3,
              },
            ],
            [
              '@babel/preset-react',
              {
                runtime: 'automatic',
              },
            ],
            '@babel/preset-typescript',
          ],
          cacheDirectory: config.dirs.path_build_tmp,
        },
      }]
  }

  clientConfig.resolve = {
    modules: [config.dirs.path_root, 'node_modules'],
    extensions: ['.tsx', '.ts', '.js', '.jsx', '...'],
    alias: {
      data: 'src/data',
      config: 'config',
      routes: 'src/routes',
      middlewares: 'src/middlewares',
      views: 'src/views',
      helpers: 'src/helpers',
      hooks: 'src/views/hooks',
      components: 'src/views/_components',
      layouts: 'src/views/_layouts',
      styles: 'src/views/_styles',
      javascripts: 'src/views/_javascripts',
    },
    fallback: {
      path: require.resolve('path-browserify'),
      stream: require.resolve('stream-browserify'),
      'process/browser': require.resolve('process/browser'),
    },
  }

When there is a change in the inputs mentioned here, the cache invalidates correctly. But when any of the imports change, the cache isn't invalidated.
Expected Behavior
When I run build:dashboard once, it should build and cache the output.
CURRENT BEHAVIOUR:
When any of the imports in the files specified in the input directive change, it constitutes a Cache Hit.
EXPECTED BEHAVIOUR:
BUT when any of the imports in the files specified in the input directive change, the cache SHOULD be INVALIDATED too. (Cache miss should occur).
To Reproduce
Run turbo build  in a repo. Ensure the result gets cached in node_modules/.cache/turbo folder.
Make a change in any of the imports in the file specified in the inputs directive (not the actual files).
Run turbo build again, the cache doesn't invalidate and result is hydrated from the cache itself.
Reproduction Repo
No response

Turborepo does not trace your code to find dependencies. You will need to specify all of the source files a particular build depends on (or omit inputs to depend on everything). So if build:dashboard can import files from your repository that are not currently specified in the inputs for build:dashboard, you will need to update the inputs to add them, or break them out into a separate workspace and add a package-level dependency on them.


How to contribute?

I have some experience finetuning LLMs and synthetic data and would like to know more about how to contribute; already submitted the form but have not received the invitation from the Slack channel.
Just want to know, is there any plan for the (outer) community? For example, if I clone this repo, add some code, and make a PR, is that OK to try?

If you fill out the form but don't receive an invitation, you may not have filled out the information completely. Let me know your email, thanks!

Thanks for the replying. I received the invitation from Junyang right after raising this issue. Looking forward to the following discussions and planning!


modify Executor side to support Multiple executor feature

The pull request contains the following features.

RestfulAPIClient
Abstract the logic from the ExectorMgr that handles the communication to Exectors via RestfulAPI into a separate class .
Code includes  -
a. A base class with all the base logic for RESTful API call inplace.
b. A wrapper class for ExecutorApiClient which parses the result from API call in JSON string.
c. unit tests.

ExecutorInfo Servlet
New servlet from the executor side that exposes the ExecutorInfo (memory, cpu, flow capacity, last assigned time etc.)
Code includes  -
a. implementation of all the getter methods that fetch the KPI value upon request.
b. unit tests.

ExecutorSelector
Mechanism to select the most suitable executor based from a list of the candidates based on the weighted statistics.
Code includes  -
a.  Framework to register filter and weighted comparator that will be used in candidate selection.
b.  Easy to use wrapper class that takes in filter name List and comparator name/wieght KVPs to initialize the selector.
c.  Implementation of system level filters for flow capacity and memory.
d.  Implementation of  comparators for the supported statistics KPIs .
e.  unit tests.

LGTM

LGTM

Tracking bug: #439


Multiple sounds for "soundfonts"

I think .memefont is a good file extension
Basically for each MIDI instrument, have a section in a .ini file (not necessarily with any specific extension).
Sample section contents
[default]
file=doot.wav
volume=100 #out of 100 instead of the 127 native
#vol is an alias for volume

Section naming/numbering
[001], [01], [1], and [acoustic-grand-piano] are all accepted ways of saying MIDI instrument #1. To cover a group of instruments, use the Wikipedia section name, e.g. [piano] is MIDI instruments 1-8.
Options
For each command-line option (which I should really organize better, BTW) you can set them in the ini file too, in a section marked [options]:
[options]
fullclip=true
transpose=12
output=dootalovania.wav

After further consideration it should just be .meme

progress with commit 0bf0358

Nah it should be .swood but it should be flexible about different ext's

Remember to implement channels (as ch1 or channel1 or channel 1 1-indexed) but other than that I'm done

I was thinking of tracks, not channels: t1 track1 track 1`

Done! It's a bit slower for simple requests as it has to generate a compatible default SoundFont (not to mention the extra lookups in tight loops, mainly in render.py), but I think that's acceptable. Once PyPy gets Py3.5/3.4 support I'll ensure the installer (issue #19) works with PyPy by default.


[WIP] Reorganise profiles into dirs for each provider

This PR is related to #453. It was actually a straightforward renaming and moving of files. Because all the functions in the profiles exist in the profiles package (both before and after the refactor), there were no code changes needed as far as I could see.
I tested this by building a local binary of kubicorn with the new profile layout and successfully created an Ubuntu cluster on AWS.
Unfortunately I wasn't able to get make test running successfully (even before making any code changes) so I wasn't able to verify that the tests also didn't need any code changes, but from reading the code this seems to be okay also.

Hi, thank you very much for the PR.
However, there are several problems.
First one is Go convention. By style guide, it's recommended to call packages by directory name.
For example, this one: https://github.com/gar/kubicorn/blob/7db7a11de523b9d86cf51b49b3a6f39af7ba2563/profiles/digitalocean/ubuntu_16_04.go#L15
Package should be called digitalocean instead of profiles, because it's located in the digitalocean subdirectory.
Second, I think I already mentioned this in the original issue, but you'll have to update other files as well. As far as I know there's cmd/create.go and tests directory that depends on profiles package. Make sure to change the package name (e.g. instead of profiles.NewUbuntuDigitalOceanCluster, it's digitalocean.NewUbuntuDigitalOceanCluster and import parts.
That should fix tests.
Optionally, you can rename functions. As package is already called digitalocean for example, we can drop DigitalOcean from structs/functions name (e.g. NewUbuntuDigitalOceanCluster to NewUbuntuCluster).
If you have any question, ping me. :)

@xmudrii  I've redone this with each profile in it's own package now and I've renamed the functions anywhere they were used. If there are any other issues with the PR just let me know and I'll fix them up!
The one thing I'm not sure about is when there was a "clash" in package names (e.g. test/amazon/ubuntu_cluster_test.go is in package amazon but needs to import the profile which is also in a package amazon). In this case I've just given the package an alias profile, which I hope is okay.

@gar This looks very good. Thank you very much for the contribution! I agree on aliasing profile and it'll do the job for now.


Fix for possibly undefined variable in CI script

This pull request includes:

[ ] adequate testing for the new functionality or fixed issue
[ ] adequate documentation informing people about the change such as

[ ] submit a PR for the guides repository if this PR changed any behavior described there: https://www.osbuild.org/guides/

@jrusz it looks like I had pushed this change in my personal fork instead of the previous PR. :-(


feat: vault selection component for issue page

Adding dropdown to allow user to choose specific vault to make issue request with.
Behaviour
If inputted BTC amount exceeds the capacity, checkbox is unchecked and disabled.
If the vault is selected and the user increases the BTC amount which then exceeds the selected vault capacity, vault is deselected.
If checkbox is checked and no vault is selected, error is shown and it's not possible to make issue request until a) vault is selected or b) checkbox is unchecked.
Preview
Checkbox is unchecked by default

Vaults dropdown

Vault is selected

After the discussion, we decided to add feature of self-issuing to Vault page reusing the PR https://github.com/interlay/interbtc-ui/pull/274/

@peterslany
Can you please resolve the conflict?

If the vault is selected and the user increases the BTC amount which then exceeds the selected vault capacity, vault is deselected.

I have a comment on this behavior. In this case, users would have no idea why that vault selection is cancelled. I guess it could be a UX problem.

I think we should be able to display an error message properly that expresses the underlying issue exactly. "The amount to issue is greater than the capacity that the selected vault can handle."
Besides that, if the vault selection is gone, there's a layout shifting which have an effect on UX as well. (CLS I mean)
It looks like "Max issuable in single request" is not aligned with the selected vault, maybe in general I guess. I'd say that max issuable amount in a single request should be dynamic according to a selected vault.

From a UX perspective, if the user input an amount mistakenly, the vault selection is gone and they have to click the checkbox and select the vault again. It's not convenient I'd say.

Reagarding this and the CLS comment, I implemented it according to the [ticket] (https://www.notion.so/interlay/Kintsugi-Bridge-Add-option-to-issue-with-a-specific-vault-cc691ae3bd6f4e379d50d811733dabf6):

IF they have entered a value that cannot be issued in one transaction by one vault, the checkbox will be deactivated.

Would you suggest showing the dropdown at all times instead?

Implementing discussed changes https://www.notion.so/Kintsugi-Bridge-Add-option-to-issue-with-a-specific-vault-cc691ae3bd6f4e379d50d811733dabf6?d=08b102c7767c45c7bdb2e287362366b5
BTC capacity information added

IF inputted BTC amount > max issuable amount, checkbox is disabled and dropdown is hidden with warning below the BTC amount input field

IF inputted BTC amount > capacity of selected vault, icon besides the vault is changed to red crossmark, dropdown border turns red and warning is displayed below the


Unable to download to ZFS mount using snap install

Storage Explorer Version: 1.14.2
Build Number: 20200715.2
Platform/OS: Ubuntu 20.04
Architecture: x64
Regression From:  unknown
Bug Description
Azure Storage Explorer snap install is unable to access zfs mounts
Steps to Reproduce
Create a ZFS pool and mount, and verify access to it via other apps (terminal, GUI apps, etc)
Install storage-explorer via Snap Store
Select item in Storage Explorer and click 'Download'
File dialog appears, but any mounted ZFS filesystems are missing
Expected Experience
Expect ZFS mounts/volumes to appear normally in dialog, and to be able to download data to them
Actual Experience
ZFS mounts are missing. Clicking a bookmark to a ZFS mount just sends the user to /

@superstator if you try downloading a file in Chrome/Chromium, does the ZFS pool show up?

Yes, other apps show the ZFS pool in the file dialog and can save files there.

What other apps did you try?

Chrome, VSCode, RStudio, Evolution, Firefox. All can access the pool fine. So far StorageExplorer is the only place I've seen I can't access it.

Ok, thanks, we'll try to look into this when we have time.

How to create a ZFS pool

@superstator Using the .tar installation, I can see a ZFS pool. I also tried connecting the removable-media interface, but that didn't work.
According to this forum post it looks like snaps are unable to see other files systems not under the /media folder. It looks like there's a way around that limitation using mount bind. Maybe give that a try? I'm attempting to do the same on my end.

I haven't heard from the author in a while after posting about the workaround, so I'm going to close. If you encounter this problem again, please open a new issue.

Sorry, this has been way down my list too. I just tried your bind mount workaround and that does appear to make it visible. Thanks!


Merge join into connect on network adapters

#156 resulted in a refactor which highlighted that when we set up each network adapter, we essentially call adapter.connect() followed immediately by adapter.join(). The join is superfluous.
This also renames leave to disconnect and removes the unused join on the network subsystem.
We could also perhaps remove leave from the subsystem too? It isn't used anywhere yet.

Leave also removed


Remove result column

BACKEND PULL REQUEST
Related Issue

#3757
#4306

Changes Proposed

Removes all references to the result (testResult) column and replaces the usage everywhere with Results (set of result)
Require Set<Result> when constructing the Test Event since it is an immutable object

Additional Information

Lazy loading the results may be a better option than eager loading them as they both really experience the same n+1 problem.

FetchType.LAZY will generate n queries to get results when requested like testEvent.getResults()
FetchType.EAGER will generate n queries  to immediately fetch results (but actually does 2n queries

It is possible to decrease the impact of n queries by setting the batch size by using @BatchSize
Hibernate.initialize is used for forcing data to be fetched before closing the session. This is just a more explanatory way of loading data rather than doing something like testEvent.getResults to force the data to be loaded.

This generates n queries
Our tests run outside of transactions/session so we need to set hibernate.enable_lazy_load_no_trans to true in order to fetch results.

To avoid n+1 JOIN FETCH will need to be applied on every @Query where you need the results eagerly. This will only generate one query. (Example)
You may be able to set @Fetch(FetchMode.JOIN) on the results field to do a similar action as above. This will eager load the data and should only issue one query, but when I tested this it was issuing n queries in our case.

Testing

Smoke test parts of the app that uses test results:

test queue
results tab

correct a result
view details
print modal
download results

Checklist for Primary Reviewer

[ ] Any large-scale changes have been deployed to test, dev, or pentest and smoke tested
[ ] Any content updates (user-facing error messages, etc) have been approved by content team
[ ] Any changes that might generate questions in the support inbox have been flagged to the support team
[ ] GraphQL schema changes are backward compatible with older version of the front-end
[ ] Changes comply with the SimpleReport Style Guide
[ ] Changes with security implications have been approved by a security engineer (changes to  authentication, encryption, handling of PII, etc.)
[ ] Any dependencies introduced have been vetted and discussed
[ ] Any changes to the startup configuration have been documented in the README

@georgehager I think most of those issues were caused because I did not merge main into this branch. I did that and redeployed to dev for you to test again!


gpu inference speed is slower than the cpu

CPU inference ： | INFO | main:process: 67-process time: 2742.7728176116943ms
GPU inference ： | INFO | main:process: 67-process time: 5317.4756874159873ms
I use the big-lama model for inference
My cpu inference speed is about 2000+ms
But using gpu inference to reach 5000+ms
Is that normal?

Abnormal, the gpu will run a little slower for the first time, but it is definitely faster than the cpu, what is your graphics card?

What is your testing script like? Have you installed the CUDA version of PyTorch correctly?

1、The core processing logic is as below：
def process(self, origin_image_bytes, mask_bytes, form):
image, alpha_channel = load_img(origin_image_bytes)
original_shape = image.shape
interpolation = cv2.INTER_CUBIC
      size_limit: Union[int, str] = form.get("sizeLimit", "1080")
      if size_limit == "Original":
          size_limit = max(image.shape)
      else:
          size_limit = int(size_limit)

      config = Config(
          ldm_steps=form["ldmSteps"],
          hd_strategy=form["hdStrategy"],
          hd_strategy_crop_margin=form["hdStrategyCropMargin"],
          hd_strategy_crop_trigger_size=form["hdStrategyCropTrigerSize"],
          hd_strategy_resize_limit=form["hdStrategyResizeLimit"],
      )
      # logger.info(f"config:{config}")
      # logger.info(f"Origin image shape: {original_shape}")
      image = resize_max_size(image, size_limit=size_limit, interpolation=interpolation)
      # logger.info(f"Resized image shape: {image.shape}")

      mask, _ = load_img(mask_bytes, gray=True)

      mask = resize_max_size(mask, size_limit=size_limit, interpolation=interpolation)

      start = time.time()
      res_np_img = self.model(image, mask, config)
      logger.info(f"process time: {(time.time() - start) * 1000}ms")

      torch.cuda.empty_cache()

      if alpha_channel is not None:
          if alpha_channel.shape[:2] != res_np_img.shape[:2]:
              alpha_channel = cv2.resize(
                  alpha_channel, dsize=(res_np_img.shape[1], res_np_img.shape[0])
              )
          res_np_img = np.concatenate(
              (res_np_img, alpha_channel[:, :, np.newaxis]), axis=-1
          )

      ext = self.get_image_ext(origin_image_bytes)
      return numpy_to_bytes(res_np_img, ext)

#############################################################
model_name = "lama"
device = "cuda"
2、 Name: torch
Version: 2.1.1+cu121
Python：3.11.5

I seem to have encountered this problem before. You can try adding these lines of code at the beginning of the file.
import torch
try:
    torch._C._jit_override_can_fuse_on_cpu(False)
    torch._C._jit_override_can_fuse_on_gpu(False)
    torch._C._jit_set_texpr_fuser_enabled(False)
    torch._C._jit_set_nvfuser_enabled(False)
except:
    pass

I seem to have encountered this problem before. You can try adding these lines of code at the beginning of the file.
https://github.com/Sanster/IOPaint/blob/a26bf7a0c9dc0f8faad77aa2591793ad20724831/iopaint/api.py#L14
import torch
try:
    torch._C._jit_override_can_fuse_on_cpu(False)
    torch._C._jit_override_can_fuse_on_gpu(False)
    torch._C._jit_set_texpr_fuser_enabled(False)
    torch._C._jit_set_nvfuser_enabled(False)
except:
    pass

thanks！