Send a SIGINT before sending a SIGKILL in DDP

What does this PR do?
This is a proposal for fixing #16285. When shutting down DDP training, we first try sending a SIGINT to the other processes -- this (by default) raises a KeyboardInterrupt exception, and since that's an exception all context managers + try/finally blocks will run as normal for graceful termination. If the processes don't shut down "soon", then we send a SIGKILL later.
Let me know if this is an acceptable approach!
TODO:

[ ] Update the CHANGELOG

Hm... this might be a bit of a dead-end. I tried running the test as an independent script, and the main process just hangs at the following backtrace (obtained via py-spy)
    __futex_abstimed_wait_common (libc.so.6)
    pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6)
    std::condition_variable::wait (libstdc++.so.6.0.30)
    std::condition_variable::wait (libstdc++.so.6.0.30)
    c10::ivalue::Future::wait (libtorch_cpu.so)
    c10d::Reducer::finalize_backward (libtorch_cpu.so)
    std::_Function_handler<void (), c10d::Reducer::mark_variable_ready(unsigned long)::{lambda()#1}>::_M_invoke (libtorch_cpu.so)
    torch::autograd::GraphTask::exec_post_processing (libtorch_cpu.so)
    torch::autograd::GraphTask::mark_as_completed_and_run_post_processing (libtorch_cpu.so)
    torch::autograd::Engine::thread_main (libtorch_cpu.so)
    torch::autograd::Engine::execute_with_graph_task (libtorch_cpu.so)
    torch::autograd::python::PythonEngine::execute_with_graph_task (libtorch_python.so)
    torch::autograd::Engine::execute (libtorch_cpu.so)
    torch::autograd::python::PythonEngine::execute (libtorch_python.so)
    THPEngine_run_backward (libtorch_python.so)
    backward (torch/autograd/__init__.py:197)
    backward (torch/_tensor.py:488)
    backward (pytorch_lightning/core/module.py:1452)
    backward (pytorch_lightning/plugins/precision/precision_plugin.py:67)
    backward (pytorch_lightning/strategies/strategy.py:207)
    _call_strategy_hook (pytorch_lightning/trainer/trainer.py:1485)
    backward_fn (pytorch_lightning/loops/optimization/optimizer_loop.py:305)
    closure (pytorch_lightning/loops/optimization/optimizer_loop.py:144)
    __call__ (pytorch_lightning/loops/optimization/optimizer_loop.py:149)
    _wrap_closure (pytorch_lightning/plugins/precision/precision_plugin.py:105)
    step (torch/optim/sgd.py:130)
    _use_grad (torch/optim/optimizer.py:23)
    wrapper (torch/optim/optimizer.py:140)
    wrapper (torch/optim/lr_scheduler.py:68)
    optimizer_step (pytorch_lightning/plugins/precision/precision_plugin.py:119)
    optimizer_step (pytorch_lightning/strategies/strategy.py:234)
    optimizer_step (pytorch_lightning/strategies/ddp.py:280)
    step (pytorch_lightning/core/optimizer.py:169)
    optimizer_step (pytorch_lightning/core/module.py:1708)
    _call_lightning_module_hook (pytorch_lightning/trainer/trainer.py:1347)
    _optimizer_step (pytorch_lightning/loops/optimization/optimizer_loop.py:370)
    _run_optimization (pytorch_lightning/loops/optimization/optimizer_loop.py:249)
    advance (pytorch_lightning/loops/optimization/optimizer_loop.py:202)
    run (pytorch_lightning/loops/loop.py:199)
    advance (pytorch_lightning/loops/batch/training_batch_loop.py:88)
    run (pytorch_lightning/loops/loop.py:199)
    advance (pytorch_lightning/loops/epoch/training_epoch_loop.py:213)
    run (pytorch_lightning/loops/loop.py:199)
    advance (pytorch_lightning/loops/fit_loop.py:267)
    run (pytorch_lightning/loops/loop.py:199)
    _run_train (pytorch_lightning/trainer/trainer.py:1205)
    _run_stage (pytorch_lightning/trainer/trainer.py:1182)
    _run (pytorch_lightning/trainer/trainer.py:1103)
    _fit_impl (pytorch_lightning/trainer/trainer.py:650)
    launch (pytorch_lightning/strategies/launchers/subprocess_script.py:88)
    _call_and_handle_interrupt (pytorch_lightning/trainer/call.py:36)
    fit (pytorch_lightning/trainer/trainer.py:608)
    <module> (script.py:30)
    __libc_start_call_main (libc.so.6)
    __libc_start_main_impl (libc.so.6)

This is weird, because I can manually hit CTRL-C on the script, which will successfully raise a KeyboardInterrupt and exit out.
 But as soon as I register a Python signal handler, the process hangs again!
I'm not sure what to make of this behavior -- do you have any ideas what could explain what this?

The code from your https://github.com/Lightning-AI/lightning/pull/16307#issuecomment-1377772171 doesn't hang for me. It exits with sigkill for me (macos).

The code from your https://github.com/Lightning-AI/lightning/pull/16307#issuecomment-1377772171 doesn't hang for me. It exits with sigkill for me (macos).

Sorry to clarify:

The SIGKILL after the time.sleep indeed works correctly
But if you comment that out (or increase the sleep to something large, like 120 second)  then the SIGINT never seems to get delivered. SIGTERMs still get delivered, but never seem to actually reach a custom signal handler if you install it in Python (i.e. the process is torn down if you use the default signal handler, but hangs if you install you're own signal handler).

@alanhdu We removed the process reconciliation logic recently (#16204). We are looking into #16410 as an alternative instead, and your code modification could eventually be included there.


Use golint as linter

Hi
Great tool,BTW, Are you planning to use golint? is it possible to change the command?

Just out of curiosity, why you may prefer golangci lint over golint?

As golint is deprecated, there is no plan to support it.


Consider removing ElementParams, SurfaceParams, etc. in favor of a simple struct for each Object within the shader.

There are 3 ideas on how to implement this:

"shallow": we do not change anything about the VulkanTracer stuff etc.
But we add structs for each Element-type both in the shader and in the CPP files.
We then write a function in the shader to reinterpret the ElementParams, SurfaceParams etc. to reconstruct the structs from it.
This way we can use conventient structs in the shader without much rebuilding, but we earn technical debt to be cleared up later.

"2 buffers": we add one double-buffer containing all data of all elements in sequence (the first double of each element would be its type).
And another buffer to express where which element starts.
This is similar to materialBuffer and materialIndexBuffer.
The advantage is that we only need 2 buffers to do everything, which is simple in VulkanTracer and in the shader.
But we need to compile things down to a double-stream to afterwards reinterpret everything back to structs in the shader.
This can probably be done using some kind of reinterpret_cast-version, or with unions in GLSL. (I'm confident I can code this reinterpreting in 1 hour).
Disadvantage: We need new buffers if compared to "shallow".

"n-buffers": we add a buffer for each element-type.
The advantage is that we don't need any reconstructions, as we can simply put an RZP-struct into the shader, which can directly be interpreted as this same RZP-struct without any effort.
Disadvantage: indices don't simply work for elements anymore, because element 0 could be the first RZP or the first PlaneMirror etc.
Disadvantage: we need to add many buffers, and even more buffers when we add new elements.

I think we should either

implement "2 buffers"
do nothing, if you think we have more pressing issues

A potentially even simpler idea would be to do "2 buffers" but merge them into one buffer.
First double would be the length of the ElementIndexBuffer,
then comes the content of the ElementIndexBuffer.
and then comes the content of the ElementBuffer (i.e. all elements reinterpreted to doubles concatenated together)

@memoryleak47 will implement the 2. option from his comment. This could also make a rework of the objects possible, where each object comprises a geometry, a surface, and a struct/class for the element parameters. This would allow for better code reusability, modularity, and clarity.

done now using the Behaviour, Cutout and Surface systems!


Running headless tests with PhantomJS as a 'browser' fails

After setting up a test suite and trying to test with PhantomJS, i get the following error:
Error: Could not find React
  at [my own path, which i removed]:linenumber:0 <- webpack:///~/marty/marty.js:44:0

Looking into this file i see the following code:
if (typeof global === "undefined" && windowDefined) {
	  window.global = window;
	}

	__webpack_require__(148).polyfill();
	__webpack_require__(151);

	var Marty = __webpack_require__(153);
	var marty = new Marty('0.10.4', react());

	marty.use(__webpack_require__(270));
	marty.use(__webpack_require__(280));
	marty.use(__webpack_require__(282));
	marty.use(__webpack_require__(299));
	marty.use(__webpack_require__(315));
	marty.use(__webpack_require__(320));
	marty.use(__webpack_require__(323));
	marty.use(__webpack_require__(327));
	marty.use(__webpack_require__(329));
	marty.use(__webpack_require__(333));
	marty.use(__webpack_require__(341));
	marty.use(__webpack_require__(344));
	marty.use(__webpack_require__(346));
	marty.use(__webpack_require__(349));
	marty.use(__webpack_require__(351));

	module.exports = marty;

	function react() {
	  try {
	    return module.parent.require("react");
	  } catch (e) {
	    try {
	      return __webpack_require__(353);
	    } catch (e) {
	      if (windowDefined && window.React) {
	        return window.React;
	      }
	    }
	  }

	  throw new Error('Could not find React');
	}
/* WEBPACK VAR INJECTION */}.call(exports, (function() { return this; }()), __webpack_require__(147)(module)))

Which is the contents of this file: https://github.com/martyjs/marty/blob/master/marty.js
It looks to me as if this has something to do with the window object missing, since using 'Chrome' as a browser works fine.
Edit:
Looking at the code again it looks like the bottom part causes the error, which seems weird since react is resolved as a dependency through webpack.

I've been able to find a workaround for this by using the following
npm install --save-dev phantomjs-polyfill
require('phantomjs-polyfill')
var React = require('react');

This issue was fixed in PhantomJS 2, though? https://github.com/ariya/phantomjs/issues/10522
Anyway, good to see that you've found a solution.


rendering window is black on some machines

has been a major problem already in 1.0.8 on some machines - some have reported that uninstalling wacom tablet drivers has fixed it for them :o
anyhow, one cause was this has been hunted down and fixed in:
https://github.com/realXtend/naali/commit/4b90c28e9ab7c590f525237a946fd2e54cf992b6
to be verified.
putting here also to inform users that the issue is known, as many seem to encounter this and perhaps end up searching here for info.

https://github.com/LudoCraft/Tundra/commit/0aa24c4fa5952a2b4414c3b430946d9b08bb16e8


[corechecks/snmp] Protect against already processed OIDs

What does this PR do?
[corechecks/snmp] Protect against non increasing / already processed OIDs
Motivation
Non increasing or already processed OIDs when using getNext or getBulk is not expected but if it happens it might lead the integration in an infinite loop.
Non increasing OIDs can happen if the device misbehave or due to configuration error (e.g. two interface with same index number, already seen in a support case).
Additional Notes
Describe how to test your changes
It seems not possible to reproduce behaviour that will return already processed OIDs with snmpsim.
I would recommend QA/check that the unit tests are testing the correct behaviour.
Checklist

[ ] A release note has been added or the changelog/no-changelog label has been applied.
[ ] The need-change/operator and need-change/helm labels has been applied if applicable.
[ ] The appropriate team/.. label has been applied, if known.
[ ] If known, an appropriate milestone has been selected; otherwise the Triage milestone is set.
[ ] The config template has been updated if applicable.

Note: Adding GitHub labels is only possible for contributors with write access.

Closing in favour of https://github.com/DataDog/datadog-agent/pull/9717


bundle info prints gem metadata

What was the end-user problem that led to this PR?
Rubygems has supported gemspec metadata since version 2.0. I think it'd be great to display this information in response to the bundle info <gem> command.
What is your fix for the problem, implemented in this PR?
Added a check for metadata, outputting this extra information if it has been included with the gem release. For example:
> bundle info rspec-mocks
  * rspec-mocks (3.8.1)
	Summary: rspec-mocks-3.8.1
	Homepage: https://github.com/rspec/rspec-mocks
	Documentation: https://rspec.info/documentation/
	Source Code: https://github.com/rspec/rspec-mocks
	Changelog: https://github.com/rspec/rspec-mocks/blob/v3.8.1/Changelog.md
	Bug Tracker: https://github.com/rspec/rspec-mocks/issues
	Mailing List: https://groups.google.com/forum/#!forum/rspec
	Path: /opt/asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/rspec-mocks-3.8.1

For better readability, we could align the values. However, this would mean a change to the existing info property lines (Summary, Homepage, Path and Default). Which would potentially break existing scripts that are coupled to the precise output.

Let's do it!
@bundlerbot r+


Build Failing: BadZipfile: File is not a zip file

Details

Project URL: http://slac-ocio-dev-processes.readthedocs.io/en/latest/
Build URL (if applicable): https://readthedocs.org/projects/slac-ocio-dev-processes/builds/4421562/
Read the Docs username (if applicable): xalg

Expected Result
My newest revision to build.
Actual Result
Collecting sphinx==1.3.5
  Using cached Sphinx-1.3.5-py2.py3-none-any.whl
Collecting Pygments==2.1.3
  Using cached Pygments-2.1.3-py2.py3-none-any.whl
Collecting setuptools==20.1.1
  Using cached setuptools-20.1.1-py2.py3-none-any.whl
Collecting docutils==0.12
Collecting mkdocs==0.15.0
  Using cached mkdocs-0.15.0-py2.py3-none-any.whl
Collecting mock==1.0.1
Collecting pillow==2.6.1
Exception:
Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/slac-ocio-dev-processes/envs/latest/local/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/home/docs/checkouts/readthedocs.org/user_builds/slac-ocio-dev-processes/envs/latest/local/lib/python2.7/site-packages/pip/commands/install.py", line 310, in run
    wb.build(autobuilding=True)
  File "/home/docs/checkouts/readthedocs.org/user_builds/slac-ocio-dev-processes/envs/latest/local/lib/python2.7/site-packages/pip/wheel.py", line 750, in build
    self.requirement_set.prepare_files(self.finder)
  File "/home/docs/checkouts/readthedocs.org/user_builds/slac-ocio-dev-processes/envs/latest/local/lib/python2.7/site-packages/pip/req/req_set.py", line 370, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/home/docs/checkouts/readthedocs.org/user_builds/slac-ocio-dev-processes/envs/latest/local/lib/python2.7/site-packages/pip/req/req_set.py", line 587, in _prepare_file
    session=self.session, hashes=hashes)
  File "/home/docs/checkouts/readthedocs.org/user_builds/slac-ocio-dev-processes/envs/latest/local/lib/python2.7/site-packages/pip/download.py", line 798, in unpack_url
    unpack_file_url(link, location, download_dir, hashes=hashes)
  File "/home/docs/checkouts/readthedocs.org/user_builds/slac-ocio-dev-processes/envs/latest/local/lib/python2.7/site-packages/pip/download.py", line 705, in unpack_file_url
    unpack_file(from_path, location, content_type, link)
  File "/home/docs/checkouts/readthedocs.org/user_builds/slac-ocio-dev-processes/envs/latest/local/lib/python2.7/site-packages/pip/utils/__init__.py", line 599, in unpack_file
    flatten=not filename.endswith('.whl')
  File "/home/docs/checkouts/readthedocs.org/user_builds/slac-ocio-dev-processes/envs/latest/local/lib/python2.7/site-packages/pip/utils/__init__.py", line 484, in unzip_file
    zip = zipfile.ZipFile(zipfp, allowZip64=True)
  File "/usr/lib/python2.7/zipfile.py", line 770, in __init__
    self._RealGetContents()
  File "/usr/lib/python2.7/zipfile.py", line 811, in _RealGetContents
    raise BadZipfile, "File is not a zip file"
BadZipfile: File is not a zip file

Steps I have tried:
- Clicked the build button again
- "Wiped" the instance
- Disabled ePub and PDF
- Review yml syntax for errors with a linter
- cursing at it

Sorry this took a while, it looks like it has been fixed (as the latest builds are passing). Did you do anything in particular to get it to work, or did it just magically start working?

@ericholscher It appears to be broken intermittendly, I got it twice in the last few days but it's not always the case. I'm guessing there's a corrupt download somewhere on one of the machines.
Here's two broken builds for your viewing pleasure:

https://readthedocs.org/projects/progressbar-2/builds/5471290/
https://readthedocs.org/projects/progressbar-2/builds/5475772/

@xalgorithm @WoLpH your latest builds are passing or failing for other reasons. Feel free to reopen if you are still having this problem,

it seems the issue hasn't occurred recently so I assume it has been fixed


100 GB Swapfile on 256GB Micro-SD not enough?

Dusty,
Thank you, you've been great. We are creating a pipeline using jetson-inference, however, even with 100Gb of swap space on a 256GB Micro-SD card, we have only once successfully trained a cat_dog model using the /jetson-inference/python/training/classification/train.py file and directory. It either hangs, is killed, with the following code:
python3 train.py --epochs=1 --model-dir=models/cat_dog data/cat_dog
Attached is a screenshot of our Gparted Analysis and memory distribution.

Not everything can go into swap (like GPU memory), so there is typically an upper limit to what is needed/effective when it comes to swap.  I think the most I ever mounted was 16GB (on an AGX Xavier)
Which Jetson are you using?
Have you tried disabling the GUI like here:  https://github.com/dusty-nv/jetson-inference/blob/master/docs/pytorch-transfer-learning.md#disabling-the-desktop-gui
And also running train.py with --batch-size=1 --workers=0 like mentioned here:  https://github.com/dusty-nv/jetson-inference/blob/master/docs/pytorch-cat-dog.md#re-training-resnet-18-model

Dusty,Great to know. We have disabled the GUI and set workers=0. If we sudo init 3 in a script, how can we bypass the username, password prompt to go straight into training? Any ideas? We are setting up a desktop application to allow us to train in one-click, for non-developers. We are using the Nano for prototyping. We have an Xavier, but we have 8 production A.I. Apps on the Xavier, as we created an A.I. Marketplace-on-a-Chip. And unfortunately the 16GB of extra RAM is a bottleneck on the Xavier, which brought us to the Nano. But still having issues with memory even when we clear the cache, and follow your suggestions.  Can we trick the memory into thinking there is more?  ---- On Wed, 10 Nov 2021 20:22:27 -0600  @.***  wrote ----
Not everything can go into swap (like GPU memory), so there is typically an upper limit to what is needed/effective when it comes to swap.  I think the most I ever mounted was 16GB (on an AGX Xavier)
Which Jetson are you using?
Have you tried disabling the GUI like here:  https://github.com/dusty-nv/jetson-inference/blob/master/docs/pytorch-transfer-learning.md#disabling-the-desktop-gui
And also running train.py with --batch-size=1 --workers=0 like mentioned here:  https://github.com/dusty-nv/jetson-inference/blob/master/docs/pytorch-cat-dog.md#re-training-resnet-18-model
—You are receiving this because you authored the thread.Reply to this email directly, view it on GitHub, or unsubscribe.Triage notifications on the go with GitHub Mobile for iOS or Android.

how can we bypass the username, password prompt to go straight into training?

I'm not sure.  Usually I just SSH into the board at that point.

Ok. Thank you. We just may write a boot up script to run at boot. Here a link for others looking to run a script on the Jetsons at boot: https://askubuntu.com/questions/290099/how-to-run-a-script-during-boot-as-root. ---- On Thu, 11 Nov 2021 08:01:28 -0600  @.***  wrote ----
how can we bypass the username, password prompt to go straight into training?
I'm not sure.  Usually I just SSH into the board at that point.
—You are receiving this because you authored the thread.Reply to this email directly, view it on GitHub, or unsubscribe.Triage notifications on the go with GitHub Mobile for iOS or Android.

You can close this.  ---- On Thu, 11 Nov 2021 08:01:28 -0600  @.***  wrote ----
how can we bypass the username, password prompt to go straight into training?
I'm not sure.  Usually I just SSH into the board at that point.
—You are receiving this because you authored the thread.Reply to this email directly, view it on GitHub, or unsubscribe.Triage notifications on the go with GitHub Mobile for iOS or Android.


why do you use Keycloak

Hi, can u tell me why u use Keycloak instead of Spring security, which feature does it have but Spring security doesn't have it ?

Please answer those questions or I'll close:

are you talking about authorization or resource server?
how you did lent on this repo?
how long have you been running an authorization-server in production for?
why are you mentioning spring-security only and not any other OpenID authorization-server?
are you really comparing an incubating project with one at maturity for years?
do you know of any company using spring security authorization server in production?

Answer ALL the questions and, maybe, I'll reopen.
For now, I suspect you have no idea what you're talking about (the link you provided does not contain any single reference to OpenID nor even OAuth2)


RHBPMS-4482: IE11 - enable ComboBox for Variable and DataType in Data…

… I/O Editor to work with single click

jenkins please retest this

The fix works pretty good on IE11. I created also PR #520 that tests the changes. +1

jenkins please retest this


fix: datafactory character encoding

Automation for azure-sdk-for-python

Automation for azure-sdk-for-go

Automation for azure-sdk-for-java

Can one of the admins verify this patch?


Impersonation example for Linux

Hello!
would you mind to add an impersonation example for Linux?

Hey :)
If you're on Linux, you should be able to use any library that can alter packets on the fly (for example: NetFilterQueue).
I will attempt to implement it myself and update the examples accordingly.

Sorry for the late response, I haven't had much time lately to test this myself.
Thankfully, @momo286 added an example for linux impersonation in this PR.
Let me know if you have any questions!


iPlayer Setup

Hi there,
In your other iPlayer repo, you advise people to use the netflix-proxy repo instead. How is this set up to access iplayer? Is it a case of just adding bbc.co.uk to the zones.override file?
Or will the iplayer-proxy repo still give good results?
Thanks

Hi there, please use the netflix-proxy repository and add bbc.co.uk block to zones.override. Just copy the Netflix definition as change the domain name, lastly run "docker restart bind". That should be all..
Cheers!

On 24 Oct 2015, at 15:29, Daniel McKay<EMAIL_ADDRESS>wrote:
Hi there,
In your other iPlayer repo, you advise people to use the netflix-proxy repo instead. How is this set up to access iplayer? Is it a case of just adding bbc.co.uk to the zones.override file?
Or will the iplayer-proxy repo still give good results?
Thanks
—
Reply to this email directly or view it on GitHub.

Thanks for the reply. It isn't working for me though. I replaced the zones.override file in this repo with the one in the iplayer repo and ran the build script. I am using a D.O. droplet. When I go to bbc.co.uk in the browser, I am not redirected to the .com site, so it works up to that point. However, when I go to view iPlayer content I get the following message - "This content doesn't seem to be working"
Perhaps they have some way of detecting that the traffic is coming through D.O.

It is possible.. try with this directive in zones.override:
zone "." {
type master;
file "/data/db.override";
};
This will confirm if BBC are using additional geo blocking (possibly on the
CDN endpoints).
If that works, you'll need to work out what domains to proxy in addition to
bbc.co.uk.
-- ab1
On 25 October 2015 at 08:11, Daniel McKay<EMAIL_ADDRESS>wrote:

Thanks for the reply. It isn't working for me though. I replaced the
zones.override file in this repo with the one in the iplayer repo and ran
the build script. I am using a D.O. droplet. When I go to bbc.co.uk in
the browser, I am not redirected to the .com site, so it works up to that
point. However, when I go to view iPlayer content I get the following
message - "This content doesn't seem to be working"
Perhaps they have some way of detecting that the traffic is coming through
D.O.
—
Reply to this email directly or view it on GitHub
https://github.com/ab77/netflix-proxy/issues/39#issuecomment-150899936.

P.S. I've added the BBC domains to the master branch now, so iPlayer should be supported out of the box, unless BBC got crafty with their geo-blocking tech.
-- ab1

BBC iPlayer doesn't work :(
Seems like you can hit 'Play' but it then results in the errors above:
This content doesn't seem to be working
Any ideas?


⚠ ESPN has degraded performance

In d88f195, ESPN (https://espn.com) experienced degraded performance:

HTTP code: 200
Response time: 2277 ms

Resolved: ESPN performance has improved in fb72ff0.


Enhancement to sumstats page new feature

Following on from goci#828
As a user I want the downloadable table of published summary statistics to include ftp path by default. Currently this can be included by selecting the additional column using the check box drop down.

Implemented in latest UI version


WIP: Contract request date intersection

#2686

Coverage decreased (-0.02%) to 90.837% when pulling<PHONE_NUMBER>c3fe0a11e6dc3fcd89c3a6aaab79f8 on contract_request_date_intersection_#2686 into 1675c164cd104674a9554aaa639dd27a1bc28df4 on master.


Hass in Docker on Synology restarts by itself with a blank log file

Home Assistant release (hass --version):
0.45.1
Description of problem:
I'm running the official Hass Docker container on a Synology DS916+ together with an Aeon Labs Z-stick Gen 5.
I'm running the container using the following command:
sudo docker run -d --name=home-assistant -v /volume1/docker/homeassistant:/config -v /etc/localtime:/etc/localtime:ro --privileged -v /dev/ttyUSB-ZStick-5G:/dev/ttyUSB-ZStick-5G --net=host --restart=always homeassistant/home-assistant
The container is running without any issues for a couple of days. Then I notice that all my Z-wave devices suddenly has disappeared. The log file has been cleared and the first entries in the new file shows Hass booting up. It seems that something is triggering the Docker container to reset/restart but it fails to make a connection to the Z-stick again. Today it happened during the early morning hours when the family was sleeping, so it doesn't seem to be triggered by a user action.
To get the container running again, I have to remove the container and run the command above to create a new container.
I have also managed to crash Hass a few times by just using the UI. This will cause Hass to crash and restart, without the Z-stick working. Log file is then purged as well.
I will try to collect more information, but since there is no log, it's hard.

Could you execute dmesg | grep seg?

@lwis:
dmesg | grep seg                                                                                [1489503.840030] python[4144]: segfault at 14 ip 00007f1f1166f102 sp 00007f1f06f327f0 error 4 in libpython3.6m.so.1.0[7f1f1150c000+29a000]       [1662300.407533] python[7548]: segfault at 8 ip 00007f0f3283e553 sp 00007ffc120e4208 error 6 in libpython3.6m.so.1.0[7f0f32688000+29a000]

You may be interested in #7752

Alright, I merged the monkey patch. So if you launch the Home Assistant docker with HASS_MONKEYPATCH_ASYNCIO=1 as environment variable it will apply the monkey patch on Home Assistant 0.46.


Valset/Data commitment requests security for QGB

For Valsets, the orchestrator is currently checking the valset by nonce and then signs it. This works if we suppose that all valset requests are valid.
However, we can have a malicious party that generates invalid valsets (duplicates with different nonce, valsets for non-significant validator set changes etc.),  and the orchestrator will keep signing those.
Thus, we will need to add validity checks on valsets before the orchestrator signs them.
These need to be defined not to allow any malicious party to flood the network with incorrect valsets.
The same applies for data commitment requests which will be added here https://github.com/celestiaorg/celestia-app/issues/268.

@evan-forbes What do you think ?

However, we can have a malicious party that generates invalid valsets (duplicates with different nonce, valsets for non-significant validator set changes etc.), and the orchestrator will keep signing those.

I thought that the only way to set an invalid one is with 2/3's malicious voting power? How would one create one in the state with out 2/3's agreement?

We're currently setting valsets in end blockers via calling:
https://github.com/celestiaorg/celestia-app/blob/95f5208c44aee2ff496d0d6e0e15c592e286fd50/x/qgb/keeper/keeper_valset.go#L22
This function doesn't have any checks in it. And similar with Data commitments.
Thus, if we implement the client requests (maybe not for valsets, but to set data commitments),  or similar, under https://github.com/celestiaorg/celestia-app/tree/master/x/qgb/client/cli,  we wouldn't have any checks whether the committed state is correct or not.
Furthermore, I created this issue to discuss adding more checks on the keeper level,  if it  makes sense.

I think its ok to add checks later if we wanted to add cli functionality, but currently we have no plans to, and even if we did, it would only be for data availability commitments.
Since the SetValsetRequest function can only be called during EndBlocker and effects state, anyone attempting to call it inappropriately will result in a different app hash, so I think its safe to not add checks. This was likely the thinking behind not adding checks in the first place.
This is good to think about though!! We might also want to consider adding invariants for this and other things. https://docs.cosmos.network/v0.45/building-modules/invariants.html

Sounds good. Should we keep the issue open to remind us in the future ?

I think we can close this for now, as there are no direct actionable items. If we ever do variable data commitment windows, then we will only handle this then.


Update dtd in gwt.xml files

According to this Stackoverlfow question these are the new correct urls to the dtd's for the gwt xml files.

That's makes sense. IntelliJ says all files I changed have now CRLF. Should I really change back some files to LF? I think different line separators are not very beautiful

You should use whatever LibGDX uses everywhere.
It's the standard thing to do, and I'm fairly sure part of the formatting guide (I'll check that soon).
It's just an advantage for getting the pull accepted.

Ok, I will do it. It's a bit weird because on my PC the files got CRLF but if I download one of these files from the commit, it got only LF

When I have to change the line ends I usually run it through a program like Notepad++ (on Windows) that allows you to change it easily.

Now it's showing a lot of Java files that I'm fairly sure you didn't modify...

Should I change trunk to the version, in this case 2.8.0?


No accelerated graphics displays working in UTM V2.2.2 (beta) x64 VMs (Debian 10 or Ubuntu 20)

After installed UTM V2.2.2, x64 all accelerated graphics displays (virtio-ramfb, virtio-gpu-gl-pci and virtio-vga-gl) do not work on Debian 10 and Ubuntu 20. Virtio-ramfb and virtio-vga-gl displays just show a black screen; virtio-gpu-gl-pci starts to boot and then changes to UEFI Interactive Shell V2.2 and stops leaving "Shell>" cursor. Spice-QXL non-accelerated graphics display still works fine in both VM with UTM V2.2.2. Fyi.
Attached 1 example debug log (Debian10 XFCE x64, using virtio-ramfb driver and after prolong wait for evidence of proper boot from black screen).
Configuration

UTM Version: 2.2.2
OS Version: 14.7.1
Device Model: iPad Pro M1 256GB
Is it jailbroken (name jailbreak used)? No
How did you install UTM? Signulous

**Debug log**
-[CSConnection finishInit]:270
-[CSInput initWithSession:]:293
2021-09-01 12:45:23.885 UTM[16388:15447919] -[CSConnection finishInit]:270
2021-09-01 12:45:23.885 UTM[16388:15447919] -[CSInput initWithSession:]:293
-[CSSession initWithSession:]:294
2021-09-01 12:45:23.885 UTM[16388:15447919] -[CSSession initWithSession:]:294
Running:  -L /private/var/containers/Bundle/Application/43DBD0AC-880F-4B8D-90D5-351343C76392/UTM.app/qemu -S -qmp tcp:<IP_ADDRESS>:4000,server,nowait -nodefaults -vga none -spice unix=on,addr=/private/var/mobile/Containers/Data/Application/E769754C-629D-4A3B-A30D-31D8385DC231/tmp/848CD8DB-6A2D-4682-A800-E5881AA795C6.spice,disable-ticketing=on,image-compression=off,playback-compression=off,streaming-video=off,gl=on -device virtio-ramfb -smp cpus=1,sockets=1,cores=1,threads=1 -machine q35, -accel tcg,tb-size=256,split-wx=on -global PIIX4_PM.disable_s3=1 -global ICH9-LPC.disable_s3=1 -drive if=pflash,format=raw,file=/private/var/containers/Bundle/Application/43DBD0AC-880F-4B8D-90D5-351343C76392/UTM.app/qemu/edk2-x86_64-code.fd,readonly=on -boot menu=on -m 4096 -device intel-hda -device hda-duplex -name "Debian 10 Xfce" -usb -device usb-tablet,bus=usb-bus.0 -device usb-mouse,bus=usb-bus.0 -device usb-kbd,bus=usb-bus.0 -device ich9-usb-ehci1,id=usb-controller-0 -device ich9-usb-uhci1,masterbus=usb-controller-0.0,firstport=0,multifunction=on -device ich9-usb-uhci2,masterbus=usb-controller-0.0,firstport=2,multifunction=on -device ich9-usb-uhci3,masterbus=usb-controller-0.0,firstport=4,multifunction=on -chardev spicevmc,name=usbredir,id=usbredirchardev0 -device usb-redir,chardev=usbredirchardev0,id=usbredirdev0,bus=usb-controller-0.0 -chardev spicevmc,name=usbredir,id=usbredirchardev1 -device usb-redir,chardev=usbredirchardev1,id=usbredirdev1,bus=usb-controller-0.0 -chardev spicevmc,name=usbredir,id=usbredirchardev2 -device usb-redir,chardev=usbredirchardev2,id=usbredirdev2,bus=usb-controller-0.0 -device ide-hd,bus=ide.0,drive=drive0,bootindex=0 -drive "if=none,media=disk,id=drive0,file=/var/mobile/Containers/Data/Application/E769754C-629D-4A3B-A30D-31D8385DC231/Documents/Debian 10 Xfce.utm/Images/disk-0.qcow2,cache=writethrough" -device ide-cd,bus=ide.1,drive=drive1,bootindex=1 -drive if=none,media=cdrom,id=drive1 -device rtl8139,mac=06:60:42:2A:4E:7E,netdev=net0 -netdev user,id=net0,net=<IP_ADDRESS>/24,host=<IP_ADDRESS>,hostname=<IP_ADDRESS>,dhcpstart=<IP_ADDRESS>,dns=<IP_ADDRESS> -device virtio-serial -device virtserialport,chardev=vdagent,name=com.redhat.spice.0 -chardev spicevmc,id=vdagent,debug=0,name=vdagent -device virtserialport,chardev=charchannel1,id=channel1,name=org.spice-space.webdav.0 -chardev spiceport,name=org.spice-space.webdav.0,id=charchannel1 -uuid 848CD8DB-6A2D-4682-A800-E5881AA795C6 -rtc base=localtime
Loading qemu-x86_64-softmmu.framework/qemu-x86_64-softmmu
2021-09-01 12:45:23.905 UTM[16388:15447919] Running:  -L /private/var/containers/Bundle/Application/43DBD0AC-880F-4B8D-90D5-351343C76392/UTM.app/qemu -S -qmp tcp:<IP_ADDRESS>:4000,server,nowait -nodefaults -vga none -spice unix=on,addr=/private/var/mobile/Containers/Data/Application/E769754C-629D-4A3B-A30D-31D8385DC231/tmp/848CD8DB-6A2D-4682-A800-E5881AA795C6.spice,disable-ticketing=on,image-compression=off,playback-compression=off,streaming-video=off,gl=on -device virtio-ramfb -smp cpus=1,sockets=1,cores=1,threads=1 -machine q35, -accel tcg,tb-size=256,split-wx=on -global PIIX4_PM.disable_s3=1 -global ICH9-LPC.disable_s3=1 -drive if=pflash,format=raw,file=/private/var/containers/Bundle/Application/43DBD0AC-880F-4B8D-90D5-351343C76392/UTM.app/qemu/edk2-x86_64-code.fd,readonly=on -boot menu=on -m 4096 -device intel-hda -device hda-duplex -name "Debian 10 Xfce" -usb -device usb-tablet,bus=usb-bus.0 -device usb-mouse,bus=usb-bus.0 -device usb-kbd,bus=usb-bus.0 -device ich9-usb-ehci1,id=usb-controller-0 -device ich9-usb-uhci1,masterbus=usb-controller-0.0,firstport=0,multifunction=on -device ich9-usb-uhci2,masterbus=usb-controller-0.0,firstport=2,multifunction=on -device ich9-usb-uhci3,masterbus=usb-controller-0.0,firstport=4,multifunction=on -chardev spicevmc,name=usbredir,id=usbredirchardev0 -device usb-redir,chardev=usbredirchardev0,id=usbredirdev0,bus=usb-controller-0.0 -chardev spicevmc,name=usbredir,id=usbredirchardev1 -device usb-redir,chardev=usbredirchardev1,id=usbredirdev1,bus=usb-controller-0.0 -chardev spicevmc,name=usbredir,id=usbredirchardev2 -device usb-redir,chardev=usbredirchardev2,id=usbredirdev2,bus=usb-controller-0.0 -device ide-hd,bus=ide.0,drive=drive0,bootindex=0 -drive "if=none,media=disk,id=drive0,file=/var/mobile/Containers/Data/Application/E769754C-629D-4A3B-A30D-31D8385DC231/Documents/Debian 10 Xfce.utm/Images/disk-0.qcow2,cache=writethrough" -device ide-cd,bus=ide.1,drive=drive1,bootindex=1 -drive if=none,media=cdrom,id=drive1 -device rtl8139,mac=06:60:42:2A:4E:7E,netdev=net0 -netdev user,id=net0,net=<IP_ADDRESS>/24,host=<IP_ADDRESS>,hostname=<IP_ADDRESS>,dhcpstart=<IP_ADDRESS>,dns=<IP_ADDRESS> -device virtio-serial -device virtserialport,chardev=vdagent,name=com.redhat.spice.0 -chardev spicevmc,id=vdagent,debug=0,name=vdagent -device virtserialport,chardev=charchannel1,id=channel1,name=org.spice-space.webdav.0 -chardev spiceport,name=org.spice-space.webdav.0,id=charchannel1 -uuid 848CD8DB-6A2D-4682-A800-E5881AA795C6 -rtc base=localtime
2021-09-01 12:45:23.905 UTM[16388:15447919] Loading qemu-x86_64-softmmu.framework/qemu-x86_64-softmmu
Stream error Error Domain=NSPOSIXErrorDomain Code=61 "Connection refused" UserInfo={_kCFStreamErrorCodeKey=61, _kCFStreamErrorDomainKey=1}
QMP stream error seen: Error Domain=NSPOSIXErrorDomain Code=61 "Connection refused" UserInfo={_kCFStreamErrorCodeKey=61, _kCFStreamErrorDomainKey=1}
2021-09-01 12:45:23.915 UTM[16388:15447956] Stream error Error Domain=NSPOSIXErrorDomain Code=61 "Connection refused" UserInfo={_kCFStreamErrorCodeKey=61, _kCFStreamErrorDomainKey=1}
2021-09-01 12:45:23.915 UTM[16388:15447956] QMP stream error seen: Error Domain=NSPOSIXErrorDomain Code=61 "Connection refused" UserInfo={_kCFStreamErrorCodeKey=61, _kCFStreamErrorDomainKey=1}
QMP connection failed, retries left: 29
Stream error Error Domain=NSPOSIXErrorDomain Code=61 "Connection refused" UserInfo={_kCFStreamErrorCodeKey=61, _kCFStreamErrorDomainKey=1}
2021-09-01 12:45:23.915 UTM[16388:15447956] QMP connection failed, retries left: 29
2021-09-01 12:45:23.915 UTM[16388:15447956] Stream error Error Domain=NSPOSIXErrorDomain Code=61 "Connection refused" UserInfo={_kCFStreamErrorCodeKey=61, _kCFStreamErrorDomainKey=1}
QMP stream error seen: Error Domain=NSPOSIXErrorDomain Code=61 "Connection refused" UserInfo={_kCFStreamErrorCodeKey=61, _kCFStreamErrorDomainKey=1}
QMP connection failed, retries left: 28
2021-09-01 12:45:23.915 UTM[16388:15447956] QMP stream error seen: Error Domain=NSPOSIXErrorDomain Code=61 "Connection refused" UserInfo={_kCFStreamErrorCodeKey=61, _kCFStreamErrorDomainKey=1}
2021-09-01 12:45:23.915 UTM[16388:15447956] QMP connection failed, retries left: 28
qemu-system: warning: global PIIX4_PM.disable_s3=1 not used
Connected to stream 0x282998900
QMP connection successful! (readStream:1)
2021-09-01 12:45:24.997 UTM[16388:15447951] Connected to stream 0x282998900
2021-09-01 12:45:24.997 UTM[16388:15447951] QMP connection successful! (readStream:1)
Debug JSON recieved <- {
  QMP =     {
      capabilities =         (
          oob
      );
      version =         {
          package = "";
          qemu =             {
              major = 6;
              micro = 0;
              minor = 1;
          };
      };
  };
}
Connected to stream 0x282998e10
QMP connection successful! (readStream:0)
2021-09-01 12:45:24.998 UTM[16388:15447951] Debug JSON recieved <- {
  QMP =     {
      capabilities =         (
          oob
      );
      version =         {
          package = "";
          qemu =             {
              major = 6;
              micro = 0;
              minor = 1;
          };
      };
  };
}
2021-09-01 12:45:24.998 UTM[16388:15447951] Connected to stream 0x282998e10
Got QMP handshake: {
  QMP =     {
      capabilities =         (
          oob
      );
      version =         {
          package = "";
          qemu =             {
              major = 6;
              micro = 0;
              minor = 1;
          };
      };
  };
}
Debug JSON send -> {
  execute = "qmp_capabilities";
}
2021-09-01 12:45:24.998 UTM[16388:15447951] QMP connection successful! (readStream:0)
2021-09-01 12:45:24.998 UTM[16388:15447951] Got QMP handshake: {
  QMP =     {
      capabilities =         (
          oob
      );
      version =         {
          package = "";
          qemu =             {
              major = 6;
              micro = 0;
              minor = 1;
          };
      };
  };
}
2021-09-01 12:45:24.998 UTM[16388:15447951] Debug JSON send -> {
  execute = "qmp_capabilities";
}
Debug JSON recieved <- {
  return =     {
  };
}
2021-09-01 12:45:24.999 UTM[16388:15447954] Debug JSON recieved <- {
  return =     {
  };
}
qemuQmpDidConnect
2021-09-01 12:45:24.999 UTM[16388:15447951] qemuQmpDidConnect
cs_channel_new:163
2021-09-01 12:45:25.006 UTM[16388:15447951] cs_channel_new:163
Debug JSON recieved <- {
  data =     {
      client =         {
          family = unix;
          host = "";
          port = "";
      };
      server =         {
          family = unix;
          host = "";
          port = "";
      };
  };
  event = "SPICE_CONNECTED";
  timestamp =     {
      microseconds = 9465;
      seconds =<PHONE_NUMBER>;
  };
}
qemu-system: warning: GSpice: main-1:0: could not set sockopt TCP_NODELAY: Operation not supported on socket
2021-09-01 12:45:25.010 UTM[16388:15447954] Debug JSON recieved <- {
  data =     {
      client =         {
          family = unix;
          host = "";
          port = "";
      };
      server =         {
          family = unix;
          host = "";
          port = "";
      };
  };
  event = "SPICE_CONNECTED";
  timestamp =     {
      microseconds = 9465;
      seconds =<PHONE_NUMBER>;
  };
}
qemu-system: info: main channel: opened
Debug JSON send -> {
  execute = cont;
}
2021-09-01 12:45:25.028 UTM[16388:15447983] Debug JSON send -> {
  execute = cont;
}
Debug JSON recieved <- {
  data =     {
      client =         {
          "channel-id" = 0;
          "channel-type" = 1;
          "connection-id" = 16807;
          family = unix;
          host = "";
          port = "";
          tls = 0;
      };
      server =         {
          auth = none;
          family = unix;
          host = "";
          port = "";
      };
  };
  event = "SPICE_INITIALIZED";
  timestamp =     {
      microseconds = 28638;
      seconds =<PHONE_NUMBER>;
  };
}
2021-09-01 12:45:25.028 UTM[16388:15447954] Debug JSON recieved <- {
  data =     {
      client =         {
          "channel-id" = 0;
          "channel-type" = 1;
          "connection-id" = 16807;
          family = unix;
          host = "";
          port = "";
          tls = 0;
      };
      server =         {
          auth = none;
          family = unix;
          host = "";
          port = "";
      };
  };
  event = "SPICE_INITIALIZED";
  timestamp =     {
      microseconds = 28638;
      seconds =<PHONE_NUMBER>;
  };
}
Debug JSON recieved <- {
  event = RESUME;
  timestamp =     {
      microseconds = 29027;
      seconds =<PHONE_NUMBER>;
  };
}
Debug JSON recieved <- {
  return =     {
  };
}
2021-09-01 12:45:25.029 UTM[16388:15447951] Debug JSON recieved <- {
  event = RESUME;
  timestamp =     {
      microseconds = 29027;
      seconds =<PHONE_NUMBER>;
  };
}
2021-09-01 12:45:25.029 UTM[16388:15447951] Debug JSON recieved <- {
  return =     {
  };
}
qemuHasResumed
2021-09-01 12:45:25.029 UTM[16388:15447955] qemuHasResumed
resizing to (1194.000000, 834.000000)
cs_channel_new:172
2021-09-01 12:45:25.029 UTM[16388:15447919] resizing to (1194.000000, 834.000000)
2021-09-01 12:45:25.029 UTM[16388:15447983] cs_channel_new:172
Debug JSON recieved <- {
  data =     {
      client =         {
          family = unix;
          host = "";
          port = "";
      };
      server =         {
          family = unix;
          host = "";
          port = "";
      };
  };
  event = "SPICE_CONNECTED";
  timestamp =     {
      microseconds = 30585;
      seconds =<PHONE_NUMBER>;
  };
}
Debug JSON recieved <- {
  data =     {
      client =         {
          family = unix;
          host = "";
          port = "";
      };
      server =         {
          family = unix;
          host = "";
          port = "";
      };
  };
  event = "SPICE_CONNECTED";
  timestamp =     {
      microseconds = 30646;
      seconds =<PHONE_NUMBER>;
  };
}
2021-09-01 12:45:25.030 UTM[16388:15447951] Debug JSON recieved <- {
  data =     {
      client =         {
          family = unix;
          host = "";
          port = "";
      };
      server =         {
          family = unix;
          host = "";
          port = "";
      };
  };
  event = "SPICE_CONNECTED";
  timestamp =     {
      microseconds = 30585;
      seconds =<PHONE_NUMBER>;
  };
}
Debug JSON recieved <- {
  data =     {
      client =         {
          family = unix;
          host = "";
          port = "";
      };
      server =         {
          family = unix;
          host = "";
          port = "";
      };
  };
  event = "SPICE_CONNECTED";
  timestamp =     {
      microseconds = 30684;
      seconds =<PHONE_NUMBER>;
  };
}
2021-09-01 12:45:25.030 UTM[16388:15447951] Debug JSON recieved <- {
  data =     {
      client =         {
          family = unix;
          host = "";
          port = "";
      };
      server =         {
          family = unix;
          host = "";
          port = "";
      };
  };
  event = "SPICE_CONNECTED";
  timestamp =     {
      microseconds = 30646;
      seconds =<PHONE_NUMBER>;
  };
}
qemu-system: warning: GSpice: display-2:0: could not set sockopt TCP_NODELAY: Operation not supported on socket
2021-09-01 12:45:25.030 UTM[16388:15447951] Debug JSON recieved <- {
  data =     {
      client =         {
          family = unix;
          host = "";
          port = "";
      };
      server =         {
          family = unix;
          host = "";
          port = "";
      };
  };
  event = "SPICE_CONNECTED";
  timestamp =     {
      microseconds = 30684;
      seconds =<PHONE_NUMBER>;
  };
}
qemu-system: warning: GSpice: usbredir-9:2: could not set sockopt TCP_NODELAY: Operation not supported on socket
Debug JSON recieved <- {
  data =     {
      client =         {
          family = unix;
          host = "";
          port = "";
      };
      server =         {
          family = unix;
          host = "";
          port = "";
      };
  };
  event = "SPICE_CONNECTED";
  timestamp =     {
      microseconds = 30713;
      seconds =<PHONE_NUMBER>;
  };
}
Debug JSON recieved <- {
  data =     {
      client =         {
          family = unix;
          host = "";
          port = "";
      };
      server =         {
          family = unix;
          host = "";
          port = "";
      };
  };
  event = "SPICE_CONNECTED";
  timestamp =     {
      microseconds = 30747;
      seconds =<PHONE_NUMBER>;
  };
}
Debug JSON recieved <- {
  data =     {
      client =         {
          family = unix;
          host = "";
          port = "";
      };
      server =         {
          family = unix;
          host = "";
          port = "";
      };
  };
  event = "SPICE_CONNECTED";
  timestamp =     {
      microseconds = 30772;
      seconds =<PHONE_NUMBER>;
  };
}
qemu-system: warning: GSpice: usbredir-9:1: could not set sockopt TCP_NODELAY: Operation not supported on socket
qemu-system: warning: GSpice: usbredir-9:0: could not set sockopt TCP_NODELAY: Operation not supported on socket
qemu-system: warning: GSpice: playback-5:0: could not set sockopt TCP_NODELAY: Operation not supported on socket
2021-09-01 12:45:25.030 UTM[16388:15447951] Debug JSON recieved <- {
  data =     {
      client =         {
          family = unix;
          host = "";
          port = "";
      };
      server =         {
          family = unix;
          host = "";
          port = "";
      };
  };
  event = "SPICE_CONNECTED";
  timestamp =     {
      microseconds = 30713;
      seconds =<PHONE_NUMBER>;
  };
}
qemu-system: warning: GSpice: record-6:0: could not set sockopt TCP_NODELAY: Operation not supported on socket
qemu-system: warning: GSpice: webdav-11:0: could not set sockopt TCP_NODELAY: Operation not supported on socket
2021-09-01 12:45:25.030 UTM[16388:15447951] Debug JSON recieved <- {
  data =     {
      client =         {
          family = unix;
          host = "";
          port = "";
      };
      server =         {
          family = unix;
          host = "";
          port = "";
      };
  };
  event = "SPICE_CONNECTED";
  timestamp =     {
      microseconds = 30747;
      seconds =<PHONE_NUMBER>;
  };
}
qemu-system: warning: GSpice: inputs-3:0: could not set sockopt TCP_NODELAY: Operation not supported on socket
Debug JSON recieved <- {
  data =     {
      client =         {
          family = unix;
          host = "";
          port = "";
      };
      server =         {
          family = unix;
          host = "";
          port = "";
      };
  };
  event = "SPICE_CONNECTED";
  timestamp =     {
      microseconds = 30810;
      seconds =<PHONE_NUMBER>;
  };
}
Debug JSON recieved <- {
  data =     {
      client =         {
          family = unix;
          host = "";
          port = "";
      };
      server =         {
          family = unix;
          host = "";
          port = "";
      };
  };
  event = "SPICE_CONNECTED";
  timestamp =     {
      microseconds = 30846;
      seconds =<PHONE_NUMBER>;
  };
}
2021-09-01 12:45:25.031 UTM[16388:15447951] Debug JSON recieved <- {
  data =     {
      client =         {
          family = unix;
          host = "";
          port = "";
      };
      server =         {
          family = unix;
          host = "";
          port = "";
      };
  };
  event = "SPICE_CONNECTED";
  timestamp =     {
      microseconds = 30772;
      seconds =<PHONE_NUMBER>;
  };
}
2021-09-01 12:45:25.031 UTM[16388:15447951] Debug JSON recieved <- {
  data =     {
      client =         {
          family = unix;
          host = "";
          port = "";
      };
      server =         {
          family = unix;
          host = "";
          port = "";
      };
  };
  event = "SPICE_CONNECTED";
  timestamp =     {
      microseconds = 30810;
      seconds =<PHONE_NUMBER>;
  };
}
2021-09-01 12:45:25.031 UTM[16388:15447951] Debug JSON recieved <- {
  data =     {
      client =         {
          family = unix;
          host = "";
          port = "";
      };
      server =         {
          family = unix;
          host = "";
          port = "";
      };
  };
  event = "SPICE_CONNECTED";
  timestamp =     {
      microseconds = 30846;
      seconds =<PHONE_NUMBER>;
  };
}
Debug JSON recieved <- {
  data =     {
      client =         {
          "channel-id" = 1;
          "channel-type" = 9;
          "connection-id" = 16807;
          family = unix;
          host = "";
          port = "";
          tls = 0;
      };
      server =         {
          auth = none;
          family = unix;
          host = "";
          port = "";
      };
  };
  event = "SPICE_INITIALIZED";
  timestamp =     {
      microseconds = 103529;
      seconds =<PHONE_NUMBER>;
  };
}
2021-09-01 12:45:25.103 UTM[16388:15447954] Debug JSON recieved <- {
  data =     {
      client =         {
          "channel-id" = 1;
          "channel-type" = 9;
          "connection-id" = 16807;
          family = unix;
          host = "";
          port = "";
          tls = 0;
      };
      server =         {
          auth = none;
          family = unix;
          host = "";
          port = "";
      };
  };
  event = "SPICE_INITIALIZED";
  timestamp =     {
      microseconds = 103529;
      seconds =<PHONE_NUMBER>;
  };
}
Debug JSON recieved <- {
  data =     {
      client =         {
          "channel-id" = 0;
          "channel-type" = 9;
          "connection-id" = 16807;
          family = unix;
          host = "";
          port = "";
          tls = 0;
      };
      server =         {
          auth = none;
          family = unix;
          host = "";
          port = "";
      };
  };
  event = "SPICE_INITIALIZED";
  timestamp =     {
      microseconds = 104320;
      seconds =<PHONE_NUMBER>;
  };
}
2021-09-01 12:45:25.104 UTM[16388:15447955] Debug JSON recieved <- {
  data =     {
      client =         {
          "channel-id" = 0;
          "channel-type" = 9;
          "connection-id" = 16807;
          family = unix;
          host = "";
          port = "";
          tls = 0;
      };
      server =         {
          auth = none;
          family = unix;
          host = "";
          port = "";
      };
  };
  event = "SPICE_INITIALIZED";
  timestamp =     {
      microseconds = 104320;
      seconds =<PHONE_NUMBER>;
  };
}
qemu-system: warning: Spice: playback:0 (0x106230120): setsockopt failed, Operation not supported on socket
Debug JSON recieved <- {
  data =     {
      client =         {
          "channel-id" = 0;
          "channel-type" = 5;
          "connection-id" = 16807;
          family = unix;
          host = "";
          port = "";
          tls = 0;
      };
      server =         {
          auth = none;
          family = unix;
          host = "";
          port = "";
      };
  };
  event = "SPICE_INITIALIZED";
  timestamp =     {
      microseconds = 105035;
      seconds =<PHONE_NUMBER>;
  };
}
2021-09-01 12:45:25.105 UTM[16388:15447954] Debug JSON recieved <- {
  data =     {
      client =         {
          "channel-id" = 0;
          "channel-type" = 5;
          "connection-id" = 16807;
          family = unix;
          host = "";
          port = "";
          tls = 0;
      };
      server =         {
          auth = none;
          family = unix;
          host = "";
          port = "";
      };
  };
  event = "SPICE_INITIALIZED";
  timestamp =     {
      microseconds = 105035;
      seconds =<PHONE_NUMBER>;
  };
}
qemu-system: warning: Spice: record:0 (0x1062301d0): setsockopt failed, Operation not supported on socket
Debug JSON recieved <- {
  data =     {
      client =         {
          "channel-id" = 0;
          "channel-type" = 6;
          "connection-id" = 16807;
          family = unix;
          host = "";
          port = "";
          tls = 0;
      };
      server =         {
          auth = none;
          family = unix;
          host = "";
          port = "";
      };
  };
  event = "SPICE_INITIALIZED";
  timestamp =     {
      microseconds = 105778;
      seconds =<PHONE_NUMBER>;
  };
}
2021-09-01 12:45:25.105 UTM[16388:15447955] Debug JSON recieved <- {
  data =     {
      client =         {
          "channel-id" = 0;
          "channel-type" = 6;
          "connection-id" = 16807;
          family = unix;
          host = "";
          port = "";
          tls = 0;
      };
      server =         {
          auth = none;
          family = unix;
          host = "";
          port = "";
      };
  };
  event = "SPICE_INITIALIZED";
  timestamp =     {
      microseconds = 105778;
      seconds =<PHONE_NUMBER>;
  };
}
Debug JSON recieved <- {
  data =     {
      client =         {
          "channel-id" = 0;
          "channel-type" = 11;
          "connection-id" = 16807;
          family = unix;
          host = "";
          port = "";
          tls = 0;
      };
      server =         {
          auth = none;
          family = unix;
          host = "";
          port = "";
      };
  };
  event = "SPICE_INITIALIZED";
  timestamp =     {
      microseconds = 106553;
      seconds =<PHONE_NUMBER>;
  };
}
2021-09-01 12:45:25.106 UTM[16388:15447954] Debug JSON recieved <- {
  data =     {
      client =         {
          "channel-id" = 0;
          "channel-type" = 11;
          "connection-id" = 16807;
          family = unix;
          host = "";
          port = "";
          tls = 0;
      };
      server =         {
          auth = none;
          family = unix;
          host = "";
          port = "";
      };
  };
  event = "SPICE_INITIALIZED";
  timestamp =     {
      microseconds = 106553;
      seconds =<PHONE_NUMBER>;
  };
}
Debug JSON recieved <- {
  data =     {
      client =         {
          "channel-id" = 0;
          "channel-type" = 3;
          "connection-id" = 16807;
          family = unix;
          host = "";
          port = "";
          tls = 0;
      };
      server =         {
          auth = none;
          family = unix;
          host = "";
          port = "";
      };
  };
  event = "SPICE_INITIALIZED";
  timestamp =     {
      microseconds = 107275;
      seconds =<PHONE_NUMBER>;
  };
}
2021-09-01 12:45:25.107 UTM[16388:15447955] Debug JSON recieved <- {
  data =     {
      client =         {
          "channel-id" = 0;
          "channel-type" = 3;
          "connection-id" = 16807;
          family = unix;
          host = "";
          port = "";
          tls = 0;
      };
      server =         {
          auth = none;
          family = unix;
          host = "";
          port = "";
      };
  };
  event = "SPICE_INITIALIZED";
  timestamp =     {
      microseconds = 107275;
      seconds =<PHONE_NUMBER>;
  };
}
Debug JSON recieved <- {
  data =     {
      client =         {
          "channel-id" = 0;
          "channel-type" = 2;
          "connection-id" = 16807;
          family = unix;
          host = "";
          port = "";
          tls = 0;
      };
      server =         {
          auth = none;
          family = unix;
          host = "";
          port = "";
      };
  };
  event = "SPICE_INITIALIZED";
  timestamp =     {
      microseconds = 107978;
      seconds =<PHONE_NUMBER>;
  };
}
2021-09-01 12:45:25.108 UTM[16388:15447954] Debug JSON recieved <- {
  data =     {
      client =         {
          "channel-id" = 0;
          "channel-type" = 2;
          "connection-id" = 16807;
          family = unix;
          host = "";
          port = "";
          tls = 0;
      };
      server =         {
          auth = none;
          family = unix;
          host = "";
          port = "";
      };
  };
  event = "SPICE_INITIALIZED";
  timestamp =     {
      microseconds = 107978;
      seconds =<PHONE_NUMBER>;
  };
}
Debug JSON recieved <- {
  data =     {
      client =         {
          "channel-id" = 2;
          "channel-type" = 9;
          "connection-id" = 16807;
          family = unix;
          host = "";
          port = "";
          tls = 0;
      };
      server =         {
          auth = none;
          family = unix;
          host = "";
          port = "";
      };
  };
  event = "SPICE_INITIALIZED";
  timestamp =     {
      microseconds = 108717;
      seconds =<PHONE_NUMBER>;
  };
}
2021-09-01 12:45:25.108 UTM[16388:15447955] Debug JSON recieved <- {
  data =     {
      client =         {
          "channel-id" = 2;
          "channel-type" = 9;
          "connection-id" = 16807;
          family = unix;
          host = "";
          port = "";
          tls = 0;
      };
      server =         {
          auth = none;
          family = unix;
          host = "";
          port = "";
      };
  };
  event = "SPICE_INITIALIZED";
  timestamp =     {
      microseconds = 108717;
      seconds =<PHONE_NUMBER>;
  };
}
-[CSDisplayMetal initWithSession:channelID:monitorID:]:480
2021-09-01 12:45:25.109 UTM[16388:15447983] -[CSDisplayMetal initWithSession:channelID:monitorID:]:480
cs_channel_new:309
2021-09-01 12:45:25.109 UTM[16388:15447983] cs_channel_new:309
qemu-system: warning: GSpice: cursor-4:0: could not set sockopt TCP_NODELAY: Operation not supported on socket
Debug JSON recieved <- {
  data =     {
      client =         {
          family = unix;
          host = "";
          port = "";
      };
      server =         {
          family = unix;
          host = "";
          port = "";
      };
  };
  event = "SPICE_CONNECTED";
  timestamp =     {
      microseconds = 109290;
      seconds =<PHONE_NUMBER>;
  };
}
2021-09-01 12:45:25.109 UTM[16388:15447954] Debug JSON recieved <- {
  data =     {
      client =         {
          family = unix;
          host = "";
          port = "";
      };
      server =         {
          family = unix;
          host = "";
          port = "";
      };
  };
  event = "SPICE_CONNECTED";
  timestamp =     {
      microseconds = 109290;
      seconds =<PHONE_NUMBER>;
  };
}
Debug JSON recieved <- {
  data =     {
      client =         {
          "channel-id" = 0;
          "channel-type" = 4;
          "connection-id" = 16807;
          family = unix;
          host = "";
          port = "";
          tls = 0;
      };
      server =         {
          auth = none;
          family = unix;
          host = "";
          port = "";
      };
  };
  event = "SPICE_INITIALIZED";
  timestamp =     {
      microseconds = 112049;
      seconds =<PHONE_NUMBER>;
  };
}
2021-09-01 12:45:25.112 UTM[16388:15447955] Debug JSON recieved <- {
  data =     {
      client =         {
          "channel-id" = 0;
          "channel-type" = 4;
          "connection-id" = 16807;
          family = unix;
          host = "";
          port = "";
          tls = 0;
      };
      server =         {
          auth = none;
          family = unix;
          host = "";
          port = "";
      };
  };
  event = "SPICE_INITIALIZED";
  timestamp =     {
      microseconds = 112049;
      seconds =<PHONE_NUMBER>;
  };
}
Debug JSON recieved <- {
  data =     {
      offset = "-1";
  };
  event = "RTC_CHANGE";
  timestamp =     {
      microseconds = 409879;
      seconds =<PHONE_NUMBER>;
  };
}
2021-09-01 12:45:25.410 UTM[16388:15447955] Debug JSON recieved <- {
  data =     {
      offset = "-1";
  };
  event = "RTC_CHANGE";
  timestamp =     {
      microseconds = 409879;
      seconds =<PHONE_NUMBER>;
  };
}
seen UIPasteboardRemovedNotification
2021-09-01 12:45:26.031 UTM[16388:15447919] seen UIPasteboardRemovedNotification
Debug JSON recieved <- {
  data =     {
      offset = "-1";
  };
  event = "RTC_CHANGE";
  timestamp =     {
      microseconds = 508920;
      seconds =<PHONE_NUMBER>;
  };
}
2021-09-01 12:45:26.410 UTM[16388:15447951] Debug JSON recieved <- {
  data =     {
      offset = "-1";
  };
  event = "RTC_CHANGE";
  timestamp =     {
      microseconds = 508920;
      seconds =<PHONE_NUMBER>;
  };
}
Debug JSON send -> {
  execute = quit;
}
2021-09-01 12:46:19.834 UTM[16388:15447953] Debug JSON send -> {
  execute = quit;
}
Debug JSON recieved <- {
  return =     {
  };
}
2021-09-01 12:46:19.835 UTM[16388:15447955] Debug JSON recieved <- {
  return =     {
  };
}
Debug JSON recieved <- {
  data =     {
      guest = 0;
      reason = "host-qmp-quit";
  };
  event = SHUTDOWN;
  timestamp =     {
      microseconds = 834941;
      seconds =<PHONE_NUMBER>;
  };
}
2021-09-01 12:46:19.835 UTM[16388:15447955] Debug JSON recieved <- {
  data =     {
      guest = 0;
      reason = "host-qmp-quit";
  };
  event = SHUTDOWN;
  timestamp =     {
      microseconds = 834941;
      seconds =<PHONE_NUMBER>;
  };
}
qemuWillQuit, reason = host-qmp-quit
2021-09-01 12:46:19.835 UTM[16388:15447953] qemuWillQuit, reason = host-qmp-quit
cs_channel_destroy:209
cs_channel_destroy:382
2021-09-01 12:46:19.836 UTM[16388:15447983] cs_channel_destroy:209
2021-09-01 12:46:19.836 UTM[16388:15447983] cs_channel_destroy:382
cs_channel_destroy:201
2021-09-01 12:46:19.837 UTM[16388:15447983] cs_channel_destroy:201

@SugarRayLua Can you try installing debian/ubuntu from a new vm from scratch? I think the issue is that I switched from BIOS to EFI.

Sorry, Osy, it still does the same as I wrote on my previous post. I installed new VM from scratch yesterday (Debian 10, x86-64 XFCE) and used UTM V2.2.2 (beta) and none fully booted:
Virtio-ramfb and virtio-vga-gl still gave me a black screen after waiting 2 minutes for a boot.
Virtio-GPU-gl-pci did start to boot (to UEFI) but then exited after a half-screen of testing and then stopped and left me with a cursor wanting input (UEFI shell?)
When I then revert back to UTM 2.2.1, all the above display drivers work fine.
Please let me know if you’d like me to do any more tests.
Have a good upcoming week.

@SugarRayLua How exactly did you build your Debian 10? Can you upload your .utm somewhere?

Hi, @osy. I built it as follows:

Downloaded Debian 10 iso (disk 1) from official Debian site
Set up a new virtual machine from iPad UTM menu with the following features:
System: Standard PC x86_64
Drives: Added standard disk-0.qwo2 and made it 40GB
Added standard removable DVD
Display: Initially Spice QXL GPU but later after built if working changed to one of the virtio displays
Enabled fit to screen
Sound: Changed to Intel HD Audio Controller ich6 (intel-hda)
Sharing: Enabled Clipboard and directory sharing

(I left all of the other settings as default when setting up the Debian VM)
3. I added the Debian 10 iso I downloaded onto the UTM VM’s CD/DVD
4. I then booted up the virtual machine by pressing the arrow on the VM
5. I then followed the Debian installer GUI. Set up all the mundane things like time zones.
Allowed Debian installer to use mirror to add additional software during installation.
Only real options I believe I chose at the end besides default “Setup Graphic Desktop” was to:
Setup XFCE
Setup SSH
Setup WebServer
Setup Printer
6. I then rebooted into the VM and detached the Debian 10 iso.
Please let me know if I made a mistake in any of the above steps. Besides the graphics displays not
working in UTM V2.2.2 everything else works in the installation and works with the prior UTMs.
Unfortunately, now that my Debian 10 was working well, I loaded it up with many applications. Its now at 18GB in size so likely hard to easily upload. Many of the apps are for fun and not critical, and I could remove them to get to a smaller size to upload the UTM if important. Otherwise, I noticed on the github issue logs, you mentioned a “config.plist” file might be helpful. Helpful to send that but couldn’t find that easily in my VM (i.e. nothing came up when I tried sudo find / -iname config.plist). If the config.plist would be helpful, and you could let me know how to find it on my system, I’d be happy to also send that to you.
Fyi.

I’ve tested an install from Ubuntu 20.04 and Debian 11 and I was able to get into the installer with virtio-vga-gl. Do you observe the same?

No, @osy, I get different results:
Setup: iPad Pro M1 256GB, iOS 14.7.1, virtio-vga-gl
VMs tested: WorkingVM (Debian 10, x86-64, 3GB, XFCE)
NewVM (Debian 10 vs. Debian 11 vs. Ubuntu 20, x86-64, 3GB, XFCE)
UTMs tested: 2.2.2 vs. 2.2.1
Results:

Able to get into installer for new VM:
UTM 2.2.2
Debian 10: no
Debian 11: no
Ubuntu 20.04: yes, but in the middle of installation, UTM logo and Ubuntu name flip over;
although when finally reach desktop setup screen back to proper orientation
UTM 2.2.1
Debian 10: no
Debian 11: no
Ubuntu 20.04: yes, but similar strange screen inversion as noted in UTM 2.2.2 install above

Able to actually render working display of my Debian 10 x86-64 VM?

UTM 2.2.2: no
UTM 2.2.1: yes
Were you also testing using M1 iPad? I wonder if that is making a difference.
Hope that helps and let me know if you’d like me to do more testing.

Please test with v2.2.3. Make sure to read the notes about the changes requires in your configuration. It may make sense to create a new VM and install from scratch so the bootloader installs correctly for UEFI boot.

Thanks, @osy. Will do and let you know.

@osy, I ran the test with v2.2.3 as you requested and made another new VM to install for scratch. Bottom line: v2.2.3 now boots my previously set up Debian 10 VM fine (with or without EFI enabled in settings). However, the problems remain with new installations as follows:
Setup: iPad Pro M1 256GB, iOS 14.7.1, virtio-vga-gl
VMs tested: MyWorkingVM (Debian 10, x86-64, 3GB, XFCE; tested with EFI enabled and not)
NewVM (Debian 10 vs. Debian 11 vs. Ubuntu 20, x86-64, 3GB, XFCE)
UTMs tested: 2.2.3
Results:
Able to get into installer for new VM:
Debian 10: no
Debian 11: no
Ubuntu 20.04: yes, but as with UTM 2.2.2, in the middle of installation, UTM logo and Ubuntu name flip over; although when finally reach desktop setup screen back to proper orientation
UTM 2.2.1
Able to actually render working display of my previously working Debian 10 x86-64 VM?
Yes (and booted fine whether EFI enabled or not).
I hope that helps. Happy to test more if needed.

For Debian 11, you were not able to boot into the installer on a newly created VM?

No.

@SugarRayLua so there seems to be an error loading graphics during EFI mode. If you wait a bit while on the black screen and then press Enter, you should be able to enter setup. But I still need to figure out why graphics aren't working in EFI.

Yes, thanks, @osy, I confirm the same thing. If I wait a bit after getting the blank screen when trying to install a new Debian 11 VM and then press return, I do get a Debian 11 setup screen.

You can try https://github.com/utmapp/UTM/actions/runs/1216445661 but you have to build the IPA yourself by either using Xcode or the package.sh script.

Thanks!


⚠ Vote (prod) has degraded performance

In 3188e92, Vote (prod) (https://vote.ageg.ca/) experienced degraded performance:

HTTP code: 500
Response time: 19671 ms

Resolved: Vote (prod) performance has improved in 83f6fb4.


adding config option (genie.disable_adapter_timeout) to disable setting request timeout for adapter in Python client

Coverage remained the same at 88.73% when pulling ab9a32a01ba2699198f74142c70a1cdd05bc76e4 on irontable:develop into 9ef093b46a5cc6d28f03d7b7498375b963325a1f on Netflix:develop.

Coverage remained the same at 88.73% when pulling 7f11fb58b194ed52b3437291aafaf572e4cd7b96 on irontable:develop into 05fac74c49dae396aae8c82a672754a596b4ac8c on Netflix:develop.

Coverage remained the same at 88.73% when pulling 7f11fb58b194ed52b3437291aafaf572e4cd7b96 on irontable:develop into 05fac74c49dae396aae8c82a672754a596b4ac8c on Netflix:develop.


QA - Site - Product Detail Page - Product Qty field is occur multiple time if user try to multiple click on Product Color Swatches.

QA - Site - Product Detail Page - Product Qty field is occur multiple time if user try to multiple click on Product Color Swatches.

Still Issue is not resolved if product not have any product color.

resolved.

It is resolved and now product color field is mandatory while importing data to pdm.

Done


Data source integration

Added questions, objectives and key points to the data source integration tutorial

@gmauro am I correct that this is explaining data-sources to a user not a dev?


Add priority to tag

2 tests disabled right now- will fix later in subsequent PR

Codecov Report

Merging #45 (0fb15f3) into master (228c5b4) will decrease coverage by 2.32%.
The diff coverage is 51.66%.

@@             Coverage Diff              @@
##             master      #45      +/-   ##
============================================
- Coverage     70.13%   67.81%   -2.33%
+ Complexity      547      540       -7
============================================
  Files            89       91       +2
  Lines          1738     1802      +64
  Branches        198      204       +6
============================================
+ Hits           1219     1222       +3
- Misses          462      530      +68
+ Partials         57       50       -7

Impacted Files
Coverage Δ

.../java/seedu/address/logic/commands/AddCommand.java
92.30% <0.00%> (-7.70%)
:arrow_down:

...ain/java/seedu/address/logic/parser/CliSyntax.java
85.71% <ø> (ø)

...n/java/seedu/address/storage/CsvAdaptedPerson.java
72.52% <0.00%> (-12.09%)
:arrow_down:

src/main/java/seedu/address/ui/PersonCard.java
0.00% <0.00%> (ø)

.../java/seedu/address/model/util/SampleDataUtil.java
20.00% <25.00%> (ø)

...in/java/seedu/address/logic/parser/ParserUtil.java
82.25% <56.52%> (-15.31%)
:arrow_down:

src/main/java/seedu/address/model/tag/Tag.java
81.81% <66.66%> (+1.81%)
:arrow_up:

...main/java/seedu/address/storage/CsvAdaptedTag.java
60.00% <66.66%> (-16.48%)
:arrow_down:

...ain/java/seedu/address/storage/JsonAdaptedTag.java
61.11% <80.00%> (+2.28%)
:arrow_up:

...rc/main/java/seedu/address/model/tag/Priority.java
100.00% <100.00%> (ø)

... and 11 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 228c5b4...0fb15f3. Read the comment docs.

LGTM. Thanks for the added feature sam


Add table support for Excel (XLSX) sheets

Implemented functionality to read and work with named tables within Excel sheets. Updated .gitignore, added corresponding tests.
One note: I'm using a clone() in the build_table method, please advise on how to get rid of this. Also, please give me feedback on eager vs. lazy loading, I believe some refinement might be necessary there.
Related issue: https://github.com/ToucanToco/fastexcel/issues/204

Oh wow thanks a lot! I'll review this ASAP. In the meantime, could you please rebase the branch and run make format ? Looks like a lot of code has been re-indented to 8 spaces, and we use 4

Very cool feature, it would also be useful for me!

Thanks a lot for your work, looks great overall 🚀
Since it's a pretty big change, I'd like @PrettyWood to have a look as well, as I might have missed some parts (he'll be back from vacation next week).
Also, I wonder wether we should introduce a new type for tables: It'd allow us to store some extra metadata, and would allow us to add to_{arrow,polars,pandas} methods as well. As a nice side effect, it would also avoid the clone there currently is without needing changes on the calamine side
Anyway, this has motivated me again to polish the README and add some proper contributor docs 😅

@lukapeschke Yup I reckon we should differentiate Sheet and Table. I don' know if you want to work on that @wdoppenberg directly in this PR. It would be great but if you don't have the time we can merge this PR like that and work on a refacto after that on our end

Thanks a lot for your work, looks great overall 🚀
Since it's a pretty big change, I'd like @PrettyWood to have a look as well, as I might have missed some parts (he'll be back from vacation next week).
Also, I wonder wether we should introduce a new type for tables: It'd allow us to store some extra metadata, and would allow us to add to_{arrow,polars,pandas} methods as well. As a nice side effect, it would also avoid the clone there currently is without needing changes on the calamine side
Anyway, this has motivated me again to polish the README and add some proper contributor docs 😅

@lukapeschke Yup I reckon we should differentiate Sheet and Table. I don' know if you want to work on that @wdoppenberg directly in this PR. It would be great but if you don't have the time we can merge this PR like that and work on a refacto after that on our end

I don't have time for this this week. So depending on your timelines you might want to merge and open a new branch. If you can wait a week I can spend some time then.

@wdoppenberg Let's merge this as is then to avoid more merge conflicts, and I'll see what I can do regarding the new Table object :slightly_smiling_face: Could you please give me push rights on your fork ? I'll fix the conflicts and merge the PR afterwards

@wdoppenberg Let's merge this as is then to avoid more merge conflicts, and I'll see what I can do regarding the new Table object 🙂 Could you please give me push rights on your fork ? I'll fix the conflicts and merge the PR afterwards

Should work:


Fix CMake CPP11_FLAG not being added to compile flags when FMT_EXTRA_TESTS is OFF.

I might have misunderstood something, but it seems wrong that you have to enable the extra tests for C++11 support.

The idea here is to test compilation with both the default flags and -std=c++11. Otherwise the library is compiled with the default flags. However, this doesn't really matter as (1) all the code that relies on C++11 features is in the header (mostly inline variadic function templates) and (2) the library is distributed in the source code and you can specify your own options.
Since this can be a bit confusing, I've added a comment in https://github.com/cppformat/cppformat/commit/8ed603145722d1266b3a30a1c0b2d71f08722c39.
BTW your PR revealed fragility of the macro-test which I fixed in https://github.com/cppformat/cppformat/commit/538c2610cb829f4ad05153b0d6b8484797a5523f.
Thanks!

OK, I understand. It was indeed confusing, but the comment helps a bit, so thanks for that.
Normally I let my submodules only use their own flags so that they are set up properly for my system, with no modification from my application project. But this results into format.cc not being compiled with the -std=c++11 flag. But if all the C++11 code is inline, then this doesn't matter.


Implement ftruncate function (IDFGH-6641)

Is your feature request related to a problem? Please describe.
truncate function was requested in https://github.com/espressif/esp-idf/issues/1980 and implemented in https://github.com/espressif/esp-idf/commit/c77f84e788819a5070a9ea63b9a73fb0aea6df15, however the related ftruncate function was not implemented.
Describe the solution you'd like

Implement ftruncate function in VFS
Implement ftruncate in FAT VFS

Describe alternatives you've considered
Instead of having both truncate and ftruncate functions in VFS interface, VFS layer could supply a truncate implementation and only require ftruncate to be implemented by the VFS driver. The VFS layer could implement truncate using driver's open, ftruncate, close methods. This, however, would be a breaking change since we would have to replace truncate with ftruncate in the VFS interface.
Additional context
Requested in a comment to the closed issue: https://github.com/espressif/esp-idf/issues/1980#issuecomment-990092159
Issues that would be solved by implementing this:

https://github.com/joltwallet/esp_littlefs/issues/57
https://github.com/kuba--/zip/issues/231

@igrr any update on this?

FYI the Rust standard library relies on ftruncate being available, first reported here: https://github.com/esp-rs/rust/issues/100.


Queue metrics

Expose a QueueMetrics interface in Vert.x core metrics.

PoolMetrics owns queue metrics (the wait queue)
HttpClientMetrics also has a behavior of queue metrics
EventBus topics can also be seen as a queue metrics

It would be good to extract this common behavior in a QueueMetrics interface, so the metrics implementations can be implemented in a more consistent manner and it could be reused for other purposes like in the RedisConnection project (Expose metrics for the Redis client)

How would components get access to the Vert.x metrics object? Currently
it's only exposed by VertxInternal.
2016-10-12 15:20 GMT+02:00 Julien Viet<EMAIL_ADDRESS>
Expose a QueueMetrics interface in Vert.x core metrics.

PoolMetrics owns queue metrics (the wait queue)
HttpClientMetrics also has a behavior of queue metrics
EventBus topics can also be seen as a queue metrics

It would be good to extract this common behavior in a QueueMetrics
interface, so the metrics implementations can be implemented in a more
consistent manner and it could be reused for other purposes like in the
RedisConnection project (Expose metrics for the Redis client
https://github.com/vert-x3/issues/issues/182)
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/vert-x3/issues/issues/204, or mute the thread
https://github.com/notifications/unsubscribe-auth/ABblts3fNjsVWp4JNk3uObJ5lZk1e_A-ks5qzN6cgaJpZM4KUvtY
.

VertxInternal is fine for components, that's what jdbc-client does

OK thanks
2016-10-13 11:59 GMT+02:00 Julien Viet<EMAIL_ADDRESS>
VertxInternal is fine for components, that's what jdbc-client does
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/vert-x3/issues/issues/204#issuecomment-253469738, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABbltrUAMdeyw_k-5sUUorGxBiBIJIo7ks5qzgD3gaJpZM4KUvtY
.


Adding Material Design 3 MaterialSegmented

I needed MaterialSegmented to work with any object and not only strings. And because I saw that the Material Design 3 version was still to be implemented, I did that, and added the change in this new version.
Related "issue": #103
Changes compared to the Material Design 2 version:

Added new rendering as per Material Design 3 guidelines,
Added the SegmentType property allows user to select from old to new rendering (as suggested in the issue),
Added a MaterialSegmentedMulti control for multi selection,
Added selection icon as per design guidelines (also for SegmentType.Filled),
Added leading and trailing icons as requested in #58
Changed ItemsSource from IEnumerable<string> to IList to allow any model
Made the leading icon, trailing icon and text property paths from the model (tried to use BindingBase, but couldn't make it work)
Leading icon is replaced by selection icon when selected if the text is set (as per guidelines)

Let me know what you think!
See demo:
https://github.com/HorusSoftwareUY/MaterialDesignControlsPlugin/assets/51406965/c25d2785-13bc-4806-b339-44c494f697e3

We are very thankful that you have contributed with modifications to MaterialSegmented. It is a shame we were already working in parallel on the same control and improvements. Still, we will review your implementation to take it into account and include it as part of the control API, so we can offer the best possible experience! We will soon launch a new version of the NuGet package with all improvements.
Moreover, we are already planning to start the next migration stage towards Material V3 (https://github.com/orgs/HorusSoftwareUY/projects/4) and all contributions are welcome! We would be delighted if you could collaborate by using your skills in any ticket from this project!
Please, feel free to write me to<EMAIL_ADDRESS>if you have any doubts and also we can stay in touch there.


Change default retention time to 1 day

Temporary change default retention time

The reason why it is like that - I can not change it in other way yet

👍

:+1:

:+1:

👍


Conditional skip support

Local build passes with these changes :)

Coverage increased (+0.04%) to 89.439% when pulling e85d395ce74a159eee6a53f9eed4b22ef896c940 on aldaris:conditional-skip into 24f7c0acd8ec27043066cc8c615fa31c159bb6a5 on cuppa-framework:master.

:shipit:

Why does Cuppa need this feature?

The idea behind this setting was to allow writing tests that run conditionally, for example when the browser is configured to Firefox/Chrome. The upside of doing this rather than the if block is that these conditions can be reused in the long term (via method references for example).

Never really got to the point where this really became that necessary.


Error while threading

I have the following issue:
I am creating a thread that is dedicated to record and post process my audio. In my main program I am streaming the real time audio according to the wire.py example. I already tried both functions separately and they work but everytime that I want to multithread them I got the following error:
StdErr: Expression 'ret' failed in 'src/hostapi/alsa/pa_linux_alsa.c', line: 1736
Expression 'AlsaOpen( &alsaApi->baseHostApiRep, params, streamDir, &self->pcm )' failed in 'src/hostapi/alsa/pa_linux_alsa.c', line: 1904
Expression 'PaAlsaStreamComponent_Initialize( &self->capture, alsaApi, inParams, StreamDirection_In, NULL != callback )' failed in 'src/hostapi/alsa/pa_linux_alsa.c', line: 2171
Expression 'PaAlsaStream_Initialize( stream, alsaHostApi, inputParameters, outputParameters, sampleRate, framesPerBuffer, callback, streamFlags, userData )' failed in 'src/hostapi/alsa/pa_linux_alsa.c', line: 2839
Exception in thread hilo_1:
Traceback (most recent call last):
  File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.9/threading.py", line 892, in run
    self._target(*self._args, **self._kwargs)
  File "/home/user/Prueba/multihilo.py", line 163, in grabacion_filtrada
    myrecording = sd.rec(int(seconds * fs), samplerate=fs, channels=1, blocksize= True)
  File "/home/user/.local/lib/python3.9/site-packages/sounddevice.py", line 274, in rec
    ctx.start_stream(InputStream, samplerate, ctx.input_channels,
  File "/home/user/.local/lib/python3.9/site-packages/sounddevice.py", line 2572, in start_stream
    self.stream = StreamClass(samplerate=samplerate,
  File "/home/user/.local/lib/python3.9/site-packages/sounddevice.py", line 1415, in __init__
    _StreamBase.__init__(self, kind='input', wrap_callback='array',
  File "/home/user/.local/lib/python3.9/site-packages/sounddevice.py", line 892, in __init__
    _check(_lib.Pa_OpenStream(self._ptr, iparameters, oparameters,
  File "/home/user/.local/lib/python3.9/site-packages/sounddevice.py", line 2737, in _check
    raise PortAudioError(errormsg, err)
sounddevice.PortAudioError: Error opening InputStream: Device unavailable [PaErrorCode -9985]

I am using Ubuntu 21.04. The device that I want to select from the sd.query_devices() is this:
   8 USB PnP Sound Device: Audio (hw:3,0), ALSA (1 in, 2 out)
and I am selecting it with this code line:
sd.default.device = 8,8
Thank you for your help

It is difficult to understand what is going wrong without the actual code. Please post a minimal example that showcases the behavior you're struggling with.

Thanks for the replay
My main code is the following:
if __name__ == '__main__':
    try:
        #THREAD
        t1=threading.Thread(name="hilo_1",target=grabacion_filtrada)
        t1.start()

        while True:
            with sd.Stream(device=(args.input_device, args.output_device),
                        samplerate=args.samplerate, blocksize=args.blocksize,
                        dtype=args.dtype, latency=args.latency,
                        channels=args.channels, callback=callback):
                #print('#' * 80)
                #print('press Return to quit')
                #print('#' * 80)
                input()

    # Reset by pressing CTRL + C
    except KeyboardInterrupt:
        print("Measurement stopped by User")

And the function that I am calling in my thread is the next one:
def grabacion_filtrada():
    while True:
        modo=1
        filename='Prueba'

        print ("recording...")
        myrecording = sd.rec(int(seconds * fs), samplerate=fs, channels=1)
        sd.wait()  # Wait until recording is finished
        print ("finished recording")

        print ("filtering...")
        raw_data = myrecording.flatten() #simple string
        kalman_filt=Kalman(raw_data,varMeas,varProcess)
        kalman_filt = np.asarray(kalman_filt)
        write(filename[0:len(filename)]+'.wav', fs,kalman_filt)

        print ("end filtering")

I get the error that I post every time my function called ¨grabacion_filtrada()¨ is trying to record. I hope this pieces of code make it more understandable. Thank you for your help.

I don't think I understand what you are trying to do or how often these loops are iterated for example.
Isn't maybe the problem that sd.Stream(... and sd.rec(... try to access the same interface simultaneously? I'm not sure right now weather that is supposed to be possible. But anyways, doing that seems like it should simply be implemented differently?
Is the objective that the input data is passed straight to output as well as simultaneously captured (+ processed + saved)? If so, then we can think of a more appropriate "signal flow" to be implemented.

Isn't maybe the problem that sd.Stream(... and sd.rec(... try to access the same interface simultaneously? I'm not sure right now weather that is supposed to be possible. But anyways, doing that seems like it should simply be implemented differently?

I was able to access to it simultaneously in windows, but when I tried it in Ubuntu it failed and my implementation will be on Ubuntu that's why I'm struggling with it.

Is the objective that the input data is passed straight to output as well as simultaneously captured (+ processed + saved)? If so, then we can think of a more appropriate "signal flow" to be implemented.

This actually what I am trying to do

I was able to access to it simultaneously in windows, but when I tried it in Ubuntu it failed and my implementation will be on Ubuntu that's why I'm struggling with it.

Interesting.

Is the objective that the input data is passed straight to output as well as simultaneously captured (+ processed + saved)? If so, then we can think of a more appropriate "signal flow" to be implemented.

This actually what I am trying to do

Then before suggesting a potentially better solution to implement this ...
The processing is efficient enough to be performed in real-time (on a block-by-block) basis?
Is the execution in the scope of a few seconds or rather of arbitrarily long length?

So I tried my processing in real time and I think that it is efficient enough to be performed. In the other hand, the execution won't have a defined duration.


[CDN] Implement hooks for DDoS protection

Reviews
The only file that actually needs review is message_hook.rs. Everything else is either removing unused code or moving it around.
This PR:
Allows for "hooking" incoming messages from both brokers and users to implement custom logic. This will be the main source of DDoS protection measures.

Breaks out push_cdn_network.rs into more relevant, smaller files
Removes some unused/confusing generator arguments

The current hooking functionality:

Deduplicates messages to the same [topic or sender] (double votes, etc)

Uses an SMA algorithm. If the connection's average bytes per second (separate for direct/broadcast) is larger than the global average bytes per second (within a sliding window of 1,000), the user is disconnected.
Testing
This PR has unit tests for expected functionality, and has been ran on the local demo to make sure it doesn't cause any issues.

Drafting while I fix up


browser.checkElement function captures someother image instead of element image

I am new to WDIO, just exploring the features for WDIO - visual testing. I tried to do visual comparison of an element ["..pluginRowBlock"] on WDIO page.
The baseline image of the element which is stored under .tmp folder is something else not the screenshot of the .pluginRowBlock [expected: image contains getstarted ,watch talk etc] actual image in tmp folder is doc,api,help that whole section.
Sorry i am not able upload the screenshot:(
here is my code
describe('Visual test',()=>{
    it('Verify visual of webdriver IO page',()=>{
    // Check a screen
    browser.url('https://webdriver.io');
    //expect(browser.checkScreen('examplePaged')).toEqual(0);
    const title = browser.getTitle();
    assert.strictEqual(title, 'WebdriverIO · Next-gen browser automation test framework for Node.js');
    //assert.equal(browser.checkScreen('examplePaged'),0,"Home page is not matching with baseline");

    assert.equal(browser.checkElement($(".pluginRowBlock"),'rowblock'),0,"row block is not matching with baseline");
    });
});

Hi @NirmalAyyappan
thanks for filing an issue, I've tried it and it just works. Can you provide the images by just dragging them into the comment field on GitHub when you add a new comment to this issue?

Hi,
thanks for the response.
I have attached actual image which taken for baseline.

But expected image for the object is

Hi @NirmalAyyappan
Can you provide the setup and screenresolution? It looks like your screen is smaller and needs to scroll to the element before taking the screenshot.

I have followed setup document given in webdriverio and the command used to download dependencies "npm install --save-dev wdio-image-comparison-service".
OS: Windows 10
Screen resolution: 1920*1080.
Thanks for the timely response.
I will check in the large screen or by changing the screen resolution.

Hi @NirmalAyyappan
Did you made any progress here?

Nope, i couldn't check as i don't have larger screen monitor with me.
I will check in different screen resolution and reopen this issue if i face the same.
thank you so much of your support.


The scheduled tasks have not run for more than 10 minutes

Question
I wanted to ask what could be wrong with the planned tasks.
I've searched everything and checked the CronJobs several times.
But don't find the error. Could it be that you have made a mistake here?
How can I proceed to find out where it hangs?

Check the tab "Scheduled Tasks Manager" that every status is at scheduled and the next executation time should be in future. otherwise it might just hang around :-)

@csaeum mostly this happens when a big product export or smth else blocks your queue. Afaik RabbitMQ is like most/all MQ a first in first out.
Product Export Message is pushed to the queue at noon but beforehand there was an export with 1000 messages this can easily block your execution for + 10 minutes.
It would be nice to configure this in the plugin config by the way :smile:
Best way to handle this longterm is to have 2 or more queues. One for high priority messages like scheduled tasks and one for example for indexing etc.

Thank you for your help!!
I have 47 products, each with different variants, tw 10 tw 60.
The preview of the GS template generated in about 3 seconds and then has about 2800 lines which should be about 1250 articles.
It could just be these exports!
So the tasks planned according to the page all run smoothly:
I'm just at a loss to find where it hacks.

@csaeum indeed the screenshot looks fine to me. Nothing is queued and all are scheduled and in the future :man_shrugging:

Is there any open issue, or can we close?


BrowserType is not set when initializing DesktopWebDriver with selenium driver instance

when initializing DesktopWebDriver need dynamically determine BrowserType by type of retrieved selenium driver instance

done in 73914f4


First third-party adoption: Amasty Label

See branch amasty-label.
It needs some design:

You need to setup the module for magento first

@vileyne could you check if it works ok with new tailwind.yaml?

@vileyne could you check if it works ok with new tailwind.yaml?

here https://github.com/lightna-project/lightna/commit/b25f303cd608836086ebd8485cc87ed40c0988dd


Use project title as filename, if available

Just request https://scratch.mit.edu/api/v1/project/<ID>/?format=json and use the title item if you get a valid response.

That would work unless if it was a 404, which means you can't get the .sb2 anyway.