[BUG] Extension hook executes on custom module

Cockpit crashes with non descriptive error.
Description
I am trying to run cockpit on a simple network trained with MSE loss (although implemented through custom modules, not nn.Sequential). As far as I can see, no unsupported operations are involved (Linear, Sequential, Tanh, Identity).
Cockpit crashes on computing the BatchGrad extension. Using backpack to compute the batch gradient works without crashes.
Steps to Reproduce

Setup

self.loss_fn = lambda pred, target: ((target - pred) ** 2).mean(dim=-1)

self._cockpit = Cockpit(model.parameters(),
                                    quantities=configuration("economy"))
model = MyCustomModel()
model = extend(model)
optimizer = torch.optim.Adam(params=model.parameters(), lr=1e-3)

Training Step

pred = model(inp)
loss = self.loss_fn(pred, target)
mean_loss = loss.mean()

optimizer.zero_grad()
info = {
    "batch_size": len(loss),
    "individual_losses": loss,
    "loss": mean_loss,
    "optimizer": optimizer,
}
with self._cockpit(step, info=info, debug=True):
    create_graph = self._cockpit.create_graph(step)
    mean_loss.backward(create_graph=create_graph)  # CRASH HERE
optimizer.step()

Source or Possible Fix

Stacktrace
[DEBUG, step 0]
 ↪Quantities  : [<cockpit.quantities.alpha.Alpha object at 0x7fa751af5f10>, <cockpit.quantities.distance.Distance object at 0x7fa74e7c0cd0>, <cockpit.quantities.grad_hist.GradHist1d object at 0x7fa74e7c0d10>, <cockpit.quantities.grad_norm.GradNorm object at 0x7fa74e7c0d90>, <cockpit.quantities.inner_test.InnerTest object at 0x7fa74e7c0dd0>, <cockpit.quantities.loss.Loss object at 0x7fa74e7cd050>, <cockpit.quantities.norm_test.NormTest object at 0x7fa74e7cd090>, <cockpit.quantities.ortho_test.OrthoTest object at 0x7fa74e7cd0d0>, <cockpit.quantities.update_size.UpdateSize object at 0x7fa74e7cd110>]
 ↪Extensions  : [<backpack.extensions.firstorder.batch_grad.BatchGrad object at 0x7fa74e7cd750>]
 ↪Hooks       : <cockpit.quantities.utils_transforms.BatchGradTransformsHook object at 0x7fa74e7931d0>
 ↪Create graph: False
 ↪Save memory : True
[DEBUG] Running extension <backpack.extensions.firstorder.batch_grad.BatchGrad object at 0x7fa74e7cd750> on Linear(in_features=128, out_features=1, bias=True)
[DEBUG] Running extension <backpack.extensions.firstorder.batch_grad.BatchGrad object at 0x7fa74e7cd750> on LinearHead(
  (l): Linear(in_features=128, out_features=1, bias=True)
)
[DEBUG] Running extension <backpack.extensions.firstorder.batch_grad.BatchGrad object at 0x7fa74e7cd750> on Identity()
[DEBUG] Running extension <backpack.extensions.firstorder.batch_grad.BatchGrad object at 0x7fa74e7cd750> on Sequential(
  (0): Linear(in_features=1, out_features=128, bias=True)
  (1): Tanh()
  (2): Linear(in_features=128, out_features=128, bias=True)
  (3): Tanh()
  (4): LinearHead(
    (l): Linear(in_features=128, out_features=1, bias=True)
  )
  (5): Identity()
)
[DEBUG] Running extension <backpack.extensions.firstorder.batch_grad.BatchGrad object at 0x7fa74e7cd750> on MLP(
  (layers): Sequential(
    (0): Linear(in_features=1, out_features=128, bias=True)
    (1): Tanh()
    (2): Linear(in_features=128, out_features=128, bias=True)
    (3): Tanh()
    (4): LinearHead(
      (l): Linear(in_features=128, out_features=1, bias=True)
    )
    (5): Identity()
  )
)
Traceback (most recent call last):
  File "./envs/pytorch/lib/python3.7/site-packages/backpack/__init__.py", line 169, in run_extension_hook
    CTX.get_extension_hook()(module)
  File "./envs/pytorch/lib/python3.7/site-packages/cockpit/quantities/hooks/base.py", line 53, in __call__
    self.run_hook(param, module)
  File "./envs/pytorch/lib/python3.7/site-packages/cockpit/quantities/hooks/base.py", line 80, in run_hook
    value = self.module_hook(param, module)
  File "./envs/pytorch/lib/python3.7/site-packages/cockpit/quantities/hooks/base.py", line 139, in module_hook
    return self.param_hook(param)
  File "./envs/pytorch/lib/python3.7/site-packages/cockpit/quantities/utils_transforms.py", line 78, in param_hook
    param.grad_batch._param_weakref = weakref.ref(param)
AttributeError: 'Parameter' object has no attribute 'grad_batch'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./code/trainers/base_trainer.py", line 112, in backward
    mean_loss.backward(create_graph=create_graph)
  File "./envs/pytorch/lib/python3.7/site-packages/torch/tensor.py", line 221, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "./envs/pytorch/lib/python3.7/site-packages/torch/autograd/__init__.py", line 132, in backward
    allow_unreachable=True)  # allow_unreachable flag
  File "./envs/pytorch/lib/python3.7/site-packages/backpack/__init__.py", line 151, in hook_run_extensions
    run_extension_hook(module)
  File "./envs/pytorch/lib/python3.7/site-packages/backpack/__init__.py", line 172, in run_extension_hook
    raise RuntimeError(f"Post extensions hook failed: {message}")
RuntimeError: Post extensions hook failed: AttributeError("'Parameter' object has no attribute 'grad_batch'")

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "code/train.py", line 215, in <module>
    metrics = main(opts)
  File "code/train.py", line 185, in main
    training_info = trainer.train(model, dataset, eval_data, logger)
  File "./code/trainers/mse_trainer.py", line 128, in train
    self.backward(global_step, loss, mean_loss, optimizer)
  File "./code/trainers/base_trainer.py", line 112, in backward
    mean_loss.backward(create_graph=create_graph)
  File "./envs/pytorch/lib/python3.7/site-packages/cockpit/context.py", line 137, in __exit__
    self.cp.track(self.global_step, protected_savefields=self.protected_savefields)
  File "./envs/pytorch/lib/python3.7/site-packages/cockpit/cockpit.py", line 178, in track
    q.track(global_step, self.params, batch_loss)
  File "./envs/pytorch/lib/python3.7/site-packages/cockpit/quantities/quantity.py", line 87, in track
    iteration, result = self.compute(global_step, params, batch_loss)
  File "./envs/pytorch/lib/python3.7/site-packages/cockpit/quantities/quantity.py", line 516, in compute
    save_result = self._compute(global_step, params, batch_loss)
  File "./envs/pytorch/lib/python3.7/site-packages/cockpit/quantities/quantity.py", line 538, in _compute
    self._compute_start(global_step, params, batch_loss)
  File "./envs/pytorch/lib/python3.7/site-packages/cockpit/quantities/alpha.py", line 280, in _compute_start
    self._save_1st_order_info(global_step, params, batch_loss, point, until)
  File "./envs/pytorch/lib/python3.7/site-packages/cockpit/quantities/alpha.py", line 326, in _save_1st_order_info
    grad_dict = {id(p): p.grad.data.clone().detach() for p in params}
  File "./envs/pytorch/lib/python3.7/site-packages/cockpit/quantities/alpha.py", line 326, in <dictcomp>
    grad_dict = {id(p): p.grad.data.clone().detach() for p in params}
AttributeError: 'NoneType' object has no attribute 'data'

Hey Max,
thanks for taking the time and reporting such issues here. This helps us a lot to improve the library. I think you've caught the same bug as #5. TL;DR: The problem is caused by using Alpha with a non-SGD optimizer. Two hotfixes:

Exclude Alpha, keep Adam
Keep Alpha, swap Adam for SGD

Moving up the #5  on my priority list :)

I see. I would suggest to remove alpha from the default configurations or to document this clearly, as this was surprising behavior to me; most people would probably not use plain SGD to train their networks.
Anyways. I removed alpha:
config = [
              quantities.Distance(schedules.linear(interval=1)),
              quantities.GradHist1d(schedules.linear(interval=1)),
              quantities.GradNorm(schedules.linear(interval=1)),
              quantities.InnerTest(schedules.linear(interval=1)),
              quantities.Loss(schedules.linear(interval=1)),
              quantities.NormTest(schedules.linear(interval=1)),
              quantities.OrthoTest(schedules.linear(interval=1)),
              quantities.UpdateSize(schedules.linear(interval=1))
 ]
self._cockpit = Cockpit(model.parameters(), quantities=config)

This did not do the trick:
[DEBUG, step 0]
 ↪Quantities  : [<cockpit.quantities.distance.Distance object at 0x7fe44176ae90>, <cockpit.quantities.grad_hist.GradHist1d object at 0x7fe43e2bffd0>, <cockpit.quantities.grad_norm.GradNorm object at 0x7fe43e2cd190>, <cockpit.quantities.inner_test.InnerTest object at 0x7fe43e2cd250>, <cockpit.quantities.loss.Loss object at 0x7fe43e2cd310>, <cockpit.quantities.norm_test.NormTest object at 0x7fe43e2cd3d0>, <cockpit.quantities.ortho_test.OrthoTest object at 0x7fe43e2cd490>, <cockpit.quantities.update_size.UpdateSize object at 0x7fe43e2cd550>]
 ↪Extensions  : [<backpack.extensions.firstorder.batch_grad.BatchGrad object at 0x7fe43e2cda10>]
 ↪Hooks       : <cockpit.quantities.utils_transforms.BatchGradTransformsHook object at 0x7fe43e28e890>
 ↪Create graph: False
 ↪Save memory : True
[DEBUG] Running extension <backpack.extensions.firstorder.batch_grad.BatchGrad object at 0x7fe43e2cda10> on Linear(in_features=128, out_features=1, bias=True)
[DEBUG] Running extension <backpack.extensions.firstorder.batch_grad.BatchGrad object at 0x7fe43e2cda10> on LinearHead(
  (l): Linear(in_features=128, out_features=1, bias=True)
)
[DEBUG] Running extension <backpack.extensions.firstorder.batch_grad.BatchGrad object at 0x7fe43e2cda10> on Identity()
[DEBUG] Running extension <backpack.extensions.firstorder.batch_grad.BatchGrad object at 0x7fe43e2cda10> on Sequential(
  (0): Linear(in_features=1, out_features=128, bias=True)
  (1): Tanh()
  (2): Linear(in_features=128, out_features=128, bias=True)
  (3): Tanh()
  (4): LinearHead(
    (l): Linear(in_features=128, out_features=1, bias=True)
  )
  (5): Identity()
)
[DEBUG] Running extension <backpack.extensions.firstorder.batch_grad.BatchGrad object at 0x7fe43e2cda10> on MLP(
  (layers): Sequential(
    (0): Linear(in_features=1, out_features=128, bias=True)
    (1): Tanh()
    (2): Linear(in_features=128, out_features=128, bias=True)
    (3): Tanh()
    (4): LinearHead(
      (l): Linear(in_features=128, out_features=1, bias=True)
    )
    (5): Identity()
  )
)
Traceback (most recent call last):
  File "./envs/pytorch/lib/python3.7/site-packages/backpack/__init__.py", line 169, in run_extension_hook
    CTX.get_extension_hook()(module)
  File "./envs/pytorch/lib/python3.7/site-packages/cockpit/quantities/hooks/base.py", line 53, in __call__
    self.run_hook(param, module)
  File "./envs/pytorch/lib/python3.7/site-packages/cockpit/quantities/hooks/base.py", line 80, in run_hook
    value = self.module_hook(param, module)
  File "./envs/pytorch/lib/python3.7/site-packages/cockpit/quantities/hooks/base.py", line 139, in module_hook
    return self.param_hook(param)
  File "./envs/pytorch/lib/python3.7/site-packages/cockpit/quantities/utils_transforms.py", line 78, in param_hook
    param.grad_batch._param_weakref = weakref.ref(param)
AttributeError: 'Parameter' object has no attribute 'grad_batch'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./code/trainers/base_trainer.py", line 122, in backward
    mean_loss.backward(create_graph=create_graph)
  File "./envs/pytorch/lib/python3.7/site-packages/torch/tensor.py", line 221, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "./envs/pytorch/lib/python3.7/site-packages/torch/autograd/__init__.py", line 132, in backward
    allow_unreachable=True)  # allow_unreachable flag
  File "./envs/pytorch/lib/python3.7/site-packages/backpack/__init__.py", line 151, in hook_run_extensions
    run_extension_hook(module)
  File "./envs/pytorch/lib/python3.7/site-packages/backpack/__init__.py", line 172, in run_extension_hook
    raise RuntimeError(f"Post extensions hook failed: {message}")
RuntimeError: Post extensions hook failed: AttributeError("'Parameter' object has no attribute 'grad_batch'")

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "code/train.py", line 215, in <module>
    metrics = main(opts)
  File "code/train.py", line 185, in main
    training_info = trainer.train(model, dataset, eval_data, logger)
  File "./code/trainers/mse_trainer.py", line 128, in train
    self.backward(global_step, loss, mean_loss, optimizer)
  File "./code/trainers/base_trainer.py", line 122, in backward
    mean_loss.backward(create_graph=create_graph)
  File "./envs/pytorch/lib/python3.7/site-packages/cockpit/context.py", line 137, in __exit__
    self.cp.track(self.global_step, protected_savefields=self.protected_savefields)
  File "./envs/pytorch/lib/python3.7/site-packages/cockpit/cockpit.py", line 178, in track
    q.track(global_step, self.params, batch_loss)
  File "./envs/pytorch/lib/python3.7/site-packages/cockpit/quantities/grad_hist.py", line 93, in track
    super().track(global_step, params, batch_loss)
  File "./envs/pytorch/lib/python3.7/site-packages/cockpit/quantities/quantity.py", line 87, in track
    iteration, result = self.compute(global_step, params, batch_loss)
  File "./envs/pytorch/lib/python3.7/site-packages/cockpit/quantities/quantity.py", line 444, in compute
    return (global_step, self._compute(global_step, params, batch_loss))
  File "./envs/pytorch/lib/python3.7/site-packages/cockpit/quantities/grad_hist.py", line 114, in _compute
    hist = sum(p.grad_batch_transforms["hist_1d"][0] for p in params).detach()
  File "./envs/pytorch/lib/python3.7/site-packages/cockpit/quantities/grad_hist.py", line 114, in <genexpr>
    hist = sum(p.grad_batch_transforms["hist_1d"][0] for p in params).detach()
AttributeError: 'Parameter' object has no attribute 'grad_batch_transforms'

Thanks for the report,
Based on the traceback, I think the problem is introduced by your custom MLP module which is not recognized as container module, for which the extension hook execution should be skipped. I pushed a fix on bug-extension-hook-executes-on-custom-module that will skip execution for any module that contains other modules.
Can you install from that branch and check if it did the trick?

Yes, now it works, very nice!

Merged to development.


Flutter / android  => Printing.listPrinters() not working

Steps to Reproduce

Hello,
In my app I need to get all printer devices connected (IP address) in the same network (because i need to choose a print and store this choice) .after saving the choice my app need to print pdf without selecting a printer so i work with the printing package in the desktop version i don't have a problem but with the android version i get this exception MissingPluginException(No implementation found for method printingInfo on channel net.nfet.printing)  / i know that android native printing forces us to select a printer, but in my case, i cant go throw this way . so I have a idea maybe I must scan all connected devices in the netowrk (using ping_discover_network.dart) but the problem how i can send the pdf to the printer throw the ip address?

Hi @OusamaDridi
Packages mentioned are third-party, this is not related to Flutter itself
It seems you are asking for assistance on personal code. Please see https://flutter.dev/community for resources and asking questions like this,
you may also get some help if you post it on Stack Overflow and if you need help with your code, please see https://www.reddit.com/r/flutterhelp/
Closing, as this isn't an issue with Flutter itself. If you disagree, please write in the comments and I will reopen it.
Thank you


[Feature Request]  Hide the app contents

Hi YL,
Very nice module. I wonder if you can add a feature to instead hide the app content in the Recent Task list, similarly to FLAG_SECURE, instead of completely removing them from the list like the module is doing right now.

That's simple, but not the target feature of this module currently, so expect another module for this.
Also I'm busy with sudy in the following year and have no time to add new features.


Add a current directory option to DotlessConfiguration

I am using dotLess in such a way i need to dynamically set the current directory Import uses to one different then the current application working directory. I currently pre-process the .less file, and am toying with the idea of forking dotLess and adding it myself.
Is this a planned feature? and/or is there a reason this hasn't been added as an option?

I'm wondering about this one... is it really necessary to set the current directory, as opposed to just treating the directory containing the "entry point" file as the "current directory"? I don't see much value in doing anything else, so maybe this is something I could implement without exposing Yet Another Configuration Switch?

Meh. I thought about this some more, and obviously that won't work when the initial source comes from something other than a file. Oh well. :-)

1d7ac01 exposes the current directory as a settable property on ILessEngine -- which passes it on to Parser, which passes it on to Importer.

The fix does not work when a LESS being compiled contains an @import link to another LESS file, e.g. "bootrstrap/bootstrap.less". It just ignores the CurrentDirectory setting when resolving the path.

Do you think you could whip up a minimal test case I could use to see what's going on?

Here is the test structure:
C:\
	sample.less
	images\
		sample.gif
	subfolder\
		module.less
		images\
			module.gif

// C:\test\sample.less
@import "subfolder/module.less";
.sample {
	background-image: url('images/sample.gif');
}

// C:\test\subfolder\module.less
.module {
	background-image: url('images/module.gif');
}

// Test.cs
var lessFile = new FileInfo("C:\\test\\sample.less"");
var lessEngine = new EngineFactory().GetEngine();
lessEngine.CurrentDirectory = lessFile.DirectoryName;
string cssContent = lessEngine.TransformToCss(lessFile.OpenText().ReadToEnd(), null);

Thanks, I'll take a look at it as soon as I can.

Oops, didn't mean to close. ;-)

Thanks. I guess it has something to do with GetAdjustedFilePath()/AggregatePaths(), however since it is called from both Import() (does not work) and AlterUrl() (works), either this has to be refactored, or a fix has to be directly applied to the Import().

Many thanks!

You're welcome. This is one of the changes I'm least confident about -- definitely needs some testing with regards to how the ASP.NET path resolvers work...


Encounter Tracker Resizing When Tokens Move

I've noticed an issue where, upon moving player tokens, the Encounter Tracker will resize itself (shrink back down). However, this issue does not occur with NPCs or Monster tokens. It also occurs when either the GM or the players move player tokens (i.e. on both the program and the browser).
I'm currently working through the modules I have installed to see what could be causing the issue.

Update - it turns out that it appears to be some sort of compatibility issue with the module "PF2e Dorako UI".


Change matplotlib color scheme

The colors for the 2D meshes can be a bit wild.

The green/blue does not work particularly well in many cases. Change it or make an option to change it.
Reduce number of colors for polygons in 2D mesh. Don't sequentially number polygons, but give them all the same label.

With #192 all plotting functions should now return the plotting / axis objects, which will help sub-plotting and make it easier to tweak the plots afterwards.

Turns out that matplotlib skips colors when creating triangle plots, which results in the poor contrast green/blue. Enforcing the default colors should fix this.


Version 3.81? You've been busy!

I see that the Downloads page is now showing version 3.81. Thanks for documenting your tests in Issue #105
But the main Github page still shows release 3.06 - that seems like ancient history.
Would you please update the CHANGELOG file and the Releases page to reflect all this good work? (Nobody knows about it if we don't tell them.) Many thanks.

Hi Rich,
I'm trying to get a number of very big changes out, as soon as I have the notepad released I'm going to fix some of your smaller issues or look again at the MAC in general.

Excellent! I'll look forward to the Notepad...

Done


Error: Could not find voice with identifier Domi

System Health details
System Information

version
core-2024.7.3

installation_type
Home Assistant OS

dev
false

hassio
true

docker
true

user
root

virtualenv
false

python_version
3.12.4

os_name
Linux

os_version
6.6.31-haos-raspi

arch
aarch64

timezone
America/Los_Angeles

config_dir
/config

Home Assistant Community Store

GitHub API
ok

GitHub Content
ok

GitHub Web
ok

GitHub API Calls Remaining
4458

Installed Version
1.34.0

Stage
running

Available Repositories
1393

Downloaded Repositories
104

AccuWeather

can_reach_server
ok

remaining_requests
29

Home Assistant Cloud

logged_in
false

can_reach_cert_server
ok

can_reach_cloud_auth
ok

can_reach_cloud
ok

Home Assistant Supervisor

host_os
Home Assistant OS 12.4

update_channel
stable

supervisor_version
supervisor-2024.06.2

agent_version
1.6.0

docker_version
26.1.4

disk_total
469.4 GB

disk_used
19.0 GB

healthy
true

supported
true

host_connectivity
true

supervisor_connectivity
true

ntp_synchronized
true

virtualization

board
yellow

supervisor_api
ok

version_api
ok

installed_addons
Matter Server (6.3.1), Mosquitto broker (6.4.1), Duck DNS (1.18.0), NGINX Home Assistant SSL proxy (3.9.0), Samba share (12.3.1), OneDrive Backup (2.3.1), ESPHome (2024.7.0), Govee to MQTT Bridge (2024.07.13-82ddc6e9), Dnsmasq (1.8.1), Piper (1.5.1), Whisper (2.1.2), Genie - Edge (edge-0c63b72), openWakeWord (1.10.0), Music Assistant (2.0.7), Studio Code Server (5.15.0)

Dashboards

dashboards
14

resources
70

views
28

mode
storage

Recorder

oldest_recorder_run
July 9, 2024 at 11:17 AM

current_recorder_run
July 19, 2024 at 6:37 PM

estimated_db_size
1587.52 MiB

database_engine
sqlite

database_version
3.45.3

Checklist

[X] I have enabled debug logging for my installation.
[X] I have filled out the issue template to the best of my ability.
[X] This issue only contains 1 issue (if you have multiple issues, open one issue for each issue).
[X] This issue is not a duplicate issue of currently previous issues..

Describe the issue
Starting today,
When loading the ElevenLabs TTS integration, the integration fails and leaves an error in the log as follows:
This error originated from a custom integration.
Logger: custom_components.elevenlabs_tts.elevenlabs
Source: custom_components/elevenlabs_tts/elevenlabs.py:116
integration: ElevenLabs TTS (documentation, issues)
First occurred: 6:37:29 PM (4 occurrences)
Last logged: 6:45:54 PM
Could not find voice with identifier Domi
Domi is not selected in configuration, but Domi is listed as Domi (Legacy) on ElevenLabs.
Reproduction steps

Reload existing configuration
Copy API key and recreate the eleven labs configuration with a different, valid voice
Reload new configuration
Witness error in logs twice

Debug logs
2024-07-19 18:37:07.805 WARNING (SyncWorker_0) [homeassistant.loader] We found a custom integration energy_meter which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-07-19 18:37:07.808 WARNING (SyncWorker_0) [homeassistant.loader] We found a custom integration epson_projector_link which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-07-19 18:37:07.811 WARNING (SyncWorker_0) [homeassistant.loader] We found a custom integration elevenlabs_tts which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-07-19 18:37:07.817 WARNING (SyncWorker_0) [homeassistant.loader] We found a custom integration smartir which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-07-19 18:37:07.821 WARNING (SyncWorker_0) [homeassistant.loader] We found a custom integration grad_vol which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-07-19 18:37:07.823 WARNING (SyncWorker_0) [homeassistant.loader] We found a custom integration plant which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-07-19 18:37:07.825 WARNING (SyncWorker_0) [homeassistant.loader] We found a custom integration hass_agent which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-07-19 18:37:07.827 WARNING (SyncWorker_0) [homeassistant.loader] We found a custom integration lennoxs30 which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-07-19 18:37:07.829 WARNING (SyncWorker_0) [homeassistant.loader] We found a custom integration sun2 which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-07-19 18:37:07.831 WARNING (SyncWorker_0) [homeassistant.loader] We found a custom integration hacs which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-07-19 18:37:07.832 WARNING (SyncWorker_0) [homeassistant.loader] We found a custom integration smartthinq_sensors which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-07-19 18:37:07.834 WARNING (SyncWorker_0) [homeassistant.loader] We found a custom integration bhyve which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-07-19 18:37:07.836 WARNING (SyncWorker_0) [homeassistant.loader] We found a custom integration flair which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-07-19 18:37:07.837 WARNING (SyncWorker_0) [homeassistant.loader] We found a custom integration scene_presets which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-07-19 18:37:07.840 WARNING (SyncWorker_0) [homeassistant.loader] We found a custom integration attributes which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-07-19 18:37:07.842 WARNING (SyncWorker_0) [homeassistant.loader] We found a custom integration openplantbook which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-07-19 18:37:07.844 WARNING (SyncWorker_0) [homeassistant.loader] We found a custom integration huesyncbox which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-07-19 18:37:16.178 WARNING (MainThread) [homeassistant.helpers.frame] Detected that custom integration 'hacs' calls hass.http.register_static_path which is deprecated because it does blocking I/O in the event loop, instead call `await hass.http.async_register_static_paths([StaticPathConfig("/hacsfiles/themes", "/config/themes", True)])`; This function will be removed in 2025.7 at custom_components/hacs/base.py, line 1182: self.hass.http.register_static_path(f"{URL_BASE}/themes", self.hass.config.path("themes")), please create a bug report at https://github.com/hacs/integration/issues
2024-07-19 18:37:16.189 WARNING (MainThread) [homeassistant.helpers.frame] Detected that custom integration 'hacs' calls hass.http.register_static_path which is deprecated because it does blocking I/O in the event loop, instead call `await hass.http.async_register_static_paths([StaticPathConfig("/hacsfiles/frontend", "/config/custom_components/hacs/hacs_frontend", False)])`; This function will be removed in 2025.7 at custom_components/hacs/frontend.py, line 54: hass.http.register_static_path(f"{URL_BASE}/frontend", locate_dir(), cache_headers=False), please create a bug report at https://github.com/hacs/integration/issues
2024-07-19 18:37:16.197 WARNING (MainThread) [homeassistant.helpers.frame] Detected that custom integration 'hacs' calls hass.http.register_static_path which is deprecated because it does blocking I/O in the event loop, instead call `await hass.http.async_register_static_paths([StaticPathConfig("/hacsfiles/iconset.js", "/config/custom_components/hacs/iconset.js", True)])`; This function will be removed in 2025.7 at custom_components/hacs/frontend.py, line 57: hass.http.register_static_path(, please create a bug report at https://github.com/hacs/integration/issues
2024-07-19 18:37:16.206 WARNING (MainThread) [homeassistant.helpers.frame] Detected that custom integration 'hacs' accesses hass.components.frontend. This is deprecated and will stop working in Home Assistant 2024.9, it should be updated to import functions used from frontend directly at custom_components/hacs/frontend.py, line 68: hass.components.frontend.async_register_built_in_panel(, please create a bug report at https://github.com/hacs/integration/issues
2024-07-19 18:37:16.214 WARNING (MainThread) [homeassistant.helpers.frame] Detected that custom integration 'hacs' calls hass.http.register_static_path which is deprecated because it does blocking I/O in the event loop, instead call `await hass.http.async_register_static_paths([StaticPathConfig("/hacsfiles", "/config/www/community", True)])`; This function will be removed in 2025.7 at custom_components/hacs/base.py, line 1162: self.hass.http.register_static_path(, please create a bug report at https://github.com/hacs/integration/issues
2024-07-19 18:37:28.713 WARNING (MainThread) [homeassistant.util.loop] Detected blocking call to open with args ('/config/custom_components/smartir/codes/media_player/1062.json',) inside the event loop by custom integration 'smartir' at custom_components/smartir/media_player.py, line 75: with open(device_json_path) as j: (offender: /config/custom_components/smartir/media_player.py, line 75: with open(device_json_path) as j:), please report it to the author of the 'smartir' custom integration
For developers, please see https://developers.home-assistant.io/docs/asyncio_blocking_operations/#open
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/src/homeassistant/homeassistant/__main__.py", line 223, in <module>
    sys.exit(main())
  File "/usr/src/homeassistant/homeassistant/__main__.py", line 209, in main
    exit_code = runner.run(runtime_conf)
  File "/usr/src/homeassistant/homeassistant/runner.py", line 190, in run
    return loop.run_until_complete(setup_and_run_hass(runtime_config))
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 674, in run_until_complete
    self.run_forever()
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 641, in run_forever
    self._run_once()
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 1990, in _run_once
    handle._run()
  File "/usr/local/lib/python3.12/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/src/homeassistant/homeassistant/helpers/entity_component.py", line 318, in async_setup_platform
    await self._platforms[key].async_setup(platform_config, discovery_info)
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 301, in async_setup
    await self._async_setup_platform(async_create_setup_awaitable)
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 361, in _async_setup_platform
    awaitable = create_eager_task(awaitable, loop=hass.loop)
  File "/usr/src/homeassistant/homeassistant/util/async_.py", line 37, in create_eager_task
    return Task(coro, loop=loop, name=name, eager_start=True)
  File "/config/custom_components/smartir/media_player.py", line 75, in async_setup_platform
    with open(device_json_path) as j:

2024-07-19 18:37:28.716 WARNING (MainThread) [homeassistant.helpers.entity] Entity None (<class 'custom_components.smartir.media_player.SmartIRMediaPlayer'>) is using deprecated supported features values which will be removed in HA Core 2025.1. Instead it should use <MediaPlayerEntityFeature.VOLUME_MUTE|PREVIOUS_TRACK|NEXT_TRACK|TURN_ON|TURN_OFF|PLAY_MEDIA|VOLUME_STEP|SELECT_SOURCE: 4024>, please report it to the author of the 'smartir' custom integration and reference https://developers.home-assistant.io/blog/2023/12/28/support-feature-magic-numbers-deprecation
2024-07-19 18:37:28.728 WARNING (MainThread) [homeassistant.helpers.entity] Entity None (<class 'custom_components.smartir.media_player.SmartIRMediaPlayer'>) is using deprecated supported features values which will be removed in HA Core 2025.1. Instead it should use <MediaPlayerEntityFeature.TURN_ON|TURN_OFF: 384>, please report it to the author of the 'smartir' custom integration and reference https://developers.home-assistant.io/blog/2023/12/28/support-feature-magic-numbers-deprecation
2024-07-19 18:37:28.740 WARNING (MainThread) [homeassistant.helpers.entity] Entity None (<class 'custom_components.smartir.media_player.SmartIRMediaPlayer'>) is using deprecated supported features values which will be removed in HA Core 2025.1. Instead it should use <MediaPlayerEntityFeature.TURN_ON|TURN_OFF: 384>, please report it to the author of the 'smartir' custom integration and reference https://developers.home-assistant.io/blog/2023/12/28/support-feature-magic-numbers-deprecation
2024-07-19 18:37:28.750 WARNING (MainThread) [homeassistant.helpers.entity] Entity None (<class 'custom_components.smartir.media_player.SmartIRMediaPlayer'>) is using deprecated supported features values which will be removed in HA Core 2025.1. Instead it should use <MediaPlayerEntityFeature.TURN_ON|TURN_OFF: 384>, please report it to the author of the 'smartir' custom integration and reference https://developers.home-assistant.io/blog/2023/12/28/support-feature-magic-numbers-deprecation
2024-07-19 18:37:29.394 WARNING (MainThread) [custom_components.elevenlabs_tts.elevenlabs] Could not find voice with identifier Domi
2024-07-19 18:37:33.367 WARNING (MainThread) [homeassistant.helpers.frame] Detected that custom integration 'scene_presets' calls hass.http.register_static_path which is deprecated because it does blocking I/O in the event loop, instead call `await hass.http.async_register_static_paths([StaticPathConfig("/scene_presets_panel.js", "/config/custom_components/scene_presets/frontend/scene_presets_panel.js", True)])`; This function will be removed in 2025.7 at custom_components/scene_presets/view.py, line 26: hass.http.register_static_path(, please create a bug report at https://github.com/Hypfer/hass-scene_presets/issues
2024-07-19 18:37:33.379 WARNING (MainThread) [homeassistant.helpers.frame] Detected that custom integration 'scene_presets' calls hass.http.register_static_path which is deprecated because it does blocking I/O in the event loop, instead call `await hass.http.async_register_static_paths([StaticPathConfig("/assets/scene_presets/e03267e7-9914-4f47-97fe-63c0bd317fe7.jpeg", "/config/custom_components/scene_presets/assets/e03267e7-9914-4f47-97fe-63c0bd317fe7.jpeg", True)])`; This function will be removed in 2025.7 at custom_components/scene_presets/view.py, line 71: hass.http.register_static_path(, please create a bug report at https://github.com/Hypfer/hass-scene_presets/issues
2024-07-19 18:37:35.276 WARNING (MainThread) [homeassistant.helpers.frame] Detected that custom integration 'scene_presets' calls hass.http.register_static_path which is deprecated because it does blocking I/O in the event loop, instead call `await hass.http.async_register_static_paths([StaticPathConfig("/assets/scene_presets/iconset.js", "/config/custom_components/scene_presets/res/iconset.js", True)])`; This function will be removed in 2025.7 at custom_components/scene_presets/view.py, line 52: hass.http.register_static_path(, please create a bug report at https://github.com/Hypfer/hass-scene_presets/issues
2024-07-19 18:37:36.019 WARNING (MainThread) [homeassistant.helpers.frame] Detected that custom integration 'lennoxs30' calls async_forward_entry_setup for integration, lennoxs30 with title: <IP_ADDRESS> and entry_id: 2ff08e5335c764b876f51d514c69cb71, which is deprecated and will stop working in Home Assistant 2025.6, await async_forward_entry_setups instead at custom_components/lennoxs30/__init__.py, line 552: self._hass.async_create_task(, please create a bug report at https://github.com/PeteRager/lennoxs30/issues
2024-07-19 18:37:40.522 WARNING (MainThread) [homeassistant.components.vizio.media_player] Lost connection to <IP_ADDRESS>:7345
2024-07-19 18:37:40.525 WARNING (MainThread) [homeassistant.components.vizio.media_player] Lost connection to <IP_ADDRESS>:7345
2024-07-19 18:37:51.477 WARNING (MainThread) [homeassistant.helpers.frame] Detected code that calls async_forward_entry_setup for integration lennoxs30 with title: <IP_ADDRESS> and entry_id: 2ff08e5335c764b876f51d514c69cb71, during setup without awaiting async_forward_entry_setup, which can cause the setup lock to be released before the setup is done. This will stop working in Home Assistant 2025.1. Please report this issue.
Stack (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/src/homeassistant/homeassistant/__main__.py", line 223, in <module>
    sys.exit(main())
  File "/usr/src/homeassistant/homeassistant/__main__.py", line 209, in main
    exit_code = runner.run(runtime_conf)
  File "/usr/src/homeassistant/homeassistant/runner.py", line 190, in run
    return loop.run_until_complete(setup_and_run_hass(runtime_config))
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 674, in run_until_complete
    self.run_forever()
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 641, in run_forever
    self._run_once()
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 1990, in _run_once
    handle._run()
  File "/usr/local/lib/python3.12/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/src/homeassistant/homeassistant/config_entries.py", line 2122, in async_forward_entry_setup
    _report_non_awaited_platform_forwards(entry, "async_forward_entry_setup")
  File "/usr/src/homeassistant/homeassistant/config_entries.py", line 1175, in _report_non_awaited_platform_forwards
    report(
  File "/usr/src/homeassistant/homeassistant/helpers/frame.py", line 151, in report
    _LOGGER.warning(msg, stack_info=True)
2024-07-19 18:37:51.618 WARNING (MainThread) [homeassistant.helpers.frame] Detected code that calls async_forward_entry_setup for integration lennoxs30 with title: <IP_ADDRESS> and entry_id: 2ff08e5335c764b876f51d514c69cb71, during setup without awaiting async_forward_entry_setup, which can cause the setup lock to be released before the setup is done. This will stop working in Home Assistant 2025.1. Please report this issue.
Stack (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/src/homeassistant/homeassistant/__main__.py", line 223, in <module>
    sys.exit(main())
  File "/usr/src/homeassistant/homeassistant/__main__.py", line 209, in main
    exit_code = runner.run(runtime_conf)
  File "/usr/src/homeassistant/homeassistant/runner.py", line 190, in run
    return loop.run_until_complete(setup_and_run_hass(runtime_config))
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 674, in run_until_complete
    self.run_forever()
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 641, in run_forever
    self._run_once()
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 1990, in _run_once
    handle._run()
  File "/usr/local/lib/python3.12/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/src/homeassistant/homeassistant/config_entries.py", line 2122, in async_forward_entry_setup
    _report_non_awaited_platform_forwards(entry, "async_forward_entry_setup")
  File "/usr/src/homeassistant/homeassistant/config_entries.py", line 1175, in _report_non_awaited_platform_forwards
    report(
  File "/usr/src/homeassistant/homeassistant/helpers/frame.py", line 151, in report
    _LOGGER.warning(msg, stack_info=True)
2024-07-19 18:37:51.853 WARNING (MainThread) [homeassistant.helpers.frame] Detected code that calls async_forward_entry_setup for integration lennoxs30 with title: <IP_ADDRESS> and entry_id: 2ff08e5335c764b876f51d514c69cb71, during setup without awaiting async_forward_entry_setup, which can cause the setup lock to be released before the setup is done. This will stop working in Home Assistant 2025.1. Please report this issue.
Stack (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/src/homeassistant/homeassistant/__main__.py", line 223, in <module>
    sys.exit(main())
  File "/usr/src/homeassistant/homeassistant/__main__.py", line 209, in main
    exit_code = runner.run(runtime_conf)
  File "/usr/src/homeassistant/homeassistant/runner.py", line 190, in run
    return loop.run_until_complete(setup_and_run_hass(runtime_config))
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 674, in run_until_complete
    self.run_forever()
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 641, in run_forever
    self._run_once()
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 1990, in _run_once
    handle._run()
  File "/usr/local/lib/python3.12/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/src/homeassistant/homeassistant/config_entries.py", line 2122, in async_forward_entry_setup
    _report_non_awaited_platform_forwards(entry, "async_forward_entry_setup")
  File "/usr/src/homeassistant/homeassistant/config_entries.py", line 1175, in _report_non_awaited_platform_forwards
    report(
  File "/usr/src/homeassistant/homeassistant/helpers/frame.py", line 151, in report
    _LOGGER.warning(msg, stack_info=True)
2024-07-19 18:37:52.013 WARNING (MainThread) [homeassistant.helpers.frame] Detected code that calls async_forward_entry_setup for integration lennoxs30 with title: <IP_ADDRESS> and entry_id: 2ff08e5335c764b876f51d514c69cb71, during setup without awaiting async_forward_entry_setup, which can cause the setup lock to be released before the setup is done. This will stop working in Home Assistant 2025.1. Please report this issue.
Stack (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/src/homeassistant/homeassistant/__main__.py", line 223, in <module>
    sys.exit(main())
  File "/usr/src/homeassistant/homeassistant/__main__.py", line 209, in main
    exit_code = runner.run(runtime_conf)
  File "/usr/src/homeassistant/homeassistant/runner.py", line 190, in run
    return loop.run_until_complete(setup_and_run_hass(runtime_config))
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 674, in run_until_complete
    self.run_forever()
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 641, in run_forever
    self._run_once()
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 1990, in _run_once
    handle._run()
  File "/usr/local/lib/python3.12/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/src/homeassistant/homeassistant/config_entries.py", line 2122, in async_forward_entry_setup
    _report_non_awaited_platform_forwards(entry, "async_forward_entry_setup")
  File "/usr/src/homeassistant/homeassistant/config_entries.py", line 1175, in _report_non_awaited_platform_forwards
    report(
  File "/usr/src/homeassistant/homeassistant/helpers/frame.py", line 151, in report
    _LOGGER.warning(msg, stack_info=True)
2024-07-19 18:37:52.083 WARNING (MainThread) [homeassistant.helpers.frame] Detected code that calls async_forward_entry_setup for integration lennoxs30 with title: <IP_ADDRESS> and entry_id: 2ff08e5335c764b876f51d514c69cb71, during setup without awaiting async_forward_entry_setup, which can cause the setup lock to be released before the setup is done. This will stop working in Home Assistant 2025.1. Please report this issue.
Stack (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/src/homeassistant/homeassistant/__main__.py", line 223, in <module>
    sys.exit(main())
  File "/usr/src/homeassistant/homeassistant/__main__.py", line 209, in main
    exit_code = runner.run(runtime_conf)
  File "/usr/src/homeassistant/homeassistant/runner.py", line 190, in run
    return loop.run_until_complete(setup_and_run_hass(runtime_config))
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 674, in run_until_complete
    self.run_forever()
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 641, in run_forever
    self._run_once()
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 1990, in _run_once
    handle._run()
  File "/usr/local/lib/python3.12/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/src/homeassistant/homeassistant/config_entries.py", line 2122, in async_forward_entry_setup
    _report_non_awaited_platform_forwards(entry, "async_forward_entry_setup")
  File "/usr/src/homeassistant/homeassistant/config_entries.py", line 1175, in _report_non_awaited_platform_forwards
    report(
  File "/usr/src/homeassistant/homeassistant/helpers/frame.py", line 151, in report
    _LOGGER.warning(msg, stack_info=True)
2024-07-19 18:37:52.270 WARNING (MainThread) [homeassistant.helpers.frame] Detected code that calls async_forward_entry_setup for integration lennoxs30 with title: <IP_ADDRESS> and entry_id: 2ff08e5335c764b876f51d514c69cb71, during setup without awaiting async_forward_entry_setup, which can cause the setup lock to be released before the setup is done. This will stop working in Home Assistant 2025.1. Please report this issue.
Stack (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/src/homeassistant/homeassistant/__main__.py", line 223, in <module>
    sys.exit(main())
  File "/usr/src/homeassistant/homeassistant/__main__.py", line 209, in main
    exit_code = runner.run(runtime_conf)
  File "/usr/src/homeassistant/homeassistant/runner.py", line 190, in run
    return loop.run_until_complete(setup_and_run_hass(runtime_config))
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 674, in run_until_complete
    self.run_forever()
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 641, in run_forever
    self._run_once()
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 1990, in _run_once
    handle._run()
  File "/usr/local/lib/python3.12/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/src/homeassistant/homeassistant/config_entries.py", line 2122, in async_forward_entry_setup
    _report_non_awaited_platform_forwards(entry, "async_forward_entry_setup")
  File "/usr/src/homeassistant/homeassistant/config_entries.py", line 1175, in _report_non_awaited_platform_forwards
    report(
  File "/usr/src/homeassistant/homeassistant/helpers/frame.py", line 151, in report
    _LOGGER.warning(msg, stack_info=True)
2024-07-19 18:37:52.698 WARNING (MainThread) [homeassistant.helpers.frame] Detected code that calls async_forward_entry_setup for integration lennoxs30 with title: <IP_ADDRESS> and entry_id: 2ff08e5335c764b876f51d514c69cb71, during setup without awaiting async_forward_entry_setup, which can cause the setup lock to be released before the setup is done. This will stop working in Home Assistant 2025.1. Please report this issue.
Stack (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/src/homeassistant/homeassistant/__main__.py", line 223, in <module>
    sys.exit(main())
  File "/usr/src/homeassistant/homeassistant/__main__.py", line 209, in main
    exit_code = runner.run(runtime_conf)
  File "/usr/src/homeassistant/homeassistant/runner.py", line 190, in run
    return loop.run_until_complete(setup_and_run_hass(runtime_config))
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 674, in run_until_complete
    self.run_forever()
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 641, in run_forever
    self._run_once()
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 1990, in _run_once
    handle._run()
  File "/usr/local/lib/python3.12/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/src/homeassistant/homeassistant/config_entries.py", line 2122, in async_forward_entry_setup
    _report_non_awaited_platform_forwards(entry, "async_forward_entry_setup")
  File "/usr/src/homeassistant/homeassistant/config_entries.py", line 1175, in _report_non_awaited_platform_forwards
    report(
  File "/usr/src/homeassistant/homeassistant/helpers/frame.py", line 151, in report
    _LOGGER.warning(msg, stack_info=True)
2024-07-19 18:38:07.144 INFO (MainThread) [homeassistant.core] Starting Home Assistant
2024-07-19 18:38:07.223 WARNING (MainThread) [homeassistant.helpers.frame] Detected that custom integration 'attributes' calls `async_track_state_change` instead of `async_track_state_change_event` which is deprecated and will be removed in Home Assistant 2025.5 at custom_components/attributes/sensor.py, line 221: async_track_state_change(, please create a bug report at https://github.com/pilotak/homeassistant-attributes/issues
2024-07-19 18:38:07.231 WARNING (MainThread) [homeassistant.helpers.frame] Detected that custom integration 'attributes' calls `async_add_job`, which is deprecated and will be removed in Home Assistant 2025.4; Please review https://developers.home-assistant.io/blog/2024/03/13/deprecate_add_run_job for replacement options at custom_components/attributes/sensor.py, line 224: self.hass.async_add_job(self.async_update_ha_state(True)), please create a bug report at https://github.com/pilotak/homeassistant-attributes/issues
2024-07-19 18:38:07.378 ERROR (MainThread) [homeassistant.helpers.event] Error while processing template: Template<template=({% set list = [
  states("sensor.gillian_s_air_monitor_calibrated_temperature") | float,
  states("sensor.bob_s_air_monitor_calibrated_temperature") | float,
  states("sensor.master_bedroom_remote_temperature") | float
  ]
%}
{{ average(list) }}) renders=2>
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/template.py", line 2258, in forgiving_float_filter
    return float(value)
           ^^^^^^^^^^^^
ValueError: could not convert string to float: 'unknown'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/template.py", line 604, in async_render
    render_result = _render_with_context(self.template, compiled, **kwargs)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/template.py", line 2639, in _render_with_context
    return template.render(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/jinja2/environment.py", line 1304, in render
    self.environment.handle_exception()
  File "/usr/local/lib/python3.12/site-packages/jinja2/environment.py", line 939, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 1, in top-level template code
  File "/usr/src/homeassistant/homeassistant/helpers/template.py", line 2261, in forgiving_float_filter
    raise_no_default("float", value)
  File "/usr/src/homeassistant/homeassistant/helpers/template.py", line 1872, in raise_no_default
    raise ValueError(
ValueError: Template error: float got invalid input 'unknown' when rendering template '{% set list = [
  states("sensor.gillian_s_air_monitor_calibrated_temperature") | float,
  states("sensor.bob_s_air_monitor_calibrated_temperature") | float,
  states("sensor.master_bedroom_remote_temperature") | float
  ]
%}
{{ average(list) }}' but no default was specified

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/template.py", line 716, in async_render_to_info
    render_info._result = self.async_render(  # noqa: SLF001
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/template.py", line 606, in async_render
    raise TemplateError(err) from err
homeassistant.exceptions.TemplateError: ValueError: Template error: float got invalid input 'unknown' when rendering template '{% set list = [
  states("sensor.gillian_s_air_monitor_calibrated_temperature") | float,
  states("sensor.bob_s_air_monitor_calibrated_temperature") | float,
  states("sensor.master_bedroom_remote_temperature") | float
  ]
%}
{{ average(list) }}' but no default was specified
2024-07-19 18:38:07.387 ERROR (MainThread) [homeassistant.components.template.template_entity] TemplateError('ValueError: Template error: float got invalid input 'unknown' when rendering template '{% set list = [
  states("sensor.gillian_s_air_monitor_calibrated_temperature") | float,
  states("sensor.bob_s_air_monitor_calibrated_temperature") | float,
  states("sensor.master_bedroom_remote_temperature") | float
  ]
%}
{{ average(list) }}' but no default was specified') while processing template 'Template<template=({% set list = [
  states("sensor.gillian_s_air_monitor_calibrated_temperature") | float,
  states("sensor.bob_s_air_monitor_calibrated_temperature") | float,
  states("sensor.master_bedroom_remote_temperature") | float
  ]
%}
{{ average(list) }}) renders=4>' for attribute '_attr_native_value' in entity 'sensor.2nd_floor_average'
2024-07-19 18:38:07.515 WARNING (MainThread) [homeassistant.helpers.frame] Detected that custom integration 'hacs' accesses hass.helpers.event. This is deprecated and will stop working in Home Assistant 2024.11, it should be updated to import functions used from event directly at custom_components/hacs/base.py, line 630: self.hass.helpers.event.async_track_time_interval(, please create a bug report at https://github.com/hacs/integration/issues
2024-07-19 18:38:41.835 WARNING (MainThread) [homeassistant.components.mqtt.light.schema_json] Deprecated flag `color_mode` used in MQTT JSON light config , the `color_mode` flag is not used anymore and should be removed.  This will stop working in Home Assistant Core 2025.3
(repeat the Govee2Mqtt block per light per scene making this log too long)
2024-07-19 18:38:58.023 WARNING (MainThread) [custom_components.elevenlabs_tts.elevenlabs] Could not find voice with identifier Domi
2024-07-19 18:42:32.593 WARNING (MainThread) [custom_components.elevenlabs_tts.elevenlabs] Could not find voice with identifier Domi
2024-07-19 18:45:02.460 WARNING (MainThread) [homeassistant.helpers.frame] Detected that custom integration 'attributes' calls `async_add_job`, which is deprecated and will be removed in Home Assistant 2025.4; Please review https://developers.home-assistant.io/blog/2024/03/13/deprecate_add_run_job for replacement options at custom_components/attributes/sensor.py, line 216: self.hass.async_add_job(self.async_update_ha_state(True)), please create a bug report at https://github.com/pilotak/homeassistant-attributes/issues
2024-07-19 18:45:54.529 DEBUG (MainThread) [custom_components.elevenlabs_tts.elevenlabs] Initializing ElevenLabs client
2024-07-19 18:45:54.993 DEBUG (MainThread) [custom_components.elevenlabs_tts.elevenlabs] Looking for voice with identifier Domi
2024-07-19 18:45:54.993 WARNING (MainThread) [custom_components.elevenlabs_tts.elevenlabs] Could not find voice with identifier Domi

Diagnostics dump
Not applicable/available

same issue

Same here

Same.
I have jeremy voice configjred.

Looks like they are having issues at Eleven Labs. I had the same error as above. I generated a new API key and now getting a client auth error.

same issue

Same here. Using custom voices.

It seems they change names (Jeremy goes legacy now)

Maybe thats the problem

change /homeassistant/custom_components/elevenlabs_tts/const.py
remove  DEFAULT_VOICE  from Domi to other available voice.  restart hass. and add again
"""Consts module."""
from homeassistant.const import Platform
################################
Do not change! Will be set by release workflow
INTEGRATION_VERSION = "2.3.0"
MIN_REQUIRED_HA_VERSION = "2023.8.1"
################################
PLATFORMS = [Platform.TTS]
DOMAIN = "elevenlabs_tts"
VERSION = "1.0.0"
DEFAULT_VOICE = "Laura"
CONF_STABILITY = "stability"
DEFAULT_STABILITY = 0.75
CONF_SIMILARITY = "similarity"
DEFAULT_SIMILARITY = 0.9
CONF_MODEL = "model"
DEFAULT_MODEL = "eleven_multilingual_v2"
CONF_OPTIMIZE_LATENCY = "optimize_streaming_latency"
DEFAULT_OPTIMIZE_LATENCY = 0
CONF_STYLE = "style"
DEFAULT_STYLE = 0.2
CONF_USE_SPEAKER_BOOST = "use_speaker_boost"
DEFAULT_USE_SPEAKER_BOOST = True

change /homeassistant/custom_components/elevenlabs_tts/const.py
remove DEFAULT_VOICE from Domi to other available voice. restart hass. and add again
"""Consts module.""" from homeassistant.const import Platform
################################
Do not change! Will be set by release workflow
INTEGRATION_VERSION = "2.3.0" MIN_REQUIRED_HA_VERSION = "2023.8.1" ################################
PLATFORMS = [Platform.TTS]
DOMAIN = "elevenlabs_tts" VERSION = "1.0.0"
DEFAULT_VOICE = "Laura" CONF_STABILITY = "stability" DEFAULT_STABILITY = 0.75 CONF_SIMILARITY = "similarity" DEFAULT_SIMILARITY = 0.9 CONF_MODEL = "model" DEFAULT_MODEL = "eleven_multilingual_v2" CONF_OPTIMIZE_LATENCY = "optimize_streaming_latency" DEFAULT_OPTIMIZE_LATENCY = 0 CONF_STYLE = "style" DEFAULT_STYLE = 0.2 CONF_USE_SPEAKER_BOOST = "use_speaker_boost" DEFAULT_USE_SPEAKER_BOOST = True

Confirmed working.

Can you expand on this fix? Bit confused what to change and the DO NOT CHANGE bit

Can you expand on this fix? Bit confused what to change and the DO NOT CHANGE bit

first , u need to go  eleven lab integration in hass and  delete your current integration entries
use text editor and edit /homeassistant/custom_components/elevenlabs_tts/const.py
change DEFAULT_VOICE = "Domi" ----> DEFAULT_VOICE = "Laura" , Save
restart hass ( i did but not sure it is necessary) and readd 11th lab integration entry

Can you expand on this fix? Bit confused what to change and the DO NOT CHANGE bit

first , u need to go  eleven lab integration in hass and  delete your current integration entries
use text editor and edit /homeassistant/custom_components/elevenlabs_tts/const.py
change DEFAULT_VOICE = "Domi" ----> DEFAULT_VOICE = "Laura" , Save
restart hass ( i did but not sure it was even necessary) and readd 11th lab integration entry

Fixed - Perfect - many thanks!

2.3.1 should fix this. Thank you all for reporting and finding a fix so quickly :)

Has the way to use this changed too I cant get it to send any data or receive it

Check if you have free quota lefts?

i do a fair bit i paid for a licence

Paid subscriptions also have quotas...

Please open new issue.


docs: update LlamaCpp input args

This PR only updates the LlamaCpp args documentation. The input arg has been flattened.

thanks @emmanuelgautier!


BitClout Improvement Proposals

As BitClout development becomes increasingly community-driven, I believe it is time for the creation of a repository to draft and track BitClout Improvement Proposals (abbreviated BCIPs).
Overall, the BitClout community is understanding of the need, early on, for the core development team to have very flexible control over the protocol. However, more and more of us in the development community are building real-world businesses on top of the BitClout protocol, and I believe it would be a benefit to everyone if such a BCIP system existed.
This proposal calls for the creation of a repo at github.com/bitclout/bcips, similar to the equivalent repo within Bitcoin, github.com/bitcoin/bips.
Individuals drafting proposals would simply fork the bitclout/bcips repo, create a folder and README for their proposal (bcip-001, bcip-002, etc), and submit a pull request back to the bitclout/bcips repo. It would be up to the core development team to edit and merge these proposals as they see fit.
These proposals should be limited in scope to the core BitClout protocol, not the backend, frontend, or any other project.
Thank you for your consideration.

Excellent idea, I've been thinking about this over the last week. The real question is should they be called BCIP or CIP?

Oh great, a chance for me to be indecisive 😂. Hmm, I'm not attached to BCIP, just used it to paint the picture. CIP has a lot going for it: shorter, more pronounceable. Either one would have my support!

+1 CIP

cip +2

Hi @maebeam
I tried running a community-version of Clout Improvement Proposals a while back, but it seems to have not been noticed
Would bitclout mind if I transferred over the repo to your org? That way, it is technically already built w/ issue templates and everything.
https://github.com/Mentors4EDU/BitClout-Proposals

Excellent idea, I've been thinking about this over the last week. The real question is should they be called BCIP or CIP?

I called it CIP (Clout Improvement Proposal) and loosely created it similar to BIP.
Here is my first one. I am hoping this can reside in the doc repo.
https://github.com/bitclout/docs/pull/10

I will make the CIP repo today or tomorrow

@maebeam If it easier, you can just clone my issue templates w/ the tags or start from scratch depending on what the devs prefer

In the spirit of BIP, I hope our CIP repo will focus entirely on the BitClout protocol. Not improvements to the BitClout.com website, nor even to a particular implementation of the blockchain.
Thanks @maebeam for taking charge on this!

Anything I can do to help to move this forward?

Anything I can do to help to move this forward?

+1
i would also love to have some clarification on the new developments in regards to the new concensus mechanism, apparently pos is being considered next? i would appreciate to understand if cip's and its voting would be affected under a pos consensus.
cheers!

Sorry for dropping the ball on this everyone. I will create this repo today.

Perrrrfect!

Sorry for dropping the ball on this everyone. I will create this repo today.

Any updates, @maebeam ?

Coming soon!

Love the idea of creating a proposals repo! One thing I had in mind though.
How do you pronounce CIP? Sip? Kip?
Why not CLIP instead? Or any other abbreviation which is not as prone to mispronounciation?

And launched!


Type narrowing of object properties by variables does not work as expected

🔎 Search Terms
"narrowing object property", "type narrowing", "narrowing", "narrowing truthiness"
🕗 Version & Regression Information
In version 4.4.4 was added support for narrowing a local variable by constant variable that had result for narrowing (last example in code)
⏯ Playground Link
Link here
💻 Code
const obj: {nestedObj: {nestedProp: string | null}} = {nestedObj: {nestedProp: "test"}};
const noNestedProp = !obj.nestedObj.nestedProp || obj.nestedObj.nestedProp === "";

//noNestedProp does not narrow nestedProp for truthiness
if(!noNestedProp) {
  const shouldBeString: string = obj.nestedObj.nestedProp; //Type 'string | null' is not assignable to type 'string'.
}

//same logic as noNestedProp
if(!(!obj.nestedObj.nestedProp || obj.nestedObj.nestedProp === "")) {
  const shouldBeString: string = obj.nestedObj.nestedProp; //works
}

//but works with the same property assigned to variable
const nestedProp = obj.nestedObj.nestedProp;
const noNestedPropVar = !nestedProp || nestedProp === "";

if(!noNestedPropVar) {
  const shouldBeString: string = nestedProp; //works for ts 4.4.4 and newer
}

🙁 Actual behavior
Error Type'string | null' is not assignable to type 'string' on line 5
🙂 Expected behavior
No error, as type should be narrowed by noNestedProp check.
Additional information about the issue
No response

It doesn’t work because the properties being checked aren’t readonly, which is one of the requirements for indirected narrowing.

const obj: {readonly nestedObj: {readonly nestedProp: string | null}} = {nestedObj: {nestedProp: "test"}};

Yeah, after after adding readonly, it works good. Thanks for clarifying.


Support multiple directives

Removes the string package
Changes Reply.js to support multiple directives
Backwards compatible support for the old audio directive style
Update the docs to show the correct directive usage style

Coverage increased (+0.6%) to 98.276% when pulling 7b3d5aef8b8f77bfb2d13197868c360bbe038ed5 on heneryville:multi-directive-support into 3f5a27d76777fb38fa00fa4eab9f43fcd40d654f on mediarain:dev.

Coverage increased (+0.6%) to 98.346% when pulling 134eee02dd579f7882b8ab539bcb60ce7bd74787 on heneryville:multi-directive-support into db1b62cdb577546ccaf23426d5900decdc75e5a6 on mediarain:dev.

Coverage increased (+0.6%) to 98.349% when pulling a877a018c63cf58dac93c1cea9c28ce237940bdc on heneryville:multi-directive-support into db1b62cdb577546ccaf23426d5900decdc75e5a6 on mediarain:dev.


feat: add FSST string compression(draft, wip)

#2415
This is a naive implementation of the FSST, it is fairly tested so that it can be used in correctness proof for future optimization, and a mark for future performance engineering.
this is not complete, but enough for one git commit, it's just, I am going through something and a bit debilitated, I hope pushing a PR and make the progress live may motivate me more.
Question 1: why don't use the original C++ implementation in the paper and do some Rust/C++ FFI?
the original C++ implementation is not tuned for our workload
the original C++ implementation doesn't work directly with arrow string_array interface
the original C++ implementation, IMO, has many flaws
Question 2: how to use this FSST implementation?
this is the designed interface,
pub fn fsst_compress(_input_buf: &[u8], _input_offsets_buf: &[i32], _output_buf: &mut Vec<u8>, _output_offsets_buf: &mut Vec<i32>) -> Result<()> {
    Ok(())
}

pub fn fsst_decompress(_input_buf: &[u8], _input_offsets_buf: &[i32], _output_buf: &mut Vec<u8>, _output_offsets_buf: &mut Vec<i32>) -> Result<()> {
    Ok(())
}

TODO:
1: more comments and documentations
2: benchmark this implementation(how can I get test data for our intended workload?)
3: push string encoding from lance logical encoding to lance physical encoding
4: lance string compression trait in lance physical encoding
5: implemente FSST for lance string compression trait
6: implemente SIMD optimization in the paper
7: maybe we could make FSST as a separate rust crate
8: predicate push down to FSST decompression
a few questions:
1: does lance intended to support big-endian machine?
2: what kind of string dataset is preferable for our use?

This is very cool!
I'll look through this in more detail tomorrow morning on Monday morning.

this is not complete, but enough for one git commit, it's just, I am going through something and a bit debilitated, I hope pushing a PR and make the progress live may motivate me more.

I think this PR is plenty for a single commit.  We don't need to tackle it all in one go, having just the compression routines is a good start.  We can help out with some of the integration into the larger system too.

Question 2: how to use this FSST implementation?
this is the designed interface,

Is there no need for a symbol table?  Or is the symbol table somehow encoded into the output buffer?

2: benchmark this implementation(how can I get test data for our intended workload?)

I think a compelling test set would be one of the training datasets used to create ML semantic search models.  For example, the MS Marco dataset is very large (probably just need a subset of it for benchmarking): https://msmarco.z22.web.core.windows.net/msmarcoranking/fulldocs.tsv.gz
These are English language strings that I think would be a good candidate for FSST.

3: push string encoding from lance logical encoding to lance physical encoding

There is an effort in progress to do this because we needed it for dictionary encoding too.  I agree it is needed: https://github.com/lancedb/lance/pull/2426

7: maybe we could make FSST as a separate rust crate

Sure, we'd be happy to host it on our org (lancedb) or use any other you have in mind.

8: predicate push down to FSST decompression

Predicate pushdown is partly wired up at the logical level now but I haven't really thought of a good way to push it down into the physical level.  One challenge is that I've been trying to avoid making datafusion a dependency of lance-file (it is large and the project benefits from fast build / debug cycles) so at the moment expressions, from the lance-file perspective, are just substrait encoded Bytes.  Some kind of lightweight typed expression library would be useful here.

Is there no need for a symbol table? Or is the symbol table somehow encoded into the output buffer?

the symbol table needs to be serialized/deserialized to the output buffer during compression/decompression, which I haven't implemented and forgot to mention, sorry for the confusion.

Sure, we'd be happy to host it on our org (lancedb) or use any other you have in mind.

I will be so honored if it can be hosted on lancedb! I have a fair amount of work to do in optimization though.

Predicate pushdown is partly wired up at the logical level now but I haven't really thought of a good way to push it down into the physical level. One challenge is that I've been trying to avoid making datafusion a dependency of lance-file (it is large and the project benefits from fast build / debug cycles) so at the moment expressions, from the lance-file perspective, are just substrait encoded Bytes. Some kind of lightweight typed expression library would be useful here.

I can only find duckDB has fsst compression, they use the original implementation in the paper and doesn't support predicate pushdown.
this might be a opportunity for us to get some state-of-art string processing benchmark results but I am not sure how good the result would be, if the benchmark improvement is not significant, we might be better off to not have this predicate push down

> Ok, took a closer look. This looks really good so far.

thank you so much for looking into these such primitive code, I feel pumped with these discussions and feedbacks!
the current code is more intended to set up the testing and benchmark framework so I can experiment with the algorithm, I expect the final version be very different

benchmark results track:
ebc8a4ef44bc4133fee4e3169a0f32bd53fa4d12

machine info:
Linux 192 5.10.0-28-amd64 #1 SMP Debian 5.10.209-2 (2024-01-31) x86_64 GNU/Linux

a few more questions @westonpace:
I am playing with the algorithm with the input buffer size 16MB, assuming we spawn a encoding task when we have 16MB arrow string value data(not counting the offsets), what's the appropriate size here?
I have a x86 CPU with AVX512, and a macbook pro 2023 with Apple M2 Pro, I don't have access to other vector extension instruction sets. I think we should prioritize AVX512?
I couldn't get the compression ratio up to 1.6 for the third column of MS Marco, this is expected and in align with the original paper results, FSST achieves better compression ratio on small similar data sets(3.6 on TPCH c_name column for 9e1f0844ef82b1666553c5977f0df162bf691f38), is this good enough or should I invest more time on this?

I am playing with the algorithm with the input buffer size 16MB, assuming we spawn a encoding task when we have 16MB arrow string value data(not counting the offsets), what's the appropriate size here?

The default right now is 8MiB so not far off.  It's configurable and will often be some value over 8MiB (e.g. if we have 7MB of data and a new array comes in with 2MB we will just make a 9MB page).

I have a x86 CPU with AVX512, and a macbook pro 2023 with Apple M2 Pro, I don't have access to other vector extension instruction sets. I think we should prioritize AVX512?

That's up to you as this is your contribution.  I think it's fine to prioritize AVX512 as long as we panic if it isn't present (we have flags to do runtime checks).  Eventually, we will probably want at least an AVX2 version as a fallback (I'm pretty sure this is our minimum supported) if we want to make this encoding a default and not opt-in.

is this good enough or should I invest more time on this(I am afraid the improvement would be marginal)?

I think it's fine.  No need to tackle everything all at once.  We can investigate other encodings for columns like that in the future.

The default right now is 8MiB so not far off. It's configurable and will often be some value over 8MiB (e.g. if we have 7MB of data and a new array comes in with 2MB we will just make a 9MB page).

just to make sure, when we spawn a encoding task with 8MiB input, the output page is much less, I assume this kind of segmentation is handled by our code?

just to make sure, when we spawn a encoding task with 8MiB input in memory, the output buffer in memory is much less, I assume this kind of segmentation is handled by our code?

At the moment, we always write the encoded buffers to a page, even if the output buffers are much smaller.  In the future we can add code to accumulate output buffers until 8MiB of encoded data has been built up.  The only tricky part is that, once we have chosen an encoding tree, we need to make sure we keep using that encoding tree until we flush a page.
We haven't needed it yet since we don't have any compressive encodings.
I think it's probably fine for now and we can optimize later.

FYI, the string encoder has moved from a field encoder to an array encoder.  This should make it considerably easier to integrate.  I will make an integration PR to allow testing FSST end-to-end soon.

FYI, the string encoder has moved from a field encoder to an array encoder. This should make it considerably easier to integrate. I will make an integration PR to allow testing FSST end-to-end soon.

Thank you so much!
excited to see the integration!
excited to see more and more encoding go into lance!

This turned out to be pretty straightforward to integrate.  Here's a commit that integrates FSST: https://github.com/lancedb/lance/commit/71ac855c232cdf952cde4661d54e515d6978901d
With FSST integrated the unit tests passed (:tada:) and I was able to encode the TPC-H lineitems table.  However, I'm currently getting a panic on the decode.  I can investigate in more detail soon (wanted to get the integration done today and didn't dig too deep into FSST code).
That being said, we should get this PR merged, as is, and then figure out the TPC-H panic and do the integration in a follow-up PR.  Working in small chunks will be helpful.  So, priority:

There are some tests that didn't compile (missing symbol from slice I think).  Let's add that method back in or remove those tests.
Please clean up the commented blocks (feel free to remove commented out tests that require setup and we can add them back in later).
Please address any warnings and CI failures

Once that is done we can merge this PR.
A few things popped up during the integration that we can then address in a follow-up:

Lance encoders receive the arrays to encode as a slice of arrays.  This is because the column encoder collects arrays until it has enough data to justify encoding.  Currently the fsst::fsst::compress function expects a single array.  This means I need to concatenate the input arrays before calling fsst::fsst::compress which I'd like to avoid.  I think this just means adding a top-level compress_many.
It would be very helpful if we can save the symbol table in a separate buffer and read the symbol table from a separate buffer instead of inserting it into the beginning of the bytes array.  This is because the symbol table is going to get stored somewhere else (the metadata) and I end up needing to do some extra copies to work around this.
The fsst::fsst::compress routine panics if given very small inputs (e.g. not enough space for the symbol table).  I don't think this is really a problem because we should probably have a size limit for using FSST anyways (and that is what I ended up doing to avoid this).
I need to change the core encoding traits to work with Vec<u8> instead of BytesMut I think.  This isn't just an FSST issue, I have encountered this issue elsewhere.  It's difficult to create a new BytesMut from existing data.  This introduced a few copies on the decode side.

cool!
I will start doing it right away!

It would be very helpful if we can save the symbol table in a separate buffer and read the symbol table from a separate buffer instead of inserting it into the beginning of the bytes array.  This is because the symbol table is going to get stored somewhere else (the metadata) and I end up needing to do some extra copies to work around this.

This is open to discussion, I don't think we need to store the symbol table somewhere else as metadata, the FSST symbol table itself doesn't offer any filter capability or statistics of a single string attribute.

The fsst::fsst::compress routine panics if given very small inputs (e.g. not enough space for the symbol table). I don't think this is really a problem because we should probably have a size limit for using FSST anyways (and that is what I ended up doing to avoid this).

the current code has this logic of store string as is when input is small, I set it to 0 during development

Codecov Report
Attention: Patch coverage is 87.89474% with 92 lines in your changes missing coverage. Please review.

Project coverage is 79.88%. Comparing base (5ab70c2) to head (9095eaa).
Report is 72 commits behind head on main.

Files
Patch %
Lines

...t/lance-encoding/compression-algo/fsst/src/fsst.rs
87.89%
82 Missing and 10 partials :warning:

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2470      +/-   ##
==========================================
+ Coverage   79.75%   79.88%   +0.12%
==========================================
  Files         205      208       +3
  Lines       56573    60350    +3777
  Branches    56573    60350    +3777
==========================================
+ Hits        45121    48211    +3090
- Misses       8770     9344     +574
- Partials     2682     2795     +113

Flag
Coverage Δ

unittests
79.88% <87.89%> (+0.12%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

This is open to discussion, I don't think we need to store the symbol table somewhere else as metadata, the FSST symbol table itself doesn't offer any filter capability or statistics of a single string in this column, it is more about substrings in this column segment.

Fair, but it is small, and if we can cache it then that is one less bit of data we need to read when searching.  LanceDB does vector search and some of our customers are bound by "max 5500 read requests per file per second".  However, we can cache it even if stored in bytes buffer I suppose, that will help keep the metadata small.

With FSST integrated the unit tests passed (:tada:) and I was able to encode the TPC-H lineitems table (and saw some compression). However, I'm currently getting a panic on the decode. I can investigate in more detail soon (wanted to get the integration done today and didn't dig too deep into FSST code).

I actually don't know how to do end-to-end test for lance, could you give some reference so I can also investigate on this? @westonpace

the current decompression panic issue does need more discussion:
there is an asymmetry in our ArrayEncoder/PageDecoder, for one output buffer that a encoding algorithm produces, we may spawn many decoding tasks, and these decoding task all tries to interpret decode meta data from these decode input buffers

there is an asymmetry in our ArrayEncoder/PageDecoder, for one output buffer that an encoding algorithm produces, we may spawn many decoding tasks, and these decoding tasks all tries to interpret decode meta data from these decode input buffers

Yes, this is correct.  We slice the buffers into batches (I normally use 8Ki and can't remember if the default is 1Ki or 8Ki).  We slice the offsets buffer into equal sized chunks and then grab the corresponding byte ranges.  Then we create a decode task for each one.  Since each chunk receives "whole strings only" (e.g. a string is not split across chunks) I think this works ok yes?  I don't remember if I cloned the symbol table for each decode task or not but we can easily wrap it in an Arc to prevent deep clones (decode should only need read-only access to symbol table)

Since each chunk receives "whole strings only" (e.g. a string is not split across chunks) I think this works ok yes?

the current fsst::decompress is designed in a way that assumes the input is a wholly compressed block and tries to interpret symbol table from the beginning of the buffer, I should think more about this

I actually don't know how to do end-to-end test for lance, could you give some reference so I can also investigate in this? @westonpace

If you merge my PR as it is applied you can install the python bindings by running the following.  maturin develop will install the wheel into your current conda / venv in editable mode.
cd python
maturin develop --release

The following python script is what I was using for testing:
from datetime import datetime
import glob
import os

import pyarrow as pa
import pyarrow.parquet as pq
import pyarrow.dataset as ds

import lance
from lance.file import LanceFileReader, LanceFileWriter

from lance.tracing import trace_to_chrome

trace_to_chrome(level="trace")

PATH = "/home/pace/dev/data/lineitem_10.parquet"

arr_dataset = ds.dataset(PATH)

schema = ds.dataset(PATH).schema

# for file in glob.glob("/home/pace/dev/data/lineitem_10.parquet.*"):
#     pq_size = os.path.getsize(file)
#     pq_size_gb = pq_size / 1024.0 / 1024.0 / 1024.0
#     print(f"Parquet file size: {pq_size_gb} GiB")

#     pq_metadata = pq.ParquetFile(file).metadata
#     num_row_groups = pq_metadata.num_row_groups
#     print(f"Parquet file num row groups: {num_row_groups}")

#     start = datetime.now()
#     arr_dataset = ds.dataset(PATH)
#     arr_dataset.to_table()
#     end = datetime.now()

#     duration = (end - start).total_seconds()
#     print(f"Parquet (pyarrow) Read Time (lineitem_10): {duration}s")

if not os.path.exists("/tmp/tpch.lancev2"):
    start = datetime.now()
    arr_dataset = ds.dataset(PATH)
    with LanceFileWriter("/tmp/tpch.lancev2", arr_dataset.schema) as writer:
        for batch in arr_dataset.to_batches():
            writer.write_batch(batch)
    end = datetime.now()
    duration = (end - start).total_seconds()
    print(f"V2 Write Time (lineitem_10): {duration}s")

lv2_size = os.path.getsize("/tmp/tpch.lancev2")
lv2_size_gb = lv2_size / 1024.0 / 1024.0 / 1024.0
print(f"Lance v2 file size: {lv2_size_gb} GiB")

start = datetime.now()
reader = LanceFileReader("/tmp/tpch.lancev2")
for batch in reader.read_all(batch_size=8 * 1024).to_batches():
    pass
end = datetime.now()
print(f"V2 Read Time (lineitem_10): {(end - start).total_seconds()}s")

The lineitem_10.parquet file can be downloaded here.

I think we can at least think about the alternatives: instead of spawn decoding task based on row number batches, what else can we do?

This is how query engines operate.  They stream small CPU-cache sized batches of data through the system.  Any other approach generates expensive cache<->CPU copies which defeats any potential SIMD / compression win.

the thing is, we need to tell fsst which row number starts a new compressed block and instruct fsst to interpret symbol table at that buffer, this can get nasty,
we can have fsst output 3 buffers, one for compressed bytes, one for offsets, one for symbol table, but we still need to tell the PageDecoder which row number begins a new compression unit and needs to fetch new symbol table

Decode batches will:

Never start in the middle of a string
Never span multiple pages / chunks

Let's assume we are decoding and we have 6MB of compressed data.  We have 5001 offsets and so we have 5000 rows.  First, we decode the offsets.  Then we split the offsets into 5 chunks of 1000 rows.  Maybe the first chunk is 2MB of data, the second one is 512KB, and the next 2 are 1MB and the last one is 1.5MB.  This is all fine.  No string is split between chunks.  We should be able to decode each of them in parallel independently I think (they will all need read access to the symbol table but none of them modify it)

I changed compression/decompression's interface to accept symbol table and merged @westonpace's integration code, but there are still many bugs
it might be a trouble for us when dealing with page fragmentation after encoding, we only have one page encoding info per page, also in page bridging
putting parallelism inside one decode task may mess our structure, I am a bit hesitate about this, another reason is that we may be able to achieve good compression ratio while keeping the decompression very very lightweight, by using only lightweight algorithms

end-to-end test result using ms marco:

there is a offset overflow bug when using full ms marco dataset(22 GiB)
the batch seems not scheduled in a column major manner but a row major manner
to reproduce, run
from datetime import datetime
import os
import pyarrow.csv as csv
import pyarrow as pa

import lance
from lance.file import LanceFileReader, LanceFileWriter

from lance.tracing import trace_to_chrome

# Path to the TSV file
file_path = "/home/x/fulldocs_500_k.tsv"

# Read the TSV file into a PyArrow table
table = csv.read_csv(file_path, read_options=csv.ReadOptions(column_names=['column1', 'column2', 'column3'], block_size = 1024 * 1024 * 1024), parse_options=csv.ParseOptions(delimiter='\t'))

with LanceFileWriter("/tmp/fulldocs.lancev2", table.schema) as writer:
    writer.write_batch(table)

original_size = os.path.getsize(file_path)
original_size_mb = original_size / 1024.0 / 1024.0
print(f"original file size: {original_size_mb} MiB")

lv2_size = os.path.getsize("/tmp/fulldocs.lancev2")
lv2_size_mb = lv2_size / 1024.0 / 1024.0
print(f"Lance v2 file size: {lv2_size_mb} MiB")

print(f"compression ratio: {original_size / lv2_size}")
start = datetime.now()

reader = LanceFileReader("/tmp/fulldocs.lancev2")
table_read = reader.read_all().to_table()

assert table.equals(table_read), "The original table and the table read back from the Lance file are not identical."

in b06b455aae1b7805f1de389f6868ec7e2887efd6

in cases like ms marco, the third column is way larger(more than 100x) than the other two column, it can be a problem if do_flush only produces one encoding task

@westonpace thank you so much for the review!
I checked out #2602  and we can see how to tackle under #2602

@broccoliSpicy I picked a random parquet file from https://huggingface.co/datasets/HuggingFaceFW/fineweb/tree/main/data
Then I converted the data with (I ran this script twice, once with FSST and once without, and saved the files to two different paths):
from lance.file import LanceFileReader, LanceFileWriter
import pyarrow.parquet as pq

data = pq.read_table("data.parquet")
with LanceFileWriter("/home/pace/dev/lance-experiments/fineweb/output.lance") as writer:
  writer.write_batch(data)

Then I queried it with:
import datetime
import pyarrow.parquet as pq
from lance.file import LanceFileReader

start = datetime.datetime.now()
tab = pq.read_table("data.parquet")
end = datetime.datetime.now()
elapsed = (end - start).total_seconds()
print(f"Parquet elapsed: {elapsed}s")

start = datetime.datetime.now()
tab2 = LanceFileReader("/home/pace/dev/lance-experiments/fineweb/data.lance").read_all().to_table()
end = datetime.datetime.now()
elapsed = (end - start).total_seconds()
print(f"Lv2(no-fsst) elapsed: {elapsed}s")

start = datetime.datetime.now()
tab2 = LanceFileReader("/home/pace/dev/lance-experiments/fineweb/data_fsst.lance").read_all().to_table()
end = datetime.datetime.now()
elapsed = (end - start).total_seconds()
print(f"Lv2(fsst) elapsed: {elapsed}s")

assert tab == tab2

print("Tables are equal")

@westonpace thanks! I will look into this tomorrow. the fsst decoder can be trivial accelerated to 1 byte per 2 - 3 CPU cycle, and I can try put the original size in symbol table.


Display link to Github Repo Draft release on console

Rationale:  Enable users to quickly navigate there and turn the draft into a final
Credits to @redcatbear

The last release I did generated the following output:
2022-08-02 17:22:52.451 [INFO   ] A GitHub release was created at: https://github.com/exasol/bucketfs-document-files-virtual-schema/releases/tag/0.5.2
2022-08-02 17:22:52.674 [INFO   ] Release on platform GITHUB is finished!
2022-08-02 17:22:52.676 [INFO   ] Validation started.
2022-08-02 17:22:52.912 [INFO   ] Releasing on JIRA platform.
2022-08-02 17:22:56.810 [INFO   ] A Jira ticket was created at: https://www.exasol.com/support/browse/EXACOMM-671
2022-08-02 17:22:57.065 [INFO   ] Release on platform JIRA is finished!
2022-08-02 17:22:57.066 [INFO   ] Removing all artifacts from 'exasol/bucketfs-document-files-virtual-schema' repository.
2022-08-02 17:22:57.831 [INFO   ] Validation was performed without any problems!
2022-08-02 17:22:57.832 [INFO   ] Release was performed without any problems!

@redcatbear: Could you please check, if this meets your requirement?

Is not suitable as a showcase, since the shown release bucketfs-document-files-virtual-schema/releases/tag/0.5.2 has already been converted from draft to final.


Add MSBuild_Logs/ to ignore list

This is a new directory which msbuild creates to store its crash dumps (ref: https://github.com/dotnet/msbuild/commit/cdb5077c451180ab38161e0b5e70f5448e70355b).

Thanks for the PR, but you failed to follow the template provided. See the project's README for help.

You could have just asked rather than closing the PR for not following template is bizarre.


fork error

🐛 Bug Report for zkSync Era In-Memory Node
📝 Description
fork cause error when i'm trying to fork from other network like https://rpc.ankr.com/zksync_era
🔄 Reproduction Steps

era_Test_node fork https://rpc.ankr.com/zksync_era

🤔 Expected Behavior
😯 Current Behavior
thread 'main' panicked at src/fork.rs:391:14:
Unable to create a client for fork: Transport(Invalid Url: Port number is missing in the URL)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

🖥 Environment
era_test_node 0.1.0-alpha.7

Hi @Ivory433 , something like the following should work:
era_test_node fork https://rpc.ankr.com:443/zksync_era

If you run into an error, try specifying a specific block you want to fork from with --fork-at 14000000


Depends on old version of importlib_resources?

Hi folks,
we're using this theme in the mamba-org and I just noted that an old sphinx version gets installed on a M1 Apple computer.
The reason is that the importlib_resources dependency is quite constrained (and not available in that low version) on macos-arm64 I believe.
Is it necessary?
Referrring to https://github.com/conda-forge/sphinx-book-theme-feedstock/blob/7cc49d3bda035668118f986030efd919a541909f/recipe/meta.yaml#L29

Current version is 5.x

yeah, that pin is taken directly from upstream... i guess we could patch it etc...
https://github.com/executablebooks/sphinx-book-theme/blob/master/setup.py#L34
I guess my feeling is fixing/discussing it upstream would be preferable...


Dont get traffic from my pods

Describe the bug
My cluster have different name than cluster.local and I cant find in mizu docs a way to set the correct cluster name. This is the log from one of the tapper daemons:
[2022-04-13T06:41:33.986+0000] INFO  ▶ socket connection to ws://mizu-api-server.mizu.svc.cluster.local/wsTapper failed: dial tcp: i/o timeout, retrying 28 out of 30 in 2 seconds... ▶ [1 main.go:323 dialSocketWithRetry]

If the cluster name is not properly set than the generated url ws://mizu-api-server.mizu.svc.cluster.local/wsTappe is not valid and the connection cannot be made.
Expected behavior
There should e option to mizu like --cluster-name that sets the correct cluster name.
Please add --cluster-name or similar option to mizu. If there is another way to set this name please put it in the help docs.

@Lamercho , thanks for taking the time to submit the issue.
I think that we can simply remove the .svc.cluster.local suffix. k8s will be able to resolve by <serviceName>.<namespace>. We'll get to it in the next few days.
If you're up to it, you are welcome to make the change yourself. You can check of wiki for project documentation, or you can reach out on slack for any question we will be glad to support you.

Thank you, just submitted pull request.


Comparing Android and Apple screens

Anant asked me too share 3 sets of screens chosen randomly for Summi to check Structural issues.
Set 1:

Duplicate issue created by mistake.