Further docker support

Hi
I've been trying to create a home assistant addon for fermentrack, building on from the docker image @mp-se has created.
I'm close to getting it working, but one of the issues I have is around persistence. I've seen the PR that detects if fermentrack is running in docker and then suffixes the base path with /db, but I wonder if we could take this a step further still...
Basically with home assistant or more specifically hassio addons, they do a lot of plumbing for you and the convention is that you get a bind-mount in your addons container for you to persist data back to the docker host at "/data". Sadly this is a bind-mount and not a volume, I spent hours not realising this and trying to get a symbolic link to work between it and the various persistence folders for fermentrack:
/home/fermentrack/fermentrack/data
/home/fermentrack/fermentrack/db
/home/fermentrack/fermentrack/log
I'll start by saying python isn't my strongest point, but I wondered what effort would be involved and if it's even possible to have it so we can specify these folders as part of the installation? That way in a docker container or even on a server, it would be possible to decide where you want these folders creating.
Thanks in advance for your help, and I really enjoy using fermentrack :)
Andy

Hey there!
What exactly does the addon do? Run Fermentrack, or interface with an external installation of Fermentrack?

Hey there!
What exactly does the addon do? Run Fermentrack, or interface with an external installation of Fermentrack?

Hi
An add-on is just what home assistant calls something that runs in a docker container, so consider add-on a container with some convenience around it's configuration and management. Basically home assistant manages the host server and also has a hypervisor to manage docker. This allows you to run other applications within docker and manage them through home assistant.
This gives various benefits such as a nice add-on store interface when you can install, update and remove add-ons. It also has a few conventions built in to cover things like providing a bind mount for your container to persist data, ways of providing configuration for your add-on through the home assistant UI and also what they call  ingress.
Ingress does some routing to proxy from the home assistant UI to the UI running in your add-on. It creates a proxy and automatically following some convention maps the a port to the add-on allowing it to not only render the add-on UI within home assistant, but also to automatically provide a menu link to the add-on.
What I've managed to do so far is take the dockerfile @mp-se has running fermentrack all in one container. Then I've tweaked it a little to fit with some of the home assistant add-on requirements.
Currently I can install the add-on and fermentrack initializes correctly, but I have an issue persisting the data. HA add-ons get a bind mount of /data but I spent hours trying to get a symbolic link working to the directories like /home/fermentrack/fermentrack/db mapped to /data/fermentrack/db and so on, but sadly bind mounts don't support symbolic links.
I've forked the fermentrack Dev branch and I've started by updating the settings.py to try to read some environment variables for data and db directories so if you specify them it uses them.
However I'm not sure if that would be sufficient and also I started looking at the log directory but that seems to define the log path where it is used. I may be wrong I'm not particularly experienced with python other than the odd lambda function.
I'm happy to put a PR in but I will definitely need your input around the issues above.
Thanks again
Andy

Hi
An add-on is just what home assistant calls something that runs in a docker container, so consider add-on a container with some convenience around it's configuration and management. Basically home assistant manages the host server and also has a hypervisor to manage docker. This allows you to run other applications within docker and manage them through home assistant.
This gives various benefits such as a nice add-on store interface when you can install, update and remove add-ons. It also has a few conventions built in to cover things like providing a bind mount for your container to persist data, ways of providing configuration for your add-on through the home assistant UI and also what they call  ingress.
Ingress does some routing to proxy from the home assistant UI to the UI running in your add-on. It creates a proxy and automatically following some convention maps the a port to the add-on allowing it to not only render the add-on UI within home assistant, but also to automatically provide a menu link to the add-on.
What I've managed to do so far is take the dockerfile @mp-se has running fermentrack all in one container. Then I've tweaked it a little to fit with some of the home assistant add-on requirements.
Currently I can install the add-on and fermentrack initializes correctly, but I have an issue persisting the data. HA add-ons get a bind mount of /data but I spent hours trying to get a symbolic link working to the directories like /home/fermentrack/fermentrack/db mapped to /data/fermentrack/db and so on, but sadly bind mounts don't support symbolic links.
I've forked the fermentrack Dev branch and I've started by updating the settings.py to try to read some environment variables for data and db directories so if you specify them it uses them.
However I'm not sure if that would be sufficient and also I started looking at the log directory but that seems to define the log path where it is used. I may be wrong I'm not particularly experienced with python other than the odd lambda function.
I'm happy to put a PR in but I will definitely need your input around the issues above.
Thanks again
Andy

Do you have any documentation on the "bind mounts" you're referring to in the context of hassio?
Additionally, do you have your dockerfile available anywhere (GitHub or otherwise) I could see it?

Do you have any documentation on the "bind mounts" you're referring to in the context of hassio?
Additionally, do you have your dockerfile available anywhere (GitHub or otherwise) I could see it?

Absolutely...
So the HA documentation is a bit light on addon development but here is a link
Bind mounts are configurable for various things such as config, backup, addons, ssl, etc there is a little in the above HA docs, but by default each addon gets a read write bind-mount of /data. Here is a much better description, but the key thing is they aren't as flexible as volumes.
My attempt so far is here
Currently the only way I can access it by setting a port mapping to the exposed port of the my addon (which you can do in home assistant) and then I can use my home assistant host with that port to access it directly. I.e. <IP_ADDRESS>:8080
This is to do with the way nginx is configured in the addons I've been told in the past, but not really had much elaboration how to get it working. There are loads of examples of things with working ingress allowing HA to render the addon ui in HA. This is for running node-red as an addon in home assistant, but sadly I have very little experience with nginx (.net developer by day) and I don't really know where to begin.
That node-red example seems a very common structure for nginx in a lot of docker containers as well as home assistant addons, but sadly I really don't understand it much.

Absolutely...
So the HA documentation is a bit light on addon development but here is a link
Bind mounts are configurable for various things such as config, backup, addons, ssl, etc there is a little in the above HA docs, but by default each addon gets a read write bind-mount of /data. Here is a much better description, but the key thing is they aren't as flexible as volumes.
My attempt so far is here
Currently the only way I can access it by setting a port mapping to the exposed port of the my addon (which you can do in home assistant) and then I can use my home assistant host with that port to access it directly. I.e. <IP_ADDRESS>:8080
This is to do with the way nginx is configured in the addons I've been told in the past, but not really had much elaboration how to get it working. There are loads of examples of things with working ingress allowing HA to render the addon ui in HA. This is for running node-red as an addon in home assistant, but sadly I have very little experience with nginx (.net developer by day) and I don't really know where to begin.
That node-red example seems a very common structure for nginx in a lot of docker containers as well as home assistant addons, but sadly I really don't understand it much.

You should be able to symlink everything to a directory in /data and have it work. What was (or wasn't) happening when you tried that?

You should be able to symlink everything to a directory in /data and have it work. What was (or wasn't) happening when you tried that?

Basically as it is a bind-mount, it was creating the symlink on the host from the container as the container has full control over the directory. So within the container all looked well and I could see my files linking from the fermentrack directories to my data directories.
However on the host, there is just 3 broken symlinks. This is because by creating the symlinks from the container, it essentially runs the same command on the host. So it was trying to link to 3 fermentrack directories on the host that don't exist as they only exist in the container.

Basically as it is a bind-mount, it was creating the symlink on the host from the container as the container has full control over the directory. So within the container all looked well and I could see my files linking from the fermentrack directories to my data directories.
However on the host, there is just 3 broken symlinks. This is because by creating the symlinks from the container, it essentially runs the same command on the host. So it was trying to link to 3 fermentrack directories on the host that don't exist as they only exist in the container.

I started playing with this idea: https://github.com/mcinnes01/fermentrack/commit/34fc2b7736085e11eb87920f4679ba1c063785e2

I started playing with this idea: https://github.com/mcinnes01/fermentrack/commit/34fc2b7736085e11eb87920f4679ba1c063785e2

Unfortunately, I'm not keen on adding configuration options for any additional folders if possible.
The logs aren't important for persistence - and honestly, I'd consider wiping them on container creation to be advantageous. Prevents old issues from showing up on a fresh version of the code. ;)
I can't find any documentation where the "magic" folders such as /data are specified in the hassio documentation - and it looks like there may be a hostconfig.json file where bind options can be specified.
Have you tried specifying binds that are more applicable to Fermentrack (or - alternatively - seeing if hostconfig.json will accept a Volumes option as I'm guessing that this configures portainer somehow?)

Unfortunately, I'm not keen on adding configuration options for any additional folders if possible.
The logs aren't important for persistence - and honestly, I'd consider wiping them on container creation to be advantageous. Prevents old issues from showing up on a fresh version of the code. ;)
I can't find any documentation where the "magic" folders such as /data are specified in the hassio documentation - and it looks like there may be a hostconfig.json file where bind options can be specified.
Have you tried specifying binds that are more applicable to Fermentrack (or - alternatively - seeing if hostconfig.json will accept a Volumes option as I'm guessing that this configures portainer somehow?)

That hostconfig.json is sadly someone just hacking around home assistants supervisor, these changes could be lost by an upgrade or restoring a backup etc or worse upset the supervisor. I came across a similar issue when trying to mount an external drive to a plex addon.
The binds you get for addons are all dictated by convention in the config.json and sadly they don't give you enough rope to hang yourself by specifying custom mounts.
Each addon essentially get's it's own sandboxed folder in /usr/share/hassio/addons/data/my-addon and there are others like the general config folder in home assistant you can grant access to and say with its read-only or read-write. But again all of these are bind-mounts.
Could you see any problem of having the fermentrack data and db folders specified somewhere outside of:
home/fermentrack/fermentrack? Also for maybe raspberry pi users would it not be useful to be able to specify a usb drive mount or network drive in case the sd card dies?

That hostconfig.json is sadly someone just hacking around home assistants supervisor, these changes could be lost by an upgrade or restoring a backup etc or worse upset the supervisor. I came across a similar issue when trying to mount an external drive to a plex addon.
The binds you get for addons are all dictated by convention in the config.json and sadly they don't give you enough rope to hang yourself by specifying custom mounts.
Each addon essentially get's it's own sandboxed folder in /usr/share/hassio/addons/data/my-addon and there are others like the general config folder in home assistant you can grant access to and say with its read-only or read-write. But again all of these are bind-mounts.
Could you see any problem of having the fermentrack data and db folders specified somewhere outside of:
home/fermentrack/fermentrack? Also for maybe raspberry pi users would it not be useful to be able to specify a usb drive mount or network drive in case the sd card dies?

Well, that's annoying. The folders should always be a subfolder of the main Fermentrack directory. It might work with a different configuration - but it's not one that I recommend/would support.
I'm sure there's some reason not to do this, but you could always change the Fermentrack user's home directory to something like /data/fermentrack. You'll end up persisting almost everything (including the codebase) - but it should work, and it would keep the data in a subfolder of the main Fermentrack directory.
To your question - if an RPi user installing onto bare metal wanted to specify a network drive/other device to save their data, I would tell them to mount the share/device at the ~fermentrack/data directory. ;)
(Or, honestly, I would tell them to set up automated backups to copy their data to their backup device. Installing Fermentrack on another device leads to weird issues when the drive accidentally gets disconnected)

Well, that's annoying. The folders should always be a subfolder of the main Fermentrack directory. It might work with a different configuration - but it's not one that I recommend/would support.
I'm sure there's some reason not to do this, but you could always change the Fermentrack user's home directory to something like /data/fermentrack. You'll end up persisting almost everything (including the codebase) - but it should work, and it would keep the data in a subfolder of the main Fermentrack directory.
To your question - if an RPi user installing onto bare metal wanted to specify a network drive/other device to save their data, I would tell them to mount the share/device at the ~fermentrack/data directory. ;)
(Or, honestly, I would tell them to set up automated backups to copy their data to their backup device. Installing Fermentrack on another device leads to weird issues when the drive accidentally gets disconnected)

The simple approach would probably be to install fermentrack into the /data dir to avoid changes to fermentrack, then you will have everything stored in one place and even git upgrades should work. You could probably modify my docker image and point the home directory towards /data instead of /home/fermentrack to make that work with minimal changes.
But if you cannot control what is mapped to your container you will not be able to utilize the firmware flashing functions within fermentrack since that requires that the /dev file system is mounted into the container. Just to be aware of the limitations that this approach will have.

The simple approach would probably be to install fermentrack into the /data dir to avoid changes to fermentrack, then you will have everything stored in one place and even git upgrades should work. You could probably modify my docker image and point the home directory towards /data instead of /home/fermentrack to make that work with minimal changes.
But if you cannot control what is mapped to your container you will not be able to utilize the firmware flashing functions within fermentrack since that requires that the /dev file system is mounted into the container. Just to be aware of the limitations that this approach will have.

Thanks, I had a similar thought about setting the root folder to data but also had that same feeling that it would mean the application code would get written to home assistant too.
I did see the details on configuring docker to allow the usb flashing to work by mounting dev. I think I have this taken care of, but it doesn't supply dev on it's own, you get: /dev/bus/usb
And then it also provides all the devices like so:

This is working for other addons such as esp-home which I also use to flash devices.

Thanks, I had a similar thought about setting the root folder to data but also had that same feeling that it would mean the application code would get written to home assistant too.
I did see the details on configuring docker to allow the usb flashing to work by mounting dev. I think I have this taken care of, but it doesn't supply dev on it's own, you get: /dev/bus/usb
And then it also provides all the devices like so:

This is working for other addons such as esp-home which I also use to flash devices.

The fermentrack code/script work directly with /dev so you will need to change that part of the  code that work with devices for this to work. Would apply to both serial brewpi integration and firmware flashing.

The fermentrack code/script work directly with /dev so you will need to change that part of the  code that work with devices for this to work. Would apply to both serial brewpi integration and firmware flashing.

Ok cool can you point me in the right direction of that?
I'm just trying a few things in regards to moving the data and db directories out of the fermentrack route on my fork, just for curiosities sake. This might seem a silly question, but if I update the settings.py file do I need to run some build or packaging or will that suffice?
And in my docker file I do a git clone to get fermentrack I've update this to my fork, but the commit output by fermentrack when it initializes is your last dev branch commit not mine. Is there something else I need to do to get it to pick that up?
Many thanks
Andy

Ok cool can you point me in the right direction of that?
I'm just trying a few things in regards to moving the data and db directories out of the fermentrack route on my fork, just for curiosities sake. This might seem a silly question, but if I update the settings.py file do I need to run some build or packaging or will that suffice?
And in my docker file I do a git clone to get fermentrack I've update this to my fork, but the commit output by fermentrack when it initializes is your last dev branch commit not mine. Is there something else I need to do to get it to pick that up?
Many thanks
Andy

One thing db wise I've not looked at yet but is possible, is to run the postgres addon and make it a dependency of the fermentrack addon. I've not seen how to do that yet, but there are other relying on other database addons already. Like I use nginx proxy manager and that uses the influxdb addon. That would only leave the data directory as an issue.
This would be a kind of hybrid between your docker implementation and the all in one approach. I.e. keep fermentrack, redis and nginx in the fermentrack container and then use a separate db container. Still have that issue of the data folder though :(

One thing db wise I've not looked at yet but is possible, is to run the postgres addon and make it a dependency of the fermentrack addon. I've not seen how to do that yet, but there are other relying on other database addons already. Like I use nginx proxy manager and that uses the influxdb addon. That would only leave the data directory as an issue.
This would be a kind of hybrid between your docker implementation and the all in one approach. I.e. keep fermentrack, redis and nginx in the fermentrack container and then use a separate db container. Still have that issue of the data folder though :(

Ok cool can you point me in the right direction of that?

Make a search for /dev in the code, you should find a starting point. I havent been digging in that area yet.

I'm just trying a few things in regards to moving the data and db directories out of the fermentrack route on my fork, just for curiosities sake. This might seem a silly question, but if I update the settings.py file do I need to run some build or packaging or will that suffice?

A restart of the application (django) should be sufficient. Either restart the image or just restart it via circusctl

And in my docker file I do a git clone to get fermentrack I've update this to my fork, but the commit output by fermentrack when it initializes is your last dev branch commit not mine. Is there something else I need to do to get it to pick that up?

The entrypoint.sh script just run a few git command to show that information. Source and last commit. If you are cloning from your own fork it should display that information. Try to connect to the docker image and see if you can figure out what is happening. If you are running docker you should be able to connect with "docker exec -it fermentrack /bin/bash" and get a bash shell.

Ok cool can you point me in the right direction of that?

Make a search for /dev in the code, you should find a starting point. I havent been digging in that area yet.

I'm just trying a few things in regards to moving the data and db directories out of the fermentrack route on my fork, just for curiosities sake. This might seem a silly question, but if I update the settings.py file do I need to run some build or packaging or will that suffice?

A restart of the application (django) should be sufficient. Either restart the image or just restart it via circusctl

And in my docker file I do a git clone to get fermentrack I've update this to my fork, but the commit output by fermentrack when it initializes is your last dev branch commit not mine. Is there something else I need to do to get it to pick that up?

The entrypoint.sh script just run a few git command to show that information. Source and last commit. If you are cloning from your own fork it should display that information. Try to connect to the docker image and see if you can figure out what is happening. If you are running docker you should be able to connect with "docker exec -it fermentrack /bin/bash" and get a bash shell.

One thing db wise I've not looked at yet but is possible, is to run the postgres addon and make it a dependency of the fermentrack addon. I've not seen how to do that yet, but there are other addons relying on other database addons already. Like I use nginx proxy manager and that uses the influxdb addon. That would only leave the data directory as an issue.
This would be a kind of hybrid between your docker implementation and the all in one approach. I.e. keep fermentrack, redis and nginx in the fermentrack container and then use a separate db container. Still have that issue of the data folder though :(

If you want to go this route you should look into the offical docker release (it's currently in the fermentrack dev branch), that uses a separate image for postgres. it's working from what I have tested so far.

One thing db wise I've not looked at yet but is possible, is to run the postgres addon and make it a dependency of the fermentrack addon. I've not seen how to do that yet, but there are other addons relying on other database addons already. Like I use nginx proxy manager and that uses the influxdb addon. That would only leave the data directory as an issue.
This would be a kind of hybrid between your docker implementation and the all in one approach. I.e. keep fermentrack, redis and nginx in the fermentrack container and then use a separate db container. Still have that issue of the data folder though :(

If you want to go this route you should look into the offical docker release (it's currently in the fermentrack dev branch), that uses a separate image for postgres. it's working from what I have tested so far.

Great I'm just picking this back up now, going to have one last play messing with the data and db folders outside of the fermentrack root the I will try relying on postrgres for the db source. That still doesn't fix my the fermentrack/data folder issue though.
Just to help with my lack of knowledge around the fermentrack implementation, what is stored in the /home/fermentrack/fermentrack/data folder and is the contents of this important if for example you want to restart the docker container? And how frequently or maybe just when are files added/update/used in this folder?
One potential idea I have with this folder is to perhaps see if I can set something up that copys files from the /data folder to /home/fermentrack/fermentrack/data at start up and either on a cron whilst running or if there is some way of triggering a script when the container is shutting down.

Great I'm just picking this back up now, going to have one last play messing with the data and db folders outside of the fermentrack root the I will try relying on postrgres for the db source. That still doesn't fix my the fermentrack/data folder issue though.
Just to help with my lack of knowledge around the fermentrack implementation, what is stored in the /home/fermentrack/fermentrack/data folder and is the contents of this important if for example you want to restart the docker container? And how frequently or maybe just when are files added/update/used in this folder?
One potential idea I have with this folder is to perhaps see if I can set something up that copys files from the /data folder to /home/fermentrack/fermentrack/data at start up and either on a cron whilst running or if there is some way of triggering a script when the container is shutting down.

/home/fermentrack/fermentrack/data contains all of the logs of your ferments (the data used to generate the graphs) so you'd likely want to retain it between container recreations.
Again, you could just do this: https://github.com/thorrak/addon-fermentrack/commit/d1f67a65287ae9e18502162b099e973f9d64f997 😉
Just sayin'.
If you can go the Postgres route I would highly recommend it - even if everything else lives in a single container. Postgres > SQLite in every way except ease of installation. I'd tackle that as a separate issue though as @mp-se 's implementation works out of the box with SQLite.

/home/fermentrack/fermentrack/data contains all of the logs of your ferments (the data used to generate the graphs) so you'd likely want to retain it between container recreations.
Again, you could just do this: https://github.com/thorrak/addon-fermentrack/commit/d1f67a65287ae9e18502162b099e973f9d64f997 😉
Just sayin'.
If you can go the Postgres route I would highly recommend it - even if everything else lives in a single container. Postgres > SQLite in every way except ease of installation. I'd tackle that as a separate issue though as @mp-se 's implementation works out of the box with SQLite.

Yep so just to confirm it hates having the data and db folders moved like I'd tried with the environment variables:

Traceback (most recent call last):
File "manage.py", line 25, in
execute_from_command_line(sys.argv)
File "/home/fermentrack/venv/lib/python3.7/site-packages/django/core/management/init.py", line 401, in execute_from_command_line
utility.execute()
File "/home/fermentrack/venv/lib/python3.7/site-packages/django/core/management/init.py", line 395, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/fermentrack/venv/lib/python3.7/site-packages/django/core/management/init.py", line 231, in fetch_command
settings.INSTALLED_APPS
File "/home/fermentrack/venv/lib/python3.7/site-packages/django/conf/init.py", line 76, in getattr
self._setup(name)
File "/home/fermentrack/venv/lib/python3.7/site-packages/django/conf/init.py", line 63, in _setup
self._wrapped = Settings(settings_module)
File "/home/fermentrack/venv/lib/python3.7/site-packages/django/conf/init.py", line 142, in init
mod = importlib.import_module(self.SETTINGS_MODULE)
File "/usr/lib/python3.7/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1006, in _gcd_import
File "", line 983, in _find_and_load
File "", line 967, in _find_and_load_unlocked
File "", line 677, in _load_unlocked
File "", line 728, in exec_module
File "", line 219, in _call_with_frames_removed
File "/home/fermentrack/fermentrack/fermentrack_django/settings.py", line 131, in
DB_DIR = env("MOUNT_DB_DIR")
File "/home/fermentrack/venv/lib/python3.7/site-packages/environ/environ.py", line 123, in call
return self.get_value(var, cast=cast, default=default, parse_default=parse_default)
File "/home/fermentrack/venv/lib/python3.7/site-packages/environ/environ.py", line 277, in get_value
raise ImproperlyConfigured(error_msg)
django.core.exceptions.ImproperlyConfigured: Set the MOUNT_DB_DIR environment variable
Applying database migration
Traceback (most recent call last):
File "/home/fermentrack/venv/lib/python3.7/site-packages/environ/environ.py", line 273, in get_value
value = self.ENVIRON[var]
File "/usr/lib/python3.7/os.py", line 678, in getitem
raise KeyError(key) from None
KeyError: 'MOUNT_DB_DIR'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "manage.py", line 25, in
execute_from_command_line(sys.argv)
File "/home/fermentrack/venv/lib/python3.7/site-packages/django/core/management/init.py", line 401, in execute_from_command_line
utility.execute()
File "/home/fermentrack/venv/lib/python3.7/site-packages/django/core/management/init.py", line 395, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/fermentrack/venv/lib/python3.7/site-packages/django/core/management/base.py", line 328, in run_from_argv
self.execute(*args, **cmd_options)
File "/home/fermentrack/venv/lib/python3.7/site-packages/django/core/management/base.py", line 366, in execute
self.check()
File "/home/fermentrack/venv/lib/python3.7/site-packages/django/core/management/base.py", line 395, in check
include_deployment_checks=include_deployment_checks,
File "/home/fermentrack/venv/lib/python3.7/site-packages/django/core/management/commands/migrate.py", line 63, in _run_checks
issues = run_checks(tags=[Tags.database])
File "/home/fermentrack/venv/lib/python3.7/site-packages/django/core/checks/registry.py", line 72, in run_checks
new_errors = check(app_configs=app_configs)
File "/home/fermentrack/venv/lib/python3.7/site-packages/django/core/checks/database.py", line 9, in check_database_backends
for conn in connections.all():
File "/home/fermentrack/venv/lib/python3.7/site-packages/django/db/utils.py", line 222, in all
return [self[alias] for alias in self]
File "/home/fermentrack/venv/lib/python3.7/site-packages/django/db/utils.py", line 219, in iter
return iter(self.databases)
File "/home/fermentrack/venv/lib/python3.7/site-packages/django/utils/functional.py", line 48, in get
res = instance.dict[self.name] = self.func(instance)
File "/home/fermentrack/venv/lib/python3.7/site-packages/django/db/utils.py", line 153, in databases
self._databases = settings.DATABASES
File "/home/fermentrack/venv/lib/python3.7/site-packages/django/conf/init.py", line 76, in getattr
self._setup(name)
File "/home/fermentrack/venv/lib/python3.7/site-packages/django/conf/init.py", line 63, in _setup
self._wrapped = Settings(settings_module)
File "/home/fermentrack/venv/lib/python3.7/site-packages/django/conf/init.py", line 142, in init
mod = importlib.import_module(self.SETTINGS_MODULE)
File "/usr/lib/python3.7/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1006, in _gcd_import
File "", line 983, in _find_and_load
File "", line 967, in _find_and_load_unlocked
File "", line 677, in _load_unlocked
File "", line 728, in exec_module
File "", line 219, in _call_with_frames_removed
File "/home/fermentrack/fermentrack/fermentrack_django/settings.py", line 131, in
DB_DIR = env("MOUNT_DB_DIR")
File "/home/fermentrack/venv/lib/python3.7/site-packages/environ/environ.py", line 123, in call
return self.get_value(var, cast=cast, default=default, parse_default=parse_default)
File "/home/fermentrack/venv/lib/python3.7/site-packages/environ/environ.py", line 277, in get_value
raise ImproperlyConfigured(error_msg)
django.core.exceptions.ImproperlyConfigured: Set the MOUNT_DB_DIR environment variable
Version/Source of fermentrack installed in image:

origin	https://github.com/mcinnes01/fermentrack.git (fetch)
origin	https://github.com/mcinnes01/fermentrack.git (push)
commit 34fc2b7736085e11eb87920f4679ba1c063785e2
Author: iamandymcinnes<EMAIL_ADDRESS>Added ability to configure data and db directories by environment variables

Starting circus deamon

I tried the example you provided moving the fermentrack root to /data, but it had an issue with the following:
The command '/bin/bash -o pipefail -c groupadd -g 2000 fermentrack && useradd -m -u 2001 -g fermentrack -d /data/fermentrack fermentrack' returned a non-zero code: 12

Yep so just to confirm it hates having the data and db folders moved like I'd tried with the environment variables:

Traceback (most recent call last):
File "manage.py", line 25, in
execute_from_command_line(sys.argv)
File "/home/fermentrack/venv/lib/python3.7/site-packages/django/core/management/init.py", line 401, in execute_from_command_line
utility.execute()
File "/home/fermentrack/venv/lib/python3.7/site-packages/django/core/management/init.py", line 395, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/fermentrack/venv/lib/python3.7/site-packages/django/core/management/init.py", line 231, in fetch_command
settings.INSTALLED_APPS
File "/home/fermentrack/venv/lib/python3.7/site-packages/django/conf/init.py", line 76, in getattr
self._setup(name)
File "/home/fermentrack/venv/lib/python3.7/site-packages/django/conf/init.py", line 63, in _setup
self._wrapped = Settings(settings_module)
File "/home/fermentrack/venv/lib/python3.7/site-packages/django/conf/init.py", line 142, in init
mod = importlib.import_module(self.SETTINGS_MODULE)
File "/usr/lib/python3.7/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1006, in _gcd_import
File "", line 983, in _find_and_load
File "", line 967, in _find_and_load_unlocked
File "", line 677, in _load_unlocked
File "", line 728, in exec_module
File "", line 219, in _call_with_frames_removed
File "/home/fermentrack/fermentrack/fermentrack_django/settings.py", line 131, in
DB_DIR = env("MOUNT_DB_DIR")
File "/home/fermentrack/venv/lib/python3.7/site-packages/environ/environ.py", line 123, in call
return self.get_value(var, cast=cast, default=default, parse_default=parse_default)
File "/home/fermentrack/venv/lib/python3.7/site-packages/environ/environ.py", line 277, in get_value
raise ImproperlyConfigured(error_msg)
django.core.exceptions.ImproperlyConfigured: Set the MOUNT_DB_DIR environment variable
Applying database migration
Traceback (most recent call last):
File "/home/fermentrack/venv/lib/python3.7/site-packages/environ/environ.py", line 273, in get_value
value = self.ENVIRON[var]
File "/usr/lib/python3.7/os.py", line 678, in getitem
raise KeyError(key) from None
KeyError: 'MOUNT_DB_DIR'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "manage.py", line 25, in
execute_from_command_line(sys.argv)
File "/home/fermentrack/venv/lib/python3.7/site-packages/django/core/management/init.py", line 401, in execute_from_command_line
utility.execute()
File "/home/fermentrack/venv/lib/python3.7/site-packages/django/core/management/init.py", line 395, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/fermentrack/venv/lib/python3.7/site-packages/django/core/management/base.py", line 328, in run_from_argv
self.execute(*args, **cmd_options)
File "/home/fermentrack/venv/lib/python3.7/site-packages/django/core/management/base.py", line 366, in execute
self.check()
File "/home/fermentrack/venv/lib/python3.7/site-packages/django/core/management/base.py", line 395, in check
include_deployment_checks=include_deployment_checks,
File "/home/fermentrack/venv/lib/python3.7/site-packages/django/core/management/commands/migrate.py", line 63, in _run_checks
issues = run_checks(tags=[Tags.database])
File "/home/fermentrack/venv/lib/python3.7/site-packages/django/core/checks/registry.py", line 72, in run_checks
new_errors = check(app_configs=app_configs)
File "/home/fermentrack/venv/lib/python3.7/site-packages/django/core/checks/database.py", line 9, in check_database_backends
for conn in connections.all():
File "/home/fermentrack/venv/lib/python3.7/site-packages/django/db/utils.py", line 222, in all
return [self[alias] for alias in self]
File "/home/fermentrack/venv/lib/python3.7/site-packages/django/db/utils.py", line 219, in iter
return iter(self.databases)
File "/home/fermentrack/venv/lib/python3.7/site-packages/django/utils/functional.py", line 48, in get
res = instance.dict[self.name] = self.func(instance)
File "/home/fermentrack/venv/lib/python3.7/site-packages/django/db/utils.py", line 153, in databases
self._databases = settings.DATABASES
File "/home/fermentrack/venv/lib/python3.7/site-packages/django/conf/init.py", line 76, in getattr
self._setup(name)
File "/home/fermentrack/venv/lib/python3.7/site-packages/django/conf/init.py", line 63, in _setup
self._wrapped = Settings(settings_module)
File "/home/fermentrack/venv/lib/python3.7/site-packages/django/conf/init.py", line 142, in init
mod = importlib.import_module(self.SETTINGS_MODULE)
File "/usr/lib/python3.7/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1006, in _gcd_import
File "", line 983, in _find_and_load
File "", line 967, in _find_and_load_unlocked
File "", line 677, in _load_unlocked
File "", line 728, in exec_module
File "", line 219, in _call_with_frames_removed
File "/home/fermentrack/fermentrack/fermentrack_django/settings.py", line 131, in
DB_DIR = env("MOUNT_DB_DIR")
File "/home/fermentrack/venv/lib/python3.7/site-packages/environ/environ.py", line 123, in call
return self.get_value(var, cast=cast, default=default, parse_default=parse_default)
File "/home/fermentrack/venv/lib/python3.7/site-packages/environ/environ.py", line 277, in get_value
raise ImproperlyConfigured(error_msg)
django.core.exceptions.ImproperlyConfigured: Set the MOUNT_DB_DIR environment variable
Version/Source of fermentrack installed in image:

origin	https://github.com/mcinnes01/fermentrack.git (fetch)
origin	https://github.com/mcinnes01/fermentrack.git (push)
commit 34fc2b7736085e11eb87920f4679ba1c063785e2
Author: iamandymcinnes<EMAIL_ADDRESS>Added ability to configure data and db directories by environment variables

Starting circus deamon

I tried the example you provided moving the fermentrack root to /data, but it had an issue with the following:
The command '/bin/bash -o pipefail -c groupadd -g 2000 fermentrack && useradd -m -u 2001 -g fermentrack -d /data/fermentrack fermentrack' returned a non-zero code: 12

Try making the /data directory first

Try making the /data directory first

Ok so just done a little more tweaking around your example, but I am having issues setting permissions on the mount point for the fermentrack user. I guess as this is essentially the directories on the host trying to change permissions on may not be possible and then when we switch to the fermentrack user and try clone it doesn't have the correct permissions and so fails: https://github.com/mcinnes01/addon-fermentrack/blob/fe83cc57bec1c50f0be3d48b9801415c815ad990/fermentrack/Dockerfile#L52
This is the error I get:
Step 12/50 : RUN git clone --single-branch --branch dev https://github.com/thorrak/fermentrack.git /data/fermentrack/fermentrack
---> Running in 986e27691215
Cloning into '/data/fermentrack/fermentrack'...
/data/fermentrack/fermentrack/.git: Permission denied

Ok so just done a little more tweaking around your example, but I am having issues setting permissions on the mount point for the fermentrack user. I guess as this is essentially the directories on the host trying to change permissions on may not be possible and then when we switch to the fermentrack user and try clone it doesn't have the correct permissions and so fails: https://github.com/mcinnes01/addon-fermentrack/blob/fe83cc57bec1c50f0be3d48b9801415c815ad990/fermentrack/Dockerfile#L52
This is the error I get:
Step 12/50 : RUN git clone --single-branch --branch dev https://github.com/thorrak/fermentrack.git /data/fermentrack/fermentrack
---> Running in 986e27691215
Cloning into '/data/fermentrack/fermentrack'...
/data/fermentrack/fermentrack/.git: Permission denied

Check what uid values you have in the container and docker host and make sure you have the same values. Check both user and group, if you have the same numbers it should work.

Check what uid values you have in the container and docker host and make sure you have the same values. Check both user and group, if you have the same numbers it should work.

Closing, to be able to create a home assistant addon I would need to be able to independently specify the location of data and db directories so I can specify them within the data mount provided by home assistant. Currently changing either the data mount path from home assistant or the db and data folders from fermentrack independently of the rest of fermentrack is not possible. Additionally I think some of the permission granted over said folders by fermentrack may not work with the mount directory from home assistant, as it is a bind mount not a volume and so changes to that directory affect the host. I.e. users would need to exist on the host too.


Update deploy script

Update the artifact action
Shorten up a long link

Thanks in advance for your review! :)

Coverage decreased (-0.01%) to 82.598% when pulling 054bd5104d9d872fb5f953cf3a3b0d99f013b69e on Luni-4:update-deploy into 01c80c2d97ac5d32e855c91c302effaf0caa4e68 on xiph:master.


feat: Edit this page will jump to Crowdin when using non-English

Preview: https://casbin.yunshu.site/

@Selflocking BTW, found a bug here: https://casbin.yunshu.site/zh/docs/syntax-for-models

@Selflocking BTW, found a bug here: https://casbin.yunshu.site/zh/docs/syntax-for-models , you can fix it in Crowdin

has been fixed


feat: support GitHub Enterprise Server environments

The api url is set by the particular installation of GHES and available via the ${{ github.api_url }} context variable.  I made the api url an input in case GHES support teams would like to explicitly test a non-prod or sandbox environment.
See<EMAIL_ADDRESS>

@samsmithnz - any chance this feature can make it in? We'd like to use this action in our company's GHES instance.

@mh-isaac-riceweber - all set! Thanks for the contribution - and thanks for pinging this - the notification was lost in a big pile and this reminder put it to the top!


Penny script won't run at all in build

Probably just a quick fix but just writing it down for later. Fix soon.

I have no idea what caused this issue, beyond the basic error message stating that PennyHost.layers had no elements, despite clearly having elements in editor. My best guess is that the BSOD I experienced earlier caused charybdis_penny_host.tscn (inherited scene from penny_example.tscn) to become corrupted somehow.
I solved the issue by removing charybdis_penny_host.tscn from use.
Moral of the story: commit often, build often, especially after a crash.


⚠ ClamAV has degraded performance

In 32b734d, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 64 ms

Resolved: ClamAV performance has improved in 295039c after 535 days, 13 hours, 46 minutes.


Update vanilla-dataTables.min.js

I thought this file was built.  Looks like we need the change here as well.  Removing "s" from modules

Can you bump your version too for npm?  Thanks

Updated to 1.6.10 👍


updateTimestamps in Eloquent/Model fails with latest isDirty

The pull request #66 created an issue.
With the latest PR, isDirty will now return true for the tests in Model->updateTimestamps. This will prevent Model from setting timestamps when creating new models. Not good.

Good catch! Maybe you could submit a pull request to fix this?

Not sure what would be the best approach here… Overriding isDirty() could have unexpected consequences in other places of the framework, and also with other packages.

Yeah, true.

Hooking into the saved event of the parent model is not necessarily the best approach either. If a model update is done, but the core model itself is not changed, only it's translation, then the saved event will not be triggered since performUpdate will not run.

I've pushed a fix in a749482cb1be65e990854c084f87a5554c2bca02. Do you think you could test again?

Seems to be working fine in my tests! 👍

@torkiljohnsen great! Then we only have the Travis tests to fix.


Wish I can watch/talk, powershell is something cool to learn from others

I was going to stick around cuz ur stream seems cool but u got followers only chat , im willing to come back and hang if that ever goes off, theres 0 reason for it and all it does is alienate potential viewers from talking and seeing if they want to follow, if ur worried bout bots, jus moderate/ban them, its simple and doesnt alienate anyone

This request is inappropriate and ludacris. Judging from your audacity in filing this issue and the boundaries that you do not have, it's clear to me that followers only chat is working as I intended.


Update README.md

@narutofan420 please close the pull request

I approved this already. Please refrain from creating duplicate pull requests.


Salsa should not place errors on comments

Testing #2218
Interesting squiggly placement strategy:
Number(3).

var x = 3;

// Just_some_good_text_that_did_nothing_wrong

This issue was moved to Microsoft/TypeScript#6653


[1.11.x] Fixed #28091 -- Re-raise original exception when closing cursor cleanup fails

Please send the PR to the master branch. We will backport it to stable/1.11.x.

This fix only applies for 1.11.x (it's only for python2), so it does not make sense to apply it on master. Can I reopen a pull request against 1.11.x?

Sorry, I didn't notice that. Please reopen the PR to the stable/1.11.x.


simplify sql numeric integration test

I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

Pull request type

Please check the type of change your PR introduces:

[ ] Bugfix
[ ] Feature
[ ] Code style update (formatting, renaming)
[x] Refactoring (no functional changes, no api changes)
[ ] Build related changes
[ ] Documentation content changes
[ ] Other (please describe):

Codecov Report
Patch coverage has no change and project coverage change: -18.55 :warning:

Comparison is base (32fe174) 63.16% compared to head (725dbce) 44.62%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
Additional details and impacted files
@@             Coverage Diff             @@
##           master     #615       +/-   ##
===========================================
- Coverage   63.16%   44.62%   -18.55%
===========================================
  Files         228      149       -79
  Lines       21775    15549     -6226
===========================================
- Hits        13754     6938     -6816
- Misses       7292     8287      +995
+ Partials      729      324      -405

Flag
Coverage Δ

1.16.x
?

1.19.x
?

1.20.x
44.62% <ø> (-18.20%)
:arrow_down:

insecure
?

integration
?

macOS-latest
?

secure
?

ubuntu-latest
?

unit
44.62% <ø> (-0.14%)
:arrow_down:

windows-latest
44.62% <ø> (-0.06%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

 Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Update KOReader

Will only happen if rot is available

Test scenarios:

Oxide is not installed
Oxide is installed but not enabled or running
Oxide is installed and running (Should automatically register with the correct settings)
Test on both rM1 and rM2

thanks for the indepth testing! the latest remux from build.rmkit.dev/master/latest has fixes for koreader which might help here

@LinusCDE could you test the suspend/resume with the v2.1 branch of oxide? It looks like there might be timing issues with the onPause/onResume there.

@LinusCDE could you test the suspend/resume with the v2.1 branch of oxide? It looks like there might be timing issues with the onPause/onResume there.

Built the v2.1 branch (make release) and copied all binaries (and the /etc/dbus thing) over my existing ones in /opt/bin (current toltect testing branch).
I still get the "screen corruption" when leaving directly to oxide (left button). It doesn't seem to happen any second time though when KOReader was not killed.
It also seems that you are not clearing the sceen as much anymore which leaves some residue when launching erode (middle button) and returning.

So it behaves a bit differently, but still has problems.

thanks for the indepth testing! the latest remux from build.rmkit.dev/master/latest has fixes for koreader which might help here

Couldn't reproduce the bug anymore with the updated binary. :+1:

Reinstalling koreader now fixes the issue. The missing onPause/onResume values in tarnish.conf were added and seem to have been the problem. Probably just ensure that people who update koreader get these changes as well and then it should work.

Hmm, well

Reinstalling koreader now fixes the issue. The missing onPause/onResume values in tarnish.conf were added and seem to have been the problem. Probably just ensure that people who update koreader get these changes as well and then it should work.

Added code to handle removing the application entry on remove that should handle this in the future. Maybe I'll also add it to the configure.

Still fighting with the code to make it more robust. It would be great if you could share functions.

Okay, so the configure script works properly, but the postremove script doesn't seem to run ever. When I source the package file on the device and run postremove it works properly. @matteodelabre thoughts?

If you mean bash functions, you can probably just nest them in this case.

It somehow still does not add the onPause and onResume to tarnish.conf. Now I fail to get it even forced in by either deleting the tarnish.conf, removing & installing koreader, fore-reinstalling koreader or some combinations.
EDIT: Using oxide 2.0.1~beta-1

With jq installed?

It somehow still does not add the onPause and onResume to tarnish.conf. Now I fail to get it even forced in by either deleting the tarnish.conf, removing & installing koreader, fore-reinstalling koreader or some combinations.
EDIT: Using oxide 2.0.1~beta-1

With jq installed?

Yes. But rot wasn't for this particular test. Maybe it should just be a dependency?

With rot, entries get added and removed properly.
Upgrading from 2020.10-2 to this version didn't update the onPause and onResume lines, so oxide was still broken when quitting.
Removing and installing this version didn't work either.

Yes. But rot wasn't for this particular test. Maybe it should just be a dependency?
* With rot, entries get added and removed properly.

* Upgrading from 2020.10-2 to this version didn't update the onPause and onResume lines, so oxide was still broken when quitting.

* Removing and installing this version didn't work either.

EDIT: Giving the tarnish service a look makes it seem that it doesn't take notice of the app getting added or removed. Maybe oxide should get updated first?

rot is a dependency of oxide now, but maybe not for the version you have installed. If rot is making the calls, it will notice that changes, since all rot does is make calls out to tarnish and have it handle it.

i should have brought this up earlier, but why does koreader package need to do special calls for oxide? (is it the onPause/onResume commands?) it seems like this should really be solved by config file (i think you are doing this because i saw config file stuff in oxide commits) + a SIGHUP to tarnish.

i should have brought this up earlier, but why does koreader package need to do special calls for oxide? (is it the onPause/onResume commands?) it seems like this should really be solved by config file (i think you are doing this because i saw config file stuff in oxide commits) + a SIGHUP to tarnish.

Because KOReader changes the bitdepth of the display by default, and oxide didn't have pgroup signals yet. v2.1 of oxide has a new file format for applications installed in a package that will solve all of this though.

Retested with the testing version of oxide. Same behaviour as before.

Retested with the testing version of oxide. Same behaviour as before.

What output are you getting during the install?

Retested with the testing version of oxide. Same behaviour as before.

What output are you getting during the install?

I get the Removed app and added app lines when removing/installing the app respectively. journalctl -fu tarnish doesn't show any lines regarding this lines though.

v2.1 will have a proper solution.

v2.1 will have a proper solution.


Apple Silicon?

Any chance this will be updated to run natively on Apple Silicon?

I don't have a Mac anymore, so I wouldn't say the chance is zero but close to zero unfortunately. The project is open sourced so anyone is able to fork it, the code is well documented and create a Silicon native version.


Use commit id for from cache detection

in performance tests. The current detection based on the teamcity build id doesn't work, since we load the teamcity build id from the jsons, which also come from the cache.

It seems to work mostly. Compare the original report vs. the one for a cached build.
I need to check out why the original report has two executions while the cached report only shows one:

Superseded by #17356.


np.array named columns and different column types not working

edit: this is for jax version 0.1.57, I'll check if this still occurs
Classical numpy arrays allow to specify column names/identifiers and assign different types to columns.
From the documentation:

Data-type consisting of more than one element:
>>> x = np.array([(1,2),(3,4)],dtype=[('a','<i4'),('b','<i4')])
>>> x['a']
array([1, 3])

The same documentation is given for jax's np.array but trying the above example yields a
TypeError: unhashable type: 'list'

for the dtype argument (raised in function _check_user_dtype_supported,  line 4563, jax/lax/lax.py).
Is there a chance that this kind of array specifications could be supported? If not, could at least the documentation be clarified?

JAX doesn’t support structured dtypes, but I agree that we provide better a error message here.

too bad, but thanks for the clarification!


Please add to kill aplication

Please add function to kill aplication

Hi @Majkysek , WebOS actually has support for an Exit button which isn't seen on the magic remote.  So if you're looking for a button to kill the currently in focus application, you can use the following.
This would assign the Exit button to the red button on the magic remote.
"red": {
    "function": "press_button",
    "inputs": {
      "button": "exit"
    }
  }


Setting file keys are not matching cmdlet parameters exactly

Raised due to this comment:
Invoke-ScriptAnalyzer has parameters IncludeRule and ExcludeRule parameters and the equivalent of those in the settings file are IncludeRules and ExcludeRules with an additional s, which makes it inconsistent, confusing and not intuitive to work with. The settings file also needs to support the singular form (since the plural form should not be removed for backcompat reasons)
Environment data
> (Get-Module -ListAvailable PSScriptAnalyzer).Version | ForEach-Object { $_.ToString() }
1.17.1

Documentation on the -Settings parameter object in general would be greatly appreciated.  It's been an increasingly frustrating experience of trial and error figuring out how the different keys/values interact especially when trying to customize formatting rules, custom rules, et al in vscode when I don't have time to dig through the source code for everything.


Compact: Offline deduplication

Currently Thanos provides dedup function in Query component, it can merge the data on the fly from Prometheus HA pairs or remote object storage.
However, the Query dedup function is not query efficiently, because the replica's blocks in tsdb/object storage are not really merged so that the Query API has to load duplicated data in each request.
With current Prometheus TSDB design, it seems difficult to implement dedup function to merge blocks in each TSDB node on Thanos side.
However, it should be easily to implement it for the object storage as it is one centralized storage, and also we already have one Compactor component running on it.
Offering dedup function on object storage side will definitely help fast metrics query latency and reduce the object storage cost.
Do we have any plan to support such requirements in Thanos? If had, I would like to know your ideas about this feature.
Thanks.

Hi :wave: thanks for raising this. I rename title to make it more clear, let me know if this makes sense.
There was always idea like this to allow compactor to deduplicate blocks offline and reduce the storage and bit the query performance. I think I am happy to do so however, we need ourselves question, is the curret deduplication algorithm a correct one for everybody?
We have seen some reports claiming that our "penalty-based" deduplication algorithm is not working for all edge cases e.g: https://github.com/improbable-eng/thanos/issues/981 It's fair as our alghorithm is very basic. The problem with this feature and not perfect algorithm is that we will deduplicate data irreversibily (unless we back up those blocks somehow).
Anyway, I think we are happy to add such feature to compactor at some point, given we can solve the gaps, for current algorithm (and add more tests/ test cases maybe).

Thanks for the reply @bwplotka. I'm glad to hear that you also want to support this feature in the future.
I'm working in eBay's monitoring infrastructure team, and we are leveraging Thanos to build eBay's high availability monitoring platform. Thanks for providing such great framework.
For the offline dedup function, we have already started to build it inside the Thanos compactor component. If needed, we are more than happy to contribute it back to Thanos community. Please feel free to let me know your thoughts as well.

Of course! PRs would be welcome, especially if you have something proven from your prod (:

Are you on slack BTW? (:

Which slack channel are your pointing here? I am only be able to see the announcements channel. I'm not able to join the thanos-dev channel which mentioned here: https://github.com/improbable-eng/thanos/blob/master/CONTRIBUTING.md. BTW, my display name is Alan Zhang in announcements channel.

I can see you in all #thanos #thanos-dev channels  (:

Remove "stale", that's an interesting feature

Stale bot, go away for now ;p
We are quite close!
@metalmatze is working on it and I am refactoring TSDB to allow us to do it
via TSDB code.
On Thu, 27 Feb 2020 at 11:14, stale[bot]<EMAIL_ADDRESS>wrote:

This issue/PR has been automatically marked as stale because it has not
had recent activity. Please comment on status otherwise the issue will be
closed in a week. Thank you for your contributions.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/thanos-io/thanos/issues/1014?email_source=notifications&email_token=ABVA3OYQGYMEUPXR4UEUVETRE6OCZA5CNFSM4HEB6SW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEND65VY#issuecomment-591916759,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABVA3OZ6KTITQNE6K3DWWMDRE6OCZANCNFSM4HEB6SWQ
.

Can someone add the label for stalebot to ignore this issue?

We block this for a reason. We want to give you update at least every month (((:+1:
Update: On master, and what will be released next week is naive offline deduplication. This means you can configure stuff to deduplicate exactly the same data. E.g produced by Thanos Receive. This was straightforward to do, so we added that as first iteration here: https://github.com/thanos-io/thanos/pull/2250
You can read changelog how to enable this and what are the consequences.
It's important to know that this will NOT work for replicated Prometheus-es. This is because data is not the same, timestamps are different. If you apply our solutions now you will malform your data. You will be able to query but data will be unreliable really
To fix proper offline dedup we need to replace deduplication algorithm in Prometheus first. I am adding interface allowing us to do so here: https://github.com/prometheus/prometheus/pull/7059 It should be ready in next week or two.
Cheers :beers:

Great news!
Unfortunately this does not work for our current setup and historical data.
We run 3 thanos-receive with --label=replica="$(NAME)"
Then in bucket inspect we have:
| 01E4N18DW7663APZ5V0B3D3CY1 | 30-03-2020 05:30:00 | 30-03-2020 06:00:00 | 30m0s    | 39h30m0s   | 545,049    | 50,824,911     | 545,054     | 1          | false       | replica=thanos-receive-0 | 0s         | receive   |
| 01E4N18DW8A4Z0ES2GCXBFEVVH | 30-03-2020 05:30:00 | 30-03-2020 06:00:00 | 30m0s    | 39h30m0s   | 545,049    | 50,824,911     | 545,054     | 1          | false       | replica=thanos-receive-2 | 0s         | receive   |
| 01E4N18DW8AR0CRXN0F4N0F7VY | 30-03-2020 05:30:00 | 30-03-2020 06:00:00 | 30m0s    | 39h30m0s   | 545,049    | 50,824,911     | 545,054     | 1          | false       | replica=thanos-receive-1 | 0s         | receive   |

And thanos-compact (master-2020-03-27-4e5c2f25) cannot process that due to: empty external labels are not allowed for Thanos block
level=info ts=2020-03-30T02:24:48.810475048Z caller=compact.go:441 msg="compact blocks" count=3 mint=1583366400000 maxt=1584576000000 ulid=01E4M5KHECGZQ1BV66E6SQGH78 sources="[01E3V4JCD9Z5BK5B8A5KSYGMV3 01E3WMG1NXY78KB6GKXR594FMS 01E3T3609JKQ5V1TKZ8NYTGE58]" duration=4h13m14.923098503s
level=info ts=2020-03-30T02:24:48.955331391Z caller=compact.go:704 compactionGroup=0@{} compactionGroupKey=0@17241709254077376921 msg="compacted blocks" blocks="[/data/compact/0@17241709254077376921/01E3V4JCD9Z5BK5B8A5KSYGMV3 /data/compact/0@17241709254077376921/01E3WMG1NXY78KB6GKXR594FMS /data/compact/0@17241709254077376921/01E3T3609JKQ5V1TKZ8NYTGE58]" duration=4h13m15.067940785s overlapping_blocks=true
level=info ts=2020-03-30T02:27:09.523306206Z caller=bucket.go:487 msg="synchronizing block metadata"
level=info ts=2020-03-30T02:27:12.627788832Z caller=bucket.go:500 msg="downloaded blocks meta.json" num=101
level=warn ts=2020-03-30T02:29:27.744301881Z caller=intrumentation.go:54 msg="changing probe status" status=not-ready reason="error executing compaction: compaction: group 0@17241709254077376921: upload of 01E4M5KHECGZQ1BV66E6SQGH78 failed: empty external labels are not allowed for Thanos block."
level=info ts=2020-03-30T02:29:27.744689058Z caller=http.go:81 service=http/server component=compact msg="internal server shutdown" err="error executing compaction: compaction: group 0@17241709254077376921: upload of 01E4M5KHECGZQ1BV66E6SQGH78 failed: empty external labels are not allowed for Thanos block."
level=info ts=2020-03-30T02:29:27.744812002Z caller=intrumentation.go:66 msg="changing probe status" status=not-healthy reason="error executing compaction: compaction: group 0@17241709254077376921: upload of 01E4M5KHECGZQ1BV66E6SQGH78 failed: empty external labels are not allowed for Thanos block."
level=error ts=2020-03-30T02:29:27.745248074Z caller=main.go:212 err="group 0@17241709254077376921: upload of 01E4M5KHECGZQ1BV66E6SQGH78 failed: empty external labels are not allowed for Thanos block.\ncompaction\nmain.runCompact.func5\n\t/go/src/github.com/thanos-io/thanos/cmd/thanos/compact.go:377\nmain.runCompact.func6.1\n\t/go/src/github.com/thanos-io/thanos/cmd/thanos/compact.go:428\ngithub.com/thanos-io/thanos/pkg/runutil.Repeat\n\t/go/src/github.com/thanos-io/thanos/pkg/runutil/runutil.go:72\nmain.runCompact.func6\n\t/go/src/github.com/thanos-io/thanos/cmd/thanos/compact.go:427\ngithub.com/oklog/run.(*Group).Run.func1\n\t/go/pkg/mod/github.com/oklog/run@v1.0.0/group.go:38\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357\nerror executing compaction\nmain.runCompact.func6.1\n\t/go/src/github.com/thanos-io/thanos/cmd/thanos/compact.go:455\ngithub.com/thanos-io/thanos/pkg/runutil.Repeat\n\t/go/src/github.com/thanos-io/thanos/pkg/runutil/runutil.go:72\nmain.runCompact.func6\n\t/go/src/github.com/thanos-io/thanos/cmd/thanos/compact.go:427\ngithub.com/oklog/run.(*Group).Run.func1\n\t/go/pkg/mod/github.com/oklog/run@v1.0.0/group.go:38\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357\ncompact command failed\nmain.main\n\t/go/src/github.com/thanos-io/thanos/cmd/thanos/main.go:212\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"

Yes, we can add one more label for new data.
But what to do with our months of historical data awaiting this feature?
Is it possible to add some --default.label to thanos-compact to apply when there are no labels after deduplication? Or can it leave first label found by --deduplication.replica-label and just remove others?

Oh. Yes, that's a bug. Help wanted to fix it. (: --label-if-empty flag
might do I think... (:
On Mon, 30 Mar 2020 at 08:13, Alex R<EMAIL_ADDRESS>wrote:

Great news!
Unfortunately this does not work for our current setup and historical data.
We run 3 thanos-receive with --label=replica="$(NAME)"
Then in bucket inspect we have:
| 01E4N18DW7663APZ5V0B3D3CY1 | 30-03-2020 05:30:00 | 30-03-2020 06:00:00 | 30m0s    | 39h30m0s   | 545,049    | 50,824,911     | 545,054     | 1          | false       | replica=thanos-receive-0 | 0s         | receive   |
| 01E4N18DW8A4Z0ES2GCXBFEVVH | 30-03-2020 05:30:00 | 30-03-2020 06:00:00 | 30m0s    | 39h30m0s   | 545,049    | 50,824,911     | 545,054     | 1          | false       | replica=thanos-receive-2 | 0s         | receive   |
| 01E4N18DW8AR0CRXN0F4N0F7VY | 30-03-2020 05:30:00 | 30-03-2020 06:00:00 | 30m0s    | 39h30m0s   | 545,049    | 50,824,911     | 545,054     | 1          | false       | replica=thanos-receive-1 | 0s         | receive   |
And thanos-compact (master-2020-03-27-4e5c2f25) cannot process that due
to: empty external labels are not allowed for Thanos block
level=info ts=2020-03-30T02:24:48.810475048Z caller=compact.go:441 msg="compact blocks" count=3 mint=1583366400000 maxt=1584576000000 ulid=01E4M5KHECGZQ1BV66E6SQGH78 sources="[01E3V4JCD9Z5BK5B8A5KSYGMV3 01E3WMG1NXY78KB6GKXR594FMS 01E3T3609JKQ5V1TKZ8NYTGE58]" duration=4h13m14.923098503s
level=info ts=2020-03-30T02:24:48.955331391Z caller=compact.go:704 compactionGroup=0@{} compactionGroupKey=0@17241709254077376921 msg="compacted blocks" blocks="[/data/compact/0@17241709254077376921/01E3V4JCD9Z5BK5B8A5KSYGMV3 /data/compact/0@17241709254077376921/01E3WMG1NXY78KB6GKXR594FMS /data/compact/0@17241709254077376921/01E3T3609JKQ5V1TKZ8NYTGE58]" duration=4h13m15.067940785s overlapping_blocks=true
level=info ts=2020-03-30T02:27:09.523306206Z caller=bucket.go:487 msg="synchronizing block metadata"
level=info ts=2020-03-30T02:27:12.627788832Z caller=bucket.go:500 msg="downloaded blocks meta.json" num=101
level=warn ts=2020-03-30T02:29:27.744301881Z caller=intrumentation.go:54 msg="changing probe status" status=not-ready reason="error executing compaction: compaction: group 0@17241709254077376921: upload of 01E4M5KHECGZQ1BV66E6SQGH78 failed: empty external labels are not allowed for Thanos block."
level=info ts=2020-03-30T02:29:27.744689058Z caller=http.go:81 service=http/server component=compact msg="internal server shutdown" err="error executing compaction: compaction: group 0@17241709254077376921: upload of 01E4M5KHECGZQ1BV66E6SQGH78 failed: empty external labels are not allowed for Thanos block."
level=info ts=2020-03-30T02:29:27.744812002Z caller=intrumentation.go:66 msg="changing probe status" status=not-healthy reason="error executing compaction: compaction: group 0@17241709254077376921: upload of 01E4M5KHECGZQ1BV66E6SQGH78 failed: empty external labels are not allowed for Thanos block."
level=error ts=2020-03-30T02:29:27.745248074Z caller=main.go:212 err="group 0@17241709254077376921: upload of 01E4M5KHECGZQ1BV66E6SQGH78 failed: empty external labels are not allowed for Thanos block.\ncompaction\nmain.runCompact.func5\n\t/go/src/github.com/thanos-io/thanos/cmd/thanos/compact.go:377\nmain.runCompact.func6.1\n\t/go/src/github.com/thanos-io/thanos/cmd/thanos/compact.go:428\ngithub.com/thanos-io/thanos/pkg/runutil.Repeat\n\t/go/src/github.com/thanos-io/thanos/pkg/runutil/runutil.go:72\nmain.runCompact.func6\n\t/go/src/github.com/thanos-io/thanos/cmd/thanos/compact.go:427\ngithub.com/oklog/run.(*Group).Run.func1\n\t/go/pkg/mod/github.com/oklog/run@v1.0.0/group.go:38\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357\nerror http://github.com/thanos-io/thanos/cmd/thanos/compact.go:377\nmain.runCompact.func6.1\n\t/go/src/github.com/thanos-io/thanos/cmd/thanos/compact.go:428\ngithub.com/thanos-io/thanos/pkg/runutil.Repeat\n\t/go/src/github.com/thanos-io/thanos/pkg/runutil/runutil.go:72\nmain.runCompact.func6\n\t/go/src/github.com/thanos-io/thanos/cmd/thanos/compact.go:427\ngithub.com/oklog/run.(*Group).Run.func1\n\t/go/pkg/mod/github.com/oklog/run@v1.0.0/group.go:38\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357\nerror executing compaction\nmain.runCompact.func6.1\n\t/go/src/github.com/thanos-io/thanos/cmd/thanos/compact.go:455\ngithub.com/thanos-io/thanos/pkg/runutil.Repeat\n\t/go/src/github.com/thanos-io/thanos/pkg/runutil/runutil.go:72\nmain.runCompact.func6\n\t/go/src/github.com/thanos-io/thanos/cmd/thanos/compact.go:427\ngithub.com/oklog/run.(*Group).Run.func1\n\t/go/pkg/mod/github.com/oklog/run@v1.0.0/group.go:38\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357\ncompact http://github.com/thanos-io/thanos/cmd/thanos/compact.go:455\ngithub.com/thanos-io/thanos/pkg/runutil.Repeat\n\t/go/src/github.com/thanos-io/thanos/pkg/runutil/runutil.go:72\nmain.runCompact.func6\n\t/go/src/github.com/thanos-io/thanos/cmd/thanos/compact.go:427\ngithub.com/oklog/run.(*Group).Run.func1\n\t/go/pkg/mod/github.com/oklog/run@v1.0.0/group.go:38\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357\ncompact command failed\nmain.main\n\t/go/src/github.com/thanos-io/thanos/cmd/thanos/main.go:212\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357 http://github.com/thanos-io/thanos/cmd/thanos/main.go:212\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"
Yes, we can add one more label for new data.
But what to do with our months of historical data awaiting this feature?
Is it possible to add some --default.label to thanos-compact to apply
when there are no labels after deduplication? Or can it leave first label
found by --deduplication.replica-label and just remove others?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/thanos-io/thanos/issues/1014#issuecomment-605824492,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABVA3O42DVURXM6HTDLRVB3RKBBB7ANCNFSM4HEB6SWQ
.

Vertical compaction works, well, but we need still something suitable for Prometheus replicas. Chasing last dedup bugs, so this is blocked for now.

Blocked by: https://github.com/thanos-io/thanos/issues/2547

@bwplotka  amazing work on Thanos. I did a POC in my testing cluster so that we have HA for Prometheus and also decoupling of data from the local cluster.
I am amazed by the arch. Good job :D
I have a few questions though, im sorry if they have already been answered before

In query component you can use the deduplication checkbox to filter data based on replica labels. So does thanos store the same metric series (replica=A and replica=B) or makes the series one while performing operations ?
I also tried your Receiver approach as one of our prometheus is in the Openshift cluster and that does not have GRPC enabled at the memory.
Noticed that all of a sudden Recieve  showed (saving wal locally, failed to upload) for one of the receiver pod.
After restart, the hashing must have changed which is the expected behaviour, but the memory shot up from 4Gi to 47Gi eventually going into OOM Killed. Do we need more memory than 47Gi or is it some kind of a memory leak  ?

I think those questions are unrelated to the issue, I think it would be better for you to ask them on slack instead. (:

Last blocker for this issue: https://github.com/thanos-io/thanos/issues/981

Hello @bwplotka,
can you please give an update on this feature?

:wave:
We are adding new improvements to runtime deduplication so we can safely use it: https://github.com/thanos-io/thanos/issues/2890 This time, improving penalty algorithm. There might be one tricky part about counter reset. In Prometheus there is not metric type information yet, so we are discussing online how we can add that thanks to recent metadata changes to block (cc @pracucci @brian-brazil). If not we will need to guess it from data which needs more testing.
In the mean time we could potentially allow deduplication with backup, so you still back up blocks in some remote location (another cold storage bucket), so we can revert things if needed. Without back up I would not be confident to allow offline dedup for our Thanos users right now.

There's some initial discussions about type metadata for remote write via the WAL, getting them into blocks is a completely different kettle of fish.

Also Chunk iterator work is almost done which will be needed for dedup during compaction.
Yea agree, super unlikely for now.

Still important

Hi @bwplotka , interested in this issue, is there anyway I can contribute on this ?

Also interested very much. Please update the status.

Still interested.

👋🏽
Recently got many questions on DM, please ask them here around offline dedup (: Some update:
We added more docs hopefully cleaning WHY and HOW: https://thanos.io/tip/components/compact.md/#vertical-compactions
Answering one DM:

I saw on the compactor docs that the compactor supports Vertical Compactions via a hidden flag. Our use-case for wanting to implement that is covered under the use-cases listed right within that doc, specifically as the realistic duplication under the Offline deduplication of series bullet point. I’m a bit confused though. Is this something we can use now using the flags:
--compact.enable-vertical-compaction
--deduplication.replica-label="prometheus_replica"
(The prometheus_replica external label is what we use to distinguish between our each Prometheus within a Prometheus HA pair)
It looks like according to the docs that should be okay, however, I also found this issue you’ve been pretty active on:

It's NOT ok, unfortunately. Vertical compaction is implemented for one-to-one duplications, which comes from https://thanos.io/tip/components/receive.md/ deduplication OR when you have for some reason duplicated block with exactly same data.
If you use this against Prometheus replicas it will most likely totally mess your querying experience as it just concatenates samples together. So scrape interval in best case is 2x of original, in the worst case it's totally unstable.
The missing part is adding the deduplication algorithm we have online on the query to the compaction stage so we can leverage that. This algorithm also works on 99% of cases which is fine for the query part, when you can just switch deduplication off. If this 1% happens on offline dedup, then you cannot revert this, that's a problem.
We are exploring different deduplication algorithms that will make this much more reliable. We also need something ideally for Query pushdown: https://docs.google.com/document/d/1ajMPwVJYnedvQ1uJNW2GDBY6Q91rKAw3kgA5fykIRrg/edit# so ... help wanted (:
We can try to enable 99% realistic dedup if we want, help wanted for this 🤗

Thank you @bwplotka for answering that question and confirming my suspicions. I'm going to look at the doc and related work and see if this is something I can help out with :) We would greatly benefit from a feature like this to dedupe the duplicated data coming from each Prometheus in the HA pair.

This issue is still needed.
@bwplotka it would be nice if Thanos would implement the 99% solution.

Please keep this open.

@2nick I am looking at the same thing. It is a little bit sophisticated and I think the grouping approach in #1276 is similar to what you mentioned.
IMO the grouping part of offline deduplication can reuse the existing tsdb planner. We don't need to group into 2 groups in this case. We can do it in 2 iterations:

block1 + block3 -> new block [1am - 5am]
new block + block2 -> result [1am - 5am]

WDYT?

After some thinking I've decided that it's "good enough" approach as it allows to move forward. :)
Really great that you are in it! Looking forward to offline dedup! :)


@qualweb/act-rules: QW-ACT-R58 - Audio element content has transcript (2eb176)

Audio element content has transcript

Closed by #127


Minor issues in math.evaluation

I found some minor issues in math.evaluation:

The Wikipedia article linked in the r_precision docstring does not seems to be correct. According to the article the metric is calculated based on the Top-N results, where N is determined by the number of relevant documents in the dataset. However, the implementation sets N as the last position of a relevant document as done here. However, all descriptions of R-Precision which I found explain the metric like in the Wikipedia article. So it should be changed to behave like this
The ndcg metric calculates the dcg score relative to the optimal dcg score. Basically, the implemention is correct, however, in most cases it is wrong to apply it in docarray.
The problem is, that optimal value needs to be calculated based on all possible results, however if I apply docarrays match function only the top-k ranking is returned. Therefore, the optimal top-value calculated by our implementation is wrong, e.g., if none of the top-20 results is relevant, the maximum score is 0, however, the actual maximum score might be higher because there relevant scores outside of the top-20. This should be at least explained in the documentation
The R-Precision also requires to know the whole ranking. So this should also be noted in the documentation.

Will u take this? @guenthermi ?
I would correct 1 and document 2 and 3

yes I will do it


Use Jdk7DirectoryWalker as default, since we don't support JDK <8 anymore

Fixes: https://github.com/gradle/gradle/issues/9075 and https://github.com/gradle/gradle/issues/9084
Since we don't support JDK 7 anymore, the alternative walker is not needed.
Removing it opens up the possibility for some minor cleanup and the related tests should also be adjusted.
Also renamed Jdk7DirectoryWalker to DefaultDirectoryWalker (and changed the package), since it's not related to JDK 7 anymore.
Please review commit-by-commit to have context for the different changes.

Merged, thank you for the valuable reviews.
If there's anything that still needs to be added, let me know and I'll do it in a separate PR.


Pictures won't load

What

For some strange reasons no pictures are loading anymore on my device does anyone else have this problem
@teolemon

Same issue here

Any Idea why @g123k I can't find any exception

SocketException: Connection refused (OS Error: Connection refused, errno = 111), address = images.openfoodfacts.org, port = 40486
Any problem with the server @stephanegigandet?

Seems to be on the server side @M123-dev

Now fixed 🥳


Cannot load under Unreal 5.0.2

Hi
I am trying to experiment with Cesium for Unreal under Linux
The machine I was using has Unreal 5.0.2 build from source.
When I put the plugin binary from 500 16 days previous from the day of this post 1.15.1 from recollection.
It says that it cannot load and it needs to be run through my IDE.
So can I get some tips about how to compile from source?
Sam

Apologies for the noise
https://github.com/CesiumGS/cesium-unreal/blob/ue5-main/Documentation/developer-setup.md


Lifetime battery degradation resets to 100% for small augmentation

Describe the bug
Battery augmentation causes the lifetime component of the cycle life model to reset degradation to 100%, regardless of the amount of augmentation. This can lead to long periods with no degradation, which does not match expectations of real world performance.
To Reproduce
Steps to reproduce the behavior:

Run the attached SAM file:
lfp-lifetime-test-btm.zip

See lack of degradation after augmentation

Expected behavior
TODO: define expected behavior
Screenshots

Desktop (please complete the following information):

Version 2022.11.21, maybe others

from conversations with Paul, simple ways to do this are:
For a 10 kwh battery that has lost 70% of its capacity (70% cycle and 70% calendar):

If the augmentation is for 3 kwh, then the battery should be back to 10kwh of capacity (no calendar or cycling degradation)
If the augmentation is for 1 kwh, then the battery should be back to ~8 kwh of capacity (figure out how to get both calendar and cycling degradation back to ~80%)

There would be two caveats with this approach:

Degradation tends to be faster in new cells, so this overestimates degradation rate, but…
This does not account for safety/reliability risks of having very low health cells (higher sudden failure rate at low health)


Error While running " ollama run llama2" or any model.

I getting a error when i am trying to setup ollama in windows. i have installed ollama successfully and even the ollama serve is also working. but when i try to run a model i am getting this error.
  That DNS server isn't resolving internet hostnames and can't resolve the ollama registry.   Hopefully whoever manages your network provides a proxy to be able to connect to the internet, and if so, you can use https://github.com/ollama/ollama/blob/main/docs/faq.md#how-do-i-use-ollama-behind-a-proxy to set up the proxy.   If your host has no possible way to connect to the internet, you could attempt to pull models on another computer on another network that does have internet connectivity, and then replicate the models directory https://github.com/ollama/ollama/blob/main/docs/faq.md#where-are-models-stored


server-version: Add server version to api endpoints.

Add server version to fetch_initial_state_data.
Add server version to register event queue api endpoint.
Add server version to get_auth_backends api endpoint.
Change source for server version in home endpoint.
Fix tests.

Fixes #3663

Automated message from Dropbox CLA bot
@kkanahin, it looks like you've already signed the Dropbox CLA. Thanks!

Merged, thanks @kkanahin!!

@borisyankov @neerajwahi FYI, clients should now have API access to the Zulip version, should they want the data.


prometheus-redis-exporter: 1.63.0 -> 1.64.0

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/oliver006/redis_exporter/releases.
meta.description for prometheus-redis-exporter is: Prometheus exporter for Redis metrics
meta.homepage for prometheus-redis-exporter is: https://github.com/oliver006/redis_exporter
Updates performed

Golang update

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact
Checks done

built on NixOS
The tests defined in passthru.tests, if any, passed
found 1.64.0 with grep in /nix/store/hqbm0agi1nb027m9xdy3l45b0zx74lga-redis_exporter-1.64.0
found 1.64.0 in filename of file in /nix/store/hqbm0agi1nb027m9xdy3l45b0zx74lga-redis_exporter-1.64.0

Rebuild report (if merged into master) (click to expand)

2 total rebuild path(s)

2 package rebuild(s)

First fifty rebuilds by attrpath

prometheus-redis-exporter

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/hqbm0agi1nb027m9xdy3l45b0zx74lga-redis_exporter-1.64.0 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A prometheus-redis-exporter https://github.com/r-ryantm/nixpkgs/archive/7a0662beb810a103a95df890f5a84b29fdcb9124.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/7a0662beb810a103a95df890f5a84b29fdcb9124#prometheus-redis-exporter

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/hqbm0agi1nb027m9xdy3l45b0zx74lga-redis_exporter-1.64.0
ls -la /nix/store/hqbm0agi1nb027m9xdy3l45b0zx74lga-redis_exporter-1.64.0/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

1 package built:

prometheus-redis-exporter

Maintainer pings
cc @eskytthe @srhb @Ma27 for testing.

[!TIP]
As a maintainer, if your package is located under pkgs/by-name/*, you can comment @NixOS/nixpkgs-merge-bot merge to automatically merge this update using the nixpkgs-merge-bot.

Add a :+1: reaction to pull requests you find important.

@ofborg test prometheus-exporters.redis

@ofborg eval

Updated to 1.65.


Add default variable for disk1 size

Now is currently hard coded with 6G

Depends on #10. Once I figure out #10, then it makes sense to implement this one.

Closing since the VM will default to the size defined in the image: 32Gb. Not a big deal anyway.


Add a server GUI report showing the full topology with object types, system names, and addresses.

Is your feature request related to a problem? Please describe.
Add a server GUI report showing the full topology with object types, system names, and addresses.
Describe the solution you'd like
Describe alternatives you've considered
Additional context

Decided to make an inventory per MO class type - Eg Decoders, LGs, Actuators, and Sensors.

Decoder inventory done.

Fixed!


python gtest_parallel.py --help doesn't do anything

This looks like a very useful utility but unfortunately I could not get it to work. I cloned the code and tried invoking the script both via 'python gtest_parallel.py` and via the script and nothing happens : no output, no error, nothing. I tried with and without arguments, same results. I tried this on a Mac (10.12.6) with Python 2.7.10 and Windows 10 with Ptyhon 2.7.8.

You should be running python gtest_parallel and not gtest_parallel.py. To be fair though, I think running gtest_parallel.py should be equivalent.


wtype: 0.3 -> 0.4

Motivation for this change
Fixes pasting into xwayland windows
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
22.05 Release Notes (or backporting 21.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

@justinlovinger @colemickens

Result of nixpkgs-review pr 162134 run on x86_64-linux 1

2 packages built:

rofimoji
wtype


YJDH-574 | Enable Suomi.fi auth with acting on behalf of a company

Description :sparkles:
This PR adds changes that are required to get the Eauthorization working with the Suomi.fi SAML authentication.
Issues :bug:
YJDH-574 Suomi.fi integration

as Juha is gone for vacation, I'll merge this


Added Responsive Honda Clone

🛠 Fixes Issue
Closes #606

👨💻 Changes proposed

Added Responsive Honda UI Clone

✔ Check List (Check all the applicable boxes)

[x] My code follows the code style of this project.
[x] This PR does not contain plagiarized content.
[x] The title of my pull request is a short description of the requested changes.

📷 Screenshots
https://user-images.githubusercontent.com/77873383/170770764-ec995514-f012-4cd6-adc3-4f9697564ccc.mp4

@gurjeetsinghvirdee please check this


I would like to use the graph API from Power Automate to interact with Outlook 365

It's kind of funny how there are about 5000 versions of THIS partially complete information on the internet (and MULTIPLE copies on Microsoft sites), but there is none, that go into depth on the individual settings, or simply explains in pain English how a reasonably technical person using Power Automate who needs to write to the Microsoft Graph to interact with Outlook (for example).
You write all this software, the marketers are out there trying to sell it, but it's like no one cares if customers can actually figure out how to use these things.  Perhaps this seems petty, but keep in mind we are SIMULTANEOUSLY trying to figure out hundreds of other things that are changing each day.  Having ONE person at Microsoft putting a little bit more effort into actually useful documentation would save millions of dollars of wasted time among your customers, and it would perhaps reduce the likelihood of the non-trivial number of people like me who trash Azure in technical forums like Hacker News every single time someone is asking for advice - I think your company should be punished for your laziness.

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 11caa291-2e98-ac3a-d5d1-ecd416b93bdf
Version Independent ID: 4f30cddd-8849-1887-ada0-a83a3d2a0027
Content: Tutorial: Register an app with Microsoft Entra ID (Microsoft Dataverse) - Power Apps
Content Source: powerapps-docs/developer/data-platform/walkthrough-register-app-azure-active-directory.md
Service: powerapps
Sub-service: dataverse-developer
GitHub Login: @paulliew
Microsoft Alias: paulliew

@torevor Our Dataverse developer documentation does not cover that specific scenario. However, I will forward your issue to our Power Automate writer for consideration.
You can also file a new GitHub issue on the Power Automate documentation yourself.
Thank you for the feedback.


Added Red Wine Quality detection

Pull Request for PyVerse 💡
Requesting to submit a pull request to the PyVerse repository.

Issue Title
Please enter the title of the issue related to your pull request.
Red wine Quality Detection

[x] I have provided the issue title.

Info about the Related Issue
What's the goal of the project?
Red wine Quality Detection by statisticall addressing and referencinf each of these features 'residual sugar', 'total sulfur dioxide', 'sulphates', 'alcohol', 'volatile acidity', 'quality' and then thoroughly taking out a decision on the overall quality.

[x] I have described the aim of the project.

Name
Please mention your name.
Janvi

[x] I have provided my name.

GitHub ID
Please mention your GitHub ID.
inkerton

[x] I have provided my GitHub ID.

Email ID
Please mention your email ID for further communication.
<EMAIL_ADDRESS>
[x] I have provided my email ID.

Identify Yourself
Mention in which program you are contributing (e.g., WoB, GSSOC, SSOC, SWOC).
GSSOC

[x] I have mentioned my participant role.

Closes
Enter the issue number that will be closed through this PR.
Closes: #issue-number #301

[x] I have provided the issue number.

Describe the Add-ons or Changes You've Made
Give a clear description of what you have added or modified.
I have added these 7 models and evaluated their accuracy

Logistic Regression (LR): A linear model that predicts categorical outcomes based on input features.
Naive Bayes (NB): A probabilistic model based on Bayes’ theorem.
K-Nearest Neighbors (KNN): A non-parametric model that classifies based on the majority vote of the nearest neighbors.
Decision Tree (DT): A tree-based model that splits the data on feature values to predict the target variable.
Support Vector Machine (SVM): A model that finds the optimal hyperplane to classify data points in high-dimensional spaces.
Linear Discriminant Analysis (LDA): A linear model that reduces dimensionality while preserving as much class-discriminatory information as possible.

[x] I have described my changes.

Type of Change
Select the type of change:

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Code style update (formatting, local variables)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

How Has This Been Tested?
Describe how your changes have been tested.
The project has been tested using the following methods:

Cross-Validation:

5-fold Stratified Cross-Validation was used to evaluate the performance of each machine learning model. This ensures that the model is tested on different subsets of the data, providing a robust evaluation of its accuracy.

Model Evaluation Metrics:

Each model was evaluated using accuracy scores, which were calculated during cross-validation. This allowed comparison of the success rates of various models like Logistic Regression, K-Nearest Neighbors, Decision Tree, etc.

Handling Class Imbalance:

The dataset's imbalanced nature was addressed using SMOTE, which oversamples minority classes, ensuring that models perform well across all wine quality levels.

[x] I have described my testing process.

Checklist
Please confirm the following:

[x] My code follows the guidelines of this project.
[x] I have performed a self-review of my own code.
[x] I have commented my code, particularly wherever it was hard to understand.
[x] I have made corresponding changes to the documentation.
[x] My changes generate no new warnings.
[x] I have added things that prove my fix is effective or that my feature works.
[ ] Any dependent changes have been merged and published in downstream modules.

@UTSAVS26
I would like the difficulty of this issue to be raised to level3 if you see fit.
Reasoning for Reclassification:
Multiple Model Evaluation: Evaluating seven different models requires understanding each model's strengths and weaknesses, as well as comparing their performance across the same dataset. This increases the effort and knowledge required, as the results must be analyzed, interpreted, and reported for multiple models.
Cross-Validation: Stratified K-Fold cross-validation for all models, rather than simple train-test splits, ensures that the models are tested rigorously. This adds both complexity and a significant time investment to the project.
Handling Imbalanced Data: Using SMOTE to address class imbalance adds another layer of difficulty, as it involves advanced preprocessing techniques to ensure model fairness across all wine quality categories.
SMOTE (Synthetic Minority Over-sampling Technique) is a technique used in machine learning to handle imbalanced datasets. When you have a classification problem where one class is significantly underrepresented compared to others (imbalanced data), models can become biased towards the majority class, leading to poor performance on the minority class.

@inkerton please go through the discussion announcement section, based on the rules if anyone is adding any project it will be categorised into level1 until they are fixing any bug or adding feature to it, but as the issue is already been assigned to level2 before making the announcement that's why I didn't change it.

@UTSAVS26  So adding a project of any complexity will now be categorized as level1??

@UTSAVS26 I have a minor specialization in data science and have many projects which are incomplete like they have the model but I have worked on explaining the result any plots or any explainatory code to explain the result. They just have they result declared as accuracy = this or like that .
I was thinking of completing them and then adding them here so should i just add them first and the add features and algorithms to them? Don't mind but I also have to earn points and if i can work on my own projects and earn points that would be beneficial to me.

@UTSAVS26  So adding a project of any complexity will now be categorized as level1??

Yes