Device-mapper does not release free space from removed images

Docker claims, via docker info to have freed space after an image is deleted, but the data file retains its former size and the sparse file allocated for the device-mapper storage backend file will continue to grow without bound as more extents are allocated.
I am using lxc-docker on Ubuntu 13.10:
Linux ergodev-zed 3.11.0-14-generic #21-Ubuntu SMP Tue Nov 12 17:04:55 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

This sequence of commands reveals the problem:
Doing a docker pull stackbrew/ubuntu:13.10 increased space usage reported docker info, before:
Containers: 0
Images: 0
Driver: devicemapper
 Pool Name: docker-252:0-131308-pool
 Data file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 291.5 Mb
 Data Space Total: 102400.0 Mb
 Metadata Space Used: 0.7 Mb
 Metadata Space Total: 2048.0 Mb
WARNING: No swap limit support

And after docker pull stackbrew/ubuntu:13.10:
Containers: 0
Images: 3
Driver: devicemapper
 Pool Name: docker-252:0-131308-pool
 Data file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 413.1 Mb
 Data Space Total: 102400.0 Mb
 Metadata Space Used: 0.8 Mb
 Metadata Space Total: 2048.0 Mb
WARNING: No swap limit support

And after docker rmi 8f71d74c8cfc, it returns:
Containers: 0
Images: 0
Driver: devicemapper
 Pool Name: docker-252:0-131308-pool
 Data file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 291.5 Mb
 Data Space Total: 102400.0 Mb
 Metadata Space Used: 0.7 Mb
 Metadata Space Total: 2048.0 Mb
WARNING: No swap limit support

Only problem is, the data file has expanded to 414MiB (849016 512-byte sector blocks) per stat. Some of that space is properly reused after an image has been deleted, but the data file never shrinks. And under some mysterious condition (not yet able to reproduce) I have 291.5 MiB allocated that can't even be reused.
My dmsetup ls looks like this when there are 0 images installed:
# dmsetup ls
docker-252:0-131308-pool        (252:2)
ergodev--zed--vg-root   (252:0)
cryptswap       (252:1)

And a du of the data file shows this:
# du /var/lib/docker/devicemapper/devicemapper/data -h
656M    /var/lib/docker/devicemapper/devicemapper/data

How can I have docker reclaim space, and why doesn't docker automatically do this when images are removed?

I'm still getting this error with Ubuntu 14.04 and Docker 1.4.1.

Don't count on the thin-pool and thin versions being >= 1.12.0 to mean you're O.K.  We're running CentOS 6.5 VMs with the 2.6.32-431.29.2.el6.x86_64 kernel and dmsetup targets reports that thin-pool and thin are both v1.12.0.  However, I'm stuck with a 40GB data file that isn't freeing itself up.

What are the implications running this on CentOS / RHEL 6.5 with this bug? Are you OK with >=100G of free space. I assume this eventually fills any size disk? Or is it bounded at 100G?

Keep in mind that 6.5 does not have the newest kernel available to centos. I would recommend a simple 'yum update' to 6.6 and a reboot to test with the 2.6.32-504.3.3 kernel.

Confirm the very latest kernel and distro updates for CentOS 6 works fine and releases space.

ripienaar: Can you explicitly state which CentOS 6 and kernel version you are using?  I just want to make sure that when I pass the info along, I have all of the info I need.
thanks.

As @reiz commented above, this is happening with latest Ubuntu 14.04 + latest docker. dmsetup targets shows thin/thin-pool of v1.9.0 on one of our instances (no active docker containers.) dmsetup targets doesn't show any thin entries on a similar instance with active docker containers. (I'm not really asking for help on this, just adding (hopefully useful) data.)

Basic OS stuff:
# uname -a
Linux vagrant-centos65 2.6.32-504.3.3.el6.x86_64 #1 SMP Wed Dec 17 01:55:02 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/redhat-release
CentOS release 6.6 (Final)

# dmsetup targets
thin-pool        v1.14.0
thin             v1.14.0
mirror           v1.12.0
striped          v1.5.6
linear           v1.1.0
error            v1.2.0

# dmsetup status
docker-8:1-393542-pool: 0 209715200 thin-pool 92343 178/524288 4664/1638400 - rw discard_passdown queue_if_no_space

Usage:
# du -h /var/lib/docker/
2.7G    /var/lib/docker/devicemapper/devicemapper
# docker rmi b39b81afc8ca
# du -h /var/lib/docker/
2.5G    /var/lib/docker/devicemapper/devicemapper

Previous to getting onto that kernel it would not recover space.

@jperrin mentions this be resolved using centos kernel 2.6.32-504.3.3.  is it known which kernel commit(set of commits?) that resolve this issue?
I am currently using one of Oracle Enterprise Linux 3.8 UEK.

Is there any workaround for this situation? at AWS is easy to just re-create the instance but at the local development environment is not good to notice the /var partition is owned in a 90% by device-mapper because of docker; soon won't have space even for logs or journals

@donrudo as above - update to latest centos 6 - the 504.3.3 kernel.

That's OK for centos but some of our devs are using Fedora and others OpenSuse.

All DM thin provisioning changes are sent upstream first.  So if Centos 6 is fixed it is also fixed upstream.  The various distros need to pull fixes back or rebase more aggressively.

@ripienaar I'll ask again, do you know which commit(set of commits) that are required for the fix?

@lexinator nope

I'm seeing this issue on Ubuntu 14.04 Trusty and kernel 3.13.0-36-generic. Not sure how to resolve this other than uninstalling docker completely and reinstalling.

Guys, this issue really hurts. CoreOS is moving to ext4 now and soon we will have all this troubles on CoreOS as well.
How people can use docker (in dev or prod) while this issue is unfixed for more than a year? Everybody has endless disks?

I believe they are moving to ext4 for overlay not devmapper. But that's
just the word on the street...
On Wednesday, January 28, 2015, Sergey<EMAIL_ADDRESS>wrote:

Guys, this issue really hurts. CoreOS is moving to ext4 now and soon we
will have all this troubles on CoreOS as well.
How people can use docker (in dev or prod) while this issue is unfixed for
more than a year? Everybody has endless disks?
—
Reply to this email directly or view it on GitHub
https://github.com/docker/docker/issues/3182#issuecomment-71800923.

I am confused. My disk space on a Google instance is just 10G, but the size of the /var/lib/docker/devicemapper/devicemapper/data is shown as 100G. I am not able to reclaim space even after deleting all the containers and images.

It's a sparse file. Use ls -sh instead of ls -l, or alternately, du -h

I can report this on Ubuntu 13.04 (probably not surprising).
Is the only real workaround to this right now to provision enough disk space to anticipate growth of devicemapper?

  One Debian user after another complaining.  If you're seeing a problem you should either switch to a distro that is properly maintained or work with your distro maintainers to get them to fix the issue.   These distros aren't keeping up with fixes or improvements.  I'll be ignoring all future pleading for someone to fix this issue on distro $foo given the fix is clearly upstream already (see CentOS or RHEL).  Please open a bug against the broken distro in question and reference this issue.

It's not that ridiculous when Docker maintains they have support on the following platforms:
Docker is supported on the following versions of Ubuntu:

Ubuntu Trusty 14.04 (LTS) (64-bit)
Ubuntu Precise 12.04 (LTS) (64-bit)
Ubuntu Raring 13.04 and Saucy 13.10 (64 bit)

I wouldn't have a blanket support call for 14.04 when this issue is obviously so prevalent.
I'm using CentOS and and kernel upgrade seems to have improved the run away devicemapper usage issue.
Regardless, thanks for your work @snitm.

For those having this issue on Ubuntu, ensure that you're appropriately cleaning up old docker images/containers. I found that my cleanup script wasn't running when I thought it was, and that's why it was using so much space.
 If your containers make use of volumes, you should use docker rm -v to have docker remove the volumes as well.

@snitm We just got bit by this as well. While it might not be Docker's fault specifically, they should have a strong interest in fixing this since it reflects poorly on the whole experience.

Also note I've bumped into this problem and this is what I did to fix it:
In my case though, due to device that /var is mounted on reaching 100% usage, the only way to stop the host from going into a kernel panic is to kill the docker process directly (i.e. issuing any docker commands would panic).
Then manually kill any processes using the devicemapper mounts using this command:
grep docker /proc/*/mounts | awk -F/ '{ print $3 }' | xargs kill -9
then I was able to umount any devicemapper mounts using:
for i in /dev/mapper/docker-*; do umount $i; dmsetup remove $i; done
those steps released all the file descriptors which allowed me to reclaim the space/delete files whatever..
At that point in my case, it made more sense to move the /var/lib/docker directory to a mount with more space and configure the docker daemon to use a different home directory using the  -g argument

Following on to @dekz comment, it would probably be a good idea if the installation guide didn't instruct users that kernel versions without this fix are acceptable as is done here: https://docs.docker.com/installation/ubuntulinux/

Thanks @JohnMorales. A little warning about this from Docker would be a great idea.

+1

+1

We recently got bit by this running Ubuntu 14.04.2 LTS (3.16.0-30-generic) and are still scrambling to find a solution. It seems like either switching to CentOS 6.6 (which supposedly has the kernel fix) or mounting a volume that has a larger disk are the only solutions.
Has anyone running Ubuntu found an acceptable workaround?
I agree - Docker should man up and address this issue - it's a serious bug that needs documentation and warning messages, so more people don't naively fall into the same trap.

I agree with @natea
I'm facing same issue with redhat 6.5 and it is putting a big question mark if docker is production ready and we are starting to look for alternatives.
I don't mind even getting a manual workaround to solve the issue, but right now, I'm not aware of any feasible ones.

@sagiegurarie be aware that we bumped the requirements for CentOS to 6.6 because of various issues with 6.5 (and the associated kernel). This requirement is probably not yet on the live docs-website, but will be with the next release, maybe before that.

so issue #11643 answer is not relevant? that means we can't use redhat 6.5?

@natea @sagiegurari What works for us is the manual "fix" suggested by @TylerMills
# Delete all containers
docker rm $(docker ps -a -q)
# Delete all images
docker rmi $(docker images -q)

The downside, of course, is that you then have to re-pull all images and run the containers again.

@bkeroackdsc I remember i deleted all containers and images before as well and that didn't solve it, but I'll give it another shot.

Hello! I would like to know if this issue still exists on ubuntu-15.04. Which is now at linux-3.19 kernel. Many thanks.

if you  are on ubuntu-15.05 with that kernel why no use overlay, it is way
better than devicemapper
On Thu, May 7, 2015 at 11:26 AM, Dreamcat4<EMAIL_ADDRESS>wrote:

Hello! I would like to know if this issue still exists on ubuntu-15.04.
Which is now at linux-3.19 kernel. Many thanks.
—
Reply to this email directly or view it on GitHub
https://github.com/docker/docker/issues/3182#issuecomment-99968299.

@jfrazelle Oh right. What's it like? Do you use overlay? I have been going under the assumption (up until now) that overlay support was still experimental as of 3.19 kernel.

Yes I use overlay :) we are hoping to move it up the list in 1.7 lost of us
use it and love it
On Thursday, May 7, 2015, Dreamcat4<EMAIL_ADDRESS>wrote:

@jfrazelle https://github.com/jfrazelle Oh right. What's it like? Do
you use overlay? I have been going under the assumption (up until now) that
overlay support was still experimental as of 3.19 kernel.
—
Reply to this email directly or view it on GitHub
https://github.com/docker/docker/issues/3182#issuecomment-99982543.

@sagiegurari Why would you stick with 6.5? There are a number of updates and fixes you're missing by NOT going to 6.6, including news-worthy CVEs with names.

@jperrin because it's not my decision :)
anyway i'm trying to check with all teams to see if redhat 7 would be ok for production.

Just ran into this issue on Centos 6.6.  It's affecting images and containers.  Both deleting or creating.
]# docker ps -a
CONTAINER ID        IMAGE               COMMAND               CREATED             STATUS                        PORTS               NAMES
9a1e328f17e8        jenkins-1:latest    "/usr/sbin/sshd -D"   54 minutes ago      Exited (255) 38 minutes ago                       trusting_feynman
[root@jenkins-sb ~]# docker rm 9a1e328f17e8
Error response from daemon: Cannot destroy container 9a1e328f17e8: Driver devicemapper failed to remove root filesystem 9a1e328f17e8a0560651dcefa8ce42679ee3b060cb11efdba88e05036cd942a3: Error running DeleteDevice dm_task_run failed
FATA[0000] Error: failed to remove one or more containers
]# docker run -t -i centos:6 /bin/bash
FATA[0000] Error response from daemon: Error running DeviceCreate (createSnapDevice) dm_task_run failed
System info:
uname -a
Linux jenkins-sb.servers.xprima.com 2.6.32-504.16.2.el6.x86_64 #1 SMP Wed Apr 22 06:48:29 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
dmsetup targets
thin-pool        v1.14.0
thin             v1.14.0
mirror           v1.12.0
striped          v1.5.6
linear           v1.1.0
error            v1.2.0
docker info
Containers: 2
Images: 61
Storage Driver: devicemapper
Pool Name: docker-253:1-659841-pool
Pool Blocksize: 65.54 kB
Backing Filesystem: extfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 2.631 GB
Data Space Total: 107.4 GB
Metadata Space Used: 4.141 MB
Metadata Space Total: 2.147 GB
Udev Sync Supported: true
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.89-RHEL6 (2014-09-01)
Execution Driver: native-0.2
Kernel Version: 2.6.32-504.16.2.el6.x86_64
Operating System:
CPUs: 2
Total Memory: 1.83 GiB
Name: jenkins-sb.servers.xprima.com
ID: DYDG:B5F7:EKJO:26HI:N4JJ:WAAB:QOTS:IA7W:4L2A:PTOY:4IKS:RADT
Installed Packages
Name        : docker-io
Arch        : x86_64
Version     : 1.5.0
Release     : 1.el6
Size        : 20 M
Repo        : installed
From repo   : xprima-epel
Summary     : Automates deployment of containerized applications
URL         : http://www.docker.com
License     : ASL 2.0
Sample device-mapper errors from dmesg:
device-mapper: space map metadata: unable to allocate new metadata block
device-mapper: thin: Deletion of thin device 165 failed.
device-mapper: space map common: dm_tm_shadow_block() failed
device-mapper: space map common: dm_tm_shadow_block() failed
device-mapper: space map common: dm_tm_shadow_block() failed
device-mapper: space map common: dm_tm_shadow_block() failed
device-mapper: space map metadata: unable to allocate new metadata block
device-mapper: space map common: dm_tm_shadow_block() failed
device-mapper: space map common: dm_tm_shadow_block() failed
device-mapper: thin: Creation of new snapshot 171 of device 15 failed.

This seems to still be an issue on CentOS7 (3.10.0-123.el7.x86_64). I setup a new CentOS box and installed docker. The devicemapper data file never decreases size after a sudo docker rmi. I am looking into getting a CentOS6.6 box to try it out on that.

v1.6.0 on Ubuntu 14.04 seemed to have a positive effect (over v1.5.0)

seems to also work good on centos 7 which I guess can also say for redhat 7
but redhat 6.5 isn't, and i suggest to state that in the docs.

This should be working in latest upstream docker. And last comment from @sagiegurari seems to imply that. (works on centos 7).
If that's the case, we should close this bug.
So is this problem still happening with latest docker? If not, lets close it.

well.... running more tests seems to provide an uncertain results.
lets say its much better, but might still be leaking, just a lot slower.
i'll run more checks to confirm.

@sagiegurari
Can you give me some more details on what's the issue you are facing and also your environment. Title of the issue says that space is not reclaimed upon image/container deletion, and latest upstream docker should have no such issues.
Can you please take upstream docker, build a dynamic binary, create a fresh thin pool (in a fresh /var/lib/docker/) directory, pull an image, delete it and look at docker info and see if space has been freed or not.

I'll run more tests hopefully next week to provide more exact info.
I admit i see it in our build machine where we create and remove a lot of images (during the build itself) + run the docker registry 2.0 (to which we always push the different images with tag 'latest') + run few containers to do initial testing that everything works fine (test is actually not supposed to run on that machine, but we are in early stage so we create/stop/rm a lot of containers).
our images are really big, sometimes i see virtual size over 2gb, so if there is an issue, it shows itself faster than with usual images that are normally around 400mg.
anyhow, i'll try to run few tests to see behavior and provide more details next week.

Using Docker version 1.6.0, build 4749651/1.6.0 on Linux kernel 3.14.35-28.38.amzn1.x86_64
Removing images decreases disk space claimed by docker.

@vbatts
Can we close this issue I don't think in latest docker we have the problem of reclaiming image/container space from oop files. If something specific comes up, one can open a new issue with specific details.

@rhvgoyal I don't see a reason to rush and close this issue after so many people complain about so many platforms/versions.

@sagiegurari
I don't know what do we gain by keeping issue open. I know it is fixed upstream. There are so many issues which are open. People are using them as dashboard to note the observations.
I feel that keeping an issue open makes sense only if there is a real reproducible problem. Otherwise this list of issues just keeps on growing and it is hard to figure out which one to focus on.
Nobody stops you from adding more data to this issue once you have that.

@rhvgoyal what tag are you referring to by latest upstream?

@rhvgoyal Say latest commit. I just clone the latest git tree and test that. Does it really matter?

So many problems are reported on old docker releases and old kernels. I think first step should be to see if same problem is visible on latest docker and relatively newer kernerl. That gives us an idea if it is still a problem or something which is not fixed in older versions.

When someone comes back and reads it in the future, yes. I was wanting to know what version it was fixed in.

We issue discards in the case of loopback devices. It was done long back actually. I think 1.6 should have it.

@stanislavb tested with 1.6 and it worked for him.

If you refer to my first comment in this issue you will see that I also tried with v1.6.0 on centos7 and still had the issue.

@alexDrinkwater
Ok, are you able to reproduce it? What kind of thin pool did you use (on top of loop devices?). Did you restart docker or it happens very first time you start docker?
In some cases it might happen that docker is using loop devices and then docker is restarted, it can happen that pool was not shutdown because some device was busy somewhere. Upon restart docker
finds the pool already there and it has no knowledge that loopback devices are being used and it does not issue discard.  I am wondering if you ran in to that situaiton?

CentOS 7
Docker version 1.6.2.el7, build c3ca5bb/1.6.2
Before image cleanup
[root@b2 ~]# docker info
Containers: 1
Images: 320
Storage Driver: devicemapper
 Pool Name: docker-8:2-1076248517-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: xfs
 Data file:
 Metadata file:
 Data Space Used: 10.74 GB
 Data Space Total: 107.4 GB
 Data Space Available: 96.63 GB
 Metadata Space Used: 22.8 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.125 GB
 Udev Sync Supported: true
 Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Kernel Version: 3.10.63-11.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 32
Total Memory: 31.52 GiB

[root@b2 ~]# df
Filesystem     1K-blocks     Used Available Use% Mounted on
/dev/sda2      483443064 13100584 470342480   3% /

After some docker rmi
[root@b2 ~]# docker info
Containers: 1
Images: 143
Storage Driver: devicemapper
 Pool Name: docker-8:2-1076248517-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: xfs
 Data file:
 Metadata file:
 Data Space Used: 7.215 GB
 Data Space Total: 107.4 GB
 Data Space Available: 100.2 GB
 Metadata Space Used: 14.68 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.133 GB
 Udev Sync Supported: true
 Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Kernel Version: 3.10.63-11.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 32
Total Memory: 31.52 GiB

[root@b2 ~]# df
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/sda2      483443064 9665924 473777140   2% /

@alexDrinkwater
Ok, just now I booted up a centos7 image and tried docker 1.6 and I don't see the problem. Here are the steps.
[root@centos7-generic ~]# docker version
Client version: 1.6.0
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): 8aae715/1.6.0
OS/Arch (client): linux/amd64
Server version: 1.6.0
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): 8aae715/1.6.0
OS/Arch (server): linux/amd64
There is already one image imported.
[root@centos7-generic ~]# docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
docker.io/fedora    latest              ded7cd95e059        2 weeks ago         186.5 MB
Here is the output of docker info.
root@centos7-generic ~]# docker info
Containers: 0
Images: 2
Storage Driver: devicemapper
Pool Name: docker-253:1-50331788-pool
Pool Blocksize: 65.54 kB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 525.5 MB
Data Space Total: 107.4 GB
Data Space Available: 20 GB
Metadata Space Used: 892.9 kB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.147 GB
Udev Sync Supported: true
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Kernel Version: 3.10.0-229.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 2
[root@centos7-generic ~]# docker rmi fedora
Untagged: fedora:latest
Deleted: ded7cd95e059788f2586a51c275a4f151653779d6a7f4dad77c2bd34601d94e4
Deleted: 48ecf305d2cf7046c1f5f8fcbcd4994403173441d4a7f125b1bb0ceead9de731
[root@centos7-generic ~]# docker info
Containers: 0
Images: 0
Storage Driver: devicemapper
Pool Name: docker-253:1-50331788-pool
Pool Blocksize: 65.54 kB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 307.2 MB
Data Space Total: 107.4 GB
Data Space Available: 20.22 GB
Metadata Space Used: 733.2 kB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.147 GB
Udev Sync Supported: true
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Kernel Version: 3.10.0-229.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 2
Notice Data usage dropped from 525MB to 307MB. So deleting an image did get me the space back.

Ok, I forgot to paste the size of loop file before and after the operation. here it is.

Before image pull.
$cd /var/lib/docker/devicemapper/devicemapper
$ du -h data
293M	data

After image pull
[root@centos7-generic devicemapper]# du -h data
499M	data

After image deletion
[root@centos7-generic devicemapper]# du -h data
293M	data

So my loop file actually shrunk after image deletion. I think that's what you are complaining about.

@stanislavb
Are you using loop files? They are not visible in your docker info. I think you are running into same issue where when docker started pool was already there.
Can you check the size of loop backing files (/var/lib/docker/devicemapper/devicemapper/data) and make sure that file shrinks after image deletion. (du -h ).

@rhvgoyal
CentOS 7, Docker version 1.6.2.el7, build c3ca5bb/1.6.2
[root@b2 ~]# du /var/lib/docker/devicemapper/devicemapper/data
7041272	/var/lib/docker/devicemapper/devicemapper/data
[root@b2 ~]# docker rmi $(docker images -q)
...
[root@b2 ~]# du /var/lib/docker/devicemapper/devicemapper/data
5617292	/var/lib/docker/devicemapper/devicemapper/data

@stanislavb
Ok, So loop device size is going down indeed.
Also I verified from the code. We will issue discard even if upon restart docker found thin pool already there.
    // By default, don't do blk discard hack on raw devices, its rarely useful and is expensive
    if !foundBlkDiscard && (devices.dataDevice != "" || devices.thinPoolDevice != "") {
            devices.doBlkDiscard = false
    }

As per above code, discards are disabled only if user either passed in a data block device or a thin pool. Given that we did not pass either, discards are still on. That's why it is working for you.
@alexDrinkwater Now we have two data points that it is fixed in 1.6. Do you still have concerns in closing this issue.

CentOS 6.6
Docker version 1.5.0, build a8a31ef/1.5.0
Before image cleanup
[root@b1 ~]# docker info
Containers: 2
Images: 2996
Storage Driver: devicemapper
 Pool Name: docker-8:2-21890120-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: extfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 43.24 GB
 Data Space Total: 107.4 GB
 Metadata Space Used: 117.1 MB
 Metadata Space Total: 2.147 GB
 Udev Sync Supported: true
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.89-RHEL6 (2014-09-01)
Execution Driver: native-0.2
Kernel Version: 2.6.32-504.16.2.el6.x86_64
Operating System: <unknown>
CPUs: 32
Total Memory: 31.33 GiB

[root@b1 ~]# df
Filesystem     1K-blocks     Used Available Use% Mounted on
/dev/sda2      475957304 53995488 397777856  12% /

Checked size of data file at 1929 images remaining
[root@b1 ~]# du -h /var/lib/docker/devicemapper/devicemapper/data
30G	/var/lib/docker/devicemapper/devicemapper/data

After some docker rmi
[root@b1 ~]# docker info
Containers: 2
Images: 497
Storage Driver: devicemapper
 Pool Name: docker-8:2-21890120-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: extfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 13.39 GB
 Data Space Total: 107.4 GB
 Metadata Space Used: 27.87 MB
 Metadata Space Total: 2.147 GB
 Udev Sync Supported: true
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.89-RHEL6 (2014-09-01)
Execution Driver: native-0.2
Kernel Version: 2.6.32-504.16.2.el6.x86_64
Operating System: <unknown>
CPUs: 32
Total Memory: 31.33 GiB

[root@b1 ~]# df
Filesystem     1K-blocks     Used Available Use% Mounted on
/dev/sda2      475957304 25606060 426167284   6% /
[root@b1 ~]# du -h /var/lib/docker/devicemapper/devicemapper/data
13G	/var/lib/docker/devicemapper/devicemapper/data

Ok, so even in 1.5 it works.

Docker version 1.6.0, build 8aae715/1.6.0
Before pull:
/dev/mapper/os-var                             3.9G  750M  2.9G  21% /var

Containers: 0
Images: 0
Storage Driver: devicemapper
 Pool Name: docker-253:3-247472-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: extfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 307.2 MB
 Data Space Total: 107.4 GB
 Data Space Available: 3.308 GB
 Metadata Space Used: 733.2 kB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.147 GB
 Udev Sync Supported: true
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Kernel Version: 3.10.0-123.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 4
Total Memory: 7.634 GiB

after pull:
/dev/mapper/os-var                             3.9G  815M  2.9G  23% /var

Containers: 0
Images: 22
Storage Driver: devicemapper
 Pool Name: docker-253:3-247472-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: extfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 639.9 MB
 Data Space Total: 107.4 GB
 Data Space Available: 3.24 GB
 Metadata Space Used: 1.438 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.146 GB
 Udev Sync Supported: true
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Kernel Version: 3.10.0-123.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 4
Total Memory: 7.634 GiB

remove image:
sudo docker rmi 617ef0e7677f
Untagged: docker.io/ghost:latest
Deleted: 617ef0e7677fbff322b8f3af29b9795314a333876d66b49e1ddede15229dccea
Deleted: e68d1ee297168a0b3c3e21ff7a9ab14f7df8edf4525e008fcaf3abc83c731387
Deleted: 4eee85e0d29b6836d3d46ff3b84b0292626df5c8428da0aeeb916d33b6fb7642
Deleted: d0e1169bd5617096df1776902e191f23c07ac0fb5b365c5664fd3d4a554e4c8e
Deleted: e983dda26a2392947fffa4cc37b36251b84bbc74c95ce9c353b80a9c8e63d70f
Deleted: 64b0f4c09fde536675106331c31636f8478ed0cf5c2c7aa274d464216e470b3f
Deleted: c0313e19f503a4770c00250049419a014972ba8f3458524d61b377b8fd289ef0
Deleted: ff1093062f402021a7574b3c40692d2c6dc7aec07d66e17caa6c35df19bad091
Deleted: 37b39063e0c0f3c1a8c90d304ad7ba6b47e14242ff60f6f5d091b280258e0ff3
Deleted: 6e878a0c2422a5496d6bfc5eaf1facbc48f66a8e437fdd7db18d8944b20f7072
Deleted: a10b93053713fb726beaea993bad52b39ca92c5ce6b646dbc7d9cd96016ee9bc
Deleted: 1324d506b72f2a602a8f55c5224708e8ff02dec86b5fe46462a1d73aafb32075
Deleted: 70014f2a472b03d0cfde21d99c601db25f62de1d6c8497cadd6e05743c09d5a1
Deleted: f81216b6a47e2f51c80ff56044a5120d4b8cb76e9ea5990ba08ed073e97fd429
Deleted: 286e04b15dcfe587da0d8b6b359d6ae74c3ef299b868183859508304153ceaca
Deleted: 1dbb5edeebca3ae23a32fcf600015df645a788747553287548ce67126b206ab7
Deleted: 8807133d30b36044c670a06b087b6b61082b660a61fef651f0871283c5505bff
Deleted: 4c0283973ca2335a9ae82168956e4add2e6f2f13fd31e16473d695912e34d974
Deleted: 95d6d696e46933c9d063b5d6328b1a92b988462f1277d74d42bbbdd568efc220
Deleted: 80565b90e8eb693f03cea0411aadb45f21f2bcfe39f6b0fda8cf351eaee1f81b
Deleted: df2a0347c9d081fa05ecb83669dcae5830c67b0676a6d6358218e55d8a45969c
Deleted: 39bb80489af75406073b5364c9c326134015140e1f7976a370a8bd446889e6f8

after delete:
/dev/mapper/os-var                             3.9G  814M  2.9G  23% /var

Containers: 0
Images: 0
Storage Driver: devicemapper
 Pool Name: docker-253:3-247472-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: extfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 307.2 MB
 Data Space Total: 107.4 GB
 Data Space Available: 3.24 GB
 Metadata Space Used: 733.2 kB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.147 GB
 Udev Sync Supported: true
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Kernel Version: 3.10.0-123.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 4
Total Memory: 7.634 GiB

Let me know if you any more details of my machine or versions etc.

@alexDrinkwater
Looks like you are mounting a separate device on /var. What's the file system on that device and what are the mount options?

Also can you give me following.

dmsetup table
dmsetup status
cat /etc/sysconfig/docker-storage

Can you try the simpler case of keeping /var on root itself with xfs file system. Trying to narrow down the problem a bit.

@alexDrinkwater
I mounted another disk with ext4 partition on /var/lib/docker so that loopback backing files are on ext4 (instead of xfs) and still things are working for me.
/dev/vdb1 on /var/lib/docker type ext4 (rw,relatime,seclabel,data=ordered)
Really not sure what's special about your setup.

To be specific,

Before pull
/dev/vdb1       9.8G  338M  8.9G   4% /var/lib/docker

After image pull
/dev/vdb1       9.8G  544M  8.7G   6% /var/lib/docker

After image delete
/dev/vdb1       9.8G  338M  8.9G   4% /var/lib/docker

@alexDrinkwater
The other major difference between your setup and my setup is kernel version. I am using "3.10.0-229.el7.x86_64" while you seem to be using "3.10.0-123.el7.x86_64". Can you upgrade your kernel and give it a try.

Thanks for the help @rhvgoyal. I wouldn't be able to update the kernel today. But here are the settings you asked for:
/dev/mapper/os-var /var ext3 rw,relatime,data=ordered 0 0

vgdocker-lvdocker: 0 62906368 linear 8:17 2048
os-tmp: 0 8388608 linear 8:2 5244928
os-var: 0 8388608 linear 8:2 13633536
os-swap: 0 5242880 linear 8:2 2048
os-root: 0 8388608 linear 8:2 22022144
docker-253:3-247472-pool: 0 209715200 thin-pool 7:1 7:0 128 32768 1 skip_block_zeroing

vgdocker-lvdocker: 0 62906368 linear
os-tmp: 0 8388608 linear
os-var: 0 8388608 linear
os-swap: 0 5242880 linear
os-root: 0 8388608 linear
docker-253:3-247472-pool: 0 209715200 thin-pool 79 179/524288 4688/1638400 - rw discard_passdown queue_if_no_space

DOCKER_STORAGE_OPTIONS=

I don't have any custom storage options configured.

Found it. I think it has something to do with ext3 file system. I can see the problem too. Try a different file system, say ext4 or xfs and you would not face the problem.
/dev/vdb1 on /var/lib/docker type ext3 (rw,relatime,seclabel,data=ordered)

Before pull image
[root@centos7-generic-2 devicemapper]# du -h data
293M	data

After pull image
[root@centos7-generic-2 devicemapper]# du -h data
501M	data

After image removal
[root@centos7-generic-2 devicemapper]# du -h data
501M	data

So for some reason if loopback file is on ext3 filesystem, it does not work. May be ext3 is old enough that loop discards don't work on it.

Thanks, as soon as I saw the filesystem was ext3 I thought that could be it. I guess if you were able to reproduce the problem on ext3 then we could call this issue closed. It will be a good reference for others.

For the reference, my working examples were Docker 1.6.2 on xfs filesystem and Docker 1.6.0 and 1.5.0 on ext4.

@vbatts
Can we close this. Space reclaim works just fine on xfs and ext4. ext3 seems to be too old to support it.

@rhvgoyal @vbatts IIRC, there's a compatibility-check that checks the underlying file-system. Perhaps we should add a check for ext3 if that is the issue here?

Look driver uses fallocate() to punch a hole in file when discard comes in. Linux man page of fallocate() verifies that it is not supported on ext3.
/*
Not all  filesystems  support  FALLOC_FL_PUNCH_HOLE;  if  a  filesystem doesn't  support the operation, an error is returned.  The operation is supported on at least the following filesystems:
*  XFS (since Linux 2.6.38)
*  ext4 (since Linux 3.0)
*  Btrfs (since Linux 3.7)
*  tmpfs (since Linux 3.5)
*/

@thaJeztah
If somebody wants to run docker on ext3, I guess we should allow that. Just that they will not get discard support hence loop file size will not shrink when image/containers are deleted.

@rhvgoyal how about showing this in docker info? Similar to the Udev Sync Supported output.

@thaJeztah
We already have an entry in docker info "Backing filesystem". I am not sure why does it say extfs instead of being precise about ext2/ext3/ext4.
May be a warning at startup time in logs should do here. Something similar to warning about using loop devices for thin pool.

@thaJeztah
And I think we should do it only if lot of people are impacted by this. If not, then we are just creating more work and code and it might not be worth it.

the syscall.Statfs_t struct, with the Type field on ext2, ext3, and ext4 all return 0xef53, and that is what we are using to detect filesystem magic.

the syscall.Statfs_t struct, with the Type field on ext2, ext3, and ext4 all return 0xef53, and that is what we are using to detect filesystem magic.

Bummer. Would have been good to have that information to make it easier to identify reported issues.
I guess, we should just close this then?

closing as this is an issue with ext3 being outdated. Please use ext4 or xfs

do you think this could be better documented in the main Docker documentation somehow?
Whether the issue is with the filesystem or not, look at how much confusion this has generated across two recent issues.
Thanks.

reterring to issue #9786

@dbabits yes, I think mentioning that ext3 has some issues could be worth mentioning in the docs. No idea yet what an appropriate location would be.

Have same/similar problem with XFS.
Was on kernel "3.10.0-123.el7.x86_64", but updated now on "3.10.0-229.el7.x86_64".
Eveything is deleted (container, images) but data still contains 100GB.
Any ideas, help?
[root@docker0101 ~]# ls -alh /data/docker/devicemapper/devicemapper/
total 80G
drwx------ 2 root root   32 Jun  8 16:48 .
drwx------ 5 root root   50 Jun  9 07:16 ..
-rw------- 1 root root 100G Jul 16 21:33 data
-rw------- 1 root root 2.0G Jul 17 09:20 metadata
[root@docker0101 ~]# uname -a
Linux docker0101 3.10.0-229.7.2.el7.x86_64 #1 SMP Tue Jun 23 22:06:11 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
[root@docker0101 ~]# cat /etc/redhat-release
CentOS Linux release 7.1.1503 (Core)
[root@docker0101 ~]# docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
[root@docker0101 ~]# docker images -a
REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
[root@docker0101 ~]# docker info
Containers: 0
Images: 0
Storage Driver: devicemapper
Pool Name: docker-253:0-268599424-pool
Pool Blocksize: 65.54 kB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 85.61 GB
Data Space Total: 107.4 GB
Data Space Available: 40.91 MB
Metadata Space Used: 211.4 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 40.91 MB
Udev Sync Supported: true
Data loop file: /data/docker/devicemapper/devicemapper/data
Metadata loop file: /data/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Kernel Version: 3.10.0-229.7.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 4
Total Memory: 11.58 GiB
Name: docker0101
[root@docker0101 ~]# dmsetup table
vg1-lvol0: 0 167772160 linear 8:16 2048
docker-253:0-268599424-pool: 0 209715200 thin-pool 7:1 7:0 128 32768 1 skip_block_zeroing
[root@docker0101 ~]# dmsetup status
vg1-lvol0: 0 167772160 linear
docker-253:0-268599424-pool: 0 209715200 thin-pool 71359 51606/524288 1306264/1638400 - ro discard_passdown queue_if_no_space
[root@docker0101 ~]# cat /etc/sysconfig/docker-storage
This file may be automatically generated by an installation program.
By default, Docker uses a loopback-mounted sparse file in
/var/lib/docker.  The loopback makes it slower, and there are some
restrictive defaults, such as 100GB max storage.
If your installation did not set a custom storage for Docker, you
may do it below.
Example: Use a custom pair of raw logical volumes (one for metadata,
one for data).
DOCKER_STORAGE_OPTIONS = --storage-opt dm.metadatadev=/dev/mylogvol/my-docker-metadata --storage-opt dm.datadev=/dev/mylogvol/my-docker-data
DOCKER_STORAGE_OPTIONS=
[root@docker0101 ~]#

@tomlux The devicemapper loopback mode you're using is mostly meant as a way to easily toy around with docker. for serious work, the loopback will be slower, and will have some limitations. I'd very highly recommend  having a read over http://www.projectatomic.io/docs/docker-storage-recommendation/
You'll get better performance, and won't hit things like this, assuming you've applied all the system updates.

@tomlux
can you add "-s" to ls. That will give you actual blocks allocated to data and metadata files. Right now it is showing apparent size of the files.
docker info output is intriguing though. It seems to show high usage.

root@docker0101# ls -als /data/docker/devicemapper/devicemapper/
total 83783524
0 drwx  2 root root                     32 Jun  8 16:48 .
0 drwx  5 root root                     50 Jun  9 07:16 ..
83496364 -rw  1 root root<PHONE_NUMBER>00 Jul 16 21:33 data
207160 -rw  1 root root<PHONE_NUMBER> Jul 17 09:20 metadata

@tomlux
So actual size of data and metadata files looks small. You can use "ls -alsh" so that sizes are more readable.
So data file size seems to be around 79MB and metadata file size is around 202KB.
I think somehow thin pool's stats about number of blocks used does not look right. Does a reboot of the machine fix the problem?

I did a reboot after the kernel update without success.

Ok, so loop files are big and pool thinks it has lots of used blocks. So something is using those blocks. Can you give me output of.

docker ps -a
docker images
ls /var/lib/dokcer/devicemapper/metadata/ | wc -l
shutdown docker and run following.
thin_dump /var/lib/docker/devicemapper/devicemapper/metadata | grep "device dev_id" | wc -l

This will tell how many thin devices are there in the pool.

Hmm,
now we hae DEAD containers but no images.
Did already a reboot.
Seems to be something wrong with the devicemapper.
I don't want to spam this issue.
I can also "rm -f /var/lib/docker" and rebuild my containers. Everything is scripted.

do "dmsetup status"
Looks like pool is in bad shape and most likely it needs to be repaired.

Hi,
I seemed to have the same problem under Ubuntu 14.04. However the cause where unwanted volumes (cf. blog http://blog.yohanliyanage.com/2015/05/docker-clean-up-after-yourself/). Running the command
docker run -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/docker:/var/lib/docker --rm martin/docker-cleanup-volumes

released a lot of disk space.

This isn't fixed in any meaningful way. On a fully up-to-date Ubuntu 14.04.x installation (the latest LTS release) and with the latest version of Docker (installed via $ wget -qO- https://get.docker.com/ | sh), Docker will continuously leak space with no easy way to reclaim. docker stop $(docker ps -q) && docker rm $(docker ps -q -a) && docker rmi $(docker images -q) only releases a small amount of space.
The only way to reclaim all space is with the following hack:
``bash
$ sudo service docker stop
$ sudo rm -rf /var/lib/docker
$ sudo service docker start

Which then requires re-pulling any images you might need.

@bkeroackdsc could that space also be related to "orphaned" volumes?

I agree with @bkeroackdsc this is not solved.
I asked @rhvgoyal  why he wanted to close this case so much.
at the end, I dropped from docker specifically because of this issue.
it is killing this product.
orphaned or not orphaned, there is no good, easy, way to cleanup space.
there needs to be a docker cli option for cleanup, a good status report cli option and also some sort of monitoring for the diskspace as this issue happens to too many people on many platforms.

@bkeroackdsc @sagiegurari
Reason I'm asking is that orphaned volumes are not related to this issue,
not related to devicemapper.
Orphaned volumes are not a bug, but a misconception that volumes defined
for a container are automatically deleted when the container is deleted.
This is not the case (and by design), because volumes can contain data
that should persist after a container is deleted.
To delete a volume together with a container, use docker rm -v [mycontainer].
Volume management functions will be added (see https://github.com/docker/docker/pull/14242 and https://github.com/docker/docker/issues/8363),
and will allow you to manage "orphaned" volumes.
A growing size of /var/lib/docker does not have to be an indication
that devicemapper is leaking data, because a grow in size of that directory
can also be the result of (orphaned) volumes that have not been cleaned up
by the user (docker stores all its data in that path).

I really hope those 2 items will give the needed capabilities.
I did understand the second item, but the first (#14242), doesn't explain at the top what it is about apart of that its a volume api (meaning, not sure what capabilities it gives).

@sagiegurari it's part of the requirements to implement image management (there are some other open PR's issues). End goals is to make volumes a first-class citizen in Docker, that can be created/deleted/management separate of the containers that make use of them.

@swachter Thanks for posting that workaround, I reclaimed 6GB with the aforementioned image.

we fixed the leaky volume issue with a dirty hack, since it was preventing the docker daemon from starting up before the service timed out on our high-churn docker hosts:
PRODUCTION<EMAIL_ADDRESS>~]$ cat /etc/systemd/system/docker.service.d/docker-high-churn.conf
[Service]
ExecStartPre=-/bin/rm -rf /var/lib/docker/containers
ExecStopPost=-/bin/rm -rf /var/lib/docker/volumes

which fixes the issue without flushing the pre-cached images.

Can we discuss the issue of leaky volumes in a separate issue. Discussing it here gives the impression that it is a device mapper issue while it is not.

@tomlux @rhvgoyal
Did you ever come to a conclusion on what occurring in the CentOS 7 box? My docker host is nearly identical, and I was experiencing the same issue. I followed through to the point in which @rhvgoyal asked to run the thin_dump command. After, I went to start the docker daemon up and it wouldn't start. I've just deleted /var/lib/docker and restarted since then, but I just wanted to know if a resolution was found as I (as well as others) may run into it again.
[root@Docker_Sandbox_00 devicemapper]# thin_dump /var/lib/docker/devicemapper/devicemapper/metadata | grep "device dev_id" | wc -l
102

[root@Docker_Sandbox_00 devicemapper]# systemctl start docker
Job for docker.service failed. See 'systemctl status docker.service' and 'journalctl -xn' for details.
 [root@Docker_Sandbox_00 devicemapper]# systemctl -l status docker.service
docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled)
   Active: failed (Result: exit-code) since Tue 2015-10-27 08:24:47 PDT; 37s ago
     Docs: https://docs.docker.com
  Process: 45244 ExecStart=/usr/bin/docker daemon -H fd:// (code=exited, status=1/FAILURE)
 Main PID: 45244 (code=exited, status=1/FAILURE)

Oct 27 08:24:45 Docker_Sandbox_00 systemd[1]: Starting Docker Application Container Engine...
Oct 27 08:24:46 Docker_Sandbox_00 docker[45244]: time="2015-10-27T08:24:46.512617474-07:00" level=info msg="[graphdriver] using prior storage driver \"devicemapper\""
Oct 27 08:24:46 Docker_Sandbox_00 docker[45244]: time="2015-10-27T08:24:46.526637164-07:00" level=info msg="Option DefaultDriver: bridge"
Oct 27 08:24:46 Docker_Sandbox_00 docker[45244]: time="2015-10-27T08:24:46.526719113-07:00" level=info msg="Option DefaultNetwork: bridge"
Oct 27 08:24:46 Docker_Sandbox_00 docker[45244]: time="2015-10-27T08:24:46.589016574-07:00" level=warning msg="Running modprobe bridge nf_nat br_netfilter failed with message: modprobe: WARNING: Module br_netfilter not found.\n, error: exit status 1"
Oct 27 08:24:46 Docker_Sandbox_00 docker[45244]: time="2015-10-27T08:24:46.625324632-07:00" level=info msg="Firewalld running: true"
Oct 27 08:24:47 Docker_Sandbox_00 docker[45244]: time="2015-10-27T08:24:47.142468904-07:00" level=fatal msg="Error starting daemon: Unable to open the database file: unable to open database file"
Oct 27 08:24:47 Docker_Sandbox_00 systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Oct 27 08:24:47 Docker_Sandbox_00 systemd[1]: Failed to start Docker Application Container Engine.
Oct 27 08:24:47 Docker_Sandbox_00 systemd[1]: Unit docker.service entered failed state.

[root@Docker_Sandbox_00 devicemapper]# df -ah
Filesystem                                  Size  Used Avail Use% Mounted on
/dev/vdb1                                    20G   20G   24K 100% /var/lib/docker
[root@Docker_Sandbox_00 devicemapper]# du -sh /var/lib/docker/devicemapper/devicemapper/data
20G     /var/lib/docker/devicemapper/devicemapper/data

@tomlux @rhvgoyal
I found the source of my issue. Despite how it appeared, it was unrelated to the issues in the thread. It's just stemmed from a misunderstanding with how docker worked. All, the dead containers were still holding on to the disk space they had allocated during their execution. I just had to remove all the container carcasses to free up the disk space. I do remember this coming up in this thread, but I thought is only regarded mounted volumes, not the disk space allocated by the container.
# Beware this removes ALL containers
docker rm -f $(docker ps -aq)

@tomlux This may have been your issue as well since your output of docker ps -a showed several Dead containers.

From this discussion, it doesn't look like the issue has been fixed. Is that the case?
I am seeing this in version 1.8.3.

docker rm is not freeing container's disk space
boot2docker recent in OS X VirtualBox.  OS X fully patched.
I'm building a fat (47 GB) container and it has a problem indicating I should rebuild the container.  So I stopped the container and did docker rm.  Double-checking using docker ssh  'df -h', I find disk usage is still 47 GB.  The container has 75 GB.
So I'm going to need to kill the docker VM again.
Can we get this done?

@awolfe-silversky is disk-space inside the VM returned? If it's outside the VM, this may be unrelated.

@awolfe-silversky also; did you remove the image as well? removing just the container may not help much if the image is still there

@awolfe-silversky this issue is about devicemapper - and if you're using docker-machine/boot2docker, then you're much more likely to be running aufs. I also wonder if you've docker rmi'd your big image.
its worth running docker images, and docker info to see if things really are as dire as you make it sound :)
(yes, if you still have the vm, and the image is removed, then we should open a new issue and debug further, as you've found a weird corner case)

I did not remove the image.  It is from an internal docker registry.
I used an awk script to size the images - total 6.9 GB.
docker images -a | awk '(FNR > 1) { imgSpace = imgSpace + $(NF - 1); }
END { print "Image space is " imgSpace; }'
Image space is 6909.01

It's dirty but I know that all the image sizes are in MB.
Here's how I was trying to diagnose usage:
 file:///Andrew-Wolfe-MacBook-Pro.local/Users/awolfe/DataStores
awolfe_10063: docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

 file:///Andrew-Wolfe-MacBook-Pro.local/Users/awolfe/DataStores
awolfe_10064: docker-machine ssh 'awolfe-dbhost' 'df -h'
Filesystem                Size      Used Available Use% Mounted on
tmpfs                     7.0G    123.8M      6.9G   2% /
tmpfs                     3.9G         0      3.9G   0% /dev/shm
/dev/sda1                71.0G     47.2G     20.2G  70% /mnt/sda1
cgroup                    3.9G         0      3.9G   0% /sys/fs/cgroup
none                    464.8G    379.9G     84.8G  82% /Users
/dev/sda1                71.0G     47.2G     20.2G  70% /mnt/sda1/var/lib/docker/aufs

Same here with ZFS backend, I started with 1.7 and updated to 1.10.0 last night;
% docker images -a | awk '(FNR > 1) { imgSpace = imgSpace + $(NF - 1); }
[pipe quote]> END { print "Image space is " imgSpace; }'
Image space is 13400
[~]-[kash@labmouse]
% sudo zfs get used extra/docker
NAME          PROPERTY  VALUE  SOURCE
extra/docker  used      25.1G  -

Right now I have box with 0 images.
docker volume ls -qf dangling=true shows nothing.
docker volume ls shows a lot of volumes, which are, by definition, orphaned, since there's no-images to own them.
docker volume rm $(docker volume ls) shows lots of such messages:
Error response from daemon: get local: no such volume
Error response from daemon: Conflict: remove 6989acc79fd53d26e3d4668117a7cb6fbd2998e6214d5c4843ee9fceda66fb14: volume is in use - [77e0eddb05f2b53e22cca97aa8bdcd51620c94acd2020b04b779e485c7563c57]

Device mapper directory eats up 30 GiG.
Docker version 1.10.2, build c3959b1
CentOS 7,  3.10.0-327.10.1.el7.x86_64

Data Space Used: 33.33 GB
 Data Space Total: 107.4 GB
 Data Space Available: 915.5 MB
 Metadata Space Used: 247 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 915.5 MB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.107-RHEL7 (2015-12-01)

Also, why is default installation uses 'strongly discouraged' storage option?
Why wasn't I told so at installation?

I have exactly the same problem here on an Amazon Linux EC2 instance.
Linux ip-172-31-25-154 4.4.5-15.26.amzn1.x86_64 #1 SMP Wed Mar 16 17:15:34 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
On instances where I install new docker images on a regular basis the only solution is to do the following:
service docker stop
yum remove docker -y
rm -rf /var/lib/docker
yum install docker -y
service docker start

I don't really think such a thing is acceptable in a production environment

As this bug is there for years and is seem that is not closed yet, could you put in the docker documentation about devidemapper how to destroy safety all docker information?
i mean, in this page: https://docs.docker.com/engine/userguide/storagedriver/device-mapper-driver/
put something like "Cleaning device mapper" and how to do it.
I will try to do rm -rf /var/lib/docker but I'm not feel confortable doing that. Can somebody tell me if is safe?

@mercuriete On your dev machine just uninstall docker, delete the directory and reinstall it. Works fine.

@ir-fuel: I just did that and now I have this:
$ sudo service docker-engine start
Redirecting to /bin/systemctl start  docker-engine.service
Failed to start docker-engine.service: Unit docker-engine.service failed to load: No such file or directory.
$ uname -a
Linux CentOS7 3.10.0-327.18.2.el7.x86_64 #1 SMP Thu May 12 11:03:55 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

I'm using service docker start

@ir-fuel thanks, it works fine. :+1:

Reinstalling docker to release disk space is the most ridiculous answer that I came across while looking for a solution for this issue. Not only that's a waste of time, it's not even allowed in most environments. It's a good way to get paid if you're an hourly worker.

I completely agree. It's amazing that a product like Docker just keeps on eating away disk space with nothing you can do about it except for uninstall/reinstall.

Checking in to this issue yet another time to see nothing has changed. +1

This issue is marked closed.  We need a resolution.  No workaround, no reconfiguration.  What is the real status, and what are the configuration settings that are implicated?  Dropping and recreating a production Docker node is not acceptable.

workaround is to avoid using the docker device-mapper driver, unfortunately.

And what is the alternative? How do we go about avoiding this?

If this is not recommended feature to use - why is it silent and default
one? If you do not care for people using devicemapper - I might be even ok
with this. But do inform the user about it! Do you realize the amount of
headache people have due to this amazing 'workaround' you settled on??
On 23 Jun 2016 4:32 p.m., "kpande"<EMAIL_ADDRESS>wrote:

workaround is to avoid using the docker device-mapper driver,
unfortunately.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/docker/docker/issues/3182#issuecomment-228068397, or mute
the thread
https://github.com/notifications/unsubscribe/ACRlBbL5BDn3qMzUW_UiaALB32anUER4ks5qOpkJgaJpZM4BTlBd
.

no one forced you to use Docker.

There's always rkt…

the general bitchy unhelpful snark is no doubt why no-one from upstream cares to give you proper answers.

no one forced you to use Docker.

That's like Oracle telling a Java developer to use PHP due to a JVM bug. That's also not consistent with the elevator pitch here

Three years ago, Docker made an esoteric Linux kernel technology called containerization simple and accessible to everyone.

I'm sure a lot of people are grateful that Docker took off like it did and that couldn't have happened without volunteering from the community. However, it shouldn't be this hard to admit that it has it's problems too without implicitly dropping the "I'm a upstream contributor so shut up and listen" line whenever someone brings up an unlikable point.

Wait. I did report an issue, provided the details of my machine and setup,
which I'm not obliged to. None of devteam responded to my and others bug
reports in half a year period. Now I stated this fact, you call my behavior
bitchy? Do you even open-source? I'm looking for Go project to work on, and
it will not be Docker, I give you that. Is this your goal?
On 23 Jun 2016 16:45, "gregory grey"<EMAIL_ADDRESS>wrote:

If this is not recommended feature to use - why is it silent and default
one? If you do not care for people using devicemapper - I might be even ok
with this. But do inform the user about it! Do you realize the amount of
headache people have due to this amazing 'workaround' you settled on??
On 23 Jun 2016 4:32 p.m., "kpande"<EMAIL_ADDRESS>wrote:

workaround is to avoid using the docker device-mapper driver,
unfortunately.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/docker/docker/issues/3182#issuecomment-228068397,
or mute the thread
https://github.com/notifications/unsubscribe/ACRlBbL5BDn3qMzUW_UiaALB32anUER4ks5qOpkJgaJpZM4BTlBd
.

First of all, if you still have this problem, please open a new issue, and

Wait. I did report an issue

You replied on a 3 year old, closed issue; following the discussion above, the original issue was resolved. Your issue may be the same, but needs more research to be sure; the errors you're reporing indicate that it may actually be something else.
I really recommend to open a new issue, not commenting on a closed issue

provided the details of my machine and setup, which I'm not obliged to.

You're not obliged to, but without any information to go on, it's unlikely to be resolved so when reporting a bug, please include the information that's asked for in the template:
https://raw.githubusercontent.com/docker/docker/master/.github/ISSUE_TEMPLATE.md

None of devteam responded to my and others bug reports in half a year period.

If you mean "one of the maintainers", please keep in mind that there's almost 24000 issues and PRs, and less than 20 maintainers, many of whom doing that besides their daily job. Not every comment will be noticed especially not if it's on a closed issue.

If this is not recommended feature to use - why is it silent and default one?

It's the default if aufs, btrfs, and zfs are not supported, you can find the priority that's used when selecting drivers; see daemon/graphdriver/driver_linux.go. It's still above overlay, because unfortunately there's some remaining issues with that driver that some people may be affected by.
Automatic selecting a graphdriver is just to get things running; the best driver for your situation depends on your use-case. Docker cannot make that decision automatically, so this is up to the user to configure.

If you do not care for people using devicemapper - I might be even ok with this.

Reading back the discussion above, I see that the upstream devicemapper maintainers have looked into this multple times, trying to assist users reporting these issues, and resolving the issue. The issue was resolved for those that reported it, or in some cases depended on distros updating devicemapper versions. I don't think that can be considered "not caring".

Also, why is default installation uses 'strongly discouraged' storage option?

Running on loop devices is fine for getting docker running, and currently the only way to set up devicemapper automatically. For production, and to get a better performance overall, use direct-lvm, as explained in the devicemapper section in the storage driver user guide.

Why wasn't I told so at installation?

That's out of scope for the installation, really. If you're going to use some software in production, it should be reasonable to assume that you get yourself familiar with that software to know what's needed to set it up for your use case. Some maintainers even argued of the warning should be output at all. Linux is not a "holding hands" OS (does your distro show a warning that data loss can occur if you're using RAID-0? If you have ports opened in your Firewall?)

+1

Deeply reluctant as I am, to once again resurrect this ancient thread, there is still no meaningful advice in it about how to work around this issue on an existing machine encountering this issue.
This is my best effort at a tldr; for the entire thread; I hope it helps others who find this thread.
Issue encountered
Your volume has a significant (and growing) amount of space which is in /var/lib/docker and you're using ext3.
Resolution
You're out of luck. Upgrade your file system or see blowing docker away at the bottom.
Issue encountered
Your volume has a significant (and growing) amount of space which is in /var/lib/docker and you're not using ext3 (eg. system currently using xfs or ext4)
Resolution
You may be able to reclaim space on your device using standard docker commands.
Read http://blog.yohanliyanage.com/2015/05/docker-clean-up-after-yourself/
Run these commands:
docker volume ls
docker ps
docker images

If you have nothing listed in any of these, see blowing docker away at the bottom.
If you see old stale images, unused containers, etc. you can perform manual cleanup with:
# Delete 'exited' containers
docker rm -v $(docker ps -a -q -f status=exited)

# Delete 'dangling' images
docker rmi $(docker images -f "dangling=true" -q)

# Delete 'dangling' volumes
docker volume rm $(docker volume ls -qf dangling=true)

This should reclaim much of the hidden container space in the devicemapper.
Blowing docker away
Didn't work? You're out of luck.
Your best bet at this point is:
service docker stop
rm -rf /var/lib/docker
service docker start

This will destroy all your docker images. Make sure to export ones you want to keep before doing this.
Ultimately, please read https://docs.docker.com/engine/userguide/storagedriver/device-mapper-driver/#configure-direct-lvm-mode-for-production; but I hope this will assist others who find this thread.
If you have problems with using advice above open a new ticket that specifically addresses the issue you encounter and link to this issue; do not post it here.

rm -rf /var/lib/docker

You can also use nuke-graph-directory.sh.

Mar 02 04:53:40 pmdc001b dockerd[29522]: Error starting daemon: error initializing graphdriver: open /var/lib/docker/devicemapper/devicemapper/data: no such file or directory

Just ran into this issue on CentOS 7.3 and didn't want to debug devmapper issues that exist for more than 3 years, so I followed this DC/OS guide, purged the original packages, switched over to overlayfs and everything seems to work fine now: https://dcos.io/docs/1.7/administration/installing/custom/system-requirements/install-docker-centos/ (had to modify the ExecStart command for docker version 17.03 though -> "dockerd --storage-driver=overlay")
Server Version: 17.03.0-ce
Storage Driver: overlay
 Backing Filesystem: extfs
 Supports d_type: true
...
Operating System: CentOS Linux 7 (Core)

(purging volumes, images and containers didn't help. Deleting stuff in /var/lib/docker led to the problem described by @gdring2 )

Hi guys, i'm a newbie and i'm pretty sure that i'm missing something... worst part is i'm new to linux aswell)
for those who like "TL;DR":
what could be the cause of "removing" and "init-removing" tags at diff files?
root@docker:/var/lib/docker/aufs/diff# du -h -d 1
163M    ./a4bd161a8c2f706c204cf9e488c5d9337dba577ccd6bbfabb4de1eeec0a01a00-removing
12K     ./46ef3cca013b9b1fb87a4aee5a4a5d4014fd8b8fcbc5c6860a020f5343ca1b8e-removing
24K     ./70ae79a1137a4cc440c27afc020d52bc59ff19216d2a8a0f09c9370bcf363d0a-removing
1,2G    ./3cbb0a650ba9fcf85383047ee5a47c85de3dd235a3b16c4c91178919e3eb2a5b-removing
4,0K    ./089ec2260a4047dabaaf357f5d3c83648214d716db599443700a3e03a29d5dee-removing
20K     ./e64c2c4fcdfa944829257b39b86a5a880f0de919dbdc8adacad0dcb5feafccba-init-removing
52K     ./1cb17f48516508489261e959446af16dc547676b8499752c529c338a6da5093e-removing
20K     ./8c381e86d51bfc20ddfb3c3b6795457993289f92a111460480005a9eb57845be-removing
4,0K    ./03c16ec732c272188c9b12ff7ec0ea0fb3fa8384c395f90f03125139c368b9d9-removing
128M    ./0960774ebb42d31cc7c278f83fbf6d2df668e2179370963a278d012219624a65-removing
1,2G    ./893580bc85a7e8e667617b14734d9872cec25614dca95d0dc124260ebd53f55c-removing
2,1M    ./b6dd12021ff0cf168e328eeb5b26e15ba1ad0554f58830881d59db08d6410500-removing
20K     ./46ef3cca013b9b1fb87a4aee5a4a5d4014fd8b8fcbc5c6860a020f5343ca1b8e-init-removing
16K     ./82c65c268841e6f6112702556057af83a6e05ba565c2782efc9a777f5f706e30-removing
84K     ./c7c2dfbd9a8760b4141c4586dfdda4384134626d3c4995939fce7ec900570f07-removing
20K     ./3cbb0a650ba9fcf85383047ee5a47c85de3dd235a3b16c4c91178919e3eb2a5b-init-removing
4,0K    ./ddeab9b48cc75a17cee194cc26f4db794ce8f21c2c48d6d435a56769f9bda6c0-removing
20K     ./93831a8230af888df9ffb1b9522e419fa15499301e3cf80a949a9dbb7b1765ce-init-removing
12K     ./93831a8230af888df9ffb1b9522e419fa15499301e3cf80a949a9dbb7b1765ce-removing
16K     ./33e3425f23336d19fc067db68932983515ef02593f539a9a93ca6bf788b80ddb-removing
12K     ./e64c2c4fcdfa944829257b39b86a5a880f0de919dbdc8adacad0dcb5feafccba-removing
16K     ./7407c3b7636dd7c11e632146822628594f216373048e6a1d478c5f25a6bfb8c0-removing
20K     ./70ae79a1137a4cc440c27afc020d52bc59ff19216d2a8a0f09c9370bcf363d0a-init-removing
2,6G    .

so, i've encountered this problem while trying to undetstand how does docker works. The strangest thing by far is that this problem sometimes (sadly, don't know when, why or how) disappears when i'm using other VM with exact same settings.
what i have:
Docker ver. 17.06.1-ce
Ubuntu 16.04.3 LTS
filesystem ext4
what do i do:
sudo docker build .
sudo docker system prune -a

it doesn't matter if i use prune or docker rm commands (for images, containers, volumes), if i run this image or not, it simply does not free disc space. The "/docker/aufs/diff" foulder only grows.
TY for your time and greetings from Russia.

Running docker system prune freed a lot of space on my machines.
https://docs.docker.com/engine/reference/commandline/system_prune/

good

after buying one of these pieces of garbage and seeing the state of support, i returned it.

There's your first problem @misterbigstuff...you bought something that's open source?

and returned it


Fix#543 add test case for scheduler manager

When related federatedtypeconfig resources is created, scheduler and
related plugin controllers are started. If they are deleted, stop
related scheduler and plugin controllers.
@marun any comments for related test cases?

My apologies in getting back to this.  I somehow missed the changes you made 6 days ago.  It looks good to me, but you will need to rebase.

Also, please avoid putting issue references in the title of PRs or commits.  The place to put them is in the body of the description as follows:
Fixes #<issue number>

See #586 for an example of this.  Following this format will ensure that github automatically closes the issue referenced issue when the PR is merged.

rebased based on new code.
@marun

Thank you!
/approve
/lgtm


Unable to use ToastAndroid on Widgets on home screen

Everything is on the title of the issue, there is some informations:

I'm using android 13, on a pixel 5 (real device)
I've also tried using emulator

There is the line of code I'm using
ToastAndroid.show('message', ToastAndroid.LONG);
This line is in my widget task handler file, on the WIDGET_CLICK event
let me know if you want more informations, this isthe first time I open an issue

Hi,
I'm getting "Suppressing toast from package com.androidwidgetexample by user request." when trying to show a toast from the widget task handler.
I found that message in NotificationManagerService.java in the Android code.
I assume that the app needs to be in foreground in order to show Toasts.


Update Jenkins install to be up-to-date + Add Secret fetching to registry

As per our discussions, here is the update to enable Croc-Hunter to work in the latest Helm Jenkins repo + add credentials for pulling images.

@lastcoolnameleft Thanks for the contrib. I'm going to pull this into dev and confirm that it doesn't break any existing work


nolibc error?

Tag @ammarfaizi2 Just updated kernel to Linux 5.15 and pulled current master, updated defines:
prefix                        /usr
includedir                    /usr/include
libdir                        /usr/lib
libdevdir                     /usr/lib
relativelibdir
mandir                        /usr/man
datadir                       /usr/share
stringop_overflow             yes
array_bounds                  yes
__kernel_rwf_t                yes
__kernel_timespec             yes
open_how                      yes
statx                         yes
C++                           yes
has_ucontext                  yes
has_memfd_create              yes
liburing_nolibc               no
CC                            gcc
CXX                           g++
running build_ext
building 'uring._uring' extension
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Ilibs/liburing/src/include -I/opt/python/3.10/include/python3.10 -c libs/liburing/src/queue.c -o build/temp.linux-x86_64-3.10/libs/liburing/src/queue.o -Os -g0
In file included from libs/liburing/src/queue.c:5:
libs/liburing/src/syscall.h: In function ‘____sys_io_uring_register’:
libs/liburing/src/syscall.h:101:15: warning: implicit declaration of function ‘syscall’ [-Wimplicit-function-declaration]
  101 |         ret = syscall(__NR_io_uring_register, fd, opcode, arg, nr_args);
      |               ^~~~~~~
libs/liburing/src/syscall.h: In function ‘uring_madvise’:
libs/liburing/src/syscall.h:170:15: warning: implicit declaration of function ‘madvise’; did you mean ‘raise’? [-Wimplicit-function-declaration]
  170 |         ret = madvise(addr, length, advice);
      |               ^~~~~~~
      |               raise
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Ilibs/liburing/src/include -I/opt/python/3.10/include/python3.10 -c libs/liburing/src/register.c -o build/temp.linux-x86_64-3.10/libs/liburing/src/register.o -Os -g0
In file included from libs/liburing/src/register.c:5:
libs/liburing/src/syscall.h: In function ‘____sys_io_uring_register’:
libs/liburing/src/syscall.h:101:15: warning: implicit declaration of function ‘syscall’ [-Wimplicit-function-declaration]
  101 |         ret = syscall(__NR_io_uring_register, fd, opcode, arg, nr_args);
      |               ^~~~~~~
libs/liburing/src/syscall.h: In function ‘uring_madvise’:
libs/liburing/src/syscall.h:170:15: warning: implicit declaration of function ‘madvise’; did you mean ‘raise’? [-Wimplicit-function-declaration]
  170 |         ret = madvise(addr, length, advice);
      |               ^~~~~~~
      |               raise
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Ilibs/liburing/src/include -I/opt/python/3.10/include/python3.10 -c libs/liburing/src/setup.c -o build/temp.linux-x86_64-3.10/libs/liburing/src/setup.o -Os -g0
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Ilibs/liburing/src/include -I/opt/python/3.10/include/python3.10 -c libs/liburing/src/syscall.c -o build/temp.linux-x86_64-3.10/libs/liburing/src/syscall.o -Os -g0
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Ilibs/liburing/src/include -I/opt/python/3.10/include/python3.10 -c src/uring.c -o build/temp.linux-x86_64-3.10/src/uring.o -Os -g0
gcc -pthread -shared build/temp.linux-x86_64-3.10/libs/liburing/src/queue.o build/temp.linux-x86_64-3.10/libs/liburing/src/register.o build/temp.linux-x86_64-3.10/libs/liburing/src/setup.o build/temp.linux-x86_64-3.10/libs/liburing/src/syscall.o build/temp.linux-x86_64-3.10/src/uring.o -o build/lib.linux-x86_64-3.10/uring/_uring.cpython-310-x86_64-linux-gnu.so
copying build/lib.linux-x86_64-3.10/uring/_uring.cpython-310-x86_64-linux-gnu.so -> uring
running clean
removing 'build/temp.linux-x86_64-3.10' (and everything under it)
removing 'build/lib.linux-x86_64-3.10' (and everything under it)
removing 'build/bdist.linux-x86_64' (and everything under it)
'build/scripts-3.10' does not exist -- can't clean it
removing 'build'

How did you build this? Can you show the full commands you executed?
Your configure output doesn't show that you're building with nolibc option.
liburing_nolibc               no

The nolibc build works just fine on the current master branch, see the build bot there, it's all green.
To build liburing nolibc, you can just simply execute these:
./configure --nolibc;
make -j$(nproc);

I have tired it with the --nolibc, I get the same error. Doesn't matter if --nolibc is set or not liburing_nolibc no is same.

It seems you've modified the build script?
From what I can see here, you don't have this flag in your build:
-include ../config-host.h

It's where the CONFIG_NOLIBC is defined.

Works just fine here. Consult your build modification needs for further assistance.
ammarfaizi2@integral:~/project/now/liburing$ ./configure --nolibc
prefix                        /usr
includedir                    /usr/include
libdir                        /usr/lib
libdevdir                     /usr/lib
relativelibdir
mandir                        /usr/man
datadir                       /usr/share
stringop_overflow             yes
array_bounds                  yes
__kernel_rwf_t                yes
__kernel_timespec             yes
open_how                      yes
statx                         yes
C++                           yes
has_ucontext                  yes
has_memfd_create              yes
liburing_nolibc               yes
CC                            gcc
CXX                           g++
ammarfaizi2@integral:~/project/now/liburing$ make -j$(nproc) V=1
make[1]: Entering directory '/home/ammarfaizi2/project/now/liburing/src'
gcc -D_GNU_SOURCE -Iinclude/ -include ../config-host.h -nostdlib -nodefaultlibs -ffreestanding -fno-stack-protector -g -fomit-frame-pointer -O2 -Wall -Wextra -Wno-unused-parameter -Wno-sign-compare -nostdlib -nodefaultlibs -ffreestanding -fno-stack-protector -c -o setup.ol setup.c
gcc -D_GNU_SOURCE -Iinclude/ -include ../config-host.h -nostdlib -nodefaultlibs -ffreestanding -fno-stack-protector -g -fomit-frame-pointer -O2 -Wall -Wextra -Wno-unused-parameter -Wno-sign-compare -nostdlib -nodefaultlibs -ffreestanding -fno-stack-protector -c -o queue.ol queue.c
gcc -D_GNU_SOURCE -Iinclude/ -include ../config-host.h -nostdlib -nodefaultlibs -ffreestanding -fno-stack-protector -g -fomit-frame-pointer -O2 -Wall -Wextra -Wno-unused-parameter -Wno-sign-compare -nostdlib -nodefaultlibs -ffreestanding -fno-stack-protector -c -o register.ol register.c
gcc -D_GNU_SOURCE -Iinclude/ -include ../config-host.h -nostdlib -nodefaultlibs -ffreestanding -fno-stack-protector -g -fomit-frame-pointer -O2 -Wall -Wextra -Wno-unused-parameter -Wno-sign-compare -nostdlib -nodefaultlibs -ffreestanding -fno-stack-protector -c -o nolibc.ol nolibc.c
gcc -D_GNU_SOURCE -Iinclude/ -include ../config-host.h -nostdlib -nodefaultlibs -ffreestanding -fno-stack-protector -fPIC -g -fomit-frame-pointer -O2 -Wall -Wextra -Wno-unused-parameter -Wno-sign-compare -nostdlib -nodefaultlibs -ffreestanding -fno-stack-protector -c -o setup.os setup.c
gcc -D_GNU_SOURCE -Iinclude/ -include ../config-host.h -nostdlib -nodefaultlibs -ffreestanding -fno-stack-protector -fPIC -g -fomit-frame-pointer -O2 -Wall -Wextra -Wno-unused-parameter -Wno-sign-compare -nostdlib -nodefaultlibs -ffreestanding -fno-stack-protector -c -o queue.os queue.c
gcc -D_GNU_SOURCE -Iinclude/ -include ../config-host.h -nostdlib -nodefaultlibs -ffreestanding -fno-stack-protector -fPIC -g -fomit-frame-pointer -O2 -Wall -Wextra -Wno-unused-parameter -Wno-sign-compare -nostdlib -nodefaultlibs -ffreestanding -fno-stack-protector -c -o register.os register.c
gcc -D_GNU_SOURCE -Iinclude/ -include ../config-host.h -nostdlib -nodefaultlibs -ffreestanding -fno-stack-protector -fPIC -g -fomit-frame-pointer -O2 -Wall -Wextra -Wno-unused-parameter -Wno-sign-compare -nostdlib -nodefaultlibs -ffreestanding -fno-stack-protector -c -o nolibc.os nolibc.c
ar r liburing.a setup.ol queue.ol register.ol nolibc.ol
gcc -fPIC -g -fomit-frame-pointer -O2 -Wall -Wextra -Wno-unused-parameter -Wno-sign-compare -nostdlib -nodefaultlibs -ffreestanding -fno-stack-protector -shared -Wl,--version-script=liburing.map -Wl,-soname=liburing.so.2 -o liburing.so.2.1.0 setup.os queue.os register.os nolibc.os  -nostdlib -nodefaultlibs
ar: creating liburing.a
ranlib liburing.a
make[1]: Leaving directory '/home/ammarfaizi2/project/now/liburing/src'

I take Liburing source and custom build .so to be used by python.
python3 setup.py build_ext --inplace (for testing)
Project is linked to whats inside liburing/src/ even the compat.h file is auto created inside liburing/src/include/liburing
This is not the project I am testing with atm but you can visit https://github.com/YoSTEALTH/Liburing/blob/master/builder.py see how its normally done.

So you create your own build script here?
https://github.com/YoSTEALTH/Liburing/blob/master/builder.py#L44-L49
Some notes for you:

For nolibc build, you should not compile the syscall.c, instead, you need to compile nolibc.c.

You must define CONFIG_NOLIBC macro in all translation units. It can simply be done by adding this to your compile flags:

-include ../config-host.h

The file ../config-host.h is generated by the configure script.

Make sure your build flags contain all necessary options for nolibc support, such as:

-nostdlib -nodefaultlibs -ffreestanding -fno-stack-protector

All the include *.h files should be in https://github.com/YoSTEALTH/Liburing/blob/master/builder.py#L49 folder! Thus its not picking up the header file.

All the include *.h files should be in https://github.com/YoSTEALTH/Liburing/blob/master/builder.py#L49 folder! Thus its not picking up the header file.

You can add
-DCONFIG_NOLIBC

to your build flags if you don't want to include config-host.h though, but the ./configure script is useless then.

If I add -DCONFIG_NOLIBC
prefix                        /usr
includedir                    /usr/include
libdir                        /usr/lib
libdevdir                     /usr/lib
relativelibdir
mandir                        /usr/man
datadir                       /usr/share
stringop_overflow             yes
array_bounds                  yes
__kernel_rwf_t                yes
__kernel_timespec             yes
open_how                      yes
statx                         yes
C++                           yes
has_ucontext                  yes
has_memfd_create              yes
liburing_nolibc               yes
CC                            gcc
CXX                           g++
running build_ext
building 'uring._uring' extension
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Ilibs/liburing/src/include -I/opt/python/3.10/include/python3.10 -c libs/liburing/src/queue.c -o build/temp.linux-x86_64-3.10/libs/liburing/src/queue.o -Os -g0 -DCONFIG_NOLIBC
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Ilibs/liburing/src/include -I/opt/python/3.10/include/python3.10 -c libs/liburing/src/register.c -o build/temp.linux-x86_64-3.10/libs/liburing/src/register.o -Os -g0 -DCONFIG_NOLIBC
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Ilibs/liburing/src/include -I/opt/python/3.10/include/python3.10 -c libs/liburing/src/setup.c -o build/temp.linux-x86_64-3.10/libs/liburing/src/setup.o -Os -g0 -DCONFIG_NOLIBC
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Ilibs/liburing/src/include -I/opt/python/3.10/include/python3.10 -c libs/liburing/src/syscall.c -o build/temp.linux-x86_64-3.10/libs/liburing/src/syscall.o -Os -g0 -DCONFIG_NOLIBC
libs/liburing/src/syscall.c:11:3: error: #error "This file should only be compiled for libc build, or for liburing tests"
   11 | # error "This file should only be compiled for libc build, or for liburing tests"
      |   ^~~~~
error: command '/usr/bin/gcc' failed with exit code 1

Read my notes, you no (1) and (3)
https://github.com/axboe/liburing/issues/475#issuecomment-962252683

If I use nolibc.c with -DCONFIG_NOLIBC it does work! Just seems sloppy.

All of these should be handled by default by the liburing build script, it is just you that don't use the provided script.
Actually you can just add extra envars like CFLAGS to modify the compilation flag.

What do you think about:

moving all the *.h files into liburing/src/include/liburing folder?
syscall.* and nolibc.* files has if CONFIG_NOLIBC enable code else hide code?

I am just thinking what type of config users system will have, I would rather not hard code cflags if i can avoid it.

What do you think about:
1. moving all the `*.h` files into `liburing/src/include/liburing` folder?

I don't think config-host.h should be included from the userland code, it's just a generated header file that is only useful for building liburing. The userland should rely on liburing.h header.
We don't use #include "config-host.h" for that anyway, we do that include from the compilation flags.

2. `syscall.*` and `nolibc.*` files has if CONFIG_NOLIBC enable code else hide code?

I don't understand the no (2). Can you rephrase that sentence?

P.s nolibc is confusing! nolibc=yes|no, maybe libc=yes|no?

There is no yes|no value for that. By default we use libc, if we specify --nolibc, this means we don't use libc.

Ok, don't worry about my comments. Thanks @ammarfaizi2 for your help resolving this issue.

@ammarfaizi2 Actually, this time I am including config-host.h so want to get both libc/nolibc right, if I can.
setup.py file
from subprocess import run
from setuptools import Extension, find_packages, setup
from Cython.Build import cythonize

LIBC = False
PACKAGE = 'uring'
SOURCES = ['src/uring.pyx',
           'libs/liburing/src/queue.c',
           'libs/liburing/src/register.c',
           'libs/liburing/src/setup.c']
CONFIGURE = ['./configure']
COMPILE_ARGS = ['-Os', '-g0',   # optimization
                '-include', 'libs/liburing/config-host.h']
if LIBC:
    SOURCES.append('libs/liburing/src/syscall.c')
else:
    SOURCES.append('libs/liburing/src/nolibc.c')
    CONFIGURE.append('--nolibc')
    # COMPILE_ARGS.extend(('-nostdlib', '-nodefaultlibs', '-ffreestanding', '-fno-stack-protector'))

# configure liburing
run(CONFIGURE, cwd='libs/liburing')
# Note: configure is only run so all the check and required files are created.
# Note: setup process does the actual compiling part.
setup(name=PACKAGE,
      packages=find_packages(),
      ext_modules=cythonize([Extension(name=f'{PACKAGE}._{PACKAGE}',  # where the `.so` will be saved.
                                       sources=SOURCES,
                                       include_dirs=['libs/liburing/src/include'],
                                       extra_compile_args=COMPILE_ARGS)],
                            annotate=True,          # for debugging only
                            language_level='3'),    # complains even though using cython 3
      zip_safe=False)

LIBC=False works good.
prefix                        /usr
includedir                    /usr/include
libdir                        /usr/lib
libdevdir                     /usr/lib
relativelibdir
mandir                        /usr/man
datadir                       /usr/share
stringop_overflow             yes
array_bounds                  yes
__kernel_rwf_t                yes
__kernel_timespec             yes
open_how                      yes
statx                         yes
C++                           yes
has_ucontext                  yes
has_memfd_create              yes
liburing_nolibc               yes
CC                            gcc
CXX                           g++
running build_ext
building 'uring._uring' extension
creating build
creating build/temp.linux-x86_64-3.10
creating build/temp.linux-x86_64-3.10/libs
creating build/temp.linux-x86_64-3.10/libs/liburing
creating build/temp.linux-x86_64-3.10/libs/liburing/src
creating build/temp.linux-x86_64-3.10/src
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Ilibs/liburing/src/include -I/opt/python/3.10/include/python3.10 -c libs/liburing/src/nolibc.c -o build/temp.linux-x86_64-3.10/libs/liburing/src/nolibc.o -Os -g0 -include libs/liburing/config-host.h -nostdlib -nodefaultlibs -ffreestanding -fno-stack-protector
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Ilibs/liburing/src/include -I/opt/python/3.10/include/python3.10 -c libs/liburing/src/queue.c -o build/temp.linux-x86_64-3.10/libs/liburing/src/queue.o -Os -g0 -include libs/liburing/config-host.h -nostdlib -nodefaultlibs -ffreestanding -fno-stack-protector
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Ilibs/liburing/src/include -I/opt/python/3.10/include/python3.10 -c libs/liburing/src/register.c -o build/temp.linux-x86_64-3.10/libs/liburing/src/register.o -Os -g0 -include libs/liburing/config-host.h -nostdlib -nodefaultlibs -ffreestanding -fno-stack-protector
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Ilibs/liburing/src/include -I/opt/python/3.10/include/python3.10 -c libs/liburing/src/setup.c -o build/temp.linux-x86_64-3.10/libs/liburing/src/setup.o -Os -g0 -include libs/liburing/config-host.h -nostdlib -nodefaultlibs -ffreestanding -fno-stack-protector
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Ilibs/liburing/src/include -I/opt/python/3.10/include/python3.10 -c src/uring.c -o build/temp.linux-x86_64-3.10/src/uring.o -Os -g0 -include libs/liburing/config-host.h -nostdlib -nodefaultlibs -ffreestanding -fno-stack-protector
creating build/lib.linux-x86_64-3.10
creating build/lib.linux-x86_64-3.10/uring
gcc -pthread -shared build/temp.linux-x86_64-3.10/libs/liburing/src/nolibc.o build/temp.linux-x86_64-3.10/libs/liburing/src/queue.o build/temp.linux-x86_64-3.10/libs/liburing/src/register.o build/temp.linux-x86_64-3.10/libs/liburing/src/setup.o build/temp.linux-x86_64-3.10/src/uring.o -o build/lib.linux-x86_64-3.10/uring/_uring.cpython-310-x86_64-linux-gnu.so
copying build/lib.linux-x86_64-3.10/uring/_uring.cpython-310-x86_64-linux-gnu.so -> uring
running clean
removing 'build/temp.linux-x86_64-3.10' (and everything under it)
removing 'build/lib.linux-x86_64-3.10' (and everything under it)
'build/bdist.linux-x86_64' does not exist -- can't clean it
'build/scripts-3.10' does not exist -- can't clean it
removing 'build'

LIBC=True back to error.
prefix                        /usr
includedir                    /usr/include
libdir                        /usr/lib
libdevdir                     /usr/lib
relativelibdir
mandir                        /usr/man
datadir                       /usr/share
stringop_overflow             yes
array_bounds                  yes
__kernel_rwf_t                yes
__kernel_timespec             yes
open_how                      yes
statx                         yes
C++                           yes
has_ucontext                  yes
has_memfd_create              yes
liburing_nolibc               no
CC                            gcc
CXX                           g++
running build_ext
building 'uring._uring' extension
creating build
creating build/temp.linux-x86_64-3.10
creating build/temp.linux-x86_64-3.10/libs
creating build/temp.linux-x86_64-3.10/libs/liburing
creating build/temp.linux-x86_64-3.10/libs/liburing/src
creating build/temp.linux-x86_64-3.10/src
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Ilibs/liburing/src/include -I/opt/python/3.10/include/python3.10 -c libs/liburing/src/queue.c -o build/temp.linux-x86_64-3.10/libs/liburing/src/queue.o -Os -g0 -include libs/liburing/config-host.h
In file included from libs/liburing/src/queue.c:5:
libs/liburing/src/syscall.h: In function ‘____sys_io_uring_register’:
libs/liburing/src/syscall.h:101:15: warning: implicit declaration of function ‘syscall’ [-Wimplicit-function-declaration]
  101 |         ret = syscall(__NR_io_uring_register, fd, opcode, arg, nr_args);
      |               ^~~~~~~
libs/liburing/src/syscall.h: In function ‘uring_madvise’:
libs/liburing/src/syscall.h:170:15: warning: implicit declaration of function ‘madvise’; did you mean ‘raise’? [-Wimplicit-function-declaration]
  170 |         ret = madvise(addr, length, advice);
      |               ^~~~~~~
      |               raise
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Ilibs/liburing/src/include -I/opt/python/3.10/include/python3.10 -c libs/liburing/src/register.c -o build/temp.linux-x86_64-3.10/libs/liburing/src/register.o -Os -g0 -include libs/liburing/config-host.h
In file included from libs/liburing/src/register.c:5:
libs/liburing/src/syscall.h: In function ‘____sys_io_uring_register’:
libs/liburing/src/syscall.h:101:15: warning: implicit declaration of function ‘syscall’ [-Wimplicit-function-declaration]
  101 |         ret = syscall(__NR_io_uring_register, fd, opcode, arg, nr_args);
      |               ^~~~~~~
libs/liburing/src/syscall.h: In function ‘uring_madvise’:
libs/liburing/src/syscall.h:170:15: warning: implicit declaration of function ‘madvise’; did you mean ‘raise’? [-Wimplicit-function-declaration]
  170 |         ret = madvise(addr, length, advice);
      |               ^~~~~~~
      |               raise
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Ilibs/liburing/src/include -I/opt/python/3.10/include/python3.10 -c libs/liburing/src/setup.c -o build/temp.linux-x86_64-3.10/libs/liburing/src/setup.o -Os -g0 -include libs/liburing/config-host.h
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Ilibs/liburing/src/include -I/opt/python/3.10/include/python3.10 -c libs/liburing/src/syscall.c -o build/temp.linux-x86_64-3.10/libs/liburing/src/syscall.o -Os -g0 -include libs/liburing/config-host.h
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Ilibs/liburing/src/include -I/opt/python/3.10/include/python3.10 -c src/uring.c -o build/temp.linux-x86_64-3.10/src/uring.o -Os -g0 -include libs/liburing/config-host.h
creating build/lib.linux-x86_64-3.10
creating build/lib.linux-x86_64-3.10/uring
gcc -pthread -shared build/temp.linux-x86_64-3.10/libs/liburing/src/queue.o build/temp.linux-x86_64-3.10/libs/liburing/src/register.o build/temp.linux-x86_64-3.10/libs/liburing/src/setup.o build/temp.linux-x86_64-3.10/libs/liburing/src/syscall.o build/temp.linux-x86_64-3.10/src/uring.o -o build/lib.linux-x86_64-3.10/uring/_uring.cpython-310-x86_64-linux-gnu.so
copying build/lib.linux-x86_64-3.10/uring/_uring.cpython-310-x86_64-linux-gnu.so -> uring
running clean
removing 'build/temp.linux-x86_64-3.10' (and everything under it)
removing 'build/lib.linux-x86_64-3.10' (and everything under it)
'build/bdist.linux-x86_64' does not exist -- can't clean it
'build/scripts-3.10' does not exist -- can't clean it
removing 'build'

You're missing -D_GNU_SOURCE.

That did it. Thanks @ammarfaizi2 works like a charm :)


Feature Request: Store current page, sort, etc. in cookie/session

Several users have complained to me that when they return to the listing page (with Smart Listing) the list has started from the beginning again.
It would be great to be able to store the current details in the users session or a cookie so that when they return to the page it would load to the same spot as before, with sort/filter/paging etc.
I have multiple pages with different lists if that makes a difference.

+1

+1

+1

+1

+1

+1

+1

+1

I'm guessing since the last reference to this was in 2015/2016, this is not even on the docket? lol

FYI: looks like this PR solves it: https://github.com/Sology/smart_listing/pull/154

@ljachymczyk are you willing to merge?

I miss this feature too much


Laravel Octane with TenantConfig Feature

Describe the bug
When using Laravel Octane with the TenantConfig feature enabled, the configuration is always from the previous request
Steps to reproduce

Enable TenantConfig feature
Map tenant id to app.name
Print app.name in a view

Expected behavior
When visiting tenant a, tenant a should appear, and the same for tenant b.
Your setup

Laravel version: 8.40
stancl/tenancy version: 3.4.3

Octane isn't officially supported yet, I'll be focusing on that in v4.

Found this thread on google, very happy using this package in our application, keep up the good work. I hope octane support comes too 👍 👍


Ensure all ValueErrors are raised

Some ValueErrors are currently not thrown (or raised) when they are detected. We want to avoid this and raise all ValueErrors given the chance.

I signed it.

Hi @anj-s  Are you going to review?

@anj-s Hi, I just want to confirm something. Would this change require tests? Or should I do something else?

There are some failures that are newly uncovered from this. Can you fix the new errors? I've retriggered the tests. There is a failure in contrib/quantize:fold_batch_norms_test

When I look at:
https://source.cloud.google.com/results/invocations/66bfcd5b-c469-4f4e-8794-0a0de124bc41/targets/%2F%2Ftensorflow%2Fcontrib%2Fquantize:fold_batch_norms_test/tests
I don't see link for the console output for the test.
How can I access the console output ?
thanks

@martinwicke Outside of this test error, should anything else be done?

@martinwicke @anj-s Should the test be retriggered? Usually, QA should not take this long.

@anj-s
Can you take another look at the PR ?
Thanks

Unless this hits unexpected flakes in testing, this should submit soon.


Function disabled if connection to rabbitmq cannot be made on startup

This may be more of a question for the function worker or function host repos, but I thought I would start here.
When starting a function (we run the dotnet isolated functions), if the rabbitmq client cannot make a connection, the function gets disabled and the function host continues to run:
[2022-05-24T21:23:25.230Z] Adding jobhost language worker channel for runtime: dotnet-isolated. workerId:d8d3aae9-1b49-4d06-9ea0-d72968c949b0
[2022-05-24T21:23:25.237Z] Worker process started and initialized.
[2022-05-24T21:23:27.439Z] Microsoft.Azure.WebJobs.Host: Error indexing method 'Functions.MyFunction'. RabbitMQ.Client: None of the specified endpoints were reachable. RabbitMQ.Client: One or more errors occurred. (Connection failed). RabbitMQ.Client: Connection failed. System.Net.Sockets: No connection could be made because the target machine actively refused it.
[2022-05-24T21:23:27.455Z] Error indexing method 'Functions.MyFunction'
[2022-05-24T21:23:27.456Z] Microsoft.Azure.WebJobs.Host: Error indexing method 'Functions.MyFunction'. RabbitMQ.Client: None of the specified endpoints were reachable. RabbitMQ.Client: One or more errors occurred. (Connection failed). RabbitMQ.Client: Connection failed. System.Net.Sockets: No connection could be made because the target machine actively refused it.
[2022-05-24T21:23:27.457Z] Function 'Functions.MyFunction' failed indexing and will be disabled.
[2022-05-24T21:23:27.461Z] No job functions found. Try making your job classes and methods public. If you're using binding extensions (e.g. Azure Storage, ServiceBus, Timers, etc.) make sure you've called the registration method for the extension(s) in your startup code (e.g. builder.AddAzureStorage(), builder.AddServiceBus(), builder.AddTimers(), etc.).

Is there a way to configure retries on initializing the functions in the function host? Or are there settings for how many times to attempt a connection to rabbitmq?
As a side question, we also run these functions in a docker container. In this error scenario above, the container and function host keep running, but we have no way of knowing the function is disabled unless we look at the logs. Ideally we would want a health check or similar so we can restart the container. Is this possible?

Hi @sherzberg, RabbitMQ extension tries to open channel with the RabbitMQ service at the time the methods are being indexed. I get that the suggestion is to add retries inside the extension code if the opening up of the channel fails. Can you share the reason that led to the connection failure, for example, was it a network issue? If possible and if RabbitMQ client allows it, we would like to retry for specific error conditions only. We are short of resources so cannot provide an ETA on getting back to this issue at the moment.
Regarding functions host running inside a container, as you said, I too am not aware of a way to identify the issues during functions host startup other than checking the pod logs. I will suggest starting a discussion thread on functions host repo for this one.

@JatinSanghvi in our case, rabbitmq was started or restarted roughly the same time as our function and was not accepting connections when the function initialized, but I could see any temporary network issue causing this as well.
It does seem that if the function gets initialized properly (rabbitmq was available during initialization) and the function loses connection to rabbitmq it does attempt a reconnection, which is good.
Thanks for the feedback on the follow up question, I will start a discussion on the function host repo.