ZFS Support

Upon executing docker run -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=<YourStrong!Passw0rd>' -p 1433:1433 microsoft/mssql-server-linux, I encounter the following message:
Configuring Microsoft(R) SQL Server(R)...
Microsoft(R) SQL Server(R) setup failed with error code 1. Please check the setup log in /var/opt/mssql/log for more information.

The setup log contains:
This is an evaluation version.  There are [154] days left in the evaluation period.
2017-01-10 23:55:24.82 Server      Setup step is copying system data file 'C:\templatedata\master.mdf' to 'C:\var\opt\mssql\data\master.mdf'.
2017-01-10 23:55:24.90 Server      Setup step is copying system data file 'C:\templatedata\mastlog.ldf' to 'C:\var\opt\mssql\data\mastlog.ldf'.
2017-01-10 23:55:24.91 Server      Setup step is copying system data file 'C:\templatedata\model.mdf' to 'C:\var\opt\mssql\data\model.mdf'.
2017-01-10 23:55:24.92 Server      Setup step is copying system data file 'C:\templatedata\modellog.ldf' to 'C:\var\opt\mssql\data\modellog.ldf'.
2017-01-10 23:55:24.94 Server      Setup step is copying system data file 'C:\templatedata\msdbdata.mdf' to 'C:\var\opt\mssql\data\msdbdata.mdf'.
2017-01-10 23:55:24.95 Server      Setup step is copying system data file 'C:\templatedata\msdblog.ldf' to 'C:\var\opt\mssql\data\msdblog.ldf'.
2017-01-10 23:55:25.02 Server      Microsoft SQL Server vNext (CTP1.1) - <IP_ADDRESS> (X64)
	Dec 10 2016 02:51:11
	Copyright (C) 2016 Microsoft Corporation. All rights reserved.
	on Linux (Ubuntu 16.04.1 LTS)
2017-01-10 23:55:25.02 Server      UTC adjustment: 0:00
2017-01-10 23:55:25.02 Server      (c) Microsoft Corporation.
2017-01-10 23:55:25.03 Server      All rights reserved.
2017-01-10 23:55:25.03 Server      Server process ID is 4116.
2017-01-10 23:55:25.03 Server      Logging SQL Server messages in file 'C:\var\opt\mssql\log\errorlog'.
2017-01-10 23:55:25.03 Server      Registry startup parameters:
	 -d C:\var\opt\mssql\data\master.mdf
	 -l C:\var\opt\mssql\data\mastlog.ldf
	 -e C:\var\opt\mssql\log\errorlog
2017-01-10 23:55:25.03 Server      Command Line Startup Parameters:
	 --setup
	 --sa-password
2017-01-10 23:55:25.04 Server      Error: 17113, Severity: 16, State: 1.
2017-01-10 23:55:25.04 Server      Error 87(The parameter is incorrect.) occurred while opening file 'C:\var\opt\mssql\data\master.mdf' to obtain configuration information at startup. An invalid startup option might have caused the error. Verify your startup options, and correct or remove them if necessary.
2017-01-10 23:55:25.05 Server      SQL Server shutdown has been initiated

The /var/opt/mssql/data directory contains:
-rw-r----- 1 root root  4194304 Jan 11 10:55 master.mdf
-rw-r----- 1 root root   524288 Jan 11 10:55 mastlog.ldf
-rw-r----- 1 root root  8388608 Jan 11 10:55 modellog.ldf
-rw-r----- 1 root root  8388608 Jan 11 10:55 model.mdf
-rw-r----- 1 root root 13959168 Jan 11 10:55 msdbdata.mdf
-rw-r----- 1 root root   524288 Jan 11 10:55 msdblog.ldf

Please advise.

uname -a
Linux tecknack-corin 4.6.5-stable #54 SMP Tue Sep 13 11:33:24 AEST 2016 x86_64 Intel(R) Core(TM) i7-5500U CPU @ 2.40GHz GenuineIntel GNU/Linux

docker version
Client:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:
 OS/Arch:      linux/amd64

@au-phiware I tried running the exact command that you provided and it worked ok for me.  In my case I was using Docker for Windows, but I've done essentially the same thing recently on RHEL and Docker for Mac.  I'm wondering - is that the exact command you used or did you possibly try mounting a volume using docker run -v?

Hi @twright-msft, you are right I did run it with -v initially (that's how I got the log contents) but before posting I made sure to run that exact command. I tried a number of variations, all had exactly the same output, and I'm assuming the setup log is the same too. So, I could be wrong, is it likely that the log would be different? I could copy the file out if you suspect there's something in that...?

I have no doubt that there must be something particular about my kernel or filesystem. I use Gentoo and build my own kernel from git (the stable branch) but I'm not doing anything special there. I use zfs and so the docker storage driver is zfs too, could it be that there's an issue with accessing metadata or something else to do with the filesystem?

Just to be sure I tried on my home machine (also Gentoo and ZFS) and used docker cp to retrieve the errorlog file.
Microsoft(R) SQL Server(R) setup failed with error code 1. Please check the setup log in /var/opt/mssql/log for more information.
$ docker ps -a | head -n 2
CONTAINER ID        IMAGE                               COMMAND                  CREATED             STATUS                      PORTS               NAMES
d66b9878ea41        microsoft/mssql-server-linux        "/bin/sh -c /opt/m..."   14 minutes ago      Exited (1) 14 minutes ago                       nervous_engelbart
$ docker cp d66b9878ea41:/var/opt/mssql/log/errorlog .
$ cat errorlog
ÿþ2017-01-11 20:59:36.77 Server      Microsoft SQL Server vNext (CTP1.1) - <IP_ADDRESS> (X64)
	Dec 10 2016 02:51:11
	Copyright (C) 2016 Microsoft Corporation. All rights reserved.
	on Linux (Ubuntu 16.04.1 LTS)
2017-01-11 20:59:36.78 Server      UTC adjustment: 0:00
2017-01-11 20:59:36.78 Server      (c) Microsoft Corporation.
2017-01-11 20:59:36.78 Server      All rights reserved.
2017-01-11 20:59:36.78 Server      Server process ID is 4116.
2017-01-11 20:59:36.78 Server      Logging SQL Server messages in file 'C:\var\opt\mssql\log\errorlog'.
2017-01-11 20:59:36.78 Server      Registry startup parameters:
	 -d C:\var\opt\mssql\data\master.mdf
	 -l C:\var\opt\mssql\data\mastlog.ldf
	 -e C:\var\opt\mssql\log\errorlog
2017-01-11 20:59:36.78 Server      Command Line Startup Parameters:
	 --setup
	 --sa-password
2017-01-11 20:59:36.79 Server      Error: 17113, Severity: 16, State: 1.
2017-01-11 20:59:36.79 Server      Error 87(The parameter is incorrect.) occurred while opening file 'C:\var\opt\mssql\data\master.mdf' to obtain configuration information at startup. An invalid startup option might have caused the error. Verify your startup options, and correct or remove them if necessary.
2017-01-11 20:59:36.80 Server      SQL Server shutdown has been initiated
$ docker version
Client:
 Version:      1.13.0-rc3
 API version:  1.25
 Go version:   go1.7.1
 Git commit:   4d92237
 Built:
 OS/Arch:      linux/amd64

Server:
 Version:      1.13.0-rc3
 API version:  1.25 (minimum version 1.12)
 Go version:   go1.7.1
 Git commit:   4d92237
 Built:
 OS/Arch:      linux/amd64
 Experimental: false
$ uname -a
Linux fibonacci 4.3.6-stable #61 SMP Sun Dec 18 14:11:45 AEDT 2016 x86_64 Intel(R) Core(TM) i7-4710HQ CPU @ 2.50GHz GenuineIntel GNU/Linux

We havent done any testing on Gentoo/zfs yet.  It's definitely possible there is an issue there with the kernel/file system.

Is there any further information that I can provide? Or turn on verbose
logging?
On Fri, 13 Jan 2017, 12:30 AM Travis Wright<EMAIL_ADDRESS>wrote:

We havent done any testing on Gentoo/zfs yet. It's definitely possible
there is an issue there with the kernel/file system.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Microsoft/mssql-docker/issues/13#issuecomment-272163319,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAx2LutMzCdMoMYWB_5SqhkOvDcjEVLOks5rRirqgaJpZM4LgDrC
.

Had the same issue with similar error log. I noticed that i used volume mapping with -v and that this is not supported for mac. Once i removed it during run of the docker image it started working.

I am having this exact same issue on Docker for windows.

The mssql image shows a clear and specific error message when docker engine has less than 4gb ram available. This is not the error I was getting. After a full docker reset the issue seems to have gone away now though.

having same issue and error message using mac and docker v1.13 with 4gb memory
when mounting volume with -v option to bind /var/opt/mssql, we can see logs but .mdf id failing with error 87.
workaround is to start without this -v option, it works, but .mdf file are inside container

@laurentleseigneur This is a known issue with macOS and -v.  We have a separate issue tracking that.  https://github.com/Microsoft/mssql-docker/issues/12
For now, you can use an attached Docker container volume for database file storage.
https://docs.docker.com/engine/tutorials/dockervolumes/#/creating-and-mounting-a-data-volume-container

thanks @twright-msft  for pointing this known issue, i will give a try to a docker compose using a container volume to store .mdf files

I had the same error and I solved it by erasing the images and containers and re-creating them.
Docker version 1.13.1, build 092cba3
Windows 10 build 15031.rs2_release.170204-1546

@twright-msft Thanks for this thread; came here because I'm experiencing the same issue. I wanted to see if my case is a part of the known issue because I'm using mounted data volume containers.
In my docker-compose.yml I have a service that is configured:
  mssql:
    build: ./mssql
    env_file:
    - .env
    ports:
    - "1433:1433"
    volumes:
    - ./mssql/volumes/data:/var/opt/mssql

But when I pull up the logs from docker logs <container iD> It spits out:
2017-03-08 20:01:34.80 Server      Error: 17113, Severity: 16, State: 1.
2017-03-08 20:01:34.80 Server      Error 87(The parameter is incorrect.) occurred while opening file 'C:\var\opt\mssql\data\master.mdf' to obtain configuration information at startup. An invalid startup option might have caused the error. Verify your startup options, and correct or remove them if necessary.
2017-03-08 20:01:34.80 Server      SQL Server shutdown has been initiated

Is this part of the the known issue?

@PatrickDePuydt Are you on MacOS or Linux?

@twright-msft I'm running OS X Sierra but the base Docker image is a Centos7

@PatrickDePuydt OK, then yes, you are running into the issue I described above with using Docker for Mac and -v (volumes: in the way you are using it in docker-compose.yml is effectively docker run -v).  Please use a 'data volume container' instead.
More info:
https://docs.docker.com/engine/tutorials/dockervolumes/#creating-and-mounting-a-data-volume-container
http://stackoverflow.com/questions/32908621/how-can-i-create-a-data-container-only-using-docker-compose-yml

@twright-msft Ok, thanks! Could you clear up one last thing? You mentioned using an attached Docker container volume for database file storage as a workaround, but isn't that what I'm doing in the volume declaration of the service:   - ./mssql/volumes/data:/var/opt/mssql ?

As I understand it what you are doing there is mounting a directory on the host to a directory in the container.  That's what doesnt work on macOS.  What you want to do is mount a volume container to a directory in the mssql container.

This is an evaluation version.  There are [156] days left in the evaluation period.
RegQueryValueEx HADR for key "Software\Microsoft\Microsoft SQL Server\MSSQL\MSSQLServer\HADR" failed.
2017-03-11 09:58:56.66 Server      Microsoft SQL Server vNext (CTP1.3) - 14.0.304.138 (X64)
Feb 13 2017 16:49:12
Copyright (C) 2016 Microsoft Corporation. All rights reserved.
on Linux (Ubuntu 16.04.1 LTS)
2017-03-11 09:58:56.66 Server      UTC adjustment: 0:00
2017-03-11 09:58:56.66 Server      (c) Microsoft Corporation.
2017-03-11 09:58:56.66 Server      All rights reserved.
2017-03-11 09:58:56.67 Server      Server process ID is 4116.
2017-03-11 09:58:56.67 Server      Logging SQL Server messages in file 'C:\var\opt\mssql\log\errorlog'.
2017-03-11 09:58:56.67 Server      Registry startup parameters:
-d C:\var\opt\mssql\data\master.mdf
-l C:\var\opt\mssql\data\mastlog.ldf
-e C:\var\opt\mssql\log\errorlog
2017-03-11 09:58:56.69 Server      Error: 17113, Severity: 16, State: 1.
2017-03-11 09:58:56.69 Server      Error 3(The system cannot find the path specified.) occurred while opening file 'C:\var\opt\mssql\data\master.mdf' to obtain configuration information at startup. An invalid startup option might have caused the error. Verify your startup options, and correct or remove them if necessary.
2017-03-11 09:58:56.70 Server      SQL Server shutdown has been initiated
I have the same problem but not in docker. it running and working in docker after pushing the docker image to heroku app it is giving the above message message. anybody has any idea about this issue.
Please help if anybody knows
Thanks

(I feels as if this thread has been hijacked by Mac users)
I decided to take ZFS out of the mix:

First I confirmed I was still receiving the error with the latest image.
I create a zvol and formatted it with ext4 and mounted it as my /var/lib/docker directory.
Started the docker daemon and ensured it wasn't using zfs as the storage device.
Tried again.

This time I didn't receive the error, the container stayed up and I have an operational mssql server on linux!
I guess the next question is: where does the fault lie? This project (seems unlikely)? Docker (more specifically the zfs storage driver)? Or ZFS?
I think I'll close this issue and go ask Docker.
For the record here's my working configuration:
$ docker info
Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 2
Server Version: 1.12.1
Storage Driver: devicemapper
 Pool Name: docker-230:1-131073-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: ext4
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 2.485 GB
 Data Space Total: 107.4 GB
 Data Space Available: 7.923 GB
 Metadata Space Used: 2.257 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.145 GB
 Thin Pool Minimum Free Space: 10.74 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.93 (2015-01-30)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.6.5-stable
Operating System: Gentoo/Linux
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.59 GiB
Name: tecknack-corin
ID: JXJV:BP5E:T547:PZTF:4JSO:6SV5:B7EO:XA3U:EPV7:7Z5H:BS2Y:SAV7
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Insecure Registries:
 <IP_ADDRESS>/8

Have you run any traces while it was running and then capturing the crash?

On May 14, 2017 at 7:23 PM,  <Corin Lawson (mailto:notifications@github.com)>  wrote:
(I feels as if this thread has been hijacked by Mac users)
I decided to take ZFS out of the mix:
First I confirmed I was still receiving the error with the latest image.
I create a zvol and formatted it with ext4 and mounted it as my /var/lib/docker directory.
Started the docker daemon and ensured it wasn't using zfs as the storage device.
Tried again.
This time I didn't receive the error, the container stayed up and I have an operational mssql server on linux!
I guess the next question is: where does the fault lie? This project (seems unlikely)? Docker (more specifically the zfs storage driver)? Or ZFS?
I think I'll close this issue and go ask Docker.
For the record here's my working configuration:
$ docker info Containers: 1 Running: 1 Paused: 0 Stopped: 0 Images: 2 Server Version: 1.12.1 Storage Driver: devicemapper Pool Name: docker-230:1-131073-pool Pool Blocksize: 65.54 kB Base Device Size: 10.74 GB Backing Filesystem: ext4 Data file: /dev/loop0 Metadata file: /dev/loop1 Data Space Used: 2.485 GB Data Space Total: 107.4 GB Data Space Available: 7.923 GB Metadata Space Used: 2.257 MB Metadata Space Total: 2.147 GB Metadata Space Available: 2.145 GB Thin Pool Minimum Free Space: 10.74 GB Udev Sync Supported: true Deferred Removal Enabled: false Deferred Deletion Enabled: false Deferred Deleted Device Count: 0 Data loop file: /var/lib/docker/devicemapper/devicemapper/data WARNING: Usage of loopback devices is strongly discouraged for production use. Use --storage-opt dm.thinpooldev to specify a custom block storage device. Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata Library Version: 1.02.93 (2015-01-30) Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host null overlay Swarm: inactive Runtimes: runc Default Runtime: runc Security Options: seccomp Kernel Version: 4.6.5-stable Operating System: Gentoo/Linux OSType: linux Architecture: x86_64 CPUs: 4 Total Memory: 15.59 GiB Name: tecknack-corin ID: JXJV:BP5E:T547:PZTF:4JSO:6SV5:B7EO:XA3U:EPV7:7Z5H:BS2Y:SAV7 Docker Root Dir: /var/lib/docker Debug Mode (client): false Debug Mode (server): false Registry: https://index.docker.io/v1/ Insecure Registries: <IP_ADDRESS>/8
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly,   view it on GitHub (https://github.com/Microsoft/mssql-docker/issues/13#issuecomment-301360919), or   mute the thread (https://github.com/notifications/unsubscribe-auth/AAk8rXrpPZxLV7L8WbCyd5L3NKHVINDvks5r57cqgaJpZM4LgDrC).

Here is a full syscall trace from machine with zfsonlinux:
Steps to reproduce

docker run -ti -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=<YourStrong!Passw0rd>' -p 1433:1433 microsoft/mssql-server-linux bash
root@f5f756e8d68c:/# apt update && apt install strace
strace -f -o /tmp/sqlservr.log /opt/mssql/bin/sqlservr

https://dl.thalheim.io/Z03MnoPGwbCXhJOXpBvgwg/sqlservr.log?dl=1
I tried to reason about the repeated sequence of syscalls (some strange enterprise retry handler), but found not failing syscall. Without access to the source code and this uninformative  error message, I have no clue what's going on here.

FWIW we have done zero testing on zfs.  @au-phiware Please let us know the issue ID if you do raise this with Docker in a way that is publicly trackable.  Thanks!

The issue can be viewed at moby/moby#33191

@au-phiware I do not think docker is the issue tracker for this, as we miss domain knowledge about mssql internals.

@Mic92 yes, well I'm really not sure where to go... I'll try recreate the issue with docker removed from the mix.
Is there a better issue tracker someplace or should I reopen this one?

this one, is probably better for the moment.

Okay, I have installed mssql-server onto my system and encounter the same error. Mounting an ext4 filesystem at /var/opt/mssql resolves the error.
I realise that only xfs and ext4 is supported. So, I guess this is a feature request?
At least I have a work around now; I look forward to giving it a spin and adding it to my workflow.

Same here. Ubuntu + ZFS + LXC.

@crazybert - Please use Ubuntu 16.04 + ext4 + Docker for now.  Keeping this issue open only as an 'enhancement' tracking issue for us to consider in a future release.

@twright-msft Thanks! Also appeared to work with ext4 on zvol. Not ideal, but good enough.

Does anyone know what exactly is the missing feature of ZFS, what stops mssql from working? Maybe we just have to enable something like ACLs or similar. ZVOLs are an option but not ideal. I would prefer a normal ZFS filesystem much more.

looks like the O_DIRECT support is missing in ZFS https://github.com/zfsonlinux/zfs/issues/224...

a quick hack would be to have a seccomp profile for the container, that masks O_DIRECT in open calls. This would only make things slightly less performant as data would be cached both by ZFS and mssql.

But to be honest, I would be better if mssql would use POSIX_FADV_DONTNEED instead of O_DIRECT.

Yes, I found something similar https://code.uplex.de/liblongpath/liblongpath/commit/2e46a921ce2b6b1caa56d39cbd58be85c5988bd0 but I did not get it to work yet

@Mic92 could you post how exactly one would set up such a seccomp profile?

Turns out seccomp alone is not enough without using ptrace. However this is quite complicated.
LD_PRELOAD is much easier to use:
#define _GNU_SOURCE
#include <dlfcn.h>
#include <fcntl.h>
typedef int (*orig_open_f_type)(const char *pathname, int flags);
int open(const char *pathname, int flags, ...) {
    static orig_open_f_type orig_open;
    if (!orig_open) {
                        orig_open = (orig_open_f_type)dlsym(RTLD_NEXT, "open");
    }
    return orig_open(pathname, flags & ~O_DIRECT);
}

$ apt update gcc && apt install gcc
$ gcc -shared -fpic -o /nodirect_open.so nodirect_open.c -ldl
$ echo /nodirect_open.so > /etc/ld.so.preload
$ /opt/mssql/bin/sqlservr --accept-eula

I had to delete the old database in /var/opt/mssql before

Thank you! I am not into the filesystem stuff, so can you tell me what effect this has, or more specifically does using this hack (over ZFS) affect stability or performance of the SQL Server in any way?

@t-oster O_DIRECT is this dirty hack that database guys introduced to bypass any caches for read/writes in the operating system. Reason for that is that it would result in double caching: both the operating system and the database cache data. However databases have in general a better idea what data needs to be cached. In the worst case your caches would be only half as efficient. In practice however it is not that bad:
If the system is under memory pressure it will drop the page cache, while the database continues to keep its own. That means that your operating system will waste some CPU cycles on maintaining unnecessary page and write back caches.

Also read this thread on the linux developer kernel mailing list: https://lkml.org/lkml/2007/1/10/231

so I would use zfs set primaraycache=metadata and be happy without concerns about other drawbacks than maybe performance?

In regards to O_DIRECT I don't see other drawbacks. I have not tested mssql intensively to see what it does elsewhere.

Thank you @Mic92 it seems to work very good, maybe I will switch our production system from ext4/zvol directly to zfs (after some tests). If anyone else needs this, here you are: https://github.com/t-oster/mssql-docker-zfs

i just ran into this today. IMO this is more than an enhancement - zfs is the default storage driver for docker on systems that support it.

Is mssql going to implement something similar to the innodb_flush_method?
It allows for the following types of methods to flush writes to disk:
fsync
O_DSYNC
littlesync
nosync
O_DIRECT
O_DIRECT_NO_FSYNC
Is there any news on this?

@Smithx10 - We don’t currently have any plans to support additional/configurable flush methods.  Do you think we need to?

@twright-msft , Is there a plan to support ZFS and other operating systemswithout forcing them to hack around the real problem using LD_PRELOAD?
:(

DirectIO via the O_DIRECT flag was originally introduced in XFS by IRIX
for database workloads. Its purpose was to allow the database to bypass
the page and buffer caches to prevent unnecessary IO operations (e.g.
readahead) while preventing contention for system memory between the
database and kernel caches.
Unfortunately, the semantics were never defined in any standard. The
semantics of O_DIRECT in XFS in Linux are as follows:

O_DIRECT requires IOs be aligned to backing device's sector size.
O_DIRECT performs unbuffered IO operations between user memory and block
device (DMA when the block device is physical hardware).
O_DIRECT implies O_DSYNC.
O_DIRECT disables any locking that would serialize IO operations.

The first is not possible in ZFS beause there is no backing device in
the general case.
The second is not possible in ZFS in the presence of compression because
that prevents us from doing DMA from user pages. If we relax the
requirement in the case of compression, we encunter another hurdle. In
specific, avoiding the userland to kernel copy risks other userland
threads modifying buffers during compression and checksum computations.
For compressed data, this would cause undefined behavior while for
checksums, this would imply we write incorrect checksums to disk.  It
would be possible to avoid those issues if we modify the page tables to
make any changes by userland to memory trigger page faults and perform
CoW operations.  However, it is unclear if it is wise for a filesystem
driver to do this.
The third is doable, but we would need to make ZIL perform indirect
logging to avoid writing the data twice.
The fourth is already done for all IO in ZFS.
Other Linux filesystems such as ext4 do not follow #3. Mac OS X does not
implement O_DIRECT, but it does implement F_NOCACHE, which is similiar
to #2 in that it prevents new data from being cached. AIX relaxes #3 by
only committing the file data to disk. Metadata updates required should
the operations make the file larger are asynchronous unless O_DSYNC is
specified.
On Solaris and Illumos, there is a library function called directio(3C)
that allows userspace to provide a hint to the filesystem that DirectIO
is useful, but the filesystem is free to ignore it. The semantics are
also entirely a filesystem decision. Those that do not implement it
return ENOTTY.
Given the lack of standardization and ZFS' heritage, one solution to
provide compatibility with userland processes that expect DirectIO is to
treat DirectIO as a hint that we ignore. This can be done trivially by
implementing a shim that maps aops->direct_IO to AIO. There is also
already code in ZoL for bypassing the page cache when O_DIRECT is
specified, but it has been inert until now.
If it turns out that it is acceptable for a filesystem driver to
interact with the page tables, the scatter-gather list work will need be
finished and we would need to utilize the page tables to make operations
on the userland pages safe.
References:
http://xfs.org/docs/xfsdocs-xml-dev/XFS_User_Guide/tmp/en-US/html/ch02s09.html
https://blogs.oracle.com/roch/entry/zfs_and_directio
https://ext4.wiki.kernel.org/index.php/Clarifying_Direct_IO's_Semantics
https://illumos.org/man/3c/directio
https://developer.apple.com/library/mac/#documentation/Darwin/Reference/ManPages/man2/fcntl.2.html
https://lists.apple.com/archives/filesystem-dev/2007/Sep/msg00010.html

:(

Using O_DIRECT well requires taking on a surprising amount of additional resource management responsibility. Also, the implementation is not portable and, combined with the additional responsibility previously mentioned, adds a ton of conditional compilation. The alignment requirements are not an issue at all; anyone using O_DIRECT is also going to be used properly aligned I/O buffers in any case.O_DIRECT can have a lot of benefits. However, while O_DIRECT may be simple to use by itself, the other design and implementation requirements that are indirectly dragged along are not trivial by any means. Consequently, you should not be using O_DIRECT unless you can manage the substantial indirect overhead in implementation. It turns off a lot of OS features most developers take for granted. | Using O_DIRECT well requires taking on a surprising amount of additional resource management responsibility. Also, the implementation is not portable and, combined with the additional responsibility previously mentioned, adds a ton of conditional compilation. The alignment requirements are not an issue at all; anyone using O_DIRECT is also going to be used properly aligned I/O buffers in any case.O_DIRECT can have a lot of benefits. However, while O_DIRECT may be simple to use by itself, the other design and implementation requirements that are indirectly dragged along are not trivial by any means. Consequently, you should not be using O_DIRECT unless you can manage the substantial indirect overhead in implementation. It turns off a lot of OS features most developers take for granted.
Using O_DIRECT well requires taking on a surprising amount of additional resource management responsibility. Also, the implementation is not portable and, combined with the additional responsibility previously mentioned, adds a ton of conditional compilation. The alignment requirements are not an issue at all; anyone using O_DIRECT is also going to be used properly aligned I/O buffers in any case.O_DIRECT can have a lot of benefits. However, while O_DIRECT may be simple to use by itself, the other design and implementation requirements that are indirectly dragged along are not trivial by any means. Consequently, you should not be using O_DIRECT unless you can manage the substantial indirect overhead in implementation. It turns off a lot of OS features most developers take for granted.

:(

Not that it's particularly bad for PostgreSQL, but as a point of possible future interest, I found out that the Linux kernel does not (and most likely will not in the future) recommend mixing O_DIRECT and non-O_DIRECT I/O on a single file.  Doing this is expected to result in very poor performance, because the use of O_DIRECT causes page cache invalidations.  I haven't been able to think of a scenario in which this will actually hurt PostgreSQL users today, because we only use O_DIRECT for WAL, and then only if wal_sync_method = open_sync or open_datasync, and then only neither archiving nor streaming replication is in use.  However, it's certainly worth keeping in mind if we ever consider expanding the use of O_DIRECT.

:(

Richrd Yao says:
July 7, 2013 at 8:56 am
Vadim, O_DIRECT was designed for in-place filesystems to allow IO to bypass the filesystem layer and caching. A literal implementation of O_DIRECT in a copy-on-write filesystem like ZFS is not possible (because checksum and parity calculations must be done). It is possible to implement it by effectively ignoring the O_DIRECT flag, but I imagine that would defeat the purpose. I imagine is the main reason the solution where O_DIRECT is ignored has not been implemented is that Linux uses a different code path for O_DIRECT and time spent implementing the separate code path is time that could be spent on other bugs.
Most of the development of ZFSOnLinux over the past two years has focused on making it ready for the first stable release. Adding O_DIRECT support did not help contribute to that, so it was a low priority. I imagine that adding O_DIRECT support would occur rather quickly if someone were to write a patch to add it that works. However, adding it would be misleading unless O_DIRECT is implemented in a way that provided some kind of tangible benefit over not using it.
Lastly, ZFSOnLinux development is done by a few professional developers at LLNL and volunteers, such as myself, that happen to use it. LLNL uses ZFSOnLinux as an OSD for the Lustre filesystem on the Sequoia suprecomputer while volunteers tend to use it on either servers or desktops. O_DIRECT is currently scheduled for a release rather far in the future because none of us have any need for O_DIRECT. It should be possible to configure your software to not use O_DIRECT, so doing it sooner does not seem like it should be a priority.

But hey there is still postgresql/mariadb which will perform faster anyway.

@Mic92  Sometimes as an operator, I get in situations where .net developers want to use mssql, or the Enterprise needs that Windows Stamp.

Well for professional usage you don't want to use the linux port anyway... It does so many strange things I have seen in strace.

@twright-msft   These are the reasons why you should support more than just Linux<EMAIL_ADDRESS>

Hmmm.. that URL isnt resolving for me.  Can you please check?

@twright-msft  https://twitter.com/justincormack/status/979540227109744640

Thanks for the heads up @Smithx10  - we are taking a look at it.

Starting with https://github.com/zfsonlinux/zfs/releases/tag/zfs-0.8.0-rc1 ZFS on Linux supports Direct IO. A quick test with the mssql image succeeded for me. Can't say anything about performance/stability though.

That's good news! I am successfully running some mssql in docker on ZFS with the LD_PRELOAD hack and did not have any problems so far. But it's better if that is not necessary anymore. I will report as soon as this version is on archzfs, so I can test.

看了一遍都是英文的,反正英文也不好.就直接上解决思路吧,出现这样的问题呢是因为启动的时候使用了-V挂载数据卷,因为也只是为了有个mssql服务器来调试环境,也就懒得去折腾数据是在本地还是容器中,直接去掉-V就开工干活了,问题也解决了

I am using macOS Mojave 10.14.1
➜  ~ docker run -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=123456a@' -p 1433:1433 -d -v /Users/donhuvy/Documents/mssql:/var/opt/mssql -d microsoft/mssql-server-linux:2017-CU8
9d4bd2a1da7a3bece6ce1052bc221538fea44b0469c19543e670b2cf5f2a6205
➜  ~ docker logs -f 9d4bd2a1da7a3bece6ce1052bc221538fea44b0469c19543e670b2cf5f2a6205
2018-11-24 07:16:46.99 Server      Microsoft SQL Server 2017 (RTM-CU8) (KB4338363) - 14.0.3029.16 (X64)
	Jun 13 2018 13:35:56
	Copyright (C) 2017 Microsoft Corporation
	Developer Edition (64-bit) on Linux (Ubuntu 16.04.4 LTS)
2018-11-24 07:16:47.01 Server      UTC adjustment: 0:00
2018-11-24 07:16:47.01 Server      (c) Microsoft Corporation.
2018-11-24 07:16:47.01 Server      All rights reserved.
2018-11-24 07:16:47.01 Server      Server process ID is 4120.
2018-11-24 07:16:47.01 Server      Logging SQL Server messages in file '/var/opt/mssql/log/errorlog'.
2018-11-24 07:16:47.01 Server      Registry startup parameters:
	 -d /var/opt/mssql/data/master.mdf
	 -l /var/opt/mssql/data/mastlog.ldf
	 -e /var/opt/mssql/log/errorlog
2018-11-24 07:16:47.04 Server      Error: 17113, Severity: 16, State: 1.
2018-11-24 07:16:47.04 Server      Error 87(The parameter is incorrect.) occurred while opening file '/var/opt/mssql/data/master.mdf' to obtain configuration information at startup. An invalid startup option might have caused the error. Verify your startup options, and correct or remove them if necessary.
➜  ~

@donhuvy - macos volume mounting is not currently working/supported.  Please see the following issue tracking adding support for this in the future:
#12

@donhuvy check this basically you have two options, a docker volumes or bind mount a local folder, by default docker doesn't create this.

Yep, what @sirio3mil linked to is the general concept.  Here's some details and examples for SQL Server specifically.  https://docs.microsoft.com/en-us/sql/linux/sql-server-linux-configure-docker?view=sql-server-2017#persist

Starting with https://github.com/zfsonlinux/zfs/releases/tag/zfs-0.8.0-rc1 ZFS on Linux supports Direct IO. A quick test with the mssql image succeeded for me. Can't say anything about performance/stability though.

That's good. Every few months, I set up a new server and forget that I have to put my SQL Server on non-ZFS volumes. On first start up, I scratch my head for an hour trying to figure it out. Hopefully this fixes the problem.
If MS fixes it on their end, would that resolve the issue for MacOS too?

I have exactly the same issue when using docker-compose up with Docker for Windows.

看了一遍都是英文的,反正英文也不好.就直接上解决思路吧,出现这样的问题呢是因为启动的时候使用了-V挂载数据卷,因为也只是为了有个mssql服务器来调试环境,也就懒得去折腾数据是在本地还是容器中,直接去掉-V就开工干活了,问题也解决了

不掛volume?那你docker重啟數據還在嗎

看了一遍都是英文的,反正英文也不好.就直接上解决思路吧,出现这样的问题呢是因为启动的时候使用了-V挂载数据卷,因为也只是为了有个mssql服务器来调试环境,也就懒得去折腾数据是在本地还是容器中,直接去掉-V就开工干活了,问题也解决了

Do not hang volume? Then your docker restart data still?

ZFS 0.8 is released and included at least in ArchLinux. I tested it and it works, so this can be closed IMHO. If you are stuck with an old version, use https://github.com/t-oster/mssql-docker-zfs

原来的

volumes:
- ../data/mssql:/var/opt/mssql

改成

volumes:
- ../data/mssql:/var/opt/mssql
- /var/opt/mssql/data

I have same issue with both versions 2017-CU8-ubuntu and  2019-CU3-ubuntu-18.04
Run Command
docker run --name mssql-2019-express-docker -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=yourStrong(!)Password' -e 'MSSQL_PID=Express' -v ~/.mssql/data:/var/opt/mssql/data -v ~/.mssql/log:/var/opt/mssql/log -v ~/.mssql/secrets:/var/opt/mssql/secrets -p 1433:1433 -d mcr.microsoft.com/mssql/server:2019-CU3-ubuntu-18.04

Error
2020-05-24 12:09:10.56 Server      Setup step is FORCE copying system data file 'C:\templatedata\model_replicatedmaster.mdf' to '/var/opt/mssql/data/model_replicatedmaster.mdf'.
2020-05-24 12:09:10.71 Server      Setup step is FORCE copying system data file 'C:\templatedata\model_replicatedmaster.ldf' to '/var/opt/mssql/data/model_replicatedmaster.ldf'.
2020-05-24 12:09:10.76 Server      Setup step is FORCE copying system data file 'C:\templatedata\model_msdbdata.mdf' to '/var/opt/mssql/data/model_msdbdata.mdf'.
2020-05-24 12:09:10.93 Server      Setup step is FORCE copying system data file 'C:\templatedata\model_msdblog.ldf' to '/var/opt/mssql/data/model_msdblog.ldf'.
2020-05-24 12:09:11.27 Server      Microsoft SQL Server 2019 (RTM-CU3) (KB4538853) - 15.0.4023.6 (X64)
Mar  4 2020 00:59:26
Copyright (C) 2019 Microsoft Corporation
Express Edition (64-bit) on Linux (Ubuntu 18.04.4 LTS) <X64>
2020-05-24 12:09:11.27 Server      UTC adjustment: 0:00
2020-05-24 12:09:11.28 Server      (c) Microsoft Corporation.
2020-05-24 12:09:11.28 Server      All rights reserved.
2020-05-24 12:09:11.28 Server      Server process ID is 40.
2020-05-24 12:09:11.28 Server      Logging SQL Server messages in file '/var/opt/mssql/log/errorlog'.
2020-05-24 12:09:11.28 Server      Registry startup parameters:
 -d /var/opt/mssql/data/master.mdf
 -l /var/opt/mssql/data/mastlog.ldf
 -e /var/opt/mssql/log/errorlog
2020-05-24 12:09:11.30 Server      Error: 17113, Severity: 16, State: 1.
2020-05-24 12:09:11.30 Server      Error 87(The parameter is incorrect.) occurred while opening file '/var/opt/mssql/data/master.mdf' to obtain configuration information at startup. An invalid startup option might have caused the error. Verify your startup options, and correct or remove them if necessary.

@dennisdaotvlk, you have exact the same issue I have, here is my docker compose file.
version: "3.8"
services:
  db:
    build:
      context: .
      dockerfile: db.Dockerfile
    image: ftop-db
    # https://docs.docker.com/compose/compose-file/#container_name
    container_name: ftop-db-server
    ports:
      - "1444:1433"
    # https://docs.docker.com/compose/compose-file/#environment
    # https://docs.microsoft.com/en-us/sql/linux/sql-server-linux-configure-environment-variables?view=sql-server-ver15#environment-variables
    environment:
      ACCEPT_EULA: "Y"
      MSSQL_SA_PASSWORD: "${SA_PASSWORD:?err}"
      MSSQL_PID: Developer
      MSSQL_DATA_DIR: /var/opt/mssql/data
      MSSQL_LOG_DIR: /var/opt/mssql/logs
      MSSQL_BACKUP_DIR: /var/opt/mssql/backups
      MSSQL_MEMORY_LIMIT_MB: 2048
      MSSQL_AGENT_ENABLED: "true" #requires string, not bool
      MSSQL_COLLATION: Thai_CI_AS
    volumes:
      - mssql_data:/var/opt/mssql/data
      - mssql_logs:/var/opt/mssql/logs
      - mssql_backups:/var/opt/mssql/backups
  # alternative syntax to just have it in local path
  # - ./sqlsystem:/var/opt/mssql/data
  # - ./sqluser:/var/opt/mssql/sqlserver
  # - ./sqlbackups:/var/opt/mssql/backups

# Top level volumes to define name volumes
# https://docs.docker.com/compose/compose-file/#volumes
volumes:
  mssql_data:
  mssql_logs:
  mssql_backups:

I have the exact same issue but I'm not using ZFS. Plain Debian 10 server. See details here -> https://github.com/microsoft/mssql-docker/issues/542#issuecomment-637229720

Nevermind, fixed it setting the container to run as root. In non-root mode, it can't create the .mdf files on a bind-mount volume (and no, I don't want to use docker volumes).
Previously:
version: '3.6'
services:

  mssql:
    restart: always
    image: mcr.microsoft.com/mssql/server:2019-latest
    ports:
      - 1433:1433
    environment:
      - ACCEPT_EULA=Y
      - SA_PASSWORD=BLAH
    volumes:
      - ./data:/var/opt/mssql/data

Status: Downloaded newer image for mcr.microsoft.com/mssql/server:2019-latest
Creating mssql_mssql_1 ... done
Attaching to mssql_mssql_1
mssql_1  | SQL Server 2019 will run as non-root by default.
mssql_1  | This container is running as user mssql.
mssql_1  | To learn more visit https://go.microsoft.com/fwlink/?linkid=2099216.
mssql_1  | 2020-06-10 09:54:56.74 Server      Setup step is copying system data file 'C:\templatedata\master.mdf' to '/var/opt/mssql/data/master.mdf'.
2020-06-10 09:54:56.94 Server      ERROR: Setup FAILED copying system data file 'C:\templatedata\master.mdf' to '/var/opt/mssql/data/master.mdf':  2(The system cannot find the file specified.)
ERROR: BootstrapSystemDataDirectories() failure (HRESULT 0x80070002)
mssql_mssql_1 exited with code 1

Now:
version: '3.6'
services:

  mssql:
    restart: always
    image: mcr.microsoft.com/mssql/server:2019-latest
    user: root
    ports:
      - 1433:1433
    environment:
      - ACCEPT_EULA=Y
      - SA_PASSWORD=BLAH
    volumes:
      - ./data:/var/opt/mssql/data

Recreating mssql_mssql_1 ... done
Attaching to mssql_mssql_1
mssql_1  | SQL Server 2019 will run as non-root by default.
mssql_1  | This container is running as user root.
mssql_1  | To learn more visit https://go.microsoft.com/fwlink/?linkid=2099216.
mssql_1  | 2020-06-10 10:11:55.69 Server      Setup step is copying system data file 'C:\templatedata\master.mdf' to '/var/opt/mssql/data/master.mdf'.
2020-06-10 10:11:55.91 Server      Did not find an existing master data file /var/opt/mssql/data/master.mdf, copying the missing default master and other system database files. If you have moved the database location, but not moved the database files, startup may fail. To repair: shutdown SQL Server, move the master database to configured location, and restart.
2020-06-10 10:11:55.94 Server      Setup step is copying system data file 'C:\templatedata\mastlog.ldf' to '/var/opt/mssql/data/mastlog.ldf'.
2020-06-10 10:11:55.95 Server      Setup step is copying system data file 'C:\templatedata\model.mdf' to '/var/opt/mssql/data/model.mdf'.
2020-06-10 10:11:55.99 Server      Setup step is copying system data file 'C:\templatedata\modellog.ldf' to '/var/opt/mssql/data/modellog.ldf'.
2020-06-10 10:11:56.02 Server      Setup step is copying system data file 'C:\templatedata\msdbdata.mdf' to '/var/opt/mssql/data/msdbdata.mdf'.
2020-06-10 10:11:56.07 Server      Setup step is copying system data file 'C:\templatedata\msdblog.ldf' to '/var/opt/mssql/data/msdblog.ldf'.
2020-06-10 10:11:56.08 Server      Setup step is FORCE copying system data file 'C:\templatedata\model_replicatedmaster.mdf' to '/var/opt/mssql/data/model_replicatedmaster.mdf'.
2020-06-10 10:11:56.12 Server      Setup step is FORCE copying system data file 'C:\templatedata\model_replicatedmaster.ldf' to '/var/opt/mssql/data/model_replicatedmaster.ldf'.
2020-06-10 10:11:56.14 Server      Setup step is FORCE copying system data file 'C:\templatedata\model_msdbdata.mdf' to '/var/opt/mssql/data/model_msdbdata.mdf'.
2020-06-10 10:11:56.19 Server      Setup step is FORCE copying system data file 'C:\templatedata\model_msdblog.ldf' to '/var/opt/mssql/data/model_msdblog.ldf'.
2020-06-10 10:11:56.43 Server      Microsoft SQL Server 2019 (RTM-CU4) (KB4548597) - 15.0.4033.1 (X64)
mssql_1 Mar 14 2020 16:10:35
mssql_1 Copyright (C) 2019 Microsoft Corporation
mssql_1 Developer Edition (64-bit) on Linux (Ubuntu 18.04.4 LTS) <X64>
2020-06-10 10:11:56.45 Server      UTC adjustment: 0:00
2020-06-10 10:11:56.45 Server      (c) Microsoft Corporation.
2020-06-10 10:11:56.45 Server      All rights reserved.
2020-06-10 10:11:56.46 Server      Server process ID is 36.
2020-06-10 10:11:56.46 Server      Logging SQL Server messages in file '/var/opt/mssql/log/errorlog'.
2020-06-10 10:11:56.47 Server      Registry startup parameters:
mssql_1  -d /var/opt/mssql/data/master.mdf
mssql_1  -l /var/opt/mssql/data/mastlog.ldf
mssql_1  -e /var/opt/mssql/log/errorlog
blah...

Success! :)

Wouldn't it be better to just chown the volume folders to whatever uid the container is using by default?

Wouldn't it be better to just chown the volume folders to whatever uid the container is using by default?

Maybe, but I don't know how to do that with docker-compose. My use-case was creating a brand new container, no existing files (i.e. ./data didn't exist). It works out-of-the-box if I use an existing data folder though, but I needed to create an empty container and found this issue.
If you know how to do that with docker-compose, and it's safer to do this than running it as root, awesome!

Hey, so I ran into this issue, and updated my ZFS just to solve it, but it didn't get rid of the issue, mssql still claims it can't access the files that are stored on the ZFS pool.

This is still an issue on other file systems that do not support O_DIRECT like ramfs.
There are a couple of toggles like -T3979 and control.writethrough and control.alternatewritethrough  (more details at https://bobsql.com/sql-server-on-linux-forced-unit-access-fua-internals/), but from my understanding they do not suppress open(O_DIRECT).
The situation that sql server crashes on ramfs is especially an issue for CI - This cares much more about speed than about consistency in case of power outages or similar.
In our case, we're running the entire /var/lib/docker in ramfs, to speed up testing by an order of magnitude even with 'quick' disks. sql server container are then nested in docker-dind. The option to volume mount 'real disk' as /var/opt/mssql is quite a bit of hassle and also leads to headaches when multiple such containers run in parallel. Not an option in our case.
But the hack from https://github.com/t-oster/mssql-docker-zfs works: It overrides open() to drop O_DIRECT flag and forces this into the system via LD_PREOLOAD. Brutal and for sure not for anything production like, but it's a simple solution for CI.
Put the nodirect_open.c next to a Dockerfile with this content:
FROM mcr.microsoft.com/mssql/server:2019-latest
ADD nodirect_open.c /
USER root
RUN apt-get update \
	&& apt-get install -y gcc \
	&& gcc -shared -fpic -o /nodirect_open.so nodirect_open.c -ldl \
	&& apt-get purge -y gcc binutils binutils-common binutils-x86-64-linux-gnu cpp cpp-7 gcc gcc-7 \
  		gcc-7-base libasan4 libatomic1 libbinutils libc-dev-bin libc6-dev \
  		libcilkrts5 libgcc-7-dev libgomp1 libisl19 libitm1 liblsan0 libmpc3 libmpx2 \
  		libquadmath0 libtsan0 libubsan0 linux-libc-dev manpages manpages-dev \
	&& apt-get clean \
	&& rm -rf /var/lib/apt/lists/* \
	&& echo "/nodirect_open.so" >> /etc/ld.so.preload

This is still an issue on other file systems that do not support O_DIRECT like ramfs.
There are a couple of toggles like -T3979 and control.writethrough and control.alternatewritethrough  (more details at https://bobsql.com/sql-server-on-linux-forced-unit-access-fua-internals/), but from my understanding they do not suppress open(O_DIRECT).
The situation that sql server crashes on ramfs is especially an issue for CI - This cares much more about speed than about consistency in case of power outages or similar.
In our case, we're running the entire /var/lib/docker in ramfs, to speed up testing by an order of magnitude even with 'quick' disks. sql server container are then nested in docker-dind. The option to volume mount 'real disk' as /var/opt/mssql is quite a bit of hassle and also leads to headaches when multiple such containers run in parallel. Not an option in our case.
But the hack from https://github.com/t-oster/mssql-docker-zfs works: It overrides open() to drop O_DIRECT flag and forces this into the system via LD_PREOLOAD. Brutal and for sure not for anything production like, but it's a simple solution for CI.
 ZFS now supports O_DIRECT out of the box, so no hack is necessary anymore.
However, it would be good to have a few infos about the best ZFS configuration (recordsize, primarycache etc) for optimizing sql-server on docker/linux.

The issue has been abused (by me as well) for other file systems, too.
afaiks, mssql still has no solution to suppress O_DIRECT itself, the hack is still required for different file systems.
We could close the issue (judging by issue title), but if so, a new issue regarding O_DIRECT should be created, I suppose.

Nevermind, fixed it setting the container to run as root. In non-root mode, it can't create the .mdf files on a bind-mount volume (and no, I don't want to use docker volumes).
Previously:
version: '3.6'
services:

  mssql:
    restart: always
    image: mcr.microsoft.com/mssql/server:2019-latest
    ports:
      - 1433:1433
    environment:
      - ACCEPT_EULA=Y
      - SA_PASSWORD=BLAH
    volumes:
      - ./data:/var/opt/mssql/data

Status: Downloaded newer image for mcr.microsoft.com/mssql/server:2019-latest
Creating mssql_mssql_1 ... done
Attaching to mssql_mssql_1
mssql_1  | SQL Server 2019 will run as non-root by default.
mssql_1  | This container is running as user mssql.
mssql_1  | To learn more visit https://go.microsoft.com/fwlink/?linkid=2099216.
mssql_1  | 2020-06-10 09:54:56.74 Server      Setup step is copying system data file 'C:\templatedata\master.mdf' to '/var/opt/mssql/data/master.mdf'.
2020-06-10 09:54:56.94 Server      ERROR: Setup FAILED copying system data file 'C:\templatedata\master.mdf' to '/var/opt/mssql/data/master.mdf':  2(The system cannot find the file specified.)
ERROR: BootstrapSystemDataDirectories() failure (HRESULT 0x80070002)
mssql_mssql_1 exited with code 1

Now:
version: '3.6'
services:

  mssql:
    restart: always
    image: mcr.microsoft.com/mssql/server:2019-latest
    user: root
    ports:
      - 1433:1433
    environment:
      - ACCEPT_EULA=Y
      - SA_PASSWORD=BLAH
    volumes:
      - ./data:/var/opt/mssql/data

Recreating mssql_mssql_1 ... done
Attaching to mssql_mssql_1
mssql_1  | SQL Server 2019 will run as non-root by default.
mssql_1  | This container is running as user root.
mssql_1  | To learn more visit https://go.microsoft.com/fwlink/?linkid=2099216.
mssql_1  | 2020-06-10 10:11:55.69 Server      Setup step is copying system data file 'C:\templatedata\master.mdf' to '/var/opt/mssql/data/master.mdf'.
2020-06-10 10:11:55.91 Server      Did not find an existing master data file /var/opt/mssql/data/master.mdf, copying the missing default master and other system database files. If you have moved the database location, but not moved the database files, startup may fail. To repair: shutdown SQL Server, move the master database to configured location, and restart.
2020-06-10 10:11:55.94 Server      Setup step is copying system data file 'C:\templatedata\mastlog.ldf' to '/var/opt/mssql/data/mastlog.ldf'.
2020-06-10 10:11:55.95 Server      Setup step is copying system data file 'C:\templatedata\model.mdf' to '/var/opt/mssql/data/model.mdf'.
2020-06-10 10:11:55.99 Server      Setup step is copying system data file 'C:\templatedata\modellog.ldf' to '/var/opt/mssql/data/modellog.ldf'.
2020-06-10 10:11:56.02 Server      Setup step is copying system data file 'C:\templatedata\msdbdata.mdf' to '/var/opt/mssql/data/msdbdata.mdf'.
2020-06-10 10:11:56.07 Server      Setup step is copying system data file 'C:\templatedata\msdblog.ldf' to '/var/opt/mssql/data/msdblog.ldf'.
2020-06-10 10:11:56.08 Server      Setup step is FORCE copying system data file 'C:\templatedata\model_replicatedmaster.mdf' to '/var/opt/mssql/data/model_replicatedmaster.mdf'.
2020-06-10 10:11:56.12 Server      Setup step is FORCE copying system data file 'C:\templatedata\model_replicatedmaster.ldf' to '/var/opt/mssql/data/model_replicatedmaster.ldf'.
2020-06-10 10:11:56.14 Server      Setup step is FORCE copying system data file 'C:\templatedata\model_msdbdata.mdf' to '/var/opt/mssql/data/model_msdbdata.mdf'.
2020-06-10 10:11:56.19 Server      Setup step is FORCE copying system data file 'C:\templatedata\model_msdblog.ldf' to '/var/opt/mssql/data/model_msdblog.ldf'.
2020-06-10 10:11:56.43 Server      Microsoft SQL Server 2019 (RTM-CU4) (KB4548597) - 15.0.4033.1 (X64)
mssql_1 Mar 14 2020 16:10:35
mssql_1 Copyright (C) 2019 Microsoft Corporation
mssql_1 Developer Edition (64-bit) on Linux (Ubuntu 18.04.4 LTS) <X64>
2020-06-10 10:11:56.45 Server      UTC adjustment: 0:00
2020-06-10 10:11:56.45 Server      (c) Microsoft Corporation.
2020-06-10 10:11:56.45 Server      All rights reserved.
2020-06-10 10:11:56.46 Server      Server process ID is 36.
2020-06-10 10:11:56.46 Server      Logging SQL Server messages in file '/var/opt/mssql/log/errorlog'.
2020-06-10 10:11:56.47 Server      Registry startup parameters:
mssql_1  -d /var/opt/mssql/data/master.mdf
mssql_1  -l /var/opt/mssql/data/mastlog.ldf
mssql_1  -e /var/opt/mssql/log/errorlog
blah...

Success! :)

this helped me solve my mistake, thank you


Add missing @SuppressWarnings to InconsistentOverloads documentation

see also https://github.com/google/error-prone/issues/717

@cushon wanna merge this? As discussed over in https://github.com/google/error-prone/issues/869 ..

@SuppressWarnings("UngroupedOverloads") now works on methods, as of e7355b5191a3259c11b2e02d39ac6b64e7399be4


Download failed rows

We've been working a store that has their data all over the place and have build a bunch of custom processors on top of the existing ones to make everything work. One issue we've seen is that if there is a significant number of failures on any particular type of import, usually that means that someone needs to go follow up in some capacity.
If it were possible to reconstruct the failed rows of the CSV and allowed that to be downloaded, that could really be useful in those failure cases.
I've personally done that with some console-fu, but it would be pretty easy to add a "Download failed rows" button to the import page. Would this enhancement be useful to others?

Still a cool thing I'd like to see.

Hey, we're going to try taking a stab at this! @Noah-Silvera


Fix a race condition when reveal is using animations

Fixes issue #9291

@Owlbertz @colin-marshall can one of you review this and merge if it looks good?  Thanks


Make initial search depth a parameter

Currently, Matcher.match_hashes only examines the top 100 most promising ref items to calculate the modal time skews and time-filtered hash counts. For larger databases, this may not be deep enough to find true matches. In any case, hard-coding a parameter like this is very poor style.

Closed via https://github.com/dpwe/audfprint/commit/0c6d7ad80e9f0a0f3be3cfddf81a985135becbeb


fedora 38: osbuild-composer installation failed because of osbuild >=93 not found

CI test job minimal-raw.sh failed on fedora38 because of the downloaded image not found, after check,  the image file has no xz as filename extension.
The check the osbuild-composer version, and found these installation failures:
Problem: package osbuild-composer-88-1.fc38.x86_64 requires osbuild-composer-worker = 88-1.fc38, but none of the providers can be installed

cannot install the best candidate for the job
nothing provides osbuild >= 93 needed by osbuild-composer-worker-88-1.fc38.x86_64
nothing provides osbuild-luks2 >= 93 needed by osbuild-composer-worker-88-1.fc38.x86_64
nothing provides osbuild-lvm2 >= 93 needed by osbuild-composer-worker-88-1.fc38.x86_64
nothing provides osbuild-ostree >= 93 needed by osbuild-composer-worker-88-1.fc38.x86_64

These failures caused a wrong version of 78 installed, so caused minimal-raw.sh failed.
[root@yih-fedora38 rhel-edge]# rpm -qa|grep osbuild
osbuild-composer-dnf-json-78-1.fc38.x86_64
osbuild-composer-core-78-1.fc38.x86_64
python3-osbuild-91-1.fc38.noarch
osbuild-selinux-91-1.fc38.noarch
osbuild-91-1.fc38.noarch
osbuild-lvm2-91-1.fc38.noarch
osbuild-ostree-91-1.fc38.noarch
osbuild-luks2-91-1.fc38.noarch
osbuild-composer-worker-78-1.fc38.x86_64
osbuild-composer-78-1.fc38.x86_64

Should be as same as bug https://github.com/virt-s1/rhel-edge/issues/4320


No documentation for .post requests

Where does one find any documentation to use this framework. I was expecting at least some information on how to create a post request with a custom body parameter. But nothing...

anything new?


semantic ui icons not loading although everything is fine at console and network log

Could be related to CORS headers on your server

I don't think so because I am loading from same origin I mean all I serving through localhost...

Everything is working fine after deleting cache both in chrome and firefox.


test(e2e): Add tests for transactions pages

Motivation
We want to create snapshots test for each native token transaction page: BTC, ETH, and ICP.
Changes

Add additional customizable function to extend HomepageLoggedIn.waitForReady
Create class TransactionPage extending HomepageLoggedIn.
Create tests for BTC, ETH and ICP.

Tests
In this PR we already updated the new screenshots. Please check committed files.

I assume not much we can do with the txs call on the ETH page?

I tried to check, but it is really a question of API keys, and for now no, unfort...


⚠ Bofrost.at has degraded performance

In 6980ce5, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 2296 ms

Resolved: Bofrost.at performance has improved in 6e4dde6 after 10 minutes.


docs: improve README.md

I added back ticks to several areas of the README to tell AnimeTask objects apart from regular text. Before this is merged, I would also like to discuss a few parts of the docs that are not very clear:

https://github.com/kyubuns/AnimeTask/blob/c0c6637a0fe69c203dcee201f8791e0f9680bb9c/README.md?plain=1#L48
What does 1 per second mean? Does it mean 1 pixel per second, 1 megapixel per second, or 1 [other unit] per second?
https://github.com/kyubuns/AnimeTask/blob/c0c6637a0fe69c203dcee201f8791e0f9680bb9c/README.md?plain=1#L38
Did you mean Easing.InCubic?

Thanks again!

What does 1 per second mean? Does it mean 1 pixel per second, 1 megapixel per second, or 1 [other unit] per second?

This is the unit of 1 on Unity. (Unity does not explicitly clarify this unit...)
In this case, should I write "1 (unity) unit per second"?

Did you mean Easing.InCubic?

Yes!

In this case, should I write "1 (unity) unit per second"?

It's up to you. If you want me to leave it alone, I'll leave it alone. You specify the changes to implement, and I'll implement them (tomorrow).

I see, It is fine as it is now.
Thanks!


[cPP ENHANCEMENT] Add Ed448 to FCS_CKM.1 and FCS_COP.1/SigGen

Provide the location of the issue
cPP FCS_CKM.1 and FCS_COP.1/SigGen
What is the enhancement request for the cPP? Please describe.
Ed25519 was added as per #127 but Ed448 was not.  Ed448 now has available ACVP testing under FIPS 186-5 and is NIST approved and therefore should be an available selection.  This is somewhat more complicated by the fact that Ed448 requires SHAKE as the hashing algorithm.  This was described in issue #272 as the primary rationale for not including it at the outset.  I'd like this issue to remain open pending addition.  Note that this issue will work in tandem with issue #326 to add SHA3/SHAKE hashing algorithms.
Describe the solution you'd like
Update FCS_CKM.1.1 to modify "ECC schemes using Ed25519 that meet the following: FIPS PUB 186-5, “Digital Signature Standard (DSS)”, Appendix A.2.3;" to be "ECC schemes using [selection: Ed25519, Ed448] that meet the following: FIPS PUB 186-5, “Digital Signature Standard (DSS)”, Appendix A.2.3;"
Update FCS_COP.1.1/SigGen to modify "For EdDSA: Ed25519" to "For EdDSA: [selection: Ed25519, Ed448]" in the second selection.
Assurance activities to be defined closer to the time that this is being added.

@mclearn One of NIAP's comments on and requirements for endorsing the NDcPPv3.0 was to remove EdDSA from FCS_CKM.1.1.


"Duplicate" hard links are skipped

If you have 2 or more files that are hard linked, erdtree will only display 1 of them.
e.g.
touch a
ln a b
et

will cause erdtree to output
mydirectory
└─ b (0 B)

The issue also happens if the files are in different subdirectories.
e.g.
mkdir 1
mkdir 2
touch 1/a
ln 1/a 2/b
et

will cause erdtree to output
mydirectory
├─ 1
│  └─ a (0 B)
└─ 2

Oh I just noticed it mentions this in the readme. Perhaps it could display them but not count towards the disk usage?

Yeah that's probably more sensible isn't it. I'll be sure to have a fix out for that in the next release :)

Just merged https://github.com/solidiquis/erdtree/pull/136 into my work-in-progress v2.0 branch. If all goes well this will be included in the next release this Sunday :]

https://github.com/solidiquis/erdtree/pull/131


Optimization: Use compact encoding of bytecode

Depends on #98
For now, every instruction is represented by an enum value. It means that each instruction occupies size required for a tag plus a size of the instruction that has the largest payload (probably BrTable) (which requires to be properly aligned). Now each instruction occupies about 24 bytes, even though most instructions are 1 byte wide.
Fortunately, we don't need to address instructions by an index and can iterate them sequentially, and branches can specify the exact byte offset for their target.
Shrinking the size of one instruction will allow us to use cache more efficiently and also will reduce memory overhead greatly.

I got this working with a crude encoding scheme. Each isa::Instruction is identified with a single u8, followed by any operands (usually u32). As of now, I'm seeing a 10-15% performance improvement in cargo bench results.
The savings aren't as large as I had hoped. A napkin-math analysis over the test suite gives me a thought:

Most of the u32 operands are slots or offsets or similar small integers. Maybe a variable length encoding scheme is worth a go?

10-15% is a very good figure actually!
This is basically a trade-off and we can easily get more overhead than we save by compacting the bytecode. What we might want to explore is introduction of a special narrow versions of opcodes that takes, say, u16 instead of u32 immediates.

Well… then I started thinking about how WebAssembly is already a compact bytecode format with variable-length encoding of operands. FunctionReader::read_instruction() swizzles around flow control, but many more instructions are validated then emitted 1:1. Maybe it's better not to produce a second bytecode, but to instead produce a mix of new/enriched flow control instructions along with references to the original bytecode.

Maybe it's better not to produce a second bytecode, but to instead produce a mix of new/enriched flow control instructions along with references to the original bytecode.

Can you expand a little on this? I'm not sure I follow.

Related:

https://github.com/paritytech/parity-wasm/issues/135
https://github.com/paritytech/parity-wasm/pull/154
https://github.com/paritytech/parity-wasm/issues/224
https://github.com/yurydelendik/wasmparser.rs/pull/45

I am going to implement a more compact encoding of wasmi bytecode for the v1 implementation introduced in https://github.com/paritytech/wasmi/pull/287.

The mentioned PR (https://github.com/paritytech/wasmi/pull/287) shrank the size of the enum bytecode from 24 bytes to 16 bytes by disentangling BrTable variant. The BrCode operand now only hosts the number of targets excluding the default target and is followed by Branch and Return operands by construction of the wasmi bytecode.
Closing this issue now since we probably won't get way lower than that given that we embed constant values into the bytecode since indirectly accessing constants (that must be 64-bits) would create too much overhead.


Add iRPlus Stats API into ADC formal API

Discussion of the ability to better count in #480
General consensus is that a lot of work went into the iRPlus Stats API - https://github.com/ireceptor-plus/WP2/blob/master/specs/stats-api.yaml
Suggested that this be considered as an official ADC API.

FYI, this is the one that VDJServer references vs the hackathon.
https://github.com/ireceptor-plus/specifications/blob/master/stats-api.yaml

@bcorrie For an official ADC API, are you suggesting that the spec would get moved into air-standards repository, or stay in its current repository? Put in the ADC API spec directly, or as another spec file?

Note that there have been some important updates to the spec past the stats-1.0.0 tag

@bcorrie For an official ADC API, are you suggesting that the spec would get moved into air-standards repository, or stay in its current repository? Put in the ADC API spec directly, or as another spec file?

I am not sure. My feeling is that this might be a separate API like /airr/stats/v1. More link the async API I would envision, but could be convinced otherwise.


Bootstrap context blocks operations in single node mode

If there is no other node, bootstrap context should never be created or cancelled properly.

Fixed in #10


AttributeError: 'User' object has no attribute 'csrf_token'

When i click on the Edit user button I got a error.

This is the content of this error:
Sorry, something went wrong
500 - Internal Server Error
Stacktrace
         Traceback (most recent call last):
  File "/home/michal/Pobrane/superset/env/lib/python2.7/site-packages/Flask-0.12-py2.7.egg/flask/app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/michal/Pobrane/superset/env/lib/python2.7/site-packages/Flask-0.12-py2.7.egg/flask/app.py", line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/michal/Pobrane/superset/env/lib/python2.7/site-packages/Flask-0.12-py2.7.egg/flask/app.py", line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/michal/Pobrane/superset/env/lib/python2.7/site-packages/Flask-0.12-py2.7.egg/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/michal/Pobrane/superset/env/lib/python2.7/site-packages/Flask-0.12-py2.7.egg/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/michal/Pobrane/superset/env/lib/python2.7/site-packages/Flask_AppBuilder-1.8.1-py2.7.egg/flask_appbuilder/security/decorators.py", line 26, in wraps
    return f(self, *args, **kwargs)
  File "/home/michal/Pobrane/superset/env/lib/python2.7/site-packages/Flask_AppBuilder-1.8.1-py2.7.egg/flask_appbuilder/views.py", line 59, in this_form_get
    self.form_get(form)
  File "/home/michal/Pobrane/superset/env/lib/python2.7/site-packages/Flask_AppBuilder-1.8.1-py2.7.egg/flask_appbuilder/security/views.py", line 102, in form_get
    form_field.data = getattr(item, key)
AttributeError: 'User' object has no attribute 'csrf_token'

All other features work good. My version of superset is 0.15.4.

This may be a bug with FAB or your specific configuration (if any)

same question

How can I reproduce this?

@mistercrunch WTF_CSRF_ENABLED = True

@gloryballet I change this option to True but i was still the same error. My solution is add '#' comment to line 102 "form_field.data = getattr(item, key)" in file located at: "/home/michal/Pobrane/superset/env/lib/python2.7/site-packages/Flask_AppBuilder-1.8.1-py2.7.egg/flask_appbuilder/security/views.py"
Now it's working good, error don't show.
Thans for help.

I got same problem with default configuration, the superset version is 0.17.2

same problem +1 ， superset version： 0.17.4

I have the same problem with version 0.17.5, python 3.6.1. How can I help to reproduce it?

This feature is part of FAB (Flask-AppBuilder), which in turn rely on Flask-WTF, and WTFForms.
As far as I discovered, those environment variable names were changed and not sync anymore by the time of writing this.
Adding this to config.py will disable CSRF, which does not work ever.
WTF_CSRF_ENABLED = False

I sent a PR to F.A.B, which should fix that.

I have merge rights on that repo, will review. I can't push a release though as I don't have rights on Pypi for that package and the unit tests are broken at the moment which is a blocker.
It would be nice if FAB was more actively maintained. It's hard for me to take on a 3rd project...

@mistercrunch, sent a PR which fixed CI on FAB, with few tests ignored.
https://github.com/dpgaspar/Flask-AppBuilder/pull/458
https://travis-ci.org/dpgaspar/Flask-AppBuilder/builds/229645773


gnuradio import droneid

Had to clone turbofec (https://github.com/ttsou/turbofec) & CRCpp (https://github.com/d-bahr/CRCpp) in order to build on Ubuntu 22.04 with gnuradio 3.10.
However when I go into the droneid_test.grc, it comes up with an error about importing droneid.
It does show the droneid grc blocks in gnuradio-companion however.

Could you share the exact version of GNU Radio that you're using and whether or not you built it from source? That would help me try to recreate the issue.
Could you also share the error that you're seeing?

Could you share the exact version of GNU Radio that you're using and whether or not you built it from source? That would help me try to recreate the issue.
Could you also share the error that you're seeing?

Gnuradio <IP_ADDRESS>-2 and the import error is in gnuradio-companion, in the import block.  Import droneid from gnuradio.

I think you may have been bitten by Python not looking in /usr/local/lib/python<VERSION>/dist-packages. If you run find /usr/local/lib/python3.* -name droneid -type d you should get a single directory where the droneid package lives. This all assumes that you didn't do any trickery when installing the module.
As an example, if the command above returned /usr/local/lib/python3.10/dist-packages/gnuradio/droneid then you could launch GNU Radio Companion from the command line with: PYTHONPATH=$PYTHONPATH:/usr/local/lib/python3.10/dist-packages gnuradio-companion and things should be happy. The $PYTHONPATH up front is there in case the PYTHONPATH environment variable has already had paths added to it.
Glad to hear a reboot seems to have solved the problem!


Include GPhotosShim like app in the Apps

Please add an app like GPhotosShim in the Graphene os Apps store, i know currently the Apps store includes only the apps made by Graphene OS team or google apps which are not easy to download. And i am not suggesting to include someone else's app also.
Considering this will be a very small but useful app if graphene os team can create an app like GPhotosShim and include in the app store it will be very helpful for people like me who dont want to sideload apps from unknown/unmaintained github repos and uses google playstore or Apps appstore for installing applications.

A lot of the time people have these apps installed and then try to install Google Photos in another profile and wonder why they can't or vice versa.

I completely agree with your points but due to some users installing this app and trying to install google photos app in the other profile and failing it doesnt mean that we have to let other users (who knows what they are doing) installing like these apps from some unknown git repos. I think giving clear app description that installing this app makes it impossible to install google photos app in current or other profiles should be sufficient.

I don't think we want to do this right now.

Again i respect your decision on this, maybe could reconsider in the future.

This would preferably be a very small OS level component installed by default rather than a optional addon, to avoid confusion, and make the UX seamless.
The Open With UI could promt to define the user's preferred gallery app when they tap the photo preview. This would avoid breaking existing setups with Google Photos installed.

This would preferably be a very small OS level component installed by default rather than a optional addon

This would block users from ever being able to install Google Photos when desired

No. Please see full context before making a point:

The Open With UI could promt to define the user's preferred gallery app when they tap the photo preview. This would avoid breaking existing setups with Google Photos installed.

Are you aware of the existence of the link handler dialogue? What prevents the user from specifying Photos as the default Gallery app?

Its not black magic. GPhotosShim simply forwards the intent to the Gallery of your choice, which could be Google Photos as well.


[bug]: keyspace name in AxonOps does not match keyspace hame in DB

Do I have the most recent version of AxonOps Workbench installed?

[X] I am using the most recent beta release

AxonOps Workbench Version
v1.0.0-beta-20240909-01
Is there an existing issue for this?

[X] I have searched the existing issues

Operating System
MacOS
Operating System Version
14.6.1 (23G93)
Apache Cassandra Version(s)
4.1.6
Bug Description
Keyspace is called geetu - but is being show as geet in the tree (tested on Mac and Windows)

Steps to Reproduce
NA
Expected Behavior
NA
Actual Behavior
NA
Severity
High
Further Information
No response

Fixed, pushed a commit for it


sqlx 查询没有记录时，不应该返回 error

sqlx 查询没有记录时，不应该返回 error；我觉得没有记录和执行错误是两个概念，但是 sqlx 把这两个东西混在一起了，对我们开发来说，有点不方便，我们很难分辨哪些是没有记录，哪些是错误的。

Please use English, there are a lot of people don't speak Chinese.
For your question, you can use  err == sqlx.ErrNoRows to determine the result is empty rather than other error. If the API does't return error, how can you know the result is empty, you can't compare returned value to default value since there are values equal default value.

it has been solved. thank you! I will close it.


LWM2M: Fix check_authorization() for Access Control Objects

I'm improving the src/test/test-lwm2m.c program to cover all new functionalities
added during GSoC, and found this issue.
This patch fixes a bug when a given LWM2M Server who owns a given
Access Control Object Instance tries to modify it (for instance,
to add or remove an ACL for a given third-party LWM2M Server).
There was no way to modify an Access Control Object Instance, since
the check_authorization() would fail, not finding an Access Control
Object Instance related to the target instance - which should not
exist anyway. To fix it, a new condition was added in case the target
obj_id == ACCESS_CONTROL_OBJECT_ID, in which we should only look
for the target instance_id itself and then inside the OWNER Resource,
not looking for ACLs or anything else.
Signed-off-by: Bruno Melo<EMAIL_ADDRESS>

lgtm

An updated version is at #2282. Thanks!


psh: sysexec whitelist feature

-added sysexec wildacrds. Allows execution only of predefined commands in /etc/wldcrds file (each command in newline)
or predefined in PSH_WILDCARDS enviroment variable (all comands in one line separated only by ';' sign)
If /etc/wldcrds file is not existing and no wildcards are specified in PSH_WILDCARDS then sysexec accepts all commands as usual
-removed #ifdef-s from psh login feature and assured that flags are defined in Makefile
JIRA: BES-148

Checks not successful  because of concurrency during building

Changes added, commits squashed to one.


[Bug]: Distribution transaction fails when trying with concurrently safe service

Description
Program exit with following error.
error: {ballerina}AsyncCallInsideLockError
	at ballerinai.transaction.0.TwoPhaseCommitTransaction:prepareParticipants(2pc_transaction.bal:163)
	   ballerinai.transaction.0.TwoPhaseCommitTransaction:twoPhaseCommit(2pc_transaction.bal:64)
	   ballerinai.transaction.0:endTransaction(transaction_block.bal:126)
	   waruna.BallerinaBasics.transactions.0:callDistributedTrxFunction(distributed_transactions.bal:43)
	   waruna.BallerinaBasics.transactions.0.$anonType$_3:$post$bookings(distributed_transactions.bal:25)

This works when I removed locks and isolated service.
Steps to Reproduce
Sample bal file

import ballerina/http;
import ballerina/io;
import ballerina/random;

type Room record {|
    int number;
|};

type Payment record {|
    decimal amount;
|};

isolated service / on new http:Listener(8081) {
    private http:Client roomsReserveClient;
    private http:Client paymentClient;

    function init() returns error? {
        self.roomsReserveClient = check new ("http://localhost:9000/reservations");
        self.paymentClient = check new ("http://localhost:9001/finance");

    }
    isolated resource function post bookings(Room room) returns error? {
        lock {
            check callDistributedTrxFunction(self.roomsReserveClient, self.paymentClient);
        }
    }
}

isolated function callDistributedTrxFunction(http:Client roomsReserveClient, http:Client paymentClient) returns error? {
    transaction {
        http:Response|http:ClientError res1 = check roomsReserveClient->/rooms.post({"number": 1});
        if (res1 is http:Response && res1.statusCode != 200) {
            io:println(res1.statusCode);
            rollback;
            check error("Error occurred while reserving rooms");
        } else {
            http:Response|http:ClientError res2 = check paymentClient->/payments.post({"amount": 1000});
            if (res2 is http:Response && res2.statusCode != 200) {
                rollback;
                fail error("Error occurred while paying for room");
            } else {
                check commit;
                io:println("Transaction completed successfully");
            }
        }
    }
}

service /reservations on new http:Listener(9000) {
    transactional resource function post rooms(Room room) returns error|http:Ok {
        io:println(`room ${room.number} is reserved`);
        transaction:onCommit(sendEmail);
        transaction:onRollback(releaseRooms);
        return http:OK;
    }
}

service /finance on new http:Listener(9001) {
    resource function post payments(Payment payment) returns error|http:Ok {
        transaction {
            int|random:Error randomNumber = random:createIntInRange(0, 5);
            if randomNumber is random:Error || randomNumber < 3 {
                rollback;
                io:println(`payment failed due to random number ${randomNumber}`);
                return error("payment failed");
            } else {
                check commit;
                io:println(`payment sucessfull with random number ${randomNumber} and amount ${payment.amount}`);
                return http:OK;
            }
        }
    }
}

Sample request
POST http://localhost:8081/bookings/
content-type: application/json

{
    "number": 101
}

Affected Version(s)
No response
OS, DB, other environment details and versions
No response
Related area
-> Compilation
Related issue(s) (optional)
No response
Suggested label(s) (optional)
No response
Suggested assignee(s) (optional)
No response

This is because futures are used in prepareParticipant to handle calls to prepare each participant in a distributed (2PC) transaction. The presence of a lock can disrupt the async nature, potentially causing deadlocks, hence, the AsyncCallInsideLockError is thrown.
Since the need for the lock here is accessing a mutable field outside a lock statement, we can make http clients final removing the need for a lock, as a workaround.


remove ctrl-enter (it's in the mainline webapp now)

Thanks for the heads up! Will remove in the next release.


Footer correction : Removed special symbol

Pull Request
Description
There are unwanted special symbols(=====) in the footer which I have successfully removed.
Closes #
Type of change

[ ] Adding a new Resource
[ ] Adding a new Category
[ ] Documentation enhancement or fixes
[ ] Website enhancement or fixes

Checklist:

[ ] I read carefully CONTRIBUTING.md
[ ] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation

Screenshots (if appropriate):
Please add screenshots to help explain your changes.
Additional

Can you solve the merge conflict? And give me the refrence issue, thans


Fixed skip-if-no-diff when triggered by workflow_dispatch

#3139  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @CatChen and the rest of your teammates on  Graphite


[PoC] create initial structure of directories for collections

Each collection repository should have at least two roles directories initialised with ansible-galaxy init command:

apply
upgrade

My personal opinion that in this step we should also add download role to each collection and as tasks progresses (around #2693) we will probably exclude this role to separate collection.

Roles in a collection cannot contain plugins any more. Plugins must live in the collection plugins directory tree. Each plugin is accessible to all roles in the collection.


Enhancement: add rest configuration to be provided from schema

Requirements
Rest configruation can be provided by schema.
It should also be allowed to be overridden from input property rest.

Fixed and tested.


Add c

Fast parser that assumes valid input
fast encoder that connects to parser to generate nibs

Parser speeds seem good so far.  Parsing 25M file in 49ms.
tim@litebrite:~/Code/nibs/c$ time ./tibs ../bench/corgis/website/datasets/json/wind_turbines/wind_turbines.json

real    0m0.054s
user    0m0.049s
sys     0m0.001s
tim@litebrite:~/Code/nibs/c$ ls -lh ../bench/corgis/website/datasets/json/wind_turbines/wind_turbines.json
-rw-r--r-- 1 tim tim 25M May 22 00:34 ../bench/corgis/website/datasets/json/wind_turbines/wind_turbines.json

The code size is good too. 39K of fully static linux binary.
tim@litebrite:~/Code/nibs/c$ ls -lh tibs
-rwxr-xr-x 1 tim tim 39K Jun 15 23:25 tibs
tim@litebrite:~/Code/nibs/c$ file tibs
tibs: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, with debug_info, not stripped


Cannot pick files from Google Drive

Description
Picker opens, I get intent popups, tapped Allow, then selected Google Drive. All image files are grayed out.
Expected behavior
Able to pick up files from Google Drive. Xamarin.Essentials picker can do it.
Configuration

Device: Galaxy Tab A SM-510
OS Version: Android 11, One UI 3.1
Plugin Version: latest

hi. Is this a problem with local files?

intent.PutExtra(Intent.ExtraLocalOnly, true);
This issue lies in this line. Unfortunately, if we remove it, we will get even more strange behavior on different Chinese devices. I faced the problem that when picking a video file from a cloud, the device returned its thumbnail (Low quality image) instead of the file itself.
I'm also surprised that Google Drive doesn't allow us to select local files. I think we can write to them about it

Sorry for the delay. I am real busy (arent we all) - I'll try to repro it when I get time. It's fifth or six in my OS github repro queue...

https://github.com/dimonovdd/Xamarin.MediaGallery/issues/93#issuecomment-1038488723

I found some time & made a sample!

Very interesting bug:
Pick from local files works fine, but when I try to pick from Google Drive, everything is gray.
Xamarin.Essentials image picker works without any extra steps.

Repro steps:

VS 2019

provided project

Set up Emulator as shown (Essentially Pixel/R/30 with 4GB RAM and forced GPU)

set up Google account there & make sure you have some images in your Google Drive app

Try to pick image from Drive. All is gray, you can only pick 'cancel'.

NOTE: Picking from Google Photos works!

MediaGallery.zip

@smalgin So far I see one solution: We can try not to display Google Drive to users at all
http://stackoverflow.com/a/27780510/106941

I thought so. Makes sense in your position.
But why Xamarin.Essentials picker doesn't have this issue???
Mystery...

But why Xamarin.Essentials picker doesn't have this issue???

Because it has other problems. Read all my messages. It describes what causes this problem

In order to hide Google drive for video files picker can be used:
intent.SetType("video/*");
intent.SetAction(Intent.ActionPick);

And to hide Google drive for image files picker
intent.SetType("image/*");
intent.SetAction(Intent.ActionPick);


Community meeting logs from 07-21

This is the community meeting logs.
Sorry i wasn't here.

Tested ok.


{background-color:"blue" }   blue background not displayed properly

| Custom  | css                             | cells |
| ------- | ------------------------------- | ----- |
| foo ~ {color:"red" }  | foo ~ {background-color:"blue" } | <     |
|         |                                 |       |

{background-color:"blue" }
blue background not displayed properly

Can you try backgroundColor instead

IsOkOkay, all the “-”  in CSS need to be removed?
Do the following characters need to be capitalized?

Yes, css styles which are accessible in JS end up replacing a-b with aB. I can try to add both as valid, but for now the plugin just uses camelCase for css styles.


Validate Code Generation as Part of CI

Description
Adds code generation validation as part of CI. Swagger configuration directories can copy the Update-Codegeneration.ps1 scripts added to Storage to enable a single script to update code generation for that Swagger. This is then consumed by Invoke-Codegeneration.ps1 in CI validation to ensure that the package has up-to-date code generation, and no handwritten changes that would be overwritten when regenerating.
All SDK Contribution checklist:

[x] The pull request does not introduce [breaking changes]
[x] CHANGELOG is updated for new features, bug fixes or other significant changes.
[x] I have read the contribution guidelines.

General Guidelines and Best Practices

[x] Title of the pull request is clear and informative.
[x] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

[x] Pull request includes test coverage for the included changes.

API change check
API changes are not detected in this pull request.

Added @weidongxu-microsoft so we can also make sure the track 2 mgmt libraries can also use this and we are consistently validating those as well.

Added @weidongxu-microsoft so we can also make sure the track 2 mgmt libraries can also use this and we are consistently validating those as well.

Most mgmt libraries starts a PR with a regen, so it mostly conform to this pattern (though not exactly the same procedure) https://dev.azure.com/azure-sdk/internal/_build?definitionId=2238
I would be thinking about how we do Cadl codegen like this (that going to be more complex, as e.g. .NET have a cadl-location file and script to do sparse checkout), and it would indeed save lots of time from dev.

@alzimmermsft this seems to have broken other pipelines like template https://dev.azure.com/azure-sdk/public/_build/results?buildId=2056475&view=logs&j=b70e5e73-bbb6-5567-0939-8415943fadb9&t=6384a917-623f-5c88-6da7-ff190e9f0bf0 We probably should make sure that all the generated code matches before enabling this.


Export fails for duplicate "extra" field

I have (at least) one item in my library that causes the BBT exporter to fail - most items export without issue. (There's nothing obvious about the item jumping out at me, although it does contain a few special characters.) Exporting e.g. using Zotero RDF works fine.
Report ID: ZMABXAF7-euc
Exporter used: Better BibLaTeX
Expected behavior: Should generate the .bib file.
Actual behavior: Fails with the following alert: "An error occurred while trying to export the selected file."
I'm on Zotero version 5.0.95 and BBT version 5.2.113 (but I noticed the behaviour before updating BBT just now). Thanks for your help!

Can you try build 377?

That works great, thanks!

Super, thanks for the confirmation (and the report).


error: package directory 'projects/PointRend/point_rend' does not exist

I get the following error when i run
error: package directory 'projects/PointRend/point_rend' does not exist
python setup.py build develop
to install detectron2. Using pytorch version 1.8.

You can copy the PointRend from detectron2 repo and place it in the BoxTeacher/projects/


System.ObjectDisposedException: Cannot access a disposed object. Object name: 'Javax.Net.Ssl.SSLSocketFactoryInvoker'.

System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'Javax.Net.Ssl.SSLSocketFactoryInvoker'.
at Java.Interop.JniPeerMembers.AssertSelf (Java.Interop.IJavaPeerable self) [0x00029] in <4085581b59834ca7b70d8a8fb4322f71>:0
at Java.Interop.JniPeerMembers+JniInstanceMethods.InvokeAbstractObjectMethod (System.String encodedMember, Java.Interop.IJavaPeerable self, Java.Interop.JniArgumentValue* parameters) [0x00000] in <4085581b59834ca7b70d8a8fb4322f71>:0
at Javax.Net.Ssl.SSLSocketFactoryInvoker.CreateSocket (Java.Net.Socket s, System.String host, System.Int32 port, System.Boolean autoClose) [0x00078] in :0
at ModernHttpClient.TlsSslSocketFactory.CreateSocket (Java.Net.Socket s, System.String host, System.Int32 port, System.Boolean autoClose) [0x00000] in <4bb329b36d5a40fd81d41dfcbe963b33>:0
at Javax.Net.Ssl.SSLSocketFactory.n_CreateSocket_Ljava_net_Socket_Ljava_lang_String_IZ (System.IntPtr jnienv, System.IntPtr native__this, System.IntPtr native_s, System.IntPtr native_host, System.Int32 port, System.Boolean autoClose) [0x00019] in :0
at (wrapper dynamic-method) Android.Runtime.DynamicMethodNameCounter.538(intptr,intptr,intptr,intptr,int,bool)
I am seeing this error occurring in Android 4.4.3 devices. However, its not occurring in other OS versions.
I am on v 3.4.2 of the package. I see that there is v3.4.3 available but cannot find the fix in its release notes. Is there any fix available for this?

Can you help me to reproduce this issue?

This error is thrown randomly, there is no common pattern.
I have seen it in Android OS 4.4.3 only so far.


Mute "NESTED_EMPHASIS" warning in html-tidy

Mute annoying warning splash window for nested_emphasis in html-tidy

I assume this happens when there are nested  tags in the source code. What does it do with them, does it remove the redundant em's? format-all is meant to be quite unopinionated, so we probably shouldn't disable warnings for all users. Does tidy provide an environment variable or configuration file you can use to mute warnings?

I assume this happens when there are nested  tags in the source code.
When I export my org file to HTML, some nested tags like something is generated.

What does it do with them, does it remove the redundant em's?
Yes, tidy turns something into something.

format-all is meant to be quite unopinionated, so we probably shouldn't disable warnings for all users.
That's a good point. But I think for this kind of common warnings and everyone definitely wants to ignore, it is nice that format-all could do this for us. At the same time, format-all might provide  interface to customize each specific formatter.
For instance, format-all-html-tidy-mute to mute any unwanted warnings.
I know this would make format-all more complex, but also more comprehensive.

Does tidy provide an environment variable or configuration file you can use to mute warnings?
Yes, it can use a configuration file. But you need to provide it via command line arguments.

I'm not sure whether everyone wants to ignore those warnings. HTML Tidy can do quite extensive rewrites to the markup when it adds or removes tags, and I've found such warnings to be helpful in those cases; I need to re-check the markup since it sometimes makes a wrong inference and messes up the rewrite.
Format-all can pass custom command line arguments to any external formatter, so one way to disable the warning is to put this in a hook function:
(setq format-all-formatters '(("HTML" (html-tidy "--mute" "NESTED_EMPHASIS"))))

I'm not sure whether everyone wants to ignore those warnings. HTML Tidy can do quite extensive rewrites to the markup when it adds or removes tags, and I've found such warnings to be helpful in those cases; I need to re-check the markup since it sometimes makes a wrong inference and messes up the rewrite.

OK. I close the issue.

Format-all can pass custom command line arguments to any external formatter, so one way to disable the warning is to put this in a hook function:
(setq format-all-formatters '(("HTML" (html-tidy "--mute" "NESTED_EMPHASIS"))))

If you have format-all-debug on, it shows in *Messages* that the args are prepended to the usual args:
Format-All: Formatting foo as HTML using html-tidy --mute NESTED_EMPHASIS
Format-All: Running: /usr/local/bin/tidy --mute NESTED_EMPHASIS -q --tidy-mark no -indent

Cool. Thanks. That's very helpful.


ToolTask: Preserve the temporary files if MSBUILDPRESERVETOOLTEMPFILES

.. environment variable is set. This affects temporary files like
script/batch files, response files generated by Exec task or similar
temporary files generated by Tasks inheriting from ToolTask .

@dotnet-bot test Windows_NT Build please

LGTM, thanks.

Do I need to do anything else to have this documented?

Historically, these environment variables have been entirely undocumented or documented only informally through blog posts. Since we're open source now that's a bit of a weird policy but I think we can stick with it for now.

@dotnet-bot test Windows_NT Build please

LGTM

@dotnet-bot test this please

@dotnet-bot test Windows_NT Build please


Styles and position won't work in 2.6.0

Describe the bug
Notifications apper unstyled and don't respect position prop when using version 2.6.0. It's all good when I use 2.5.0.
To Reproduce
Steps to reproduce the behavior:

Create a new Vue 3 project using npm init vue@latest (choose TypeScript support).
Install the latest version of the library (2.6.0)
Use Notifications plugin and add notifications component to the app. Set position prop to "top center", for example.
Call notify to see the notification appears in the top right corner unstyled (see screenshot).

Expected behavior
A styled notification to appear in the top center position.
Screenshots
This is 2.6.0:

And this is 2.5.0:

Additional context
I didn't check any another props, maybe they're not working too.

Hi, sorry for that. Fixed in 2.6.1


feat(cli): generate changelog

What
Implements a changelog CLI that has the features that we expect it to have out-of-the-box.
Why
So we can get off that old version of conventional-changelog and remove all the unnecessary scripts we have in multiple repos.
Notes

I think this is fully complete. You can try it out by cloning it, checking out this branch, and then running ../generate-changelog/bin/generate -h in another repo. It makes a changelog that is very similar to ours but now has several new types:

chore
docs
feat
fix
other
refactor
style
test

At first, I had it printing to stdout so that we can just >> into the CHANGELOG.md file, but I forgot that we (and most people) want it to prepend to the file instead of append, and since there's no easy way to prepend, I decided to build that functionality in. I left these as separate commits though mostly so that when v1.0.0 is cut, the changelog generated for this repo will be more than one commit.
I setup travis and coveralls from the get-go so this package started off fresh with 100% coverage.

Happy to hear any comments about style or functionality! I'm also down to do a peer code review as well.
@elnaz

This is awesome! What would be an example of a chore? I can't think of one that wouldn't be covered by docs, refactor, style, or test.

@elnaz hmm I can't really think of anything off the top of my head, but these were the categories in git-changelog so I thought it'd be a good starting point. It's there if we need it, but if we don't, then we won't use it.

Wanna update this for Node v5.1.1, too?

I might consider one-off data migrations (like backdating) or converting from controller -> plugin to be a chore... :)

@brianseitel I thought about it, but since it's open source, I'd rather it compatible with multiple versions of node. That's why I'm testing it again 0.10, 0.12, 4, and 5 in travis.

:+1:

LGTM

sounds good. LGTM


Create guidance sections on 'the makeup of...'

Need guidance on the composition of a questionnaire, internal service etc.
To do:
Include guidance on making labels and legends headings in eQ guidance

This is more of a nice but might be worth looking into after we're done with higher priority sections (community, accessibility, pattern and component docs)
Would encompass user centered survey design guidance: https://analysisfunction.civilservice.gov.uk/policy-store/a-user-centred-design-approach-to-surveys/
As well as creating a process re' patterns such as 'survey introduction page', 'survey section introduction page' etc.


Add backgroundColor to change background fill color

from @emmasuzuki https://github.com/jaketrent/react-social-icons/issues/18#issuecomment-448686185

Definitely a pretty cool change if it makes it in.

This is now supported in 4.x using fgColor.  color also renamed to bgColor.  See site for usage details: https://jaketrent.github.io/react-social-icons/
Thanks for contributing to this!