mount: data inconsistencies between multiple hosts

Describe the bug
Mount SeaweedFS on two hosts. Modify the contents of the file on the first host. Data cannot be synchronized to the second host.
But by changing the contents of the file on the second host, the data can be synchronized to the first host.
# 1. init seaweedfs
<EMAIL_ADDRESS>weed server -filer=true
<EMAIL_ADDRESS>weed mount -filer=<IP_ADDRESS>:8888 -cacheDir=./cache/t1 -dir=./t1 -filer.path=/
<EMAIL_ADDRESS>weed mount -filer=<IP_ADDRESS>:8888 -cacheDir=./cache/t2 -dir=./t2 -filer.path=/

# 2. init test data
<EMAIL_ADDRESS>echo "1" > ./t1/1
<EMAIL_ADDRESS>cat ./t1/1
1
<EMAIL_ADDRESS>cat ./t2/1
1

# 3. host2 => host3
<EMAIL_ADDRESS>echo "2" >> ./t1/1
<EMAIL_ADDRESS>cat ./t1/1
1
2
<EMAIL_ADDRESS>cat ./t2/1
1

# 4. host3 => host2
<EMAIL_ADDRESS>echo "3" >> ./t2/1
<EMAIL_ADDRESS>cat ./t2/1
1
3
<EMAIL_ADDRESS>cat ./t1/1
1
3

System Setup

<EMAIL_ADDRESS>weed server -filer=true
<EMAIL_ADDRESS>weed mount -filer=<IP_ADDRESS>:8888 -cacheDir=./cache/t1 -dir=./t1 -filer.path=/
<EMAIL_ADDRESS>weed mount -filer=<IP_ADDRESS>:8888 -cacheDir=./cache/t2 -dir=./t2 -filer.path=/
linux amd64
version 30GB 2.11  linux amd64

Expected behavior
The data on the two hosts are consistent
Additional context
When the three commands are run on the same host, the data is consistent
<EMAIL_ADDRESS>weed server -filer=true
<EMAIL_ADDRESS>weed mount -filer=<IP_ADDRESS>:8888 -cacheDir=./cache/t1 -dir=./t1 -filer.path=/
<EMAIL_ADDRESS>weed mount -filer=<IP_ADDRESS>:8888 -cacheDir=./cache/t2 -dir=./t2 -filer.path=/

docker-compose -f seaweedfs-compose.yml -p seaweedfs up

seaweedfs-compose.yml is:
version: '2'

services:
  master:
    image: chrislusf/seaweedfs:2.21 # use a remote image
    ports:
      - 9333:9333
      - 19333:19333
    command: "master -ip=master"
  volume:
    image: chrislusf/seaweedfs:2.21 # use a remote image
    ports:
      - 8080:8080
      - 18080:18080
      - 9325:9325
    command: 'volume -mserver="master:9333" -port=8080  -metricsPort=9325'
    depends_on:
      - master
  filer:
    image: chrislusf/seaweedfs:2.21 # use a remote image
    ports:
      - 8888:8888
      - 18888:18888
      - 9326:9326
    command: 'filer -master="master:9333"  -metricsPort=9326'
    tty: true
    stdin_open: true
    depends_on:
      - master
      - volume
  s3:
    image: chrislusf/seaweedfs:2.21 # use a remote image
    ports:
      - 8333:8333
      - 9327:9327
    command: 's3 -filer="filer:8888" -metricsPort=9327'
    depends_on:
      - master
      - volume
      - filer

docker-compose -f seaweedfs-compose.yml -p seaweedfs up

seaweedfs-compose.yml is:
version: '2'

services:
  master:
    image: chrislusf/seaweedfs:2.21 # use a remote image
    ports:
      - 9333:9333
      - 19333:19333
    command: "master -ip=master"
  volume:
    image: chrislusf/seaweedfs:2.21 # use a remote image
    ports:
      - 8080:8080
      - 18080:18080
      - 9325:9325
    command: 'volume -mserver="master:9333" -port=8080  -metricsPort=9325'
    depends_on:
      - master
  filer:
    image: chrislusf/seaweedfs:2.21 # use a remote image
    ports:
      - 8888:8888
      - 18888:18888
      - 9326:9326
    command: 'filer -master="master:9333"  -metricsPort=9326'
    tty: true
    stdin_open: true
    depends_on:
      - master
      - volume
  s3:
    image: chrislusf/seaweedfs:2.21 # use a remote image
    ports:
      - 8333:8333
      - 9327:9327
    command: 's3 -filer="filer:8888" -metricsPort=9327'
    depends_on:
      - master
      - volume
      - filer

seaweedfs-compose.yml is:
version: '2'

services:
  master:
    image: chrislusf/seaweedfs:2.21 # use a remote image
    ports:
      - 9333:9333
      - 19333:19333
    command: "master -ip=master"
  volume:
    image: chrislusf/seaweedfs:2.21 # use a remote image
    ports:
      - 8080:8080
      - 18080:18080
      - 9325:9325
    command: 'volume -mserver="master:9333" -port=8080  -metricsPort=9325'
    depends_on:
      - master
  filer:
    image: chrislusf/seaweedfs:2.21 # use a remote image
    ports:
      - 8888:8888
      - 18888:18888
      - 9326:9326
    command: 'filer -master="master:9333"  -metricsPort=9326'
    tty: true
    stdin_open: true
    depends_on:
      - master
      - volume
  mount_1:
    image: chrislusf/seaweedfs:2.21
    privileged: true
    entrypoint: '/bin/sh -c "mkdir -p t1 && mkdir -p cache/t1 && weed mount -filer=filer:8888 -cacheDir=./cache/t1 -dir=./t1 -filer.path=/"'
    depends_on:
      - master
      - volume
      - filer
  mount_2:
    image: chrislusf/seaweedfs:2.21
    privileged: true
    entrypoint: '/bin/sh -c "mkdir -p t2 && mkdir -p cache/t2 && weed mount -filer=filer:8888 -cacheDir=./cache/t2 -dir=./t2 -filer.path=/"'
    depends_on:
      - master
      - volume
      - filer
      - mount_1

test.sh is:
#!/bin/bash

echo "-------------------"
echo "# 1. init seaweedfs"
echo "-------------------"
docker-compose -f seaweedfs-compose.yml -p seaweedfs up -d
sleep 1s

echo ""
echo ""
echo "-------------------"
echo "# 2. init test data"
echo "-------------------"
echo "     echo 1 >> t1/1"
echo "-------------------"
docker exec -ti $(docker ps  | grep mount_1 | awk '{print $1}') sh -c "echo 1 > t1/1"
echo "# cat t1/1"
docker exec -ti $(docker ps  | grep mount_1 | awk '{print $1}') cat t1/1
echo "# cat t2/1"
docker exec -ti $(docker ps  | grep mount_2 | awk '{print $1}') cat t2/1
sleep 1s

echo ""
echo ""

echo "-----------------------"
echo "# 3. mount_1 => mount_2"
echo "-----------------------"
echo "         echo 2 >> t1/1"
echo "-----------------------"
docker exec -ti $(docker ps  | grep mount_1 | awk '{print $1}') sh -c "echo 2 >>  t1/1"
echo "# cat t1/1"
docker exec -ti $(docker ps  | grep mount_1 | awk '{print $1}') cat t1/1
echo "# cat t2/1"
docker exec -ti $(docker ps  | grep mount_2 | awk '{print $1}') cat t2/1
sleep 1s

echo ""
echo ""

echo "-----------------------"
echo "# 3. mount_2 => mount_1"
echo "-----------------------"
echo "        echo 3 >>  t2/1"
echo "-----------------------"
docker exec -ti $(docker ps  | grep mount_2 | awk '{print $1}') sh -c "echo 3 >>  t2/1"
echo "# cat t1/1"
docker exec -ti $(docker ps  | grep mount_1 | awk '{print $1}') cat t1/1
echo "# cat t2/1"
docker exec -ti $(docker ps  | grep mount_2 | awk '{print $1}') cat t2/1
sleep 1s

echo ""
echo ""

echo "-----------------------"
echo "# 5. stop seaweedfs"
echo "-----------------------"
docker-compose -f seaweedfs-compose.yml -p seaweedfs down

output:
-------------------
# 1. init seaweedfs
-------------------
Creating network "seaweedfs_default" with the default driver
Creating seaweedfs_master_1 ... done
Creating seaweedfs_volume_1 ... done
Creating seaweedfs_filer_1  ... done
Creating seaweedfs_mount_1_1 ... done
Creating seaweedfs_mount_2_1 ... done

-------------------
# 2. init test data
-------------------
     echo 1 >> t1/1
-------------------
# cat t1/1
1
# cat t2/1
cat: can't open 't2/1': No such file or directory

-----------------------
# 3. mount_1 => mount_2
-----------------------
         echo 2 >> t1/1
-----------------------
# cat t1/1
1
2
# cat t2/1
1
2

-----------------------
# 3. mount_2 => mount_1
-----------------------
        echo 3 >>  t2/1
-----------------------
# cat t1/1
1
2
# cat t2/1
1
2
3

-----------------------
# 5. stop seaweedfs
-----------------------
Stopping seaweedfs_mount_2_1 ... done
Stopping seaweedfs_mount_1_1 ... done
Stopping seaweedfs_filer_1   ... done
Stopping seaweedfs_volume_1  ... done
Stopping seaweedfs_master_1  ... done
Removing seaweedfs_mount_2_1 ... done
Removing seaweedfs_mount_1_1 ... done
Removing seaweedfs_filer_1   ... done
Removing seaweedfs_volume_1  ... done
Removing seaweedfs_master_1  ... done
Removing network seaweedfs_default

seaweedfs-compose.yml is:
version: '2'

services:
  master:
    image: chrislusf/seaweedfs:2.21 # use a remote image
    ports:
      - 9333:9333
      - 19333:19333
    command: "master -ip=master"
  volume:
    image: chrislusf/seaweedfs:2.21 # use a remote image
    ports:
      - 8080:8080
      - 18080:18080
      - 9325:9325
    command: 'volume -mserver="master:9333" -port=8080  -metricsPort=9325'
    depends_on:
      - master
  filer:
    image: chrislusf/seaweedfs:2.21 # use a remote image
    ports:
      - 8888:8888
      - 18888:18888
      - 9326:9326
    command: 'filer -master="master:9333"  -metricsPort=9326'
    tty: true
    stdin_open: true
    depends_on:
      - master
      - volume
  mount_1:
    image: chrislusf/seaweedfs:2.21
    privileged: true
    entrypoint: '/bin/sh -c "mkdir -p t1 && mkdir -p cache/t1 && weed mount -filer=filer:8888 -cacheDir=./cache/t1 -dir=./t1 -filer.path=/"'
    depends_on:
      - master
      - volume
      - filer
  mount_2:
    image: chrislusf/seaweedfs:2.21
    privileged: true
    entrypoint: '/bin/sh -c "mkdir -p t2 && mkdir -p cache/t2 && weed mount -filer=filer:8888 -cacheDir=./cache/t2 -dir=./t2 -filer.path=/"'
    depends_on:
      - master
      - volume
      - filer
      - mount_1

test.sh is:
#!/bin/bash

echo "-------------------"
echo "# 1. init seaweedfs"
echo "-------------------"
docker-compose -f seaweedfs-compose.yml -p seaweedfs up -d
sleep 1s

echo ""
echo ""
echo "-------------------"
echo "# 2. init test data"
echo "-------------------"
echo "     echo 1 >> t1/1"
echo "-------------------"
docker exec -ti $(docker ps  | grep mount_1 | awk '{print $1}') sh -c "echo 1 > t1/1"
echo "# cat t1/1"
docker exec -ti $(docker ps  | grep mount_1 | awk '{print $1}') cat t1/1
echo "# cat t2/1"
docker exec -ti $(docker ps  | grep mount_2 | awk '{print $1}') cat t2/1
sleep 1s

echo ""
echo ""

echo "-----------------------"
echo "# 3. mount_1 => mount_2"
echo "-----------------------"
echo "         echo 2 >> t1/1"
echo "-----------------------"
docker exec -ti $(docker ps  | grep mount_1 | awk '{print $1}') sh -c "echo 2 >>  t1/1"
echo "# cat t1/1"
docker exec -ti $(docker ps  | grep mount_1 | awk '{print $1}') cat t1/1
echo "# cat t2/1"
docker exec -ti $(docker ps  | grep mount_2 | awk '{print $1}') cat t2/1
sleep 1s

echo ""
echo ""

echo "-----------------------"
echo "# 3. mount_2 => mount_1"
echo "-----------------------"
echo "        echo 3 >>  t2/1"
echo "-----------------------"
docker exec -ti $(docker ps  | grep mount_2 | awk '{print $1}') sh -c "echo 3 >>  t2/1"
echo "# cat t1/1"
docker exec -ti $(docker ps  | grep mount_1 | awk '{print $1}') cat t1/1
echo "# cat t2/1"
docker exec -ti $(docker ps  | grep mount_2 | awk '{print $1}') cat t2/1
sleep 1s

echo ""
echo ""

echo "-----------------------"
echo "# 5. stop seaweedfs"
echo "-----------------------"
docker-compose -f seaweedfs-compose.yml -p seaweedfs down

output:
-------------------
# 1. init seaweedfs
-------------------
Creating network "seaweedfs_default" with the default driver
Creating seaweedfs_master_1 ... done
Creating seaweedfs_volume_1 ... done
Creating seaweedfs_filer_1  ... done
Creating seaweedfs_mount_1_1 ... done
Creating seaweedfs_mount_2_1 ... done

-------------------
# 2. init test data
-------------------
     echo 1 >> t1/1
-------------------
# cat t1/1
1
# cat t2/1
cat: can't open 't2/1': No such file or directory

-----------------------
# 3. mount_1 => mount_2
-----------------------
         echo 2 >> t1/1
-----------------------
# cat t1/1
1
2
# cat t2/1
1
2

-----------------------
# 3. mount_2 => mount_1
-----------------------
        echo 3 >>  t2/1
-----------------------
# cat t1/1
1
2
# cat t2/1
1
2
3

-----------------------
# 5. stop seaweedfs
-----------------------
Stopping seaweedfs_mount_2_1 ... done
Stopping seaweedfs_mount_1_1 ... done
Stopping seaweedfs_filer_1   ... done
Stopping seaweedfs_volume_1  ... done
Stopping seaweedfs_master_1  ... done
Removing seaweedfs_mount_2_1 ... done
Removing seaweedfs_mount_1_1 ... done
Removing seaweedfs_filer_1   ... done
Removing seaweedfs_volume_1  ... done
Removing seaweedfs_master_1  ... done
Removing network seaweedfs_default

It was caused by linux page cache. Now it is fixed.

It was caused by linux page cache. Now it is fixed.

@nivekuil Is this reproducible on freebsd? Not too sure since this is a linux only change. It will be nice to have a FreeBSD docker compose file that I can test with later.

@nivekuil Is this reproducible on freebsd? Not too sure since this is a linux only change. It will be nice to have a FreeBSD docker compose file that I can test with later.

@nivekuil Is this reproducible on freebsd? Not too sure since this is a linux only change. It will be nice to have a FreeBSD docker compose file that I can test with later.

I don't know anything about freebsd but having written a similar fuse driver before I agree that using O_DIRECT and handling any caching in userspace is a good idea. libfuse should just do the right thing on all platforms if you tell it to do direct io.

@nivekuil Is this reproducible on freebsd? Not too sure since this is a linux only change. It will be nice to have a FreeBSD docker compose file that I can test with later.

I don't know anything about freebsd but having written a similar fuse driver before I agree that using O_DIRECT and handling any caching in userspace is a good idea. libfuse should just do the right thing on all platforms if you tell it to do direct io.

@nivekuil Sorry. I mistaken you with another contributor who worked on FreeBSD. :)  But thanks for your answer!

@nivekuil Sorry. I mistaken you with another contributor who worked on FreeBSD. :)  But thanks for your answer!

@chrislusf Thanks for your quick fix, But another problem I found using the latest version 2.22 validation is that the value in mount_1 overrides the value in mount_2
$ ls -l
-rw-r--r-- 1 root root 1245 Jan 22 19:16 seaweedfs-compose.yml
-rwxr-xr-x 1 root root 1660 Jan 22 20:00 test.sh*

$ ./test.sh

seaweedfs-compose.yml:
version: '2'

services:
  master:
    image: chrislusf/seaweedfs:2.22 # use a remote image
    ports:
      - 9333:9333
      - 19333:19333
    command: "master -ip=master"
  volume:
    image: chrislusf/seaweedfs:2.22 # use a remote image
    ports:
      - 8080:8080
      - 18080:18080
      - 9325:9325
    command: 'volume -mserver="master:9333" -port=8080  -metricsPort=9325'
    depends_on:
      - master
  filer:
    image: chrislusf/seaweedfs:2.22 # use a remote image
    ports:
      - 8888:8888
      - 18888:18888
      - 9326:9326
    command: 'filer -master="master:9333"  -metricsPort=9326'
    tty: true
    stdin_open: true
    depends_on:
      - master
      - volume
  mount_1:
    image: chrislusf/seaweedfs:2.22
    privileged: true
    entrypoint: '/bin/sh -c "mkdir -p t1 && mkdir -p cache/t1 && weed mount -filer=filer:8888 -cacheDir=./cache/t1 -dir=./t1 -filer.path=/"'
    depends_on:
      - master
      - volume
      - filer
  mount_2:
    image: chrislusf/seaweedfs:2.22
    privileged: true
    entrypoint: '/bin/sh -c "mkdir -p t2 && mkdir -p cache/t2 && weed mount -filer=filer:8888 -cacheDir=./cache/t2 -dir=./t2 -filer.path=/"'
    depends_on:
      - master
      - volume
      - filer
      - mount_1

test.sh:
#!/bin/bash

echo "-------------------"
echo "# 0. init seaweedfs"
echo "-------------------"
docker-compose -f seaweedfs-compose.yml -p seaweedfs up -d
sleep 5s

echo ""
echo ""
echo "-------------------"
echo "# 1. mount_1 => mount_2"
echo "-------------------"
echo "     echo 1 > t1/1"
echo "------------i-------"
echo -e "expect value: \n1"
docker exec -ti $(docker ps  | grep mount_1 | awk '{print $1}') sh -c "echo 1 > t1/1"
echo "# cat t1/1"
docker exec -ti $(docker ps  | grep mount_1 | awk '{print $1}') cat t1/1
echo "# cat t2/1"
docker exec -ti $(docker ps  | grep mount_2 | awk '{print $1}') cat t2/1
sleep 1s

echo ""
echo ""

echo "-----------------------"
echo "# 2. mount_2 => mount_1"
echo "-----------------------"
echo "        echo 2 >>  t2/1"
echo "-----------------------"
echo -e "expect value: \n1\n2"
docker exec -ti $(docker ps  | grep mount_2 | awk '{print $1}') sh -c "echo 2 >>  t2/1"
echo "# cat t1/1"
docker exec -ti $(docker ps  | grep mount_1 | awk '{print $1}') cat t1/1
echo "# cat t2/1"
docker exec -ti $(docker ps  | grep mount_2 | awk '{print $1}') cat t2/1
sleep 1s

echo ""
echo ""

echo "-----------------------"
echo "# 3. mount_1 => mount_2"
echo "-----------------------"
echo "         echo 3 >> t1/1"
echo "-----------------------"
echo -e "expect value: \n1\n2\n3"
docker exec -ti $(docker ps  | grep mount_1 | awk '{print $1}') sh -c "echo 3 >>  t1/1"
echo "# cat t1/1"
docker exec -ti $(docker ps  | grep mount_1 | awk '{print $1}') cat t1/1
echo "# cat t2/1"
docker exec -ti $(docker ps  | grep mount_2 | awk '{print $1}') cat t2/1
sleep 1s

echo ""
echo ""

echo "-----------------------"
echo "# 4. mount_2 => mount_1"
echo "-----------------------"
echo "         echo 4 >> t2/1"
echo "-----------------------"
echo -e "expect value: \n1\n2\n3\n4"
docker exec -ti $(docker ps  | grep mount_2 | awk '{print $1}') sh -c "echo 4 >>  t2/1"
echo "# cat t1/1"
docker exec -ti $(docker ps  | grep mount_1 | awk '{print $1}') cat t1/1
echo "# cat t2/1"
docker exec -ti $(docker ps  | grep mount_2 | awk '{print $1}') cat t2/1
sleep 1s

echo ""
echo ""

echo "-----------------------"
echo "# 5. mount_1 => mount_2"
echo "-----------------------"
echo "         echo 5 >> t1/1"
echo "-----------------------"
echo -e "expect value: \n1\n2\n3\n4\n5"
docker exec -ti $(docker ps  | grep mount_1 | awk '{print $1}') sh -c "echo 5 >>  t1/1"
echo "# cat t1/1"
docker exec -ti $(docker ps  | grep mount_1 | awk '{print $1}') cat t1/1
echo "# cat t2/1"
docker exec -ti $(docker ps  | grep mount_2 | awk '{print $1}') cat t2/1
sleep 1s

echo ""
echo ""

echo "-----------------------"
echo "# 5. stop seaweedfs"
echo "-----------------------"
docker-compose -f seaweedfs-compose.yml -p seaweedfs down

output:
-------------------
# 0. init seaweedfs
-------------------
Creating network "seaweedfs_default" with the default driver
Creating seaweedfs_master_1 ... done
Creating seaweedfs_volume_1 ... done
Creating seaweedfs_filer_1  ... done
Creating seaweedfs_mount_1_1 ... done
Creating seaweedfs_mount_2_1 ... done

-------------------
# 1. mount_1 => mount_2
-------------------
     echo 1 > t1/1
------------i-------
expect value:
1
# cat t1/1
1
# cat t2/1
1

-----------------------
# 2. mount_2 => mount_1
-----------------------
        echo 2 >>  t2/1
-----------------------
expect value:
1
2
# cat t1/1
1
2
# cat t2/1
1
2

-----------------------
# 3. mount_1 => mount_2
-----------------------
         echo 3 >> t1/1
-----------------------
expect value:
1
2
3
# cat t1/1
1
3
# cat t2/1
1
3

-----------------------
# 4. mount_2 => mount_1
-----------------------
         echo 4 >> t2/1
-----------------------
expect value:
1
2
3
4
# cat t1/1
1
3
4
# cat t2/1
1
3
4

-----------------------
# 5. mount_1 => mount_2
-----------------------
         echo 5 >> t1/1
-----------------------
expect value:
1
2
3
4
5
# cat t1/1
1
3
5
# cat t2/1
1
3
5

-----------------------
# 5. stop seaweedfs
-----------------------
Stopping seaweedfs_mount_2_1 ... done
Stopping seaweedfs_mount_1_1 ... done
Stopping seaweedfs_filer_1   ... done
Stopping seaweedfs_volume_1  ... done
Stopping seaweedfs_master_1  ... done
Removing seaweedfs_mount_2_1 ... done
Removing seaweedfs_mount_1_1 ... done
Removing seaweedfs_filer_1   ... done
Removing seaweedfs_volume_1  ... done
Removing seaweedfs_master_1  ... done
Removing network seaweedfs_default

@chrislusf Thanks for your quick fix, But another problem I found using the latest version 2.22 validation is that the value in mount_1 overrides the value in mount_2
$ ls -l
-rw-r--r-- 1 root root 1245 Jan 22 19:16 seaweedfs-compose.yml
-rwxr-xr-x 1 root root 1660 Jan 22 20:00 test.sh*

$ ./test.sh

seaweedfs-compose.yml:
version: '2'

services:
  master:
    image: chrislusf/seaweedfs:2.22 # use a remote image
    ports:
      - 9333:9333
      - 19333:19333
    command: "master -ip=master"
  volume:
    image: chrislusf/seaweedfs:2.22 # use a remote image
    ports:
      - 8080:8080
      - 18080:18080
      - 9325:9325
    command: 'volume -mserver="master:9333" -port=8080  -metricsPort=9325'
    depends_on:
      - master
  filer:
    image: chrislusf/seaweedfs:2.22 # use a remote image
    ports:
      - 8888:8888
      - 18888:18888
      - 9326:9326
    command: 'filer -master="master:9333"  -metricsPort=9326'
    tty: true
    stdin_open: true
    depends_on:
      - master
      - volume
  mount_1:
    image: chrislusf/seaweedfs:2.22
    privileged: true
    entrypoint: '/bin/sh -c "mkdir -p t1 && mkdir -p cache/t1 && weed mount -filer=filer:8888 -cacheDir=./cache/t1 -dir=./t1 -filer.path=/"'
    depends_on:
      - master
      - volume
      - filer
  mount_2:
    image: chrislusf/seaweedfs:2.22
    privileged: true
    entrypoint: '/bin/sh -c "mkdir -p t2 && mkdir -p cache/t2 && weed mount -filer=filer:8888 -cacheDir=./cache/t2 -dir=./t2 -filer.path=/"'
    depends_on:
      - master
      - volume
      - filer
      - mount_1

test.sh:
#!/bin/bash

echo "-------------------"
echo "# 0. init seaweedfs"
echo "-------------------"
docker-compose -f seaweedfs-compose.yml -p seaweedfs up -d
sleep 5s

echo ""
echo ""
echo "-------------------"
echo "# 1. mount_1 => mount_2"
echo "-------------------"
echo "     echo 1 > t1/1"
echo "------------i-------"
echo -e "expect value: \n1"
docker exec -ti $(docker ps  | grep mount_1 | awk '{print $1}') sh -c "echo 1 > t1/1"
echo "# cat t1/1"
docker exec -ti $(docker ps  | grep mount_1 | awk '{print $1}') cat t1/1
echo "# cat t2/1"
docker exec -ti $(docker ps  | grep mount_2 | awk '{print $1}') cat t2/1
sleep 1s

echo ""
echo ""

echo "-----------------------"
echo "# 2. mount_2 => mount_1"
echo "-----------------------"
echo "        echo 2 >>  t2/1"
echo "-----------------------"
echo -e "expect value: \n1\n2"
docker exec -ti $(docker ps  | grep mount_2 | awk '{print $1}') sh -c "echo 2 >>  t2/1"
echo "# cat t1/1"
docker exec -ti $(docker ps  | grep mount_1 | awk '{print $1}') cat t1/1
echo "# cat t2/1"
docker exec -ti $(docker ps  | grep mount_2 | awk '{print $1}') cat t2/1
sleep 1s

echo ""
echo ""

echo "-----------------------"
echo "# 3. mount_1 => mount_2"
echo "-----------------------"
echo "         echo 3 >> t1/1"
echo "-----------------------"
echo -e "expect value: \n1\n2\n3"
docker exec -ti $(docker ps  | grep mount_1 | awk '{print $1}') sh -c "echo 3 >>  t1/1"
echo "# cat t1/1"
docker exec -ti $(docker ps  | grep mount_1 | awk '{print $1}') cat t1/1
echo "# cat t2/1"
docker exec -ti $(docker ps  | grep mount_2 | awk '{print $1}') cat t2/1
sleep 1s

echo ""
echo ""

echo "-----------------------"
echo "# 4. mount_2 => mount_1"
echo "-----------------------"
echo "         echo 4 >> t2/1"
echo "-----------------------"
echo -e "expect value: \n1\n2\n3\n4"
docker exec -ti $(docker ps  | grep mount_2 | awk '{print $1}') sh -c "echo 4 >>  t2/1"
echo "# cat t1/1"
docker exec -ti $(docker ps  | grep mount_1 | awk '{print $1}') cat t1/1
echo "# cat t2/1"
docker exec -ti $(docker ps  | grep mount_2 | awk '{print $1}') cat t2/1
sleep 1s

echo ""
echo ""

echo "-----------------------"
echo "# 5. mount_1 => mount_2"
echo "-----------------------"
echo "         echo 5 >> t1/1"
echo "-----------------------"
echo -e "expect value: \n1\n2\n3\n4\n5"
docker exec -ti $(docker ps  | grep mount_1 | awk '{print $1}') sh -c "echo 5 >>  t1/1"
echo "# cat t1/1"
docker exec -ti $(docker ps  | grep mount_1 | awk '{print $1}') cat t1/1
echo "# cat t2/1"
docker exec -ti $(docker ps  | grep mount_2 | awk '{print $1}') cat t2/1
sleep 1s

echo ""
echo ""

echo "-----------------------"
echo "# 5. stop seaweedfs"
echo "-----------------------"
docker-compose -f seaweedfs-compose.yml -p seaweedfs down

output:
-------------------
# 0. init seaweedfs
-------------------
Creating network "seaweedfs_default" with the default driver
Creating seaweedfs_master_1 ... done
Creating seaweedfs_volume_1 ... done
Creating seaweedfs_filer_1  ... done
Creating seaweedfs_mount_1_1 ... done
Creating seaweedfs_mount_2_1 ... done

-------------------
# 1. mount_1 => mount_2
-------------------
     echo 1 > t1/1
------------i-------
expect value:
1
# cat t1/1
1
# cat t2/1
1

-----------------------
# 2. mount_2 => mount_1
-----------------------
        echo 2 >>  t2/1
-----------------------
expect value:
1
2
# cat t1/1
1
2
# cat t2/1
1
2

-----------------------
# 3. mount_1 => mount_2
-----------------------
         echo 3 >> t1/1
-----------------------
expect value:
1
2
3
# cat t1/1
1
3
# cat t2/1
1
3

-----------------------
# 4. mount_2 => mount_1
-----------------------
         echo 4 >> t2/1
-----------------------
expect value:
1
2
3
4
# cat t1/1
1
3
4
# cat t2/1
1
3
4

-----------------------
# 5. mount_1 => mount_2
-----------------------
         echo 5 >> t1/1
-----------------------
expect value:
1
2
3
4
5
# cat t1/1
1
3
5
# cat t2/1
1
3
5

-----------------------
# 5. stop seaweedfs
-----------------------
Stopping seaweedfs_mount_2_1 ... done
Stopping seaweedfs_mount_1_1 ... done
Stopping seaweedfs_filer_1   ... done
Stopping seaweedfs_volume_1  ... done
Stopping seaweedfs_master_1  ... done
Removing seaweedfs_mount_2_1 ... done
Removing seaweedfs_mount_1_1 ... done
Removing seaweedfs_filer_1   ... done
Removing seaweedfs_volume_1  ... done
Removing seaweedfs_master_1  ... done
Removing network seaweedfs_default

Turns out direct_io was not the right fix. Need to invalidate directory entry in kernel node cache.

Turns out direct_io was not the right fix. Need to invalidate directory entry in kernel node cache.

@chrislusf Thanks for your quick fix, but there are still problems when adding the parameters -collection and -filer.path. The problem was discovered during validation in seaweedfs-csi-driver.
$ ls -l
-rw-r--r-- 1 root root 1287 Jan 27 17:47 seaweedfs-compose.yml
-rwxr-xr-x 1 root root 2738 Jan 27 17:50 test.sh*

$ ./test.sh

seaweedfs-compose.yml:
p.s remember to change the image.
version: '2'

services:
  master:
    image: chrislusf/seaweedfs:latest
    ports:
      - 9333:9333
      - 19333:19333
    command: "master -ip=master"
  volume:
    image: chrislusf/seaweedfs:latest
    ports:
      - 8080:8080
      - 18080:18080
      - 9325:9325
    command: 'volume -mserver="master:9333" -port=8080  -metricsPort=9325'
    depends_on:
      - master
  filer:
    image: chrislusf/seaweedfs:latest
    ports:
      - 8888:8888
      - 18888:18888
      - 9326:9326
    command: 'filer -master="master:9333"  -metricsPort=9326'
    tty: true
    stdin_open: true
    depends_on:
      - master
      - volume
  mount_1:
    image: chrislusf/seaweedfs:latest
    privileged: true
    entrypoint: '/bin/sh -c "mkdir -p t1 && mkdir -p cache/t1 && weed mount -filer=filer:8888 -dir=./t1 -collection=c1 -filer.path=/buckets/c1"'
    depends_on:
      - master
      - volume
      - filer
  mount_2:
    image: chrislusf/seaweedfs:latest
    privileged: true
    entrypoint: '/bin/sh -c "mkdir -p t2 && mkdir -p cache/t2 && weed mount -filer=filer:8888 -dir=./t2 -collection=c1 -filer.path=/buckets/c1"'
    depends_on:
      - master
      - volume
      - filer
      - mount_1

test.sh: same as above
output: same as above

@chrislusf Thanks for your quick fix, but there are still problems when adding the parameters -collection and -filer.path. The problem was discovered during validation in seaweedfs-csi-driver.
$ ls -l
-rw-r--r-- 1 root root 1287 Jan 27 17:47 seaweedfs-compose.yml
-rwxr-xr-x 1 root root 2738 Jan 27 17:50 test.sh*

$ ./test.sh

seaweedfs-compose.yml:
p.s remember to change the image.
version: '2'

services:
  master:
    image: chrislusf/seaweedfs:latest
    ports:
      - 9333:9333
      - 19333:19333
    command: "master -ip=master"
  volume:
    image: chrislusf/seaweedfs:latest
    ports:
      - 8080:8080
      - 18080:18080
      - 9325:9325
    command: 'volume -mserver="master:9333" -port=8080  -metricsPort=9325'
    depends_on:
      - master
  filer:
    image: chrislusf/seaweedfs:latest
    ports:
      - 8888:8888
      - 18888:18888
      - 9326:9326
    command: 'filer -master="master:9333"  -metricsPort=9326'
    tty: true
    stdin_open: true
    depends_on:
      - master
      - volume
  mount_1:
    image: chrislusf/seaweedfs:latest
    privileged: true
    entrypoint: '/bin/sh -c "mkdir -p t1 && mkdir -p cache/t1 && weed mount -filer=filer:8888 -dir=./t1 -collection=c1 -filer.path=/buckets/c1"'
    depends_on:
      - master
      - volume
      - filer
  mount_2:
    image: chrislusf/seaweedfs:latest
    privileged: true
    entrypoint: '/bin/sh -c "mkdir -p t2 && mkdir -p cache/t2 && weed mount -filer=filer:8888 -dir=./t2 -collection=c1 -filer.path=/buckets/c1"'
    depends_on:
      - master
      - volume
      - filer
      - mount_1

test.sh: same as above
output: same as above

After verification, the problem has been fixed.


WIP: probe-rs-server

This PR adds an initial implementation of a probe-rs server.
The server is based on a JSON RPC API that is served via a websocket.
It provides a large part of the functionality Core and Session usually expose via the network to potentially several clients, allowing us to do GDB, RTT, ITM and what not at the same time for the same chip.
Closes #258

The websocket server does not have allowed_origins configured, so it'll allow connections from any origin.
This means any website you visit in a browser in the same machine can connect to ws://localhost:1234 (or whatever port) and mess with your probe.

@FrozenDroid GDB and RTT are possible to be used together on master. There is some kinks to it tho. You need to disable logging for this to work and even then there is spurious println!().
I have cargo embed v2 (will be called differently) in the makings, which basically acts as a daemon for all services and can be connected to via WebSocket/TCP/Stdout.
@hargoniX It was not followed further because there were questions how well this would work in practice when multiple participants would interfere with each other.
My current approach involves exposing RTT, ITM, GDB, Microsoft DAP via a structured API (JSON) and a non structured API (plain text) which can be relayed to WS, TCP or Stdout/In configurably. This will enable folks to use the UI client I am building for tracing etc. Or use a normal socket CLI like socat to use RTT. You wont have to use the clumsy TUI and you also wont have any clashes with different tools.


use testFile in some tests

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1600/
Test PASSed.

I can't speak for @heuermh but I never use Intellij either and do all my editing in emacs. I tried switching to Intellij several years back but it's emacs compat mode is pretty rough. I code in a variety of other languages, so cutting over and switching to Intellij doesn't make sense as an investment for me.

Will merge on tests rerunning and passing.

Emacs and whatever scala mode comes in via melpa for me.  I try to look at all my prs in IntelliJ for the static code checking at least once, but the way it handles git makes me nervous, so I don't use it full time.  And I prefer running in terminals.  :)

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1603/
Test PASSed.

Jenkins, retest this please.

why retest OOC?

@ryan-williams I just merged #1135 which was +~2500 -~600 so I am somewhat distrustful of old Jenkins results is all.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1607/
Test PASSed.

Merged! Thanks @ryan-williams!