mmdetection example does not work

I am very interested in using Label Studio and its Machine Learning backend, particularly for Object Detection with Bounding Boxes. However, I cannot get the example to work, and as such, I am stuck.
I have looked at the following documentation (amid many other sections), and none of which was able to help me:
https://labelstud.io/guide/ml.html
https://labelstud.io/tutorials/object-detector.html
https://labelstud.io/guide/ml_troubleshooting.html
Can anyone help me understand why this is? Any feedback, solutions, or ideas would be greatly appreciated.
Steps to reproduce:

Run "git clone https://github.com/heartexlabs/label-studio-ml-backend"
Traverse to label-studio-ml-backend/label_studio_ml/examples/mmdetection/
Turn on the ml backend"docker-compose up"
In Label Studio, add the ml backend into a Object Detection with Bounding Boxes project
In Label Studio, open the first image with Auto-Annotation
Turn off the ml backend with "docker-compose down"

You will see how none of the images is auto-labeled, and no predictions are generated while the ml backend is connected.
Checking the log file of the ml backend, it is full of errors. (log file provided below.)
First Error:
[2022-10-11 03:28:36,789] [ERROR] [label_studio_ml.exceptions::exception_f::53] Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/label_studio_ml/exceptions.py", line 39, in exception_f
	return f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/label_studio_ml/api.py", line 32, in _predict
	predictions, model = _manager.predict(tasks, project, label_config, force_reload, try_fetch, **params)
  File "/usr/local/lib/python3.7/site-packages/label_studio_ml/model.py", line 582, in predict
	predictions = m.model.predict(tasks, **kwargs)
  File "./mmdetection.py", line 89, in predict
	assert len(tasks) == 1
AssertionError

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/label_studio_ml/exceptions.py", line 39, in exception_f
	return f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/label_studio_ml/api.py", line 32, in _predict
	predictions, model = _manager.predict(tasks, project, label_config, force_reload, try_fetch, **params)
  File "/usr/local/lib/python3.7/site-packages/label_studio_ml/model.py", line 582, in predict
	predictions = m.model.predict(tasks, **kwargs)
  File "./mmdetection.py", line 89, in predict
	assert len(tasks) == 1
AssertionError

Full Logs file:
uwsgi logs.txt
Full Logs file transcript:
*** Starting uWSGI <IP_ADDRESS> (64bit) on [Tue Oct 11 03:28:25 2022] ***
compiled with version: 10.2.1 20210110 on 11 October 2022 02:44:17
os: Linux-5.15.0-50-generic #56-Ubuntu SMP Tue Sep 20 13:23:26 UTC 2022
nodename: a122b259b43f
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 11
current working directory: /app
writing pidfile to /tmp/uwsgi.pid
detected binary path: /usr/local/bin/uwsgi
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
your memory page size is 4096 bytes
detected max file descriptor number: 1048576
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address <IP_ADDRESS>:9090 fd 3
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
Python version: 3.7.14 (default, Oct  5 2022, 16:09:13)  [GCC 10.2.1 20210110]
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x557f021398a0
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 145840 bytes (142 KB) for 1 cores
*** Operational MODE: single process ***
WSGI app 0 (mountpoint='') ready in 3 seconds on interpreter 0x557f021398a0 pid: 10 (default app)
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 10)
spawned uWSGI worker 1 (pid: 32, cores: 1)
[pid: 32|app: 0|req: 1/1] <IP_ADDRESS> () {32 vars in 662 bytes} [Tue Oct 11 03:28:35 2022] GET /health => generated 54 bytes in 3 msecs (HTTP/1.1 200) 2 headers in 71 bytes (1 switches on core 0)
Load new model from:  faster_rcnn_r50_fpn_1x_coco.py faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth
[pid: 32|app: 0|req: 2/2] <IP_ADDRESS> () {36 vars in 714 bytes} [Tue Oct 11 03:28:35 2022] POST /setup => generated 31 bytes in 717 msecs (HTTP/1.1 200) 2 headers in 71 bytes (1 switches on core 0)
[pid: 32|app: 0|req: 3/3] <IP_ADDRESS> () {32 vars in 665 bytes} [Tue Oct 11 03:28:36 2022] GET /health => generated 54 bytes in 0 msecs (HTTP/1.1 200) 2 headers in 71 bytes (1 switches on core 0)
[pid: 32|app: 0|req: 4/4] <IP_ADDRESS> () {36 vars in 717 bytes} [Tue Oct 11 03:28:36 2022] POST /setup => generated 31 bytes in 0 msecs (HTTP/1.1 200) 2 headers in 71 bytes (1 switches on core 0)
[2022-10-11 03:28:36,789] [ERROR] [label_studio_ml.exceptions::exception_f::53] Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/label_studio_ml/exceptions.py", line 39, in exception_f
	return f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/label_studio_ml/api.py", line 32, in _predict
	predictions, model = _manager.predict(tasks, project, label_config, force_reload, try_fetch, **params)
  File "/usr/local/lib/python3.7/site-packages/label_studio_ml/model.py", line 582, in predict
	predictions = m.model.predict(tasks, **kwargs)
  File "./mmdetection.py", line 89, in predict
	assert len(tasks) == 1
AssertionError

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/label_studio_ml/exceptions.py", line 39, in exception_f
	return f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/label_studio_ml/api.py", line 32, in _predict
	predictions, model = _manager.predict(tasks, project, label_config, force_reload, try_fetch, **params)
  File "/usr/local/lib/python3.7/site-packages/label_studio_ml/model.py", line 582, in predict
	predictions = m.model.predict(tasks, **kwargs)
  File "./mmdetection.py", line 89, in predict
	assert len(tasks) == 1
AssertionError

[pid: 32|app: 0|req: 5/5] <IP_ADDRESS> () {36 vars in 723 bytes} [Tue Oct 11 03:28:36 2022] POST /predict => generated 703 bytes in 3 msecs (HTTP/1.1 500) 2 headers in 91 bytes (2 switches on core 0)
[pid: 32|app: 0|req: 6/6] <IP_ADDRESS> () {32 vars in 674 bytes} [Tue Oct 11 03:28:36 2022] GET /health => generated 54 bytes in 0 msecs (HTTP/1.1 200) 2 headers in 71 bytes (1 switches on core 0)
[pid: 32|app: 0|req: 7/7] <IP_ADDRESS> () {36 vars in 726 bytes} [Tue Oct 11 03:28:36 2022] POST /setup => generated 31 bytes in 0 msecs (HTTP/1.1 200) 2 headers in 71 bytes (1 switches on core 0)
[2022-10-11 03:28:36,934] [ERROR] [label_studio_ml.exceptions::exception_f::53] Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn
	(self._dns_host, self.port), self.timeout, **extra_kw
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection
	raise err
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection
	sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen
	chunked=chunked,
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request
	conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request
	super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/local/lib/python3.7/http/client.py", line 1281, in request
	self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request
	self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders
	self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output
	self.send(msg)
  File "/usr/local/lib/python3.7/http/client.py", line 976, in send
	self.connect()
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect
	conn = self._new_conn()
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 187, in _new_conn
	self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7ff42cb26190>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 499, in send
	timeout=timeout,
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen
	method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 592, in increment
	raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /data/upload/3/testing_image_10.png (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff42cb26190>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/label_studio_ml/exceptions.py", line 39, in exception_f
	return f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/label_studio_ml/api.py", line 32, in _predict
	predictions, model = _manager.predict(tasks, project, label_config, force_reload, try_fetch, **params)
  File "/usr/local/lib/python3.7/site-packages/label_studio_ml/model.py", line 582, in predict
	predictions = m.model.predict(tasks, **kwargs)
  File "./mmdetection.py", line 92, in predict
	image_path = self.get_local_path(image_url)
  File "/usr/local/lib/python3.7/site-packages/label_studio_ml/model.py", line 322, in get_local_path
	return get_local_path(url, project_dir=project_dir, hostname=self.hostname, access_token=self.access_token)
  File "/usr/local/lib/python3.7/site-packages/label_studio_tools/core/utils/io.py", line 91, in get_local_path
	r = requests.get(url, stream=True, headers=headers)
  File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 73, in get
	return request("get", url, params=params, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 59, in request
	return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 587, in request
	resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 701, in send
	r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 565, in send
	raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /data/upload/3/testing_image_10.png (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff42cb26190>: Failed to establish a new connection: [Errno 111] Connection refused'))

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn
	(self._dns_host, self.port), self.timeout, **extra_kw
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection
	raise err
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection
	sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen
	chunked=chunked,
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request
	conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request
	super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/local/lib/python3.7/http/client.py", line 1281, in request
	self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request
	self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders
	self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output
	self.send(msg)
  File "/usr/local/lib/python3.7/http/client.py", line 976, in send
	self.connect()
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect
	conn = self._new_conn()
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 187, in _new_conn
	self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7ff42cb26190>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 499, in send
	timeout=timeout,
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen
	method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 592, in increment
	raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /data/upload/3/testing_image_10.png (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff42cb26190>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/label_studio_ml/exceptions.py", line 39, in exception_f
	return f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/label_studio_ml/api.py", line 32, in _predict
	predictions, model = _manager.predict(tasks, project, label_config, force_reload, try_fetch, **params)
  File "/usr/local/lib/python3.7/site-packages/label_studio_ml/model.py", line 582, in predict
	predictions = m.model.predict(tasks, **kwargs)
  File "./mmdetection.py", line 92, in predict
	image_path = self.get_local_path(image_url)
  File "/usr/local/lib/python3.7/site-packages/label_studio_ml/model.py", line 322, in get_local_path
	return get_local_path(url, project_dir=project_dir, hostname=self.hostname, access_token=self.access_token)
  File "/usr/local/lib/python3.7/site-packages/label_studio_tools/core/utils/io.py", line 91, in get_local_path
	r = requests.get(url, stream=True, headers=headers)
  File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 73, in get
	return request("get", url, params=params, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 59, in request
	return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 587, in request
	resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 701, in send
	r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 565, in send
	raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /data/upload/3/testing_image_10.png (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff42cb26190>: Failed to establish a new connection: [Errno 111] Connection refused'))

[pid: 32|app: 0|req: 8/8] <IP_ADDRESS> () {36 vars in 730 bytes} [Tue Oct 11 03:28:36 2022] POST /predict => generated 5616 bytes in 6 msecs (HTTP/1.1 500) 2 headers in 92 bytes (1 switches on core 0)
...brutally killing workers...
worker 1 buried after 1 seconds
binary reloading uWSGI...
chdir() to /app
closing all non-uwsgi socket fds > 2 (max_fd = 1048576)...
found fd 3 mapped to socket 0 (<IP_ADDRESS>:9090)
running /usr/local/bin/uwsgi
[uWSGI] getting INI configuration from /etc/uwsgi/uwsgi.ini
*** Starting uWSGI <IP_ADDRESS> (64bit) on [Tue Oct 11 03:29:02 2022] ***
compiled with version: 10.2.1 20210110 on 11 October 2022 02:44:17
os: Linux-5.15.0-50-generic #56-Ubuntu SMP Tue Sep 20 13:23:26 UTC 2022
nodename: a122b259b43f
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 11
current working directory: /app
detected binary path: /usr/local/bin/uwsgi
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
your memory page size is 4096 bytes
detected max file descriptor number: 4096
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 inherited INET address <IP_ADDRESS>:9090 fd 3
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
Python version: 3.7.14 (default, Oct  5 2022, 16:09:13)  [GCC 10.2.1 20210110]
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x55b29e7d02d0
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 145840 bytes (142 KB) for 1 cores
*** Operational MODE: single process ***
WSGI app 0 (mountpoint='') ready in 3 seconds on interpreter 0x55b29e7d02d0 pid: 10 (default app)
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
*** uWSGI is running in multiple interpreter mode ***
gracefully (RE)spawned uWSGI master process (pid: 10)
spawned uWSGI worker 1 (pid: 60, cores: 1)

Hi @Michael-Loia
Assertion error is not dangerous, it's only telling that previous training job is failed (we already have a fix for it in the future version).
The error you get is: Connection refused - your ML backend instance can't connect to your LS instance to get task data. Please check connectivity from ML backend to LS instance.

Hi @KonstantinKorotaev, thank you for your prompt response.
Can you help me to understand these issue's further?
Below is a screenshot of the LS instance:

The LS instance shows the ML backend as connected, yet it does not generate predictions, and the logs show errors initially mentioned above.
I do not understand how the LS instance is connected to the ML backend, yet the ML backend is not connected to the LS instance. Also, several images are already labeled, and I'm not sure why the training jobs failed.
Looking into the error logs, we can see that the ML backend was able to talk to the LS instance to get the image URL path
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 499, in send
	timeout=timeout,
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen
	method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 592, in increment
	raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /data/upload/3/testing_image_10.png (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff42cb26190>: Failed to establish a new connection: [Errno 111] Connection refused'))

Is there a way I can configure the ML backend the with the host address of the LS instance, as opposed to it assuming the host is localhost? Unfortunately, I was unable to find documentation regarding this.
Best,
Michael

Hi @Michael-Loia
You can modify mmdetection example on line 92 to replace hostname.

Hi @KonstantinKorotaev,
I went into the mmdectectiop.py file an replaced line 92 with the following:
 image_path = self.get_local_path('<IP_ADDRESS>')

Hi @KonstantinKorotaev,
I edited line 92 to be the following:
        image_path = self.get_local_path('<IP_ADDRESS>')

<IP_ADDRESS> is the LS instance.
The ML backend appears to have connected and exchanged information; however, it now shows different errors while continuing not to make predictions.
Do you have any insight as to why this is?
Relevant logs:
[pid: 32|app: 0|req: 1/1] <IP_ADDRESS> () {32 vars in 662 bytes} [Sat Oct 15 02:11:31 2022] GET /health => generated 54 bytes in 4 msecs (HTTP/1.1 200) 2 headers in 71 bytes (1 switches on core 0)
Load new model from:  faster_rcnn_r50_fpn_1x_coco.py faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth
[pid: 32|app: 0|req: 2/2] <IP_ADDRESS> () {36 vars in 714 bytes} [Sat Oct 15 02:11:31 2022] POST /setup => generated 31 bytes in 992 msecs (HTTP/1.1 200) 2 headers in 71 bytes (1 switches on core 0)
[pid: 32|app: 0|req: 3/3] <IP_ADDRESS> () {32 vars in 665 bytes} [Sat Oct 15 02:11:32 2022] GET /health => generated 54 bytes in 0 msecs (HTTP/1.1 200) 2 headers in 71 bytes (1 switches on core 0)
[pid: 32|app: 0|req: 4/4] <IP_ADDRESS> () {36 vars in 717 bytes} [Sat Oct 15 02:11:32 2022] POST /setup => generated 31 bytes in 0 msecs (HTTP/1.1 200) 2 headers in 71 bytes (1 switches on core 0)
[2022-10-15 02:11:33,112] [ERROR] [label_studio_ml.exceptions::exception_f::53] Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/label_studio_ml/exceptions.py", line 39, in exception_f
	return f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/label_studio_ml/api.py", line 32, in _predict
	predictions, model = _manager.predict(tasks, project, label_config, force_reload, try_fetch, **params)
  File "/usr/local/lib/python3.7/site-packages/label_studio_ml/model.py", line 582, in predict
	predictions = m.model.predict(tasks, **kwargs)
  File "./mmdetection.py", line 89, in predict
	assert len(tasks) == 1
AssertionError

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/label_studio_ml/exceptions.py", line 39, in exception_f
	return f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/label_studio_ml/api.py", line 32, in _predict
	predictions, model = _manager.predict(tasks, project, label_config, force_reload, try_fetch, **params)
  File "/usr/local/lib/python3.7/site-packages/label_studio_ml/model.py", line 582, in predict
	predictions = m.model.predict(tasks, **kwargs)
  File "./mmdetection.py", line 89, in predict
	assert len(tasks) == 1
AssertionError

[pid: 32|app: 0|req: 5/5] <IP_ADDRESS> () {36 vars in 723 bytes} [Sat Oct 15 02:11:33 2022] POST /predict => generated 703 bytes in 3 msecs (HTTP/1.1 500) 2 headers in 91 bytes (1 switches on core 0)

Full logs:
*** Starting uWSGI <IP_ADDRESS> (64bit) on [Sat Oct 15 02:10:44 2022] ***
compiled with version: 10.2.1 20210110 on 11 October 2022 02:44:17
os: Linux-5.15.0-50-generic #56-Ubuntu SMP Tue Sep 20 13:23:26 UTC 2022
nodename: 1c3a7457f248
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 11
current working directory: /app
writing pidfile to /tmp/uwsgi.pid
detected binary path: /usr/local/bin/uwsgi
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
your memory page size is 4096 bytes
detected max file descriptor number: 1048576
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address <IP_ADDRESS>:9090 fd 3
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
Python version: 3.7.14 (default, Oct  5 2022, 16:09:13)  [GCC 10.2.1 20210110]
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x55fc156c28a0
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 145840 bytes (142 KB) for 1 cores
*** Operational MODE: single process ***
WSGI app 0 (mountpoint='') ready in 3 seconds on interpreter 0x55fc156c28a0 pid: 10 (default app)
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 10)
spawned uWSGI worker 1 (pid: 32, cores: 1)
[pid: 32|app: 0|req: 1/1] <IP_ADDRESS> () {32 vars in 662 bytes} [Sat Oct 15 02:11:31 2022] GET /health => generated 54 bytes in 4 msecs (HTTP/1.1 200) 2 headers in 71 bytes (1 switches on core 0)
Load new model from:  faster_rcnn_r50_fpn_1x_coco.py faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth
[pid: 32|app: 0|req: 2/2] <IP_ADDRESS> () {36 vars in 714 bytes} [Sat Oct 15 02:11:31 2022] POST /setup => generated 31 bytes in 992 msecs (HTTP/1.1 200) 2 headers in 71 bytes (1 switches on core 0)
[pid: 32|app: 0|req: 3/3] <IP_ADDRESS> () {32 vars in 665 bytes} [Sat Oct 15 02:11:32 2022] GET /health => generated 54 bytes in 0 msecs (HTTP/1.1 200) 2 headers in 71 bytes (1 switches on core 0)
[pid: 32|app: 0|req: 4/4] <IP_ADDRESS> () {36 vars in 717 bytes} [Sat Oct 15 02:11:32 2022] POST /setup => generated 31 bytes in 0 msecs (HTTP/1.1 200) 2 headers in 71 bytes (1 switches on core 0)
[2022-10-15 02:11:33,112] [ERROR] [label_studio_ml.exceptions::exception_f::53] Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/label_studio_ml/exceptions.py", line 39, in exception_f
	return f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/label_studio_ml/api.py", line 32, in _predict
	predictions, model = _manager.predict(tasks, project, label_config, force_reload, try_fetch, **params)
  File "/usr/local/lib/python3.7/site-packages/label_studio_ml/model.py", line 582, in predict
	predictions = m.model.predict(tasks, **kwargs)
  File "./mmdetection.py", line 89, in predict
	assert len(tasks) == 1
AssertionError

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/label_studio_ml/exceptions.py", line 39, in exception_f
	return f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/label_studio_ml/api.py", line 32, in _predict
	predictions, model = _manager.predict(tasks, project, label_config, force_reload, try_fetch, **params)
  File "/usr/local/lib/python3.7/site-packages/label_studio_ml/model.py", line 582, in predict
	predictions = m.model.predict(tasks, **kwargs)
  File "./mmdetection.py", line 89, in predict
	assert len(tasks) == 1
AssertionError

[pid: 32|app: 0|req: 5/5] <IP_ADDRESS> () {36 vars in 723 bytes} [Sat Oct 15 02:11:33 2022] POST /predict => generated 703 bytes in 3 msecs (HTTP/1.1 500) 2 headers in 91 bytes (1 switches on core 0)

Best,
Michael

Hi @KonstantinKorotaev,
I am following up to see if you have any insight into why the example is not working with the recommended changes.
I did not intend to mark this as closed/completed; that was a misclick.
I am still interested in using this LS, but I have been unable to get the example to work and have been stuck since then.
Is there any documentation that gives a step by step setup instructions for the docker-compose that is for the ML backend mmdetection example for Object Detection with Bounding Boxes? Unfortunately, I have been unable to find anything.
I would greatly appreciate any and all recommendations.
Best,
Michael

Hi @KonstantinKorotaev,
I've set line 89 to 500 (There are not more than 7 objects that should be detected, they are in plain view and close)
I set it to 500 because that is when stopped giving another error that said it was set to 1 and when I set it to 5.
assert len(tasks) == 500

I've set line 92 to <IP_ADDRESS> (LS Instance IP)
image_path = self.get_local_path('<IP_ADDRESS>')

Yet there are still not predictions and despite the change and a restart of the docker, it had the following error(s)
I can't seem to get anything to work and I can't find any documentation.
I also don not understand the connection errors. It is set in the file and it gets the IP with the health checks, it should know exactly what to talk to with that information.
[pid: 32|app: 0|req: 1/1] <IP_ADDRESS> () {32 vars in 674 bytes} [Fri Oct 21 01:12:57 2022] GET /health => generated 54 bytes in 3 msecs (HTTP/1.1 200) 2 headers in 71 bytes (1 switches on core 0)

Is there any documentation that describes this example in depth? Or perhaps a pre-setup example with a simple config file as a docker container?
I greatly appreciate your continued efforts and recommendations.
Relevant logs:
[2022-10-21 01:12:58,708] [ERROR] [label_studio_ml.exceptions::exception_f::53] Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn
	(self._dns_host, self.port), self.timeout, **extra_kw
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection
	raise err
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection
	sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen
	chunked=chunked,
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request
	conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request
	super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/local/lib/python3.7/http/client.py", line 1281, in request
	self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request
	self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders
	self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output
	self.send(msg)
  File "/usr/local/lib/python3.7/http/client.py", line 976, in send
	self.connect()
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect
	conn = self._new_conn()
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 187, in _new_conn
	self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fe33fb9e790>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 499, in send
	timeout=timeout,
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen
	method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 592, in increment
	raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /data/upload/3/f491ea10-IMG_6823_jpg.rf.28533445ef71ad35b6ac5a8837bcb80a.jpg (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe33fb9e790>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/label_studio_ml/exceptions.py", line 39, in exception_f
	return f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/label_studio_ml/api.py", line 32, in _predict
	predictions, model = _manager.predict(tasks, project, label_config, force_reload, try_fetch, **params)
  File "/usr/local/lib/python3.7/site-packages/label_studio_ml/model.py", line 582, in predict
	predictions = m.model.predict(tasks, **kwargs)
  File "./mmdetection.py", line 92, in predict
	image_path = self.get_local_path(image_url)
  File "/usr/local/lib/python3.7/site-packages/label_studio_ml/model.py", line 322, in get_local_path
	return get_local_path(url, project_dir=project_dir, hostname=self.hostname, access_token=self.access_token)
  File "/usr/local/lib/python3.7/site-packages/label_studio_tools/core/utils/io.py", line 91, in get_local_path
	r = requests.get(url, stream=True, headers=headers)
  File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 73, in get
	return request("get", url, params=params, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 59, in request
	return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 587, in request
	resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 701, in send
	r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 565, in send
	raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /data/upload/3/f491ea10-IMG_6823_jpg.rf.28533445ef71ad35b6ac5a8837bcb80a.jpg (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe33fb9e790>: Failed to establish a new connection: [Errno 111] Connection refused'))

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn
	(self._dns_host, self.port), self.timeout, **extra_kw
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection
	raise err
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection
	sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen
	chunked=chunked,
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request
	conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request
	super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/local/lib/python3.7/http/client.py", line 1281, in request
	self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request
	self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders
	self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output
	self.send(msg)
  File "/usr/local/lib/python3.7/http/client.py", line 976, in send
	self.connect()
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect
	conn = self._new_conn()
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 187, in _new_conn
	self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fe33fb9e790>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 499, in send
	timeout=timeout,
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen
	method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 592, in increment
	raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /data/upload/3/f491ea10-IMG_6823_jpg.rf.28533445ef71ad35b6ac5a8837bcb80a.jpg (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe33fb9e790>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/label_studio_ml/exceptions.py", line 39, in exception_f
	return f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/label_studio_ml/api.py", line 32, in _predict
	predictions, model = _manager.predict(tasks, project, label_config, force_reload, try_fetch, **params)
  File "/usr/local/lib/python3.7/site-packages/label_studio_ml/model.py", line 582, in predict
	predictions = m.model.predict(tasks, **kwargs)
  File "./mmdetection.py", line 92, in predict
	image_path = self.get_local_path(image_url)
  File "/usr/local/lib/python3.7/site-packages/label_studio_ml/model.py", line 322, in get_local_path
	return get_local_path(url, project_dir=project_dir, hostname=self.hostname, access_token=self.access_token)
  File "/usr/local/lib/python3.7/site-packages/label_studio_tools/core/utils/io.py", line 91, in get_local_path
	r = requests.get(url, stream=True, headers=headers)
  File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 73, in get
	return request("get", url, params=params, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 59, in request
	return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 587, in request
	resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 701, in send
	r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 565, in send
	raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /data/upload/3/f491ea10-IMG_6823_jpg.rf.28533445ef71ad35b6ac5a8837bcb80a.jpg (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe33fb9e790>: Failed to establish a new connection: [Errno 111] Connection refused'))

Full Logs:
*** Starting uWSGI <IP_ADDRESS> (64bit) on [Fri Oct 21 01:12:46 2022] ***
compiled with version: 10.2.1 20210110 on 11 October 2022 02:44:17
os: Linux-5.15.0-50-generic #56-Ubuntu SMP Tue Sep 20 13:23:26 UTC 2022
nodename: 25d846ef2678
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 11
current working directory: /app
writing pidfile to /tmp/uwsgi.pid
detected binary path: /usr/local/bin/uwsgi
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
your memory page size is 4096 bytes
detected max file descriptor number: 1048576
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address <IP_ADDRESS>:9090 fd 3
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
Python version: 3.7.14 (default, Oct  5 2022, 16:09:13)  [GCC 10.2.1 20210110]
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x55d1076658a0
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 145840 bytes (142 KB) for 1 cores
*** Operational MODE: single process ***
WSGI app 0 (mountpoint='') ready in 2 seconds on interpreter 0x55d1076658a0 pid: 10 (default app)
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 10)
spawned uWSGI worker 1 (pid: 32, cores: 1)
[pid: 32|app: 0|req: 1/1] <IP_ADDRESS> () {32 vars in 674 bytes} [Fri Oct 21 01:12:57 2022] GET /health => generated 54 bytes in 3 msecs (HTTP/1.1 200) 2 headers in 71 bytes (1 switches on core 0)
Load new model from:  faster_rcnn_r50_fpn_1x_coco.py faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth
[pid: 32|app: 0|req: 2/2] <IP_ADDRESS> () {36 vars in 726 bytes} [Fri Oct 21 01:12:57 2022] POST /setup => generated 31 bytes in 1242 msecs (HTTP/1.1 200) 2 headers in 71 bytes (1 switches on core 0)
[2022-10-21 01:12:58,708] [ERROR] [label_studio_ml.exceptions::exception_f::53] Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn
	(self._dns_host, self.port), self.timeout, **extra_kw
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection
	raise err
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection
	sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen
	chunked=chunked,
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request
	conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request
	super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/local/lib/python3.7/http/client.py", line 1281, in request
	self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request
	self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders
	self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output
	self.send(msg)
  File "/usr/local/lib/python3.7/http/client.py", line 976, in send
	self.connect()
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect
	conn = self._new_conn()
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 187, in _new_conn
	self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fe33fb9e790>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 499, in send
	timeout=timeout,
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen
	method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 592, in increment
	raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /data/upload/3/f491ea10-IMG_6823_jpg.rf.28533445ef71ad35b6ac5a8837bcb80a.jpg (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe33fb9e790>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/label_studio_ml/exceptions.py", line 39, in exception_f
	return f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/label_studio_ml/api.py", line 32, in _predict
	predictions, model = _manager.predict(tasks, project, label_config, force_reload, try_fetch, **params)
  File "/usr/local/lib/python3.7/site-packages/label_studio_ml/model.py", line 582, in predict
	predictions = m.model.predict(tasks, **kwargs)
  File "./mmdetection.py", line 92, in predict
	image_path = self.get_local_path(image_url)
  File "/usr/local/lib/python3.7/site-packages/label_studio_ml/model.py", line 322, in get_local_path
	return get_local_path(url, project_dir=project_dir, hostname=self.hostname, access_token=self.access_token)
  File "/usr/local/lib/python3.7/site-packages/label_studio_tools/core/utils/io.py", line 91, in get_local_path
	r = requests.get(url, stream=True, headers=headers)
  File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 73, in get
	return request("get", url, params=params, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 59, in request
	return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 587, in request
	resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 701, in send
	r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 565, in send
	raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /data/upload/3/f491ea10-IMG_6823_jpg.rf.28533445ef71ad35b6ac5a8837bcb80a.jpg (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe33fb9e790>: Failed to establish a new connection: [Errno 111] Connection refused'))

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn
	(self._dns_host, self.port), self.timeout, **extra_kw
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection
	raise err
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection
	sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen
	chunked=chunked,
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request
	conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request
	super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/local/lib/python3.7/http/client.py", line 1281, in request
	self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request
	self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders
	self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output
	self.send(msg)
  File "/usr/local/lib/python3.7/http/client.py", line 976, in send
	self.connect()
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect
	conn = self._new_conn()
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 187, in _new_conn
	self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fe33fb9e790>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 499, in send
	timeout=timeout,
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen
	method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 592, in increment
	raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /data/upload/3/f491ea10-IMG_6823_jpg.rf.28533445ef71ad35b6ac5a8837bcb80a.jpg (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe33fb9e790>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/label_studio_ml/exceptions.py", line 39, in exception_f
	return f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/label_studio_ml/api.py", line 32, in _predict
	predictions, model = _manager.predict(tasks, project, label_config, force_reload, try_fetch, **params)
  File "/usr/local/lib/python3.7/site-packages/label_studio_ml/model.py", line 582, in predict
	predictions = m.model.predict(tasks, **kwargs)
  File "./mmdetection.py", line 92, in predict
	image_path = self.get_local_path(image_url)
  File "/usr/local/lib/python3.7/site-packages/label_studio_ml/model.py", line 322, in get_local_path
	return get_local_path(url, project_dir=project_dir, hostname=self.hostname, access_token=self.access_token)
  File "/usr/local/lib/python3.7/site-packages/label_studio_tools/core/utils/io.py", line 91, in get_local_path
	r = requests.get(url, stream=True, headers=headers)
  File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 73, in get
	return request("get", url, params=params, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 59, in request
	return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 587, in request
	resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 701, in send
	r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 565, in send
	raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /data/upload/3/f491ea10-IMG_6823_jpg.rf.28533445ef71ad35b6ac5a8837bcb80a.jpg (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe33fb9e790>: Failed to establish a new connection: [Errno 111] Connection refused'))

[pid: 32|app: 0|req: 3/3] <IP_ADDRESS> () {36 vars in 730 bytes} [Fri Oct 21 01:12:58 2022] POST /predict => generated 5604 bytes in 18 msecs (HTTP/1.1 500) 2 headers in 92 bytes (1 switches on core 0)

Best,
Michael

assert len(tasks) == 500

You can comment this line.

requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /data/upload/3/f491ea10-IMG_6823_jpg.rf.28533445ef71ad35b6ac5a8837bcb80a.jpg (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe33fb9e790>: Failed to establish a new connection: [Errno 111] Connection refused'))

Exception tells that there is no connection by the link.

Is there any documentation that describes this example in depth? Or perhaps a pre-setup example with a simple config file as a docker container?

This method is used to access data on LS side:
https://github.com/heartexlabs/label-studio-tools/blob/master/label_studio_tools/core/utils/io.py#L34
Examples of use:
Here is the method to access Label Studio tasks: https://github.com/heartexlabs/label-studio-ml-backend/blob/master/label_studio_ml/examples/simple_text_classifier/simple_text_classifier.py#L97
And to access task data use methods from label-studio-tools: https://github.com/heartexlabs/label-studio-ml-backend/blob/master/label_studio_ml/examples/mmdetection/mmdetection.py#L92

I am facing the similar issue as well.

Hi, if I face the similar problem. I start both label-studio and mmdetection by docker.
According to the iohttps://github.com/heartexlabs/label-studio-tools/blob/master/label_studio_tools/core/utils/io.py here, when detecting the file from local, it will add hostname mmdetection docker detect, which is localhost:8080 for me.
Here comes the issure, for mmdetection docker, it can not access localhost for host machine.
To solve this, edit [https://github.com/heartexlabs/label-studio-ml-backend/blob/master/label_studio_ml/examples/mmdetection/mmdetection.py#L92]
image_path = self.get_local_path("http://label-studio:8080" + image_url)
For me, I start label-studio and mmdetection container in the same docker net so that mmdetection can access label-studio by container's name.
Maybe you can try
image_path = self.get_local_path("http://host.docker.internal:8080" + image_url)
to get localhost for host machine

@abinashsinha330

I am facing the similar issue as well.

Do you have requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8080)  in your logs?
Can you ping localohost:8080 from your ML backend instance?

Hello,
Did you solved it ?
I have the same issue to get this tutorial working.
I have a conda env with mmdetection installed and when I run the script:
label-studio-ml init coco-detector --from label_studio_ml/examples/mmdetection/mmdetection.py
I get the error message:
Can't import module "mmdetection", reason: No module named 'mmdet'.

Hi @ODAncona
Which version of mmdetection do you use?

So here is how I got mmdetection-3 working, I am writing this here so that someone else can take some hints from it and reverse engineer their own setup.
Note: I do not use docker, rather I just run the backend in a terminal (instructions below)

Clone the Label Studio ML Backend repository in your directory of choice
git clone https://github.com/heartexlabs/label-studio-ml-backend

After some reverse engineering, I figured out the following patch:
label_studio_ml.patch
Here's the patch in text form if you don't feel comfortable downloading the one attached above:
diff --git a/label_studio_ml/examples/mmdetection-3/mmdetection.py b/label_studio_ml/examples/mmdetection-3/mmdetection.py
index 9d8df46..f7f6309 100644
--- a/label_studio_ml/examples/mmdetection-3/mmdetection.py
+++ b/label_studio_ml/examples/mmdetection-3/mmdetection.py
@@ -38,6 +38,7 @@ class MMDetection(LabelStudioMLBase):
         :param device: device (cpu, cuda:0, cuda:1, ...)
         :param kwargs:
         """
+        kwargs["label_config"] = '<View>\n  <View style="display:flex;align-items:start;gap:8px;flex-direction:column-reverse"><Image name="image" value="$image" zoom="true" zoomControl="true" rotateControl="true"/><RectangleLabels name="label" toName="image" showInline="true">\n    \n    \n  <Label value="stop sign" background="#D4380D"/></RectangleLabels></View>\n  \n\</View>'
         super(MMDetection, self).__init__(**kwargs)
         config_file = config_file or os.environ['config_file']
         checkpoint_file = checkpoint_file or os.environ['checkpoint_file']
diff --git a/label_studio_ml/examples/mmdetection-3/requirements.txt b/label_studio_ml/examples/mmdetection-3/requirements.txt
index c836942..148b031 100644
--- a/label_studio_ml/examples/mmdetection-3/requirements.txt
+++ b/label_studio_ml/examples/mmdetection-3/requirements.txt
@@ -7,10 +7,10 @@ werkzeug==2.0.2
 itsdangerous==2.0.1

 boto3
-torch
-torchvision
+torch==1.13.0
+torchvision==0.14.0
 openmim

 opencv-contrib-python-headless
-mmcv==2.0.0rc3
+# mmcv==2.0.0rc3
 mmdet==3.0.0rc5

You can apply this patch right after cloning the Label Studio ML Backend repo:
patch -ruN -p1 -d <path/to/repo>/label-studio-ml-backend < <path/to/patch>/label_studio_ml.patch

Change directory to the available models directory
cd label-studio-ml-backend/label_studio_ml/examples/mmdetection-3

Create a virtual environment using venv and install all dependencies using pip
python -m venv ml-backend
source ml-backend/bin/activate # assuming you're on bash or zsh
pip install -r requirements.txt

Install and Download mmdet related depedencies in the virtual environment
mim install mmengine
mim download mmdet --config yolov3_mobilenetv2_8xb24-320-300e_coco --dest .
mim install mmcv==2.0.0rc3

Export required variables
export checkpoint_file=yolov3_mobilenetv2_320_300e_coco_20210719_215349-d18dff72.pth
export config_file=yolov3_mobilenetv2_8xb24-320-300e_coco.py

Run the following command to start your ML backend
label-studio-ml start --root-dir .. mmdetection-3 --kwargs hostname=http://<ip.of.your.label.studio.server>:8080 access_token=<access_token> [score_threshold=0.5] [--check]

Use this guide to find out your access token: https://labelstud.io/guide/api.html
If you're running label studio and the ML backend on the same machine, then the IP of your label studio server could also be "localhost". Try it, I haven't tested this.
score_threshold and --check are optional.
You can use and increased value of score_threshold parameter when you see a lot of unwanted detections or lower its value if you don't see any detections.
The --check parameter allows you to check if all requirements to run the backend have been fulfilled

@raash1d Thank you for your great instruction! I will include it soon into README.md for this model.
Only one thing I am not sure:
kwargs["label_config"] = ...

I believe it's not a good idea to hardcode label config into the model, it should be retrieved automatically when you connect the model.

Hi @makseq, I agree the hardcoding shouldn't be done but the label_config wasn't being retrieved. My solution is by no means the "proper" way and is 100% hacky. But after 5 days of pulling my hair out, I convinced myself to go with it. I'm sure you'll do a better a job at the README :)

@raash1d it would be great if you could check the latest changes in this model on your end and confirmed that it works as expected.


Book time to focus - auto sets Teams to "focusing/DND"

I am a big fan of the Book time to focus feature with Insights. What I do want to do is change a setting so when I am in my scheduled focus time my Teams status is not set to Focusing or DND by default. Is there an option that I am missing to make this change?

Hi rogersda322,
Sorry about the delay. I've contacted the product team about this. I'll reply back here as soon as I have a definitive answer.
Thanks for your interest in the feature,
-- Paul

Hi rogersda322,
The product team responded and said that a feature is being developed that will affect Focus-plan settings. After it ships (perhaps this year), it will let you configure whether your Teams "presence" state will show as DND or simply as "busy" during focus time. When it appears, this feature will be available on the Focus page of your MyAnalytics dashboard.
-- Paul


Pipeline parallel not supported in benchmark?

I built a model with tp 2 pp 2, that's world size 4, and benchmark it. It wars reported that expected world size is 2, and I check the code:
https://github.com/NVIDIA/TensorRT-LLM/blob/release/0.5.0/benchmarks/python/base_benchmark.py#L70C30-L70C30
It seems pipeline parallel is not supported in benchmark yet?

PP support has been really enabled only for the C++ backend. There’s a “naive” implementation in Python to help us debug but it should not be used for performance studies.

Hello, is PP support roadmapped for the python benchmarks?

For now, the Python runtime contains a simple but not efficient implementation of PP. It's been implemented only for debugging purposes. We do not plan to update it in a near future. We are working on wrappers of the C++ using Python to offer a fast implementation to our Python users.


Custom data feed

How do I add my own data feed

Are there any updates on this? @rafaelklaessen

The trading view widget is not for custom data feed. https://www.tradingview.com/widget/advanced-chart/, this is not a limitation of this React module, but trading view.

you could try
https://tradingview.github.io/lightweight-charts/tutorials/react/simple


Scale block proposal scores based on pre-existing descendants

As per @stefa2k https://beaconcha.in/block/573311 was orphaned because the parent root was for a block a long way back in the chain, which already had descendants.
The beacon block proposal strategy must take this distance in to account, scaling down the score for such proposals so that they are less favourable than those that are for the root of the current head as defined by consensus across all nodes.

Addressed in release 0.9.0


Remove version: Peppy.Osu! version 2023.121.0

https://github.com/microsoft/winget-pkgs/issues/152768#issuecomment-2102735187

Resolves #152768

Microsoft Reviewers: Open in CodeFlow

URL: https://github.com/ppy/osu/releases/download/2023.121.0/install.exe
OK
(Automated message - build 882)


Port //engine/sky/tools/flutter_gdb to a subcommand of "flutter"

This seems unlikely to be desired, given our move on top of XCode and Android Studio.  I suspect when folks want to connect gdb they should do so from one of those, no?


Use the Idempotent Publishing feature of the Ably SDK

We should use the Idempotent Publishing feature to avoid message duplication while resending location updates. To do that we need to enable the option in the Ably SDK configuration and assign IDs for the location update messages.
This was done on Android as a part of this PR
┆Issue is synchronized with this Jira Uncategorised by Unito

I'm not sure if implementing this will change anything as the idempotent publishing seems to only be applicable to the REST publishing. We are using realtime so I'm not sure if enabling the REST idempotent publishing will change anything for the AblyRealtime.

I guess that means that the work to be done under this issue is actually to remove this line, needlessly setting the idempotentRestPublishing client option, from the Android implementation.

Looked at the iOS code, we're indeed using only Realtime here
And api specs confirm that idempotent publishing "does not apply in Realtime publishing": (RTL6a1)
So in my opinion we can close this issue, and as Quintin mentioned above - we might want to remove the line from Android implementation @KacperKluka

I've created an issue for it in the Android repo, thanks for checking 🙇


[New feature] Add support for ldap access

Just to support ldap authorizations

Could you describe the feature you are asking for more in detail? For instance, which new workflows are expected? How would the new feature work with the rest of functionality, etc?


I don't find in this repository.

you can see font codes in demo html

I need the codes too (in the config.json file, not shown on my screen on demo.html) to use them in an SVG object.
Am I missing something here? Is there another way to do it?
What's the use of the code shown in config.json by the way? Can't find an answer.
I asked the question on Stackoverflow too, no answer so far.
@puzrin would you know about that?
https://stackoverflow.com/questions/57819421/how-do-i-retrieve-a-fontello-icons-code-programatically
Thank you

I figured it, the codes in the json file are a decimal version of the hex code that can be found in the css file. You have to write code.toString(16) to get the hex value. I guess it was obvious to erveryone else since it's not mentionned anywhere :(

https://github.com/fontello/fontello/tree/master/lib/embedded_fonts (autogenerated)


Group ACL support

Corresponds with solid-server PR https://github.com/solid/node-solid-server/pull/719

I can't get group ACLs working yet: https://pastebin.com/92cB4BB8

Here's my group listing:
@prefix acl: <http://www.w3.org/ns/auth/acl#>.
@prefix vcard: <http://www.w3.org/2006/vcard/ns#> .

<> a acl:GroupListing.

<#meandi>
    a vcard:Group;
    vcard:hasUID <urn:uuid:8831CBAD-1111-2222-8563-F0F4787EG398:ABGroup> ;

  # Simply local and remote identity for myself
  vcard:hasMember </profile/card#me>;
  vcard:hasMember <https://solid.kjernsmo.net/profile/card#me>.

I don't know what the UID is, so I just put something in there. Isn't this about the minimal group listing there could be?


zone-lockdown: paginating list response

Description
I'm adding pagination for ListZoneLockdowns API. Additionally, I'm bringing Lockdown APIs to a common pattern. Instead of returning ZoneLockdownResponse they will return ZoneLockdown so they are easier to use.
Has your change been tested?
Unit tests are working
Screenshots (if appropriate):
Types of changes
What sort of change does your code introduce/modify?

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[x] I have added tests to cover my changes.
[x] All new and existing tests passed.
[x] This change is using publicly documented (api.cloudflare.com or developers.cloudflare.com) and stable APIs.

instead of introducing a change that takes us further away from the experimental method signatures, let's update it to move towards it.
check out https://github.com/cloudflare/cloudflare-go/blob/master/tunnel.go for an example of where we should head.

That's very useful, thanks Jacob! I've opened an issue for further clarifications: 1023


Store all redirects completely resolved

I don't believe we should need this kind of code when resolving on a resolved graph:
https://github.com/denoland/deno_graph/blob/2356edc5d69fedbeea82ecd7d82ce45687c2411f/src/graph.rs#L1185-L1202
Resolving once the graph is resolved can be one step to the destination for performance reasons. I don't believe storing all the steps is necessary. If it's infinite, then we should store that in the graph as an error at the redirect.

We actually already do this.


ci: fix rust build target settings in CI

Follow up #2790

Sorry, Prettier doesn't work with my VS Code and I can't format the yaml file.

It's weird that it now just works!

I have reverted to musl only for prqlc builds. The tests are still on gnu.
As you know, the default target on Linux is gnu, and this seems to be because musl requires additional tools to be installed, etc.? (you should be able to find various articles on this if you look it up)

Ah that explains it!
That doesn't share the cache, so this undoes some of the work. But the ubuntu builds are fast anyway, so it only takes the total time from 2.5 to 3.5 minutes. If you want to try and get the tests working on musl then that could be good. But overall we've overall done a lot here and it's in a good state, we can leave it for a while.
Thank you!

(and FYI I downloaded it and tested it — it works. Kinda weird the tests don't work then; I'm fairly confident I had added the apt-get command)


Component out of order in a deconstruction pattern

Just capturing an idea; not sure if it is high-value. In something like
case MyRecord(int foo, _, var bar)
A warning could be issued if the name of any pattern variable (here, "foo" or "bar") has a shorter edit distance to some other component of the record than the one it is matched up with.
Positional APIs can be hard to remember. This applies to instanceof as well. Currently this only applies to record patterns, but in the future there may be deconstructors for non-record classes too.

I think we may already have this via ArgumentSelectionDefectChecker and its friends.
We should certainly make sure that applies to records as well! I'm suspicious that there's a separate AutoValueConstructorOrderChecker, which makes me wonder if we do need to special-case records somehow.

I realised on a second view you're specifically referring to deconstruction not constructing records. I shall go clean my glasses. :)


Can this actually be used in Kubernetes?

I am trying to  implement cote in kubernetes on IBM bluemix and am having alot of trouble getting it to work.
The bluemix kubernetes implementation has Calico plugin at network layer and Pods can definitely access each other via IP.
I have put a redis service in place and set cote to use that for discovery which works:
2017-09-13T05:23:41.920741203Z core space requester > service.online idea service space responder#7528508b-a087-414e-ade8-2a95825472e1 on 8002

however once it starts actually messaging it seems to drop back to hostnames that do not resolve inside kubernetes, e.g.
2017-09-13T05:23:46.925401243Z Error: getaddrinfo EAI_AGAIN idea:8000
2017-09-13T05:23:46.925441235Z     at Object._errnoException (util.js:1041:11)
2017-09-13T05:23:46.925447197Z     at errnoException (dns.js:58:15)
2017-09-13T05:23:46.925452669Z     at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:95:26)

I've trawled through the implementation of node-discover and can see the hostname getting encoded in the message but I'm unclear if that hostname is used to respond.
Is there a way to insist on using IP addresses for communication rather than hostnames or am I looking down the wrong path here?

Apparently, the host name configuration was a preliminary move and a mistake for K8S. One thing you can do to bypass it and use IP addresses again is to pass in a config object when you require cote (it needs to be done only once, other requires don't need it);
const cote = require('cote', { useHostNames: false });

Having said that, please take a look at https://github.com/dashersw/cote-workshop/pull/5 which details a K8S setup. It will be official soon.

Thanks for your quick reply!!
Unfortunately neither of these approaches seem to work for me.
I have tried installing weave CNI plugin however it does not appear to be compatible with the bluemix implementation of Kubernetes. and the Calico implementation there does not support multicasting.
The useHostNames: false change has stopped the error I was seeing however no messages are being received (I assume this is because the reported address when using redis for discovery is <IP_ADDRESS>).
To check that anything at all is working, I have configured a kubernetes subdomain as per their doco (https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/), using this subdomain the pods can see each other using hostname.subdomain however I can't see a clear way to get cote to use that (other than forking cote and node-discover and change the hostName var to accept an environment variable).
Any advise/help you can provide would be amazing!

Would you consider changing or accepting a Pull Request in node-discover? something to the effect of:
(lib/network.js#11)
var hostName = process.env.DISCOVERY_HOSTNAME || os.hostname();

@drubin what do you think about the issue?
@NickyTope if we can't find a solution that works here, we could have an environment variable as you suggested to set the discovery names of each service individually.

@NickyTope I also had a similar issue with weave networking lucky for me it was simple as switching from broadcast to multicast. (and reading the docs again).
I will try next week to see if I can get a cluster running Calico and see if I can get a setup working.
As for the hostname approach, k8s and hostnames are a little iffy since a pod's DNS is pretty much its IP with -'s in it. I had a similar suggestion of using the below to inject the discover_name but I found a work around using Weave net.
    - name: MY_POD_IP
      valueFrom:
        fieldRef:
          fieldPath: status.podIP

Also from https://docs.projectcalico.org/v2.5/usage/openstack/semantics

Note: Calico’s connectivity design, based on IP routing, allows unicast IP and anycast IP. Anycast IP also requires support for allowed-address-pairs, or some other way of assigning the same IP address to more than one instance; work for allowed-address-pairs support is in progress at https://review.openstack.org/#/c/344008/. Multicast IP support is on our roadmap but not yet implemented. Broadcast IP is not possible because it depends on L2 adjacency.

It seems like a safe bet will be to make the redis integration work with correct ips.

Thanks for looking @drubin and @dashersw I'm glad that what I've been coming up with is similar to what you have found.
I have currently deployed successfully using a fork of node-discover (and cote to point at it) with the env.DISCOVERY_HOSTNAME || os.hostname() change above and am using pretty much exactly what you put above for the Deployment yaml:
- name: DISCOVERY_HOSTNAME
    valueFrom:
      fieldRef:
        fieldPath: status.podIP

I also found that multicast is on the way for Calico (and many other of the available network plugins for kube), so am looking forward to going back to less configuration for cote in future, however in my current situation (deployment via IBM bluemix) I don't have enough control over the host environment to change network plugins so would really appreciate you guys considering this change, or perhaps you can see a better way to go back to useHostNames: false in combination with redis discovery and an environment variable so the correct IP can be reported to redis?
Really appreciate you guys looking into this!

I think the easiest approach for this would be what you have suggested above.
Also there potentially 2 ways of getting the IP address

From the packet. (I found with k8s setup with broadcast the source address is the internal router at not the actual pod.)
From the pod its self and send it through. (Like your method above)

The issue with useHostNames: false with redis is that each node has to pick what their IP (because of the router issue above) should be and most pods have multiple IP addresses, <IP_ADDRESS>, <IP_ADDRESS>, their internal network, some cases IPV6 address and so on. So I originally had a proposal to "just pick the first IPV4 address that is a not internal" but I think this is also pretty error-prone for a fallback.
I think if the autodiscovery doesn't work it makes the most sense to allow the operator to explicitly provide a config + broker (in this case redis).
Obviously, this is a fallback/worst case but in my opinion, provides way more stable predictable method of discovery than trying to build a few more heuristics that might not work all the time anyway.
What are your thoughts? @dashersw @NickyTope

Thanks for your thoughts and comments.
Totally agree that this is a fallback / worst case scenario for cote discovery but it allows at least a working environment in a network where the current options are restricted or not viable.
The changes mentioned above have been implemented in my dev environment and are working flawlessly.
@dashersw I have created a pull request with the above in your fork of node-discover for your consideration.
I still don't think it's ideal that I'm sending the IP and calling it 'hostname' however as @drubin mentioned above, that is part of the way hostnames can be assigned in k8s anyway and the proposed change does not at all mention sending the IP, you could also setup a k8s subdomain as mentioned above and provide something like . as the hostname.

Hello.
I'm trying use cote in k8s cluster. But service discovery not work. I used DISCOVERY_HOSTNAME  and
const cote = require('cote', { useHostNames: false });

Also start workshop example cluster, but services also not find each other.
May be need additional configuration for k8s?

@LeoVS09 as mentioned https://github.com/dashersw/cote-workshop/pull/5#issuecomment-482807015
Right now the easiest way to get this working in either docker-swarm or k8s is to use the redis service discovery (with DISCOVERY_HOSTNAME env param).
I can update the example workshop to work k8s with redis, but will only get to that next week.

Thank you! With redis all start working!

I set up a minimal kubernetes cluster with weave network. discovery seems to be working but the actual messages don't get through. what could be the problem? is there some way to turn on more verbose logging?

@orgalaf  this should help
https://github.com/dashersw/cote#controlling-cote-with-environment-variables
COTE_LOG=true  as an env var.
I had a similar issue it was trying to use BROADCAST or ANYCAST instead of COTE_MULTICAST_ADDRESS.
@orgalaf what is your kubernetes cluster setup? How did you set it up, what is it running on?

Hi, so after more reading through this issue I got it to work on a local
cluster with weave + DISCOVERY_HOSTNAME as detailed above +
COTE_USE_HOSTNAMES true. Did not need to switch to multicast or use  redis.
Today will try it on a cluster in aws.
On Wed, 17 Apr 2019 at 21:10 David Rubin<EMAIL_ADDRESS>wrote:

@orgalaf https://github.com/orgalaf this should help
https://github.com/dashersw/cote#controlling-cote-with-environment-variables
COTE_LOG=true as an env var.
I had a similar issue it was trying to use BROADCAST or ANYCAST instead of
COTE_MULTICAST_ADDRESS.
@orgalaf https://github.com/orgalaf what is your kubernetes cluster
setup? How did you set it up, what is it running on?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dashersw/cote/issues/66#issuecomment-484201693, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AK76OY7MPG3L3UF6GB2ZLZ3PQ5RY3ANCNFSM4D2WWNOQ
.


🚧 add WSL 2 Named pipe strategy

Adding WSL 2 support
resolves #1766
Currently seeing this issue
20:41:33.476 DEBUG org.testcontainers.utility.TestcontainersConfiguration - Testcontainers configuration overrides will be loaded from file:/C:/Users/josep/.testcontainers.properties
20:41:33.482 DEBUG org.testcontainers.utility.TestcontainersConfiguration - Testcontainers configuration overrides loaded from TestcontainersConfiguration(properties={docker.client.strategy=org.testcontainers.dockerclient.WslNpipeSocketClientProviderStrategy})
20:41:33.516 INFO  org.testcontainers.dockerclient.DockerClientProviderStrategy - Loaded org.testcontainers.dockerclient.WslNpipeSocketClientProviderStrategy from ~/.testcontainers.properties, will try it first
20:41:33.574 INFO  org.testcontainers.dockerclient.DockerClientProviderStrategy - Will use 'okhttp' transport
20:41:33.749 DEBUG org.testcontainers.dockerclient.DockerClientProviderStrategy - Pinging docker daemon...
20:41:34.041 INFO  org.testcontainers.dockerclient.WslNpipeSocketClientProviderStrategy - Accessing docker with local WSL Npipe socket (npipe:////./pipe/docker_wsl)
20:41:34.041 INFO  org.testcontainers.dockerclient.DockerClientProviderStrategy - Found Docker environment with local WSL Npipe socket (npipe:////./pipe/docker_wsl)
20:41:34.041 DEBUG org.testcontainers.dockerclient.DockerClientProviderStrategy - Checking Docker OS type for local WSL Npipe socket (npipe:////./pipe/docker_wsl)
20:41:34.375 INFO  org.testcontainers.DockerClientFactory - Docker host IP address is localhost
20:41:34.673 INFO  org.testcontainers.DockerClientFactory - Connected to docker:
  Server Version: 19.03.1
  API Version: 1.40
  Operating System: Ubuntu 18.04.2 LTS
  Total Memory: 31040 MB
20:41:34.725 DEBUG org.testcontainers.utility.RegistryAuthLocator - Looking up auth config for image: quay.io/testcontainers/ryuk:0.2.3
20:41:34.726 DEBUG org.testcontainers.utility.RegistryAuthLocator - RegistryAuthLocator has configFile: C:\Users\josep\.docker\config.json (exists) and commandPathPrefix:
20:41:34.732 DEBUG org.testcontainers.utility.RegistryAuthLocator - registryName [quay.io] for dockerImageName [quay.io/testcontainers/ryuk:0.2.3]
20:41:34.732 DEBUG org.testcontainers.utility.RegistryAuthLocator - Executing docker credential provider: docker-credential-desktop to locate auth config for: quay.io
20:41:34.915 DEBUG org.testcontainers.utility.RegistryAuthLocator - Got credentials not found error message from docker credential helper - credentials not found in native keychain
20:41:34.915 INFO  org.testcontainers.utility.RegistryAuthLocator - Credential helper/store (docker-credential-desktop) does not have credentials for quay.io
20:41:34.916 DEBUG org.testcontainers.utility.RegistryAuthLocator - no matching Auth Configs - falling back to defaultAuthConfig [null]
20:41:34.916 DEBUG org.testcontainers.dockerclient.auth.AuthDelegatingDockerClientConfig - Effective auth config [null]

java.lang.ExceptionInInitializerError
        Caused by:
        com.github.dockerjava.api.exception.InternalServerErrorException: rpc error: code = Unknown desc = wslpath failed; stderr is "wslpath: //var/run/docker.sock:/var/run/docker.sock\n": exit status 1
            at org.testcontainers.dockerclient.transport.okhttp.OkHttpInvocationBuilder.execute(OkHttpInvocationBuilder.java:276)
            at org.testcontainers.dockerclient.transport.okhttp.OkHttpInvocationBuilder.execute(OkHttpInvocationBuilder.java:254)
            at org.testcontainers.dockerclient.transport.okhttp.OkHttpInvocationBuilder.post(OkHttpInvocationBuilder.java:125)
            at com.github.dockerjava.core.exec.CreateContainerCmdExec.execute(CreateContainerCmdExec.java:33)
            at com.github.dockerjava.core.exec.CreateContainerCmdExec.execute(CreateContainerCmdExec.java:13)
            at com.github.dockerjava.core.exec.AbstrSyncDockerCmdExec.exec(AbstrSyncDockerCmdExec.java:21)
            at com.github.dockerjava.core.command.AbstrDockerCmd.exec(AbstrDockerCmd.java:35)
            at com.github.dockerjava.core.command.CreateContainerCmdImpl.exec(CreateContainerCmdImpl.java:1139)
            at org.testcontainers.utility.ResourceReaper.start(ResourceReaper.java:82)
            at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:131)
            at org.testcontainers.containers.GenericContainer.<init>(GenericContainer.java:175)
            at org.testcontainers.vault.VaultContainer.<init>(VaultContainer.java:38)
            at org.testcontainers.vault.VaultContainer.<init>(VaultContainer.java:34)
            at org.testcontainers.vault.VaultContainerTest.<clinit>(VaultContainerTest.java:26)

Not experienced enough the inner workings of either test containers nor docker as to why this is an issue.

Hi @casz,
Any progress?

Nah, hopefully with some of the recent WSL improvements I might give it another go 😄

See https://github.com/testcontainers/testcontainers-java/issues/1766
Basically works as intended as named pipe is back to being called //./pipe/docker_engine


Support arm64

Summary
M1 Mac development now possible
Purpose

Make QuickMPC multi-arch supported

Contents

Use TARGETPLATFORM for multi-arch build
Use couchbase 7.1.1
Use bazelisk as bazel launcher
Update several packages, tools for multi-arch build

Testing Methods Performed

CI https://github.com/arukuka/QuickMPC/commit/143c62a8e705827f38bd56a8aecde17e83898c48
local medium CC test on M1 Mac
checked that an binary can be launched on dep-runner image

I checked CI status.  I will check details later.

steps containing bazel are failed
steps pushing image to registry are failed because it seems to be not allowed

@acompany-develop/coreengine
I cannot run CI expected because protecting safety.
After checking the changelist, could you please create a PR to check the CI status?

I cannot run CI expected because protecting safety.
After checking the changelist, could you please create a PR to check the CI status?

@arukuka
Sorry, I didn't understand... Why is this? What does it mean?

@mdonaka

Is there any way to check the operation on a non-M1 machine?

I have to add test run on arm64.
I will add it, so I make it draft.
However, GitHub Actions cannot run test on arm64 by default.
We need to use self-hosted runner, or use other CI service.

Will future maintenance be the responsibility of the person with the M1 machine?

I think so.

Sorry, I didn't understand... Why is this? What does it mean?

I opened this PR from forked repository because

I wrote this using hobby time, so I think that using fork is better way, and
I'd like to check that a flow run correctly when PR is opened from forked repository.

Also, PR from forked repositry cannot access secrets used in GitHub Actions.

doc: https://docs.github.com/ja/actions/security-guides/security-hardening-for-github-actions#accessing-secrets
actual behavior: https://github.com/acompany-develop/QuickMPC/runs/7934124110#step:3:7

I apologize for the inconvenience.

@arukuka
It would be ideal to have a test, but If the person who has the M1 machine can maintain it responsibly, I think this PR is good.
I think the arm64 test addition can be a separate PR.

I opened this PR from forked repository

I understood.

@mdonaka
Thank you.  I will add description about this support as "experimental" on README.
After its commit, I will make this PR open.


teleconsole from android

is there a way to access a session from android? More specifically I would like to access a web page on the machine that started the session.
thank you.

No, this is not supported.


[GSB] Basic infrastructure for delaying and reprocessing requirements

Start staging in support for delaying the processing of a requirement in the GenericSignatureBuilder when one of the involved types cannot be immediately resolved. In such cases, stash the requirement to be resolved later. Most of this infrastructure is unused at this point, but is important for the detection of recursion in constraint systems.
While here... fix an issue where the GenericSignatureBuilder was placing conformance requirements on the representative of the equivalence class rather than on the potential archetype that was named, which breaks some invariants. That in turn uncovered a latent idempotency problem in the GSB, where we wouldn't necessarily have the appropriate anchor within a component within the implied same-type constraint graphs.

@swift-ci please smoke test and merge

Foundation failure looks like a spurious networking issue.

@swift-ci please smoke test Linux

I mis-clicked the request for review button. Sorry : (

The Linux failure will be addressed by https://github.com/apple/swift-corelibs-foundation/pull/945

@swift-ci please smoke test Linux

@slavapestov RequirementResult would be better, probably. Will update


⚠ File access (beta) has degraded performance

In eef409d, File access (beta) (https://beta.fyle.uk/file/fish) experienced degraded performance:

HTTP code: 200
Response time: 3376 ms

Resolved: File access (beta) performance has improved in 0991966.


Unable to access dict Attributes in train/data/base.py and train/model/base.py

In aframev2/projects/train/train/data/base.py and aframev2/train/train/model/base.py instances of self.hparams.{attribute} caused AttributeErrors with errors similar to,
AttributeError: 'AttributeDict' object has no attribute 'data_dir'

It started to work by changing all instances of self.hparams.{attribute} to self.hparams.init_args["{attribute}"]

Hey @sjhend03 encountered this too. Turns out it was a bug with pytorch lightning. (see here).
This should already be fixed in the train project by constraining lightning to <2.3.0.
So, try re-building your train container and try again.

@sjhend03 going to close this but feel free to reopen if issue persists for whatever reason


Async method naming

There's a bit of inconsistency in the project with its use of asynchronous naming conventions:
Sometimes a method follows the Microsoft naming guidelines, including an -Async suffix (e.g. OpenAsync) and other times the suffix is not present (e.g. IHtmlFormElement.Submit()).
What is the preferred coding style and should there be a cleanup to make the project conform?

I think its pretty much coherent. Yes, Submit should most probably be renamed, however, this has a historic reason as at first methods (of DOM interfaces / classes) have been named exactly (ignore case) as they should have been. Now this is no longer the case (thanks to attributes), we may give methods like Submit their proper name.
Can you name more examples where the suffix is missing? Just from the top of my head I would say its fairly consistent.
Two more things to mention here:

The suffix only makes sense if its a hot task.
It was mostly chosen to not conflict with existing methods that have the same parameters, but do not return a (hot) task (and may be blocking).

So in a world with overload resolution that takes into account the sink we would probably not have seen it.

Whether the Task is "hot" or not is an implementation detail that should be hidden to the caller. There are many scenarios in asynchronous programming where the call can be asynchronous or synchronous depending on the input parameters or whether values are being cached, so it's best to just treat all Task instances equally.
The following methods are implemented to return a Task but don't have an Async suffix:
Dom\Css\CssStyleEngine.cs
Line 162:         async Task<CssStyleSheet> Parse(CssParser parser, CssStyleSheet sheet, TextSource source)
Dom\Css\Rules\CssImportRule.cs
Line 59:         internal async Task LoadStylesheetFrom(Document document)
Dom\Document.cs
Line 1610:         internal void DelayLoad(Task task)
Line 1627:         internal async Task FinishLoading()
Dom\Html\HtmlFormElement.cs
Line 174:         public Task<IDocument> Submit()
Line 183:         public Task<IDocument> Submit(IHtmlElement sourceElement)
Dom\Html\HtmlFrameElementBase.cs
Line 167:             public async Task Perform(Action<IDocument> callback)
Dom\Window.cs
Line 679:         async Task DoTimeout(Action<IWindow> callback, Int32 timeout, CancellationToken token)
Line 689:         async Task DoInterval(Action<IWindow> callback, Int32 timeout, CancellationToken token)
\Extensions\ApiExtensions.cs
Line 106:         public static async Task<Event> AwaitEvent<TEventTarget>(this TEventTarget node, String eventName)
Line 302:         public static Task<IDocument> Navigate<TElement>(this TElement element)
Line 315:         public static Task<IDocument> Navigate<TElement>(this TElement element, CancellationToken cancel)
Line 335:         public static Task<IDocument> Submit(this IHtmlFormElement form, Object fields)
Line 347:         public static Task<IDocument> Submit(this IHtmlFormElement form, IDictionary<String, String> fields)
Extensions\DocumentExtensions.cs
Line 286:         public static async Task WaitForReady(this Document document)
Extensions\ElementExtensions.cs
Line 811:         public static Task<Boolean> ProcessResource<TResource>(this Element element, IDownload download, Action<TResource> callback)
Line 842:         public static async Task<Boolean> ProcessResponse(this Element element, IDownload download, Action<IResponse> callback)
Line 921:         public static async Task<IDocument> NavigateTo(this Element element, DocumentRequest request)
Extensions\PortableExtensions.cs
Line 33:         public static Task Delay(this CancellationToken token, Int32 timeout)
Interfaces\Html\IHtmlFormElement.cs
Line 84:         Task<IDocument> Submit();
Line 90:         Task<IDocument> Submit(IHtmlElement sourceElement);
Parser\Html\HtmlDomBuilder.cs
Line 3439:         async Task RunScript(HtmlScriptElement script)
TextSource.cs
Line 276:         public Task Prefetch(Int32 length, CancellationToken cancellationToken)
Line 286:         public async Task PrefetchAll(CancellationToken cancellationToken)
I didn't scrutinize the test suites.

Whether the Task is "hot" or not is an implementation detail that should be hidden to the caller.

Actually, no. This is crucial, as a hot task is not started and still needs to be triggered. You don't call Start on tasks returned from Async methods, right? Actually, this is where this convention really shines. But I think it does not matter for the discussion.
Most things from the list above are actually internal and pretty clear from the context (there is also no synchronous counter-part).
I would not put to much weight on this. After all, this is cosmetics and there are much more important issues on the list.

Sorry, I didn't appreciate there were places where Task.Start was being invoked. I don't believe there's any reason to use Task.Start. I would strongly advocate refactoring to remove this pattern.
From a calling perspective, I expect any Task given to me to be something encapsulating a piece of work which was set up by the method call. Either the task is scheduled, is already running, or has already completed. The key thing is that the task should be created and scheduled before it returns to the caller.
Stephen Cleary has a quick read on the topic of starting tasks which I feel is relevant:
http://blog.stephencleary.com/2015/02/a-tour-of-task-part-8-starting.html

Clearly we are discussing different things.

Implemented, see PR #301.