Support service which should use: subscription_bound=False

https://github.com/Azure/azure-cli-extensions/blob/master/src/resource-graph/azext_resourcegraph/_client_factory.py#L10

duplicate of #222


Training stopping because of  BufferError: Existing exports of data: object cannot be re-sized or something wrong with tornado

Click to expand!
Issue Type
Bug
Have you reproduced the bug with TF nightly?
No
Source
source
Tensorflow Version
2.12.0
Custom Code
Yes
OS Platform and Distribution
NAME="CentOS Linux" VERSION="7 (Core)"
Mobile device
NAME="CentOS Linux" VERSION="7 (Core)"
Python version
3.9.16
Bazel version
No response
GCC/Compiler version
No response
CUDA/cuDNN version
11.8.0
GPU model and memory
No response
Current Behaviour?
The model training would just stop abruptly
https://colab.research.google.com/drive/1WiqyF7dCdnNBIANEY80Pxw_mVz4fyV-S?usp=sharing
Standalone code to reproduce the issue
Voxelmoprh library training

Relevant log output
(tf) vr-lab@pop-os:~$ jupyter notebook

  _   _          _      _
 | | | |_ __  __| |__ _| |_ ___
 | |_| | '_ \/ _` / _` |  _/ -_)
  \___/| .__/\__,_\__,_|\__\___|
       |_|

Read the migration plan to Notebook 7 to learn about the new features and the actions to take if you are using extensions.

https://jupyter-notebook.readthedocs.io/en/latest/migrate_to_notebook7.html

Please note that updating to Notebook 7 might break some of your extensions.

[I 00:02:49.290 NotebookApp] Serving notebooks from local directory: /home/vr-lab
[I 00:02:49.290 NotebookApp] Jupyter Notebook 6.5.4 is running at:
[I 00:02:49.290 NotebookApp] http://localhost:8888/?token=697572ae046e4388d22c7be946cefcb261064994d2f99466
[I 00:02:49.290 NotebookApp]  or http://<IP_ADDRESS>:8888/?token=697572ae046e4388d22c7be946cefcb261064994d2f99466
[I 00:02:49.290 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 00:02:49.334 NotebookApp]

    To access the notebook, open this file in a browser:
        file:///home/vr-lab/.local/share/jupyter/runtime/nbserver-405435-open.html
    Or copy and paste one of these URLs:
        http://localhost:8888/?token=697572ae046e4388d22c7be946cefcb261064994d2f99466
     or http://<IP_ADDRESS>:8888/?token=697572ae046e4388d22c7be946cefcb261064994d2f99466
[I 00:03:15.170 NotebookApp] Kernel started: 4915aa8a-d4aa-4d50-885f-810d53eae7db, name: python3
[I 00:03:20.670 NotebookApp] Kernel restarted: 4915aa8a-d4aa-4d50-885f-810d53eae7db
[W 00:03:20.684 NotebookApp] Replacing stale connection: 4915aa8a-d4aa-4d50-885f-810d53eae7db:e6146c4b818f471185049a02ac632f6d
[W 00:03:21.180 NotebookApp] zmq message arrived on closed channel
[I 00:03:21.181 NotebookApp] Starting buffering for 4915aa8a-d4aa-4d50-885f-810d53eae7db:e6146c4b818f471185049a02ac632f6d
[I 00:03:21.183 NotebookApp] Restoring connection for 4915aa8a-d4aa-4d50-885f-810d53eae7db:e6146c4b818f471185049a02ac632f6d
[I 00:03:21.689 NotebookApp] Replaying 1 buffered messages
[E 00:03:21.761 NotebookApp] Uncaught exception, closing connection.
    Traceback (most recent call last):
      File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/iostream.py", line 702, in _handle_events
        self._handle_write()
      File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/iostream.py", line 976, in _handle_write
        self._write_buffer.advance(num_bytes)
      File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/iostream.py", line 182, in advance
        assert 0 < size <= self._size
    AssertionError
[W 00:03:21.764 NotebookApp] Write error on <socket.socket [closed] fd=-1, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6>: [Errno 9] Bad file descriptor
[W 00:03:21.766 NotebookApp] zmq message arrived on closed channel
[W 00:03:21.767 NotebookApp] zmq message arrived on closed channel
Exception in callback None()
handle: <Handle cancelled>
Traceback (most recent call last):
  File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/platform/asyncio.py", line 206, in _handle_events
    handler_func(fileobj, events)
  File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/iostream.py", line 702, in _handle_events
    self._handle_write()
  File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/iostream.py", line 976, in _handle_write
    self._write_buffer.advance(num_bytes)
  File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/iostream.py", line 182, in advance
    assert 0 < size <= self._size
AssertionError
[I 00:03:21.768 NotebookApp] Starting buffering for 4915aa8a-d4aa-4d50-885f-810d53eae7db:e6146c4b818f471185049a02ac632f6d
2023-04-11 00:03:22.084618: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F AVX512_VNNI FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-04-11 00:03:22.225493: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
[I 00:03:22.803 NotebookApp] Restoring connection for 4915aa8a-d4aa-4d50-885f-810d53eae7db:e6146c4b818f471185049a02ac632f6d
[I 00:03:22.803 NotebookApp] Replaying 1 buffered messages
2023-04-11 00:03:22.815590: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: :/home/vr-lab/anaconda3/envs/tf/lib/:/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/nvidia/cudnn/lib
2023-04-11 00:03:22.815709: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: :/home/vr-lab/anaconda3/envs/tf/lib/:/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/nvidia/cudnn/lib
2023-04-11 00:03:22.815716: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2023-04-11 00:03:25.015062: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1613] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 12776 MB memory:  -> device: 0, name: NVIDIA RTX A4000, pci bus id: 0000:af:00.0, compute capability: 8.6
2023-04-11 00:03:40.078576: I tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:428] Loaded cuDNN version 8600
[I 00:05:15.159 NotebookApp] Saving file at /Music/HybridMorph Please don't delete/HybridMorph_proof of concept.ipynb
Task exception was never retrieved
future: <Task finished name='Task-76' coro=<WebSocketProtocol13.write_message.<locals>.wrapper() done, defined at /home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/websocket.py:1090> exception=WebSocketClosedError()>
Traceback (most recent call last):
  File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/websocket.py", line 1092, in wrapper
    await fut
tornado.iostream.StreamClosedError: Stream is closed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/asyncio/tasks.py", line 256, in __step
    result = coro.send(None)
  File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/websocket.py", line 1094, in wrapper
    raise WebSocketClosedError()
tornado.websocket.WebSocketClosedError
[E 01:03:52.904 NotebookApp] Exception in callback <bound method WebSocketMixin.send_ping of ZMQChannelsHandler(4915aa8a-d4aa-4d50-885f-810d53eae7db)>
    Traceback (most recent call last):
      File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/ioloop.py", line 921, in _run
        val = self.callback()
      File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/notebook/base/zmqhandlers.py", line 188, in send_ping
        self.ping(b'')
      File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/websocket.py", line 445, in ping
        self.ws_connection.write_ping(data)
      File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/websocket.py", line 1101, in write_ping
        self._write_frame(True, 0x9, data)
      File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/websocket.py", line 1061, in _write_frame
        return self.stream.write(frame)
      File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/iostream.py", line 540, in write
        self._write_buffer.append(data)
      File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/iostream.py", line 157, in append
        b += data  # type: ignore
    BufferError: Existing exports of data: object cannot be re-sized
[E 01:13:22.812 NotebookApp] Uncaught exception in ZMQStream callback
    Traceback (most recent call last):
      File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/zmq/eventloop/zmqstream.py", line 584, in _run_callback
        f = callback(*args, **kwargs)
      File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/zmq/eventloop/zmqstream.py", line 308, in stream_callback
        return callback(self, msg)
      File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/notebook/services/kernels/handlers.py", line 572, in _on_zmq_reply
        super()._on_zmq_reply(stream, msg)
      File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/notebook/base/zmqhandlers.py", line 256, in _on_zmq_reply
        self.write_message(msg, binary=isinstance(msg, bytes))
      File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/websocket.py", line 339, in write_message
        return self.ws_connection.write_message(message, binary=binary)
      File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/websocket.py", line 1086, in write_message
        fut = self._write_frame(True, opcode, message, flags=flags)
      File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/websocket.py", line 1061, in _write_frame
        return self.stream.write(frame)
      File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/iostream.py", line 540, in write
        self._write_buffer.append(data)
      File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/iostream.py", line 157, in append
        b += data  # type: ignore
    BufferError: Existing exports of data: object cannot be re-sized
[E 01:13:22.815 NotebookApp] Uncaught exception in zmqstream callback
    Traceback (most recent call last):
      File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/zmq/eventloop/zmqstream.py", line 634, in _handle_events
        self._handle_recv()
      File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/zmq/eventloop/zmqstream.py", line 663, in _handle_recv
        self._run_callback(callback, msg)
      File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/zmq/eventloop/zmqstream.py", line 584, in _run_callback
        f = callback(*args, **kwargs)
      File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/zmq/eventloop/zmqstream.py", line 308, in stream_callback
        return callback(self, msg)
      File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/notebook/services/kernels/handlers.py", line 572, in _on_zmq_reply
        super()._on_zmq_reply(stream, msg)
      File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/notebook/base/zmqhandlers.py", line 256, in _on_zmq_reply
        self.write_message(msg, binary=isinstance(msg, bytes))
      File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/websocket.py", line 339, in write_message
        return self.ws_connection.write_message(message, binary=binary)
      File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/websocket.py", line 1086, in write_message
        fut = self._write_frame(True, opcode, message, flags=flags)
      File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/websocket.py", line 1061, in _write_frame
        return self.stream.write(frame)
      File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/iostream.py", line 540, in write
        self._write_buffer.append(data)
      File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/iostream.py", line 157, in append
        b += data  # type: ignore
    BufferError: Existing exports of data: object cannot be re-sized
[E 01:13:22.815 NotebookApp] Exception in callback functools.partial(<function ZMQStream._update_handler.<locals>.<lambda> at 0x7f1de4ff4b80>)
    Traceback (most recent call last):
      File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/ioloop.py", line 740, in _run_callback
        ret = callback()
      File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/zmq/eventloop/zmqstream.py", line 718, in <lambda>
        self.io_loop.add_callback(lambda: self._handle_events(self.socket, 0))
      File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/zmq/eventloop/zmqstream.py", line 634, in _handle_events
        self._handle_recv()
      File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/zmq/eventloop/zmqstream.py", line 663, in _handle_recv
        self._run_callback(callback, msg)
      File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/zmq/eventloop/zmqstream.py", line 584, in _run_callback
        f = callback(*args, **kwargs)
      File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/zmq/eventloop/zmqstream.py", line 308, in stream_callback
        return callback(self, msg)
      File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/notebook/services/kernels/handlers.py", line 572, in _on_zmq_reply
        super()._on_zmq_reply(stream, msg)
      File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/notebook/base/zmqhandlers.py", line 256, in _on_zmq_reply
        self.write_message(msg, binary=isinstance(msg, bytes))
      File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/websocket.py", line 339, in write_message
        return self.ws_connection.write_message(message, binary=binary)
      File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/websocket.py", line 1086, in write_message
        fut = self._write_frame(True, opcode, message, flags=flags)
      File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/websocket.py", line 1061, in _write_frame
        return self.stream.write(frame)
      File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/iostream.py", line 540, in write
        self._write_buffer.append(data)
      File "/home/vr-lab/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/iostream.py", line 157, in append
        b += data  # type: ignore
    BufferError: Existing exports of data: object cannot be re-sized

@CaffineAddic,
I tried to execute the mentioned code with the tensorflow v2.12 and it was executed without any issues. Kindly find the gist of it here and also looks like the error which you stated was related to tensorflow.
The error which was mentioned was discussed in Tornado issue 2008. The current theory is that it only happens when threads are being used incorrectly, but this is not certain.
Reference. Thank you!

Thanks for the reply, actually the code works well for 150 epochs with 100 steps per epoch but it stops with this error at any range of epochs from 70 to 200.

@CaffineAddic,
While I was accessing the issue, I was unable to view any code in the above link. Could you please provide the colab gist which helps to analyse the issue in an effective way. Thank you!

https://github.com/CaffineAddic/HybridMorph-proof-of-concept-/blob/main/HybridMorph_proof of concept.ipynb

I am running it on my local machine

Could you please provide the model_loc = 'Models/ and csv_loc = 'CSV/' datasets and the models which you are trying to execute the code in the reproducible format. Thank you!

There is no data-set needed to run this
One of the data-set is provided by the library other one is generated during the execution.
Those two location are just to store the model weights as .h5 file and the other has the location of the CSV file where it will store errors per step.
Just keep both as random temp folder.
Thank you

Anyone got any fix??

@CaffineAddic,
Apologies for the delay. We are working on the issue and will update the status here. Thank you!

Thanks a lot for the reply, Good luck.

(also seeing this issue)

@CaffineAddic @akellehe , Could you please try to provide more information on this to debug the root cause of the issue.
Also, are you facing the similar behavior in other environments as well?

I am getting the same error and stacktrace with a pytorch model with MPS backend from a jupyter notebook. The model continues training, but output stops streaming to jupyter.
I suspect the problem is actually with jupyter and that websocket that allows streaming data from the python backend to the output cell.

Did you find any fix for it ??

Same here. The error just random shows up. Some times after 100 epochs, sometimes at much earlier.
2023-07-26 17:43:54 [E 00:43:54.737 NotebookApp] Uncaught exception in zmqstream callback
2023-07-26 17:43:54     Traceback (most recent call last):
2023-07-26 17:43:54       File "/usr/local/lib/python3.8/dist-packages/zmq/eventloop/zmqstream.py", line 634, in _handle_events
2023-07-26 17:43:54         self._handle_recv()
2023-07-26 17:43:54       File "/usr/local/lib/python3.8/dist-packages/zmq/eventloop/zmqstream.py", line 663, in _handle_recv
2023-07-26 17:43:54         self._run_callback(callback, msg)
2023-07-26 17:43:54       File "/usr/local/lib/python3.8/dist-packages/zmq/eventloop/zmqstream.py", line 584, in _run_callback
2023-07-26 17:43:54         f = callback(*args, **kwargs)
2023-07-26 17:43:54       File "/usr/local/lib/python3.8/dist-packages/zmq/eventloop/zmqstream.py", line 308, in stream_callback
2023-07-26 17:43:54         return callback(self, msg)
2023-07-26 17:43:54       File "/usr/local/lib/python3.8/dist-packages/notebook/services/kernels/handlers.py", line 572, in _on_zmq_reply
2023-07-26 17:43:54         super()._on_zmq_reply(stream, msg)
2023-07-26 17:43:54       File "/usr/local/lib/python3.8/dist-packages/notebook/base/zmqhandlers.py", line 256, in _on_zmq_reply
2023-07-26 17:43:54         self.write_message(msg, binary=isinstance(msg, bytes))
2023-07-26 17:43:54       File "/usr/local/lib/python3.8/dist-packages/tornado/websocket.py", line 339, in write_message
2023-07-26 17:43:54         return self.ws_connection.write_message(message, binary=binary)
2023-07-26 17:43:54       File "/usr/local/lib/python3.8/dist-packages/tornado/websocket.py", line 1086, in write_message
2023-07-26 17:43:54         fut = self._write_frame(True, opcode, message, flags=flags)
2023-07-26 17:43:54       File "/usr/local/lib/python3.8/dist-packages/tornado/websocket.py", line 1061, in _write_frame
2023-07-26 17:43:54         return self.stream.write(frame)
2023-07-26 17:43:54       File "/usr/local/lib/python3.8/dist-packages/tornado/iostream.py", line 540, in write
2023-07-26 17:43:54         self._write_buffer.append(data)
2023-07-26 17:43:54       File "/usr/local/lib/python3.8/dist-packages/tornado/iostream.py", line 157, in append
2023-07-26 17:43:54         b += data  # type: ignore
2023-07-26 17:43:54     BufferError: Existing exports of data: object cannot be re-sized
2023-07-26 17:43:54 Exception in callback BaseAsyncIOLoop._handle_events(28, 1)
2023-07-26 17:43:54 handle: <Handle BaseAsyncIOLoop._handle_events(28, 1)>
2023-07-26 17:43:54 Traceback (most recent call last):
2023-07-26 17:43:54   File "/usr/lib/python3.8/asyncio/events.py", line 81, in _run
2023-07-26 17:43:54     self._context.run(self._callback, *self._args)
2023-07-26 17:43:54   File "/usr/local/lib/python3.8/dist-packages/tornado/platform/asyncio.py", line 206, in _handle_events
2023-07-26 17:43:54     handler_func(fileobj, events)
2023-07-26 17:43:54   File "/usr/local/lib/python3.8/dist-packages/zmq/eventloop/zmqstream.py", line 634, in _handle_events
2023-07-26 17:43:54     self._handle_recv()
2023-07-26 17:43:54   File "/usr/local/lib/python3.8/dist-packages/zmq/eventloop/zmqstream.py", line 663, in _handle_recv
2023-07-26 17:43:54     self._run_callback(callback, msg)
2023-07-26 17:43:54   File "/usr/local/lib/python3.8/dist-packages/zmq/eventloop/zmqstream.py", line 584, in _run_callback
2023-07-26 17:43:54     f = callback(*args, **kwargs)
2023-07-26 17:43:54   File "/usr/local/lib/python3.8/dist-packages/zmq/eventloop/zmqstream.py", line 308, in stream_callback
2023-07-26 17:43:54     return callback(self, msg)
2023-07-26 17:43:54   File "/usr/local/lib/python3.8/dist-packages/notebook/services/kernels/handlers.py", line 572, in _on_zmq_reply
2023-07-26 17:43:54     super()._on_zmq_reply(stream, msg)
2023-07-26 17:43:54   File "/usr/local/lib/python3.8/dist-packages/notebook/base/zmqhandlers.py", line 256, in _on_zmq_reply
2023-07-26 17:43:54     self.write_message(msg, binary=isinstance(msg, bytes))
2023-07-26 17:43:54   File "/usr/local/lib/python3.8/dist-packages/tornado/websocket.py", line 339, in write_message
2023-07-26 17:43:54     return self.ws_connection.write_message(message, binary=binary)
2023-07-26 17:43:54   File "/usr/local/lib/python3.8/dist-packages/tornado/websocket.py", line 1086, in write_message
2023-07-26 17:43:54     fut = self._write_frame(True, opcode, message, flags=flags)
2023-07-26 17:43:54   File "/usr/local/lib/python3.8/dist-packages/tornado/websocket.py", line 1061, in _write_frame
2023-07-26 17:43:54     return self.stream.write(frame)
2023-07-26 17:43:54   File "/usr/local/lib/python3.8/dist-packages/tornado/iostream.py", line 540, in write
2023-07-26 17:43:54     self._write_buffer.append(data)
2023-07-26 17:43:54   File "/usr/local/lib/python3.8/dist-packages/tornado/iostream.py", line 157, in append
2023-07-26 17:43:54     b += data  # type: ignore
2023-07-26 17:43:54 BufferError: Existing exports of data: object cannot be re-sized

I was having this problem training the keras models.
Some random times i was getting this error and the training just stopped.
Seems like the problem is on streaming the large output to jupyter.
So, turning off the verbose of fit() method worked for me.
i.e:  model.fit(trainX, trainY, ... , verbose=0)
I guess you can also use verbose=2 for showing just the final details for each epoch and it will work fine.

I will test it thank you @tgoMota

Uncaught exception in ZMQStream callback
Traceback (most recent call last):
File "/home/saumya/anaconda3/envs/tf/lib/python3.9/site-packages/zmq/event                                                                                loop/zmqstream.py", line 584, in _run_callback
f = callback(*args, **kwargs)
File "/home/saumya/anaconda3/envs/tf/lib/python3.9/site-packages/zmq/event                                                                                loop/zmqstream.py", line 308, in stream_callback
return callback(self, msg)
File "/home/saumya/anaconda3/envs/tf/lib/python3.9/site-packages/notebook/                                                                                services/kernels/handlers.py", line 572, in _on_zmq_reply
super()._on_zmq_reply(stream, msg)
File "/home/saumya/anaconda3/envs/tf/lib/python3.9/site-packages/notebook/                                                                                base/zmqhandlers.py", line 256, in _on_zmq_reply
self.write_message(msg, binary=isinstance(msg, bytes))
File "/home/saumya/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/w                                                                                ebsocket.py", line 339, in write_message
return self.ws_connection.write_message(message, binary=binary)
File "/home/saumya/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/w                                                                                ebsocket.py", line 1086, in write_message
fut = self._write_frame(True, opcode, message, flags=flags)
File "/home/saumya/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/w                                                                                ebsocket.py", line 1061, in _write_frame
return self.stream.write(frame)
File "/home/saumya/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/i                                                                                ostream.py", line 546, in write
self._handle_write()
File "/home/saumya/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/i                                                                                ostream.py", line 976, in _handle_write
self._write_buffer.advance(num_bytes)
File "/home/saumya/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/i                                                                                ostream.py", line 182, in advance
assert 0 < size <= self._size
AssertionError
[E 16:53:16.352 NotebookApp] Uncaught exception in zmqstream callback
Traceback (most recent call last):
File "/home/saumya/anaconda3/envs/tf/lib/python3.9/site-packages/zmq/event                                                                                loop/zmqstream.py", line 634, in _handle_events
self._handle_recv()
File "/home/saumya/anaconda3/envs/tf/lib/python3.9/site-packages/zmq/event                                                                                loop/zmqstream.py", line 663, in _handle_recv
self._run_callback(callback, msg)
File "/home/saumya/anaconda3/envs/tf/lib/python3.9/site-packages/zmq/event                                                                                loop/zmqstream.py", line 584, in _run_callback
f = callback(*args, **kwargs)
File "/home/saumya/anaconda3/envs/tf/lib/python3.9/site-packages/zmq/event                                                                                loop/zmqstream.py", line 308, in stream_callback
return callback(self, msg)
File "/home/saumya/anaconda3/envs/tf/lib/python3.9/site-packages/notebook/                                                                                services/kernels/handlers.py", line 572, in _on_zmq_reply
super()._on_zmq_reply(stream, msg)
File "/home/saumya/anaconda3/envs/tf/lib/python3.9/site-packages/notebook/                                                                                base/zmqhandlers.py", line 256, in _on_zmq_reply
self.write_message(msg, binary=isinstance(msg, bytes))
File "/home/saumya/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/w                                                                                ebsocket.py", line 339, in write_message
return self.ws_connection.write_message(message, binary=binary)
File "/home/saumya/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/w                                                                                ebsocket.py", line 1086, in write_message
fut = self._write_frame(True, opcode, message, flags=flags)
File "/home/saumya/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/w                                                                                ebsocket.py", line 1061, in _write_frame
return self.stream.write(frame)
File "/home/saumya/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/i                                                                                ostream.py", line 546, in write
self._handle_write()
File "/home/saumya/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/i                                                                                ostream.py", line 976, in _handle_write
self._write_buffer.advance(num_bytes)
File "/home/saumya/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/i                                                                                ostream.py", line 182, in advance
assert 0 < size <= self._size
AssertionError
Exception in callback BaseAsyncIOLoop._handle_events(33, 1)
handle: <Handle BaseAsyncIOLoop._handle_events(33, 1)>
Traceback (most recent call last):
File "/home/saumya/anaconda3/envs/tf/lib/python3.9/asyncio/events.py", line 80                                                                                , in _run
self._context.run(self._callback, *self._args)
File "/home/saumya/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/platf                                                                                orm/asyncio.py", line 206, in _handle_events
handler_func(fileobj, events)
File "/home/saumya/anaconda3/envs/tf/lib/python3.9/site-packages/zmq/eventloop                                                                                /zmqstream.py", line 634, in _handle_events
self._handle_recv()
File "/home/saumya/anaconda3/envs/tf/lib/python3.9/site-packages/zmq/eventloop                                                                                /zmqstream.py", line 663, in _handle_recv
self._run_callback(callback, msg)
File "/home/saumya/anaconda3/envs/tf/lib/python3.9/site-packages/zmq/eventloop                                                                                /zmqstream.py", line 584, in _run_callback
f = callback(*args, **kwargs)
File "/home/saumya/anaconda3/envs/tf/lib/python3.9/site-packages/zmq/eventloop                                                                                /zmqstream.py", line 308, in stream_callback
return callback(self, msg)
File "/home/saumya/anaconda3/envs/tf/lib/python3.9/site-packages/notebook/serv                                                                                ices/kernels/handlers.py", line 572, in _on_zmq_reply
super()._on_zmq_reply(stream, msg)
File "/home/saumya/anaconda3/envs/tf/lib/python3.9/site-packages/notebook/base                                                                                /zmqhandlers.py", line 256, in _on_zmq_reply
self.write_message(msg, binary=isinstance(msg, bytes))
File "/home/saumya/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/webso                                                                                cket.py", line 339, in write_message
return self.ws_connection.write_message(message, binary=binary)
File "/home/saumya/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/webso                                                                                cket.py", line 1086, in write_message
fut = self._write_frame(True, opcode, message, flags=flags)
File "/home/saumya/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/webso                                                                                cket.py", line 1061, in _write_frame
return self.stream.write(frame)
File "/home/saumya/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/iostr                                                                                eam.py", line 546, in write
self._handle_write()
File "/home/saumya/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/iostr                                                                                eam.py", line 976, in _handle_write
self._write_buffer.advance(num_bytes)
File "/home/saumya/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/iostr                                                                                eam.py", line 182, in advance
assert 0 < size <= self._size
AssertionError
Still this error is persistent @tgoMota

I've encountered the same or a similar issue with TF 2.13.
docker run --gpus all  --rm -u $(id -u):$(id -g) -p 8888:8888 -p 6006:6006 -v $PWD/:/tf/david_home   tensorflow/tensorflow:2.13.0-gpu-jupyter
In my case output of the notebook to Firefox has frozen but the training job still seems to be running.
E 12:40:01.689 NotebookApp] Uncaught exception in ZMQStream callback
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/zmq/eventloop/zmqstream.py", line 584, in _run_callback
f = callback(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/zmq/eventloop/zmqstream.py", line 308, in stream_callback
return callback(self, msg)
File "/usr/local/lib/python3.8/dist-packages/notebook/services/kernels/handlers.py", line 572, in _on_zmq_reply
super()._on_zmq_reply(stream, msg)
File "/usr/local/lib/python3.8/dist-packages/notebook/base/zmqhandlers.py", line 256, in _on_zmq_reply
self.write_message(msg, binary=isinstance(msg, bytes))
File "/usr/local/lib/python3.8/dist-packages/tornado/websocket.py", line 334, in write_message
return self.ws_connection.write_message(message, binary=binary)
File "/usr/local/lib/python3.8/dist-packages/tornado/websocket.py", line 1081, in write_message
fut = self._write_frame(True, opcode, message, flags=flags)
File "/usr/local/lib/python3.8/dist-packages/tornado/websocket.py", line 1056, in _write_frame
return self.stream.write(frame)
File "/usr/local/lib/python3.8/dist-packages/tornado/iostream.py", line 533, in write
self._write_buffer.append(data)
File "/usr/local/lib/python3.8/dist-packages/tornado/iostream.py", line 157, in append
b += data  # type: ignore
BufferError: Existing exports of data: object cannot be re-sized
[E 12:40:01.691 NotebookApp] Uncaught exception in zmqstream callback
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/zmq/eventloop/zmqstream.py", line 634, in _handle_events
self._handle_recv()
File "/usr/local/lib/python3.8/dist-packages/zmq/eventloop/zmqstream.py", line 663, in _handle_recv
self._run_callback(callback, msg)
File "/usr/local/lib/python3.8/dist-packages/zmq/eventloop/zmqstream.py", line 584, in _run_callback
f = callback(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/zmq/eventloop/zmqstream.py", line 308, in stream_callback
return callback(self, msg)
File "/usr/local/lib/python3.8/dist-packages/notebook/services/kernels/handlers.py", line 572, in _on_zmq_reply
super()._on_zmq_reply(stream, msg)
File "/usr/local/lib/python3.8/dist-packages/notebook/base/zmqhandlers.py", line 256, in _on_zmq_reply
self.write_message(msg, binary=isinstance(msg, bytes))
File "/usr/local/lib/python3.8/dist-packages/tornado/websocket.py", line 334, in write_message
return self.ws_connection.write_message(message, binary=binary)
File "/usr/local/lib/python3.8/dist-packages/tornado/websocket.py", line 1081, in write_message
fut = self._write_frame(True, opcode, message, flags=flags)
File "/usr/local/lib/python3.8/dist-packages/tornado/websocket.py", line 1056, in _write_frame
return self.stream.write(frame)
File "/usr/local/lib/python3.8/dist-packages/tornado/iostream.py", line 533, in write
self._write_buffer.append(data)
File "/usr/local/lib/python3.8/dist-packages/tornado/iostream.py", line 157, in append
b += data  # type: ignore
BufferError: Existing exports of data: object cannot be re-sized
ERROR:asyncio:Exception in callback BaseAsyncIOLoop._handle_events(29, 1)
handle: <Handle BaseAsyncIOLoop._handle_events(29, 1)>
Traceback (most recent call last):
File "/usr/lib/python3.8/asyncio/events.py", line 81, in _run
self._context.run(self._callback, *self._args)
File "/usr/local/lib/python3.8/dist-packages/tornado/platform/asyncio.py", line 192, in _handle_events
handler_func(fileobj, events)
File "/usr/local/lib/python3.8/dist-packages/zmq/eventloop/zmqstream.py", line 634, in _handle_events
self._handle_recv()
File "/usr/local/lib/python3.8/dist-packages/zmq/eventloop/zmqstream.py", line 663, in _handle_recv
self._run_callback(callback, msg)
File "/usr/local/lib/python3.8/dist-packages/zmq/eventloop/zmqstream.py", line 584, in _run_callback
f = callback(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/zmq/eventloop/zmqstream.py", line 308, in stream_callback
return callback(self, msg)
File "/usr/local/lib/python3.8/dist-packages/notebook/services/kernels/handlers.py", line 572, in _on_zmq_reply
super()._on_zmq_reply(stream, msg)
File "/usr/local/lib/python3.8/dist-packages/notebook/base/zmqhandlers.py", line 256, in _on_zmq_reply
self.write_message(msg, binary=isinstance(msg, bytes))
File "/usr/local/lib/python3.8/dist-packages/tornado/websocket.py", line 334, in write_message
return self.ws_connection.write_message(message, binary=binary)
File "/usr/local/lib/python3.8/dist-packages/tornado/websocket.py", line 1081, in write_message
fut = self._write_frame(True, opcode, message, flags=flags)
File "/usr/local/lib/python3.8/dist-packages/tornado/websocket.py", line 1056, in _write_frame
return self.stream.write(frame)
File "/usr/local/lib/python3.8/dist-packages/tornado/iostream.py", line 533, in write
self._write_buffer.append(data)
File "/usr/local/lib/python3.8/dist-packages/tornado/iostream.py", line 157, in append
b += data  # type: ignore
BufferError: Existing exports of data: object cannot be re-sized

as @tgoMota mentioned try keeping
verbose=2
This worked for me

Ya essentially allowed me to start training but I cannot comment on the rest of users, also shouldn't training should happen with the default verbose value, tornado errors are still there if you want to I can close this issue, Thank you for your time and support.

Exception in callback <bound method WebSocketMixin.send_ping of ZMQChannelsHandler(bccd1408-a4a4-4810-805f-d10d0d4585df)>
Traceback (most recent call last):
File "/home/saumya/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/ioloop.py", line 921, in _run
val = self.callback()
File "/home/saumya/anaconda3/envs/tf/lib/python3.9/site-packages/notebook/base/zmqhandlers.py", line 188, in send_ping
self.ping(b'')
File "/home/saumya/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/websocket.py", line 445, in ping
self.ws_connection.write_ping(data)
File "/home/saumya/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/websocket.py", line 1101, in write_ping
self._write_frame(True, 0x9, data)
File "/home/saumya/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/websocket.py", line 1061, in _write_frame
return self.stream.write(frame)
File "/home/saumya/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/iostream.py", line 540, in write
self._write_buffer.append(data)
File "/home/saumya/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/iostream.py", line 157, in append
b += data  # type: ignore
BufferError: Existing exports of data: object cannot be re-sized
[I 01:36:07.003 NotebookApp] Saving file at /New_BRaTS/Brain_data/HybridMorph_proof of concept.ipynb
[E 01:36:28.774 NotebookApp] Uncaught exception in ZMQStream callback
Traceback (most recent call last):
File "/home/saumya/anaconda3/envs/tf/lib/python3.9/site-packages/zmq/eventloop/zmqstream.py", line 584, in _run_callback
f = callback(*args, **kwargs)
File "/home/saumya/anaconda3/envs/tf/lib/python3.9/site-packages/zmq/eventloop/zmqstream.py", line 308, in stream_callback
return callback(self, msg)
File "/home/saumya/anaconda3/envs/tf/lib/python3.9/site-packages/notebook/services/kernels/handlers.py", line 572, in _on_zmq_reply
super()._on_zmq_reply(stream, msg)
File "/home/saumya/anaconda3/envs/tf/lib/python3.9/site-packages/notebook/base/zmqhandlers.py", line 256, in _on_zmq_reply
self.write_message(msg, binary=isinstance(msg, bytes))
File "/home/saumya/anaconda3/envs/tf/lib/python3.9/site-packages/tornado/websocket.py", line 339, in write_message
return self.ws_connection.write_message(message, binary=binary
Same error with verbose = 2. Any fixes

Its interesting, I got this error when apply a function on pandas dataframe.

I am getting the same error and stacktrace with a pytorch model with MPS backend from a jupyter notebook. The model continues training, but output stops streaming to jupyter. I suspect the problem is actually with jupyter and that websocket that allows streaming data from the python backend to the output cell.

Faced the same error and tried a lot of troubleshooting steps mentioned here including but not limited to clearing pytorch cuda memory. I agree that the training actually continues because I could hear my GPU wheezing. Ultimately ran the training through a .py script and that worked as expected. I know this isn't technically a solution but I hope it saves someone time. :)

@tgoMota comment about changing verbose to 2 works out well.
After this I plan to migrate to pyTorch


error: failed to get numpy as a dependency of package `tokenizers-python v0.10.3

When I installled package transformers, I met this error. Does anyone met this problem before?
error: failed to get numpy as a dependency of package tokenizers-python v0.10.3 (/tmp/pip-install-ulgbj6di/tokenizers_2c687e1980e640e483b11bc6ace87ebc)
  Caused by:
    failed to load source for dependency `numpy`

  Caused by:
    Unable to update https://github.com/pyo3/rust-numpy/?rev=e331befa27fede78d4662edf08fa0508db39be01

  Caused by:
    failed to fetch into: /home/fmy/.cargo/git/db/rust-numpy-ea13d29ac0b89ab6

  Caused by:
    network failure seems to have happened
    if a proxy or similar is necessary `net.git-fetch-with-cli` may help here
    https://doc.rust-lang.org/cargo/reference/config.html#netgit-fetch-with-cli

  Caused by:
    SSL error: received early EOF; class=Ssl (16); code=Eof (-20)
  error: `cargo rustc --lib --message-format=json-render-diagnostics --manifest-path Cargo.toml --release -v --features pyo3/extension-module --crate-type cdylib --` failed with code 101
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects

Install transformers==4.31.0  rather than 4.15.0 can work.


Security and Privacy: Location of where the AttachmentInput.cs uploads file to

Hi team, this is regarding to this action in Composer.
https://github.com/microsoft/botbuilder-dotnet/blob/main/libraries/Microsoft.Bot.Builder.Dialogs.Adaptive/Input/AttachmentInput.cs
We observed that the attachment is always uploaded to a remote location that not owned by us, from checking the returned content url, which is something like
"contentUrl": "https://webchat.botframework.com/attachments/[CrlLiBoBCjTFPwD9Z1hzoU-us/0000004/0/2020-11-0519-19-07.mp4?t=xxxxxxxxxxxxxxxxxxxxxxxxxxxxx(https://webchat.botframework.com/attachments/CrlLiBoBCjTFPwD9Z1hzoU-us/0000004/0/2020-11-0519-19-07.mp4?t=xxxxxxxxxxxxxxxxxxxxxxxxxxxxx)

But I could not find where to specify a custom storage since we don't expect to upload files to somewhere not owned by us for security and privacy reasons.
Would you please help to understand how the current code works and where the files are actually uploaded?
Any way we could specify a custom storage for it?
Thanks!

This is a duplicate of https://github.com/microsoft/BotFramework-Services/issues/350, raised by the same customer. Case 350 was closed with the comment: 'The attachment is stored in our internal blob storage and the TTL is 24 hours. We don't have a mechanism to use custom storage for this scenario. The customer confirmed that this meets their security requirements.'


Fix constructConstantSurface when checkMinMaxRanges was not called

headerInfo.maxValues is undefined in constructConstantSurface at
https://github.com/Esri/lerc/blob/a79aba07cd7ef9b718e2f77d44bab3655d98ba74/OtherLanguages/js/LercDecode.js#L1624
when it is called without calling checkMinMaxRanges by: https://github.com/Esri/lerc/blob/a79aba07cd7ef9b718e2f77d44bab3655d98ba74/OtherLanguages/js/LercDecode.js#L1947-L1950

@tmaurer3 After merging, is it possible to do a proper npm release? It's going to be used in https://github.com/geotiffjs/geotiff.js/pull/206

@zakjan  Thanks for the fix! Yes, we will look into getting the npm site updated more regularly.

@tmaurer3 Thanks! Looking forward to the release.


Hollow Knight Fandom/Wikia -> Hollow Knight Wiki

The Hollow Knight Wiki on Fandom (Formerly Wikia) has moved to hollowknight.wiki: https://hollowknight.fandom.com/wiki/Hollow_Knight_Wiki:Moving_from_Fandom

There's a whole bunch of similar cases for other Fandom wikis:

Terraria Wiki: https://forums.terraria.org/index.php?threads/new-official-terraria-wiki-launches-today.111239/
Don't Starve Wiki: https://www.reddit.com/r/dontstarve/comments/15z7bbl/dont_starve_wiki_moved_to_wikigg/
Dungeons & Dragons Wiki: https://dungeons.fandom.com/wiki/Forum:We're_Moving
Hearthstone Wiki: https://www.reddit.com/r/hearthstone/comments/16l9vzo/hearthstone_wiki_has_migrated_from_fandom_to/
Noita Wiki: https://www.reddit.com/r/noita/comments/xnxx22/noita_wiki_has_migrated_to_wikigg/
Thorium Wiki (Linked from mod forum post https://forums.terraria.org/index.php?threads/the-thorium-mod.40788/)
PUBG: https://www.reddit.com/r/PUBATTLEGROUNDS/comments/127ln33/pubg_wiki_is_moving_to_wikigg/
Toontown Rewritten Wiki: https://www.reddit.com/r/toontownrewritten/comments/17s1kux/the_toontown_rewritten_wiki_has_migrated/
Zelda Wiki: https://zelda.fandom.com/wiki/Zelda_Wiki:Fork_Announcement
JoJo's Bizarre Encyclopedia: https://jojo.fandom.com/f/p/3678094237892900919
Koei Tecmo Wiki: https://koei.fandom.com/wiki/User_blog:Humble_Novice/Very_Important_Announcement!_The_Koei_Tecmo_Wiki_Is_Officially_Moving!

Would it be fine to just do the rest of those in one pull request?
There's also a couple of cases where the official wiki seems to have changed - probably safe to update, but I don't know enough to say which is better
Risk of Rain 2 Fandom -> riskofrain2.wiki.gg
Path of Exile Fandom -> poewiki.net

Yeah it'd be fine to do those in one PR, thank you!


Use linked list for captured tokens

The is essential because the list of captured tokens can branch with state. Currently I use a single mutable array, which is just wrong.

Closed by bec46a8d62ee3062c99fd36f8b916fc777167f60


Group datasets into "channels" when publishing

Problem
Right now, we have a pattern where every dataset we care about will have two versions in the ETL, an unharmonized one in meadow and a harmonized one in garden. So when you search for a dataset, it is not clear at first glance which one you should use.
Secondly, the open numbers datasets are so many that they overwhelm the search results.
Proposed solution

We group datasets into channels based on their folder inside the data/ directory

We should have garden, meadow and open_numbers channels initially

A dataset doesn't know which channel it came from, but it is stored as a categorical variable in the published index
When you search the catalog, you search "garden" by default; but you can also select any other channel, or "all"

catalog.channels should be a list of the options

With this solution, you should be able find data quickly, and default to the best version of the data that we have.

All is pretty clear, a couple of questions

should we move open_numbers from meadow into its own open_numbers folder? (so that the folder structure aligns with channel definitions) We'd end up with channels: garden, meadow, open_numbers, examples (and possibly backported). I think yes
reference datasets have their own folder, I'd load them by default together with garden

I guess that lazy loading of channels and better logging to show why is it taking so long are in scope of this.

should we move open_numbers from meadow into its own open_numbers folder? (so that the folder structure aligns with channel definitions) We'd end up with channels: garden, meadow, open_numbers, examples (and possibly backported). I think yes

I agree, open numbers deserves its own whole channel.

reference datasets have their own folder, I'd load them by default together with garden

I'm happy with this too, though I increasingly think we should end up making it just another standard step in garden that depends on an initial walden snapshot, rather than something checked-in to git.

I guess that lazy loading of channels and better logging to show why is it taking so long are in scope of this.

Seems like it would make sense. I'd probably break it into multiple steps. Something like:

Move open_numbers out of meadow and into its own folder (and update scripts)
Move reference into garden (and update scripts)
Index and publish channels individually
Add channels and lazy loading to owid-catalog-py

Feel free to edit the issue description and put your todo list there


Delete Files from Existing Snapshot

In cases of accidential backup of e.g. too large files, I would like to be able to delete specific files or directories (incl. recursion) from existing snapshots

That'd be really nice.

It would also allow removing sensitive data that got included unwittingly.

This would be a great feature!

Suggestion for the command name: restic purge.
I'm looking forward to this feature.  Thanks

I'm looking forward to this too ! I propose using something like the base structure for restic find.
restic purge [flags] PATTERN
Where you could limit the purge to host (-H) snapshots (-s) or paths (--path)
Then maybe a restic prune would afterward do the actual delete
This would be soooo helpful when a unforseen file gets backed up by error (a large video in a document folder or maybe a some confidential file) Right now, I run a restic find then delete every snapshot containing the file... This is less than desirable if the file is far in the repo (in time)
Thanks !

No update, sorry. You'll get notified by subscribing to this issue when something happens.

Not too bad, @nullcake.
Though, based on my past experience, it was usually that I detect that I backup loads of worthless stuff only days or weeks later. When I have some time to investigate. What this means is that by the time I understand I need some specific --exclude, there's probably a dozen or more backups impacted.
Of course, even if any kind of cleanup is implemented based on a single backup, like you suggest, it would still be a great step forward. We, of course, know how to script. ;)
So, thumbs up. :)

CrashPlan had an interesting behavior that when a file is excluded, it is purged from all existing snapshots. That could be something to consider.

@fd0 has there been any progress on this?  I just discovered that I wasn't excluding caches like I thought and would love to remove them.

Yet another suggestion for a command name: scrub (though I'm fine with purge as well, the --rename flag would seem odd to me). My thoughts:
restic scrub [--dry-run] [--replace=<clean|prune>] [--diff] [--all | snapshot-ID...]

Where --replace=clean removes any modified snapshot, prune cleans and runs prune afterwards. --diff shows a diff for any modified snapshots. --dry-run avoids committing any changes to the repo.
Also valid are all of the --exclude flags from restic backup. I guess --host and --time also make sense (each replacing the values of the preexisting snapshot); --tag editing is already handled by restic tag.
Do any developers have guidance on how this could be implemented? It seems to me (from a cursory look) that most of the code can go in a cmd_scrub.go file; maybe just a few API additions to the internal library necessary since it seems to be mainly index operations, but maybe that's naive. Any estimated difficulty (I assume testing will be the bulk of it in any case)?

since this is a very very old issue.. is there any chance of getting this feature?

For all that monitor this for updates and hit it from Google, there's no need to wait for this issue to never go into fruition, just use duplicati for the meantime, it has first class support for removing files post fact from snapshots.

For all that monitor this for updates and hit it from Google, there's no need to wait for this issue to never go into fruition, just use duplicati for the meantime, it has first class support for removing files post fact from snapshots.

I've been using restic for about a year now and I stopped waiting for features to be implemented. I don't mean that everything should be added into restic, but there is basic things that should be there. I'm considering moving away from restic: the repository is very fragile and can get broken very easily.
Yesterday I deleted a snapshots because it included files that should not have been in the backup (I forgot to add an exclude). Since then I have errors in my repository and I haven't been able to repair it yet. I should not have to delete a whole snapshots because some files where included by mistake.

@MorgothSauron I usually just removed snapshots that contained it too, which is the only solution it seems in restic, but again, duplicati can do it via a single command for a while now, so I've changed since and had no issues.

I wish to thank everyone for their input on this matter. As we've seen, many people have wanted in particular the ability to remove files from a snapshot. I guess we all make mistakes once in a while when backing up ;)
At this point in time the available maintainer and developer time is needed on other parts of restic, so I do not foresee this issue being implemented in the foreseeable future. Personally I'm trying to tie some knots together and the next step is getting @ifedorenko's restore optimizations merged, because that is one of the things that brings much value to many users. I'm also going to release a new rest-server as soon as I can, and will then start to look into some other issues.
That said, if someone makes a solid PR that is nicely and clearly written, well tested and bug free, and produced in coordination with maintainers, it will definitely be considered for inclusion. This specific issue is one where @fd0 has already given his blessing on the direction, so focus can be mainly on producing a solid implementation (that we know won't corrupt repos) rather than "should we add this feature", which is good.
Such a PR should be basic and act as a starting point which if needed can be built upon. An example of what I mean by that is it should for starters:

Just be one new command (e.g. rewrite since that's the most voted for in this issue).
Take a list of snapshots as its primary argument(s) (including support for all), e.g. all or 098db9d5 or 098db9d5 af92db33.
Take a list of one or more --exclude <pattern> to list the paths that should be excluded/removed form the snapshot (in other words, here's the --exclude that was missing when backing up), e.g. --exclude="*.o", --exclude=*.unwanted, --exclude="*.o" --exclude=*.unwanted --exclude=.DS_Store.

The rationale here is to get a minimal start as a proof of concept and minimum viable product. Once being tested we can adjust it as needed, e.g. by adding the other --exclude-* arguments from the backup command. If we make a rewrite command like this, it will have pretty much the same interface as the backup command that it's meant to "correct":
restic -r /some/repo rewrite all --exclude="*.o" --exclude=*.unwanted --exclude=.DS_Store
restic -r /some/repo rewrite 098db9d5 af92db33 --exclude="*.o" --exclude=*.unwanted --exclude=.DS_Store

On a related note, perhaps the work done by @middelink in https://github.com/restic/restic/issues/323 could be used as inspiration or a basis for the implementation, as it does some processing of existing snapshots. I'm going to see if we can get moving with this one too soon.

@rawtaz
Thanks for the thoughtful feedback!

Hi there.
I've added draft rewrite implementation close to comment by @rawtaz
It works here with test repo, passes restic check --read-data without errors, but have not tested it much. So I strongly suggest to not use it with important data.
I've tried to get syntax very close to backup command. So --exclude, --iexclude and --exclude-file are supported (but not tested). Ideally I also want to see --exclude-if-present option (ideal workflow for me is something like 'oops, not needed to backup, add CACHEDIR.TAG and restic rewrite'). But it's pretty complex because in such case we'll need to rewrite on same host where backup was made and access filesystem to collect these files (plus tons of magic with relative paths). So not right now...
Also I don't like idea to replace snapshots by default, so currently default behavior is to just create new snapshot with rewrite tag. But replacing is also possible with --inplace arg.
Any feedback would be greatly appreciated.

Hey Dmitry,
Thanks for this implementation, great work !
So far it works perfectly on Linux with a small test repo of 600 files + several test snapshots. Restore works and diff shows correctly excluded folders. I will be doing more intensive tests on a "clone" real repo with many GB of data with more 100's of snapshots. I will also try Windows sourced repos.
One proposition : have the option to specify a tag for the snapshots that contained the exclusions on a rewrite pass. (keeping the "rewrite" tag on newly created snapshots.)
restic rewrite --add-tag mytag -i thisfileshouldberemoved.txt all
This would help identify those snapshots that still contains "thisfileshouldberemoved.txt". On the other hand, the more direct --inplace works like expected.
Again very good work.

@NovacomExperts Yes, my initial motivation was to keep 'history editing as safe as possible. It's very easy to exclude something important with --exclude * and almost no way to recover from this (with backup it's just matter of start new backup again). Something like --dry-run but with ability to get actual snapshot and explicitly delete source snapshot after checking that it's ok.
I fully agree that currently this is not fully achieved.  It's easy to 'observe' new snapshots, but too difficult to delete old one. Plus I don't like hardcoded rewrite snapshot name.  Maybe it's better to have --inplace by default and and ---keep-source-tagged before-rewrite --tag-destination after-rewrite or something like this. (--add-tag is a bit unclear, whether it's old or new snapshot).
In any case I'll wait for feedback from maintainers. Don't want to spend much time if it's move in wrong direction.
PS. My primary restic repo is around ~2TB now. Will try on it later after making LVM snapshot.

@dionorgua Your initial motivation is fully correct. I'll cast my vote to keep it like that, with the "dangerous" option --inplace as far as possible from the user (definitely not by default). I would prefer a missing argument error on --keep-source-tagged / --tag-destination than --inplace by default.
But I agree, let's wait for feedback on this.
Yesterday, I forgot the cloned test repo (65 GB) inside a folder that was backed up by restic overnight. I could have forget yesterday's snapshot but went "all in" and tried your implementation. After forget + prune , I successfully removed the 65GB from a 400GB repo. All good, no error found.
I test more intensively with data that spans across multiple snapshots.
Cheers

I've replaced that wrong #2720 pull request with new one because old one was created from master branch. Just added one missing error check. Sorry for extra noise

Hm, modify then?

Very late for this, but rectify is my suggestion for the delete-specific-file-from-backup command.

#2731 is exciting, thanks a bunch!

Hi, if it was possible to add, remove folders or files to an existing snapshot, restic could be like a dedupe filesystem, as OpenDedup. An interesting use case could be to save multiple versions of vhd files.

The thing should be simple, e.g.
restic rm /srv/git/linux

It will delete the directory from all snapshots where it exists.

Such a destructive action should not be so trivial. FWIW, I think the approach currently taken is the right approach (editing snapshots to remove references to paths then using forget to age the old snapshots out of the repository permanently).

What do you mean  "the approach currently taken"?  Is it taken for future release or it has already implemented?

There's a PR for this which incorporates outcomes of discussion here. See #2731.

It would be great for Restic to have functionality analogous to borg recreate. https://borgbackup.readthedocs.io/en/stable/usage/recreate.html

Any updates on this? Pull request https://github.com/restic/restic/pull/2731 seems not maintained any more.
I'd also really like to have this feature.

Because it seems there is some interest in this issue for a long time now, I'll post my crude workaround using Python which I used a couple of days ago and worked perfectly.
The basic idea is to rewrite all snapshots but with an 'exclude' filter to exclude the files you want to scrub/purge. Depending on the amount and size of the snapshots, this might take some time because it will rescan the metadata of every file in all of your snapshots. It uses the restic mount function for this so fusermount must be working on your system. Also if the files you want to purge moved location in between the snapshots, make sure to specify multiple '--exclude=' filters with each location the file or directory could be in. Also probably only works on Linux.
You will lose some information from your snapshots but this can be changed by adapting the script to your needs. As far as I know (might be more) with the current script you will lose:

Timezone and milliseconds precision from the time the snapshot was taken
Existing tags
The listed backup path will be incorrect

Requirements:

Mount your repository somewhere
Get a list of the snapshots you want to rewrite in JSON format: restic -r /srv/restic-repo snapshots --json
Prepare your restic command which you normally use, however with some modifications:

Add your new '--exclude=home/user/directory_to_scrub' filter. Make sure this filter does not use an absolute path. We will change into the backup directory and therefore all paths will be relative. You can remove your existing filters as the files the normally affect simply won't be present in the snapshots.
Make sure the command is not interactive (no sudo, no password input) as this won't work when scripting.
Remove your targets (for example '~/work') from the command

In the script substitute:

SNAPSHOTS_JSON_MAPPING, with your list of snapshots in JSON format
RESTIC_COMMAND, with your modified command, leaving every existing option inplace.
RESTIC_MOUNT_DIR, with the absolute path to the directory you mounted your repository to
PRUNE_TAG, with a tag of choice

If you now run this Python script it will change into the directory of each snapshot and perform a backup again using the current snapshot as parent and tagging it to make it distinguishable. Might be wise to test it with just one snapshot if it works to your liking. When restic is making a backup it will show you the 'current_file' it's processing and the total amount of files increasing. These numbers should increase roughly at the same rate as this script is not really writing new files to your repository but only metadata (which is quite fast).
#!/usr/bin/env python3
import datetime
import os

mapping = SNAPSHOTS_JSON_MAPPING

for i in mapping:
	command = """ RESTIC_COMMAND
		--parent {parent_id} \
		--ignore-inode \
		--time "{time}" \
		--tag "PRUNE_TAG" \
		. """
	repo_dir = "RESTIC_MOUNT_DIR"

	os.chdir(f'{repo_dir}/ids/{i["short_id"]}')
	# We are forced to lose the timezone and some seconds precision
	command = command.format(parent_id=i["id"],
		time=datetime.datetime.fromisoformat(i["time"][0:19]))
	print(f'---- Processing snapshot {i["short_id"]} ----')
	os.system(command)

Example:
I normally use this command restic --no-cache -p /etc/resticpasswd -r "/mnt/vg2-backup_lvol1/" --exclude="home/user/downloads" backup . and I mounted my repository at /mnt/resticmnt/.
The script will become:
#!/usr/bin/env python3
import datetime
import os

mapping = [
  {
    "time": "2022-08-03T03:04:10.22434835+02:00",
    "parent": "6b0a4ca9cbc8bce824588c6343e347405aac3d2bf196ca29b0d59234fc5e4da2",
    "tree": "8d33292e2d616d855e1dfba601abaf0e02f61404ae09075462eb6496e5a7eeba",
    "paths": [
      "/mnt/resticmnt"
    ],
    "hostname": "big-server",
    "username": "root",
    "id": "f4c4093a743a1ce3eb7f6e7a1914f9b13fca7bab87de6fe1bed0c3d0a2cd314c",
    "short_id": "f4c4093a"
  }
]

for i in mapping:
	command = """ restic --no-cache -p /etc/resticpasswd -r "/mnt/vg2-backup_lvol1/" backup \
		--exclude "home/user/directory_i_also_dont_want" \
		--parent {parent_id} \
		--ignore-inode \
		--time "{time}" \
		--tag "prune_downloads" \
		. """
	repo_dir = "/mnt/resticmnt"

	os.chdir(f'{repo_dir}/ids/{i["short_id"]}')
	# We are forced to lose the timezone and some seconds precision
	command = command.format(parent_id=i["id"],
		time=datetime.datetime.fromisoformat(i["time"][0:19]))
	print(f'---- Processing snapshot {i["short_id"]} ----')
	os.system(command)

Afterwards you can use restic diff between an old and a 'new' snapshot to make sure your excluded files are not in the new snapshot, but everything else is fine. If everything is fine you can restic forget everything which does not contain the PRUNE_TAG you chose. For example: restic -p /etc/resticpasswd -r "/mnt/vg2-backup_lvol1/" forget --keep-tag "prune_downloads". And finally after pruning your files will be permanently removed from the repository.
Afterwards you first new backup should contain both the --ignore-inode and --parent {REPLACE_WITH_LATEST_SNAPSHOT_ID} parameters as restic will not recognize the parent because the path is different. Also don't forget your new exclude filter, otherwise the files will be backed up again in new snapshots...
I'm writing this after the fact so I might have forgotton some steps or requirements. Let me know and will update this post. Also feel free to make the script more robust if you so desire.

Is there anything we can do to get this rolling? Would a bounty work?

Hi all, just checking in to see if this has had any movement?
I have several TB of data I need to remove.  They're video editing movie data (massive files) that was put into the wrong folder by someone who simply messed up.  Human error.  Our online backup is now MASSIVE and costing us some coin.  We need to remove these files as they're costing us each month to have it there.
Just wanted to see if there's an implementation yet?
Thanks so much.

@therealrobster there is a pull request in #2731 that is still being
worked on, but until it is finished, you may want to try the workaround
documented above (note the data loss mentioned in the post).
--
bye,
pabs
https://bonedaddy.net/pabs3/

I'd say stay tuned because that PR is shaping up pretty well.


Add test mode that only returns the uncompressed size and checks it and maybe also the CRC32

Getting the uncompressed size is a common use case.
Avoiding the copies for backward pointers might improve performance for this particular use case.
And marker replacement is not necessary in this case! Also, allocations can be avoided and copying out of the deflate decompressor. All together this should combine to some significant performance increases.
A --count option similar to --count-lines also wouldn't be out of line (pun intended).

I couldn't observe any speed improvements for the single-core case even in a highly compressed file:
m pragzip && src/tools/pragzip -v --count test-files/small/small.gz
    Decompressed in total 536870912 B in 2.17922 s -> 246.36 MB/s$

m pragzip && src/tools/pragzip -v -d -o /dev/null -P 1 test-files/small/small.gz
    Decompressed in total 536870912 B in 2.17986 s -> 246.287 MB/s
    Decompressed in total 536870912 B in 2.73646 s -> 196.192 MB/s
    Decompressed in total 536870912 B in 2.15831 s -> 248.746 MB/s

m pragzip && src/tools/pragzip -v --count test-files/small/small.gz
    Decompressed in total 536870912 B in 2.08869 s -> 257.037 MB/s
m pragzip && src/tools/pragzip -v --count test-files/small/small.gz
    Decompressed in total 536870912 B in 2.17042 s -> 247.358 MB/s

src/tools/pragzip -d -o /dev/null CTU-13-Dataset.tar.gz
    Decompressed in total<PHONE_NUMBER>0 B in 43.8358 s -> 1819.23 MB/s
src/tools/pragzip --count CTU-13-Dataset.tar.gz
    Decompressed in total<PHONE_NUMBER>0 B in 207.621 s -> 384.102 MB/s
src/tools/pragzip -d -P 1 -o /dev/null CTU-13-Dataset.tar.gz
    Decompressed in total<PHONE_NUMBER>0 B in 204.375 s -> 390.202 MB/s

And the added code is not negligible even for the sequential case. Adding a specialization for the parallel case does not seem worth it.
Another aspect would be memory usage, which could be hugely reduced. This should also integrate more easily into the existing parallel gzip reader. There could simply be a flag that skips the copy of the ranges returned by deflate::Block::read and simply accumulates the size.

Implemented with e9b27b21.


add vex typings

@gdcohan thank you your contributions.
please rename vex/vex.d.ts to vex-js/vex-js.d.ts. http://definitelytyped.org/guides/contributing.html#naming-the-file
https://www.npmjs.com/package/vex-js

@vvakame sorry about that--should be good to go now

@gdcohan thanks mate!


Switch Endonym and Language in "Explore" Cards

User suggestion to switch the position of Endonym and Language in the "Explore" cards where they appear together, to match the prominence of Endonym in the Language Profile and in Details. This would not apply to card footers, which are Language only. It would apply at the level of Explore/World Region/Something, Explore/Neighborhood/Something etc. Would also want to have it in Explore/Languages and change "Common language name in English" to something like "Common endonym (used by community) and name in English"

I think I did this one in under 3 minutes 🏆


Eddystone for Bluetooth 5.0

Hi,
Can Eddystone run on nRF52840? Its running quite well on nRF52832. Need some URLs to be pushed through BLE 5.0. Right now I can't find the right files for the 840 chip.

I am assuming you want to put larger URL's in the advertising packet. If so, make sure you test that the large installed base of BT4.X phone and tablets will be able to receive those longer advertising packets.

Hi, I was thinking if we can use long range feature of Bluetooth 5, that would really give some aspects as well. I would try the larger advertisements as well.


[feature request] ollama website API? or maybe docs for it...

is it possible we have a way of fetching/searching models list from the official ollama website? so that other programs can integrate into it as well? if it already exists, maybe a small docs about it.

We'll definitely publish something. The API is still under pretty active development, so we didn't want to publish it, have people rely on it, and then change it and break everything.

https://github.com/jmorganca/ollama/issues/1473

Closing for https://github.com/ollama/ollama/issues/1070 thanks for the issue!


fixed a small typo in example

Would it be possible to resubmit this PR against razorsedge:develop?

Merged into develop in commit 600a538a1a4a5b3148b1c47a6f1de212d470c9d2.  Thanks!


Customize now playing with metadata/tags

It would be nice if I could configure the now playing screen to add in some metadata/tags. Probably around the area where the bitrate & codec is (e.g. "804kbps VBR • flc").
It would be awesome if I could modify the album list to show tags as well.

No, sorry - not interested in adding more data to nowplaying screen. What else would be required?
As to albums, Material shows the relevant info that is obtainable from the JSON API as is. So again, no.


Inference code

Hello.
Thanks for awesome work and sharing the code.
Can you please share the inference/demo code?
Thanks

I need it too.
Thanks.


Add "flip_all_pepflips" to libcoot api

This change is no longer required in libcoot api as it was implemented in the front end in #273


Disable HTTP Upgrade feature when WebSockets are not available

Out-of-proc

ANCM sets ASPNETCORE_WEBSOCKETS_SUPPORTED environment variable to true if WebSockets module is installed and OS Version > Windows 7. Or false otherwise
IIS Integration middleware reads this environment variable (defaults to true if not present)
If ASPNETCORE_WEBSOCKETS_SUPPORTED = false: Remove the IHttpUpgradeFeature from the request features
Else: do nothing

In-proc

IISHttpServer checks if WebSockets if WebSockets module is installed and OS Version > Windows 7 (either in the managed code, or by reading data populated by the native code, whichever is easier/possible)
If true: Install the IHttpUpgradeFeature as before
Else: Do not install the IHttpUpgradeFeature

/cc @muratg for pre-triaged visibility

@anurse for the inproc and out of proc scenarios, they are both on a per request basis? Therefore,the IISHttpContext should be the one to remove the IHttpUgradeFeature?

IIS should be modifying the request features yes. However, it's certainly the case that the decision to allow the upgrade feature can be made once per process, since it's not possible to enable the WebSockets module or upgrade the OS while the process is running ;). So yeah, it would be in the IISHttpContext that you would remove the feature, but you could use a global static variable to determine it.


Include Norm in Bracket behaviour

the idea would be to add “|” and “{“ “}” as a bracket type, so that the auto-enlarge behaviour applies to norms.
Your plugin is great.

Good idea, I'll try to add this in one of the next releases.

This will be easy to do for \{ and \}, because the open and close symbols are different.
However, "norm"  uses the same symbol | to denote the start and end. So I don't think it's possible, exactly, to determine which open symbol | should be paired with which close symbol | in the case of nested norms (e.g. $|a+b|c|d|$).

Ooh that is very subtle. I see, it doesn’t seem possible.

1.4.2 adds \{ and \} to the auto-enlarge brackets feature.

Amazing thanks!

Hello, I think this can be done via the \lVert \rVert commands for the norm instead of | |. I just learned about these.

Hello, I think this can be done via the \lVert \rVert commands for the norm instead of | |. I just learned about these.

Thanks for the info! Added in 1.6.5.

Amazing, thanks a lot!


rate limit exceeded

Worked fine for a few days, then I started getting this error. I tried it two days in a row so far. There are two sites, south and west facing arrays on my roof. If it helps I could split the list comprehension to find out which it's failing on.
python3 ElmBankSolar.py
Traceback (most recent call last):
File "/Users/peland/Documents/pp/ElmBankSolar.py", line 125, in
forecasts = [site.get_forecasts() for site in sites]
File "/Users/peland/Documents/pp/ElmBankSolar.py", line 125, in
forecasts = [site.get_forecasts() for site in sites]
File "/Users/peland/Library/Python/3.9/lib/python/site-packages/pysolcast/rooftop.py", line 21, in get_forecasts
return self._get_data(self._create_uri(self.base_uri, endpoint), params)
File "/Users/peland/Library/Python/3.9/lib/python/site-packages/pysolcast/base.py", line 39, in _get_data
raise RateLimitExceeded(
pysolcast.exceptions.RateLimitExceeded: Rate limit exceeded. Reset time: None

Update: it started working again and I dismissed it as a server problem, but I just tried it again and on the third or fourth call I got the same error.

From the trace, it Looks like the rate limit exceeded error is being thrown by pysolcast rather than octopus energy. You should probably look there.

I'm beginning to suspect I have a (very small) quota on the server, could that be the case? Or am I somehow making far more requests than I think I am?

Oops! Your reply appeared as I typed and I didn't notice - I'll follow that up, thanks!

My apologies, I thought I was posting on the pysolcast page!


wrong format for https://github.com/kubevirt/kubevirt/releases/download/v0.16.0/kubevirt.yaml ?

Trying to deploy with same instructions as for https://kubevirt.io/user-guide/docs/latest/administration/intro.html#1-stable-flow:
kubectl apply -f https://github.com/kubevirt/kubevirt/releases/download/v0.16.0/kubevirt.yaml
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply

Not sure whether this is worrying...

Hey @olberger just did a quick test and didn't get the warning.

Did you apply the manifest on a clean cluster? (clean as in it didn't have kubevirt deployed before)
What are the k8s client/server versions? (kubectl version)

@olberger it can be that you did a kubectl create … on a previous run, then oyu get the warning if you do a kubectl apply later on.
This is normal ..


No dependency for "fs"

file-fetch requires "fs" but doesn't list it as a dependency in the package.json.

See https://github.com/linkeddata/rdflib.js/issues/311#issuecomment-479966204 (towards the bottom) for the error we are getting.
ModuleNotFoundError: Module not found: Error: Can't resolve 'fs' in '/home/mrh/ttt/rdflib.js/node_modules/file-fetch'

This is almost certainly a webpack issue, unrelated to file-fetch.  Sorry for the false alarm.


Replace Array with ActiveRecord::Relation partial in MiqWidget::ContentGenerator specs

Currently the specs for MiqWidget::ContentGenerator break with partial double verification because records is an Array trying to call find_by.
This PR updates it so that it uses an actual AR relation. I also updated the outer describe block to RSpec.describe for future compliance with rspec 4.

@miq-bot add_labels test, refactoring, hammer/no, changelog/no

@miq-bot add_reviewer @kbrock


Fix bad merge in YamlError

Bad merge in https://github.com/VirtusLab/scala-yaml/pull/309

@lbialy @tgodzik ready to be merged :pray:


Refactor interfaces to use composition over inheritance

For instance the Reporter class becomes a module that can be included in any class, without enforcing the class to inherit from the old abstract class.
Basically this implies that any class can act as a 'reporter', yet it
does not need actualy to inherits from it, if implements the necessary
interface.
Affected interfaces: Matcher, Order, Output, Reporter, Runner

Looks good to me.
PS: this will break spec2-mocks.cr library, since it defines matcher in old style.

I have released 0.5.0 with this change and updated the README accordingly.
Now I will update spec2-mocks.cr library to use new style of matcher.

I have released 0.2.0 of spec2-mocks.

@waterlink thanks! :dancers:


Core: Let one pass authentication header to JsonFeeder

Main class to modify: https://github.com/gatling/gatling/blob/master/gatling-core/src/main/scala/io/gatling/core/feeder/JsonFeederFileParser.scala

I'm in to take this issue !

Which kind of API would you expose ?
Something like: def jsonUrl(url: String, authenticationHeader: String)(implicit jsonParsers: JsonParsers) ?

@blemale Is this something you need, or an issue you picked in order to contribute? I mean that somebody asked for this once, and I never heard about it again, so I'd rather not bring a feature that nobody would need/use.
If we were to have this, I'd rather go with the same design as request auth: http://gatling.io/docs/2.1.7/http/http_request.html#authentication

@slandelle, I just pick an issue with "Please contribute" tag.

Someone's feeling his Scala rusty before quitting for a new Scala dev position? :stuck_out_tongue_winking_eye:
If you want to implement this one, I think this feeder json w/ url should move to the http module so it can reuse some AHC utils for the header computation.

Do you have another more useful issue to unjam my scala skills ?

Actually, could you please first tackle this one? I have some ideas, but I have to first finish some other tasks so you can build upon.

Hello guys, I came from the future just to ask one question, I couldn't found in the documentation anything about this feature, it's that possible to call json feed using some sort of authentication like oauth2?
did you get this feature?
Thanks

The idea was dropped because of lack of interest. Feel free to contribute.


Pin dind dep

@aaronlehmann can we pin https://github.com/docker/distribution/blob/master/contrib/docker-integration/Dockerfile#L28 to a fixed commit?
I don't like the smell of unbound master, specifically in light of https://github.com/docker/docker/pull/15596

Not really into "dind", but according to https://github.com/docker/docker/pull/15596, dind is no longer needed when using Docker 1.8+, don't know if that applies for distribution, but may be worth checking :+1:

I think we can avoid using dind in this Dockerfile.


Maturing the API + adding verification

Maturing the API
In my mind the goal of this library is clear: The library should aim to deal with only HTTP Signatures and not deal with other related but not directly part of the signature issues. This means there should be no attempts made to do things like calculate or verify digest headers, nor attempt to worry about validating other parts of HTTP requests/responses except the http signature header(s).
This means there is no longer any need to worry about the body that will be passed to the HTTP request or that comes from the HTTP Response. The only parts of a request that we are concerned with are the headers, absolute url, and method. All components for a request signature can be derived from these parts alone.
At the moment the support for complex headers and components is limited, for example query parameters are not currently supported. The introduction of the structured-headers library should greatly help in this regard as the parsing of headers is not as straightforward as it may seem.
It should also be possible to add request-response binding to signatures.
Verification of signatures
At the moment verification of headers is completely missing from the library. This feature should be added.
Both key lookup and signature verification should be asynchronous.
Verification, like generation, of signatures does not require the full response, just the headers and status code. Further, to enable request-response binding the verifier needs access to the original request as well.
New requirements
Given the need to support request/response binding, it makes sense that there are separate function for signing/verifying requests and responses. At the moment there is an attempt to unify these use-cases in single functions, but that feels like the API is less clear and the code base needs to work in extra checks to be able to infer what is going on.
Therefore we need 4 distinct pieces of functionality:

Sign requests
Verify requests
Sign responses (with request binding)
Verify responses (with request binding)

Compatibility
The library aims to be compatible with built in node requests and other popular libraries like axios and should be simple to implement in an express middleware

The introduction of the structured-headers library should greatly help in this regard as the parsing of headers is not as straightforward as it may seem.

The latest release of structured-headers is ready to pull in as a dependency now and replace the existing parsing and serialisation logic.


Non-max suppression is not implemented

Since Non-max suppression is not implemented in the inference.py, some ships are recognized several times and disrupt the model output.

PR merged, I'm closing the issue.


Make -> CMake + Workflows for Windows & macOS

Thank you very much for the nice work. I think that developer community of Dear ImGUI will benefit a lot from this project.
I made tiny modifications to the C++ part so that it can compile cleanly using MSVC 2019 and replaced the Makefile with CMake. Also added workflow/actions for Windows and macOS.
You now need to install glfw only if the target platform is Linux. Windows & macOS will automatically download GLFW during the build process and build a static version that can then be linked against. This means that no brew is needed or macOS. It also means that native Windows runtime from VS2019 can be utilized and there's no need for mingw or msys runtimes. The app works pretty nicely on Windows. I haven't tried it yet on macOS, but the workflow builds with a few warnings.
Hope this helps. :)

Closing this one and creating another to reduce spam as I work on creating installer for Windows.


Error when generating object keys [JIRA: CLIENTS-900]

Using riak 2.1.4 with the python client 2.5.3 with Python 3.4.3:
from riak import RiakClient

c = RiakClient(nodes=[{
    'host': '<IP_ADDRESS>',
    'pb_port': '8087'
}])

maps = c.bucket_type('maps')

nodes = maps.bucket('nodes')

node = nodes.new()  # expecting key to be generated by riak
# ...
node.store()

It seems that the generated key is of type bytes instead of str in Python3:
Traceback (most recent call last):
  File "graph.py", line 77, in <module>
    node.store()
  File "/home/linkdd/devel/projects/link-framework/venv3/lib/python3.4/site-packages/riak/datatypes/datatype.py", line 165, in update
    self.bucket._client.update_datatype(self, **params)
  File "/home/linkdd/devel/projects/link-framework/venv3/lib/python3.4/site-packages/riak/client/operations.py", line 1143, in update_datatype
    include_context=include_context)
  File "/home/linkdd/devel/projects/link-framework/venv3/lib/python3.4/site-packages/riak/transports/tcp/transport.py", line 505, in update_datatype
    msg = codec.encode_update_datatype(datatype, **kwargs)
  File "/home/linkdd/devel/projects/link-framework/venv3/lib/python3.4/site-packages/riak/codecs/pbuf.py", line 1211, in encode_update_datatype
    req.key = str_to_bytes(datatype.key)
  File "/home/linkdd/devel/projects/link-framework/venv3/lib/python3.4/site-packages/riak/util.py", line 118, in str_to_bytes
    return value.encode(encoding)
AttributeError: 'bytes' object has no attribute 'encode'

Hello -
Could you please run this command and provide the output in a response?
riak-admin bucket-type status maps

Thanks. I'd like to see how the maps bucket type is configured.

Here is the output:
maps is active

young_vclock: 20
w: quorum
small_vclock: 50
rw: quorum
r: quorum
pw: 0
precommit: []
pr: 0
postcommit: []
old_vclock: 86400
notfound_ok: true
n_val: 3
linkfun: {modfun,riak_kv_wm_link_walker,mapreduce_linkfun}
last_write_wins: false
dw: quorum
dvv_enabled: true
chash_keyfun: {riak_core_util,chash_std_keyfun}
big_vclock: 50
basic_quorum: false
allow_mult: true
datatype: map
active: true
claimant<EMAIL_ADDRESS>
FYI, I followed the documentation when creating the bucket:
$ riak-admin bucket-type create maps '{"props":{"datatype":"map"}}'
$ riak-admin bucket-type activate maps

OK. I asked because there is an integration test that tests a Riak-generated key on a non-datatype bucket, but I couldn't find something for a bucket configured for a datatype. I suspect that's the issue.


Bugs & new feature idea

Bugs :

Can't play render file if it's sequence ,because file address in write node it's not the exact file name (%d or ###)

Not Compatible with linux : - UI has some problems (because of texts)
- Buttons (open render file and copy to clipboard function) don't work

New future : Add a button to "create read node from rendered file"

v2.0 is ready
All is done in v2.0 :)


FAI-13016: Remove lerna and migrate to Turbo repo

Description

Bye bye lerna

Type of change

[X] Bug fix
[X] New feature
[X] Breaking change

Yay!!!


Can't run the private_gpt module.

I almost done everything perfectly. But once i try to run the private_gpt module t show me some errors:
"TypeError: BertModel.init() got an unexpected keyword argument 'safe_serialization'"
How to solve this error ?

For me, just upgrade llama-index-embeddings-huggingface
llama-index-embeddings-huggingface 0.1.5, solved in 0.2.0

Fixed in last main version :)


failed to complie

/var/folders/b_/crw_x7vj52g1g39w4dr5pv800000gp/T/CocoaPods-Lint-20180821-16308-ax4kba-R.swift/Pods/R.swift.Library/Library/UIKit/NibResource+UIKit.swift:22:84:
error:
'OptionsKey' is not a member type of 'UINib'
public func instantiate(withOwner ownerOrNil: Any?, options optionsOrNil: [UINib.OptionsKey : Any]? = [:]) -> [Any]

ERROR | xcodebuild: Returned an unsuccessful exit code.
- ERROR | xcodebuild:  R.swift.Library/Library/UIKit/NibResource+UIKit.swift:22:84: error: 'OptionsKey' is not a member type of 'UINib'
i cloned R.swift to my private pod, while R.swift.Library's source is still from github.
In the terminal, i typed pod repo push, error came out. i check the code,OptionsKey is not available. Any idea?

The master branch of R.swift and R.swift.Library uses Swift 4.2.
If you want the Swift 4.0 compatible version, use the v4.0.0 tag of both repositories.

fixed. Thanks.


curvebs: fix mount status check fail during format chunkserver

Closes: #55
When data dir is umounted, the check will failed on CentOS, because
we check suffix must be 'not mounted', but on CentOS it is 'not mounted.',
then the format will be failed.

LGTM！


Frontend audit

Fixes #1027

Description

Updated: packages and configurations

Checklist
- [ ] I have made corresponding changes to the documentation.
- [ ] I have updated the types definition of modified exports.
- [ ] I have verified the functionality in some of the neeto web-apps.
~~- [ ] I have added the necessary label (patch/minor/major - If package publish
is required).~
Reviewers
@AbhayVAshokan _a

@AbhayVAshokan
“postcss”: “8.4.31",
“postcss-import”: “15.1.0",
“postcss-loader”: “7.3.3",
“postcss-preset-env”: “9.2.0",
“@babel/core”: “7.23.2"
“@babel/eslint-parser”: “7.22.15",
“@babel/plugin-transform-runtime”: “7.23.2",
these packages are causing issues while building storybook, which i was not able to figure out completely could you please take a look at it.

The issue is with the incompatible storybook and babel versions. We shall fix this in a separate issue. Merging this PR to unblock #1024.


Configuration management

Problem
We want to have flexibility to shift towards declarative configuration management. Some benefits:

expose every hyperparameter to enhance reproducibility
build a library/zoo of models
facilitate the creation of user models by modification of existing ones.

Solution
We considered: (a) roll-our-own and (b) Google's GIN.
(a) would lead us to reinvent, in particular, the function call syntax that thinc and GIN already provide. And create a non-core code liability.
(b) seems to be too complex / less well adapted for our objective.
Adopt the config module of thinc, possibly without depending on the entirety of thinc.
Follow-up work

deprecate get_sbi_posterior in favor of config mechanism - consider rename (density_estimator).

NOTE: we may want to start at pyknós level, defining nflows models via thinc.

Should we close this in favour of #195?

@janfb @jan-matthis any objections if I close this? Configuration management should be handled by the user imo.

I agree, but we need to give all the flexibility to do so, e.g., open it up for all kwargs to be set. in neural_nets some kwargs are still hard coded, e.g., dropout and num_blocks. We could add them to the list of changeable kwargs?
But then, all this nflows code will go out soon, so maybe that doesn't matter anymore. But when building the new flows, we should keep in mind to give access to all kwargs.

Yes, I agree. I think it's fine to close this issue now though.


Rancher shows "ServerFaultCode: A specified parameter was not correct" when setting console in cloud config

RancherOS Version: (ros os version)
v1.4.2
Where are you running RancherOS? (docker-machine, AWS, GCE, baremetal, etc.)
vSphere 5.5
If I set the console in the cloud config as described in the documentation, then the node will start successfully and installs the console. But Rancher shows: "Error creating machine: Error in driver during machine creation: ServerFaultCode: A specified parameter was not correct.". Rancher then tries to provision the node again and again. I have tried to set the console to centos and ubuntu.
My current config:
#cloud-config
rancher:
  console: ubuntu
  registry_auths:
    https://registry.domain.com/:
      username: "user"
      password: "***"

ssh_authorized_keys:
  - ssh-rsa ***<EMAIL_ADDRESS>
write_files:
  - path: /etc/docker/certs.d/registry.domain.com/ca.crt
    permissions: "0644"
    owner: root
    content: |
      -----BEGIN CERTIFICATE-----
      ***
      -----END CERTIFICATE-----

I use exactly this ISO. And I create the nodes with Rancher and vSphere node templates.

If I enable the console after the node has started, then rancher loses also the connection to the node. The UI shows "Kubelet stopped posting node status"

The user docker will be stopped when switching the console.  I think it may be caused by this.

@niusmallnan
I reproduced this issue
After rancheros start, it will switch the console immediately. If the switch is successful after rancher connects to the node, this error will occur.

Hitting this problem as well. Is there any timescale to look at it? It prevents a lot of custom configuration OS-wise during provisioning which is annoying (in our case we want to install things like cifs-utils to handle smb mounts).

Looking to see if there is a possibility of this being looked into.
Thanks,


feat/preview-comply-with-photo-resolution-when-video-is-disabled

What
This PR handles the Android preview based on the format and camera configuration, using the proper resolution if the video output is disabled.
Use case: I want to see exactly what the final photo will look like in the preview.
To achieve this you can use the selected format and calculate the ratio, then you can render your camera view like this:
const aspectRatio = selectedFormat.photoHeight / selectedFormat.photoWidth; // if sensor is rotated

{
  !!device && (
    <View
      style={{
        width: '100%',
        aspectRatio,
      }}>
      <Camera
        ref={cameraRef}
        style={StyleSheet.absoluteFill}
        device={device}
        isActive={true}
        resizeMode="contain"
        format={selectedFormat}
        photo
      />
    </View>
  );
}

Changes

This PR sets the preview size based on the camera configuration
This PR updates the documentation of the preview section

Tested on

iPhone 15 Pro, iOS 17.6.1
Android Moto G52, android 13

Related issues

it still not reverted

Do you mean to remove all changes from the docs?
Here I specify that the preview on Android will follow the photo or video prop

my bad sorry I just quickly peeked at it and thought it was a formatting change. sorry I need sleep.

cool :) don't worry, go rest, thank you


Please add spotify

good day of your project I like but spotify ba there personally all fit

Spotify requires an API key, which cannot be provided in the bot.

it's nice to have a key so put into the config every one of my keys

scope = 'playlist-modify-public'
token = util.prompt_for_user_token("USERNAME", scope, "3b7eba6edc674bb4a78d62c111abb0f6",
"9902c953bc9d4ab38a2343caf", "web")
sp = spotipy.Spotify(auth=token)
#DISPUTES IS JESUS ;) added checks for URI format
#URI must be in format spotify:user:USERNAME:playlist:PLAYLISTID
if URI.split(":")[1] == "user":
username = URI.split(":")[2]
playlist = URI.split(":")[4]
res = sp.user_playlist_tracks(username, playlist)
for items in res["items"]:
song = items["track"]["name"]
artist = items["track"]["artists"][0]["name"]
songToPlay = str(song) + " by " + str(artist) + " Lyrics"
await self.cmd_play(player, channel, author, permissions, leftover_args, songToPlay)
#URI must be in format spotify:track:TRACKID
elif URI.split(":")[1] == "track":
track = URI.split(":")[2]
res = sp.track(track)
song = res['name']
artist = res['artists'][0]['name']
songToPlay = str(song) + " by " + str(artist) + " Lyrics"
await self.cmd_play(player, channel, author, permissions, leftover_args, songToPlay)
#URI must be in format spotify:album:ALBUMID
elif URI.split(":")[1] == "album":
album = URI.split(":")[2]
res = sp.album_tracks(album)
for items in res["items"]:
song = items["name"]
artist = items["artists"][0]["name"]
songToPlay = str(song) + " by " + str(artist) + " Lyrics"
await self.cmd_play(player, channel, author, permissions, leftover_args, songToPlay)
    # TODO checks to see if it fails

    #TODO Check if everything was added to queue
    return Response(songToPlay+" Added")

Can you not leak the key?

it is not a good idea to implement

I've stated on multiple occasions that I will not be adding features that require additional API keys at this time.

thank you for your embarrassment


add "zh_TW" and moving "zh" to "zh_CN"

What's going on? I just add docs of Traditional Chinese......

@Sentimentron
Can you help me figure out the problem? I think that's a bug of travis. BTW, someone told me that he faced the same problem when building gin-gonic/gin without changing anything.
the problem is : every versions pass the test, while tip fail.

I've restarted the build. If tip fails again, I don't think there will be a problem merging this, because it's not a code change.

Thank a lot.

@Sentimentron It seemed to fail again...

OK, as a work-around, I've disabled testing against the latest go tip.


how to get id and name for a selected value in model

I am trying to get id and name for a selected value in single select drop down model.
this.myOptions = [       { id: 1, name: 'Option 1' },       { id: 2, name: 'Option 2' },     ];
 console.log(this.optionsModel);
for e.g if I select Option 1 from drop down i need to get id:1 and Option 1 both values in OptionsModel.

I tried many ways, I always get Id value/values, as an array of numbers, how do I get the array of names or whole objects? Ofc I can filter through this.optionsModel and search by id, but is there some "out of the box" way? With all those other fancy options I feel like there should be some "easy" way, otherwise, it's all wasted.

+1

+1

I got this done by appending the value of the name field within id itself. For my implementation, when I am building the this.myOptions array, I am populating it in this way:
this.myOptions = [ { id: 1===myValue1, name: 'myValue1' }, { id: 2===myValue2, name: 'myValue2' }, ];
and later, you can do a split("===") on the id field to get both id and value.
Not the best implementation one can use, but it works!

The thing is current implementation does return only selected id's.
I agree that selecting whole objects makes more sense but this change will be breaking.
So if we will consider implementing it the best option would be to enable it under the setting flag.
@softsimon what do you think about it?

Btw I just realized how you can easily achieve this behavior without too much "hacking" (though I did not test it yet but should work fine).
When you get your array of options, just update it once to set each item's id to itself like so:
const myOptions = [{id: 1, name: 'Option #1'},...];

// One time post-processing on options
myOptions.forEach(obj => {
  obj._id = obj.id // You might want to save original id
  obj.id = obj;
});

And in this case you will get selected items as objects and then you can access stored real id selectedOption._id


Emails are decoded

I encode emails using sites like http://www.davidegrayson.com/emailhider/ to hide my email from harvesters that do not decode emails.
This plugin decodes email for the final result, can this be disabled?

project is deprecated https://github.com/stereobooster/html_press/issues/21