Version of urllib3 and HTTPSConnection error

When I save the result of video, I have a problem:
Example:

Error:
> Finished chain.
Moviepy - Building video /data/GitHub/DriveLikeAHuman/results-video/highwayv0-episode-0.mp4.
Moviepy - Writing video /data/GitHub/DriveLikeAHuman/results-video/highwayv0-episode-0.mp4

Moviepy - Done !
Moviepy - video ready /data/GitHub/DriveLikeAHuman/results-video/highwayv0-episode-0.mp4

---------------------------------------------------------------------------
TimeoutError                              Traceback (most recent call last)
File ~/anaconda3/envs/llama/lib/python3.10/site-packages/urllib3/connection.py:203, in HTTPConnection._new_conn(self)
    202 try:
--> 203     sock = connection.create_connection(
    204         (self._dns_host, self.port),
    205         self.timeout,
    206         source_address=self.source_address,
    207         socket_options=self.socket_options,
    208     )
    209 except socket.gaierror as e:

File ~/anaconda3/envs/llama/lib/python3.10/site-packages/urllib3/util/connection.py:85, in create_connection(address, timeout, source_address, socket_options)
     84 try:
---> 85     raise err
     86 finally:
     87     # Break explicitly a reference cycle

File ~/anaconda3/envs/llama/lib/python3.10/site-packages/urllib3/util/connection.py:73, in create_connection(address, timeout, source_address, socket_options)
     72     sock.bind(source_address)
---> 73 sock.connect(sa)
     74 # Break explicitly a reference cycle

TimeoutError: [Errno 110] Connection timed out

The above exception was the direct cause of the following exception:

ConnectTimeoutError                       Traceback (most recent call last)
File ~/anaconda3/envs/llama/lib/python3.10/site-packages/urllib3/connectionpool.py:790, in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, preload_content, decode_content, **response_kw)
    789 # Make the request on the HTTPConnection object
--> 790 response = self._make_request(
    791     conn,
    792     method,
    793     url,
    794     timeout=timeout_obj,
    795     body=body,
    796     headers=headers,
    797     chunked=chunked,
    798     retries=retries,
    799     response_conn=response_conn,
    800     preload_content=preload_content,
    801     decode_content=decode_content,
    802     **response_kw,
    803 )
    805 # Everything went great!

File ~/anaconda3/envs/llama/lib/python3.10/site-packages/urllib3/connectionpool.py:491, in HTTPConnectionPool._make_request(self, conn, method, url, body, headers, retries, timeout, chunked, response_conn, preload_content, decode_content, enforce_content_length)
    490         new_e = _wrap_proxy_error(new_e, conn.proxy.scheme)
--> 491     raise new_e
    493 # conn.request() calls http.client.*.request, not the method in
    494 # urllib3.request. It also calls makefile (recv) on the socket.

File ~/anaconda3/envs/llama/lib/python3.10/site-packages/urllib3/connectionpool.py:467, in HTTPConnectionPool._make_request(self, conn, method, url, body, headers, retries, timeout, chunked, response_conn, preload_content, decode_content, enforce_content_length)
    466 try:
--> 467     self._validate_conn(conn)
    468 except (SocketTimeout, BaseSSLError) as e:

File ~/anaconda3/envs/llama/lib/python3.10/site-packages/urllib3/connectionpool.py:1092, in HTTPSConnectionPool._validate_conn(self, conn)
   1091 if conn.is_closed:
-> 1092     conn.connect()
   1094 if not conn.is_verified:

File ~/anaconda3/envs/llama/lib/python3.10/site-packages/urllib3/connection.py:611, in HTTPSConnection.connect(self)
    610 sock: socket.socket | ssl.SSLSocket
--> 611 self.sock = sock = self._new_conn()
    612 server_hostname: str = self.host

File ~/anaconda3/envs/llama/lib/python3.10/site-packages/urllib3/connection.py:212, in HTTPConnection._new_conn(self)
    211 except SocketTimeout as e:
--> 212     raise ConnectTimeoutError(
    213         self,
    214         f"Connection to {self.host} timed out. (connect timeout={self.timeout})",
    215     ) from e
    217 except OSError as e:

ConnectTimeoutError: (<urllib3.connection.HTTPSConnection object at 0x7f0c0bb01480>, 'Connection to openaipublic.blob.core.windows.net timed out. (connect timeout=None)')

The above exception was the direct cause of the following exception:

MaxRetryError                             Traceback (most recent call last)
File ~/anaconda3/envs/llama/lib/python3.10/site-packages/requests/adapters.py:486, in HTTPAdapter.send(self, request, stream, timeout, verify, cert, proxies)
    485 try:
--> 486     resp = conn.urlopen(
    487         method=request.method,
    488         url=url,
    489         body=request.body,
    490         headers=request.headers,
    491         redirect=False,
    492         assert_same_host=False,
    493         preload_content=False,
    494         decode_content=False,
    495         retries=self.max_retries,
    496         timeout=timeout,
    497         chunked=chunked,
    498     )
    500 except (ProtocolError, OSError) as err:

File ~/anaconda3/envs/llama/lib/python3.10/site-packages/urllib3/connectionpool.py:844, in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, preload_content, decode_content, **response_kw)
    842     new_e = ProtocolError("Connection aborted.", new_e)
--> 844 retries = retries.increment(
    845     method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2]
    846 )
    847 retries.sleep()

File ~/anaconda3/envs/llama/lib/python3.10/site-packages/urllib3/util/retry.py:515, in Retry.increment(self, method, url, response, error, _pool, _stacktrace)
    514     reason = error or ResponseError(cause)
--> 515     raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
    517 log.debug("Incremented Retry for (url='%s'): %r", url, new_retry)

MaxRetryError: HTTPSConnectionPool(host='openaipublic.blob.core.windows.net', port=443): Max retries exceeded with url: /encodings/cl100k_base.tiktoken (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f0c0bb01480>, 'Connection to openaipublic.blob.core.windows.net timed out. (connect timeout=None)'))

During handling of the above exception, another exception occurred:

ConnectTimeout                            Traceback (most recent call last)
File /data/GitHub/DriveLikeAHuman/HELLM.py:105
    103 while not (done or truncated):
    104     sce.upateVehicles(obs, frame)
--> 105     DA.agentRun(output)
    106     da_output = DA.exportThoughts()
    107     output = outputParser.agentRun(da_output)

File /data/GitHub/DriveLikeAHuman/LLMDriver/driverAgent.py:72, in DriverAgent.agentRun(self, last_step_decision)
     70     last_step_explanation = "Not available"
     71 with get_openai_callback() as cb:
---> 72     self.agent.run(
     73         f"""
     74         You, the 'ego' car, are now driving a car on a highway. You have already drive for {self.sce.frame} seconds.
     75         The decision you made LAST time step was `{last_step_action}`. Your explanation was `{last_step_explanation}`.
     76         Here is the current scenario: \n ```json\n{self.sce.export2json()}\n```\n.
     77         Please make decision for the `ego` car. You have to describe the state of the `ego`, then analyze the possible actions, and finally output your decision.
     78
     79         There are several rules you need to follow when you drive on a highway:
     80         {TRAFFIC_RULES}
     81
     82         Here are your attentions points:
     83         {DECISION_CAUTIONS}
     84
     85         Let's think step by step. Once you made a final decision, output it in the following format: \n
     86         ```
     87         Final Answer:
     88             "decision":{{"ego car's decision, ONE of the available actions"}},
     89             "expalanations":{{"your explaination about your decision, described your suggestions to the driver"}}
     90         ``` \n
     91         """,
     92         callbacks=[self.ch]
     93     )
     94 print(cb)
     95 print('[cyan]Final decision:[/cyan]')

File ~/anaconda3/envs/llama/lib/python3.10/site-packages/langchain/chains/base.py:315, in Chain.run(self, callbacks, tags, metadata, *args, **kwargs)
    313     if len(args) != 1:
    314         raise ValueError("`run` supports only one positional argument.")
--> 315     return self(args[0], callbacks=callbacks, tags=tags, metadata=metadata)[
    316         _output_key
    317     ]
    319 if kwargs and not args:
    320     return self(kwargs, callbacks=callbacks, tags=tags, metadata=metadata)[
    321         _output_key
    322     ]

File ~/anaconda3/envs/llama/lib/python3.10/site-packages/langchain/chains/base.py:183, in Chain.__call__(self, inputs, return_only_outputs, callbacks, tags, metadata, include_run_info)
    181     raise e
    182 run_manager.on_chain_end(outputs)
--> 183 final_outputs: Dict[str, Any] = self.prep_outputs(
    184     inputs, outputs, return_only_outputs
    185 )
    186 if include_run_info:
    187     final_outputs[RUN_KEY] = RunInfo(run_id=run_manager.run_id)

File ~/anaconda3/envs/llama/lib/python3.10/site-packages/langchain/chains/base.py:257, in Chain.prep_outputs(self, inputs, outputs, return_only_outputs)
    255 self._validate_outputs(outputs)
    256 if self.memory is not None:
--> 257     self.memory.save_context(inputs, outputs)
    258 if return_only_outputs:
    259     return outputs

File ~/anaconda3/envs/llama/lib/python3.10/site-packages/langchain/memory/token_buffer.py:48, in ConversationTokenBufferMemory.save_context(self, inputs, outputs)
     46 # Prune buffer if it exceeds max token limit
     47 buffer = self.chat_memory.messages
---> 48 curr_buffer_length = self.llm.get_num_tokens_from_messages(buffer)
     49 if curr_buffer_length > self.max_token_limit:
     50     pruned_memory = []

File ~/anaconda3/envs/llama/lib/python3.10/site-packages/langchain/chat_models/openai.py:499, in ChatOpenAI.get_num_tokens_from_messages(self, messages)
    497 if sys.version_info[1] <= 7:
    498     return super().get_num_tokens_from_messages(messages)
--> 499 model, encoding = self._get_encoding_model()
    500 if model.startswith("gpt-3.5-turbo"):
    501     # every message follows <im_start>{role/name}\n{content}<im_end>\n
    502     tokens_per_message = 4

File ~/anaconda3/envs/llama/lib/python3.10/site-packages/langchain/chat_models/openai.py:477, in ChatOpenAI._get_encoding_model(self)
    475 # Returns the number of tokens used by a list of messages.
    476 try:
--> 477     encoding = tiktoken_.encoding_for_model(model)
    478 except KeyError:
    479     logger.warning("Warning: model not found. Using cl100k_base encoding.")

File ~/anaconda3/envs/llama/lib/python3.10/site-packages/tiktoken/model.py:67, in encoding_for_model(model_name)
     65     for model_prefix, model_encoding_name in MODEL_PREFIX_TO_ENCODING.items():
     66         if model_name.startswith(model_prefix):
---> 67             return get_encoding(model_encoding_name)
     69 if encoding_name is None:
     70     raise KeyError(
     71         f"Could not automatically map {model_name} to a tokeniser. "
     72         "Please use `tiktok.get_encoding` to explicitly get the tokeniser you expect."
     73     ) from None

File ~/anaconda3/envs/llama/lib/python3.10/site-packages/tiktoken/registry.py:63, in get_encoding(encoding_name)
     60     raise ValueError(f"Unknown encoding {encoding_name}")
     62 constructor = ENCODING_CONSTRUCTORS[encoding_name]
---> 63 enc = Encoding(**constructor())
     64 ENCODINGS[encoding_name] = enc
     65 return enc

File ~/anaconda3/envs/llama/lib/python3.10/site-packages/tiktoken_ext/openai_public.py:64, in cl100k_base()
     63 def cl100k_base():
---> 64     mergeable_ranks = load_tiktoken_bpe(
     65         "https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken"
     66     )
     67     special_tokens = {
     68         ENDOFTEXT: 100257,
     69         FIM_PREFIX: 100258,
   (...)
     72         ENDOFPROMPT: 100276,
     73     }
     74     return {
     75         "name": "cl100k_base",
     76         "pat_str": r"""(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}{1,3}| ?[^\s\p{L}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+""",
     77         "mergeable_ranks": mergeable_ranks,
     78         "special_tokens": special_tokens,
     79     }

File ~/anaconda3/envs/llama/lib/python3.10/site-packages/tiktoken/load.py:116, in load_tiktoken_bpe(tiktoken_bpe_file)
    114 def load_tiktoken_bpe(tiktoken_bpe_file: str) -> dict[bytes, int]:
    115     # NB: do not add caching to this function
--> 116     contents = read_file_cached(tiktoken_bpe_file)
    117     return {
    118         base64.b64decode(token): int(rank)
    119         for token, rank in (line.split() for line in contents.splitlines() if line)
    120     }

File ~/anaconda3/envs/llama/lib/python3.10/site-packages/tiktoken/load.py:48, in read_file_cached(blobpath)
     45     with open(cache_path, "rb") as f:
     46         return f.read()
---> 48 contents = read_file(blobpath)
     50 os.makedirs(cache_dir, exist_ok=True)
     51 tmp_filename = cache_path + "." + str(uuid.uuid4()) + ".tmp"

File ~/anaconda3/envs/llama/lib/python3.10/site-packages/tiktoken/load.py:24, in read_file(blobpath)
     22         return f.read()
     23 # avoiding blobfile for public files helps avoid auth issues, like MFA prompts
---> 24 resp = requests.get(blobpath)
     25 resp.raise_for_status()
     26 return resp.content

File ~/anaconda3/envs/llama/lib/python3.10/site-packages/requests/api.py:73, in get(url, params, **kwargs)
     62 def get(url, params=None, **kwargs):
     63     r"""Sends a GET request.
     64
     65     :param url: URL for the new :class:`Request` object.
   (...)
     70     :rtype: requests.Response
     71     """
---> 73     return request("get", url, params=params, **kwargs)

File ~/anaconda3/envs/llama/lib/python3.10/site-packages/requests/api.py:59, in request(method, url, **kwargs)
     55 # By using the 'with' statement we are sure the session is closed, thus we
     56 # avoid leaving sockets open which can trigger a ResourceWarning in some
     57 # cases, and look like a memory leak in others.
     58 with sessions.Session() as session:
---> 59     return session.request(method=method, url=url, **kwargs)

File ~/anaconda3/envs/llama/lib/python3.10/site-packages/requests/sessions.py:589, in Session.request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
    584 send_kwargs = {
    585     "timeout": timeout,
    586     "allow_redirects": allow_redirects,
    587 }
    588 send_kwargs.update(settings)
--> 589 resp = self.send(prep, **send_kwargs)
    591 return resp

File ~/anaconda3/envs/llama/lib/python3.10/site-packages/requests/sessions.py:703, in Session.send(self, request, **kwargs)
    700 start = preferred_clock()
    702 # Send the request
--> 703 r = adapter.send(request, **kwargs)
    705 # Total elapsed time of the request (approximately)
    706 elapsed = preferred_clock() - start

File ~/anaconda3/envs/llama/lib/python3.10/site-packages/requests/adapters.py:507, in HTTPAdapter.send(self, request, stream, timeout, verify, cert, proxies)
    504 if isinstance(e.reason, ConnectTimeoutError):
    505     # TODO: Remove this in 3.0.0: see #2811
    506     if not isinstance(e.reason, NewConnectionError):
--> 507         raise ConnectTimeout(e, request=request)
    509 if isinstance(e.reason, ResponseError):
    510     raise RetryError(e, request=request)

ConnectTimeout: HTTPSConnectionPool(host='openaipublic.blob.core.windows.net', port=443): Max retries exceeded with url: /encodings/cl100k_base.tiktoken (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f0c0bb01480>, 'Connection to openaipublic.blob.core.windows.net timed out. (connect timeout=None)'))

MY Version:
python==3.10
urllib3==2.0.4
moviepy==1.0.3
tiktoken==0.4.0

I guess it may be a problem with the urllib3 version, my current version is urllib3==2.0.4,
I would like to ask your the version of urllib3 , python, moviepy and tiktoken.
Thank you.

Hello, I noticed you're using LLama instead of our original GPT-3/4 in the repo.
The error you posted, especially the last line mentioning a connection timeout to openaipublic.blob.core.windows.net, seems like a network issue rather than a problem with saving movies. Please verify your network connection to the LLama server.

Thank you for your reply.

LLama is just the Conda environment name. The model I used was "gpt-3.5-turbo-16k-0613".  And I can use GPT3.5 directly through the proxy.
Some people experienced similar problems and changed the urllib3 version from 2.0.4 to 1.25.11 https://github.com/Significant-Gravitas/Auto-GPT/issues/1909

For your information, My current urllib3 version is 1.26.15.

Thank you very much.
Finally, I solved the problem by specifying urllib3==1.26.15 and tiktoken==0.4.0;

Still experiencing this issue someone please help out with the same. Tried matching the versions of tiktoken and urllib3 to the versions above and Im trying to count tokens in my prompt using vscode locally.

I have the same problem here.
I have already aligned my environment to the suggested with
 !pip install urllib3==1.26.15
!pip install tiktoken==0.4.0

TimeoutError                              Traceback (most recent call last)
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/urllib3/connection.py:174, in HTTPConnection._new_conn(self)
173 try:
--> 174     conn = connection.create_connection(
175         (self._dns_host, self.port), self.timeout, **extra_kw
176     )
178 except SocketTimeout:
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/urllib3/util/connection.py:95, in create_connection(address, timeout, source_address, socket_options)
94 if err is not None:
---> 95     raise err
97 raise socket.error("getaddrinfo returns an empty list")
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/urllib3/util/connection.py:85, in create_connection(address, timeout, source_address, socket_options)
84     sock.bind(source_address)
---> 85 sock.connect(sa)
86 return sock
TimeoutError: [Errno 110] Connection timed out
During handling of the above exception, another exception occurred:
NewConnectionError                        Traceback (most recent call last)
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/urllib3/connectionpool.py:715, in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
714 # Make the request on the httplib connection object.
--> 715 httplib_response = self._make_request(
716     conn,
717     method,
718     url,
719     timeout=timeout_obj,
720     body=body,
721     headers=headers,
722     chunked=chunked,
723 )
725 # If we're going to release the connection in finally:, then
726 # the response doesn't need to know about the connection. Otherwise
727 # it will also try to release it and we'll have a double-release
728 # mess.
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/urllib3/connectionpool.py:404, in HTTPConnectionPool._make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw)
403 try:
--> 404     self._validate_conn(conn)
405 except (SocketTimeout, BaseSSLError) as e:
406     # Py2 raises this as a BaseSSLError, Py3 raises it as socket timeout.
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/urllib3/connectionpool.py:1058, in HTTPSConnectionPool._validate_conn(self, conn)
1057 if not getattr(conn, "sock", None):  # AppEngine might not have  .sock
-> 1058     conn.connect()
1060 if not conn.is_verified:
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/urllib3/connection.py:363, in HTTPSConnection.connect(self)
361 def connect(self):
362     # Add certificate verification
--> 363     self.sock = conn = self._new_conn()
364     hostname = self.host
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/urllib3/connection.py:186, in HTTPConnection._new_conn(self)
185 except SocketError as e:
--> 186     raise NewConnectionError(
187         self, "Failed to establish a new connection: %s" % e
188     )
190 return conn
NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f64568f1fd0>: Failed to establish a new connection: [Errno 110] Connection timed out
During handling of the above exception, another exception occurred:
MaxRetryError                             Traceback (most recent call last)
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/requests/adapters.py:486, in HTTPAdapter.send(self, request, stream, timeout, verify, cert, proxies)
485 try:
--> 486     resp = conn.urlopen(
487         method=request.method,
488         url=url,
489         body=request.body,
490         headers=request.headers,
491         redirect=False,
492         assert_same_host=False,
493         preload_content=False,
494         decode_content=False,
495         retries=self.max_retries,
496         timeout=timeout,
497         chunked=chunked,
498     )
500 except (ProtocolError, OSError) as err:
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/urllib3/connectionpool.py:799, in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
797     e = ProtocolError("Connection aborted.", e)
--> 799 retries = retries.increment(
800     method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
801 )
802 retries.sleep()
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/urllib3/util/retry.py:592, in Retry.increment(self, method, url, response, error, _pool, _stacktrace)
591 if new_retry.is_exhausted():
--> 592     raise MaxRetryError(_pool, url, error or ResponseError(cause))
594 log.debug("Incremented Retry for (url='%s'): %r", url, new_retry)
MaxRetryError: HTTPSConnectionPool(host='openaipublic.blob.core.windows.net', port=443): Max retries exceeded with url: /encodings/cl100k_base.tiktoken (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f64568f1fd0>: Failed to establish a new connection: [Errno 110] Connection timed out'))
During handling of the above exception, another exception occurred:
ConnectionError                           Traceback (most recent call last)
Cell In[24], line 15
11 # Update this line based on the type of client you are using
12 # For example, for a persistent client:
13 client = chromadb.PersistentClient(path=f"{path_for_directory_of_pdf}_db")
---> 15 vectorstore = Chroma.from_documents(
16     documents, embeddings, client=client  # Pass the client here
17 )
19 # saving the current state of the vectorstore object to disk.
20 vectorstore.persist()
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/langchain/vectorstores/chroma.py:462, in Chroma.from_documents(cls, documents, embedding, ids, collection_name, persist_directory, client_settings, client, **kwargs)
460 texts = [doc.page_content for doc in documents]
461 metadatas = [doc.metadata for doc in documents]
--> 462 return cls.from_texts(
463     texts=texts,
464     embedding=embedding,
465     metadatas=metadatas,
466     ids=ids,
467     collection_name=collection_name,
468     persist_directory=persist_directory,
469     client_settings=client_settings,
470     client=client,
471 )
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/langchain/vectorstores/chroma.py:430, in Chroma.from_texts(cls, texts, embedding, metadatas, ids, collection_name, persist_directory, client_settings, client, **kwargs)
406 """Create a Chroma vectorstore from a raw documents.
407
408 If a persist_directory is specified, the collection will be persisted there.
(...)
421     Chroma: Chroma vectorstore.
422 """
423 chroma_collection = cls(
424     collection_name=collection_name,
425     embedding_function=embedding,
(...)
428     client=client,
429 )
--> 430 chroma_collection.add_texts(texts=texts, metadatas=metadatas, ids=ids)
431 return chroma_collection
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/langchain/vectorstores/chroma.py:149, in Chroma.add_texts(self, texts, metadatas, ids, **kwargs)
147 embeddings = None
148 if self._embedding_function is not None:
--> 149     embeddings = self._embedding_function.embed_documents(list(texts))
150 self._collection.upsert(
151     metadatas=metadatas, embeddings=embeddings, documents=texts, ids=ids
152 )
153 return ids
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/langchain/embeddings/openai.py:452, in OpenAIEmbeddings.embed_documents(self, texts, chunk_size)
440 """Call out to OpenAI's embedding endpoint for embedding search docs.
441
442 Args:
(...)
448     List of embeddings, one for each text.
449 """
450 # NOTE: to keep things simple, we assume the list may contain texts longer
451 #       than the maximum context and use length-safe embedding function.
--> 452 return self._get_len_safe_embeddings(texts, engine=self.deployment)
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/langchain/embeddings/openai.py:280, in OpenAIEmbeddings._get_len_safe_embeddings(self, texts, engine, chunk_size)
278 model_name = self.tiktoken_model_name or self.model
279 try:
--> 280     encoding = tiktoken.encoding_for_model(model_name)
281 except KeyError:
282     logger.warning("Warning: model not found. Using cl100k_base encoding.")
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/tiktoken/model.py:75, in encoding_for_model(model_name)
69 if encoding_name is None:
70     raise KeyError(
71         f"Could not automatically map {model_name} to a tokeniser. "
72         "Please use tiktok.get_encoding to explicitly get the tokeniser you expect."
73     ) from None
---> 75 return get_encoding(encoding_name)
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/tiktoken/registry.py:63, in get_encoding(encoding_name)
60     raise ValueError(f"Unknown encoding {encoding_name}")
62 constructor = ENCODING_CONSTRUCTORS[encoding_name]
---> 63 enc = Encoding(**constructor())
64 ENCODINGS[encoding_name] = enc
65 return enc
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/tiktoken_ext/openai_public.py:64, in cl100k_base()
63 def cl100k_base():
---> 64     mergeable_ranks = load_tiktoken_bpe(
65         "https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken"
66     )
67     special_tokens = {
68         ENDOFTEXT: 100257,
69         FIM_PREFIX: 100258,
(...)
72         ENDOFPROMPT: 100276,
73     }
74     return {
75         "name": "cl100k_base",
76         "pat_str": r"""(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}{1,3}| ?[^\s\p{L}\p{N}]+[\r\n]|\s[\r\n]+|\s+(?!\S)|\s+""",
77         "mergeable_ranks": mergeable_ranks,
78         "special_tokens": special_tokens,
79     }
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/tiktoken/load.py:116, in load_tiktoken_bpe(tiktoken_bpe_file)
114 def load_tiktoken_bpe(tiktoken_bpe_file: str) -> dict[bytes, int]:
115     # NB: do not add caching to this function
--> 116     contents = read_file_cached(tiktoken_bpe_file)
117     return {
118         base64.b64decode(token): int(rank)
119         for token, rank in (line.split() for line in contents.splitlines() if line)
120     }
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/tiktoken/load.py:48, in read_file_cached(blobpath)
45     with open(cache_path, "rb") as f:
46         return f.read()
---> 48 contents = read_file(blobpath)
50 os.makedirs(cache_dir, exist_ok=True)
51 tmp_filename = cache_path + "." + str(uuid.uuid4()) + ".tmp"
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/tiktoken/load.py:24, in read_file(blobpath)
22         return f.read()
23 # avoiding blobfile for public files helps avoid auth issues, like MFA prompts
---> 24 resp = requests.get(blobpath)
25 resp.raise_for_status()
26 return resp.content
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/requests/api.py:73, in get(url, params, **kwargs)
62 def get(url, params=None, **kwargs):
63     r"""Sends a GET request.
64
65     :param url: URL for the new :class:Request object.
(...)
70     :rtype: requests.Response
71     """
---> 73     return request("get", url, params=params, **kwargs)
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/requests/api.py:59, in request(method, url, **kwargs)
55 # By using the 'with' statement we are sure the session is closed, thus we
56 # avoid leaving sockets open which can trigger a ResourceWarning in some
57 # cases, and look like a memory leak in others.
58 with sessions.Session() as session:
---> 59     return session.request(method=method, url=url, **kwargs)
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/requests/sessions.py:589, in Session.request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
584 send_kwargs = {
585     "timeout": timeout,
586     "allow_redirects": allow_redirects,
587 }
588 send_kwargs.update(settings)
--> 589 resp = self.send(prep, **send_kwargs)
591 return resp
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/requests/sessions.py:703, in Session.send(self, request, **kwargs)
700 start = preferred_clock()
702 # Send the request
--> 703 r = adapter.send(request, **kwargs)
705 # Total elapsed time of the request (approximately)
706 elapsed = preferred_clock() - start
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/requests/adapters.py:519, in HTTPAdapter.send(self, request, stream, timeout, verify, cert, proxies)
515     if isinstance(e.reason, _SSLError):
516         # This branch is for urllib3 v1.22 and later.
517         raise SSLError(e, request=request)
--> 519     raise ConnectionError(e, request=request)
521 except ClosedPoolError as e:
522     raise ConnectionError(e, request=request)
ConnectionError: HTTPSConnectionPool(host='openaipublic.blob.core.windows.net', port=443): Max retries exceeded with url: /encodings/cl100k_base.tiktoken (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f64568f1fd0>: Failed to establish a new connection: [Errno 110] Connection timed out'))

I have the same problem here. I have already aligned my environment to the suggested with
 !pip install urllib3==1.26.15
!pip install tiktoken==0.4.0

TimeoutError Traceback (most recent call last) File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/urllib3/connection.py:174, in HTTPConnection._new_conn(self) 173 try: --> 174 conn = connection.create_connection( 175 (self._dns_host, self.port), self.timeout, **extra_kw 176 ) 178 except SocketTimeout:
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/urllib3/util/connection.py:95, in create_connection(address, timeout, source_address, socket_options) 94 if err is not None: ---> 95 raise err 97 raise socket.error("getaddrinfo returns an empty list")
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/urllib3/util/connection.py:85, in create_connection(address, timeout, source_address, socket_options) 84 sock.bind(source_address) ---> 85 sock.connect(sa) 86 return sock
TimeoutError: [Errno 110] Connection timed out
During handling of the above exception, another exception occurred:
NewConnectionError Traceback (most recent call last) File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/urllib3/connectionpool.py:715, in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw) 714 # Make the request on the httplib connection object. --> 715 httplib_response = self._make_request( 716 conn, 717 method, 718 url, 719 timeout=timeout_obj, 720 body=body, 721 headers=headers, 722 chunked=chunked, 723 ) 725 # If we're going to release the connection in finally:, then 726 # the response doesn't need to know about the connection. Otherwise 727 # it will also try to release it and we'll have a double-release 728 # mess.
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/urllib3/connectionpool.py:404, in HTTPConnectionPool._make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw) 403 try: --> 404 self._validate_conn(conn) 405 except (SocketTimeout, BaseSSLError) as e: 406 # Py2 raises this as a BaseSSLError, Py3 raises it as socket timeout.
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/urllib3/connectionpool.py:1058, in HTTPSConnectionPool._validate_conn(self, conn) 1057 if not getattr(conn, "sock", None): # AppEngine might not have .sock -> 1058 conn.connect() 1060 if not conn.is_verified:
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/urllib3/connection.py:363, in HTTPSConnection.connect(self) 361 def connect(self): 362 # Add certificate verification --> 363 self.sock = conn = self._new_conn() 364 hostname = self.host
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/urllib3/connection.py:186, in HTTPConnection._new_conn(self) 185 except SocketError as e: --> 186 raise NewConnectionError( 187 self, "Failed to establish a new connection: %s" % e 188 ) 190 return conn
NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f64568f1fd0>: Failed to establish a new connection: [Errno 110] Connection timed out
During handling of the above exception, another exception occurred:
MaxRetryError Traceback (most recent call last) File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/requests/adapters.py:486, in HTTPAdapter.send(self, request, stream, timeout, verify, cert, proxies) 485 try: --> 486 resp = conn.urlopen( 487 method=request.method, 488 url=url, 489 body=request.body, 490 headers=request.headers, 491 redirect=False, 492 assert_same_host=False, 493 preload_content=False, 494 decode_content=False, 495 retries=self.max_retries, 496 timeout=timeout, 497 chunked=chunked, 498 ) 500 except (ProtocolError, OSError) as err:
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/urllib3/connectionpool.py:799, in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw) 797 e = ProtocolError("Connection aborted.", e) --> 799 retries = retries.increment( 800 method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] 801 ) 802 retries.sleep()
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/urllib3/util/retry.py:592, in Retry.increment(self, method, url, response, error, _pool, _stacktrace) 591 if new_retry.is_exhausted(): --> 592 raise MaxRetryError(_pool, url, error or ResponseError(cause)) 594 log.debug("Incremented Retry for (url='%s'): %r", url, new_retry)
MaxRetryError: HTTPSConnectionPool(host='openaipublic.blob.core.windows.net', port=443): Max retries exceeded with url: /encodings/cl100k_base.tiktoken (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f64568f1fd0>: Failed to establish a new connection: [Errno 110] Connection timed out'))
During handling of the above exception, another exception occurred:
ConnectionError Traceback (most recent call last) Cell In[24], line 15 11 # Update this line based on the type of client you are using 12 # For example, for a persistent client: 13 client = chromadb.PersistentClient(path=f"{path_for_directory_of_pdf}_db") ---> 15 vectorstore = Chroma.from_documents( 16 documents, embeddings, client=client # Pass the client here 17 ) 19 # saving the current state of the vectorstore object to disk. 20 vectorstore.persist()
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/langchain/vectorstores/chroma.py:462, in Chroma.from_documents(cls, documents, embedding, ids, collection_name, persist_directory, client_settings, client, **kwargs) 460 texts = [doc.page_content for doc in documents] 461 metadatas = [doc.metadata for doc in documents] --> 462 return cls.from_texts( 463 texts=texts, 464 embedding=embedding, 465 metadatas=metadatas, 466 ids=ids, 467 collection_name=collection_name, 468 persist_directory=persist_directory, 469 client_settings=client_settings, 470 client=client, 471 )
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/langchain/vectorstores/chroma.py:430, in Chroma.from_texts(cls, texts, embedding, metadatas, ids, collection_name, persist_directory, client_settings, client, **kwargs) 406 """Create a Chroma vectorstore from a raw documents. 407 408 If a persist_directory is specified, the collection will be persisted there. (...) 421 Chroma: Chroma vectorstore. 422 """ 423 chroma_collection = cls( 424 collection_name=collection_name, 425 embedding_function=embedding, (...) 428 client=client, 429 ) --> 430 chroma_collection.add_texts(texts=texts, metadatas=metadatas, ids=ids) 431 return chroma_collection
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/langchain/vectorstores/chroma.py:149, in Chroma.add_texts(self, texts, metadatas, ids, **kwargs) 147 embeddings = None 148 if self._embedding_function is not None: --> 149 embeddings = self._embedding_function.embed_documents(list(texts)) 150 self._collection.upsert( 151 metadatas=metadatas, embeddings=embeddings, documents=texts, ids=ids 152 ) 153 return ids
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/langchain/embeddings/openai.py:452, in OpenAIEmbeddings.embed_documents(self, texts, chunk_size) 440 """Call out to OpenAI's embedding endpoint for embedding search docs. 441 442 Args: (...) 448 List of embeddings, one for each text. 449 """ 450 # NOTE: to keep things simple, we assume the list may contain texts longer 451 # than the maximum context and use length-safe embedding function. --> 452 return self._get_len_safe_embeddings(texts, engine=self.deployment)
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/langchain/embeddings/openai.py:280, in OpenAIEmbeddings._get_len_safe_embeddings(self, texts, engine, chunk_size) 278 model_name = self.tiktoken_model_name or self.model 279 try: --> 280 encoding = tiktoken.encoding_for_model(model_name) 281 except KeyError: 282 logger.warning("Warning: model not found. Using cl100k_base encoding.")
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/tiktoken/model.py:75, in encoding_for_model(model_name) 69 if encoding_name is None: 70 raise KeyError( 71 f"Could not automatically map {model_name} to a tokeniser. " 72 "Please use tiktok.get_encoding to explicitly get the tokeniser you expect." 73 ) from None ---> 75 return get_encoding(encoding_name)
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/tiktoken/registry.py:63, in get_encoding(encoding_name) 60 raise ValueError(f"Unknown encoding {encoding_name}") 62 constructor = ENCODING_CONSTRUCTORS[encoding_name] ---> 63 enc = Encoding(**constructor()) 64 ENCODINGS[encoding_name] = enc 65 return enc
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/tiktoken_ext/openai_public.py:64, in cl100k_base() 63 def cl100k_base(): ---> 64 mergeable_ranks = load_tiktoken_bpe( 65 "https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken" 66 ) 67 special_tokens = { 68 ENDOFTEXT: 100257, 69 FIM_PREFIX: 100258, (...) 72 ENDOFPROMPT: 100276, 73 } 74 return { 75 "name": "cl100k_base", 76 "pat_str": r"""(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}{1,3}| ?[^\s\p{L}\p{N}]+[\r\n]|\s[\r\n]+|\s+(?!\S)|\s+""", 77 "mergeable_ranks": mergeable_ranks, 78 "special_tokens": special_tokens, 79 }
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/tiktoken/load.py:116, in load_tiktoken_bpe(tiktoken_bpe_file) 114 def load_tiktoken_bpe(tiktoken_bpe_file: str) -> dict[bytes, int]: 115 # NB: do not add caching to this function --> 116 contents = read_file_cached(tiktoken_bpe_file) 117 return { 118 base64.b64decode(token): int(rank) 119 for token, rank in (line.split() for line in contents.splitlines() if line) 120 }
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/tiktoken/load.py:48, in read_file_cached(blobpath) 45 with open(cache_path, "rb") as f: 46 return f.read() ---> 48 contents = read_file(blobpath) 50 os.makedirs(cache_dir, exist_ok=True) 51 tmp_filename = cache_path + "." + str(uuid.uuid4()) + ".tmp"
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/tiktoken/load.py:24, in read_file(blobpath) 22 return f.read() 23 # avoiding blobfile for public files helps avoid auth issues, like MFA prompts ---> 24 resp = requests.get(blobpath) 25 resp.raise_for_status() 26 return resp.content
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/requests/api.py:73, in get(url, params, **kwargs) 62 def get(url, params=None, **kwargs): 63 r"""Sends a GET request. 64 65 :param url: URL for the new :class:Request object. (...) 70 :rtype: requests.Response 71 """ ---> 73 return request("get", url, params=params, **kwargs)
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/requests/api.py:59, in request(method, url, **kwargs) 55 # By using the 'with' statement we are sure the session is closed, thus we 56 # avoid leaving sockets open which can trigger a ResourceWarning in some 57 # cases, and look like a memory leak in others. 58 with sessions.Session() as session: ---> 59 return session.request(method=method, url=url, **kwargs)
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/requests/sessions.py:589, in Session.request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json) 584 send_kwargs = { 585 "timeout": timeout, 586 "allow_redirects": allow_redirects, 587 } 588 send_kwargs.update(settings) --> 589 resp = self.send(prep, **send_kwargs) 591 return resp
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/requests/sessions.py:703, in Session.send(self, request, **kwargs) 700 start = preferred_clock() 702 # Send the request --> 703 r = adapter.send(request, **kwargs) 705 # Total elapsed time of the request (approximately) 706 elapsed = preferred_clock() - start
File /opt/saturncloud/envs/saturn/lib/python3.9/site-packages/requests/adapters.py:519, in HTTPAdapter.send(self, request, stream, timeout, verify, cert, proxies) 515 if isinstance(e.reason, _SSLError): 516 # This branch is for urllib3 v1.22 and later. 517 raise SSLError(e, request=request) --> 519 raise ConnectionError(e, request=request) 521 except ClosedPoolError as e: 522 raise ConnectionError(e, request=request)
ConnectionError: HTTPSConnectionPool(host='openaipublic.blob.core.windows.net', port=443): Max retries exceeded with url: /encodings/cl100k_base.tiktoken (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f64568f1fd0>: Failed to establish a new connection: [Errno 110] Connection timed out'))

Hi! The same problem here with SaturnCloud. Have you found a solution?


Segment Geometry

A geometry for line segments/vectors/arrows. There is an example in the file geom_segment.md. This feature was previously requested in Issues #608 and #791. My current code will work for regular plots, but not yet for subplot_grid.  I'll attempt to implement that sometime soon!

Hey @Mattriks, if you have time, can you rebase this on the latest master? Lets check if it works with all the recent changes and make sure it's all okay on 0.5. I think this would be a cool geometry to have.

Coverage decreased (-0.4%) to 65.072% when pulling 05d8ef1218372763d79747408e3bfd383a9e7c51 on Mattriks:Geom_segment into 4a3683797227463b2a5bb4f736ce64fc19fd016d on dcjones:master.

Coverage decreased (-0.4%) to 65.072% when pulling 05d8ef1218372763d79747408e3bfd383a9e7c51 on Mattriks:Geom_segment into 4a3683797227463b2a5bb4f736ce64fc19fd016d on dcjones:master.

Coverage decreased (-0.4%) to 65.072% when pulling ef675b356410a833cc774e67f8465583a8449cf5 on Mattriks:Geom_segment into 4a3683797227463b2a5bb4f736ce64fc19fd016d on dcjones:master.

Coverage decreased (-0.4%) to 65.072% when pulling ef675b356410a833cc774e67f8465583a8449cf5 on Mattriks:Geom_segment into 4a3683797227463b2a5bb4f736ce64fc19fd016d on dcjones:master.

Coverage decreased (-0.4%) to 65.072% when pulling ef675b356410a833cc774e67f8465583a8449cf5 on Mattriks:Geom_segment into 4a3683797227463b2a5bb4f736ce64fc19fd016d on dcjones:master.

I've tested this now  on Julia 0.5-rc2. The example in the file geom_segment.md works.  Also, this Segment Geometry works when using with Geom.subplot_grid.

Just note: When i click on the issue, i first get

We went looking everywhere, but couldn’t find those commits.
Sometimes commits can disappear after a force-push. Head back to the latest changes here.

@Mattriks Awesome. Could you add some tests?

I got to play around with this and looks good!
The only thing is that the arrowheads when calling Geom.vector look a little funky:
plot(x=rand(1:10, 10), y=rand(1:10, 10), xend=rand(1:10, 10), yend=rand(1:10, 10), Geom.vector)

And they don't behave properly when one of the axes is logged

Actually, the first one is caused by not having a square axis ratio. But in general, I would think that you don't want scaling and transformations to apply to the arrowheads.

I've implemented a new and improved Geom.vector (I've also included tests). In Gadfly, the current situation is that no information about Scales is passed to the geometry render functions. But for rendering vectors properly, info about the x and y scales must be provided. That is, the user must manually provide a Scale object (with a minvalue and maxvalue) for both axes. I will look into automating this in the future. Here is a current working example:
srand(1234)
D = convert(DataFrame, 99*rand(4, 4)+0.5)
xsc  = Scale.x_continuous(minvalue=0.0, maxvalue=100)
xlogsc  = Scale.x_log10(minvalue=1, maxvalue=100)
ysc  = Scale.y_continuous(minvalue=0, maxvalue=100)
xsqrtsc  = Scale.x_sqrt(minvalue=0, maxvalue=100)

pa = plot(D, x=:x1, y=:x2, xend=:x3, yend=:x4, Geom.vector, xsc, ysc,
Guide.title("Plain scales"), Guide.ylabel(nothing), Guide.xlabel(" "))
pb = plot(D, x=:x1, y=:x2, xend=:x3, yend=:x4, Geom.vector, xlogsc, ysc,
Guide.title("Semi-log"), Guide.ylabel(nothing), Guide.xlabel("x"))
pc = plot(D, x=:x1, y=:x2, xend=:x3, yend=:x4, Geom.vector, xsqrtsc, ysc,
Guide.title("√x"), Guide.ylabel(nothing), Guide.xlabel(" "))

Coverage increased (+0.3%) to 65.863% when pulling 1670e747aaa7f7354f97de7ffbbc8b11db49b4fb on Mattriks:Geom_segment into f6183acb6015c8e133e335d32227f551056c59f8 on dcjones:master.

Very nice. What do you think about filled-in arrowheads? Whenever I think of vector fields, I always think of something like this:

(https://en.wikipedia.org/wiki/Vector_field)

Some extra attributes (example below). Wrt gradient vectorfields, that could be implemented as function Geom.gradvectorfield(z, x, y) (returning a SegmentGeometry), just as Geom.contour(z,x,y) returns a LineGeometry. That's a feature for the future, need to get Geom.segment merged first.
srand(123)
D = convert(DataFrame, 99*rand(4, 4)+0.5)
D[:col] = ["a","b","c","d"]

xsc  = Scale.x_continuous(minvalue=0.0, maxvalue=100)
ysc  = Scale.y_continuous(minvalue=0.0, maxvalue=100)
theme1 = Theme(line_width=3pt, line_style=[4mm,1mm], key_position=:none)
lf(x) = layer(x=:x1, y=:x2, xend=:x3, yend=:x4, x, color=:col)

pa = plot(D, lf(Geom.vector), xsc, ysc, Theme(key_position=:none))
pb = plot(D, lf(Geom.vector(filled=true)), xsc, ysc, theme1)

Coverage increased (+0.3%) to 65.864% when pulling 9adbbf906ed23d984bd1d3ce22a67d1a9ded2ea6 on Mattriks:Geom_segment into 30670b35298222a4d7d2fc651382c745118ec10b on dcjones:master.

That looks awesome. Personally, I like filled arrows to be the default, but that can changed in the future.
I've looked through this and I think it all looks good. LGTM. @shashi what do you think?

Scales are needed to stop arrowheads looking strange. Segments without arrowheads are not affected  i.e.  Geom.segment(arrow=false)

I see. I guess eventually we'll want to get rid of that when we pass Scales to render.

Alright. LGTM.


Reusable Zephyr setup GitHub workflow + multiple platform CI testing

Introduction
Hi, some time ago I've put together a reusable github action to prepare a stock GitHub CI image to build Zephyr projects. It really started as an exercise to learn how to use the workflow, then to figure out how fast it would be to prepare a Zephyr environment for building a single application from scratch, and lastly I added multi-OS support as well.
The thing turns out to work pretty decently and I'm thinking it may be nice for the project to support it officially, to facilitate running a CI on GitHub for end users.
Proposed change
The first proposal is to move over the project under the Zephyr GitHub organization: that is creating a new repository for it under the org and then move over the code from https://github.com/fabiobaltieri/zephyr-setup.
After that I'm thinking we could reuse it for testing:

in our example application: https://github.com/zephyrproject-rtos/example-application/pull/49, to show an actual example usage, it also happen to be faster than the current container-based solution
in the main project CI, to test a build on all supported platforms (Linux, macOS and Windows, all SDKs): https://github.com/zephyrproject-rtos/zephyr/pull/64802

The two above seems to be running in a reasonable time, and as far as I understand they are free within the GitHub limitations (20 parallel runs or something like that).
Dependencies
The workflow right now has an optional property for specifying the SDK version, I think it'd be nice to be able to pull a "latest" one automatically that pulls whatever was currently used in the Zephyr checkout. This could be done by having the version in the main repository in a known file, so I'd like to propose to do that as well (something like dumping a .github/current-sdk.txt with 0.16.3 or something like that).

cc @kartben @nashif @stephanosio @carlescufi @gmarull

@carlescufi put this up for Arch WG. Feel free to put it at the bottom of the list, it's a bit of a side project of mine, don't have much time to do major rework at the moment but if we are happy with the current state and some minor tweaks we can go with it.
I think at this stage the request is to create the org repo and move the code there.

As a random note, it's been working pretty smoothly for me since I added it to my artificial nose project, building with a couple different toolchains and all. https://github.com/kartben/artificial-nose-zephyr/actions

@fabiobaltieri https://github.com/zephyrproject-rtos/action-zephyr-setup has been created.

Thanks @stephanosio, I've done an initial empty push and opened https://github.com/zephyrproject-rtos/action-zephyr-setup/pull/1 to import the files from my original repo.


my ./gb is using only one cpu core?

i have 8 core processor. is there a way to ask the spider or gb to run on multiple processors?

Did you even bother to read the documentation?

I read the documentation on cluster scaling, it doesn't make sense to config clustering on individual server for 4 cores just to use 4 cores. Is it possible for it to use all four cores directly? this is because if it's used on multiple servers with multiple cores, that will be difficult to do the settings.

I'm not a Gigablast developer, but what I recall from documentation, the Gigablast is a very old project and essentially requires an overhaul. I might be wrong, but some hazy memory region of mine claims that the Gigablast does not use even the Linux built-in threads, posix threads, because at the time, when the Gigablast project started, Linux was not mature enough and the original author of the Gigablast wrote his own threading implementation.
That is to say, Gigablast needs a serious overhaul, "upgrade". May be the original, old, threading should be replaced with industry standard OpenMP, which is supported by all C++ compilers for at least 10 years, and if You are REALLY after a new threading model, then I suggest to check out Tucker Taft's ParaSail, which I repackaged for servers.
The ParaSail Forum. is also a de facto ParaSail upstream wiki.
Tucker Taft is the person, who developed the modern Ada programming language to the state that it is today (2017) and the ParaSail is his pet project, where he wants to summarize all of his life's work as a programming language designer, without taking any legacy with the project. He is a bit like the inventor of the Pascal programming language, the Niklaus Wirth: remarkable at programming langague development and, to say it mildly, not-so-remarkable at making a proper home page for distributing his awesome beauty. That explains, why I created a ParaSail wiki for myself, where I try to link everything to a somewhat coherent whole. The Niklaus Wirth also has his fansite to compensate his lack of ability to create a proper home page for himself. The Tucker Taft has at least a domain, parasail-lang.org, but basically, my wiki is like the fan site of the Niklaus Wirth  :-D


Enable cache

This potentially fixes #1538 (I'm still waiting for user test results).
The potential cause for this error was no cache table (no cache at all
was set up), so the Django-select2 AutoResponseView was unable to find
a correct widget by field_id GET parameter.
https://github.com/applegrew/django-select2/blob/master/django_select2/views.py#L82

According to this comment, the issue was fixed.


Automated cherry pick of #5526: Rename skip flag in custom Test Options

Cherry pick of #5526 on release-1.12.
#5526: Rename skip flag in custom Test Options
For details on the cherry pick process, see the cherry pick requests page.

/skip-all


add bulk upsert feature

This adds a bulk_upsert method. This will first fetch the latest rev for each document that does not have a rev set. It will then update the passed-in docs with that rev and insert all documents into the database.

Thanks Frando.
Merging this into "develop" to test this new feature.


[doc] Improve Pulsar Security Kerberos

Motivation
Improve the language and the overall descriptive style of the Pulsar Security document ( Security-kerberos section): http://pulsar.apache.org/docs/en/next/security-kerberos/
Modifications
Adjust the tone, personal pronouns, voice also some typo errors of some sentences in the document.

@Jennifer88huang  Please review, thank you~

run java8 tests


Trait rework: Variables should not be an associated type, but a marker trait

GraphQLQuery::Variables is the way currently used to represent that some type is, once serialized, a correct variables set for the given query.
However we could imagine that several structures, once serialized, would be valid for that query, e.g. GraphQLQuery::Variables and &GraphQLQuery::Variables, or the same structure but with &strs instead of Strings.
So what we see from that is that fundamentally, "being a valid variable set for query Q once serialized" is a property of some structures, and there may be several of them, so a single associated type is not the proper way to represent them. Instead, this property should be represented using a marker trait.
The same applies to Response, with an additional lifetime in the trait (and would help when trying to implement #30). In that case though, we may still want to provide a DefaultResponseData as an associated type, because it may still be useful when writing generic code running the query and returning deserialized owned data. (fn send_some_query<Q: GraphQLQuery>() -> Q::DefaultResponseData)
Here is what I think the traits should look like:
pub trait GraphQLQuery: Sized {
    /// The top-level shape of the response data (the `data` field in the GraphQL response). In practice this should be generated, since it is hard to write by hand without error.
    type DefaultResponseData: for<'de> IsValidResponseFor<'de, Self>;

    /// Produce a GraphQL query struct that can be JSON serialized and sent to a GraphQL API.
    fn build_query<V>(variables: V) -> QueryBody<V>
    where
        V: IsValidVariablesFor<Self>;
}

/// This struct is, once serialized, a valid `variables` parameter for this GraphQL query
///
/// `struct`s implementing this should be generated `struct`s most of the time.
pub trait IsValidVariablesFor<Q: GraphQLQuery>: serde::Serialize {}

/// This struct can be deserialized from the response data (the `data` field in the GraphQL response)
///
/// `struct`s implementing this should be generated `struct`s, since they are hard to write by hand without error.
pub trait IsValidResponseFor<'de, Q: GraphQLQuery>: serde::Deserialize<'de> {}

// With this new trait system, one could preserve some variables from one query to the other if needed
impl<'a, Q: GraphQLQuery, V: IsValidVariablesFor<Q>> IsValidVariablesFor<Q> for &'a V {}

With this new trait system, we could also generate non-owned versions of the structures using &str instead of Strings and serialize/deserialize from/to them. (#30 🎉), or better yet, replace the current generated structures with generic ones where you could specify the types for each of the fields, and:
//  ------ IN graphql_client ------
pub trait StringSer: Serialize {}
impl StringSer for String {}
impl<'a> StringSer for &'a str {}
// + std::borrow::Cow, etc, and lib users could add their own
// Also, maybe there's already a lib doing that kind of thing?

pub trait StringDe<'de>: Deserialize<'de> {}
impl<'de> StringDe<'de> for String {}
impl<'de> StringDe<'de> for &'de str {}
// + std::borrow::Cow, etc, and lib users could add their own
// Also, maybe there's already a lib doing that kind of thing?

// ------ CODEGEN ------
pub struct Query;
impl GraphQLQuery for Query {
    type DefaultResponseData = Response<String, String>;
    fn build_query<V>(variables: V) -> QueryBody<V>
    where
        V: IsValidVariablesFor<Self>,
    {
        todo!() // Same as currently implemented
    }
}

#[derive(Serialize)]
pub struct Variables<T1, T2> {
    pub field1: T1,
    pub field2: T2,
}
impl<T1, T2> IsValidVariablesFor<Query> for Variables<T1, T2>
where
    T1: StringSer,
    T2: StringSer,
{
}

#[derive(Deserialize)]
pub struct Response<T1, T2> {
    pub field1: T1,
    pub field2: T2,
}
impl<'de, T1, T2> IsValidResponseFor<'de, Query> for Response<T1, T2>
where
    T1: StringDe<'de>,
    T2: StringDe<'de>,
{
}

Obviously this would imply a breaking change but given the current download count this seems reasonable, and better do it now than later! 😉
What do you think ?

I think this would massively complicate things. What would IsValidVariablesFor be? Do we lose type safety because of this? Same with the response trait.
Other than Cow<'a, str>, what types do you forsee being valid here? Would it instead be sufficient to just use Cow<'a, str> (maybe optionally using 'static if wanted?) instead of String?

Hello @mathstuf, hope you are having a great day.
I will try to address your concerns here.

I think this would massively complicate things.

It's important to note that what is really important here is the first (small) code block, and the second one with StringDe and the like are just examples of additional user-friendliness improvements that could then be made in a semver-compatible way.
Now, although this would indeed complicate things a bit as it introduces new traits, I think it would not complicate them too much: it's just a slightly different way to express the notion "SomeStruct is a valid Variables for Query Q". (You would use f(variables: V) where V: IsValidVariablesFor<Q> instead of f(variables: Q::Variables).) I also beleive the benefit to doing it this way is actually very high as it is actually the correct way to represent what we need to be able to represent (I will explain what I mean by "correct way" a few lines below).

What would IsValidVariablesFor be? Do we lose type safety because of this?

I will try to explain in more detail than I did in my previous message:

IsValidVariablesFor<Q> is a trait that provides type safety for the notion that "SomeStruct, once serialized is a valid Variables for Query Q".
It does not lose type safety: if you provide SomeOtherStruct which does not implement IsValidVariablesFor<Q>, it will be rejected by the constraint on build_query: V: IsValidVariablesFor<Self>, in the same way as it would currently be rejected by the constraint build_query(variables: Self::Variables). And of course, generated code would implement IsValidVariablesFor<Query> on the Variables struct in the generated module of query Query and any pair Query/struct for which it would work (that is, the server would accept it), and not for all pairs of Query/struct.
The main advantage compared to the current representation is that it allows the typesystem to represent when there are several (eg. SomeStruct1 and SomeStruct2) that, once serialized, are valid json Variables for Query Q.
A good example of that last point is: if currently we have Q: GraphQLQuery<Variables=SomeStruct1> with this system we would have Q: Query and SomeStruct1: IsValidVariablesFor<Q>, but also &SomeStruct1: IsValidVariablesFor<Q> (thanks to the generic impl at the bottom of my first code block). Our current implementation does not allow to represent that we could call build_query with &SomeStruct1, but if it did, as &SomeStruct1 serializes in the exact same way as SomeStruct1 (enforced by serde), the generated json would also be valid, so why not allowing the user to call build_query passing a reference ? That would enable them to reuse the contents of the query they sent later, which may be useful in some scenarios.
Another example of several structures being valid is if you write SomeStruct2 that is basically the same as SomeStruct1 (the current Variables struct for a given query), but you replace a type String inside by a type &str. Again, once serialized, it would be considered valid by the server, as String serializes in the exact same way as &str (again, enforced by serde). But it would also remove the need for the user to write code that reallocates a String when they already have an &str, when building Variables ! 😃
These two needs (and I can think of others) cannot be represented with the current system, even though they would be valid, because we are constrained by our representation that uses an associated type, hence enforcing that a single type can be used as variables. The representation described in the previous post however, allows for this possibility that conceptually exists, making it the exact correct way to represent our concepts.

Same with the response trait.

The response trait is basically the mirror of IsValidVariablesFor, for responses. There may be several structs we could deserialize to (e.g. owned with Strings, or borrowed when implementing #30), but thanks to the IsValidResponseFor constraint, when deserializing a response one knows they can only attempt to deserialize it in a struct that implements IsValidResponseFor<TheirQuery>, in the same way as currently they would know they can only attempt to deserialize it in TheirQuery::Response. Again, we do not lose type safety, and can properly represent any scenario where a response for a particular query could be successfully deserialized into several different types, making it, here again, the correct way to represent that a response to the query Q could be successfully deserialized into Response.

Other than Cow<'a, str>, what types do you forsee being valid here? Would it instead be sufficient to just use Cow<'a, str> (maybe optionally using 'static if wanted?) instead of String?

I could indeed imagine that one could want to build their query for Cows when generating part of their data and borrowing another part. However, I think most common use-cases would be:

Replacing Strings by &strs, as you can just borrow the data from elsewhere and do not need to clone it inside the Variables. Creating the Cow would introduce some boilerplate and runtime overhead.
Sometimes you'd still need the Strings when you need your Variables struct to be 'static, because you want e.g. to send it across threads to send the query later. (hence the need for both String and &str representations). Again, creating the Cow would introduce some boilerplate and runtime overhead.
One could also want to create newtypes that represent IDs for a particular object of a particular graphql service, that would propagate the Deserialize implementation of the inner object and hence still serialize as a json string. One may want to use these directly, instead of having to unpack the &str/String inside everytime they build a Variables struct. Cow would restrict you to only String and &'a str.
As a conclusion regarding that last point, I think simply replacing the Strings by Cows would introduce a lot of boilerplate for the user (need to create the Cow), as well as runtime overhead (need to handle the Cow serialization/deallocation), still require some form of genericity (need to be generic on 'a or give up on not cloning when using 'static), while also allowing for less use-cases (no newtype) compared to the solution with generic presented in the second code block of the initial post.

As a general conclusion, I would like to underline that these are only a few examples I could think of now, but there could be many more I didn't think of: I beleive this trait system is simply the one that perfectly matches the reality of our concepts (by the fact the concept of trait is the notion of "some property/capability satisfied by some structure", which is precisely what we're trying to represent here, where the property is "once serialized, will be accepted as a valid variables by the graphql server for this query"), making it the correct way to represent them. And once designed, as that is always the case when you're designing a computer system and your representation matches your concepts perfectly, it naturally comes with tons of advantages, among which user-friendliness (generic Variables structs, allowing to build query by reference on Variables), performance (#30), better type safety (allowing for newtype patterns), etc.
I hope I was able to answer all of your concerns as well as better explain what I tried to design and the philosophy behind it, and would love to have more feedback.
Do you have any other question, comment, remark ?

It does not lose type safety: if you provide SomeOtherStruct which does not implement IsValidVariablesFor, it will be rejected by the constraint on build_query: V: IsValidVariablesFor, in the same way as it would currently be rejected by the constraint build_query(variables: Self::Variables). And of course, generated code would implement IsValidVariablesFor on the Variables struct in the generated module of query Query and any pair Query/struct for which it would work (that is, the server would accept it), and not for all pairs of Query/struct.

What enforces that IsValidVariablesFor<Q> is actually true? What is stopping me from doing (the moral equivalent of):
impl<Q> IsValidVariablesFor<Q> for () {
    fn as_variables(&self) -> serde_json::Value {
        json!({})
    }
}

What enforces that IsValidVariablesFor<Q> is actually true? What is stopping me from doing (the moral equivalent of):
impl<Q> IsValidVariablesFor<Q> for () {
    fn as_variables(&self) -> serde_json::Value {
        json!({})
    }
}

Saying () is valid a valid Variables for any query Q is prevented by the fact we can implement a trait on a type only if either the trait or the type is local to our crate, which will not be the case for any user.
For other moral equivalents (typically, where you own the query or the type you're trying to implement it on), a user could specify by themselves that they know that some type is a valid Variables for a given query, which they could know for very good reasons, e.g. because that type transfers the Serialize impl to that of the original code-generated Variables that already implemented IsValidVariablesFor<TheirQuery>, by building it in the Serialize impl.
It seems to be a legitimate use case (especially since it would be zero-cost if we had non-owned versions of Variables).
If a user implements that trait themselves, they take responsibility that they're saying "I know this type is valid for that query" (which they can only do if they own the query or the given type). It provides them flexibility and allows again for more user-friendly interfaces.
I'm not sure I understand in which scenario a user would manually impl IsValidVariablesFor<Q> for one of their types that does not satisfy the predicate this trait is meant for, hence breaking some of their type safety, if they don't have a legitimate scenario for trying to manually implement it for their type/query pair in the first place.
(Also, just as a side note, IsValidVariablesFor depends on Serialize and does not need as_variables or anything like it.)

Saying () is valid a valid Variables for any query Q is prevented by the fact we can implement a trait on a type only if either the trait or the type is local to our crate, which will not be the case for any user.

Replace () with MyLocalZST. It's isomorphic.

If a user implements that trait themselves, they take responsibility that they're saying "I know this type is valid for that query" (which they can only do if they own the query or the given type). It provides them flexibility and allows again for more user-friendly interfaces.

That's not how Rust traits work in practice. At least it's not how I'd want traits I'm relying on to ensure that I got my types right to work. It's certainly not an improvement to the status quo in my eyes. I'd find it far more interesting to replace String usages with Cow<'a, str> if you're really concerned about unnecessary allocations. Cow wrapping is likely going to be trivial overhead compared to the JSON building that is going to happen at the end of this chain anyways.

(Also, just as a side note, IsValidVariablesFor simply inherits Serialize and does not need as_variables or anything like it.)

I saw that. It means that I can't actually use this trait in practice because it doesn't actually guarantee what I'd want. And it wouldn't work anyways because how to I signal that in Query1 I skip some field while the same struct as a variables that Query2 needs? Why am I not ending up with exactly one struct per Query in this new pattern? And why do I now have to write the code and manually keep in sync with my query code.
I find it far more useful for the compiler to tell me "you forgot to add this field you added to your query" than the GraphQL server to tell me after I deploy it. We may as well just take raw JSON for the variables parameter for how much it actually helps me get my types right.
Maybe this could be an option for a way to generate code, but I certainly don't think it should be the default.

Oh, I beleive I found where the misunderstanding is coming from.

And why do I now have to write the code and manually keep in sync with my query code.

As I tried to explain in my first post, the IsValidVariablesFor<Query> implementation for the automatically generated Variables struct would still be automatically generated in the same way the associated type Variables = Variables was, so you you wouldn't have anything more to manually write than you previously did.

I find it far more useful for the compiler to tell me "you forgot to add this field you added to your query" than the GraphQL server to tell me after I deploy it.

As a consequence, that's precisely what it still does.

Replace () with MyLocalZST. It's isomorphic.

A user could define a type, and declare that they want it to typecheck with any query, but again, people would have to do that on purpose and use that type on purpose, so they wouldn't do it unless they have a legitimate use case for that. So as a prerequisite to doing that they would need that feature. So "somebody does that" implies "this feature is needed for him". (Though I don't know what use case that would be since it kinda defeats the whole point of this library).

That's not how Rust traits work in practice.

I'm afraid I don't see how they work any differently from that. The Send trait means "your type can be sent from a thread to another without breaking everything". The FusedIterator trait means "calling next on a fused iterator that has returned None once is guaranteed to return None again". And the IsValidVariablesFor<Query> trait means "calling .serialize() will produce a variable the server will consider as valid for Query". You can find tons of other examples of that in the great library that is diesel.

At least it's not how I'd want traits I'm relying on to ensure that I got my types right to work.

I beleive (as per the examples I've pointed out) that's unfortunately the idiomatic way Rust works.
Why do you think that is not desirable ?

I saw that. It means that I can't actually use this trait in practice because it doesn't actually guarantee what I'd want.

I beleive I've addressed this: it does.
But if I'm mistaken and that wasn't the misunderstanding I thought, can you further explain what property you think is not guaranteed unless you specifically write additional code for removing these guarantees ?

Why am I not ending up with exactly one struct per Query in this new pattern?

As I said, because several different structures could be valid. What's the point of chosing a single one of them for your user, when you could allow him to use any that is valid (while still preventing the ones that are not valid) ?


chore: release 4.4.0

:robot: I have created a release beep boop
wombat: 4.4.0
4.4.0 (2024-11-18)
Features

add days support to timerange (#165) (2ae5510)

Miscellaneous Chores

add plugins (d907df8)
polyfill changelog (d044180)

tauri-backend: 4.4.0
4.4.0 (2024-11-18)
Miscellaneous Chores

add plugins (d907df8)
polyfill changelog (d044180)

This PR was generated with Release Please. See documentation.

:robot: Created releases:

wombat-v4.4.0
:sunflower:


Update  "emoji-datasource" to 7.x.x

Please update  "emoji-datasource" to latest 7.x.x
Thank you.

Main change in v7: Updated to Emoji 13.1.
Previous major update (Emoji v13):

https://github.com/serebrov/emoji-mart-vue/pull/108
https://github.com/serebrov/emoji-mart-vue/pull/109

We've updated to v14 datasource in #205, closing this issue.


Vulnerability in "class-transformer" package

This showed up in my GitHub repo, and it looks like it's coming from Warthog's dependencies:

I did a quick search through warthog's code and it doesn't seem like the package is being used anywhere.

This is pretty common in the npm ecosystem. I’ll stay on top of this as much as I can but note that you can explicitly require this version in your code (ie yarn add class-transformer@latest).


GCC 13 support

Resolves issue #263

Hum ..?

Ah sorry, lemme merge into that one and close this one


docs: Add maintainers file

@lucasfcosta : For me this is perfectly fine. You can hit the merge button when you like. ;)

Oh, I haven't seen this.
It's fine for me too, thanks for mentioning me on this one @davelosert


html code in python code

for example, the following code

re.findall('.*?', page)

can not be show normally because there is some HTML code in python code.
How can I solve this problem?

#259 #373 #572 #984

Thanks @uranusjr for digging out those issues ;)
@bojone The solutions explained in those will fix your current problem. You might also wanna check the Unescaped markup plugin.


Create Observability overview in cluster-administration

As suggested by @sftim in https://github.com/kubernetes/website/pull/39970#issuecomment-1466459781, we should create an overview detected to Observability in https://kubernetes.io/docs/concepts/cluster-administration/.

/assign
/sig instrumentation docs
/kind cleanup

Want to work on this? If you know how to draft a PR, I suggest starting there and mention the draft PR here with a request for early feedback.

I assigned it to myself, but I have limited bandwidth to work on this.
/unassign
/help

/assign

@aritraghosh Are you still working on this issue?


Sparse matrix addition/subtraction eliminates explicit zeros

<<Please describe the issue in detail here, and for bug reports fill in the fields below.>>
Addition of sparse matrices removes explicit zeros. While the result is correct the structure of the resulting sparse matrix ignores the explicit zeros that come from the original matrices. This might be a problem for subroutines that rely on the structure of the linear system (e.g sparse linear solver).
I believe the issue can be corrected by modifying the _cs_matrix._binopt(other, 'plus') subroutine
Reproducing code example:
from scipy.sparse import coo_matrix
import numpy as np

row  = np.array([0, 3, 1, 0])
col  = np.array([0, 3, 1, 2])
data = np.array([0, 5, 7, 9])
m = coo_matrix((data, (row, col)), shape=(4, 4))
print(m.nnz) # output = 4
mm = m + m
print(mm.nnz)  # output = 3

Scipy/Numpy/Python version information:
# scipy.__version__ = '1.1.0'
# numpy.__version__ = '1.15.1'
# sys.version_info(major=3, minor=5, micro=5, releaselevel='final', serial=0)

Any comments on this?

The design decision iirc as per previous discussions was to preserve explicit zeros.
So now it's up for someone to roll their sleeves to fix it.

Has someone had the time to fix this?


No longer able to search directly on focus without click

In 3.x I was able to tab to a select2 field and directly start typing, you can try it out here:
http://select2.github.io/select2/
In 4.0 this is is no longer possible. You have to click an item or press enter before being able to search. This is a downgrade and I don't see it anywhere in the release notes, so probably a bug.
Proof that it no longer works can be found by trying here: https://select2.github.io/examples.html

Duplicate, explained at https://github.com/select2/select2/issues/3279#issuecomment-96450069.


feat: 🎸 Add new pallet, extrinsics, events for 5.3.0 chain

Description
This adds new NFT pallet, its new extrinsics and events to existing enums.
Breaking Changes
NA
JIRA Link
DA-554
Checklist

[ ] Updated the Readme.md (if required) ?

:tada: This PR is included in version 9.2.0-alpha.1 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Plot learning curve of SGD and GP

R^2 vs training point #.

Progress made in ba42d23 (#169).

Do passive learning curves second.

I have obtained the following curve for passive learning. I am working on unifying SGD and GP and producing a better one.
From this, we can see that GP works very well on small training sets. Perhaps there is no need for SGD regression?

At long last, I produced correct curve! I take back what I said about not needing SGD regression: you can clearly see that the accuracy of SGD regression keeps increasing beyond 1.5K training points, which is where we stop being able to train GP regression.

Completed for passive learning in 529ff9b.


Bind UIWebView in JSB

Related PR: https://github.com/cocos2d/cocos2d-js/pull/1518

duplicated https://github.com/cocos2d/cocos2d-js/issues/1661


Wrong calculation of Closed findings

Bug description
After reimport DefectDojo closed all findings, but not really mitigated.
I had 23 findings (see screenshot 1), then 17 issues was solved and new report was reimported, and I see that calculation is wrong (not expected).
Calculation of Closed findings and Total findings is wrong.
Two issues:

I see 23 closed findings, but in reality only 17 was solved. See screenshot 2.
I see 29 total findings, but I expect 23, because 17 closed + 6 open. See screenshot 2.

Steps to reproduce
Steps to reproduce the behavior:

Upload first report with 23 findings.
Upload second report with 6 findings (this 6 is the same as in first report)
See that closed 23 findings, but not 17.
See that Total findings is 29, but not 23 (17 closed and 6 open)

Expected behavior
If the finding is the same (same CVE) it should not be closed after reimport report.
I expect in Closed findings to see only unique closed CVE.
Deployment method

[x] Docker Compose

Environment information

DefectDojo version v. 2.39.0

Sample scan files
report1.json - has 23 Critical findings
report2.json - has 6 Critical findings (6 same as first report has)
Total findings - DD report.pdf - Report from DefectDojo where you can see that open CVE is the same as closed.
Screenshots

Additional info
I use close_old_findings argument while reimport. See all arguments I use below.

Script which I use for reimport
export DOJO_HOST=xyz.com
export DOJO_ENV_NAME=Production
export DOJO_LEAD_ID=1
export DOJO_SCANNER="Trivy Scan"
export DOJO_API_TOKEN=xxx
export DOJO_ENGAGEMENT_NAME="Weekly docker image scanning"

run this to import report1.json to aa-ruby-test project
python reimport.py --host ${DOJO_HOST} --api_key ${DOJO_API_TOKEN} --result_file report1.json --scanner "${DOJO_SCANNER}" --engagement_name "${DOJO_ENGAGEMENT_NAME}" --product_name aa-ruby-test --lead_id ${DOJO_LEAD_ID} --environment ${DOJO_ENV_NAME}

#!/usr/bin/python3

import argparse
from datetime import datetime
import requests
import urllib3
urllib3.disable_warnings()

def upload_results(host, api_key, scanner, result_file, engagement_name, lead_id, environment, verify=False): # set verify to False if ssl cert is self-signed
  API_URL = "https://"+host+"/api/v2"
  IMPORT_SCAN_URL = API_URL+"/reimport-scan/"
  AUTH_TOKEN = "Token "+api_key

  headers = dict()
  json = dict()
  files = dict()

  headers['Authorization'] = AUTH_TOKEN
  print(headers)

  json['minimum_severity'] = "Low"
  json['scan_date'] = datetime.now().strftime("%Y-%m-%d")
  json['verified'] = True
  json['active'] = True
  json['lead'] = lead_id
  json['scan_type'] = scanner
  json['environment'] = environment
  json['close_old_findings'] = True
  json['deduplication_on_engagement'] = True
  json['product_name'] = product_name
  json['product_type_name'] = "microservices"
  json['engagement_name'] = engagement_name
  json['auto_create_context'] = True
  print(json)

  # Prepare file data to send to API
  files['file'] = open(result_file)

  # Make a request to API
  response = requests.post(IMPORT_SCAN_URL, headers=headers, files=files, data=json, verify=verify)
  # Uncomment these lines to debug
  # print(response.request.body)
  # print(response.request.headers)
  # print(response.status_code)
  # print(response.text)
  return response.status_code

if __name__ == "__main__":
  parser = argparse.ArgumentParser(description='CI/CD integration for DefectDojo')
  parser.add_argument('--host', help="DefectDojo Hostname", required=True)
  parser.add_argument('--api_key', help="API v2 Key", required=True)
  parser.add_argument('--engagement_name', help="Engagement name", required=True)
  parser.add_argument('--result_file', help="Scanner file", required=True)
  parser.add_argument('--scanner', help="Type of scanner", required=True)
  parser.add_argument('--product_name', help="DefectDojo Product name", required=True)
  parser.add_argument('--lead_id', help="ID of the user conducting the testing", required=True)
  parser.add_argument('--environment', help="Environment name", required=True)
  parser.add_argument('--build_id', help="Reference to external build id", required=False)

  # Parse out arguments
  args = vars(parser.parse_args())
  host = args["host"]
  api_key = args["api_key"]
  product_name = args["product_name"]
  result_file = args["result_file"]
  scanner = args["scanner"]
  engagement_name = args["engagement_name"]
  lead_id = args["lead_id"]
  environment = args["environment"]
  build_id = args["build_id"]

  result = upload_results(host, api_key, scanner, result_file, engagement_name, lead_id, environment)
  if result == 201 :
       print("✅ Successfully uploaded the results to Defect Dojo")
  else:
       print("Something went wrong, please debug " + str(result))

@optimistic5

If the finding is the same (same CVE) it should not be closed after reimport report.
I expect in Closed findings to see only unique closed CVE.

DefectDojo doesn't necessarily use only CVE to determine which findings are closed on reimport. What is used is determined on a tool by tool basis.
For Trivy Scans, the fields used to determine the 'same' finding are:
https://github.com/DefectDojo/django-DefectDojo/blob/master/dojo/settings/settings.dist.py#L1236
You can adjust how this works by overriding the defaults in a local_settings.py file you create so you can modify it to only use CVE if you want. BTW, CVE would be the first item stored under "vulnerability_ids" if a tool provides a CVE. You can find out more about the deduplication algorithms at https://documentation.defectdojo.com/usage/features/#deduplication-algorithms
However, I don't think this is actually a bug if your expectation was that CVE alone was used to close reimported findings.
If you want to more interactively discuss how reimport & dedup works, I'd suggest joining the OWASP Slack and ask in the #defectdojo channel. Details to join are at https://github.com/DefectDojo/django-DefectDojo/tree/master?tab=readme-ov-file#community-getting-involved-and-updates
HTH

I did investigation.
Here is my findings.
In trivy report of ruby we can see this vulnerability:

vulnerability from ruby 3.1.4
{
        "VulnerabilityID": "CVE-2023-6879",
        "PkgID"<EMAIL_ADDRESS>        "PkgName": "libaom3",
        "PkgIdentifier": {
          "PURL": "pkg:deb/debian/libaom3@3.6.0-1?arch=amd64\u0026distro=debian-12.5",
          "UID": "b2afd5c5ffd23406"
        },
        "InstalledVersion": "3.6.0-1",
        "Status": "affected",
        "Layer": {
          "Digest": "sha256:b4580645a8e50b87a19330da289a9b1540022379f2c99d3f0112e3c5c4a8d051",
          "DiffID": "sha256:893507f6057f8f0532ae8907fb5470f4a0181181bdbe139c2706d47cc4a88600"
        },
        "SeveritySource": "nvd",
        "PrimaryURL": "https://avd.aquasec.com/nvd/cve-2023-6879",
        "DataSource": {
          "ID": "debian",
          "Name": "Debian Security Tracker",
          "URL": "https://salsa.debian.org/security-tracker-team/security-tracker"
        },
        "Title": "aom: heap-buffer-overflow on frame size change",
        "Description": "Increasing the resolution of video frames, while performing a multi-threaded encode, can result in a heap overflow in av1_loop_restoration_dealloc().\n\n",
        "Severity": "CRITICAL",
        "CweIDs": [
          "CWE-787",
          "CWE-20"
        ],
        "VendorSeverity": {
          "nvd": 4,
          "redhat": 3
        },
        "CVSS": {
          "nvd": {
            "V3Vector": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H",
            "V3Score": 9.8
          },
          "redhat": {
            "V3Vector": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H",
            "V3Score": 9.8
          }
        },
        "References": [
          "https://access.redhat.com/security/cve/CVE-2023-6879",
          "https://aomedia.googlesource.com/aom/+/refs/tags/v3.7.1",
          "https://crbug.com/aomedia/3491",
<EMAIL_ADDRESS><EMAIL_ADDRESS>          "https://nvd.nist.gov/vuln/detail/CVE-2023-6879",
          "https://www.cve.org/CVERecord?id=CVE-2023-6879"
        ],
        "PublishedDate": "2023-12-27T23:15:07.53Z",
        "LastModifiedDate": "2024-02-02T02:27:15.863Z"
      },

vulnerability from ruby 3.1.4
{
        "VulnerabilityID": "CVE-2023-6879",
        "PkgID"<EMAIL_ADDRESS>        "PkgName": "libaom3",
        "PkgIdentifier": {
          "PURL": "pkg:deb/debian/libaom3@3.6.0-1%2Bdeb12u1?arch=amd64\u0026distro=debian-12.7",
          "UID": "ac5b595bbc65735b"
        },
        "InstalledVersion": "3.6.0-1+deb12u1",
        "Status": "affected",
        "Layer": {
          "Digest": "sha256:ad1c7cfc347f5c86fc2678b58f6a8fb6c6003471405760532fc3240b9eb1b343",
          "DiffID": "sha256:d23b5e6144a796978b150684f10a986aa5dc30d4729f5af29864847008211613"
        },
        "SeveritySource": "nvd",
        "PrimaryURL": "https://avd.aquasec.com/nvd/cve-2023-6879",
        "DataSource": {
          "ID": "debian",
          "Name": "Debian Security Tracker",
          "URL": "https://salsa.debian.org/security-tracker-team/security-tracker"
        },
        "Title": "aom: heap-buffer-overflow on frame size change",
        "Description": "Increasing the resolution of video frames, while performing a multi-threaded encode, can result in a heap overflow in av1_loop_restoration_dealloc().\n\n",
        "Severity": "CRITICAL",
        "CweIDs": [
          "CWE-787",
          "CWE-20"
        ],
        "VendorSeverity": {
          "nvd": 4,
          "redhat": 3
        },
        "CVSS": {
          "nvd": {
            "V3Vector": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H",
            "V3Score": 9.8
          },
          "redhat": {
            "V3Vector": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H",
            "V3Score": 9.8
          }
        },
        "References": [
          "https://access.redhat.com/security/cve/CVE-2023-6879",
          "https://aomedia.googlesource.com/aom/+/refs/tags/v3.7.1",
          "https://crbug.com/aomedia/3491",
<EMAIL_ADDRESS><EMAIL_ADDRESS>          "https://nvd.nist.gov/vuln/detail/CVE-2023-6879",
          "https://www.cve.org/CVERecord?id=CVE-2023-6879"
        ],
        "PublishedDate": "2023-12-27T23:15:07.53Z",
        "LastModifiedDate": "2024-02-02T02:27:15.863Z"
      },
      {
        "VulnerabilityID": "CVE-2023-5841",
        "PkgID"<EMAIL_ADDRESS>        "PkgName": "libopenexr-3-1-30",
        "PkgIdentifier": {
          "PURL": "pkg:deb/debian/libopenexr-3-1-30@3.1.5-5?arch=amd64\u0026distro=debian-12.7",
          "UID": "5c104a106ff50b8d"
        },
        "InstalledVersion": "3.1.5-5",
        "Status": "affected",
        "Layer": {
          "Digest": "sha256:ad1c7cfc347f5c86fc2678b58f6a8fb6c6003471405760532fc3240b9eb1b343",
          "DiffID": "sha256:d23b5e6144a796978b150684f10a986aa5dc30d4729f5af29864847008211613"
        },
        "SeveritySource": "nvd",
        "PrimaryURL": "https://avd.aquasec.com/nvd/cve-2023-5841",
        "DataSource": {
          "ID": "debian",
          "Name": "Debian Security Tracker",
          "URL": "https://salsa.debian.org/security-tracker-team/security-tracker"
        },
        "Title": "OpenEXR: Heap Overflow in Scanline Deep Data Parsing",
        "Description": "Due to a failure in validating the number of scanline samples of a OpenEXR file containing deep scanline data, Academy Software Foundation OpenEX image parsing library version 3.2.1 and prior is susceptible to a heap-based buffer overflow vulnerability. This issue was resolved as of versions v3.2.2 and v3.1.12 of the affected library.\n",
        "Severity": "CRITICAL",
        "CweIDs": [
          "CWE-787",
          "CWE-122"
        ],
        "VendorSeverity": {
          "amazon": 2,
          "nvd": 4,
          "redhat": 3
        },
        "CVSS": {
          "nvd": {
            "V3Vector": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:N",
            "V3Score": 9.1
          },
          "redhat": {
            "V3Vector": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:N",
            "V3Score": 9.1
          }
        },
        "References": [
          "https://access.redhat.com/security/cve/CVE-2023-5841",
<EMAIL_ADDRESS><EMAIL_ADDRESS>          "https://nvd.nist.gov/vuln/detail/CVE-2023-5841",
          "https://takeonme.org/cves/CVE-2023-5841.html",
          "https://takeonme.org/cves/CVE-2023-5841.html#",
          "https://www.cve.org/CVERecord?id=CVE-2023-5841"
        ],
        "PublishedDate": "2024-02-01T19:15:08.097Z",
        "LastModifiedDate": "2024-02-26T16:27:49.42Z"
      },

note this part in ruby:3.1.4:
"VulnerabilityID": "CVE-2023-6879",
"PkgID"<EMAIL_ADDRESS>"PkgName": "libaom3",
"InstalledVersion": "3.6.0-1",

and same part in ruby:3.1.6:
"VulnerabilityID": "CVE-2023-6879",
"PkgID"<EMAIL_ADDRESS>"PkgName": "libaom3",
"InstalledVersion": "3.6.0-1+deb12u1",

PkgID and InstalledVersion is different. But as for me it's still same finding. Suffix +deb12u1 does not really change the vulnerability.
But Trivy Scan has configuration:
"Trivy Scan": ["title", "severity", "vulnerability_ids", "cwe", "description"],
where title is
title = f"{vuln_id} {package_name} {package_version}"

which in JSON report would be: VulnerabilityID, PkgName, InstalledVersion.
It means that title would be different in ruby:3.1.4 and ruby:3.1.6, which is as for me should not be like this.
This is the main thing which was confusing - title consist from these parts.
For me I found configuration which suites my needs. I added to uwsgi container variables:
DD_DEDUPLICATION_ALGORITHM_PER_PARSER: '{"Trivy Scan": "hash_code"}'
DD_HASHCODE_FIELDS_PER_SCANNER: '{"Trivy Scan": ["vulnerability_ids", "component_name"]}'

And I received desired result:

Thank you.

@optimistic5 Glad you were able to configure DD to fit your needs. Just FYI: You can also keep those dedup changes in local_settings.py but not settings.dist.py - details here.
Each AppSec/Product Security/DevSecOps place I've worked at or talked to people who worked at different places, the more I've realized that what works at one place won't necessarily work at the next place. Different programming languages, programming styles, environments, ... can impact how you need to use tool output.
We've tried to keep DefectDojo as flexible as possible so you can change DefectDojo vs changing how you do things where you're working. :+1:
I'm going to go ahead and close this issue but feel free to reopen it if needed.


[stable/kong] Cannot install 0.3.X ingress controller version with helm

Describe the bug
When trying to set the version manually of the ingress controller, kong-ingress is crashing. The log message is "Please specify --default-backend-service"
I believe it is because when releasing the 0.4, the parameter --default-backend-service has been removed, so it is not specified in the helm chart anymore. I think the helm chart should handle this
Version of Helm and Kubernetes:
helm version:
Client: &version.Version{SemVer:"v2.13.1", GitCommit:"618447cbf203d147601b4b9bd7f8c37a5d39fbb4", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.13.1", GitCommit:"618447cbf203d147601b4b9bd7f8c37a5d39fbb4", GitTreeState:"clean"}

Kubernete version:
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.6", GitCommit:"abdda3f9fefa29172298a2e42f5102e777a8ec25", GitTreeState:"clean", BuildDate:"2019-05-08T13:53:53Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.6-gke.13", GitCommit:"fcbc1d20b6bca1936c0317743055ac75aef608ce", GitTreeState:"clean", BuildDate:"2019-06-19T20:50:07Z", GoVersion:"go1.11.5b4", Compiler:"gc", Platform:"linux/amd64"}

Which chart:
kong-0.12.1
What happened:
Running helm install stable/kong --name=kong --set ingressController.enabled=true --set proxy.externalTrafficPolicy=Local --set ingressController.image.tag=0.3.0   --namespace kong
What you expected to happen:
Pods in namespace kong running normally
How to reproduce it (as minimally and precisely as possible):
Install kong via the chart using the 0.3 version of kong-ingress

This is due to a breaking change in Kong Ingress Controller 0.4.0.
Either use an older version of the chart or upgrade your Ingress Controller.

Thanks,
In case someone have a similar issue, i fixed my issue with:
helm install stable/kong --name=kong --set ingressController.enabled=true --set ingressController.image.tag=0.3.0 --namespace kong --version v0.9.8


Cleanup of some of the hardware node stuff

Added EPOSes back to robot hardware
fixed some bugs
switched to steady_clock for control loop
respawn nodes
other stuff

Add message server


Support vscode new explorer.experimental.fileNesting

[x] I'm sure this issue is not a duplicate.

vscode's new version  support experimental nesting files. @see: https://code.visualstudio.com/updates/v1_64#_explorer-file-nesting
bug when enabled this exp feature with vscode-icons 's Hide Explorer Arrows: true the files expand arrow disapled.
want screenshot:

when set Hide Explorer Arrows: true:

There is no way to expand/unexpand nesting files.

HI @zzjin, this is something completely out of our control and should be informed to the vscode team. I suggest you to open a new issue in their repo and reference this one. Thanks for reporting this, though.

@cc https://github.com/microsoft/vscode/issues/141428


⚠ ClamAV has degraded performance

In 0e1901f, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 51 ms

Resolved: ClamAV performance has improved in 77b35be.


fix: error badge

This is an intended breaking change, to align with the custom containers ::: danger
See https://vuepress2.netlify.app/guide/migration.html#default-theme-change.