Possible to load your own trained models with internet disabled?

Was wondering if there is a way to load a model in a kaggle notebook that I trained myself. There's currently a NER competition going on, and I wanted to try using the SpanMarker library to compete. Training went fine, but now to submit, I need to have the kaggle notebook have internet disabled. When trying to load my checkpoint, I get this error:
model_checkpoint = "/kaggle/input/pii-train-1-cp3000/Kaggle Checkpoints/checkpoint 3000"
model = SpanMarkerModel.from_pretrained(model_checkpoint,local_files_only = True,
labels = [
'1-EMAIL', '1-ID_NUM', '1-NAME_STUDENT', '1-PHONE_NUM', '1-STREET_ADDRESS',
'1-URL_PERSONAL', '1-USERNAME', '2-ID_NUM', '2-NAME_STUDENT', '2-PHONE_NUM',
'2-STREET_ADDRESS', '2-URL_PERSONAL', 'O'
])
OSError: We couldn't connect to 'https://huggingface.co/' to load this file, couldn't find it in the cached files and it looks like bert-base-uncased is not the path to a directory containing a file named config.json.
Checkout your internet connection or see how to run the library in offline mode at 'https://huggingface.co/docs/transformers/installation#offline-mode'.
Kaggle notebook here: https://www.kaggle.com/jdonnelly0804/pii-infer

Hello!
I'm struggling to reproduce the issue - I seem to be able to load a local model without internet. I can't reproduce it with your notebook either, as the model is private. Could you provide a larger stack trace perhaps? It seems like it wants to load the config for bert-base-uncased, which seems a bit odd as this is likely the original non-spanmarker model; I'm not sure why it wants to load this one.

Tom Aarsen

Thank you for the quick response! I've made the dataset public, so you can use it now when trying to reproduce the error. Also, here is the link to the training notebook as well: https://www.kaggle.com/code/jdonnelly0804/pii-train  - Training notebook its ok for the internet to be on, and everything goes smoothly, I simply save a checkpoint from the trained model, download the files, upload it as a dataset, and use it in the inference notebook that you we're trying to recreate (that was the hidden dataset)
Finally, here is the full traceback:
model_checkpoint = "/kaggle/input/pii-train-1-cp3000/Kaggle Checkpoints/checkpoint 3000"
model = SpanMarkerModel.from_pretrained(model_checkpoint,local_files_only = True,
labels = [
'1-EMAIL', '1-ID_NUM', '1-NAME_STUDENT', '1-PHONE_NUM', '1-STREET_ADDRESS',
'1-URL_PERSONAL', '1-USERNAME', '2-ID_NUM', '2-NAME_STUDENT', '2-PHONE_NUM',
'2-STREET_ADDRESS', '2-URL_PERSONAL', 'O'
])
gaierror                                  Traceback (most recent call last)
File /opt/conda/lib/python3.10/site-packages/urllib3/connection.py:174, in HTTPConnection._new_conn(self)
173 try:
--> 174     conn = connection.create_connection(
175         (self._dns_host, self.port), self.timeout, **extra_kw
176     )
178 except SocketTimeout:
File /opt/conda/lib/python3.10/site-packages/urllib3/util/connection.py:72, in create_connection(address, timeout, source_address, socket_options)
68     return six.raise_from(
69         LocationParseError(u"'%s', label empty or too long" % host), None
70     )
---> 72 for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
73     af, socktype, proto, canonname, sa = res
File /opt/conda/lib/python3.10/socket.py:955, in getaddrinfo(host, port, family, type, proto, flags)
954 addrlist = []
--> 955 for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
956     af, socktype, proto, canonname, sa = res
gaierror: [Errno -3] Temporary failure in name resolution
During handling of the above exception, another exception occurred:
NewConnectionError                        Traceback (most recent call last)
File /opt/conda/lib/python3.10/site-packages/urllib3/connectionpool.py:703, in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
702 # Make the request on the httplib connection object.
--> 703 httplib_response = self._make_request(
704     conn,
705     method,
706     url,
707     timeout=timeout_obj,
708     body=body,
709     headers=headers,
710     chunked=chunked,
711 )
713 # If we're going to release the connection in finally:, then
714 # the response doesn't need to know about the connection. Otherwise
715 # it will also try to release it and we'll have a double-release
716 # mess.
File /opt/conda/lib/python3.10/site-packages/urllib3/connectionpool.py:386, in HTTPConnectionPool._make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw)
385 try:
--> 386     self._validate_conn(conn)
387 except (SocketTimeout, BaseSSLError) as e:
388     # Py2 raises this as a BaseSSLError, Py3 raises it as socket timeout.
File /opt/conda/lib/python3.10/site-packages/urllib3/connectionpool.py:1042, in HTTPSConnectionPool._validate_conn(self, conn)
1041 if not getattr(conn, "sock", None):  # AppEngine might not have  .sock
-> 1042     conn.connect()
1044 if not conn.is_verified:
File /opt/conda/lib/python3.10/site-packages/urllib3/connection.py:363, in HTTPSConnection.connect(self)
361 def connect(self):
362     # Add certificate verification
--> 363     self.sock = conn = self._new_conn()
364     hostname = self.host
File /opt/conda/lib/python3.10/site-packages/urllib3/connection.py:186, in HTTPConnection._new_conn(self)
185 except SocketError as e:
--> 186     raise NewConnectionError(
187         self, "Failed to establish a new connection: %s" % e
188     )
190 return conn
NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x783edf828070>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution
During handling of the above exception, another exception occurred:
MaxRetryError                             Traceback (most recent call last)
File /opt/conda/lib/python3.10/site-packages/requests/adapters.py:486, in HTTPAdapter.send(self, request, stream, timeout, verify, cert, proxies)
485 try:
--> 486     resp = conn.urlopen(
487         method=request.method,
488         url=url,
489         body=request.body,
490         headers=request.headers,
491         redirect=False,
492         assert_same_host=False,
493         preload_content=False,
494         decode_content=False,
495         retries=self.max_retries,
496         timeout=timeout,
497         chunked=chunked,
498     )
500 except (ProtocolError, OSError) as err:
File /opt/conda/lib/python3.10/site-packages/urllib3/connectionpool.py:787, in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
785     e = ProtocolError("Connection aborted.", e)
--> 787 retries = retries.increment(
788     method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
789 )
790 retries.sleep()
File /opt/conda/lib/python3.10/site-packages/urllib3/util/retry.py:592, in Retry.increment(self, method, url, response, error, _pool, _stacktrace)
591 if new_retry.is_exhausted():
--> 592     raise MaxRetryError(_pool, url, error or ResponseError(cause))
594 log.debug("Incremented Retry for (url='%s'): %r", url, new_retry)
MaxRetryError: HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /bert-base-uncased/resolve/main/config.json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x783edf828070>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))
During handling of the above exception, another exception occurred:
ConnectionError                           Traceback (most recent call last)
File /opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py:1238, in hf_hub_download(repo_id, filename, subfolder, repo_type, revision, library_name, library_version, cache_dir, local_dir, local_dir_use_symlinks, user_agent, force_download, force_filename, proxies, etag_timeout, resume_download, token, local_files_only, legacy_cache_layout, endpoint)
1237 try:
-> 1238     metadata = get_hf_file_metadata(
1239         url=url,
1240         token=token,
1241         proxies=proxies,
1242         timeout=etag_timeout,
1243         library_name=library_name,
1244         library_version=library_version,
1245         user_agent=user_agent,
1246     )
1247 except EntryNotFoundError as http_error:
1248     # Cache the non-existence of the file and raise
File /opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py:118, in validate_hf_hub_args.._inner_fn(*args, **kwargs)
116     kwargs = smoothly_deprecate_use_auth_token(fn_name=fn.name, has_token=has_token, kwargs=kwargs)
--> 118 return fn(*args, **kwargs)
File /opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py:1631, in get_hf_file_metadata(url, token, proxies, timeout, library_name, library_version, user_agent)
1630 # Retrieve metadata
-> 1631 r = _request_wrapper(
1632     method="HEAD",
1633     url=url,
1634     headers=headers,
1635     allow_redirects=False,
1636     follow_relative_redirects=True,
1637     proxies=proxies,
1638     timeout=timeout,
1639 )
1640 hf_raise_for_status(r)
File /opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py:385, in _request_wrapper(method, url, follow_relative_redirects, **params)
384 if follow_relative_redirects:
--> 385     response = _request_wrapper(
386         method=method,
387         url=url,
388         follow_relative_redirects=False,
389         **params,
390     )
392     # If redirection, we redirect only relative paths.
393     # This is useful in case of a renamed repository.
File /opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py:408, in _request_wrapper(method, url, follow_relative_redirects, **params)
407 # Perform request and return if status_code is not in the retry list.
--> 408 response = get_session().request(method=method, url=url, **params)
409 hf_raise_for_status(response)
File /opt/conda/lib/python3.10/site-packages/requests/sessions.py:589, in Session.request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
588 send_kwargs.update(settings)
--> 589 resp = self.send(prep, **send_kwargs)
591 return resp
File /opt/conda/lib/python3.10/site-packages/requests/sessions.py:703, in Session.send(self, request, **kwargs)
702 # Send the request
--> 703 r = adapter.send(request, **kwargs)
705 # Total elapsed time of the request (approximately)
File /opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_http.py:67, in UniqueRequestIdAdapter.send(self, request, *args, **kwargs)
66 try:
---> 67     return super().send(request, *args, **kwargs)
68 except requests.RequestException as e:
File /opt/conda/lib/python3.10/site-packages/requests/adapters.py:519, in HTTPAdapter.send(self, request, stream, timeout, verify, cert, proxies)
517         raise SSLError(e, request=request)
--> 519     raise ConnectionError(e, request=request)
521 except ClosedPoolError as e:
ConnectionError: (MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /bert-base-uncased/resolve/main/config.json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x783edf828070>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))"), '(Request ID: 36d0930c-5de6-410e-962b-fee0e9868975)')
The above exception was the direct cause of the following exception:
LocalEntryNotFoundError                   Traceback (most recent call last)
File /opt/conda/lib/python3.10/site-packages/transformers/utils/hub.py:389, in cached_file(path_or_repo_id, filename, cache_dir, force_download, resume_download, proxies, token, revision, local_files_only, subfolder, repo_type, user_agent, _raise_exceptions_for_missing_entries, _raise_exceptions_for_connection_errors, _commit_hash, **deprecated_kwargs)
387 try:
388     # Load from URL or cache if already cached
--> 389     resolved_file = hf_hub_download(
390         path_or_repo_id,
391         filename,
392         subfolder=None if len(subfolder) == 0 else subfolder,
393         repo_type=repo_type,
394         revision=revision,
395         cache_dir=cache_dir,
396         user_agent=user_agent,
397         force_download=force_download,
398         proxies=proxies,
399         resume_download=resume_download,
400         token=token,
401         local_files_only=local_files_only,
402     )
403 except GatedRepoError as e:
File /opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py:118, in validate_hf_hub_args.._inner_fn(*args, **kwargs)
116     kwargs = smoothly_deprecate_use_auth_token(fn_name=fn.name, has_token=has_token, kwargs=kwargs)
--> 118 return fn(*args, **kwargs)
File /opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py:1371, in hf_hub_download(repo_id, filename, subfolder, repo_type, revision, library_name, library_version, cache_dir, local_dir, local_dir_use_symlinks, user_agent, force_download, force_filename, proxies, etag_timeout, resume_download, token, local_files_only, legacy_cache_layout, endpoint)
1369     else:
1370         # Otherwise: most likely a connection issue or Hub downtime => let's warn the user
-> 1371         raise LocalEntryNotFoundError(
1372             "An error happened while trying to locate the file on the Hub and we cannot find the requested files"
1373             " in the local cache. Please check your connection and try again or make sure your Internet connection"
1374             " is on."
1375         ) from head_call_error
1377 # From now on, etag and commit_hash are not None.
LocalEntryNotFoundError: An error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.
The above exception was the direct cause of the following exception:
OSError                                   Traceback (most recent call last)
Cell In[11], line 2
1 model_checkpoint = "/kaggle/input/pii-train-1-cp3000/Kaggle Checkpoints/checkpoint 3000"
----> 2 model = SpanMarkerModel.from_pretrained(model_checkpoint,local_files_only = True,
3                                        labels = [
4     '1-EMAIL', '1-ID_NUM', '1-NAME_STUDENT', '1-PHONE_NUM', '1-STREET_ADDRESS',
5     '1-URL_PERSONAL', '1-USERNAME', '2-ID_NUM', '2-NAME_STUDENT', '2-PHONE_NUM',
6     '2-STREET_ADDRESS', '2-URL_PERSONAL', 'O'
7 ])
9 device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
11 model.to(device);
File /opt/conda/lib/python3.10/site-packages/span_marker/modeling.py:269, in SpanMarkerModel.from_pretrained(cls, pretrained_model_name_or_path, labels, config, model_card_data, *model_args, **kwargs)
262     if parse(model_span_marker_version) < Version("1.0.0.dev"):
263         logger.warning(
264             f"Loading a model trained using SpanMarker v{model_span_marker_version},"
265             f" while SpanMarker v{span_marker_version} is installed. Due to large changes"
266             " introduced in v1.0.0, this is not recommended. Either retrain your model for"
267             f" v{span_marker_version}, or install span_marker < 1.0.0."
268         )
--> 269     model = super().from_pretrained(
270         pretrained_model_name_or_path, *model_args, config=config, **kwargs, model_card_data=model_card_data
271     )
273 # If 'pretrained_model_name_or_path' refers to an encoder (roberta, bert, distilbert, electra, etc.),
274 # then initialize it and create the SpanMarker config and model using the encoder and its config.
275 else:
276     encoder = SpanMarkerModel._load_encoder_with_kwargs(
277         pretrained_model_name_or_path, config, *model_args, **kwargs.copy()
278     )
File /opt/conda/lib/python3.10/site-packages/transformers/modeling_utils.py:3462, in PreTrainedModel.from_pretrained(cls, pretrained_model_name_or_path, config, cache_dir, ignore_mismatched_sizes, force_download, local_files_only, token, revision, use_safetensors, *model_args, **kwargs)
3456 config = cls._autoset_attn_implementation(
3457     config, use_flash_attention_2=use_flash_attention_2, torch_dtype=torch_dtype, device_map=device_map
3458 )
3460 with ContextManagers(init_contexts):
3461     # Let's make sure we don't run the init function of buffer modules
-> 3462     model = cls(config, *model_args, **model_kwargs)
3464 # make sure we use the model's config since the init call might have copied it
3465 config = model.config
File /opt/conda/lib/python3.10/site-packages/span_marker/modeling.py:80, in SpanMarkerModel.init(self, config, encoder, model_card_data, **kwargs)
71 # encoder will be specified if this Model is initializer via .from_pretrained with an encoder
72 # If .from_pretrained is called with a SpanMarkerModel instance, then we use the "traditional"
73 # PreTrainedModel.from_pretrained, which won't include an encoder keyword argument. In that case,
74 # we must create an "empty" encoder for PreTrainedModel.from_pretrained to fill with the correct
75 # weights.
76 if encoder is None:
77     # Load the encoder via the Config to prevent having to use AutoModel.from_pretrained, which
78     # could load e.g. all of roberta-large from the Hub unnecessarily.
79     # However, use the SpanMarkerModel updated vocab_size
---> 80     encoder_config = AutoConfig.from_pretrained(self.config.encoder["_name_or_path"], **self.config.encoder)
81     encoder = AutoModel.from_config(encoder_config)
82 self.encoder = encoder
File /opt/conda/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py:1082, in AutoConfig.from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
1079 trust_remote_code = kwargs.pop("trust_remote_code", None)
1080 code_revision = kwargs.pop("code_revision", None)
-> 1082 config_dict, unused_kwargs = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
1083 has_remote_code = "auto_map" in config_dict and "AutoConfig" in config_dict["auto_map"]
1084 has_local_code = "model_type" in config_dict and config_dict["model_type"] in CONFIG_MAPPING
File /opt/conda/lib/python3.10/site-packages/transformers/configuration_utils.py:644, in PretrainedConfig.get_config_dict(cls, pretrained_model_name_or_path, **kwargs)
642 original_kwargs = copy.deepcopy(kwargs)
643 # Get config dict associated with the base config file
--> 644 config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs)
645 if "_commit_hash" in config_dict:
646     original_kwargs["_commit_hash"] = config_dict["_commit_hash"]
File /opt/conda/lib/python3.10/site-packages/transformers/configuration_utils.py:699, in PretrainedConfig._get_config_dict(cls, pretrained_model_name_or_path, **kwargs)
695 configuration_file = kwargs.pop("_configuration_file", CONFIG_NAME)
697 try:
698     # Load from local folder or from cache or download from model Hub and cache
--> 699     resolved_config_file = cached_file(
700         pretrained_model_name_or_path,
701         configuration_file,
702         cache_dir=cache_dir,
703         force_download=force_download,
704         proxies=proxies,
705         resume_download=resume_download,
706         local_files_only=local_files_only,
707         token=token,
708         user_agent=user_agent,
709         revision=revision,
710         subfolder=subfolder,
711         _commit_hash=commit_hash,
712     )
713     commit_hash = extract_commit_hash(resolved_config_file, commit_hash)
714 except EnvironmentError:
715     # Raise any environment error raise by cached_file. It will have a helpful error message adapted to
716     # the original exception.
File /opt/conda/lib/python3.10/site-packages/transformers/utils/hub.py:429, in cached_file(path_or_repo_id, filename, cache_dir, force_download, resume_download, proxies, token, revision, local_files_only, subfolder, repo_type, user_agent, _raise_exceptions_for_missing_entries, _raise_exceptions_for_connection_errors, _commit_hash, **deprecated_kwargs)
427     if not _raise_exceptions_for_missing_entries or not _raise_exceptions_for_connection_errors:
428         return None
--> 429     raise EnvironmentError(
430         f"We couldn't connect to '{HUGGINGFACE_CO_RESOLVE_ENDPOINT}' to load this file, couldn't find it in the"
431         f" cached files and it looks like {path_or_repo_id} is not the path to a directory containing a file named"
432         f" {full_filename}.\nCheckout your internet connection or see how to run the library in offline mode at"
433         " 'https://huggingface.co/docs/transformers/installation#offline-mode style="color:rgb(175,0,0)">'."
434     ) from e
435 except EntryNotFoundError as e:
436     if not _raise_exceptions_for_missing_entries:
OSError: We couldn't connect to 'https://huggingface.co/' to load this file, couldn't find it in the cached files and it looks like bert-base-uncased is not the path to a directory containing a file named config.json.
Checkout your internet connection or see how to run the library in offline mode at 'https://huggingface.co/docs/transformers/installation#offline-mode'.

That's very useful information right there! I see the issue now. This should fix it for you:

import json
from typing import Optional
from transformers import PreTrainedModel, BertModel, BertConfig
from span_marker import SpanMarkerModel
from span_marker.configuration import SpanMarkerConfig
from span_marker.model_card import SpanMarkerModelCardData

class BertSpanMarkerModel(SpanMarkerModel):
    def __init__(
        self,
        config: SpanMarkerConfig,
        encoder: Optional[PreTrainedModel] = None,
        model_card_data: Optional[SpanMarkerModelCardData] = None,
        **kwargs,
    ) -> None:
        if encoder is None:
            encoder = BertModel(BertConfig(**config.encoder))
        super().__init__(config, encoder, model_card_data, **kwargs)

with open(r"/kaggle/input/pii-train-1-cp3000/Kaggle Checkpoints/checkpoint 3000/config.json") as f:
    data = json.load(f)
try:
    del data["encoder"]["_name_or_path"]
    with open(r"/kaggle/input/pii-train-1-cp3000/Kaggle Checkpoints/checkpoint 3000/config.json", "w") as f:
        json.dump(data, f)
except Exception:
    pass

model = BertSpanMarkerModel.from_pretrained("/kaggle/input/pii-train-1-cp3000/Kaggle Checkpoints/checkpoint 3000", local_files_only=True, labels = [
'1-EMAIL', '1-ID_NUM', '1-NAME_STUDENT', '1-PHONE_NUM', '1-STREET_ADDRESS',
'1-URL_PERSONAL', '1-USERNAME', '2-ID_NUM', '2-NAME_STUDENT', '2-PHONE_NUM',
'2-STREET_ADDRESS', '2-URL_PERSONAL', 'O'
])

If you're interested in the mistakes, here they are: The model makes two errors when trying to load without internet:

Tries to load the "base encoder" (e.g. bert-base-uncased) config when initializing a trained SpanMarker model; this is necessary to create an "empty" SpanMarker model that can then be filled with the saved weights. It uses the base encoder config to infer the base encoder model class, but we can help it out with the above BertSpanMarkerModel. We know our model is a BertModel, so we use that. (You'd use RobertaModel if you use a Roberta model here, for example)
Tries to load the "base encoder" tokenizer. This is kind of unnecessary; the tokenizer is also included in the saved SpanMarker model, which is the "second option" in my code. We want it to skip the first option (as that one crashes) and go straight to option 2. We can do this by opening our config.json and removing the _name_or_path from the encoder.

Tom Aarsen

Hi Tom, unfortunately after copy pasting the code, that issue still persists: https://www.kaggle.com/jdonnelly0804/pii-infer
Were you able to make it work on your end?
import json
from typing import Optional
from transformers import PreTrainedModel, BertModel, BertConfig
from span_marker import SpanMarkerModel
from span_marker.configuration import SpanMarkerConfig
from span_marker.model_card import SpanMarkerModelCardData
class BertSpanMarkerModel(SpanMarkerModel):
def init(
self,
config: SpanMarkerConfig,
encoder: Optional[PreTrainedModel] = None,
model_card_data: Optional[SpanMarkerModelCardData] = None,
**kwargs,
) -> None:
if encoder is None:
encoder = BertModel(BertConfig(**config.encoder))
super().init(config, encoder, model_card_data, **kwargs)
with open(r"/kaggle/input/pii-train-1-cp3000/Kaggle Checkpoints/checkpoint 3000/config.json") as f:
data = json.load(f)
try:
del data["encoder"]["_name_or_path"]
with open(r"/kaggle/input/pii-train-1-cp3000/Kaggle Checkpoints/checkpoint 3000/config.json", "w") as f:
json.dump(data, f)
except Exception:
pass
model = BertSpanMarkerModel.from_pretrained("/kaggle/input/pii-train-1-cp3000/Kaggle Checkpoints/checkpoint 3000", local_files_only=True, labels = [
'1-EMAIL', '1-ID_NUM', '1-NAME_STUDENT', '1-PHONE_NUM', '1-STREET_ADDRESS',
'1-URL_PERSONAL', '1-USERNAME', '2-ID_NUM', '2-NAME_STUDENT', '2-PHONE_NUM',
'2-STREET_ADDRESS', '2-URL_PERSONAL', 'O'
])

LocalEntryNotFoundError                   Traceback (most recent call last)
File /opt/conda/lib/python3.10/site-packages/transformers/utils/hub.py:389, in cached_file(path_or_repo_id, filename, cache_dir, force_download, resume_download, proxies, token, revision, local_files_only, subfolder, repo_type, user_agent, _raise_exceptions_for_missing_entries, _raise_exceptions_for_connection_errors, _commit_hash, **deprecated_kwargs)
387 try:
388     # Load from URL or cache if already cached
--> 389     resolved_file = hf_hub_download(
390         path_or_repo_id,
391         filename,
392         subfolder=None if len(subfolder) == 0 else subfolder,
393         repo_type=repo_type,
394         revision=revision,
395         cache_dir=cache_dir,
396         user_agent=user_agent,
397         force_download=force_download,
398         proxies=proxies,
399         resume_download=resume_download,
400         token=token,
401         local_files_only=local_files_only,
402     )
403 except GatedRepoError as e:
File /opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py:118, in validate_hf_hub_args.._inner_fn(*args, **kwargs)
116     kwargs = smoothly_deprecate_use_auth_token(fn_name=fn.name, has_token=has_token, kwargs=kwargs)
--> 118 return fn(*args, **kwargs)
File /opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py:1362, in hf_hub_download(repo_id, filename, subfolder, repo_type, revision, library_name, library_version, cache_dir, local_dir, local_dir_use_symlinks, user_agent, force_download, force_filename, proxies, etag_timeout, resume_download, token, local_files_only, legacy_cache_layout, endpoint)
1361 if local_files_only:
-> 1362     raise LocalEntryNotFoundError(
1363         "Cannot find the requested files in the disk cache and outgoing traffic has been disabled. To enable"
1364         " hf.co look-ups and downloads online, set 'local_files_only' to False."
1365     )
1366 elif isinstance(head_call_error, RepositoryNotFoundError) or isinstance(head_call_error, GatedRepoError):
1367     # Repo not found => let's raise the actual error
LocalEntryNotFoundError: Cannot find the requested files in the disk cache and outgoing traffic has been disabled. To enable hf.co look-ups and downloads online, set 'local_files_only' to False.
The above exception was the direct cause of the following exception:
OSError                                   Traceback (most recent call last)
Cell In[7], line 29
26 except Exception:
27     pass
---> 29 model = BertSpanMarkerModel.from_pretrained("/kaggle/input/pii-train-1-cp3000/Kaggle Checkpoints/checkpoint 3000", local_files_only=True, labels = [
30 '1-EMAIL', '1-ID_NUM', '1-NAME_STUDENT', '1-PHONE_NUM', '1-STREET_ADDRESS',
31 '1-URL_PERSONAL', '1-USERNAME', '2-ID_NUM', '2-NAME_STUDENT', '2-PHONE_NUM',
32 '2-STREET_ADDRESS', '2-URL_PERSONAL', 'O'
33 ])
File /opt/conda/lib/python3.10/site-packages/span_marker/modeling.py:302, in SpanMarkerModel.from_pretrained(cls, pretrained_model_name_or_path, labels, config, model_card_data, *model_args, **kwargs)
298     model = cls(config, encoder, *model_args, **kwargs, model_card_data=model_card_data)
300 # Pass the tokenizer directly to the model for convenience, this way the user doesn't have to
301 # make it themselves.
--> 302 tokenizer = SpanMarkerTokenizer.from_pretrained(
303     config.encoder.get("_name_or_path", pretrained_model_name_or_path), config=config, **kwargs
304 )
305 model.set_tokenizer(tokenizer)
306 # Since transformers 4.32.0 we should use pad_to_multiple_of=8.
307 # That'll fail for earlier versions, so we try-except it.
File /opt/conda/lib/python3.10/site-packages/span_marker/tokenizer.py:277, in SpanMarkerTokenizer.from_pretrained(cls, pretrained_model_name_or_path, config, *inputs, **kwargs)
275 @classmethod
276 def from_pretrained(cls, pretrained_model_name_or_path: Union[str, os.PathLike], *inputs, config=None, **kwargs):
--> 277     tokenizer = AutoTokenizer.from_pretrained(
278         pretrained_model_name_or_path, *inputs, **kwargs, add_prefix_space=True
279     )
280     # XLM-R is known to have some tokenization issues, so be sure to also split on punctuation.
281     # Strictly required for inference, shouldn't affect training.
282     if isinstance(tokenizer, XLMRobertaTokenizerFast):
File /opt/conda/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py:752, in AutoTokenizer.from_pretrained(cls, pretrained_model_name_or_path, *inputs, **kwargs)
750 if config_tokenizer_class is None:
751     if not isinstance(config, PretrainedConfig):
--> 752         config = AutoConfig.from_pretrained(
753             pretrained_model_name_or_path, trust_remote_code=trust_remote_code, **kwargs
754         )
755     config_tokenizer_class = config.tokenizer_class
756     if hasattr(config, "auto_map") and "AutoTokenizer" in config.auto_map:
File /opt/conda/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py:1082, in AutoConfig.from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
1079 trust_remote_code = kwargs.pop("trust_remote_code", None)
1080 code_revision = kwargs.pop("code_revision", None)
-> 1082 config_dict, unused_kwargs = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
1083 has_remote_code = "auto_map" in config_dict and "AutoConfig" in config_dict["auto_map"]
1084 has_local_code = "model_type" in config_dict and config_dict["model_type"] in CONFIG_MAPPING
File /opt/conda/lib/python3.10/site-packages/transformers/configuration_utils.py:644, in PretrainedConfig.get_config_dict(cls, pretrained_model_name_or_path, **kwargs)
642 original_kwargs = copy.deepcopy(kwargs)
643 # Get config dict associated with the base config file
--> 644 config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs)
645 if "_commit_hash" in config_dict:
646     original_kwargs["_commit_hash"] = config_dict["_commit_hash"]
File /opt/conda/lib/python3.10/site-packages/transformers/configuration_utils.py:699, in PretrainedConfig._get_config_dict(cls, pretrained_model_name_or_path, **kwargs)
695 configuration_file = kwargs.pop("_configuration_file", CONFIG_NAME)
697 try:
698     # Load from local folder or from cache or download from model Hub and cache
--> 699     resolved_config_file = cached_file(
700         pretrained_model_name_or_path,
701         configuration_file,
702         cache_dir=cache_dir,
703         force_download=force_download,
704         proxies=proxies,
705         resume_download=resume_download,
706         local_files_only=local_files_only,
707         token=token,
708         user_agent=user_agent,
709         revision=revision,
710         subfolder=subfolder,
711         _commit_hash=commit_hash,
712     )
713     commit_hash = extract_commit_hash(resolved_config_file, commit_hash)
714 except EnvironmentError:
715     # Raise any environment error raise by cached_file. It will have a helpful error message adapted to
716     # the original exception.
File /opt/conda/lib/python3.10/site-packages/transformers/utils/hub.py:429, in cached_file(path_or_repo_id, filename, cache_dir, force_download, resume_download, proxies, token, revision, local_files_only, subfolder, repo_type, user_agent, _raise_exceptions_for_missing_entries, _raise_exceptions_for_connection_errors, _commit_hash, **deprecated_kwargs)
427     if not _raise_exceptions_for_missing_entries or not _raise_exceptions_for_connection_errors:
428         return None
--> 429     raise EnvironmentError(
430         f"We couldn't connect to '{HUGGINGFACE_CO_RESOLVE_ENDPOINT}' to load this file, couldn't find it in the"
431         f" cached files and it looks like {path_or_repo_id} is not the path to a directory containing a file named"
432         f" {full_filename}.\nCheckout your internet connection or see how to run the library in offline mode at"
433         " 'https://huggingface.co/docs/transformers/installation#offline-mode style="color:rgb(175,0,0)">'."
434     ) from e
435 except EntryNotFoundError as e:
436     if not _raise_exceptions_for_missing_entries:
OSError: We couldn't connect to 'https://huggingface.co/' to load this file, couldn't find it in the cached files and it looks like bert-base-uncased is not the path to a directory containing a file named config.json.
Checkout your internet connection or see how to run the library in offline mode at 'https://huggingface.co/docs/transformers/installation#offline-mode'.


Adding message truncation so you don't accidentally crash your softwa…

First, this a a great log handler!
Second, I think software should have good defaults. When presented with large log entries (ex: dumping an http request) that exceeds Cloudwatch event size limits, this log handler will crash your software. Crashing is not a good default.
This commit adds truncation.
In some cases you might want to split the log entry into multiple parts. If someone is interested, I would suggest you extend this commit and give this handler two different modes 1) truncate 2) split log message.
But for now, here is the truncate piece.

Hi @computercolin,
Thanks for your contribution! I will take your branch and add a couple of things listed here:

The default option will continue to raise an error. Unfortunatelly, we can't push such a critical behaviour change without affecting others.
I will add a new parameter with two options as you suggest: truncate and split
I will change the formatting to adhere to PEP guideliness

Unfortunately I will need to reject the PR to make sure all of the above are covered in the same version. But I wanted to let you know that I appreciate very much your contribution!
Thanks!


how to directly use checkpoint from stylegan2-ada-pytorch

Hi, thanks for your great work and easy to use streamlit app. The current available checkpoint has impressed me  a lot, however, i find it hard to use other checkpoint from nvlab official repositorys such as stylegan2-ada-pytorch. Detail list following:
different checkpoint type
genfporce checkpoint
I try to download some checkpoint from 'model zoo', the checkpoint end with .pth and load through torch.load()
stylegan2-ada-checkpoint
My trained checkpoint end with .pkl. It seems to be saved in nvlab's unique method.
my effort

Directly use torch.load() to load checkpoint

Search for some method to convert the pkl checkpoint in genforce repository issues.
But I failed to find a resolution.

You only need to figure out how to visit the weight you would like to factorize. No matter what type of checkpoint you are dealing with, loading the parameters as numpy.ndarray will be good enough. The key implementation is only with two lines.

Thanks, I have deployed this algorithm in implicit ae, and it is well to DIY in nvlab's repository.

Hi @tianyilt, did you get this to work?
Have a stylegan2-ada .pkl file that i would like to run as well using Sefa


Cannot open url on any browser

I am trying to use google-drive-ocamlfuse on a vps without display (Ubuntu server 20.04 TLS) and its trying to opena browser, I tried to install chromium to avoid this issue, but then its trying to open chromium as getting error.

Check this comment (https://github.com/astrada/google-drive-ocamlfuse/issues/764#issuecomment-1076281507), maybe you can do the same.

I just ran into this on Ubuntu Server and #764 helped me out here. Follow the directions provided there by creating a web client with the redirect URI of http:<IP_ADDRESS>:8080/oauth2callback. Then try running the command with these options:
google-drive-ocaml-fuse -id *CLIENT_ID* -secret *CLIENT_SECRET* -headless -redirect_uri http:<IP_ADDRESS>:8080/oauth2callback
You should land on an error page after you've authenticated the application. This is where you can nab the verification code you need from the URL params of that address.

I literally just ran into this problem but didn't want to bother with creating a client, etc. My workaround: create a "firefox" executable that dumps the auth url somewhere I can grab and that I can easily close "remotely":
cat <<-EOF > ./firefox
	#! /bin/env bash
	echo "$@" > "${HOME}/url"
	while ! [ -e "${HOME}/go" ]; do sleep 1; done
	rm "${HOME}/go"
EOF
chmod a+x ./firefox
sudo mv ./firefox /usr/bin/firefox

... Now when you run google-drive-ocamlfuse for auth, it halts while you cat out the "url" file for copying, paste it into the browser on your actual desktop, run through auth, then touch ~/go to terminate the "browser" so the tool can write the config file.
Hacky, yes... but it works, and it's faster and easier than creating a custom OAuth client.  🤷


Chinese text can't auto split

When I try to add some Traditional Chinese font into the text() function, the text() function DIDN'T split the text normally

Code
$image->text(
    "大家小時候都有寫過紀念冊嗎？通過紀念冊上的文字和圖案，可以將大家美好的回憶記錄下來。今年我們希望製作出所有星詠者都能夠参與的線下活動。透過成員互相分享自己與星街同行的經歷與回憶，讓這些窩心的信息化為各地星詠者對星街的支持，藉此加強星街與星詠者以及星詠者之間的連繫與羈絆。",
    array(
        'x'          => $ix,
        'y'          => $iy,
        'width'      => $mx,
        'height'     => $my,
        'horizontal' => 'start',
        'vertical'   => 'start',
        'fontpath'   => 'NotoSansTC-Regular.otf',
        'lineheight' => 1.75,
        'color'      => '#ffffff',
        'opacity'    => 1,
        'debug'      => false,
    ),
    $boundary
);

@TaddeoLeung Hi there and thank you for report.
I will try to fix it soon.

Fast update:
The problem may be that you don't have spaces in the text.
Can you try to change your text with:
"大家小時候都有寫過紀念冊嗎? 通過紀念冊上的文字和圖案, 可以將大家美好的回憶記錄下來。今年我們希望製作出所有星詠者都能夠参與的線下活動。透過成員互相分享自己與星街同行的經歷與回憶, 讓這些窩心的信息化為各地星詠者對星街的支持, 藉此加強星街與星詠者以及星詠者之間的連繫與羈絆。",
Unfortanally I do not know Chinese, but it seems that symbols ，。 withoud required spaces.

Update:
It work if i add " " manually
But It's just not convenient to use in the picture generator system which design for chinese users.

@TaddeoLeung can you send me all characters that should be delimiters instead of spaces, please?
I can add them to a new version of this class.

，。？！
There you Go, These 4 types of characters should be delimiters

@TaddeoLeung check this examples, please:
https://github.com/antonlukin/poster-editor/blob/master/example/chinese.php
https://github.com/antonlukin/poster-editor/blob/master/example/samples/chinese.jpg

Look Good !


Add validation on comments model

We need to add the validations on the comments model.

Text is required
Add minimum of 10 characters limit
Add maximum of 500 characters limit

#19

Validation added on the comment model, as described above. And also the error messages are shown on Add Comment, Edit Comment and Reply Modal.
But there is only one UI Bug that needs to be fixed, is that the , is displayed while showing the error messages on the Reply Form Modal.
Below is the screenshot:

Hey! @ShehrozIrfan!
I want to fix this issue.


Creating instance of TestServer throws exception after upgrading to VS 2017

Hi, after successfully converting a VS2015 ASP.NET Core solution to VS2017, the only thing that I can't get to work is the integration test I had set up using Microsoft.AspNetCore.TestHost.TestServer (which, when it works is awesome by the way). The code I use to run is very simple:
var builder = new WebHostBuilder().UseStartup<Startup>();
var server = new TestServer(builder);
At the point at which it tries to construct TestServer, the following exception is thrown:
System.IO.FileLoadException: 'Could not load file or assembly 'Microsoft.Extensions.Configuration.Abstractions, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=adb9793829ddae60' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)'
Extra info under 'FusionLog' states:
Pre-bind state information  LOG: DisplayName = Microsoft.Extensions.Configuration.Abstractions, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=adb9793829ddae60  (Fully-specified) LOG: Appbase = file:///C:/<ommitted>/bin/Debug/net451 LOG: Initial PrivatePath = NULL Calling assembly : Microsoft.ApplicationInsights.AspNetCore, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=31bf3856ad364e35.
LOG: This bind starts in default load context. LOG: Using application configuration file: C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO\2017\ENTERPRISE\COMMON7\IDE\COMMONEXTENSIONS\MICROSOFT\TESTWINDOW\vstest.executionengine.x86.exe.Config LOG: Using host configuration file:  LOG: Using machine configuration file from C:\Windows\Microsoft.NET\Framework\v4.0.30319\config\machine.config. LOG: Post-policy reference: Microsoft.Extensions.Configuration.Abstractions, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=adb9793829ddae60 LOG: Attempting download of new URL file:///C:/<ommitted>/bin/Debug/net451/Microsoft.Extensions.Configuration.Abstractions.DLL. WRN: Comparing the assembly name resulted in the mismatch: Build Number ERR: Failed to complete setup of assembly (hr = 0x80131040). Probing terminated.
I am riding the LTS train of dotnet core (v 1.0.3) and using Microsoft.AspNetCore.TestHost v 1.0.2. Looking at the other projects in my solution, any references to Microsoft.Extensions.Configuration.Abstractions are transitively included through other dependencies and are all version 1.0.2. All projects target framework dotnet451 (not appcore). This all works fine in my VS2015 solution and I haven't changed the versions going to VS2017. What I have tried:

Deleting all bin and obj directories, restart and recompile after everything I changed.
Upgrading Microsoft.AspNetCore to v 1.0.4
targeting dotnet452 instead of dotnet451
Reproducing the situation by creating a simple new VS2017 project. This error was reproducable, but was solved when I upgraded Microsoft.AspNetCore to v 1.0.4. Unfortunately this same action did not solve the problem in my main solution.
Adding a bindingRedirect in web.config to force the said library to use version 1.0.2. but noticed no difference, in fact, I then saw that the application config file in the bin directory already contains this redirect (and others) as generated by dot net.

The next thing I'm going to try is to upgrade everything to the 1.1.1 FTS version, although I feel that really the LTS should just work.
Any help would be much appreciated.

Can you post your project file so I can try to reproduce the issue?

Hi John,
Thanks for your reply. I have done some more digging and found that what is happening is that the test project doesn't pick up the config of the host application, and so it is missing the bindingRedirects which would normally be applied. Copying the config file of the host application into the test project, renaming it to app.config and setting it to 'copy always' fixed the problem. However, this led to a second problem where it would seem that any call to a Controller in a different assembly returns 404. Maybe I this should be registered as a separate issue, but I can't tell if this is related or not.
I have created a project in which you can reproduce both problems. I have published the code here.
The file 'README.txt' explains the exact steps to reproduce the issue.
I look forward to your reply,
Chris

By the way, if you find in my sample that the test doesn't show up after converting to vs2017, change the following references in the MyTest project:
"Microsoft.NET.Test.Sdk" Version="15.0.0-preview-20170106-08" ---> "Microsoft.NET.Test.Sdk" Version="15.0.0"
"xunit.runner.visualstudio" Version="2.2.0-beta5-build1225" ---> "xunit.runner.visualstudio" Version="2.2.0"

@anurse has the binding redirect workaround

https://github.com/Microsoft/vstest/issues/428 describes the issue and the workaround.

Ok, thanks for the info. And is there any info on the second problem I reported?

Seems like there are a few other issues involved.
https://github.com/Microsoft/vstest/issues/196
https://github.com/NuGet/Home/issues/4412
We work around the issue with
https://github.com/aspnet/Mvc/blob/dev/test/Microsoft.AspNetCore.Mvc.FunctionalTests/Microsoft.AspNetCore.Mvc.FunctionalTests.csproj#L30-L34
https://github.com/aspnet/Mvc/blob/dev/test/Microsoft.AspNetCore.Mvc.FunctionalTests/Microsoft.AspNetCore.Mvc.FunctionalTests.csproj#L75-L85
Gives these a try.

Many thanks guys, these workarounds solved my problem and allow me to get on with enjoying VS2017. Just for anybody else stumbling on this issue, I'll summarize what I did below:
Add a new file 'xunit.runner.json' to the test project containg the following code:
{
  "shadowCopy": false
}

Then add the following code to the test project:
<!-- Solves Problem#1 (binding error) https://github.com/Microsoft/vstest/issues/428. -->
<PropertyGroup>
    <AutoGenerateBindingRedirects>true</AutoGenerateBindingRedirects>
    <GenerateBindingRedirectsOutputType>true</GenerateBindingRedirectsOutputType>
</PropertyGroup>

<!-- Solves Problem#2 (404 when executing service calls hosted in other assemblies) -->
<!-- https://github.com/Microsoft/vstest/issues/196.-->
<ItemGroup>
    <None Update="xunit.runner.json">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
    </None>
</ItemGroup>
<!-- https://github.com/NuGet/Home/issues/4412. -->
<Target Name="CopyDepsFiles" AfterTargets="Build" Condition="'$(TargetFramework)'!=''">
    <ItemGroup>
        <DepsFilePaths Include="$([System.IO.Path]::ChangeExtension('%(_ResolvedProjectReferencePaths.FullPath)', '.deps.json'))" />
    </ItemGroup>

    <Copy SourceFiles="%(DepsFilePaths.FullPath)" DestinationFolder="$(OutputPath)" Condition="Exists('%(DepsFilePaths.FullPath)')" />
</Target>

Closing since the issue is not in Hosting and there are tracking issues and workarounds for the known problems.

@Chris197  an extra kudo from me!! 😎


Rewrite @import paths like we do url() paths

Added regex replacement for @import "[path]" pattern
Added @import "[path]" test
Added @import url("[path]") test
Created a helper that waits for <style> tags to finish loading @import statements
Deferred running tests until after @import's are loaded
Rewrote existing test messages to be more descriptive

Closes #12.

Tested in Chrome, Chrome Canary, Firefox, and Safari.


Add web/agencies/ and update app.yaml to accept json

What's changed, or what was fixed?

Adding /web/agencies/ page
Create a directory widget which includes a map with pins and cards based on a JSON file - NOTE: This is within a framebox so the JavaScript should be allowed.
Updated app.yaml to accept JSON (to be able to host this as a static file) - NOTE: If there is a better way of doing this without changing app.yaml, let me know.  This seemed to be the only way to be able to host this JSON as well.  I also would like to ideally host via Firebase where it would be retrieving realtime data vs. me having to create a PR each time this data needs to be updated.

Staging link : https://devsite.googleplex.com/web/agencies/
Target Live Date: 2018-03-15

[X] This has been reviewed and approved by (sgeer@google, fguerrero@google)
[X] I have run gulp test locally and all tests pass.
[X] I have added the appropriate type-something label.
[X] I've staged the site and manually verified that my content displays correctly.

CC: @petele

Whoops!
There were 2 warnings that will prevent this PR from being merged. Please take a look, and either fix, or provide a justification for why they can't be fixed.
WARNINGS
app.yaml - 'app.yaml' was changed, was that intentional?
src/content/en/agencies/js/directory.js - JavaScript files are generally not allowed.

Whoops!
There were 2 warnings that will prevent this PR from being merged. Please take a look, and either fix, or provide a justification for why they can't be fixed.
WARNINGS
app.yaml - 'app.yaml' was changed, was that intentional?
src/content/en/agencies/js/directory.js - JavaScript files are generally not allowed.

Whoops!
There were 2 warnings that will prevent this PR from being merged. Please take a look, and either fix, or provide a justification for why they can't be fixed.
WARNINGS
app.yaml - 'app.yaml' was changed, was that intentional?
src/content/en/agencies/js/directory.js - JavaScript files are generally not allowed.

Whoops!
There were 2 warnings that will prevent this PR from being merged. Please take a look, and either fix, or provide a justification for why they can't be fixed.
WARNINGS
app.yaml - 'app.yaml' was changed, was that intentional?
src/content/en/agencies/js/directory.js - JavaScript files are generally not allowed.

@ebidel @petele Hi guys! Can you review this again, I made changes last week. Thanks!

Whoops!
There were 5 warnings that will prevent this PR from being merged. Please take a look, and either fix, or provide a justification for why they can't be fixed.
WARNINGS
app.yaml - 'app.yaml' was changed, was that intentional?
src/content/en/agencies/_partials/web-directory.html#L22 - '

Whoops!
There were 6 warnings that will prevent this PR from being merged. Please take a look, and either fix, or provide a justification for why they can't be fixed.
WARNINGS
app.yaml - 'app.yaml' was changed, was that intentional?
src/content/en/agencies/_partials/web-directory.html#L22 - '

:tada: This has been pushed live to https://developers.google.com/web/


Cache keys based on HTTP response headers.

close #1

Question: is this cachable? What do we do with key results that aren't cachable?
$ curl -v https://login.salesforce.com/id/keys
*   Trying <IP_ADDRESS>...
* Connected to login.salesforce.com (<IP_ADDRESS>) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* ALPN/NPN, server did not agree to a protocol
* SSL connection using TLS_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* 	subject: CN=login.salesforce.com,OU=Applications,O="Salesforce.com, Inc",L=San Francisco,ST=California,C=US
* 	start date: Mar 03 00:00:00 2016 GMT
* 	expire date: Jul 01 23:59:59 2018 GMT
* 	common name: login.salesforce.com
* 	issuer: CN=Symantec Class 3 Secure Server CA - G4,OU=Symantec Trust Network,O=Symantec Corporation,C=US
> GET /id/keys HTTP/1.1
> Host: login.salesforce.com
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Fri, 15 Jul 2016 05:18:38 GMT
< Strict-Transport-Security: max-age=10886400; includeSubDomains; preload
< Content-Security-Policy-Report-Only: default-src https:; script-src https: 'unsafe-inline' 'unsafe-eval'; style-src https: 'unsafe-inline'; img-src https: data:; font-src https: data:; report-uri /_/ContentDomainCSPNoAuth?type=login
< Set-Cookie: BrowserId=hyXyLx3qTWCH2kXOOR69rA;Path=/;Domain=.salesforce.com;Expires=Tue, 13-Sep-2016 05:18:38 GMT
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Content-Type: application/json;charset=UTF-8
< Pragma: no-cache
< Cache-Control: no-cache, no-store
< Transfer-Encoding: chunked
<


request_transformer not adding key:value to message body

I've setup two services, one I wrote in Java with an interceptor as well as a mockbin service, both of which show the insertion into the query string and header by the request_transformer.  I can't get confirmation that the insertion into the message body is taking place.
The mockbin is located at:  http://www.mockbin.com/bin/e4dee421-0e16-482d-ba69-4b507bef3346/log
Below is my Kong configuration:
$ curl -X GET http://localhost:8001/apis

{
  "data": [
    {
      "path": "\/mock",
      "target_url": "http:\/\/www.mockbin.com\/bin\/e4dee421-0e16-482d-ba69-4b507bef3346",
      "created_at":<PHONE_NUMBER>000,
      "id": "07ecccc6-e7a2-494d-c95d-f4e8bcc436c9",
      "name": "mock",
      "strip_path": true
    }
  ]
}

$ curl -X GET http://localhost:8001/apis/mock/plugins

{
  "data": [{
    "api_id": "07ecccc6-e7a2-494d-c95d-f4e8bcc436c9",
    "id": "5ea3f322-f77e-4dbf-cd7d-70a353089b1e",
    "value": {
      "add": {
        "querystring": ["testQueryStrKey:20u3ihf2ku1khe89"],
        "headers": ["testHeaderKey:20u3ihf2ku1khe89"],
        "form": ["testBodyKey:20u3ihf2ku1khe89"]
      }
    },
    "enabled": true,
    "name": "request_transformer",
    "created_at":<PHONE_NUMBER>000
  }]
}

Below are the 3 commands I issued:
curl -X POST -H "Content-Type: application/json" -H "Accept: application/json" http://localhost:8000/mock/ -d '{"name":"Mike"}'

curl -X GET -H "Content-Type: application/json" -H "Accept: application/json" http://localhost:8000/mock/

curl -X PUT -H "Content-Type: application/json" -H "Accept: application/json" http://localhost:8000/mock/ -d '{"name":"Mike"}'

@bisoldi seems like your example of the JSON output from /apis is duplicated, can you please update the ticket?
P.S. I cleaned up the formatting.

as for the actual issue, I just tested locally and reviewed your mockbin example, it seems you're sending an application/json content-type for the request ... which is the problem
the request_transformer plugin currently only supports Form Data (a.k.a content type of: application/x-www-form-urlencoded)

Thank you for the help with that.  I think it would be beneficial if the documentation for the plugin (likely the response transformer as well?) clearly stipulated that requirement.

@thefosk ^

@thefosk ^

Thank you for the help with that. I think it would be beneficial if the documentation for the plugin (likely the response transformer as well?) clearly stipulated that requirement.

agreed

Closing this in favor of #569.


bugfix: 修复插件显示逻辑

Codecov Report

Merging #841 (d292ecd) into v2.6.x_develop (f0833fe) will not change coverage.
The diff coverage is n/a.

@@               Coverage Diff               @@
##           v2.6.x_develop     #841   +/-   ##
===============================================
  Coverage           61.15%   61.15%
===============================================
  Files                 461      461
  Lines               29292    29292
===============================================
  Hits                17914    17914
  Misses              11378    11378

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f0833fe...d292ecd. Read the comment docs.


V 3/Add wui-grid component

Changes

feat: Add wui-grid component

When gap is set, it overrules the column-gap and row-gap. I've added an undefined option in Storybook for spacing properties.
Removed duplicate value


When compiling TensorFlow 2.13 using bazel+clang, an error was reported as fatal error: 'stddef. h' file not found

Issue type
Build/Install
Have you reproduced the bug with TensorFlow Nightly?
No
Source
source
TensorFlow version
2.13
Custom code
No
OS platform and distribution
No response
Mobile device
No response
Python version
No response
Bazel version
No response
GCC/compiler version
No response
CUDA/cuDNN version
No response
GPU model and memory
No response
Current behavior?
I use tensorflow/build:2.13-python3.11 image
When compiling TensorFlow 2.13 using bazel+clang, an error was reported as fatal error: 'stddef. h' file not found.
but I can find 'stddef. h' in docker container path /usr/lib/llvm-16/lib/clang/16/include ，if I manual append -isystem /usr/lib/llvm-16/lib/clang/16/include to clang compile command, the bellow code can run success
Standalone code to reproduce the issue
"cd /root/.cache/bazel/_bazel_root/645133528a7a8476e9bedcf54eb858b4/execroot/org_tensorflow && \
  exec env - \
    DOCKER_HOST_CACHEBUSTER=1682977560680045781 \
    LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/lib \
    PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
    PWD=/proc/self/cwd \
  /usr/bin/clang-16 -MD -MF bazel-out/host/bin/external/zlib/_objs/zlib/uncompr.d '-frandom-seed=bazel-out/host/bin/external/zlib/_objs/zlib/uncompr.o' -iquote external/zlib -iquote bazel-out/host/bin/external/zlib  -isystem external/zlib -isystem bazel-out/host/bin/external/zlib -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -fPIE -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -Wall -Wno-invalid-partial-specialization -fno-omit-frame-pointer -no-canonical-prefixes -DNDEBUG -g0 -O2 -ffunction-sections -fdata-sections '--cuda-path=/usr/local/cuda-11.8' -g0 -w -Wno-shift-negative-value -DZ_HAVE_UNISTD_H -c external/zlib/uncompr.c -o bazel-out/host/bin/external/zlib/_objs/zlib/uncompr.o"
this code will get fatal error: 'stddef. h' file not found.

Relevant log output
No response

same issue as i met

@mars1248,
Could you please confirm the bazel version you are trying to compile and also whether you are building from source or any other way? Thank you!

I use tensorflow/build:2.13-python3.11 image, with bazel 5.3.0. and build from source code.
I add this inc_dirs.append("/usr/lib/llvm-16/lib/clang/16/include") code to third_party/gpus/rocm_configure.bzl but get this error.
ERROR: /root/.cache/bazel/_bazel_root/645133528a7a8476e9bedcf54eb858b4/external/llvm-project/llvm/BUILD.bazel:184:11: Compiling llvm/lib/Support/Atomic.cpp failed: (Exit 1): clang failed: error executing command /usr/bin/clang -MD -MF bazel-out/k8-opt-exec-50AE0418/bin/external/llvm-project/llvm/_objs/Support/Atomic.d ... (remaining 87 arguments skipped) In file included from external/llvm-project/llvm/lib/Support/Atomic.cpp:13: In file included from external/llvm-project/llvm/include/llvm/Support/Atomic.h:20: In file included from external/llvm-project/llvm/include/llvm/Support/DataTypes.h:19: In file included from external/llvm-project/llvm/include/llvm-c/DataTypes.h:43: /usr/include/x86_64-linux-gnu/sys/types.h:144:10: fatal error: 'stddef.h' file not found #include <stddef.h>          ^~~~~~~~~~ 1 error generated.

Use /usr/lib/llvm-16/bin/clang instead of /usr/bin/clang-16,this may help.
my build env(not using Docker):

Ubuntu 22.04
Clang  16.0.6 from https://apt.llvm.org/
Python 3.10
Bazel  5.3.0
cuDnn  8.6
CUDA   11.8
configure:
No ROCm support
cuda and tensorrt support

Has anyone figured this out? I'm running Fedora 40 and specifying:
Please specify the path to clang executable. [Default is /usr/bin/clang]: **/usr/lib64/llvm16/bin/clang**
Having same sort of issues with not finding system headers, but I notice it's not looking in the llvm directory for the headers.
Does the build need to be containerized?
Sorry if this is a noob question, but I am new to tensorflow.


Unable To Use Latest jQuery

I am using the test_page.html as my cookie bar setup. But cannot switch out https://github.com/bmihelac/django-cookie-consent/blob/master/tests/core/templates/test_page.html#L58 with the latest jQuery:
<script src="https://code.jquery.com/jquery-3.5.1.slim.min.js" integrity="sha384-DfXdz2htPH0lsSSs5nCTpuj/zy4C+OGpamoFVy38MVBnE+IbbVYUew+OrCXaRkfj" crossorigin="anonymous"></script>
The accept and decline links for the cookie bar on line 85 does not work.
It is kind of undesirable to use 2 versions of jQuery 3.3.1 and the newest in my code which is a quick way around this when you should just be able to use the latest. Otherwise the jQuery that this app uses needs to be modernized.
Maybe there is a way to completely avoid jQuery and use JavaScript only?

@9mido thanks for report, this should be resolved with https://github.com/bmihelac/django-cookie-consent/pull/20
please reopen if it is not

@bmihelac Awesome! I will have to test it and check it out thanks!


HP-Tracker Statblocks Rolling Twice

Hi,
I've recently run into the problem of the dice prompts on statblocks rolling twice (without hitting advantage. When I use advantage it rolls with advantage twice (4 dice)).
Let me know if there's anyway I can fix this issue.
thanks,
Jordan

Hmm it might be something to do with click propagation, I will look into it

Was because HP Tracker and dddice extension were both active and so the dice rolls were rendered by both extensions. Disabling one renderer solved the problem.


using C++ api in windows10 to extract feature error

i'm just using mxnet C++ api to extract feature  ,show the codes below
cv::Mat img = _img.getMat();
img.convertTo(img, CV_32F);
img = (img - 127.5) / 128;
static NDArray input(Shape(1, img.channels(), img.rows, img.cols), ctx);	//
	NDArray temp(Shape(1, img.channels(), img.rows, img.cols), Context(kCPU, 0),false);
	NDArray::WaitAll();
	auto ptr = temp.GetData();
	std::vector<cv::Mat> chs(3);
	for (int i = img.channels() - 1; i >= 0; i--)
	{
		chs[img.channels() - i - 1] = cv::Mat(img.size(), CV_32FC1, (float*)ptr + i*img.size().area());
	}
	cv::split(img, chs);
	//input = temp.Copy(ctx);
	input.SyncCopyFromCPU(temp.GetData(), temp.Size());
	NDArray::WaitAll();
	return Extract(input);

when ctx = Context(kGPU, 0);
the code     executor = net.SimpleBind(ctx, args_map, map<string, NDArray>(), map<string, OpReqType>(), aux_map);  occurs  error. but cpu mode all of them are OK.
I don't know how to solve it ,anybody could help me ?thanks a lot

@ghgggg : Could you submit this issue as a issue template? It isn't clear what the exact error is from the issue. For reference https://github.com/apache/incubator-mxnet/blob/master/.github/ISSUE_TEMPLATE.md
@mxnet-label-bot: [C++, Windows, Question]

@vdantu  i has recommit the issue as a new one ,could you help me to solve it ?thansk a lot. i will close this one. the new issue is #12267 .thank you


Plugin breaks key mapping

cmp-buffer breaks this mapping:
:nnoremap <Space> i_<Esc>r|                     " Insert single char in normal mode with space + char

Specifically, pressing "" results in "__" being inserted instead of a single space " ".
I am not sure why this happens, but it looks like the same issue happens if I load any of these:
cmp-nvim-lsp
cmp-path
Happy to help testing ideas if that's useful!

cmp-cmdline also causes problem


Trigger notification is repeating

Hi!
I am trying to set schedule notification to run every day on specific hour. And it runs fine at the first time, but when the next notification takes place it loops, and sending over and over notification to the user. Someone has any ideas why it is happening? I'm also logging it with notifee.onBackgroundEvent and it shows the same notification takes places.
My config:
const trigger = {
    type: TriggerType.TIMESTAMP,
    timestamp: time,
    repeatFrequency: RepeatFrequency.DAILY,
    alarmManager: true,
  };

  await notifee.createTriggerNotification(
    {
      title,
      body,
      android: {
        channelId: channelId,
        category: AndroidCategory.REMINDER,
        importance: AndroidImportance.HIGH,
        visibility: AndroidVisibility.PUBLIC,
        autoCancel: false,
        ongoing: true,
        pressAction: {
          id: 'default',
        },
      },
    },
    trigger,
  );

Someone has any idea why it is working that way?
I've tried to remove autoCancel, when I remove alarmManager the notification runs only once and doesnt repeat.

Hi there!
What versions of everything are you running? What platform do you see this? real device or emulator/simulator?
https://stackoverflow.com/help/how-to-ask

Im running
@notifee/react-native - 3.0.4,
react-native: 0.66.2
And it happens on Pixel 5a emulator from android studio

Just found that it was fixed in 4.0.1, sorry for opening an issue. I will do better research at first next time


Permssion control of observability maps dashboards

Summary
Currently, anyone in the same account can modify an observability map. This may not be desirable, especially in large accounts
Desired Behaviour
Add access control to the observability maps app so that only the creator and (say) admins can change a map.
Possible Solution
Add an 'export as dashboard' feature to export the map as a dashboard and have it governed by the typical access rules for all dashboards
NOTE: # ( Not required. Suggest how to implement the addition or change. )
Additional context

Hey @sekrish1 unfortunately this is something that I cannot control or modify myself because of how NerdStorage permissions work for nerdpacks. I'm going to close this issue, but I will share it internally as I also think it would be great to have this functionality available.


Is this repo a duplicate of reconbot/streaming-iterables?

Hi. Curious, what's the purpose of this current repository? It looks like a duplicate of the main one (and is not a fork at the same time): on NPM package of the library (https://www.npmjs.com/package/streaming-iterables), another repo is mentioned, not this one: https://github.com/reconbot/streaming-iterables

?


[OCL][MI100][MI200] GEMM convolution failures with OCL backend (Staging 95b58f72f)

make check convolution failures. Preconditions:

Mainline build 9148
OpenCL backend
MI200 or MI100 target

Failures: make-check.9148-opencl.float.gfx90a.failures.txt.
As you can see, all failures happen with GemmBwd1x1_stride1 and GemmBwdRest.
The exact technical reason of failures is unknown. However we know that MIOpen leverages MIOpenGemm library for GEMM when backend is OpenCL. But MIOpenGEMM was never tested with MI100 and MI200 because it is deprecated and not supported anymore. So the most likely reason is issue in MIOpenGEMM.
The solution is disabling MIOpenGEMM in convolutions. The side effect is some performance drop. This can be disregarded as MI100 and MI200 are not designed to provide the best possible performance with an OpenCL backend. If performance will become important, then we can consider enabling back GEMM solvers except GemmBwd1x1_stride1 and GemmBwdRest.

Full description provided.

#1321 merged. This specific issue can be now closed. We already have the goal to replace MIOpenGemm with MiopenTensile including OCL BE).


Current nuget package does not work with ASP.Net Core 2.0 project

Looks like the AWSSDK dependencies do not support .Net standard based on the version in nuget. Any estimate on when a latest nuget package will be released with the latest updated .Net Standard support?

@anandja Get latest . . .should be fixed now.

Thanks. It looks good now.


Added Code of Conduct

Solves the issue #455

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Hey @gr8shivam, I am sorry but it looks like #456 beat you to this one!


[SPARK-6910] [SQL] Support for pushing predicates down to metastore for partition pruning

@marmbrus @liancheng per request, I am reopening PR that contains #7216 and #7386.
Can you help me to understand unit test failures?

Merged build triggered.

Merged build started.

Test build #37372 has started for   PR 7421 at commit 69eb136.

Test build #37372 has finished for   PR 7421 at commit 69eb136.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Hmm, jenkins failed for a unknown reason-
[error] running /home/jenkins/workspace/SparkPullRequestBuilder@2/build/sbt -Pyarn -Phadoop-2.3 -Dhadoop.version=2.3.0 -Phive-thriftserver -Phive sql/test mllib/test hive-thriftserver/test hive/test catalyst/test examples/test ; received return code 143
Archiving unit tests logs...
> Send successful.
Attempting to post to Github...
 > Post successful.
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
Finished: FAILURE

ok to test

Merged build triggered.

Merged build started.

Merged build finished. Test FAILed.

Merged build triggered.

Merged build started.

Test build #37421 has started for   PR 7421 at commit 5599cc4.

Test build #37421 has finished for   PR 7421 at commit 5599cc4.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
abstract class StandaloneRecoveryModeFactory(conf: SparkConf, serializer: Serializer)
class LDAModel(JavaModelWrapper):
class LDA(object):
trait ImplicitCastInputTypes extends ExpectsInputTypes
abstract class BinaryOperator extends BinaryExpression with ExpectsInputTypes
case class UnaryMinus(child: Expression) extends UnaryExpression with ExpectsInputTypes
case class UnaryPositive(child: Expression) extends UnaryExpression with ExpectsInputTypes
case class Abs(child: Expression) extends UnaryExpression with ExpectsInputTypes
case class Pmod(left: Expression, right: Expression) extends BinaryArithmetic
case class BitwiseNot(child: Expression) extends UnaryExpression with ExpectsInputTypes
      final class SpecificRow extends $
case class Factorial(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
case class Hex(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
case class Unhex(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
case class Round(child: Expression, scale: Expression)
case class Md5(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
case class Sha1(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
case class Crc32(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
case class Not(child: Expression)
case class And(left: Expression, right: Expression) extends BinaryOperator with Predicate
case class Or(left: Expression, right: Expression) extends BinaryOperator with Predicate
trait StringRegexExpression extends ImplicitCastInputTypes
trait String2StringExpression extends ImplicitCastInputTypes
trait StringComparison extends ImplicitCastInputTypes
case class StringSpace(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
case class StringLength(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
case class Ascii(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
case class Base64(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
case class UnBase64(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
case class Exchange(newPartitioning: Partitioning, child: SparkPlan) extends UnaryNode

Merged build finished. Test PASSed.

Test build #1083 has started for   PR 7421 at commit 5599cc4.

Test build #1085 has started for   PR 7421 at commit 5599cc4.

Test build #1084 has started for   PR 7421 at commit 5599cc4.

Test build #1086 has started for   PR 7421 at commit 5599cc4.

Test build #1087 has started for   PR 7421 at commit 5599cc4.

  Lets see what happens...

Test build #1087 has finished for   PR 7421 at commit 5599cc4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #1083 has finished for   PR 7421 at commit 5599cc4.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
abstract class StandaloneRecoveryModeFactory(conf: SparkConf, serializer: Serializer)
case class Pmod(left: Expression, right: Expression) extends BinaryArithmetic
      final class SpecificRow extends $

Test build #1085 has finished for   PR 7421 at commit 5599cc4.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
abstract class StandaloneRecoveryModeFactory(conf: SparkConf, serializer: Serializer)

Test build #1086 has finished for   PR 7421 at commit 5599cc4.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
abstract class StandaloneRecoveryModeFactory(conf: SparkConf, serializer: Serializer)
case class Pmod(left: Expression, right: Expression) extends BinaryArithmetic
      final class SpecificRow extends $

Test build #1084 has finished for   PR 7421 at commit 5599cc4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

4/5 passed... Do you think this is because of multithreading in unit tests? Otherwise, I have no explanation.

getPartitionsByFilter is really a great improvement, normally in a production hive data warehouse, there are tables with huge amount of partitions. looking forward to see this will be included in next release :)

I am repeatedly running the sql/hive unit tests after synchronizing getPartitionsByFilterMethod.invoke. I'll report back how it goes.

I am pretty confused about why this this failing, but only sometimes.  I don't think it could be locking because getPartitionsByFilter is guarded by the locks in withHiveState, right?
Have you been able to reproduce the failure locally?

Investigated the following 3 build failure samples:

https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/37233/testReport/
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/37232/testReport/
https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1083/testReport/

Firstly, this issue couldn't be steadily reproduced, and only showed up on Jenkins occasionally.  An obvious guess is that it's probably a concurrency bug and only occurs within highly concurrent jobs (the Jenkins server has 32 cores, while our laptops have only 8 or less).
Secondly, all 3 build failures behaved extremely consistently: 18 ParquetDataSourceOffMetastoreSuite test cases involving partitioned Hive metastore Parquet tables failed altogether.  It seems that some internal Hive state got corrupted before this test suite was executed.  However, this PR only updates the read path and doesn't introduce any extra state.  So my guess is that, this PR doesn't introduce but just somehow triggers an existing issue.  The root cause probably lies in some initialization phase, e.g. HiveContext initialization, or testing partitioned table creation in ParquetDataSourceOffMetastoreSuite.beforeAll().
And I got another interesting finding after single step debugging a failed test case.  The following stacktrace snippet appears in all 3 build failures:
Caused by: MetaException(message:Filtering is supported only on partition keys of type string)
      .----
      | at org.apache.hadoop.hive.metastore.parser.ExpressionTree$FilterBuilder.setError(ExpressionTree.java:185)
      | at org.apache.hadoop.hive.metastore.parser.ExpressionTree$LeafNode.getJdoFilterPushdownParam(ExpressionTree.java:452)
      | at org.apache.hadoop.hive.metastore.parser.ExpressionTree$LeafNode.generateJDOFilterOverPartitions(ExpressionTree.java:357)
      | at org.apache.hadoop.hive.metastore.parser.ExpressionTree$LeafNode.generateJDOFilter(ExpressionTree.java:279)
      | at org.apache.hadoop.hive.metastore.parser.ExpressionTree.generateJDOFilterFragment(ExpressionTree.java:590)
      | at org.apache.hadoop.hive.metastore.ObjectStore.makeQueryFilterString(ObjectStore.java:2417)
      | at org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsViaOrmFilter(ObjectStore.java:2029)
      | at org.apache.hadoop.hive.metastore.ObjectStore.access$500(ObjectStore.java:146)
      | at org.apache.hadoop.hive.metastore.ObjectStore$4.getJdoResult(ObjectStore.java:2332)
      | at org.apache.hadoop.hive.metastore.ObjectStore$4.getJdoResult(ObjectStore.java:2317)
      `----
        at org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2214)

The marked code path showed above is actually NEVER executed in normal cases.  To be more specific, the getJdoResult() method in [the anonymous GetListHelper object] 1 is never called in [GetHelper<T>.run()] 2.  Instead, only the getSqlResult() method is called.  And we can see that this behavior is controlled by doUseDirectSql, which is [partially decided] 3 by [ObjectStore.directSql.isCompatibleDatastore] 4.  Since ObjectStore is initialized while initializing HiveContext, ObjectStore.directSql.isCompatibleDatastore is probably the corrupted Hive internal state.
Haven't got any clue how this state gets corrupted yet.  My guess is that there is a race condition during HiveContext initialization.  For example, maybe the underlying Derby database is not fully created while ObjectStore is been initialized.

It seems that Hive prefers to access the underlying metastore database via direct SQL, and uses JDO ORM as a fallback. The existing bug doesn't show up because usually either direct SQL or JDO ORM is capable to do the work. But in case of getPartitionByFilter, the ORM one doesn't support predicates involving integral types by default, and thus leads to build failure.
Before fixing the root cause, I guess we can workaround this issue by setting hive.metastore.integral.jdo.pushdown to true to let the JDO ORM code path be able to handle integral partition columns.

Experimenting the workaround mentioned above in PR #7492.

Closing as it is merged as part of #7492.

@liancheng @piaozhexiu Have you cherry-pick this PR to spark branch-1.5?

@litao-buptsse yes, this patch is committed in branch-1.5. You need to set spark.sql.hive.metastorePartitionPruning to true to enable it, which is false by default.

@piaozhexiu OK, I got it, thank you very much!


CTCP0-5940

https://jira.tools.tax.service.gov.uk/browse/CTCP-5940
Upped min coverage to 90%

rebuild


Feature: Run kor in docker

Using a multistage Dockerfile create a Docker image that can run kor.
The final image should be based on alpine

Can I be assigned to this issue?


isAuthenticated returning false after successful login

Hi,
Its my first time using the auth0-react SDK. I've implemented the login flow using guide, and have configured my app on Auth0 with Social Connection to LinkedIn. First login, the user is redirected to the UniversalLogin when they can login through LinkedIn, but as the user is redirected back to the application, the isAuthenticated value still remains false, and I see the POST require to oauth/token returning 401 unauthorized.  On the Auth0 dashboard it also showed a successful login attempt with the user account.
I've provided the Auth0Provider with the appropriate props & values filled in of course,
    <Auth0Provider
            domain=""
            clientId=""
            redirectUri={window.location.origin}
            audience=""
            onRedirectCallback={onRedirectCallback}>

Am I missing anything?
The request header for the failing oauth/token request includes grant-type: authorization-code, with the code values from successful login request. On the Auth0 documentation, it seems that client-side applications should only be using grant-type: implicit. Could this be the issue? Is there any way to specific the request sent by the SDK to oauth/token as grant-type=implicit? Any help in how to fix this issue is greatly appreciated 🙏🏼

Hi @alicemao23 - thanks for raising your issue
This library uses the Authorization Code Flow with PKCE (https://auth0.com/docs/flows/concepts/auth-code-pkce) so grant_type: "authorization_code" is expected.

I see the POST require to oauth/token returning 401 unauthorized

You should get an error and error_description properties back in the body of the response which you can check. Also, you should also get a "Failed exchange" in your Auth0 tenant logs with some more information https://manage.auth0.com/#/logs
From what I can see your config looks fine. If you have a running version of an app, or can share the code, I might be able to help you debug the issue.

Hi @adamjmcgrath!
Thanks for getting back. So the error is access_denied & error description is unauthorized.  The behaviour that I have been consistently able to replicate is

On initial login, loginWithRedirect() will direct screen to log in, the flow goes through, on successful login, the user is redirected back to the app as expected. But Network call to "/oauth/token" returns 401 unauthorized with the error message and description above. isAuthorized from useAuth0 returns false.

Then user tries to use loginWithRedirect() again, the user is not taken to the login widget, page refreshes, I assume from the redirect, and then the network call to /oauth/token is also made and returns the same error response in the body.

I would have to manually call logout if I want to be able to use the login flow again from 1.

I don't have a running version of the app right now, just started up and is trying to work out the authentication part. I could definitely share the code, I just don't know of a way of me sharing it safely without needing to share sensitive data like my client id. Do you have any suggestions?
Thank you!

Hi @alicemao23
clientId, domain and audience are not considered confidential, if you were to share a running app I would be able to see them just like I would if you were to share the app's sourcecode - that's because a SPA is a public client and there is no way to hide them.

So the error is access_denied & error description is unauthorized.

I'm not sure what your issue might be, do you have any rules configured on your tenant perhaps? Do the other values you are sending in the /oauth/token request look sensible? Are you sending the request more than once?
If you feel able to share your code, or a sample app where I can reproduce the issue, I might be able to help. But from the information you've given me I wont be able to figure much out

You could also try running the basic sample app in this repo: https://github.com/auth0/auth0-react/blob/master/CONTRIBUTING.md#local-development replacing our clientId and domain for yours and seeing if that works

Hi @adamjmcgrath ,

You could also try running the basic sample app in this repo: https://github.com/auth0/auth0-react/blob/master/CONTRIBUTING.md#local-development replacing our clientId and domain for yours and seeing if that works

I tried that. My management logs show Success Login and Success Exchange.

Issue moved here #51

Hi @alicemao23 - let me know if you've had any luck fixing your issue.
Another thing to check would be that you have "Allowed Web Origins" set correctly in your dashboard.
And that your Auth0 Application in your dashboard is set to "Single Page Application"

Let me know if either of those suggestions helps

I am also experiencing this same issue. I cloned the sample app (https://github.com/auth0-samples/auth0-react-samples) and added a config file with my Auth0 application's domain, clientId, and audience (the audience may be incorrect - I'm not sure what to put for that because it's not documented anywhere).
When I start my react app and visit localhost:3000, it redirects me to the Auth0 login page as expected and I successfully log in - the logs even show a successful login - but then there is the "Failed Exchange" error message as mentioned above with the 401 to /oauth/token. I also see the "Oops... Unauthorized" message in the sample app.
What is going on? How can I fix this? Is my application configured incorrectly in Auth0?

I'm having this same issue over the last two days.  Same behavior as mentioned above:  the user is directed to the lock login page, redirected back to the app, user authentication is never returned.

{
  "date": "2020-07-05T03:13:29.663Z",
  "type": "s",
  "connection_id": "",
  "client_id": "7o21rTYAsoK0xKec9UxK2olZppzy7j56",
  "client_name": "", <removed>
  "ip": "<IP_ADDRESS>",
  "user_agent": "Chrome 83.0.4103 / Windows 10.0.0",
  "details": {
    "prompts": [],
    "completedAt":<PHONE_NUMBER>662,
    "elapsedTime": null,
    "session_id": "X6LOaGTzDP35GD7I9HjGm57iH5wqhLtz"
  },
  "hostname": "", <removed>
  "user_id": "auth0|5efce1eda15b7b001361c588",
  "user_name": "", <removed>
  "auth0_client": {
    "name": "auth0-react",
    "version": "1.0.0"
  },
  "log_id": "90020200705031335800043426049758653529361151510378971138",
  "_id": "90020200705031335800043426049758653529361151510378971138",
  "isMobile": false,
  "description": "Successful login"
}

  "prompts": [],
  "completedAt":<PHONE_NUMBER>662,
  "elapsedTime": null,
  "session_id": "X6LOaGTzDP35GD7I9HjGm57iH5wqhLtz"
}

This has been a very disappointing way of spending the holiday weekend. :(

@avala-HFI we hired a consultant familiar with Auth0 to fix our application in the Auth0 dashboard - he hasn't gotten to me with the changes that he made to get it working, but as soon as I know I will post it here and submit a request to Auth0 to update their documentation for this SDK since their steps do not work OOB

Hi @avala-HFI
Could you check if you have  "Allowed Web Origins" set to the origin of your webapp (eg http://localhost:3000)?
And that your Auth0 Application in your dashboard is set to "Single Page Application":

Let me know if either of those suggestions helps

as soon as I know I will post it here and submit a request to Auth0 to update their documentation for this SDK

Thanks @mwarnerdotme! please let us know if there's anything we can do to improve our docs

@mwarnerdotme you are my personal hero!  I'm looking forward to hearing what your consultant finds!
@adamjmcgrath, I have two applications that are both set to Single Page Application and the Allowed Web Origins are set to http://localhost:3000.

Thanks for confirming @avala-HFI
Can you share a link to one of these running apps or the code so I can run one myself and debug the issue?

@adamjmcgrath I'll have something out later this afternoon.

@adamjmcgrath I've created a fairly basic React-Express-Node app using the same Auth0 quickstart guide and Auth0 application settings as our private projects.  In all cases using the Quickstart guide, I'm experiencing the exact same issue where isAuthenticated returning false.  You can find the example project here.  Please let me know if you need any additional information.
React Developer Tools

Console Error stack

@adamjmcgrath  Hi everyone, actually I have found a workaround on this issue even though I'm not exactly sure why this was the way it was. I also cannot re-replicate it, so maybe this will help (fingers crossed).  My tenant was configured to be SinglePageApplication with Token End Point Authentication to be None
I actually changed it to be a few other variations of values including "Regular Web Application": POST & "Regular Web Application": None, saved my changes, and tried using the SDK to log in a couple more times.
Obviously the configurations above weren't valid, so I switched back toSingle Page Application: None, and the SDK started returning the expected authentication values for me. You could try re-updating your tenant configuration, not sure if this will help anyone else experiencing this issue.

@alicemao23 @adamjmcgrath That worked!
I first toggled our application to "Regular Web Application": None and that worked.  I then toggled to "Regular Web Application": POST and Basic, neither of which worked.  I then toggled it back to Single Page Application: None and it worked.

Ah, thanks @alicemao23 @avala-HFI! I'm not sure why both your tenants were incorrectly configured even though they were showing the correct configuration Single Page Application: None.
I've just tested this myself by creating a new SPA Application in the dashboard and it seems to be working now, but I'll keep an eye out to see if anyone else runs into this and raise it internally.
Closing this issue, feel free to reopen if you run into this again

I was facing the same issue, now fixed with the workaround @alicemao23 provided. Thanks.

I have the same issue but @alicemao23's fix didn't work for me. Any updates on this?

I faced the same issue and @alicemao23's fix worked for me as well. 👍

Hi @alicemao23 - let me know if you've had any luck fixing your issue.
Another thing to check would be that you have "Allowed Web Origins" set correctly in your dashboard.
And that your Auth0 Application in your dashboard is set to "Single Page Application"

Let me know if either of those suggestions helps

after trying lots of alternatives this one has worked for me. from application type drop down menu you have to select single page application and the problem you are facing will probably disppear


[Mobile] Android client should be updated to the latest version

I have managed to get the Android client to work by editing the site config to include
<add key="MS_SkipVersionCheck" value="true" />
in the appSettings.  It's a bit of a faff but it is working correctly on the Nexus 5X emulator.

@alecouteurbisson A better fix would be to use the the Azure Mobile Apps Android library. In gradle: compile 'com.microsoft.azure:azure-mobile-android:3.1.0'.


chore(main): release 1.7.0

:robot: I have created a release beep boop
1.7.0 (2024-06-27)
Features

cjs to esm + imrpovement ... (10884a9)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/Basalt-Lab/basalt-logger/releases/tag/v1.7.0 :sunflower:


hdmi-output: not working in RPi4

HDMI audio is not working on Pi 4.

Bump. I am experiencing the same issue

Fixed by adding RESIN_HOST_CONFIG_hdmi_mode set to 2 in the configuration settings.

@tmigone works the comment from zchbndcc9 for you?

All fixes are included in the troubleshooting: https://landr-balenalabs-repo-balena-sound.netlify.app/docs/troubleshooting/ Closing.

it did not work for me with a clean install

it did not work for me with a clean install

@AlexdrFontes could you give any more information? I've just done a fresh deploy of the latest version of the software onto a Pi 4 here and it worked fine without any changes.

@AlexdrFontes could you give any more information? I've just done a fresh deploy of the latest version of the software onto a Pi 4 here and it worked fine without any changes.

Hi Chris.
I’ve manage to use balenasound and reproduce music on several speakers, but the jack(3.5mm) signal is really really weak and to play on my amps i have to crank the volume all the way up.
I bought a regular HDMI2AV transmitter, self powered and tried with raspbian to play sound on the transmitter. Everything went ok, the sound volume is much better that with the jack.
After this I flashed balenaSound and tried to connect the transmitter to the amp, but unfortunately with no luck.
I spent several hours editing config variables that other people on this forum say it would solve the issue but I had no success.
I booted the pi with the cable on HDMI 0 slot but unfortunately the sound comes ALWAYS from the jack.
Have you been able to force HDMI to pass the sound onto one of these receivers?
My Setup would be just BalenaSound flashed directly onto the sd card and connected to the amp with HDMI - > HDMI2AV transmitter - > LEFT and RIGHT sound cable to AMP.

Hi Chris.
I’ve manage to use balenasound and reproduce music on several speakers, but the jack(3.5mm) signal is really really weak and to play on my amps i have to crank the volume all the way up.
I bought a regular HDMI2AV transmitter, self powered and tried with raspbian to play sound on the transmitter. Everything went ok, the sound volume is much better that with the jack.
After this I flashed balenaSound and tried to connect the transmitter to the amp, but unfortunately with no luck.
I spent several hours editing config variables that other people on this forum say it would solve the issue but I had no success.
I booted the pi with the cable on HDMI 0 slot but unfortunately the sound comes ALWAYS from the jack.
Have you been able to force HDMI to pass the sound onto one of these receivers?
My Setup would be just BalenaSound flashed directly onto the sd card and connected to the amp with HDMI - > HDMI2AV transmitter - > LEFT and RIGHT sound cable to AMP.

@AlexdrFontes ah, I'm replying to you on the forum here https://forums.balena.io/t/hdmi-audio-output-only-on-balenasound/232278

@AlexdrFontes ah, I'm replying to you on the forum here https://forums.balena.io/t/hdmi-audio-output-only-on-balenasound/232278

Thank you

Thank you

This seems to be reoccuring: https://github.com/balena-os/balena-raspberrypi/issues/586

This seems to be reoccuring: https://github.com/balena-os/balena-raspberrypi/issues/586

Hi @AlexdrFontes @chrisys, this is consequence of changes made by the Pi foundation on the linux kernel. It's still working but they changed the behaviour of the snd_bcm2835  audio hardware. Let me elaborate.
Historically the snd_bcm2835 hardware was initialised in a way that the OS would see one sound card with multiple devices (3.5mm jack, HDMI). This meant you could output audio to both audio jack and HDMI by default with no configuration required.
As of kernel version 5.4, the Pi foundation changed this so the OS now lists two sound cards, one for the audio jack and another one for HDMI (HDMI only showing up if you got a cable plugged in on boot). The default is the audio jack, so you won't get HDMI audio unless you select it as the output interface of your choice. I believe this was made so users get to select which one they want with a nice UI dropdown on Raspberry Pi OS (see below).
Some references:

Pi Foundation's changes to the kernel
balenaOS 2.54.3+rev1 incorporating those changes only for the Pi4. This explains why it "doesn't work" on 2.60 but it does in 2.50.
Pi official docs on how to change output audio device
Related GitHub issues in the audio block repo with further details: #39 and #4

So, how do we select which audio interface we want to use now?
Easy, use the AUDIO_OUTPUT environment variable and set it to RPI_HDMI0 for example for HDMI0 on the Pi4. This is described in detail in the balenaSound docs and the audio block readme.
I hope this helps @AlexdrFontes, I'll reply to you in the forums too with this information so that others can find it there too. Let's continue the conversation over there if needed. Thanks!


Create the books page

Implements #49.
InnerSource Books

Getting Started With InnerSource
Understanding the InnerSource Checklist
Managing InnerSource Projects
Adopting InnerSource - Principles and Case Studies
InnerSource Patterns

TODO

[ ] add PDFs of books to the repo (e.g. at /static/documents/books)
[ ] add images of book covers to the repo (high-resolution; same size, so that they look nice side-by-side on the overview page
[ ] confirm what happened to the "Getting Started With InnerSource" book
[ ] get confirmation of the intro text for all books. specifically: Which books do we list here, and why?
[ ] show books on the overview page in the same order as listed above (chronological order)
[ ] create a detail view for the book pages that is somewhat consistent across all books
[ ] decide whether to link to books directly from the overview page (i.e. to the PDF)
[ ] (bonus) improve short description of each book on the overview page, so that readers can more quickly decide what they need which book for
[ ] (bonus) get a cover image for "Managing InnerSource Projects"
[ ] (bonus) move "Managing InnerSource Projects" into the https://innersourcecommons.gitbook.io/ org

Further info that we could work in:

They were written in the following order:

Getting Started With InnerSource
Understanding the InnerSource Checklist
Managing InnerSource Projects
Adopting InnerSource - Principles and Case Studies

Stay tuned for Adopting InnerSource Vol. 2 (more case studies)

Adopting InnerSource - Principles and Case Studies - is good for why it's important.

@voborgus can you help me with something related to hugo?
I could not figure out how to create a new list layout, that would only be used for the books?

@spier I saw you already sorted the problem out. If no, here are my observations on the point:
According to this doc, Hugo takes only the first root section layout and not looks for children. It means, in the case of Books he takes only the parent Learn layout. You can create a new Books layout and specify a special tag on the top of the page like here (uses this layout) or make changes in the Learn layout inserting if clauses only for changes for Books pages.

I have created a Doc here with shortened descriptions of the books for the overview page, hope this helps. Let me know if I can do anything else to help

@Katiegilroyy when you say overview page, you mean this one, right? https://innersourcecommons.net/learn/books/
I am in the process of updating that page. Currently it looks like this (see image below).
I think we might have to shorten the description text for the books that you are proposing, because there isn't that much space on the website.
Also one book is missing from your list: "Adopting InnerSource - Principles and Case Studies"

I will add that book in now, Them short descriptions are great, but Is there any way have a more detailed description of the books on a separate page,  similar to what is on the current website? What would you think of having something similar to that?

Yeah, in my current approach I am keeping one dedicated page per book. So longer text descriptions can go there.
My goal is to get a first version of this done and deployed in the coming days.
Then you can review it and improve the text further.

@spier I saw you already sorted the problem out. If no, here are my observations on the point:
According to this doc, Hugo takes only the first root section layout and not looks for children. It means, in the case of Books he takes only the parent Learn layout. You can create a new Books layout and specify a special tag on the top of the page like here (uses this layout) or make changes in the Learn layout inserting if clauses only for changes for Books pages.

Thank you for these tips @voborgus. I tried it according to this and got it to work. Thank you!

I think this version is now good enough.
While there are some things that could still be improved (see unfinished tasks in the checklist), I would opt to ignore them for now and just get this branch merged as is.
Overview Page

Details Page


Console messages appearing even when debug is set to false

Hi,
I'm handling this situation via the error object in a callback.  I really don't want the console messages being redundantly generated but I don't know how to turn them off.  Debug mode is set to false, I'm not sure what else I can do:

In the above I was calling validate manually per the meteor guide.  I've switched to using collection2 with automatic validation and the above console messages have gone away (but I still get a trappable error message in my code.)  Closing this since it's no longer an issue for me.


Pods should be updated flaky

Pods should be updated (Failed 8 times in the last 100 runs. Stability: 92 %): Test or product?
Actually, this one's disturbing, since it looks new?

This appears to be referring to test/e2e/pods.go

Some possibly-related discussion on #4528

This seems to have been fixed by #4587, as there are no recent failures.  Re-open this issue if the problem recurs.

This is actually still flaking.

I looked at all of our jenkins projects and this test hasn't flaked any time recently (last 30 runs) on any of them. Marking as fixed.