SPDM 1.0 - What is correct order for the responder to send the GET_CERTIFICATE response?

I've the following in my certificate chain (Root_Cert, intermediate_cert, device_cert, alias0_cert, alias1_cert), working on implemting the GET_CERTIFICATE response, so when I receive the GET_CERTIFICATE request, I will be sending the response in the below order..
GET_CERTIFICATE_RESPONSE_HEADER
Certificate_chain_format
root_cert
intermediate_cert
device_Cert
alias0_Cert
alias1_cert
(Please note- alias1_Cert ismy leaf cert)
Am I sending the certificate in the correct order? If not, please correct. Thanks in Advance!!

I’m confused, how can both very be at least significant bytes

Edited it. :)

Thank you

@steven-bellock : Sorry for the late reply, according to your reply "root_cert is at the least significant bytes and alias1_cert is at the most significant bytes." then should I  be sending the response in the below order, right ??
GET_CERTIFICATE_RESPONSE_HEADER
Certificate_chain_format
alias1_cert
alias0_Cert
device_Cert
intermediate_cert
root_cert
"OR this order...
GET_CERTIFICATE_RESPONSE_HEADER
Certificate_chain_format
root_cert
intermediate_cert
device_Cert
alias0_Cert
alias1_cert

GET_CERTIFICATE_RESPONSE_HEADER
Certificate_chain_format
root_cert
intermediate_cert
device_Cert
alias0_Cert
alias1_cert

Thanks

Shouldn't it be the reverse order ? with the leaf appearing first ( when traversing from top to bottom )-
GET_CERTIFICATE_RESPONSE_HEADER
Certificate_chain_format
alias1_cert
alias0_Cert
device_Cert
intermediate_cert
root_cert
This is how the other certs exist with root at the bottom of the file. Example - https://www.digicert.com/kb/ssl-support/pem-ssl-creation.htm

Please read SPDM 1.2, 10.6.1 Certificates and certificate chains:
In the context of this specification, a complete certificate chain is one where: (i) the first certificate is either signed by a Root Certificate (a certificate that specifies a trust anchor) or is a Root Certificate itself, (ii) each subsequent certificate is signed by the preceding certificate, and (iii) the final certificate contains the public key used to authenticate the SPDM endpoint. The final certificate is called the leaf certificate.

Please read SPDM 1.2, 10.6.1 Certificates and certificate chains:
In the context of this specification, a complete certificate chain is one where: (i) the first certificate is either signed by a Root Certificate (a certificate that specifies a trust anchor) or is a Root Certificate itself, (ii) each subsequent certificate is signed by the preceding certificate, and (iii) the final certificate contains the public key used to authenticate the SPDM endpoint. The final certificate is called the leaf certificate.

Totally agree on the order of signing and the cert chain creation . My comment was indicative of the non standard way in which the cert file gets generated here when compared to other standard cert files which have cert chain.


[Hydrogen reference]: Tailwind benefits

This PR:

Adds some context about the benefits of Tailwind to the CSS support topic
Relates to https://github.com/Shopify/shopify-dev/pull/14657 and https://github.com/Shopify/shopify-dev/issues/10171

I like these changes. Thanks! 👍


Enhancement: Improvements to fetch quick panel

This PR addresses a couple annoyances I've noticed with GsFetchCommand:

The quick panel prompts you to select a remote even when the repository only has a single remote, which feels redundant.
The quick panel can remember the last choice you selected if it's an individual remote, but can't remember if your last choice was All remotes.

Changes:

If there is only one remote, fetch from that immediately without requiring a second user interaction. To support this without impacting other remote commands (e.g. GsRemoteRemoveCommand) I added an allow_direct keyword argument to show_remote_panel.
If there are multiple remotes, make the preselected choice "All remotes." if GitCommand.last_remote_used hasn't been set.
Allow the quick panel to remember All remotes. as the previous selection.

Closes #601

The maintainers of this repo require that all pull request submitters agree and adhere to the following:

[x] I have read and agree to the contribution guidelines.
(required)
[x] All related documentation has been updated to reflect the changes made. (required)
[x] My commit messages are cleaned up and ready to merge. (required)

The maintainers of this repository require you to select the semantic version type that
the changes in this pull request represent.  Please select one of the following:

[ ]  major
[x]  minor
[ ]  patch
[ ]  documentation only

For consistency it might make sense to add allow_direct=True to the following commands:

GsPushToBranchNameCommand
GsBranchesPushAllCommand
GsOpenFileOnRemoteCommand
GsOpenGithubRepoCommand

Fetch was the main one for me as I'm fetching a lot more than I use the other commands, and on the repos I work on with multiple remotes, my muscle memory for git: fetch followed by Return means I frequently need to run the command a second time to change my selection to All remotes. 😁

Closing and reopening to re-trigger Travis

Thanks for the PR @ddevlin ! Not sure why travis.ci was failing back there, but it looks like it's may pass now. By the way, you can also retrigger build manually in travis, e.g:

I tried restarting your PR build earlier this morning, with the same failure result

Thanks @asfaltboy, unfortunately I don't have that option, I think because I don't have write access to the repo.

Earlier failures seemed to be sh travis.sh run_tests --coverage timing out while waiting for Sublime Text.

should we need to select semver when we merge to dev?

should we need to select semver when we merge to dev?
Hi @stoivo, is that a question for me? I marked it as minor as it’s not a breaking change and not a bug fix. Not sure what the current policy is with the dev branch, it’s been over a year since my last contribution to GitSavvy.

it can be a bit counter intuitive, but i believe maintainerd will tag a version if we merge a PR with a semver selected

Ah, so I should leave the semver options in the description unselected?

We will mark it documentation onlyto avoid maintainerd tagging a new release.
Sorry @ddevlin, it was a question to @asfaltboy and @randy3k.
Thanks @ddevlin  for your contribution!

The CI tests may fail under some rare occasions when Package Control server doesn't respond. I also have no control of it.


Bring mypy out of pre-commit to type check against imports

Testing on CI; I can't figure out how to add a section to tox.ini. I added (locally, didn't commit)
[testenv:typing]
extras = test
commands=
    python -m mypy --config-file setup.cfg -p rio_tiler
deps=
    numpy

but it didn't seem to get run.

Interestingly.. there are no numpy type errors with numpy 1.19; a few on 1.20; and many on 1.21. Tox is installing numpy 1.19.5, but I can't figure out why. There's no restriction on numpy, so I'd expect it to install 1.21 (and that's what it does for me locally).

I tried to pin to numpy 1.21 in tests only... and tox can't understand that apparently 🤷♂ (though pip can!).

ERROR: Could not find a version that satisfies the requirement numpy<1.22,>=1.21
ERROR: No matching distribution found for numpy<1.22,>=1.21

zero clue why pip can't find any numpy version past 1.19.5

zero clue why pip can't find any numpy version past 1.19.5

This is because 1.19.5 was the last one to support Python 3.6.
🤦♂

done in #461


Slave discovery

This PR is just a suggestion for some changes to the mesos_exporter, namely:

Only need to point the exporter to a Mesos master, elected master and slaves will be discovered automatically.
The exporter scrapes all slave in a cluster periodical (every scrape interval).
Number of active slaves are updated every 10 minutes.
Replaced the config with cmdline flags, which means you don't have the option to have per-slave scrape intervals or select which slaves to collect metrics from :disappointed:
Added an Exporter which implements the Collector interface.

this I believe also fixes #2 , but also means if the exporter fails to fetch a slaves metrics it will disappear until it can be reached again.
Do these changes sound reasonable @juliusv @antonlindstrom ?
and thanks for writing this @antonlindstrom !

@tommyulfsparre Thanks for sending this in! We don't have Mesos experience ourselves, so I'm hoping for @antonlindstrom to give his opinion here on the general concept.
Happy to help review the code itself later on, but can't really help with Mesos-specific things.

From an architectural standpoint I wonder if this would be better done as a mesos_exporter per mesos slave, and then using some service discovery mechanism to find them all on the prometheus server.

@brian-brazil just curious, how is that better than outsourcing the collection/discovery to the mesos_exporter? What is the benefits?
The prometheus server might already have a bunch of targets to scrape and the only service discovery you can do ATM is via SRV records, right?

@tommyulfsparre I think this looks really nice! I like that you've limited the concurrent requests.
One thing that concerns me is that the performance of the exporter once you have a huge number of slaves and tasks. What do you think about adding a way to connect to a single slave as well? That way, it's possible to run an exporter on every node as @brian-brazil suggests.
I also think the Dockerfile has to change to use the flags instead of the config as well.
Otherwise, I think this is super! Well done, @tommyulfsparre!

@antonlindstrom thanks!
That only shifts the burden to the prometheus server instead, you only amortize the overhead of fetching and deserialising the statistics.json I guess.
Anyway small change to support both modes, so I added it.

@tommyulfsparre Spreads the load better, and is easier to manage as you remove a SPOF.

@brian-brazil fair enough, but I would assume that the mesos_exporter would be run as a task on Mesos.
@juliusv @antonlindstrom I think this is ready to be reviewed now :)

@tommyulfsparre LGTM! Nice work! :bow:

@juliusv thanks for the review! Fixed and pushed

This is awesome! LGTM.
@tommyulfsparre: Does this mean Battlefield is now powered by Prometheus? :)

@discordianfish this is something I'm toying with on my spare time
@juliusv address -> url

@tommyulfsparre Aww, too bad. Hope you'll toy with it at work sometime soon :)

:+1: @tommyulfsparre Woot, thanks for bearing with me all this time! Merging!


Add Instructions to Operate on Cloud Servers

I've installed the node on Digitalocean which works fine. The only issue is, when I exit SSH terminal, the node stops running as well. Is there a way to keep it alive?

If you deploy the node in a Docker container, it will keep running as long as the container and the VM continue to run. You can find the instructions for running the node in Docker here: https://docs.chain.link/docs/running-a-chainlink-node/
Alternatively, you could start the node using a service, which can automatically re-start if you reboot the VM. This guide might help with that, but I haven't tested it myself. https://betterprogramming.pub/unleashing-your-daemons-creating-services-on-ubuntu-731cd933e02e
If you are just doing testing and development, you can use screen to keep your process running or you could start the node using a detached background process. I wouldn't recommend running production services this way. The following guides might help with that.
https://linuxhint.com/screen_command_ubuntu/
https://www.tecmint.com/run-linux-command-process-in-background-detach-process/

Closing this issue as it seems to be resolved.
Please direct future questions like these to Stack Overflow: https://stackoverflow.com/questions/tagged/chainlink


Cannot create BertQA model with  tflite-model-maker==0.4.2 (or 4.3)

Issue type
Bug
Have you reproduced the bug with TensorFlow Nightly?
No
Source
source
TensorFlow version
2.8.4
Custom code
Yes
OS platform and distribution
Fedora 40
Mobile device
No response
Python version
3.8.19
Bazel version
No response
GCC/compiler version
No response
CUDA/cuDNN version
CPU
GPU model and memory
No response
Current behavior?
I'm trying to train a BERTQA model (Squad v2 format) with tflite-model-maker in order to use it on Android
Here is the code:
from tflite_model_maker import model_spec
from tflite_model_maker import question_answer
from tflite_model_maker.question_answer import DataLoader

import tensorflow as tf
assert tf.__version__.startswith('2')
tf.get_logger().setLevel('ERROR')

spec = model_spec.get('mobilebert_qa_squad')

train_data = DataLoader.from_squad(filename='final.json',model_spec=spec,is_training=True,version_2_with_negative=True)
test_data = DataLoader.from_squad(filename='final_val.json',model_spec=spec,is_training=False,version_2_with_negative=True)

model = question_answer.create(train_data, model_spec=spec, epochs=2)

print(model.summary())

res = model.evaluate(test_data) # <==== RES IS NONE

print('Loss:', res[0])
print('Accuracy:', res[1])

model.export(export_dir='mobilebert',
    tflite_filename='ingredients.tflite',
    label_filename='labels.txt',
    vocab_filename='vocab.txt',
    saved_model_filename='saved_model',
    tfjs_folder_name='tfjs',
    export_format=None)

final_val.json
final.json
Standalone code to reproduce the issue
pip install tflite-model-maker==0.4.2
pip install tensorflow-addons==0.17.1
pip install tensorflow-cpu==2.8.4

Here is the output of the previous script:
(.venv)  main ±  python train.py
WARNING:absl:Could not find answer: 'd'ail,' vs. 'd' ail,'
2024-07-11 09:07:50.409590: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
Epoch 1/2
80/80 [==============================] - 528s 6s/step - loss: 0.6571 - start_positions_loss: 0.5835 - end_positions_loss: 0.7307
Epoch 2/2
80/80 [==============================] - 505s 6s/step - loss: 0.1952 - start_positions_loss: 0.1642 - end_positions_loss: 0.2262
Model: "model"
__________________________________________________________________________________________________
 Layer (type)                   Output Shape         Param #     Connected to
==================================================================================================
 input_word_ids (InputLayer)    [(None, 384)]        0           []

 input_mask (InputLayer)        [(None, 384)]        0           []

 input_type_ids (InputLayer)    [(None, 384)]        0           []

 hub_keras_layer_v1v2 (HubKeras  {'end_logits': (Non  24582914   ['input_word_ids[0][0]',
 LayerV1V2)                     e, 384),                          'input_mask[0][0]',
                                 'start_logits': (N               'input_type_ids[0][0]']
                                one, 384)}

 start_positions (Lambda)       (None, 384)          0           ['hub_keras_layer_v1v2[0][1]']

 end_positions (Lambda)         (None, 384)          0           ['hub_keras_layer_v1v2[0][0]']

==================================================================================================
Total params: 24,582,914
Trainable params: 24,582,914
Non-trainable params: 0
__________________________________________________________________________________________________
None
Traceback (most recent call last):
  File "train.py", line 25, in <module>
    print('Loss:', res[0])
KeyError: 0

Relevant log output
No response

@AmarOk1412 Creating a BERT QA model with a TensorFlow Lite Model Maker can sometimes be tricky due to compatibility issues. Could you please upgrade to the latest TF version and let us know the outcome?
Thank you!

I can't because I can't install it with pip (tried python 3.8, 3.9, 3.10, 3.11 and 3.12)
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
scann 1.2.6 requires tensorflow~=2.8.0, but you have tensorflow 2.11.0 which is incompatible.

# Then running:
  warnings.warn(
Traceback (most recent call last):
  File "train.py", line 1, in <module>
    from tflite_model_maker import model_spec
  File "/home/amarok/Projects/PlanEat/models/ingredients/.venv/lib64/python3.8/site-packages/tflite_model_maker/__init__.py", line 51, in <module>
    from tflite_model_maker import searcher
  File "/home/amarok/Projects/PlanEat/models/ingredients/.venv/lib64/python3.8/site-packages/tflite_model_maker/searcher/__init__.py", line 25, in <module>
    from tensorflow_examples.lite.model_maker.core.task.searcher import ExportFormat
  File "/home/amarok/Projects/PlanEat/models/ingredients/.venv/lib64/python3.8/site-packages/tensorflow_examples/lite/model_maker/core/task/searcher.py", line 30, in <module>
    from tensorflow_examples.lite.model_maker.core.utils import ondevice_scann_builder
  File "/home/amarok/Projects/PlanEat/models/ingredients/.venv/lib64/python3.8/site-packages/tensorflow_examples/lite/model_maker/core/utils/ondevice_scann_builder.py", line 17, in <module>
    from scann.proto import scann_pb2
  File "/home/amarok/Projects/PlanEat/models/ingredients/.venv/lib64/python3.8/site-packages/scann/__init__.py", line 2, in <module>
    from scann.scann_ops.py import scann_ops
  File "/home/amarok/Projects/PlanEat/models/ingredients/.venv/lib64/python3.8/site-packages/scann/scann_ops/py/scann_ops.py", line 23, in <module>
    _scann_ops_so = tf.load_op_library(
  File "/home/amarok/Projects/PlanEat/models/ingredients/.venv/lib64/python3.8/site-packages/tensorflow/python/framework/load_library.py", line 54, in load_op_library
    lib_handle = py_tf.TF_LoadLibrary(library_filename)
tensorflow.python.framework.errors_impl.NotFoundError: /home/amarok/Projects/PlanEat/models/ingredients/.venv/lib64/python3.8/site-packages/scann/scann_ops/cc/_scann_ops.so: undefined symbol: _ZN4absl12lts_2021032416numbers_internal9kHexTableE

But here is a version with 2.13 (Scann 1.2.6 doesn't like Python 3.10, so I used 3.8 and 2.13 is the max)
``
PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python  python train.py
2024-07-12 10:29:57.563757: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable TF_ENABLE_ONEDNN_OPTS=0.
2024-07-12 10:29:57.565045: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2024-07-12 10:29:57.590564: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2024-07-12 10:29:57.590860: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-07-12 10:29:58.233409: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
/home/amarok/Projects/PlanEat/.venv3.9/lib64/python3.9/site-packages/tensorflow_addons/utils/tfa_eol_msg.py:23: UserWarning:
TensorFlow Addons (TFA) has ended development and introduction of new features.
TFA has entered a minimal maintenance and release mode until a planned end of life in May 2024.
Please modify downstream libraries to take dependencies from other repositories in our TensorFlow community (e.g. Keras, Keras-CV, and Keras-NLP).
For more information see: https://github.com/tensorflow/addons/issues/2807
warnings.warn(
Epoch 1/2
2024-07-12 10:38:04.953281: W tensorflow/core/framework/op_kernel.cc:1805] OP_REQUIRES failed at cast_op.cc:121 : UNIMPLEMENTED: Cast string to float is not supported
2024-07-12 10:38:05.100766: W tensorflow/core/framework/op_kernel.cc:1805] OP_REQUIRES failed at cast_op.cc:121 : UNIMPLEMENTED: Cast string to float is not supported
Traceback (most recent call last):
File "/home/amarok/Projects/PlanEat/models/ingredients/train.py", line 19, in
model = question_answer.create(train_data, model_spec=spec, epochs=2, steps_per_epoch=50)
File "/home/amarok/Projects/PlanEat/.venv3.9/lib64/python3.9/site-packages/tensorflow_examples/lite/model_maker/core/task/question_answer.py", line 228, in create
model.train(train_data, epochs, batch_size, steps_per_epoch)
File "/home/amarok/Projects/PlanEat/.venv3.9/lib64/python3.9/site-packages/tensorflow_examples/lite/model_maker/core/task/question_answer.py", line 79, in train
self.model = self.model_spec.train(
File "/home/amarok/Projects/PlanEat/.venv3.9/lib64/python3.9/site-packages/tensorflow_examples/lite/model_maker/core/task/model_spec/text_spec.py", line 944, in train
bert_model.fit(x=train_ds, epochs=epochs, **kwargs)
File "/home/amarok/Projects/PlanEat/.venv3.9/lib64/python3.9/site-packages/keras/src/utils/traceback_utils.py", line 70, in error_handler
raise e.with_traceback(filtered_tb) from None
File "/home/amarok/Projects/PlanEat/.venv3.9/lib64/python3.9/site-packages/tensorflow/python/eager/execute.py", line 53, in quick_execute
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.UnimplementedError: Graph execution error:
Detected at node 'Adam/Cast' defined at (most recent call last):
File "/home/amarok/Projects/PlanEat/models/ingredients/train.py", line 19, in
model = question_answer.create(train_data, model_spec=spec, epochs=2, steps_per_epoch=50)
File "/home/amarok/Projects/PlanEat/.venv3.9/lib64/python3.9/site-packages/tensorflow_examples/lite/model_maker/core/task/question_answer.py", line 228, in create
model.train(train_data, epochs, batch_size, steps_per_epoch)
File "/home/amarok/Projects/PlanEat/.venv3.9/lib64/python3.9/site-packages/tensorflow_examples/lite/model_maker/core/task/question_answer.py", line 79, in train
self.model = self.model_spec.train(
File "/home/amarok/Projects/PlanEat/.venv3.9/lib64/python3.9/site-packages/tensorflow_examples/lite/model_maker/core/task/model_spec/text_spec.py", line 944, in train
bert_model.fit(x=train_ds, epochs=epochs, **kwargs)
File "/home/amarok/Projects/PlanEat/.venv3.9/lib64/python3.9/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
return fn(*args, **kwargs)
File "/home/amarok/Projects/PlanEat/.venv3.9/lib64/python3.9/site-packages/keras/src/engine/training.py", line 1742, in fit
tmp_logs = self.train_function(iterator)
File "/home/amarok/Projects/PlanEat/.venv3.9/lib64/python3.9/site-packages/keras/src/engine/training.py", line 1338, in train_function
return step_function(self, iterator)
File "/home/amarok/Projects/PlanEat/.venv3.9/lib64/python3.9/site-packages/keras/src/engine/training.py", line 1322, in step_function
outputs = model.distribute_strategy.run(run_step, args=(data,))
File "/home/amarok/Projects/PlanEat/.venv3.9/lib64/python3.9/site-packages/keras/src/engine/training.py", line 1303, in run_step
outputs = model.train_step(data)
File "/home/amarok/Projects/PlanEat/.venv3.9/lib64/python3.9/site-packages/keras/src/engine/training.py", line 1084, in train_step
self.optimizer.minimize(loss, self.trainable_variables, tape=tape)
File "/home/amarok/Projects/PlanEat/.venv3.9/lib64/python3.9/site-packages/keras/src/optimizers/optimizer.py", line 544, in minimize
self.apply_gradients(grads_and_vars)
File "/home/amarok/Projects/PlanEat/.venv3.9/lib64/python3.9/site-packages/official/nlp/optimization.py", line 175, in apply_gradients
return super(AdamWeightDecay, self).apply_gradients(
File "/home/amarok/Projects/PlanEat/.venv3.9/lib64/python3.9/site-packages/keras/src/optimizers/optimizer.py", line 1230, in apply_gradients
return super().apply_gradients(grads_and_vars, name=name)
File "/home/amarok/Projects/PlanEat/.venv3.9/lib64/python3.9/site-packages/keras/src/optimizers/optimizer.py", line 650, in apply_gradients
self._apply_weight_decay(trainable_variables)
File "/home/amarok/Projects/PlanEat/.venv3.9/lib64/python3.9/site-packages/keras/src/optimizers/optimizer.py", line 1249, in _apply_weight_decay
tf.internal.distribute.interim.maybe_merge_call(
File "/home/amarok/Projects/PlanEat/.venv3.9/lib64/python3.9/site-packages/keras/src/optimizers/optimizer.py", line 1245, in distributed_apply_weight_decay
distribution.extended.update(
File "/home/amarok/Projects/PlanEat/.venv3.9/lib64/python3.9/site-packages/keras/src/optimizers/optimizer.py", line 1241, in weight_decay_fn
wd = tf.cast(self.weight_decay, variable.dtype)
Node: 'Adam/Cast'
Cast string to float is not supported
[[{{node Adam/Cast}}]] [Op:__inference_train_function_181018]
(.venv3.9)  ✘ amarok@L-SBLIN  ~/Projects/PlanEat/models/ingredients   main ±  pip freeze
absl-py==1.4.0
array_record==0.5.1
astunparse==1.6.3
attrs==23.2.0
audioread==3.0.1
bleach==6.1.0
cachetools==5.3.3
certifi==2024.7.4
cffi==1.16.0
charset-normalizer==3.3.2
click==8.1.7
cycler==0.12.1
Cython==3.0.10
dataclasses==0.6
decorator==5.1.1
dm-tree==0.1.8
etils==1.5.2
fire==0.6.0
flatbuffers==24.3.25
fsspec==2024.6.1
gast==0.4.0
gin-config==0.5.0
google-api-core==2.19.1
google-api-python-client==2.137.0
google-auth==2.32.0
google-auth-httplib2==0.2.0
google-auth-oauthlib==1.0.0
google-cloud-bigquery==3.25.0
google-cloud-core==2.4.1
google-crc32c==1.5.0
google-pasta==0.2.0
google-resumable-media==2.7.1
googleapis-common-protos==1.63.1
grpcio==1.64.1
grpcio-status==1.48.2
h5py==3.11.0
httplib2==0.22.0
idna==3.7
importlib_metadata==8.0.0
importlib_resources==6.4.0
joblib==1.4.2
kaggle==1.6.14
keras==2.13.1
Keras-Preprocessing==1.1.2
kiwisolver==1.4.5
libclang==18.1.1
librosa==0.8.1
llvmlite==0.43.0
lxml==5.2.2
Markdown==3.6
markdown-it-py==3.0.0
MarkupSafe==2.1.5
matplotlib==3.4.3
mdurl==0.1.2
ml-dtypes==0.4.0
namex==0.0.8
neural-structured-learning==1.4.0
numba==0.60.0
numpy==1.23.0
oauthlib==3.2.2
opencv-python-headless==<IP_ADDRESS>
opt-einsum==3.3.0
optree==0.12.1
packaging==20.9
pandas==2.2.2
pillow==10.4.0
platformdirs==4.2.2
pooch==1.8.2
promise==2.3
proto-plus==1.24.0
protobuf==4.25.3
psutil==6.0.0
py-cpuinfo==9.0.0
pyasn1==0.6.0
pyasn1_modules==0.4.0
pybind11==2.13.1
pycparser==2.22
Pygments==2.18.0
pyparsing==3.1.2
python-dateutil==2.9.0.post0
python-slugify==8.0.4
pytz==2024.1
PyYAML==6.0.1
requests==2.32.3
requests-oauthlib==2.0.0
resampy==0.4.3
rich==13.7.1
rsa==4.9
scann==1.2.10
scikit-learn==1.5.1
scipy==1.13.1
sentencepiece==0.2.0
six==1.16.0
sounddevice==0.4.7
soundfile==0.12.1
tensorboard==2.13.0
tensorboard-data-server==0.7.2
tensorboard-plugin-wit==1.8.1
tensorflow==2.13.1
tensorflow-addons==0.23.0
tensorflow-datasets==4.9.0
tensorflow-estimator==2.13.0
tensorflow-hub==0.12.0
tensorflow-io-gcs-filesystem==0.37.1
tensorflow-metadata==1.13.0
tensorflow-model-optimization==0.8.0
tensorflowjs==3.18.0
termcolor==2.4.0
text-unidecode==1.3
tf-models-official==2.3.0
tf-slim==1.1.0
tflite-model-maker==0.4.3
tflite-support==0.4.4
threadpoolctl==3.5.0
toml==0.10.2
tqdm==4.66.4
typeguard==2.13.3
typing_extensions==4.5.0
tzdata==2024.1
uritemplate==4.1.1
urllib3==1.25.11
webencodings==0.5.1
Werkzeug==3.0.3
wrapt==1.16.0
zipp==3.19.2

Then with tensorflow 2.17:

python train.py
2024-07-12 10:59:02.240329: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable TF_ENABLE_ONEDNN_OPTS=0.
2024-07-12 10:59:02.240648: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2024-07-12 10:59:02.243211: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2024-07-12 10:59:02.249174: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-07-12 10:59:02.258851: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-07-12 10:59:02.261519: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-07-12 10:59:02.269337: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-07-12 10:59:02.878013: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Traceback (most recent call last):
File "/home/amarok/Projects/PlanEat/models/ingredients/train.py", line 1, in
from tflite_model_maker import model_spec
File "/home/amarok/Projects/PlanEat/.venv3.9/lib64/python3.9/site-packages/tflite_model_maker/init.py", line 44, in
from tflite_model_maker import audio_classifier
File "/home/amarok/Projects/PlanEat/.venv3.9/lib64/python3.9/site-packages/tflite_model_maker/audio_classifier/init.py", line 24, in
from tensorflow_examples.lite.model_maker.core.data_util.audio_dataloader import DataLoader
File "/home/amarok/Projects/PlanEat/.venv3.9/lib64/python3.9/site-packages/tensorflow_examples/lite/model_maker/core/data_util/audio_dataloader.py", line 27, in
from tensorflow_examples.lite.model_maker.core.task.model_spec import audio_spec
File "/home/amarok/Projects/PlanEat/.venv3.9/lib64/python3.9/site-packages/tensorflow_examples/lite/model_maker/core/task/model_spec/init.py", line 20, in
from tensorflow_examples.lite.model_maker.core.task.model_spec import audio_spec
File "/home/amarok/Projects/PlanEat/.venv3.9/lib64/python3.9/site-packages/tensorflow_examples/lite/model_maker/core/task/model_spec/audio_spec.py", line 30, in
from tensorflow_examples.lite.model_maker.core.task import model_util
File "/home/amarok/Projects/PlanEat/.venv3.9/lib64/python3.9/site-packages/tensorflow_examples/lite/model_maker/core/task/model_util.py", line 28, in
from tensorflowjs.converters import converter as tfjs_converter
File "/home/amarok/Projects/PlanEat/.venv3.9/lib64/python3.9/site-packages/tensorflowjs/init.py", line 21, in
from tensorflowjs import converters
File "/home/amarok/Projects/PlanEat/.venv3.9/lib64/python3.9/site-packages/tensorflowjs/converters/init.py", line 21, in
from tensorflowjs.converters.converter import convert
File "/home/amarok/Projects/PlanEat/.venv3.9/lib64/python3.9/site-packages/tensorflowjs/converters/converter.py", line 37, in
from tensorflowjs.converters import tf_saved_model_conversion_v2
File "/home/amarok/Projects/PlanEat/.venv3.9/lib64/python3.9/site-packages/tensorflowjs/converters/tf_saved_model_conversion_v2.py", line 42, in
import tensorflow_hub as hub
File "/home/amarok/Projects/PlanEat/.venv3.9/lib64/python3.9/site-packages/tensorflow_hub/init.py", line 88, in
from tensorflow_hub.estimator import LatestModuleExporter
File "/home/amarok/Projects/PlanEat/.venv3.9/lib64/python3.9/site-packages/tensorflow_hub/estimator.py", line 62, in
class LatestModuleExporter(tf.compat.v1.estimator.Exporter):
File "/home/amarok/Projects/PlanEat/.venv3.9/lib64/python3.9/site-packages/tensorflow/python/util/module_wrapper.py", line 232, in _getattr
attr = getattr(self._tfmw_wrapped_module, name)
AttributeError: module 'tensorflow.compat.v1' has no attribute 'estimator'

And 2.15.0

2024-07-12 11:01:56.289875: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable TF_ENABLE_ONEDNN_OPTS=0.
2024-07-12 11:01:56.291246: I external/local_tsl/tsl/cuda/cudart_stub.cc:31] Could not find cuda drivers on your machine, GPU will not be used.
2024-07-12 11:01:56.313500: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-07-12 11:01:56.313534: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-07-12 11:01:56.314475: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-07-12 11:01:56.318960: I external/local_tsl/tsl/cuda/cudart_stub.cc:31] Could not find cuda drivers on your machine, GPU will not be used.
2024-07-12 11:01:56.319116: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-07-12 11:01:56.967406: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Traceback (most recent call last):
File "/home/amarok/Projects/PlanEat/models/ingredients/train.py", line 1, in
from tflite_model_maker import model_spec
File "/home/amarok/Projects/PlanEat/.venv3.9/lib64/python3.9/site-packages/tflite_model_maker/init.py", line 44, in
from tflite_model_maker import audio_classifier
File "/home/amarok/Projects/PlanEat/.venv3.9/lib64/python3.9/site-packages/tflite_model_maker/audio_classifier/init.py", line 24, in
from tensorflow_examples.lite.model_maker.core.data_util.audio_dataloader import DataLoader
File "/home/amarok/Projects/PlanEat/.venv3.9/lib64/python3.9/site-packages/tensorflow_examples/lite/model_maker/core/data_util/audio_dataloader.py", line 27, in
from tensorflow_examples.lite.model_maker.core.task.model_spec import audio_spec
File "/home/amarok/Projects/PlanEat/.venv3.9/lib64/python3.9/site-packages/tensorflow_examples/lite/model_maker/core/task/model_spec/init.py", line 20, in
from tensorflow_examples.lite.model_maker.core.task.model_spec import audio_spec
File "/home/amarok/Projects/PlanEat/.venv3.9/lib64/python3.9/site-packages/tensorflow_examples/lite/model_maker/core/task/model_spec/audio_spec.py", line 30, in
from tensorflow_examples.lite.model_maker.core.task import model_util
File "/home/amarok/Projects/PlanEat/.venv3.9/lib64/python3.9/site-packages/tensorflow_examples/lite/model_maker/core/task/model_util.py", line 28, in
from tensorflowjs.converters import converter as tfjs_converter
File "/home/amarok/Projects/PlanEat/.venv3.9/lib64/python3.9/site-packages/tensorflowjs/init.py", line 21, in
from tensorflowjs import converters
File "/home/amarok/Projects/PlanEat/.venv3.9/lib64/python3.9/site-packages/tensorflowjs/converters/init.py", line 21, in
from tensorflowjs.converters.converter import convert
File "/home/amarok/Projects/PlanEat/.venv3.9/lib64/python3.9/site-packages/tensorflowjs/converters/converter.py", line 37, in
from tensorflowjs.converters import tf_saved_model_conversion_v2
File "/home/amarok/Projects/PlanEat/.venv3.9/lib64/python3.9/site-packages/tensorflowjs/converters/tf_saved_model_conversion_v2.py", line 42, in
import tensorflow_hub as hub
File "/home/amarok/Projects/PlanEat/.venv3.9/lib64/python3.9/site-packages/tensorflow_hub/init.py", line 93, in
from tensorflow_hub.feature_column_v2 import text_embedding_column_v2
File "/home/amarok/Projects/PlanEat/.venv3.9/lib64/python3.9/site-packages/tensorflow_hub/feature_column_v2.py", line 24, in
from tensorflow_hub import keras_layer
File "/home/amarok/Projects/PlanEat/.venv3.9/lib64/python3.9/site-packages/tensorflow_hub/keras_layer.py", line 27, in
from tensorflow.python.training.tracking import data_structures
ModuleNotFoundError: No module named 'tensorflow.python.training.tracking'
(.venv3.9)  ✘ amarok@L-SBLIN  ~/Projects/PlanEat/models/ingredients   main ±  pip freeze
absl-py==1.4.0
array_record==0.5.1
astunparse==1.6.3
attrs==23.2.0
audioread==3.0.1
bleach==6.1.0
cachetools==5.3.3
certifi==2024.7.4
cffi==1.16.0
charset-normalizer==3.3.2
click==8.1.7
cycler==0.12.1
Cython==3.0.10
dataclasses==0.6
decorator==5.1.1
dm-tree==0.1.8
etils==1.5.2
fire==0.6.0
flatbuffers==24.3.25
fsspec==2024.6.1
gast==0.4.0
gin-config==0.5.0
google-api-core==2.19.1
google-api-python-client==2.137.0
google-auth==2.32.0
google-auth-httplib2==0.2.0
google-auth-oauthlib==1.0.0
google-cloud-bigquery==3.25.0
google-cloud-core==2.4.1
google-crc32c==1.5.0
google-pasta==0.2.0
google-resumable-media==2.7.1
googleapis-common-protos==1.63.1
grpcio==1.64.1
grpcio-status==1.48.2
h5py==3.11.0
httplib2==0.22.0
idna==3.7
importlib_metadata==8.0.0
importlib_resources==6.4.0
joblib==1.4.2
kaggle==1.6.14
keras==3.4.1
Keras-Preprocessing==1.1.2
kiwisolver==1.4.5
libclang==18.1.1
librosa==0.8.1
llvmlite==0.43.0
lxml==5.2.2
Markdown==3.6
markdown-it-py==3.0.0
MarkupSafe==2.1.5
matplotlib==3.4.3
mdurl==0.1.2
ml-dtypes==0.2.0
namex==0.0.8
neural-structured-learning==1.4.0
numba==0.60.0
numpy==1.23.4
oauthlib==3.2.2
opencv-python-headless==<IP_ADDRESS>
opt-einsum==3.3.0
optree==0.12.1
packaging==20.9
pandas==2.2.2
pillow==10.4.0
platformdirs==4.2.2
pooch==1.8.2
promise==2.3
proto-plus==1.24.0
protobuf==4.25.3
psutil==6.0.0
py-cpuinfo==9.0.0
pyasn1==0.6.0
pyasn1_modules==0.4.0
pybind11==2.13.1
pycparser==2.22
Pygments==2.18.0
pyparsing==3.1.2
python-dateutil==2.9.0.post0
python-slugify==8.0.4
pytz==2024.1
PyYAML==6.0.1
requests==2.32.3
requests-oauthlib==2.0.0
resampy==0.4.3
rich==13.7.1
rsa==4.9
scann==1.2.10
scikit-learn==1.5.1
scipy==1.13.1
sentencepiece==0.2.0
six==1.16.0
sounddevice==0.4.7
soundfile==0.12.1
tensorboard==2.15.2
tensorboard-data-server==0.7.2
tensorboard-plugin-wit==1.8.1
tensorflow==2.15.0
tensorflow-addons==0.23.0
tensorflow-datasets==4.9.0
tensorflow-estimator==2.15.0
tensorflow-hub==0.12.0
tensorflow-io-gcs-filesystem==0.37.1
tensorflow-metadata==1.13.0
tensorflow-model-optimization==0.8.0
tensorflowjs==3.18.0
termcolor==2.4.0
text-unidecode==1.3
tf-models-official==2.3.0
tf-slim==1.1.0
tflite-model-maker==0.4.3
tflite-support==0.4.4
threadpoolctl==3.5.0
toml==0.10.2
tqdm==4.66.4
typeguard==2.13.3
typing_extensions==4.5.0
tzdata==2024.1
uritemplate==4.1.1
urllib3==1.25.11
webencodings==0.5.1
Werkzeug==3.0.3
wrapt==1.14.1
zipp==3.19.2

So the question is, is there a tuple python/tensorflow/tflite-model-maker/numpy/keras I can try cause compatibility seems a clusterfuck

I also tried https://www.tensorflow.org/lite/models/modify/model_maker/question_answer that give the same result (because my model is Squad v2.0, so I tried Squad v1).

Hi @pkgoogle ,
I have also been running into tflite model maker installation issues, where it just goes in a loop of downloading bunch of files.
!pip install -q tflite-model-maker
 Preparing metadata (setup.py) ... done      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.2/5.2 MB 48.1 MB/s eta 0:00:00      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.7/1.7 MB 74.9 MB/s eta 0:00:00      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 80.3/80.3 kB 7.8 MB/s eta 0:00:00      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 632.0/632.0 MB 2.1 MB/s eta 0:00:00      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 621.0/621.0 MB 2.2 MB/s eta 0:00:00      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 620.6/620.6 MB 1.6 MB/s eta 0:00:00      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 620.6/620.6 MB 2.2 MB/s eta 0:00:00      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 620.6/620.6 MB 2.2 MB/s eta 0:00:00      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 620.6/620.6 MB 1.9 MB/s eta 0:00:00      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 620.6/620.6 MB 1.6 MB/s eta 0:00:00      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 620.5/620.5 MB 2.2 MB/s eta 0:00:00      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 607.5/607.5 MB 1.7 MB/s eta 0:00:00      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 607.5/607.5 MB 2.3 MB/s eta 0:00:00      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 606.2/606.2 MB 2.0 MB/s eta 0:00:00      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 606.2/606.2 MB 940.0 kB/s eta 0:00:00      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 606.2/606.2 MB 2.5 MB/s eta 0:00:00      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 605.8/605.8 MB 2.5 MB/s eta 0:00:00      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 604.7/604.7 MB 2.0 MB/s eta 0:00:00      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 604.5/604.5 MB 2.1 MB/s eta 0:00:00      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 604.6/604.6 MB 2.0 MB/s eta 0:00:00      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 604.5/604.5 MB 2.1 MB/s eta 0:00:00      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 604.4/604.4 MB 2.3 MB/s eta 0:00:00      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 604.4/604.4 MB 1.2 MB/s eta 0:00:00      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 604.4/604.4 MB 2.0 MB/s eta 0:00:00      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 600.4/600.4 MB 1.3 MB/s eta 0:00:00      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 600.3/600.3 MB 2.2 MB/s eta 0:00:00      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 600.0/600.0 MB 1.4 MB/s eta 0:00:00      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 590.9/590.9 MB 2.0 MB/s eta 0:00:00      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 590.9/590.9 MB 1.9 MB/s eta 0:00:00      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 591.0/591.0 MB 1.3 MB/s eta 0:00:00      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 591.0/591.0 MB 1.6 MB/s eta 0:00:00      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 591.0/591.0 MB 1.8 MB/s eta 0:00:00      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 591.0/591.0 MB 1.6 MB/s eta 0:00:00      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 590.9/590.9 MB 1.2 MB/s eta 0:00:00      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 590.9/590.9 MB 2.5 MB/s eta 0:00:00      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 590.7/590.7 MB 2.4 MB/s eta 0:00:00      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 590.7/590.7 MB 1.8 MB/s eta 0:00:00      ━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 175.1/590.7 MB 51.7 MB/s eta 0:00:09 ERROR: Operation cancelled by user
I am not sure if this issue can be resolved through mediapipe model maker , can you please take a look ?

Hi @AmarOk1412, please either use mediapipe or please use a modern LLM for your use case, BertQA is quite outdated so the tools and infrastructure to help it generally are less stable, may I recommend https://huggingface.co/google/gemma-2b-it for your use case?

I'll, converting the models may take a while as i'm in vacations. But will try. For now, I tried llama2 but it wasn't good, will try gemma

Hi @AmarOk1412, awesome, let us know if somehow you feel Gemma isn't up to par as well so that we may be able to debug whether it is a use issue or not.

Hi @AmarOk1412  It appears that there might be some issues with the installation process. Let's explore potential solutions:
1.Dependency Resolution:
- The error you encountered might be related to dependency conflicts. To address this, consider the following steps:
- Loosen Version Constraints: Modify your package version constraints to allow more flexibility. Sometimes, specifying a narrower range can lead to conflicts.
- Remove Specific Versions: Remove specific package versions to let pip attempt to resolve the dependency conflict algorithmically.
https://stackoverflow.com/questions/77537307/error-cannot-install-tflite-model-maker-the-conflict-is-caused-by-other-module
2.Fallback Runtime Version in Colab:
- If you're using Colab, try using the fallback runtime version option. Choose Python 3.9 and install tflite-model-maker. You might encounter a runtime error, but it can be safely ignored.
https://discuss.tensorflow.org/t/tflite-model-maker-installation/16577/26
3.Nightly Version:
- While you mentioned concerns about stability, consider using the nightly version of tflite-model-maker. It might provide a workaround until the issue is resolved.
https://github.com/tensorflow/tensorflow/issues/71666


Specify ns not project

Part of DOC-122

Closed in favor of https://github.com/loft-sh/vcluster-docs/pull/201.


Better suggestions for error handling

Hi.
This should improve the readme, as suggested by #1351

This PR has not been touched in at least 6 months. If this is still being worked on, please respond and we can continue the review process (you will probably have to sync / rebase). Otherwise I intend to close this pull request in a week.


Missing exclamation mark in override (Null Safety migration tool)

Consider the following snippet:
dynamic doSth() => 'something';

abstract class Foo {
  String /*!*/ foo();
}

class Bar extends Foo {
  @override
  String foo() => doSth()?.toString();
}

The migration tool correctly sets the return type of the Bar.foo to be String since it's explicitly hinted. But it does not add an exclamation mark to the body, which means that the generated code doesn't compile:
dynamic doSth() => 'something';

abstract class Foo {
  String /*!*/ foo();
}

class Bar extends Foo {
  @override
  String  foo() => doSth()?.toString();
}

@stereotype441

As of 1c7fe7196dd7a542f7f91cd6ea2ac81e1a13146c, the null safety migration tool has been removed from active development and retired. No further work on the tool is planned.
If you still need help, or you believe this issue has been closed in error, please feel free to reopen.


How does Divine Sense work?

I've got the spell with an AE of Detect Evil and Good. It applies to the caster when I cast the spell. It's being added to the detection mode tab on the token config sheet. But it's not really showing anything on the map. How do the "evil" tokens have to be configured to trigger this? I've got the alignments in the proper boxes...

But it doesn't seem to be doing anything. Divine sense also doesn't seem to work. It's not picking up anything.

Detect Evil and Good is poorly named in 5E…
For the duration, you know if there is an aberration, celestial, elemental, fey, fiend, or undead within 30 feet of you, as well as where the creature is located.

ah, that makes sense. Sorry. But Divine Sense is also not showing anything.

It's working for me... if you want to chat about it I'll see you on Posney's discord?
This Commoner is undead... 😱

I assume this was just a misunderstanding of the rules of Divine Sense. If not, let me now.

That it was. My apologies.

On Jul 8, 2023, at 12:52 PM, dev7355608 @.***> wrote:
I assume this was just a misunderstanding of the rules of Divine Sense. If not, let me now.
—
Reply to this email directly, view it on GitHub https://github.com/dev7355608/vision-5e/issues/17#issuecomment-1627408056, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABCORQT5KZKMJ3MT2Q7NSZ3XPGF4RANCNFSM6AAAAAAZ36ZNUA.
You are receiving this because you authored the thread.

Christopher Allbritton
+1<PHONE_NUMBER>
@.*** @.***>
christopherallbritton.com https://www.christopherallbritton.com/
https://www.twitter.com/chrisallbritton https://www.instagram.com/christopherallbritton/ https://www.linkedin.com/in/callbritton/
B57D DD90 57FD A0D8 52C5  AF86 FA58 861F FAB2 F8B9


Clean up prologue/epilogue functions

There is probably a good amount of consolidation and general clean up that can be done with these to make them easier to understand.

For my own work, I was thinking of calling through a vector of unique_ptr's to functions with mutexes between them. Would you like to see my implementation once its ready? It would be a pleasure to pitch in.

Do you mean in regards to prologue/epilogue functions?  I'll look at anything cereal related that someone does, though your approach would introduce too much overhead to be a general solution we could include in cereal - we prefer to keep everything resolvable at compile time if possible.


Locate git repository upwards

this patch locates the git repository - a real one or a work tree - upwards in the directory tree, if the solution file is not in the root of the repository.

Nice! This is actually something I discovered while I was working to add VS2022 support, but left as a comment to fix later.
Thanks, let me test it and I will merge soon.


Support additional block adapters in unit tests

Add the following block adapters to the unit test infrastructure:

S3
GCS
Mem

@N-o-Z  can close this one? as far as I remember you added this capability.

Added the unit tests, but we didn't add these block adapters

Missing mem adapted unit tests


pre-commit catch for embargoed structures

We should do a better job at catching confidential structures before they're committed to this public space. We'll need to build something that:

grabs the list of confidential structures from CDD (both VC IDs and SMILES)
for any commit, checks if there are any matching VC IDs
for any commit, checks if there are any matching SMILES/ molecular matches
for any commit, extract ligands from PDB files and do the same as above
reject a commit if a match is found

There might be edge cases such as ligands living in JSON or other formats, but catching the above will prevent the vast majority of leaks.

Tagging @ijpulidos

Having thought about this more, I am not sure centralising all our sensitive information is a good idea.
Here are some initial practical steps, the rest we can discuss in private.

Add all .pdb .mol and .sdf extensions to gitignore so that they are hard to commit (must use force)
We should pull down datafiles from an S3 bucket rather than have them committed to the repo, that way we can yank them at any time without having to rewrite history.


Inserting lots of data to Google BigQuery randomly throws an SSL Exception

👋🏻 G'Day!
Problem
An SSL error is randomly thrown while importing a lot of data into Google BigQuery.
Details
I've got a (once off) maintenance task which is trying to insert about 10mil rows into a Google BigQuery table.
Randomly (well, it feels random) an SSL error thrown which crashes my app.

Authentication failed because the remote party sent a TLS alert: 'DecryptError'.

I've added in some retrying ability (using Polly for .NET) and we're back on track.
I'm not really sure how to reproduce it, but I've provided some info here to maybe help. It's happened after 30/40k have been pushed up. Other times, hundreds of thousands pushed up. With Polly, it retries and it works again/continues ... until the next random failure .. which Polly retries OK and we rinse/repeat.

Environment details
PS C:\Users\justi> dotnet --info
.NET SDK:
 Version:   7.0.304
 Commit:    7e794e2806

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.22621
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\7.0.304\

Host:
  Version:      7.0.7
  Architecture: x64
  Commit:       5b20af47d9

Package name and version: Google.Cloud.BigQuery.V2 v3.2.0

Here's some sample code I've got which is doing this:
// Configure the retry policy with Polly
var retryPolicy = Policy.Handle<Exception>()
    .WaitAndRetryAsync(3, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)), (exception, timeSpan, retryCount, context) =>
    {
        // Log the exception to the console
        Console.WriteLine($"Retry #{retryCount} after {timeSpan.TotalSeconds} seconds due to: {exception}");
    });

try
{
    // Execute the retry policy for the HTTP request
    var pollyResult = await retryPolicy.ExecuteAsync(async () =>
    {
        var result = await client.InsertRowsAsync(destinationTable, rows, null, cancellationToken);

        return result;
    });

    _logger.LogDebug("Finished sending all rows to Google BigQuery. Status: {bigQueryInsertStatus}", pollyResult.Status);

    pollyResult.ThrowOnAnyError();

}
catch (Exception exception)
{
    // Log the final exception to the console
    Console.WriteLine($"Error occurred after retries: {exception}");

    throw;
}

Hi @PureKrome,
We've discussed this in the team, and we'd like to at least try to reproduce the problem for the sake of completeness, but we suspect we won't want to actually make any changes to how retries are performed... while it may be safe to automatically retry in your particular case, at the point where the network transport has problems we're in unusual territory, and in many cases retrying would be the wrong approach.
Could you let us know:

What your insert code looks like (just which methods you're using, and ideally a rough idea of how how large each row is, in terms of fields and total data)
What your network is like - are you going through a proxy which might be causing problems, for example? (That seems to be the most likely cause of problems, to be honest.)
How often you see the errors in terms of elapsed time (due to some internal details, I wouldn't be entirely surprised to see issues once every hour, for example)

I've tried to reproduce this in https://github.com/jskeet/google-cloud-dotnet/tree/issue-10625 - I've not seen the problem yet.
My most recently attempt consisted of:

10 million rows
Inserting 1000 rows per request
Each row consists of an integer ID, and a text field with 20 random ASCII characters

(The test took about 25 minutes from home.)
If you'd be happy to run the repro code in your environment, the results would be interesting.

Thanks @jskeet for showing interest into this issue - I really really appreciate it.
I'll try and answer as much of the questions as possible:

What your insert code looks like (just which methods you're using, and ideally a rough idea of how how large each row is, in terms of fields and total data)

var result = await client.InsertRowsAsync(destinationTable, rows, null, cancellationToken);

i have 7 tables i'm inserting data into. So I just spent the last 2 hours running my app to see when/where it fails and if a specific table is always the cause. Two different tables had the same error at random times. The first one was at rows 83k -> 84k. The 2nd one was at rows 2.433m -> 2.434m. I've also seen it happen at various/random places.

What your network is like - are you going through a proxy which might be causing problems, for example? (That seems to be the most likely cause of problems, to be honest.)

The errors occurred both from my home network and through the office network. Neither has anything special. Home has a pi-hole, but that's DNS. Office has nothing like that. I don't have anything special setup on my dev box here. E.g. no custom firewalls, etc.

How often you see the errors in terms of elapsed time (due to some internal details, I wouldn't be entirely surprised to see issues once every hour, for example)

Pretty random. Sometimes it's 5 - 10 mins. other times .. an hour+ it's usually never longer than an hour -> tonight's run was unique how i got so far (from 80k -> 2.4m)

If you'd be happy to run the repro code in your environment, the results would be interesting.

Test 1
Ok. So I've done this and it worked 100% from my home env/location. Here's some data to help explain what I did (so we are both on the same page with no assumptions)

PS C:\Users\justi\Downloads\google-cloud-dotnet-issue-10625\issues\Issue10625> dotnet run <snipped> issue10625 test_table 20 10000000 1000
12:31:11 Inserted 0 rows
12:31:13 Inserted 10000 rows
...
12:39:22 Inserted 9970000 rows
12:39:22 Inserted 9980000 rows
12:39:22 Inserted 9990000 rows
PS C:\Users\justi\Downloads\google-cloud-dotnet-issue-10625\issues\Issue10625>

and this was with a simple WIFI connection to home internet (25 up).
I had to change the global.json to:
{
    "sdk": {
        "version": "6.0.100",
        "rollForward": "latestFeature"
    }
}

and this was how i did my creds:

the datasetId was in the same region (australia-southeast2) as the other datasetId which was showing my issue. The only difference between the two data set info was the new one (i just created) has Time travel window: 7 days (maybe this is something new?)
Test 2
This time, I'm going to try and use the async methods, to get similar to my code which uses async (i'm getting desperate..)

.. and this worked 💯 fine, like before.
Question.

rough idea of how how large each row is, in terms of fields and total data

How could I calculate how large each row is?
With the 2x errors that occurred in the run above,

Exception 1 : table has 21 columns
Exception 2 : table has 9 columns

cheers!

Thanks for the info. Will try with more columns next week, although it's hard to see how that would affect things.
I may also try with a delay to deliberately run the test over a long time period without actually having to insert billions of rows.

How could I calculate how large each row is?

Just roughly in terms of what the field types are, and how long any strings or byte arrays are - very roughly, as in are we talking about 100 bytes per row or 100K...

👋🏻 G'Day @jskeet
sorry for the slow/late reply. Here's some more info:
  **** Reading in rows: 1  to 1,000 ****
Total size of data (in bytes): 142,460. MaxSize:200 MinSize:100
  **** Reading in rows: 1,001  to 2,000 ****
Total size of data (in bytes): 140,902. MaxSize:206 MinSize:102
  **** Reading in rows: 2,001  to 3,000 ****
Total size of data (in bytes): 140,830. MaxSize:212 MinSize:100
  **** Reading in rows: 3,001  to 4,000 ****
Total size of data (in bytes): 140,628. MaxSize:210 MinSize:94
  **** Reading in rows: 4,001  to 5,000 ****
Total size of data (in bytes): 141,728. MaxSize:208 MinSize:94
  **** Reading in rows: 5,001  to 6,000 ****
Total size of data (in bytes): 139,766. MaxSize:198 MinSize:100
  **** Reading in rows: 6,001  to 7,000 ****
Total size of data (in bytes): 141,180. MaxSize:240 MinSize:96
  **** Reading in rows: 7,001  to 8,000 ****
Total size of data (in bytes): 140,360. MaxSize:196 MinSize:98
  **** Reading in rows: 8,001  to 9,000 ****
Total size of data (in bytes): 146,100. MaxSize:210 MinSize:98
  **** Reading in rows: 9,001  to 10,000 ****
Total size of data (in bytes): 140,604. MaxSize:208 MinSize:92
  **** Reading in rows: 10,001  to 11,000 ****
Total size of data (in bytes): 139,920. MaxSize:196 MinSize:98
  **** Reading in rows: 11,001  to 12,000 ****
Total size of data (in bytes): 140,680. MaxSize:216 MinSize:102
  **** Reading in rows: 12,001  to 13,000 ****
Total size of data (in bytes): 140,378. MaxSize:196 MinSize:90
  **** Reading in rows: 13,001  to 14,000 ****
Total size of data (in bytes): 140,988. MaxSize:210 MinSize:98
  **** Reading in rows: 14,001  to 15,000 ****
Total size of data (in bytes): 138,344. MaxSize:244 MinSize:102
  **** Reading in rows: 15,001  to 16,000 ****
Total size of data (in bytes): 139,504. MaxSize:198 MinSize:106
  **** Reading in rows: 16,001  to 17,000 ****
Total size of data (in bytes): 142,420. MaxSize:196 MinSize:104
  **** Reading in rows: 17,001  to 18,000 ****
Total size of data (in bytes): 139,298. MaxSize:228 MinSize:98
  **** Reading in rows: 18,001  to 19,000 ****
Total size of data (in bytes): 140,160. MaxSize:206 MinSize:104
  **** Reading in rows: 19,001  to 20,000 ****
Total size of data (in bytes): 143,954. MaxSize:196 MinSize:104
  **** Reading in rows: 20,001  to 21,000 ****

so out of the 10mil odd db rows, here's some data for the first 20k rows.
feels like each poco is just over 200 bytes.

the `record` has **21** properties. (click to see code)
public record ListingData(
        int ListingId,
        DateTime UpdatedOn,
        string StatusType,
        string CategoryType,
        string? StreetNumber,
        string? Street,
        string Suburb,
        string? City,
        string State,
        int Postcode,
        bool IsStreetDisplayed,
        DateTime? OnMarketOn,
        string ProviderName,
        bool IsVisible,
        bool IsLive,
        string Tier,
        int BedroomsCount,
        int BathroomsCount,
        int CarsCount,
        int OfficeId,
        DateTime ImportedOn)
    {
        public int CalculateMemorySize()
        {
            // calculate the memory size as bytes of this single instance
            var sizeOfStatusType = Encoding.Unicode.GetByteCount(StatusType);
            var sizeOfCategoryType = Encoding.Unicode.GetByteCount(CategoryType);
            var sizeOfStreetNumber = string.IsNullOrEmpty(StreetNumber) ? 0 : Encoding.Unicode.GetByteCount(StreetNumber);
            var sizeOfStreet = string.IsNullOrEmpty(Street) ? 0 : Encoding.Unicode.GetByteCount(Street);
            var sizeOfSuburb = Encoding.Unicode.GetByteCount(Suburb);
            var sizeOfCity = string.IsNullOrEmpty(City) ? 0 : Encoding.Unicode.GetByteCount(City);
            var sizeOfState = Encoding.Unicode.GetByteCount(State);
            var sizeOfProviderName = Encoding.Unicode.GetByteCount(ProviderName);
            var sizeOfTier = Encoding.Unicode.GetByteCount(Tier);

            var sizeOfInstance = System.Runtime.CompilerServices.Unsafe.SizeOf<ListingData>();

            var totalSize = sizeOfInstance +
                            sizeOfStatusType +
                            sizeOfCategoryType +
                            sizeOfStreetNumber +
                            sizeOfStreet +
                            sizeOfSuburb +
                            sizeOfCity +
                            sizeOfState +
                            sizeOfProviderName +
                            sizeOfTier;

            return totalSize;
        }
    }

does this help?

You're using Encoding.Unicode.GetByteCount() there, which will mean two bytes per character... I suspect they're mostly ASCII characters, so that'll be 1 byte per character in UTF-8. That suggests it's more like 50-100 bytes per row in terms of data - with some overhead for the number of fields. I'll edit my test accordingly, but I'd be surprised if that made a difference. Worth testing though.

That suggests it's more like 50-100 bytes per row in terms of data

kewl. which suggests it's not much data then, versus thousands and thousands of bytes, per row.
does the "region" have any difference? could there be some rogue SSL cert or server, in the region I'm pushing the data up to?

does the "region" have any difference? could there be some rogue SSL cert or server, in the region I'm pushing the data up to?

The region may well be relevant, although I think the chances of a rogue cert on Google's side are slim.
I'll definitely try with australia-southeast2 as well though.

Okay, next test - still in multi-region, US:
Text fields per row (in addition to ID): 20
Size of text per field: 20
Rows per insert: 1000
Total rows inserted: 10000000
Time taken (seconds): 4337
(Code is in the branch.) Now starting to run against a dataset in Australia (inserting from London).

@PureKrome: I'm not sure what the next step is at this point. If you can run the tests (which are now checked in at https://github.com/googleapis/google-cloud-dotnet/tree/main/issues/Issue10625) some more to see if the problem is reproduced that way, that would be useful. (I'd suggest using the GOOGLE_APPLICATION_CREDENTIALS environment variable to refer to a JSON file with your service account rather than modifying the code.)
I'm on holiday for a week now, so won't be able to run any more tests (although I'll still monitor this issue). Looking at the stack trace again, it's interesting that it's failing when establishing the connection... I'm not sure why that would happen during the run. Unless you're creating a new BigQueryClient for each batch of rows? If you're able to provide a similar standalone program that does reproduce the issue for you, that would be really helpful.

👋🏻 G'Day Jon!

Unless you're creating a new BigQueryClient for each batch of rows?

YES! This was what I was doing, actually! These lines:
// Id's: the current batch of Id's. e.g. 1->1000, 1001->2000, etc.
public async Task UploadDataAsync(IList<int> ids, CancellationToken cancellationToken)
{
    // Do some business logic + MS-SQL call.
    var (users, uniSubjects) = await _repository.GetDataAsync(ids, cancellationToken);

    var client = BigQueryClient.Create(_projectId, _googleCredential);
    var usersTableReference = client.GetTableReference(_datasetId, "users");
    var subjectsTableReference = client.GetTableReference(_datasetId, "subjects");

    // NOTE: there's no AWAIT here. This is just -the task-
    var usersTask = UploadToBigQueryAsync(
        client,
        usersTableReference,
        users,
        ConvertToUsersHandler,
        cancellationToken);

    // Again, no AWAIT.
    var subjectsTask = UTBQA(...);

    await Task.WhenAll(usersTask, subjectsTask);
}

So as you can see here, each time we did a batch of 1000 rows to upload to BQ, I would create the client AND the table references!
So now I've changed the code to be something like this:
// CTOR
public BigQueryService(...)
{
    _client = new Lazy<BigQueryClient>(() => BigQueryClient.Create(projectId, googleCredential));
    _usersTableReference = new Lazy<TableReference>(() => _client.Value.GetTableReference(_datasetId, "users"));
    _subjectsTableReference = new Lazy<TableReference>(() => _client.Value.GetTableReference(_datasetId, "subjects"));
}

public async Task UploadDataAsync(IList<int> ids, CancellationToken cancellationToken)
{
    // Do some business logic + MS-SQL call.
    var (users, uniSubjects) = await _repository.GetDataAsync(ids, cancellationToken);

    // Create the client and table references when FIRST TIME required/requested/called.
    var usersTask = UploadToBigQueryAsync(
        _client.Value,
        _usersTableReference.Value,
        users,
        ConvertToUsersHandler,
        cancellationToken);

    // Again, no AWAIT.
    var subjectsTask = UTBQA(...);

    await Task.WhenAll(usersTask, subjectsTask);
}

and the instance of this class is a singleton / single instance, in my main app. The Lazy<T> could be an overkill but I was just wanting to give that a crack.
So .. given this information ... I'm still trying to understand the relationship between the error (which the stack trace suggests it's during BQ.InsertRowsAsync and creating a client (too many times) ?
I'll also give your code another run, as request. I'll report back.

Each client has its own HttpClient, which would create a separate set of network connections. I suspect you're running out of TCP ports or similar. I strongly suspect this will clear up the problem. I might try making a change to the repro to create a new client for each batch and see if that fixes the problem - then I could document it accordingly.

I tried my code with the client initialization within the loop, and it was still working fine - but I didn't let it do a huge run due to other constraints. It's definitely worth changing your code to only create a single client though. The table reference is simple though - no need to worry about caching/reusing those.

Hi @jskeet - I really hope you're having a rest on your break so maybe read this reply when you get back 🏖 🌞
When I refactored to a single client I never got the issue, again. So it feels like this is not really an issue about INSERTING data. It's really about creating heaps of clients.
Maybe something like this might be used to see if this is an issue with the client creation?
using Google.Cloud.BigQuery.V2;

string projectId = "...";

// Create 1,000 clients
var tasks = new List<Task>(1000);

for (var i = 0; i < 100_000; i++)
{
    tasks.Clear();

    for (var j = 0; j < 1000; j++)
    {
        var task = CreateTaskAndRunASingleThing(projectId);
        tasks.Add(task);
    }

    await Task.WhenAll(tasks);

    Console.WriteLine($"Iteration {i:N0} completed");

    async Task CreateTaskAndRunASingleThing(string projectId)
    {
        // Credentials provided via json file via GOOGLE_APPLICATION_CREDENTIALS env var.
        var client = await BigQueryClient.CreateAsync(projectId);

        // NOTE: not sure if this is needed?
        await client.GetBigQueryServiceAccountEmailAsync(); // So -some- type of network traffic is executed.
    }
}

Not really answering... but I should check when I'm back whether we cache a service for default credentials, which would make my current test code unsuitable. At the very least, using the same credential for each client will be a bit different. And we don't use self-signed JWTs for BigQuery.

Hmm... I still can't reproduce this when inserting, even with loads of clients each with a new credential. But it may well depend on precise environmental factors at that point. I'm really glad to hear that using a single client seems to have resolved the problem - but I'll leave this issue open and have another go at reproducing it next week with the code you've suggested above.
Just for clarity, does that code reproduce the issues on your machine?

Just for clarity, does that code reproduce the issues on your machine?

Sorry - I don't understand. Which code do you mean, causes the SSL issue?
I also might see if i can reproduce it by packaging up my original code (1x client per upload-to-bigquery) into a single exe and then run it from some VM in azure. This way, I can see if it's a weird issue from my machine to isolate some hardware out of the equation.

Sorry - I don't understand. Which code do you mean, causes the SSL issue?

Yup. I haven't been able to reproduce that anywhere yet. Using a single client is a better idea anyway, but at the moment I don't know whether it's really related to this issue.

@PureKrome: Have you seen enough data now with a single client to know whether this has actually fixed the issue? While I still haven't reproduced the problem myself, I can imagine it could be related, which would be good to know in case we see other users with the same issue. It's a shame that I can't reproduce it, but if your code now works, I'd like to close this issue. If you're still seeing the error, we should absolutely continue to pursue it...

👋🏻 G'Day @jskeet - ok. interesting! I tried again running my code on an Azure VM and it didn't seem to error at all. Ran it on my localhost and yeah .. took a while but I got the error popping up again.
So i think it's then fair to say one (or more) of the follow:

It's something weird with my laptop
It's something weird with my home network (wifi? my pi-hole handling DNS requests?)

I was expecting the error to pop up when I did it on the Azure VM - but it didn't.
I feel really bad about this as I honestly thought it might be a rogue setting somewhere outside of my little bubble. Serious lessons learnt here.
I'm going to close this card.
Again Jon - thank you so much for showing some interest in this post. Keep up the awesome work.
Thanks, mate.

I feel really bad about this as I honestly thought it might be a rogue setting somewhere outside of my little bubble. Serious lessons learnt here.

Please don't feel bad about this at all - I thought it was a great collaboration. Okay, we didn't find anything in the end - but we still ended up with a code improvement for you, and I'll definitely remember this error if it comes up for another customer.
Please keep reporting bugs :)

Thanks Jon for the kind words. Yep - an action item here was definitely how to reuse the Client instead of creating a new one for each item to save. So there was some good that came out of this.
Thank you again for all your patience and great questions in trying to resolve this. Really appreciated it. 👏🏻 🍰 👍🏻


Reduce parse API variants with flexible return types

This reduces our various parse methods down to 2 major variants, document::parse() and parser.parse(), eliminating variants like try_parse by returning a more flexible result.
It enables these scenarios:
Error Codes
// Parse and create a new document
auto [doc, error] = document::parse("[1,2,3]");
if (error) { exit(1); }

// Use the parser's internal document
auto parser = document::parser();
auto [doc, error] = parser.parse("[1,2,3]");
if (error) { exit(1); }

// Steal the document from the parser so you can store it elsewhere
auto [doc, error] = parser.parse("[1,2,3]");
if (error) { exit(1); }
document my_doc = doc; // take the document (doc is document&, not document)

Exceptions
// Parse and create a new document
document doc = document::parse("[1,2,3]");

// Use the parser's internal document
auto parser = document::parser();
document &doc = parser.parse("[1,2,3]");

// Steal the document from the parser so you can store it elsewhere
document doc = parser.parse("[1,2,3]");

This reduces the number of parse() variants from 30 to 8, which makes me feel better :)

This looks very good (sexy). How is the performance... any effect?

I didn't check actually. I'll look through the benchmarks and see if there's a relevant one (parse isn't really designed to measure calling overhead, and doesn't call this method anyway since it splits up stage 1 and 2).

@jkeiser  you are doing good work and changing a lot of files. And you seem to understand the key concepts of valstat very well. Allow me to express few comments before I dive into the code :)
As ever I might be pretty sure it is just terminology.

what is the valstat type you are using? I seem to remember this is discussed

// very basic sketch
template <typename V, typename S>
struct valstat_template final {
  V * val;
  S * stat;
};

After which types can be defined. That type will slow down anything for approx 0 nano seconds.

That example above I would do differently

// Parse and create a new document
auto [doc, status] = document::parse("[1,2,3]");
// key is there is no obuject called "error"
// error is just one of four states of an valstat intance
// communicated in code somewhat like this:
//  { {} , status }
// no value and yes status, ditto:
if ( ! doc ) { exit(1); }

// Use the parser's internal document
auto parser = document::parser();
auto doc  = parser.parse("[1,2,3]");
// { {}, stat }  was returned
if (  ! doc.val ) { exit(1); }

// Steal the document from the parser so you can store it elsewhere
// on this "consuming site" I choose not to use structured binding
// doc is here an instance of some valstat type
auto doc = parser.parse("[1,2,3]");
if ( ! doc.val ) { exit(1); }
document my_doc = doc.val ;  // take the document (doc is document&, not document)

I am busy on the documentation branch.

@lemire @jkeiser
I would like us not to throw anything from simdjson.

@lemire I'm working through a new benchmark (benchmark/bench_parse_call.cpp), but here's some results I got out of the exception path vs. error path vs. build_parsed_json and json_parse (existing APIs):
--------------------------------------------------------------
Benchmark                    Time             CPU   Iterations
--------------------------------------------------------------
document_parse_take        286 ns          285 ns      2357895
document_parse_try         302 ns          305 ns      2357895
build_parsed_json          279 ns          276 ns      2488889
parser_parse               116 ns          117 ns      6400000
parser_parse_try           112 ns          112 ns      5600000
json_parse                 109 ns          107 ns      6400000

document.parse() is in the same range of build_parsed_json(), and parser.parse() in the range of json_parse(), as expected--the originals might be a tiny bit faster, but it's not entirely clear.
@DBJDBJ the change should be more manageable to read now; it included commits from another PR which is now merged. Of course, that's caused a build error, but I'll deal with that separate.
The big reason for using the word error instead of status is that in this case, doc and error are mutually exclusive, and the word error is more specific (and therefore conveys more information) than status. And the big reason for checking error instead of doc is that most error-handling code is going to do if (something) { cerr << error; } (or something else that uses the error). It's weird to say if (!doc) { cerr << error; exit(1); } when what you wanted was to check for error a la if (error) { cerr << error; exit(1); }
Apart from that, I don't understand one change in your example: why do you use auto doc = ...; if (!doc.val) in your later examples instead of using auto [doc, status] = ...; if (!doc)? Is there some reason not to make the examples use the same form each time?

@jkeiser if I can read those results "exceptions" are faster than "error codes"?  Well, that would be a novelty for the C++ community at large :)   The way(s) to measure and compare exceptions vs no exceptions are based on comparing applications built with or without. http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2019/p1640r1.html.
"valstatish" solution. Again, I might be wrong but you are returning:
class document::doc_result {
private:
  doc_result(document &&_value, error_code _error) : value(std::move(_value)), error(_error) { }
  doc_result(document &&_value) : value(std::move(_value)), error(SUCCESS) { }
  doc_result(error_code _error) : value(), error(_error) { }
  friend class document;
public:
  ~doc_result()=default;

  operator bool() noexcept { return error == SUCCESS; }
  operator document() {
    if (!*this) {
      throw invalid_json(error);
    }
    return std::move(value);
  }
  document value;
  error_code error;
};

Hardly an "error code" :) No wonder this is slow.  Among other things,
in there you are keeping the copy of document and having the implicit casting operator from which you are moving the document out to the caller. Making it unusable for any next calls. This class is not 'error code' and certainly not a valstat.
the terminology and the concept -- the valstat behaviour exhibits four states
// return two optional values
// value and status
// there is no 'special' return type visible
 auto [ value, status ] = function () ;
// four possible states
 if (   value &&   status )  { /* info */ }
 if (   value && ! status )  { /* ok   */ }
 if ( ! value &&   status )  { /* error*/ }
 if ( ! value && ! status )  { /* empty*/ }

The lack of value indicates an "error". "error" is a state not an object.  both value and status can be returned, that is not an error. example: HTTP returns consuming.
But, if you in your code want to call the state "error" you might as well do that.  In any case you do not need special method on that object to check if it is an error , it is enough to check for its existence or lack of. for example
// valstat is returned
// both doc and error are simple pointers
auto [ doc, error ] = parser.parse("[1,2,3]");
if ( ! error  ) { /* all is fine */ }
 else { exit(1); }

Apart from that, I don't understand one change in your example: why do you use auto doc = ...; if (!doc.val) in your later examples instead of using auto [doc, status] = ...; if (!doc)? Is there some reason not to make the examples use the same form each time?

This is because I wanted to show, structured binding is not mandatory. In your example, it is getting in the way:
auto [people, error] = json::load("people.json");
if (error) { cerr << "Error loading people.json: " << error << endl; return error; }
auto [person, error] = people[1];
if (error) { cerr << "Error getting people[1]: " << error << endl; return error; }
auto [name, error] = person.get<string>("Name");
if (error) { cerr << "Error getting people[1].Name: " << error << endl; return error; }
auto [legs, error] = person.get("Legs", 2);
if (error) { cerr << "Error getting people[1].Legs: " << error << endl; return error; }
cout << "Name: " << name << ", Legs: " << legs << endl;

That will not compile and will be a pain to change, as you have righly noted. If you do it without structued returns binding things are suddenly simple:
/// valstats are returned amd consumed
auto people = json::load("people.json");
if (! people.val) { cerr << "Error loading people.json: " << people.stat << endl; return ; }

auto person = people.val[1];
if (! person.val ) { cerr << "Error getting people[1]: " << person.stat << endl; return ; }

auto name = person.get<string>("Name");
if ( ! name.val ) { cerr << "Error getting people[1].Name: " << name.stat << endl; return ; }

auto legs = person.get("Legs", 2);
if ( ! legs.val ) { cerr << "Error getting people[1].Legs: " << legs.stat << endl; return ; }

cout << "Name: " << *(name.val) << ", Legs: " << *(legs.val) << endl;

That is the preffered way. No macros. Clean code.
Where do you want to go from here? I am here to help.

@dbjdbj note that there are two result classes. One is the one you mentioned, which is part of the "I want to trade efficiency for ease of use right now" document::parse API. That one must return a concrete value, because there is nothing else that could own the memory. The other is the result of document::parser::parse and returns a reference to a document owned by the parser object.
Away right now so can't do much responding, though :)

@DBJDBJ note that there are two result classes. One is the one you mentioned, which is part of the "I want to trade efficiency for ease of use right now" document::parse API. That one must return a concrete value, because there is nothing else that could own the memory. The other is the result of document::parser::parse and returns a reference to a document owned by the parser object.
Away right now so can't do much responding, though :)

I do sympathize,  but as in Indy Racing, one has to trade the comfort for the speed :)
I suggest we mail or chat,  to sort this out. Kind regards ...

We already have both of those interfaces--at any rate, it turns out different people make different choices :)

Here's the interface you probably actually care about:

json_parse - this is the existing interface. It fills in a document owned by the parser object and returns an error code. No moves occur.
parser::parse - this is the replacement. It fills in a document owned by the parser object and returns a (document&, error) pair. No moves occur.

I believe doc_ref_result is less likely to trigger your performance concerns. But given that it doesn't make doc optional, I understand why referring to it as valstat-ish might cause confusion.

if I can read those results "exceptions" are faster than "error codes"? Well, that would be a novelty for the C++ community at large :) The way(s) to measure and compare exceptions vs no exceptions are based on comparing applications built with or without. Please peruse this paper.

I'm really not trying to measure whether exceptions are faster here, and I don't need to be convinced that they are going to be slower overall :) I'm just trying to answer @lemire's performance question by measuring the overhead of the parse call, It seems like these interfaces are in the ballpark with the old ones (I don't trust the benchmarks to say much more than "ballpark" yet, especially with respect to exceptions).

OK. I've slimmed down the benchmark, particularly making the input a const padded_string to avoid the padding overhead. We're not just measuring call overhead, still, but we're getting a lot closer.
What it shows me:

parser.parse()--the equivalent of the core function json_parse--has some overhead I didn't expect, given that it does nothing more than reformat the two outputs into a single one. I'd have expected the optimizer to do a better job.
--------------------------------------------------------------------
Benchmark                          Time             CPU   Iterations
--------------------------------------------------------------------
json_parse                      47.9 ns         47.4 ns     11200000
parser_parse_error_code         54.2 ns         54.4 ns     11200000
parser_parse_exception          54.2 ns         54.4 ns     11200000

document::parse()--the equivalent of the "convenience method" build_parsed_json--and its equivalents have roughly the same extra overhead:

--------------------------------------------------------------------
Benchmark                          Time             CPU   Iterations
--------------------------------------------------------------------
build_parsed_json                147 ns          146 ns      4480000
document_parse_error_code        150 ns          150 ns      4480000
document_parse_exception         150 ns          150 ns      4480000

I'll see if I can track down this overhead; I'm guessing a non-inline function got in there somewhere.
Note: @DBJDBJ's warning stands: this is not compiled with exceptions off. All we're measuring is the difference between the error_code path and exceptions path in an executable compiled with the ability to do exceptions. This shouldn't be taken as "users who use exceptions will be just as well off as users who don't."

@lemire it was indeed non-inline functions causing the overhead:
--------------------------------------------------------------------
Benchmark                          Time             CPU   Iterations
--------------------------------------------------------------------
json_parse                      47.9 ns         47.4 ns     11200000
parser_parse_error_code         48.7 ns         48.8 ns     11200000
parser_parse_exception          47.7 ns         47.6 ns     14451613

--------------------------------------------------------------------
Benchmark                          Time             CPU   Iterations
--------------------------------------------------------------------
build_parsed_json                147 ns          146 ns      4480000
document_parse_error_code        146 ns          143 ns      4480000
document_parse_exception         146 ns          146 ns      4480000

it was indeed non-inline functions causing the overhead

I fell off my chair!!! :-)

Just cleaning up the compiler errors I introduced when adding a Google Benchmark, now.

Google Benchmark doesn't pass cmake on the Ubuuntu 18 machines. I'm unsure why--the cmake version is the obvious thing to check, but I'm running the same cmake version on my machine. It also fails on sanitize. I've disabled the google benchmark compilation on those CI machines.

Aaand that one CI machine is once again showing a perf degradation that I can't explain or really investigate. I looked back through the changes to json_parse itself, and I don't see anything that ought to have affected it. I have a feeling small variations in the compiler output (say, register names) are causing unpredicted outsized effects.

Aaand that one CI machine is once again showing a perf degradation that I can't explain or really investigate.  My machine, skylake, skylake-x and ampere all show zero performance degradation, and code inspection does not indicate any reason existing methods should be either slower or faster.
I'm calling this baked, pending review.

(appveyor is having the same problem compiling Google Benchmarks as ubuntu 18--just turned it off there.)

Regarding the parse benchmarks, I see small variations in the number of cycles but nothing that is systematic or worrisome.

The new Google dependency adds increases the build time.
Master:
 cmake ..  -DCMAKE_BUILD_TYPE=Release
 time make

 real	0m17.215s
user	0m15.788s
sys	0m0.940s

So about 17 seconds.
With this PR:
cmake ..  -DCMAKE_BUILD_TYPE=Release  -DSIMDJSON_GOOGLE_BENCHMARKS=ON
time make
real	1m19.016s
user	1m14.076s
sys	0m3.420s

So almost 5 times longer to build.

It looks like the time necessary for the initial build is the same whether one does
cmake ..  -DCMAKE_BUILD_TYPE=Release  -DSIMDJSON_GOOGLE_BENCHMARKS=ON

or
cmake ..  -DCMAKE_BUILD_TYPE=Release  -DSIMDJSON_GOOGLE_BENCHMARKS=OFF

And I think that when -DSIMDJSON_GOOGLE_BENCHMARKS=... is omitted, it defaults to OFF.

Regarding benchmark/bench_parse_call,  is there a flag to get instrumentation: cycles, branch predictions, instructions...? I find it hard to reason from a single number (running time). In this instance, we have what looks like flat numbers, but what if there were a difference...?

Regarding the code samples in the README, I think it would be prudent to have matching tests to ensure that the suggested code works (otherwise, it can be very frustrating for users).

Are the new dependencies a step forward? They multiply by five the initial build time.

That's bad. I'll think it through, but my inclination is not to include the benchmark this go around. It proves to us that we haven't lost anything, but it's not something I expect to break every time we check in. We can then either work at getting gbench build times back to normal, or using other bench libs / revert to raw benchmarks.

If one suggests code samples (in the README or elsewhere), one really ought to add corresponding tests because nothing irritates a programmer more than documentation that does match the actual software.

You are 100% correct. There are actually ways to write unit tests that run the documentation examples directly; I hate it when the README drifts from the tests, too :) I'll get one or the other in here.

it is not supported at the legacy Makefile level

I feel like the right thing to do here is to make the Makefile generated from cmake, as I think we've talked about doing before ... but it's a valid concern, and causes confusion until that time. I don't think this is the only place we do it, though.

apparently needs to be disabled in some instances in CI.

To be fair to them, this one is probably more my unfamiliarity with cmake causing me trouble debugging the issue :)

When SIMDJSON_GOOGLE_BENCHMARKS is off (either because it is unspecified or manually set to OFF), I would expect the build to be faster... maybe five times faster?

Yeah, I am surprised about that one. I'll take a look.

One thing that we really ought to have is instrumentation (cycles, instructions, misses): merely measuring the running time is not so useful.

It's definitely add-able. They've got a whole thing for custom metrics, designed for this.

I tend to want to write my own benchmarking code, and my own instrumentation... because I like or I need to go deep. You can always do the instrumentation and the fine analysis using other tools, but, again, due to the specific focus of simdjson, I think we want, we need, very specific benchmarking tools...

You really do know more about it than everyone else here combined :) The reason I tend to reach for benchmarking libs is, my understanding is benchmarking has a lot of gotchas I don't know yet, and not having correct numbers to compare to, I won't even know that I've got it wrong.
The other reason is some of the benchmark libraries automagically give you nice graphs and statistics which have been vetted by, like, statisticians, and other quality of life features like comparisons over time. I know we can build it, and I'm quite sure you already have; it's mostly down to me not writing benchmarks every day for breakfast :)

Checked something in which gets rid of the vast majority of the build time difference. cmake still takes a while (an extra 10s) due to the extra ./configure tests it does, though. @lemire make sure when you are checking OFF vs. ON times you explicitly specify -DSIMDJSON_GOOGLE_BENCHMARKS=OFF/ON, or remove CMakeCache.txt, because it remembers the last choice you made otherwise :)
I'll work through the tests when I get my next moment.

Updated the README (honestly, it looks like I had started updating the README with aspirational goals and never updated it to what it actually was :)), Added readme_examples tests as well to validate that they work.

Link to the updated README section.

I also turned the google benchmarks off by default until we decide what to do with them--you can still turn them on with -DSIMDJSON_GOOGLE_BENCHMARKS=ON

Ok. Let us merge, then... (please do so at your convenience)
I am not at all opposed to using a dependency for the benchmarks... but I will stick with the earlier statement: we need very fine/specific benchmarks.
Cheers!

Another thing I'd like to call out here: even though this modifies the README to match the API, I think it's still evolving. If we'd like a more stable README (and if the README is designed to work with the public release), I can totally go in for that and leave it as is until we feel like we've got a final 0.3 API. I'll merge for now just to get it out of mind, but changing it back is super easy and I'm willing.

I am not at all opposed to using a dependency for the benchmarks... but I will stick with the earlier statement: we need very fine/specific benchmarks.

I hear you. I only have guesses what the actual benefit of a framework would be, either--the benefit could all be illusion.
I am curious though, if you have thoughts already: what kind of benchmarks tend to run badly on Google Benchmarks or other standard benchmarking frameworks? Are you talking about microbenchmarks that try to measure the effects of individual instructions or a small series of instructions? It feels like the parse style benchmarks could be done just fine in Google Benchmarks, including the instructions/cycles/etc., but I don't think that's what you mean by fine/specific :)

I am curious though, if you have thoughts already: what kind of benchmarks tend to run badly on Google Benchmarks or other standard benchmarking frameworks? Are you talking about microbenchmarks that try to measure the effects of individual instructions or a small series of instructions? It feels like the parse style benchmarks could be done just fine in Google Benchmarks, including the instructions/cycles/etc., but I don't think that's what you mean by fine/specific :)

We need to be able to benchmark specific steps in the code. Though this sounds trivial, this makes it hard to work with external frameworks like perf, xperf, vtune... That is, we often need to instrument the code itself to know precisely what we are measuring.
We need to be able to control the number of iterations. This is often critical. If you are working with tiny inputs or large inputs, you want as few iterations as possible to deal properly with branches and cache. I frequently vary the number of iterations to see how the metrics behave. If they change with respect to the number of iterations, then I need to understand why.
We need to be able to do the aggregates we need... for some problems, an average with a standard deviation is best... often times, a minimum is best, and so forth. So you really want access to the raw numbers.
We need low-overhead measures.. That is, it is not acceptable for us to call the function many many times to make the overhead go away... because the number of iterations plays into the branch predictions and cache misses...

Regarding quality of life... I run benchmarks all over the place, often on fresh installs... I just benchmarked the earlier CMake build on an ARM server, and I got over 5 mins to build simdjson... We definitively want to be mindful of the build times. I don't mind the gap between 10s and 20s, but going from 45 s to 5 mins is the difference with "ok" and "f*cking unbearable". Note that long build times negatively impact CI latency. It also harms the quality of life of some users who would like to contribute. I wouldn't say that build times are of primary importance, but it is also something that can strongly discourage you from working on the project.

I utterly agree we have to keep sane build times to keep ourselves sane. 5 minutes is ridiculous (I hadn't run a full make after cmake and didn't realize that was happening). Note that I did fix that, and the build times are not far off now--but taking dependencies does and always will increase build time, so we have to be mindful and selective.

I've read your points, and will keep them in mind. In fact, I'mma make an issue to evaluate Google Benchmarks and remove it from the benchmark I wrote if it doesn't make the bar, and link to this.


Need to be able to designate Element to preserve whitespace when Document uses XML Parser

I use Jsoup to output XML Documents, and have come across the unfortunate need to output:
<dateline>May 20 — </dateline>
where the trailing space character is required.  I could do something ugly like wedge a zero-width space character into the text at the last character position and then replaceAll with a regular space, but having Jsoup handle this would be a much better solution.
Per this Oracle white paper on whitespace, XML elements support a reserved attribute xml:space which can be set to "default" or "preserve".
http://www.oracle.com/technetwork/articles/wang-whitespace-092897.html
Since the default value is also implementation-dependent, both values can result in preservation of whitespace unless an additional implementation-specific configuration sets the default to specifically not preserve whitespace.
I realize that Jsoup is primarily used for HTML and have seen the previous Github issue [https://github.com/jhy/jsoup/issues/276] request for finer-grained control over whitespace, so I request as a distinctly different improvement, the ability to generate XML from a Document configured with the XML Parser in a way that allows either:

Designation of the entire Document as whitespace-preserving (this may be easier and also applicable to generating the DOM from XML, not just generating XML from the DOM)

or

Designation of each Element within such a Document to be whitespace-preserving. (seems like this might be an add-on to 1)

Closing issue, as there is a pretty simple way to preserve whitespace when generating a document:
element.appendChild(new TextNode(stringWithTrailingWhitespace, baseUri));