GH Actions Fail for py v3.7

Manually ran unit testing workflow and python version 3.7 fails for both mac and linux os.  See job details here.
Reminder to look into.

I ran the workflow again and it still failed. https://github.com/NOAA-OWP/lstm/actions/runs/7830614382
@madMatchstick, It looks like the Python 3.7 workflow is accessing a different file for netCDF4 versus 3.8 and 3.9, even though it's using the same version of the package.
3.7 file: Downloading netCDF4-1.6.5.tar.gz
3.8 file:  Downloading netCDF4-1.6.5-cp38-cp38-macosx_10_9_x86_64.whl.metadata
Any idea why 3.7 would download a tarball versus 3.8 and 3.9 accessing a whl.metadata file? I don't know enough about Python to answer the question. Also tagging @jmframe, @peckhams, and @ajkhattak

@SnowHydrology - A few things,

I too have no idea why python v3.7 is pulling a tarball vs building wheel, nor do I know enough about python to answer this question :).  It's yelling about No package 'hdf5' found which I suppose we could try installing directly.
After some stack overflow trolling, this can happen with the use of actions/setup-python@v<*> which is where the ${{ matrix.python-version }} currently lives.
Know that the environment.yml specifically calls out python v3.7.  I believe this has been the case from our lstm_bmi genesis.
When moving the version matrix to the conda-incubator/setup-miniconda@v<*> block the workflow runs for v3.7 but fails for all other because of dependency python=3.7 mentioned in (3).  Removing this version dependency from the env file allows for all python versions (3.7 - 3.12) to run okay.  Where as before (via actions/setup-python@v<*>), versions 3.9+ would break, as noted in comment.
I updated some of the "reusable workflow" versions (e.g. uses: actions/checkout@v4, etc.) but don't think it's relevant to this issue.  Although we should change it to point to the latest, regardless.

See actions in my debug branch.
So, I suppose the question is, do we want to keep python pinned to v3.7?  For standalone local builds, gh workflows, ngen integration (the dependancies doc list python at version > 3.6.8), etc.?  Tagging @jmframe & @hellkite500.
Thanks!

The Python version info probably needs to be changed. see this page:
https://github.com/NOAA-OWP/ngen/wiki/Building. On UCS machines, we use
3.8.18. t-route requires 3.9. That probably is going to become a standard
soon. Nels can answer this more authoritatively.
On Thu, Feb 29, 2024 at 1:19 PM JessicaGarrett-NOAA <
@.***> wrote:

@SnowHydrology https://github.com/SnowHydrology - A few things,

I too have no idea why python v3.7 is pulling a tarball vs building
wheel, nor do I know enough about python to answer this question :). It's
yelling about No package 'hdf5' found which I suppose we could try
installing directly.
After some stack overflow trolling, this can happen with the use of
@.**<> which is where the ${{ matrix.python-version
}} currently lives.
Know that the environment.yml
https://github.com/NOAA-OWP/lstm/blob/8d33cc9d0900984833ab0f311e932c9cc595e510/environment.yml#L15
specifically calls out python v3.7. I believe this has been the case from
our lstm_bmi genesis.
When moving the version matrix to the
@.<> block the workflow runs for v3.7
but fails for all other because of dependency python=3.7 mentioned in
(3). Removing this version dependency from the env file allows for all
python versions (3.7 - 3.12) to run okay. Where as before (via
@.<>), versions 3.9+ would break, as noted in
comment
https://github.com/NOAA-OWP/lstm/blob/8d33cc9d0900984833ab0f311e932c9cc595e510/.github/workflows/test_unit_standalone.yml#L26
.
I updated some of the "reusable workflow" versions (e.g. uses:
@.***, etc.) but don't think it's relevant to this
issue. Although we should change it to point to the latest, regardless.

See actions https://github.com/NOAA-OWP/lstm/actions/runs/8100995217 in
my debug branch.
So, I suppose the question is, do we want to keep python pinned to v3.7?
For standalone local builds, gh workflows, ngen integration (the
dependancies doc list python at version > 3.6.8), etc.? Tagging @jmframe
https://github.com/jmframe & @hellkite500
https://github.com/hellkite500.
Thanks!
—
Reply to this email directly, view it on GitHub
https://github.com/NOAA-OWP/lstm/issues/40#issuecomment-1971804138, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ACA4SRO35JAIP7MMYLZUDRLYV57KHAVCNFSM6AAAAAA74H2B7GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNZRHAYDIMJTHA
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

Python 3.7 is officially EoL (https://devguide.python.org/versions/). I wouldn't invest energy into maintaining tests with it, just remove it from the matrix and start testing on a newer version instead.

I am clearly a bit behind the times.  That settles it then.  Thanks @hellkite500 & @stcui007!

I think the issue of pinning to Python 3.7 may have been because LSTM was
using some PKL (pickle) files, and these aren't guaranteed to work with
future versions of Python.  Perhaps that use of PKL files should be
reconsidered.
Best,
Scott
On Fri, Mar 1, 2024 at 9:11 AM JessicaGarrett-NOAA @.***>
wrote:

I am clearly a bit behind the times. That settles it then. Thanks
@hellkite500 https://github.com/hellkite500 & @stcui007
https://github.com/stcui007!
—
Reply to this email directly, view it on GitHub
https://github.com/NOAA-OWP/lstm/issues/40#issuecomment-1973464289, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AB2T4L6H5B2F3OFG6DY2DTLYWCSBHAVCNFSM6AAAAAA74H2B7GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNZTGQ3DIMRYHE
.
You are receiving this because you were mentioned.Message ID:
@.***>

I think the issue of pinning to Python 3.7 may have been because LSTM was using some PKL (pickle) files, and these aren't guaranteed to work with future versions of Python. Perhaps that use of PKL files should be reconsidered. Best, Scott

@peckhams - Thanks for bringing this up.  I know the serialization work was heavily based on pickle.  I'll have to test the lstm_serialization_test.py on later versions of python.  For now, this workflow only considers the "standalone" version and the bmi unit testing script.  But to your point we may have to revisit the use pickle  for loading and saving model state, etc. ... -thanks!

@@.*** @.***> - I wasn't thinking
about the serialization work, but that is also the case.  Jonathan had
saved some training data or other required data in a PKL file as well,
which imposed some version dependencies.
Best, Scott
On Tue, Mar 5, 2024 at 9:23 AM JessicaGarrett-NOAA @.***>
wrote:

I think the issue of pinning to Python 3.7 may have been because LSTM was
using some PKL (pickle) files, and these aren't guaranteed to work with
future versions of Python. Perhaps that use of PKL files should be
reconsidered. Best, Scott
@peckhams https://github.com/peckhams - Thanks for bringing this up. I
know the serialization work was heavily based on pickle. I'll have to
test the lstm_serialization_test.py
https://github.com/NOAA-OWP/lstm/blob/master/lstm/lstm_serialization_test.py
on later versions of python. For now, this workflow
https://github.com/NOAA-OWP/lstm/blob/master/.github/workflows/test_unit_standalone.yml
only considers the "standalone" version and the bmi unit testing script.
But to your point we may have to revisit the use pickle for loading and
saving model state, etc. ... -thanks!
—
Reply to this email directly, view it on GitHub
https://github.com/NOAA-OWP/lstm/issues/40#issuecomment-1979154174, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AB2T4LYBOD2EVEUT7ABDC63YWXWOTAVCNFSM6AAAAAA74H2B7GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNZZGE2TIMJXGQ
.
You are receiving this because you were mentioned.Message ID:
@.***>

Also - I just realized that neuralhydrology requires python >= v3.10 and relies strongly on pickle to save and load files (e.g. here).  Looking at the latest 3.12.2 docs too. Thinking we should be okay?

But the training done in NeuralHydrology shouldn't impact how the model is run in NextGen, right? So even if there are dependency issues and formatting issues from the NeuralHydrology training, the files that are used for the LSTM module in NextGen don't use them, right?

But the training done in NeuralHydrology shouldn't impact how the model is run in NextGen, right? So even if there are dependency issues and formatting issues from the NeuralHydrology training(, the files that are used for the LSTM module in NextGen don't use them, right?

Correct, good clarification.  Perhaps a poorly placed reference on my part.  All I was getting at was that pickle should work fine with later (3.7+) python versions :)
-thanks!

@madMatchstick, to confirm, we can remove Python 3.7 tests?

@madMatchstick, to confirm, we can remove Python 3.7 tests?

Correct.  And update the env.yml to something like python=3.10.  I'll cleanup my debug branch and submit pr. -thanks


Requirements.txt update - pynetbox==7.3.4

Providing fix on version of requirements.txt according to #134

whomever can approve and merge - others likely may benefit from the improvement when attempting to build against using this importer.

Please merge! Mine was failing until I saw I needed to update pynetbox


[dyld cache] Consider the actual size of header

The contents of the dyld_cache_header changes with each dyld version and its size.
Therefore, when analysing different versions of the dyld cache, there may be cases where incorrect values are obtained by using properties of the header that should not exist.
The actual size of the header can be obtained from dyld_cache_header.mappingOffset.
(dyld_cache_mapping_info exists after the header.)
Example: https://github.com/apple-oss-distributions/dyld/blob/65bbeed63cec73f313b1d636e63f243964725a9d/common/DyldSharedCache.cpp#L562
We would like to improve the use of header properties that seem to have been added relatively recently, using the above method to check whether they actually exist or not before using them.

The following properties also require improvement
https://github.com/p-x9/MachOKit/blob/351aaae2e77d8ca148a2086d56c698403ce3a9e3/Sources/MachOKit/DyldCache.swift#L16-L18


Stuck in login password when tryed to run as instructed.

I did (ubuntu 18.04 lts):
git clone https://github.com/collab-project/videojs-record.git
cd videojs-record/
npm install
npm run build
npm run start
when I point my browser (chrome, newest) to:
http://localhost:8080/examples/audio-video.html
It asks for login and password, neither of them known to me.
I'm not sure if it is an npm issue or videosj.
Any help would be appretiated.

that's odd, you probably already have something else running on port 8080. webpack-dev-server automatically increases the port number when 8080 is in use, so try http://localhost:8081 instead.

Perfect, this is exactly what was happening.
Thank you for your help, I was already able to test recording a video with soud.


the channels of mask pred in mask-rcnn

My English is really poor and this is the first time I post an issue on github, so please forgive me for any confuse caused by my words.
Now this is the problem, when I debug the mask r-cnn, I find out that the mask_pred from fcn_mask_head doesn't have a background class. For example, I use a dataset which have 2 classes to debug mask r-cnn, then the mask_pred output from fcn_mask_head has 2 channels; while I switch to a 5 classes dataset, and the mask_pred has 5 channels. That confuse me beause if the mask_pred don't have a background class, then the "one-hot" gt of background pixels mast be a whole-zero veator, that will never be matched by mask_pred through a softmax, because softmax will make mask_pred normalizes rather than becoming an all zero vector.
So my question is how mask r-cnn in mmdetection deal with the background class exactly?

The same question as https://github.com/open-mmlab/mmdetection/issues/8775.


first draft working retune for my crosstrek. primarily actuator delay

I was messing with the basePoints as well, bumped to about 70mph. seemed strange that the interpolation stopped at ~45mph before.
Otherwise this is a tune that is working pretty well on my crosstrek which was having a slow weave/drift on the Hwy @ 70+mph
It was also previously taking turns late (Actuator delay seemed to take care of that.)
If this tune doesn't work for you then we could try just adjusting the Actuator Delay. (0.2 was still a touch late for me.  0.15 was cutting the corner) 0.18 seemed about right.
I started with Actuator delay watching the timing on OP taking a corner/turn.
FF alone should be an open loop controller so,
Then I bumped "FF" up until it oversteered and brought it back down.
I then bumped "P" up until it oversteered and brought it back down.
I did not do a lot to "I" yet but bumping it up helped smooth out the straight line performance so far.
I also will need to try more P/I values at highway speed. I checked them to make sure it was performing well but did not do a second set of tuning for the 32m/s breakpoint I have currently.

Started Tuning with actual PID instead of PI. feels much better. not sure why it's not upstream.
I did notice the steering ratio and weight seemed off for impreza.
SR = 13:1 , weight = ~1400
still using actuator delay = .2
I'm not  sure why it was so high before.


[Clipclops] Temp codegenerated lexicon

Stacked on https://github.com/bluesky-social/social-app/pull/3754
Don't merge yet

Merging this into https://github.com/bluesky-social/social-app/pull/3754 so I have access to the types there.


using tensorflow-hub model elmo ensorflow.python.framework.errors_impl.ResourceExhaustedError

Python 3.6.10
tensorflow 1.12.0 gpu_py36he68c306_0
tensorflow-base 1.12.0 gpu_py36h8e0ae2d_0
tensorflow-gpu 1.12.0 h0d30ee6_0
tensorflow-hub 0.5.0 pypi_0 pypi cudatoolkit 9.0 h13b8566_0
cudnn 7.6.5  cuda9.0_0
NVIDIA-SMI 418.87.01 Driver Version: 418.87.01 CUDA Version: 10.1
0 Tesla V100-SXM2... Off | 00000000:18:00.0 Off | 0 | | N/A 37C P0 43W / 300W | 11MiB / 32480MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 1 Tesla V100-SXM2... Off | 00000000:3B:00.0 Off | 0 | | N/A 34C P0 42W / 300W | 11MiB / 32480MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 2 Tesla V100-SXM2... Off | 00000000:86:00.0 Off | 0 | | N/A 34C P0 43W / 300W | 11MiB / 32480MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 3 Tesla V100-SXM2... Off | 00000000:AF:00.0 Off | 0 | | N/A 38C P0 43W / 300W | 11MiB / 32480MiB | 0% Default
error: Using TensorFlow backend. /share/nishome/19930072_0/anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:523: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint8 = np.dtype([("qint8", np.int8, 1)]) /share/nishome/19930072_0/anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:524: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint8 = np.dtype([("quint8", np.uint8, 1)]) /share/nishome/19930072_0/anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint16 = np.dtype([("qint16", np.int16, 1)]) /share/nishome/19930072_0/anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint16 = np.dtype([("quint16", np.uint16, 1)]) /share/nishome/19930072_0/anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint32 = np.dtype([("qint32", np.int32, 1)]) /share/nishome/19930072_0/anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:532: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. np_resource = np.dtype([("resource", np.ubyte, 1)]) INFO:bert_bilstm_softmax.py:running bert_bilstm_softmax.py INFO:root:loading data... INFO:absl:Using /tmp/tfhub_modules to cache modules. INFO:tensorflow:Saver not created because there are no variables in the graph to restore INFO:tensorflow:Saver not created because there are no variables in the graph to restore INFO:tensorflow:Saver not created because there are no variables in the graph to restore INFO:tensorflow:Saver not created because there are no variables in the graph to restore 38 7 (2742, 38, 1024) (392, 38, 1024) (2742, 38, 2) (392, 38, 2)
Layer (type) Output Shape Param #
input_1 (InputLayer) (None, 38, 1024) 0
batch_normalization_1 (Batch (None, 38, 1024) 4096
masking_1 (Masking) (None, 38, 1024) 0
bidirectional_1 (Bidirection (None, 38,<PHONE_NUMBER>
bidirectional_2 (Bidirection (None, 38, 256) 394240
dropout_1 (Dropout) (None, 38, 256) 0
dense_1 (Dense) (None, 38, 2) 514
Total params: 1,579,522 Trainable params: 1,577,474 Non-trainable params: 2,048
Train on 2742 samples, validate on 392 samples Epoch 1/30 2020-02-28 19:42:36.337714: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA 2020-02-28 19:42:36.787351: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties: name: Tesla V100-SXM2-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53 pciBusID: 0000:18:00.0 totalMemory: 31.72GiB freeMemory: 31.41GiB 2020-02-28 19:42:36.957634: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 1 with properties: name: Tesla V100-SXM2-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53 pciBusID: 0000:3b:00.0 totalMemory: 31.72GiB freeMemory: 31.41GiB 2020-02-28 19:42:37.143501: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 2 with properties: name: Tesla V100-SXM2-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53 pciBusID: 0000:86:00.0 totalMemory: 31.72GiB freeMemory: 31.41GiB 2020-02-28 19:42:37.336031: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 3 with properties: name: Tesla V100-SXM2-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53 pciBusID: 0000:af:00.0 totalMemory: 31.72GiB freeMemory: 31.41GiB 2020-02-28 19:42:37.336143: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0, 1, 2, 3 2020-02-28 19:42:41.617032: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix: 2020-02-28 19:42:41.617079: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 1 2 3 2020-02-28 19:42:41.617090: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N Y Y Y 2020-02-28 19:42:41.617122: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1: Y N Y Y 2020-02-28 19:42:41.617132: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 2: Y Y N Y 2020-02-28 19:42:41.617158: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 3: Y Y Y N 2020-02-28 19:42:41.617526: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30472 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-32GB, pci bus id: 0000:18:00.0, compute capability: 7.0) 2020-02-28 19:42:41.634656: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 30472 MB memory) -> physical GPU (device: 1, name: Tesla V100-SXM2-32GB, pci bus id: 0000:3b:00.0, compute capability: 7.0) 2020-02-28 19:42:41.635333: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 30472 MB memory) -> physical GPU (device: 2, name: Tesla V100-SXM2-32GB, pci bus id: 0000:86:00.0, compute capability: 7.0) 2020-02-28 19:42:41.635518: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 30472 MB memory) -> physical GPU (device: 3, name: Tesla V100-SXM2-32GB, pci bus id: 0000:af:00.0, compute capability: 7.0) 2020-02-28 19:43:06.568339: W tensorflow/core/common_runtime/bfc_allocator.cc:267] Allocator (GPU_0_bfc) ran out of memory trying to allocate 8.94GiB. Current allocation summary follows. 2020-02-28 19:43:06.568442: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (256): Total Chunks: 147, Chunks in use: 145. 36.8KiB allocated for chunks. 36.2KiB in use in bin. 1.1KiB client-requested in use in bin.
2020-02-28 19:43:06.568541: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (65536): Total Chunks: 36, Chunks in use: 36. 2.27MiB allocated for chunks. 2.27MiB in use in bin. 2.25MiB client-requested in use in bin. 2020-02-28 19:43:06.568552: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (131072): Total Chunks: 18, Chunks in use: 18. 2.42MiB allocated for chunks. 2.42MiB in use in bin. 2.38MiB client-requested in use in bin. 2020-02-28 19:43:06.568564: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (262144): Total Chunks: 19, Chunks in use: 18. 5.06MiB allocated for chunks. 4.69MiB in use in bin. 4.69MiB client-requested in use in bin. 2020-02-28 19:43:06.568577: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (524288): Total Chunks: 31, Chunks in use: 28. 16.48MiB allocated for chunks. 14.59MiB in use in bin. 14.59MiB client-requested in use in bin. 2020-02-28 19:43:06.568588: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1048576): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2020-02-28 19:43:06.568600: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2097152): Total Chunks: 7, Chunks in use: 7. 14.00MiB allocated for chunks. 14.00MiB in use in bin. 14.00MiB client-requested in use in bin. 2020-02-28 19:43:06.568611: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4194304): Total Chunks: 2, Chunks in use: 1. 8.00MiB allocated for chunks. 4.00MiB in use in bin. 4.00MiB client-requested in use in bin. 2020-02-28 19:43:06.568624: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8388608): Total Chunks: 4, Chunks in use: 4. 32.00MiB allocated for chunks. 32.00MiB in use in bin. 32.00MiB client-requested in use in bin. 2020-02-28 19:43:06.568636: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16777216): Total Chunks: 5, Chunks in use: 4. 80.00MiB allocated for chunks. 64.00MiB in use in bin. 64.00MiB client-requested in use in bin. 2020-02-28 19:43:06.568648: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (33554432): Total Chunks: 1, Chunks in use: 0. 32.02MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2020-02-28 19:43:06.568661: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (67108864): Total Chunks: 5, Chunks in use: 4. 340.04MiB allocated for chunks. 256.00MiB in use in bin. 256.00MiB client-requested in use in bin. 2020-02-28 19:43:06.568670: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2020-02-28 19:43:06.568679: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (268435456): Total Chunks: 13, Chunks in use: 7. 29.24GiB allocated for chunks. 18.58GiB in use in bin. 18.58GiB client-requested in use in bin. 2020-02-28 19:43:06.568687: I tensorflow/core/common_runtime/bfc_allocator.cc:613] Bin for 8.94GiB was 256.00MiB, Chunk State: 2020-02-28 19:43:06.568701: I tensorflow/core/common_runtime/bfc_allocator.cc:619] Size: 317.98MiB | Requested Size: 317.98MiB | in_use: 0, prev: Size: 256B | Requested Size: 4B | in_use: 1, next: Size: 317.98MiB | Requested Size: 317.98MiB | in_use: 1 2020-02-28 19:43:06.568712: I tensorflow/core/common_runtime/bfc_allocator.cc:619] Size: 635.96MiB | Requested Size: 623.24MiB | in_use: 0, prev: Size: 317.98MiB | Requested Size: 317.98MiB | in_use: 1, next: Size: 635.96MiB | Requested Size: 635.96MiB | in_use: 1 2020-02-28 19:43:06.568721: I tensorflow/core/common_runtime/bfc_allocator.cc:619] Size: 1.19GiB | Requested Size: 1.17GiB | in_use: 0, prev: Size: 623.24MiB | Requested Size: 623.24MiB | in_use: 1, next: Size: 1.19GiB | Requested Size: 1.19GiB | in_use: 1 2020-02-28 19:43:06.568729: I tensorflow/core/common_runtime/bfc_allocator.cc:619] Size: 1.62GiB | Requested Size: 80.0KiB | in_use: 0, prev: Size: 8.94GiB | Requested Size: 8.94GiB | in_use: 1 2020-02-28 19:43:06.568739: I tensorflow/core/common_runtime/bfc_allocator.cc:619] Size: 2.33GiB | Requested Size: 1.43GiB | in_use: 0, prev: Size: 1.19GiB | Requested Size: 1.19GiB | in_use: 1, next: Size: 2.33GiB | Requested Size: 2.33GiB | in_use: 1 2020-02-28 19:43:06.568776: I tensorflow/core/common_runtime/bfc_allocator.cc:619] Size: 4.57GiB | Requested Size: 4.57GiB | in_use: 0, prev: Size: 2.33GiB | Requested Size: 2.33GiB | in_use: 1, next: Size: 4.57GiB | Requested Size: 4.57GiB | in_use: 1 2020-02-28 19:43:06.568791: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f566e000000 of size 1280 0x7f568efea100 of size 65536 2020-02-28 19:43:06.570830: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f568effa100 of size 65536 2020-02-28 19:43:06.570835: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f568f00a100 of size 65536 2020x7f568f3ea100 of size<PHONE_NUMBER>-02-28 19:43:06.571038: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f568f46a100 of size 65536<PHONE_NUMBER>544 totalling 2.33GiB 2020-02-28 19:43:06.571302: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size<PHONE_NUMBER> totalling 4.57GiB 2020-02-28 19:43:06.571308: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size<PHONE_NUMBER> totalling 8.94GiB 2020-02-28 19:43:06.571314: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 18.97GiB 2020-02-28 19:43:06.571322: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats: Limit:<PHONE_NUMBER>6 InUse:<PHONE_NUMBER>6 MaxInUse:<PHONE_NUMBER>6 NumAllocs: 582 MaxAllocSize:<PHONE_NUMBER>
2020-02-28 19:43:06.571344: W tensorflow/core/common_runtime/bfc_allocator.cc:271] _________________________****************************************_____ 2020-02-28 19:43:06.571376: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at conv_ops.cc:746 : Resource exhausted: OOM when allocating tensor with shape[2742,512,38,45] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc 2020-02-28 19:43:16.573348: W tensorflow/core/common_runtime/bfc_allocator.cc:267] Allocator (GPU_0_bfc) ran out of memory trying to allocate 17.49GiB. Current allocation summary follows. 2020-02-28 19:43:16.573412: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (256): Total Chunks: 147, Chunks in use: 145. 36.8KiB allocated for chunks. 36.2KiB in use in bin. 1.1KiB client-requested in use in bin. 2020-02-28 19:43:16.573425: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (512): Total Chunks: 19, Chunks in use: 17. 9.5KiB allocated for chunks. 8.5KiB in use in bin. 8.5KiB client-requested in use in bin. 2020-02-28 19:43:16.573434: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1024): Total Chunks: 2, Chunks in use: 2. 2.2KiB allocated for chunks. 2.2KiB in use in bin. 2.0KiB client-requested in use in bin. 2020-02-28 19:43:16.573443: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2048): Total Chunks: 24, Chunks in use: 22. 48.2KiB allocated for chunks. 44.0KiB in use in bin. 44.0KiB client-requested in use in bin. 2020-02-28 19:43:16.573453: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4096): Total Chunks: 15, Chunks in use: 15. 60.0KiB allocated for chunks. 60.0KiB in use in bin. 60.0KiB client-requested in use in bin. 2020-02-28 19:43:16.573468: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8192): Total Chunks: 11, Chunks in use: 6. 111.5KiB allocated for chunks. 53.5KiB in use in bin. 53.4KiB client-requested in use in bin. 2020-02-28 19:43:16.573478: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16384): Total Chunks: 1, Chunks in use: 0. 16.2KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2020-02-28 19:43:16.573486: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (32768): Total Chunks: 3, Chunks in use: 0. 96.2KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2020-02-28 19:43:16.573494: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (65536): Total Chunks: 36, Chunks in use: 36. 2.27MiB allocated for chunks. 2.27MiB in use in bin. 2.25MiB client-requested in use in bin. 2020-02-28 19:43:16.573503: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (131072): Total Chunks: 17, Chunks in use: 16. 2.24MiB allocated for chunks. 2.00MiB in use in bin. 2.00MiB client-requested in use in bin. 2020-02-28 19:43:16.573512: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (262144): Total Chunks: 18, Chunks in use: 18. 4.69MiB allocated for chunks. 4.69MiB in use in bin. 4.69MiB client-requested in use in bin. 2020-02-28 19:43:16.573520: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (524288): Total Chunks: 30, Chunks in use: 28. 15.60MiB allocated for chunks. 14.59MiB in use in bin. 14.59MiB client-requested in use in bin. 2020-02-28 19:43:16.573528: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1048576): Total Chunks: 1, Chunks in use: 0. 1.44MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2020-02-28 19:43:16.573537: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2097152): Total Chunks: 7, Chunks in use: 7. 14.00MiB allocated for chunks. 14.00MiB in use in bin. 14.00MiB client-requested in use in bin. 2020-02-28 19:43:16.573545: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4194304): Total Chunks: 2, Chunks in use: 1. 8.00MiB allocated for chunks. 4.00MiB in use in bin. 4.00MiB client-requested in use in bin. 2020-02-28 19:43:16.573553: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8388608): Total Chunks: 4, Chunks in use: 4. 32.00MiB allocated for chunks. 32.00MiB in use in bin. 32.00MiB client-requested in use in bin. 2020-02-28 19:43:16.573562: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16777216): Total Chunks: 5, Chunks in use: 4. 80.00MiB allocated for chunks. 64.00MiB in use in bin. 64.00MiB client-requested in use in bin. 2020-02-28 19:43:16.573570: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (33554432): Total Chunks: 1, Chunks in use: 0. 32.02MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2020-02-28 19:43:16.573579: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (67108864): Total Chunks: 5, Chunks in use: 4. 340.04MiB allocated for chunks. 256.00MiB in use in bin. 256.00MiB client-requested in use in bin. 2020-02-28 19:43:16.573587: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2020-02-28 19:43:16.573595: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (268435456): Total Chunks: 12, Chunks in use: 6. 29.24GiB allocated for chunks. 9.64GiB in use in bin. 9.64GiB client-requested in use in bin. 2020-02-28 19:43:16.573604: I tensorflow/core/common_runtime/bfc_allocator.cc:613] Bin for 17.49GiB was 256.00MiB, Chunk State: 2020-02-28 19:43:16.573618: I tensorflow/core/common_runtime/bfc_allocator.cc:619] Size: 317.98MiB | Requested Size: 317.98MiB | in_use: 0, prev: Size: 256B | Requested Size: 4B | in_use: 1, next: Size: 317.98MiB | Requested Size: 317.98MiB | in_use: 1 2020-02-28 19:43:16.573630: I tensorflow/core/common_runtime/bfc_allocator.cc:619] Size: 635.96MiB | Requested Size: 623.24MiB | in_use: 0, prev: Size: 317.98MiB | Requested Size: 317.98MiB | in_use: 1, next: Size: 635.96MiB | Requested Size: 635.96MiB | in_use: 1 2020-02-28 19:43:16.573641: I tensorflow/core/common_runtime/bfc_allocator.cc:619] Size: 1.19GiB | Requested Size: 1.17GiB | in_use: 0, prev: Size: 623.24MiB | Requested Size: 623.24MiB | in_use: 1, next: Size: 1.19GiB | Requested Size: 1.19GiB | in_use: 1 2020-02-28 19:43:16.573650: I tensorflow/core/common_runtime/bfc_allocator.cc:619] Size: 2.33GiB | Requested Size: 1.43GiB | in_use: 0, prev: Size: 1.19GiB | Requested Size: 1.19GiB | in_use: 1, next: Size: 2.33GiB | Requested Size: 2.33GiB | in_use: 1 2020-02-28 19:43:16.573695: I tensorflow/core/common_runtime/bfc_allocator.cc:619] Size: 4.57GiB | Requested Size: 4.57GiB | in_use: 0, prev: Size: 2.33GiB | Requested Size: 2.33GiB | in_use: 1, next: Size: 4.57GiB | Requested Size: 4.57GiB | in_use: 1 2020-02-28 19:43:16.573704: I tensorflow/core/common_runtime/bfc_allocator.cc:619] Size: 10.57GiB | Requested Size: 8.94GiB | in_use: 0, prev: Size: 4.57GiB | Requested Size: 4.57GiB | in_use: 1 2020-02-28 19:43:16.573714: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f566e000000 of size 1280 2020-02-28 19:43:16.573720: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f566e000500 of size 256
2020-02-28 19:43:16.576222: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1024 totalling 1.0KiB 666854400 totalling 635.96MiB 2020-02-28 19:43:16.576352: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size<PHONE_NUMBER> totalling 1.19GiB 2020-02-28 19:43:16.576358: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size<PHONE_NUMBER> totalling 2.33GiB 2020-02-28 19:43:16.576364: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size<PHONE_NUMBER> totalling 4.57GiB 2020-02-28 19:43:16.576371: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 10.02GiB 2020-02-28 19:43:16.576380: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats: Limit:<PHONE_NUMBER>6 InUse:<PHONE_NUMBER>0 MaxInUse:<PHONE_NUMBER>6 NumAllocs: 582 MaxAllocSize:<PHONE_NUMBER>
2020-02-28 19:43:16.576403: W tensorflow/core/common_runtime/bfc_allocator.cc:271] _____________________**********___________________________________ 2020-02-28 19:43:16.576430: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at conv_ops.cc:446 : Resource exhausted: OOM when allocating tensor with shape[2742,1024,38,44] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc Traceback (most recent call last): File "bert_bilstm_softmax.py", line 321, in model.fit(train_vectors, val_train_labels, epochs=nb_epoch,steps_per_epoch=1,validation_data=[test_vectors, val_test_labels],validation_steps=1) File "/share/nishome/19930072_0/anaconda3/envs/py36/lib/python3.6/site-packages/keras/engine/training.py", line 1039, in fit validation_steps=validation_steps) File "/share/nishome/19930072_0/anaconda3/envs/py36/lib/python3.6/site-packages/keras/engine/training_arrays.py", line 154, in fit_loop outs = f(ins) File "/share/nishome/19930072_0/anaconda3/envs/py36/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2715, in __call return self._call(inputs) File "/share/nishome/19930072_0/anaconda3/envs/py36/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2675, in _call fetched = self._callable_fn(*array_vals) File "/share/nishome/19930072_0/anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1439, in call run_metadata_ptr) File "/share/nishome/19930072_0/anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 528, in exit c_api.TF_GetCode(self.status.status)) tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[2742,512,38,45] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[{{node module_apply_default/bilm/CNN/Conv2D_5}} = Conv2D[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="VALID", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](module_apply_default/bilm/CNN/Conv2D-0-TransposeNHWCToNCHW-LayoutOptimizer, module_apply_default/bilm/CNN/Conv2D_5/ReadVariableOp/_299)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[[{{node bidirectional_2/transpose_4/_313}} = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1962_bidirectional_2/transpose_4", tensor_type=DT_BOOL, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

@DavidInWuhanChina,
Looks like the issue is related to TensorFlow-Hub. Could you please open a new issue in the Tensorflow-Hub repository from this link. Thanks!

Any updates regarding this issue? Thanks!

I have solve this problem，thanks！


Fix for difficulty comparator

ShiftDate and ShiftType comparisons are done by name and id. It make more sence to order shift assigments by RequiredEmployeeSize. It gives better results too.

Can one of the admins verify this PR? Comment with 'ok to test' to start the build.

Can one of the admins verify this PR? Comment with 'ok to test' to start the build.

ok to test

I think date/type is not true difficulty comparator as it is based on the date/type description/id.

Closing this PR because it does not improve the solution quality, at least in my benchmarks. If you have benchmarks that show the opposite, please share and let's discuss :) Feel free to reopen if you disagree.
I did improve the comment for better clarity.
http://github.com/droolsjbpm/optaplanner/commit/9595a4877


New command: spo navigation node get

Usage
m365 spo navigation node get [options]
Description
Gets information about a specific navigation node
Options

Option
Description

-u, --webUrl <webUrl>
Absolute URL of the site.

--id <id>
Id of the navigation node.

Examples
Retrieve information for a specific navigation node.
m365 spo navigation node get --id 2209

Default properties
No response
Additional Info
API request:
GET https://contoso.sharepoint.com/sites/demo/_api/web/navigation/GetNodeById(<node ID>)

looks good. 👍
just one small mistake in the usage. I think it should be get instead of list 🤔

Fixed it, thx!

Can I work on this?

All yours 👍


Neil/201 docs folder

Description
This Pr adds a github action and docs folder to allow the new docsite to render .md and .mdx from this repo
related: onflow/next-docs-v1#201

@jribbink @gregsantos @justinbarry is this PR okay to merge? It's needed for the docsite


weaviate[major],langchain[patch],community[patch]: Adds weaviate vector store integration package

@bracesproul thank you for your contribution!
It's impressive to see the inclusion of both unit and integration tests in your PR.
Could you provide more details on the current test coverage for this integration? Understanding the extent of the coverage will help us ensure the robustness of the implementation and identify any potential areas that might need additional tests

Hey @hsm207, these tests are just the same as in @langchain/community (I pretty much just copy-pasted the weaviate code into this new package).
As for the tests, the non integration test only checks the flattenObjectForWeaviate which flattens metadata for the properties field in the API requests.
The integration test tests:

similaritySearch
similaritySearch with metadata
upserting via texts
upserting via Documents
deleting
deleting with filters

Let me know if this helps, and if you need anything else!

Hey @jacoblee93, could you please let us know how do we check the test coverage in this project?
Given that this PR  is about the integration with Weaviate, it's important for us to ensure that the code meets our quality benchmarks.
Insight into the test coverage would be very helpful for identifying any areas we might need to address.

Hi @hsm207, we don't currently measure code coverage but that would be a great thing to add. Let me get back to you once it's set up!

Hey @hsm207 I'll be setting up test coverage this weekend, I'll tag you once it's in and we have a concrete number!

@hsm207 The current weaviate integration has 20% test coverage.

@bracesproul thanks for the update.
20%  is unexpectedly low 🤣
Can you share the test coverage report? I'd like to see what we're missing. Ideally, we should be closer to 90% coverage before we proceed to review.

@bracesproul thanks for the update.
20% is unexpectedly low 🤣
Can you share the test coverage report? I'd like to see what we're missing. Ideally, we should be closer to 90% coverage before we proceed to review.

Our code coverage tool doesn't have a share option, so instead I've included screenshots from the report. Let me know if you need anything else!

@bracesproul thanks for sharing the screenshots.
How do I interpret them? Does red mean the line is not covered by any tests? What about yellow?

@bracesproul thanks for sharing the screenshots.
How do I interpret them? Does red mean the line is not covered by any tests? What about yellow?

Sorry for the delay, yes the red lines are lines which codecov has determined are not tested, and yellow means partial test coverage. Let me know if you need anything else!

@bracesproul, thanks for the info.
Just had another look at the coverage report and, honestly, I'm puzzled. Even basic stuff like the constructor (line 109) is left untested. 🤔
More importantly, critical functions for the integration – like adding documents, searching, etc. – are also in the no-coverage zone. This doesn't just raise red flags; it's practically a stop sign 😝
Given this, I can't endorse merging this PR in its current state. It's not my call to make, though since I'm not a maintainer. @jacoblee93, over to you for the final decision.

@bracesproul can we fix?

@bracesproul can we fix?

Taking a look now!

Hey @hsm207 so I pushed up some more tests, however in doing so I realized our codecov percentage was so low because it was only checking the tests we run in ci (non integration tests). Most of the tests in the langchain repo are integration tests, and we don't run those in CI for various reasons. We do however run them before releasing new versions, so they end up getting run a few times a week. Running jest coverage locally gives these results (includes integration tests):

One other thing to note, we're still unable to use the default export from weaviate-ts-client, even on the latest version without casting it as any.
The error it throws it:
Property 'client' does not exist on type 'typeof import("/Users/bracesproul/code/lang-chain-ai/wt/brace/weaviate-integration-pkg/node_modules/weaviate-ts-client/dist/index")'.ts(2339)

Is there anything we can do about this?

@bracesproul I'll defer to @jacoblee93 to decide if the coverage is sufficient.

Is there anything we can do about this?

This is related to your typescript settings. I suggest to compare your tsconfig with this project and look for the difference:
https://github.com/weaviate/weaviate-breadboard-kit
this project also has a weaviate.client call but it does not throw any compile erorr. See here


quick question of table data extraction

Is this only for identifying the layout or is there an extraction framework to extract the identified tables?

The short answer is yes!

Currently, you could use the model trained on PubLayNet to detect the table regions, refer to  https://layout-parser.readthedocs.io/en/latest/notes/modelzoo.html

import layoutparser as lp
model = lp.Detectron2LayoutModel(
            config_path ='lp://PubLayNet/faster_rcnn_R_50_FPN_3x/config', # In model catalog
            label_map   ={0: "Text", 1: "Title", 2: "List", 3:"Table", 4:"Figure"}, # In model`label_map`
            extra_config=["MODEL.ROI_HEADS.SCORE_THRESH_TEST", 0.8] # Optional
        )
layout = model.detect(image)
table_regions = [ele for ele in layout if ele.type == 'Table']

We are working on some dedicated table detection models and will update our repositories very soon. Please stay tuned.

Hey ! Can we train this model ourselves? In such a way that it will have existing features and additional to detect the table format. If yes, please do let me know how?
Thank you :)

Hey ! Can we train this model ourselves? In such a way that it will have existing features and additional to detect the table format. If yes, please do let me know how?
Thank you :)

The short answer is yes - you could find more details in this repository: https://github.com/Layout-Parser/layout-model-training. And we will provide more details soon.

Also see #19 - the table region detection model trained on the docbank dataset has been added.


Error: The mode has failed while tokenizing the input

VSCode Version: code-insiders_1.14.0-1498021996_amd64.deb
OS VErsion: Debian GNU/Linux 7.11
Steps to Reproduce:
Edit any file
You should then see an error popup, and you see this on the window gaining focus:
"The mode has failed while tokenizing the input"
I have tried uninstalling and reinstalling. Have also tried the same with the standard vs code (not insiders). Have also tried the trick mentioned in other forums to no avail (sudo rm -rf /usr/share/code/resources/app/node_modules/vscode-textmate/node_modules).
Any suggestions? Many thanks

I wonder if this issue is being caused by an installed extension. Can you try to run VS Code without extensions? From the command line (NOT the integrated terminal in Code), execute: code --disable-extensions and try your steps again to see if it reproduces. If you see it is an issue with the extension, please file it against the extension repository itself.

@peterhinners Does the console show anything? (F1 > Developer: Toggle Developer Tools). Does this happen for any file type or only for a specific file type? Does this happen for any file or only a specific file?

@bpasero Thank you for the suggestion. I tried it and am still getting the error.
@alexandrudima Great suggestion. This happens for any file by the way. What I see in the console is the following (and it continues to appear repeatedly...
I believe the ENOSPC error messages in the console are a red herring, and possibly unrelated to the initial error message -- it is a known problem for our file watcher under Linux -- #7950 . I believe the solution is to increase the maximum watchers in inotify .
Once you get rid of the file watcher noise in the console, could you please check the console again, hopefully more information about the exception will be present there.

@alexandrudima Thank you for your continued help... I increased the watchers and am indeed getting a different message now:
shell.ts:439 /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.14' not found (required by /usr/share/code-insiders/resources/app/node_modules/oniguruma/build/Release/onig_scanner.node): Error: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.14' not found (required by /usr/share/code-insiders/resources/app/node_modules/oniguruma/build/Release/onig_scanner.node)
at process.module.(anonymous function) [as dlopen] (ELECTRON_ASAR.js:173:20)
at Object.Module._extensions..node (module.js:598:18)
at Object.module.(anonymous function) [as .node] (ELECTRON_ASAR.js:173:20)
at Module.load (module.js:488:32)
at tryModuleLoad (module.js:447:12)
at Function.Module._load (module.js:439:3)
at Module.require (module.js:498:17)
at n (file:///usr/share/code-insiders/resources/app/out/vs/loader.js:4:10609)
at Object. (/usr/share/code-insiders/resources/app/node_modules/oniguruma/lib/oniguruma.js:4:17)
at Object. (/usr/share/code-insiders/resources/app/node_modules/oniguruma/lib/oniguruma.js:69:4)

@peterhinners Would this help?
@Tyriar Is there a way for us to express that we need libc6 in the linux distributions ?

@alexandrudima yes and we currently do this for .rpm packages https://github.com/Microsoft/vscode/blob/master/resources/linux/rpm/dependencies.json#L54 - for .deb this is covered by https://github.com/Microsoft/vscode/issues/13089

:+1: @Tyriar I leave it up to you to decide if you want to merge this issue into #13089


Something wrong with Elixir

I try perfectly and exactly as you write but I got an issue like this..
file_get_contents(C:\xampp\htdocs\lvmaster\public/build/rev-manifest.json): failed to open stream: No such file or directory

Can you give me a shot to solve this?

Not tested thought, try this:
.version([
      'public/build/css/all.css',
      'public/build/js/all.js'
    ])
//.copy(...)
//.copy(...);

Yeah it solved the error shown when I run gulp command.
But the main problem still there.
rev-manifest.json not found.

I've come across rev-manifest.json not found a couple of times myself. Running composer install --prefer-dist has always fixed this for me.

I've tried clean install and use composer install --prefer-dist but still rev-manifest.json not generated..
Any idea?

Consider this issue comment. https://github.com/Zemke/starter-laravel-angular/issues/23#issuecomment-125245170

Ditto. Solved my issue by reinstalling npm, then installing gulp globally, running npm install laravel-elixir on the app folder, then modifying gulpfile.js as mentioned above, then finally running gulp.
Out of curiosity why not commit that change to the gulpfile, since it seems to be an issue?


Consider allowing use of Vagrant with docker provider to run and provision Drupal VM

Issue Type

Feature Idea

Summary
In #1366, we added the ability to use Drupal VM in a monolithic Docker container, completely independent of Vagrant. Vagrant itself allows the use of the Docker provider in lieu of other providers (the most commonly used for Drupal VM is currently VirtualBox...).
After we solidify the experimental Docker support, it might be a nice improvement to let Vagrant manage the building/provisioning of Docker containers, especially if it solves some of the more annoying/difficult bits, like managing the hosts file using vagrant-hostsupdater.
Tagging @thom8, who was the original suggester of such an integration.

Played around with this a bit during the weekend. A few notes:

Vagrants synced folders do not work (https://github.com/hashicorp/vagrant/issues/8021). Dockers is the way to go anyways.
vagrant-hostsupdater correctly edits the hostfile while vagrant-hostmanager doesn't (https://github.com/devopsgroup-io/vagrant-hostmanager/issues/243)
Pretty flexible as we can map the current config to docker params ala d.ports = vconfig['firewall_allowed_tcp_ports'].map { |p| "#{p}:#{p}" }
Networking is a pain on macos ofc. Really hoping for an upstream fix here as it's still sort of deal breaker for me.

Closing this out; if I go further down the Docker path, I go further down Docker alone since it's faster/easier to manage than Vagrant with Docker inside at this point, for all things but the networking setup.


PowerToys Run: Translated App Names Wouldn't Be Found Anymore

Microsoft PowerToys version
v0.63.0
Running as admin

[X] Yes

Area(s) with issue?
PowerToys Run
Steps to reproduce
I use Windows in brazilian portuguese. In previous versions of the software, I was able to open Notepad by searching for de string "Bloco de Notas" or just a first slice of it.
✔ Expected Behavior
Find Notepad aplication with the portuguese traslated name as "Bloco de Notas".
❌ Actual Behavior
Isn't found. It turns out that asking in English for "Notepad" it is working instead as if the software was renamed.
Other Software
No response

@baboseiro
Can you please close PT and open it again. Sometimes the translation isn't working correctly.


Fixing errors importing FHIR resources from NDJSON file containing empty lines

Resolves #298

This has been merged into release/4.0.0.


Add net7 to testing matrix

Description
Add net7 into the testing framework to ensure compatibility.
Signed-off-by: halspang<EMAIL_ADDRESS>Issue reference
We strive to have all PR being opened based on an issue, where the problem or feature have been discussed prior to implementation.
Please reference the issue this PR will close: NA
Checklist
Please make sure you've completed the relevant tasks for this PR, out of the following list:

[x] Code compiles correctly
[x] Created/updated tests
[ ] Extended the documentation

You should be able to also remove .NET 5 from your testing matrix as it was EOL in May 2022. This is not a breaking change since it doesn't break anything for a .NET 5 user. We'll just stop testing it.


⚠ Bofrost.be has degraded performance

In e47a748, Bofrost.be (https://www.bofrost.be) experienced degraded performance:

HTTP code: 200
Response time: 1896 ms

Resolved: Bofrost.be performance has improved in 19506eb.


Signed binaries

Would it be feasible to extend cargo-auditable to signed binaries ?
I would love to sign all the information and then use auditable also to check that the auditable information was signed correctly.
https://rust-lang.zulipchat.com/#narrow/stream/146229-wg-secure-code/topic/Signed.20binaries/near/293060631

You can sign a binary after running cargo auditable the same way you sign a regular binary for the respective OS, right?

Correct you could post-auditable.
I was additionally thinking that auditable could provide a mechanism to inject a signatures of the dependencies.
Nonetheless I would be keen to have builder to have it's own signing information in auditable format as well.

Dependencies don't have signatures, right?

Not currently but those still get linked. Especially in repeatable builds one would expect to get the same dependency.

You mean including a hash of all compiled rlibs that are linked into the executable?

Yep.

Binaries compiled with cargo auditable certainly benefit from whole-binary signing, since it also makes the audit info impossible to tamper with after the binary is compiled.
I'm not sure how this proposal is different from regular binary signing, and why should it be handled by cargo auditable instead of a separate tool. Could you elaborate?

in zulip also Q came up:

what's the use case? if the intent is to publish the binary, it might be interesting to use something like Sigstore

My A:
yeah publish.
I would prefer it in the binary instead of separate signature text file that ~never gets checked due to convenience factor cargo-auditable and tooling could provide it.
The tooling would just provide convenience to both insert and check the opinionated checksums without separate sign & check -
I could just tell the users just to use to verify it came signed auditable with our sig with audit - also @tarcieri Sigstore sounds nice - does it support multisig flow or smth ?

Embedding the signature into the binary in a way that leaves the binary easy to check is not easy. You are literally modifying the data you wanted to sign after you signed it. The procedure to revert the injection of the signature data is both security-critical and non-trivial.
I agree that it would synergize nicely with cargo auditable, but that project is sufficiently non-trivial to deserve its own tool. I'm going to close this as out of scope for cargo auditable, but you're welcome to get a repo for cargo sig going.

I also proposed that the dependencies in reproducible builds could be signed separately not just the whole binary.


Add missing dependency on generated header files

Specifically the GetSerial.h is generated from the srv file and is
needed by astra_driver.h. Without this there is nothing to ensure that
GetSerial.h is generated before the code that uses it gets compiled and
can fail as follows:
Errors     << astra_camera:make .../build/logs/astra_camera/build.make.000.log
In file included from
.../build/src/ros_astra_camera/src/astra_device.cpp:39:0:
.../build/src/ros_astra_camera/include/astra_camera/astra_driver.h:55:36:
fatal error: astra_camera/GetSerial.h: No such file or directory

It looks like another fix was added for the same issue a week after I pushed this request so I'm going to close it as the issue has been fixed.


Add operation "Extract EXIF"

This aims to close #65.
It uses the npm module exif-parser for the heavy lifting of actually getting the EXIF data.
Screenshot:

Excellent work. It would be nice to have an operation that strips EXIF data from files as well. I'll create a new issue.


[Feature][python] Slim dist package by migrate python gatewar into apiserver

Search before asking

[X] I had searched in the issues and found no similar feature requirement.

Description
For now, our distribution package is more than 700 MB, with too big distribute package make our users difficult to download to deploy service from one machine to another. We find out python gateway server libs is the subset of API server libs, and also python gateway server is another kind of API service, So I think we should migrate the python gateway server into api server.
Please make sure we have to change our doc and release note about this change
Use case
No response
Related issues
No response
Are you willing to submit a PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Closed by #9372


Write an about section for this repository

I'll leave this to anyone that wants to take a shot at it. Should include something about OGPC 2023 and summarize our game in under 3 sentences.

Done as of today. Will update it upon feedback.


Add eslint check to the pipeline

We should not be able to deploy code that does not work. We need to make sure automated checks are run every time we want to deploy something.
-> issue on v3 only

this is a technical issue, maybe @fernandolucchesi can take a look on this before i merge it to main :)


on IOS the native call returns "Not Flight Mode" and a late initialization error occurs

see this code snippet:

you fixed this in https://github.com/14h4i/airplane_mode_checker/pull/3, but is the pub.dev version upgraded?

you fixed this in #3, but is the pub.dev version upgraded?

Sorry for the slow response, I updated the package to a new version

Sorry for the slow response, I updated the package to a new version. I will close this issue.


Made black ones appear on grey lines #265

z-index attribute of the grey lines fixed this issue.

@therealryan  is this the way to raise a PR

Ok ill change and will let you know when done

pr #276

The sonar test is failing, but that's because the fork doesn't have access to the auth token for the upload to sonar. It'd beggar belief that this failure was masking sonar issues in the change.

Congratulations to @Chaitu-Tatipamula for being the first external contributor to this project!
🥇

Thank you..😀

@therealryan hey can you tell me what this open source is i kinda like this kind of fixing issues but does it really help me for my future career like direct hiring if i really contribute to great projects

@Chaitu-Tatipamula While I can't really offer useful career advice, it seems to me that having a public history of self-motivated exposure to a bunch of different technologies, problem solving, etc, can only be a good thing.

@therealryan thanks for the information ☺


Tumblr subdomain takeover

Tumblr is vulnerable to subdomain takeover, please reference Issue #240

Thank you, @diophant0x!


path planner for this package

I am trying to create a 3d path planner using elevation mapping and was wondering if there are any packages you use or suggest for that purpose.

Do you plan to do full 3d planning with drones? or do with ground robot?

yeah i have a robot that can fly and also do mobile navigation and i am planning to add a factor which focus on more mobile navigation and use aerial capability when surface is non traversable


Set Environment Properly for Execute Resources & Adjust for new Cask Behavior

Fixes #85

LGTM, brew install brew-cask is now causing errors because the package was blanked.

@tas50 Any reason in particular we didn't merge this in with the other fix before?

This one simply slipped by.  I'm prepping a release now


[Python] LLaMA/Vicuna 7B: Different results with iree-run-module and Python for local-sync vs local-task

What happened?
The python binding and iree-run-module is giving different results for the same module. If we pass device=local-sync instead of local-task then it's giving correct result for both.
Steps to reproduce your issue

Download the .mlir from https://storage.googleapis.com/shark-public/prashant/traced_vicuna/vicuna_fp32.mlir
Compile the file with iree-compile --iree-input-type=none --iree-hal-target-backends=llvm-cpu --iree-stream-resource-index-bits=64 --iree-vm-target-index-bits=64   vicuna_fp32.mlir -o vicuna_fp32_cpu.vmfb or directly download the compiled .vmfb from https://storage.googleapis.com/shark-public/prashant/traced_vicuna/vicuna_fp32_cpu.vmfb
Download the inputs: inp1: https://storage.googleapis.com/shark-public/prashant/traced_vicuna/inp1.npy and inp2: https://storage.googleapis.com/shark-public/prashant/traced_vicuna/inp2.npy
Run via iree-module: iree-run-module --device=local-task --function=forward<EMAIL_ADDRESS><EMAIL_ADDRESS> --module=vicuna_fp32_cpu.vmfb
Run via python file present here: https://gist.github.com/pashu123/7f6ac404e16df6e5f5ee69904a06e172 (Make sure .vmfb and inputs.npy is present in the working directory).

What component(s) does this issue relate to?
Python
Version information
No response
Additional context
No response

@stellaraccident Do you know if there are some hard coded limits for large models in Python ? It seems to work with local-sync but  local-task is broken.

@stellaraccident bumping up this question for you.

Also seeing this on OPT-1.3b causalLM in SHARK (scripts)
(Though with the minimal reproducer here I don't get zero outputs anymore, there is still a large numerical discrepancy between the CLI and python results). I can post in a separate issue if needed.
First, download OPT1-3b_causallm_torch.mlir and opt_inputs.npy from google storage.
Next, compile the .vmfb via iree-compile CLI:
$ iree-compile   --iree-hal-target-backends=llvm-cpu   --iree-llvmcpu-target-cpu-features=host   --iree-input-type=tm_tensor  ./OPT1-3b_causallm_torch.mlir -o opt_1-3b_causallm_cpu.vmfb

Then, compare outputs.
 I'll try with the compile/runtime flags.
For clarification:
Python produces this result on local-sync: (non-truncated -> local_sync_opt_logits.txt)
local-sync result: 7x50272=
[[ 2.9483373  14.57561     9.309269   ...  3.8072436   2.736043
  -0.48549324]
 [ 2.8499205  15.748621    7.3167763  ...  3.9052784   3.8763328
   0.42264798]
 [ 2.8242998  17.309875    7.2879767  ...  3.4833791   3.7424765
  -0.1637713 ]
 ...
 [ 2.660748   18.589664    5.18831    ...  4.1783724   3.6274369
  -0.9794496 ]
 [ 1.9488358  22.832752    7.8331327  ...  3.781442    3.3019094
   0.1572762 ]
 [ 2.2833219  14.766316    5.9369116  ...  2.6796591   3.9609249
  -0.6819936 ]]

Python produces this result on local-task: (non-truncated ->local_task_opt_logits.txt)
local-task result: 7x50272=
[[ 3.194312   14.64125     9.192374   ...  3.8184016   2.64964
  -0.5783603 ]
 [ 3.0902762  15.804577    7.2782545  ...  3.9580245   3.7676458
   0.40852466]
 [ 3.0401552  17.358082    7.2172465  ...  3.5345457   3.6020813
  -0.20694567]
 ...
 [ 2.854957   18.644693    5.1286902  ...  4.240262    3.4892058
  -1.0118058 ]
 [ 2.166676   22.891552    7.766478   ...  3.8540208   3.1496656
   0.1291914 ]
 [ 2.5119016  14.818969    5.8827424  ...  2.7389405   3.8110926
  -0.7152195 ]]

the local-sync result appears the same as the output of iree-run-module (with --device=local-task) given above.
The result from pytorch that we hope to validate both against:
golden_out.txt

ok cool so nothing produces the right results? (that makes it easier to diagnose as we can ignore that it's different between them and focus on where the compiler is having issues)

In the case of OPT-1.3b, yes. I believe LLaMa/vicuna may be a different case.

Skimming, is it safe to assume that this is just broken in all configurations tested? (i.e. none of python, local-sync, local-task are producing correct results)

Skimming, is it safe to assume that this is just broken in all configurations tested? (i.e. none of python, local-sync, local-task are producing correct results)

I'm actually hoping we move the OPT numerics discussion elsewhere as these may not be the same issue and I don't want to lose track of the LLaMa/vicuna numerics if the problem there is different. @powderluv do you know more about the LLaMa issue? Do our local-sync numerics validate against pytorch for vicuna?

Two things I'd try are passing --iree-stream-partitioning-favor=debug to iree-compile (should disable the compiler's attempts at concurrently issuing work even when running with local-task) and passing --task_topology_group_count=1 to the runtime to make local-task single-threaded (still may have some behavioral differences but rules out threads stomping on each other and limits it to ordering).

These two flags didn't seem to affect the local-task results from python.
Here is the modified script: gist

Have you tried non-llvm-cpu backends (CUDA?)? knowing that may isolate it to HLO/linalg/etc vs the particular platform codegen vs the runtime.

CUDA works! (I'll show the pytorch vs. SHARK python numerics at-a-glance):
SHARK logits have shape:  (7, 50272) : [[  1.8319876    1.5199093    6.082119   ...   2.109685     1.9062941
    1.9811553 ]
 [-12.1053505  -11.442817     0.08139505 ... -10.795685   -11.532479
  -11.433617  ]
 [-12.078147   -12.59089      2.0352333  ... -11.382179   -11.428994
  -11.703665  ]
 ...
 [ -7.102992    -7.6976204    4.072431   ...  -7.376941    -6.719622
   -7.269612  ]
 [-11.234247   -11.439148    -6.903964   ... -11.79036    -11.258774
  -11.402507  ]
 [ -2.255033    -2.3394425   16.018326   ...  -1.0540967   -2.1388423
   -1.4458579 ]]
PyTorch logits have shape: torch.Size([7, 50272]) : tensor([[  1.8320,   1.5199,   6.0821,  ...,   2.1097,   1.9063,   1.9812],
        [-12.1054, -11.4428,   0.0814,  ..., -10.7957, -11.5325, -11.4336],
        [-12.0782, -12.5909,   2.0352,  ..., -11.3822, -11.4291, -11.7037],
        ...,
        [ -7.1030,  -7.6976,   4.0724,  ...,  -7.3770,  -6.7196,  -7.2696],
        [-11.2342, -11.4391,  -6.9040,  ..., -11.7903, -11.2587, -11.4024],
        [ -2.2550,  -2.3394,  16.0183,  ...,  -1.0541,  -2.1388,  -1.4458]],
       grad_fn=<MmBackward0>)

Well that's a change, GPU works and CPU doesn't. And this is all F32?

just to summarize (@pashu123 can correct if wrong) lets ignore the OPT 1.3B issue for now
For Vicuna/LLaMA:
Pytorch, IREE Python local-sync, CUDA match in FP32.  Just local-task fails.  So can't be a overflow in Python land since the CUDA and local-sync work ok.

just to summarize (@pashu123 can correct if wrong) lets ignore the OPT 1.3B issue for now
For Vicuna/LLaMA: Pytorch, IREE Python local-sync, CUDA match in FP32. Just local-task fails. So can't be an overflow in Python land since the CUDA and local-sync work ok.

Also, local-task works fine via iree-run-module and outputs right results but gives wrong result via the python's local-task.

There isn't much different from the Python API level between the working/not-working cases...

I think the OPT issue is what we should tackle first - having any correctness issues like that scares me and would explain all kinds of other failures (if uninitialized memory is being read then what's in the uninitialized memory that is read will vary between local-sync and local-task, whether python/the long-lived process is recycling memory instead of using 0-initialized OS allocs, etc).

@stellaraccident @jpienaar Looking to get this triaged and understand the priority here. Do we need immediate assignment or can this wait?
@powderluv let us know your needs as well.

OK, after cleaning up and trying to run OPT with iree-run-module CLI, I was able to get correct results for local-task.
But loading the flatbuffer into the iree-runtime python context produces zero output just like the llama/vicuna case.
This means we can rule out any underlying correctness issues specific to (or exposed by) OPT and probably focus entirely on why the flatbuffer results are getting zeroed out. @MaheshRavishankar mentioned that this may be a garbage collection issue.
For clarity, this issue seems to encapsulate the remaining problems with opt1.3b.

@monorimet can you also attach your OPT 1.3B failure case here. It will be easier to debug with a 1.3B parameter model than the 7B in the first comment.

I have uploaded a gist for a minimal reproducer of the local-task zeros issue with the OPT1.3B model in python: OPT_1.3B_zeroes_case.py
To run the script you will want the following files in the script directory:

opt-1.3b_causallm_64_torch_cpu.vmfb -- google storage link
This .vmfb can also be generated via the following iree-compile command (with this .mlir file)
iree-compile   --iree-hal-target-backends=llvm-cpu   --iree-llvmcpu-target-cpu-features=host   --iree-input-type=tm_tensor  ./opt-1.3b_causallm_64_torch.mlir -o opt-1.3b_causallm_64_torch_cpu.vmfb

model_inputs_0.npy and model_inputs_1.npy

In my sentence completion script I can work around this (for example) by running iree-run-module via the CLI.
It also works if I run in python via local-sync (can be tested on OPT_1.3B_zeroes_case.py by changing to local-sync)
Here's what I'm doing to run in a subprocess in case it's relevant for debug:
    inputs = (
        model_inputs["input_ids"],
        model_inputs["attention_mask"],
    )
    np.save("model_inputs_0.npy", inputs[0])
    np.save("model_inputs_1.npy", inputs[1])
    #output = shark_model("forward", inputs)
    cmd = [
        "iree-run-module",
        f"--module={vmfb_path}",
<EMAIL_ADDRESS><EMAIL_ADDRESS>        f"--device=local-task",
<EMAIL_ADDRESS><EMAIL_ADDRESS>    ]
    run_cmd(cmd)
    output = torch.FloatTensor(np.load("model_outputs_0.npy"))

Ok, I have successfully repro'd on my mac.

The plot thickens. In an attempt to shorten the memory thrash on my poor laptop, I switched to mmap:
import mmap

vmfb_file = open(os.path.join("vicuna_fp32_cpu_local.vmfb"), "r+b")
mf = mmap.mmap(vmfb_file.fileno(), 0)
flatbuffer_blob = mf

# print("Loading")
# with open(os.path.join("vicuna_fp32_cpu_local.vmfb"), "rb") as mlir_file:
#     flatbuffer_blob = mlir_file.read()

... and it gets the right answers.

Ok, I've redone the patch (see above) and tested it comprehensively on Windows. It should now be impossible to use the Python API in a way that violates alignment constraints, but the existing from_flatbuffer will now make an extra copy if necessary (and issue a warning).
I expect that most such uses that are coming from files anyway should just use VmModule.mmap. But you can also use VmModule.copy_buffer (to unconditionally make a copy) or VmModule.wrap_buffer if you've been (somehow) able to guarantee that it is aligned (i.e. if you mmap'd it yourself or something).
I'm not sure that I can make it more automatic than that: these exceptionally large modules must be used in a way that reduces loading thrash, and the default way folks have been doing it (i.e. read from a file) internally involves multiple copies anyway. I found this to carry large systemic costs no matter what, so I didn't feel too badly about at least making an aligned copy.
If using one of these in-memory APIs (vs mapping), you'll want to make sure that you del any references to the backing buffer to limit the possibility of having committed memory for multiple copies.


bug: ERR_CLEARTEXT_NOT_PERMITTED livereload Android api 28

Bug Report
Ionic version:
[x] 4.12.0
Android API version
28 (Android 9.0)
Cordova Version
9.0.0
Current behavior:
the command ionic cordova run android -l deploys on my emulator but I have an alert :
Application Error
net::ERR_CLEARTEXT_NOT_PERMITTED
(http://<IP_ADDRESS>:8100)

and the application does not launch
If I deploy without the livereload option, it works fine
Expected behavior:
No error
Steps to reproduce:

create a new blank app
try to deploy it with livereload option on an emulator with Android 9.0

ionic cordova run android -l

Other information:
I have already tried to add the attribut android:usesCleartextTraffic="true"  to the  tag in the config.xml (and it apears in the manifest.xml) but it did not fix my problem
Also tried to add
<IP_ADDRESS>
to the network_security_config.xml but the file is overwritten so I can not say if it would work
=> Maybe the tag LOCAL_IP could be added to the network_security_config.xml durring build
Ionic info:
Ionic:
ionic (Ionic CLI)             : 4.12.0 (C:\Users\Jérôme\AppData\Roaming\npm\node_modules\ionic)
Ionic Framework               : @ionic/angular 4.4.0
@angular-devkit/build-angular : 0.13.9
@angular-devkit/schematics    : 7.3.9
@angular/cli                  : 7.3.9
@ionic/angular-toolkit        : 1.5.1
Cordova:
cordova (Cordova CLI) : 9.0.0<EMAIL_ADDRESS>Cordova Platforms     : android 8.0.0
Cordova Plugins       : cordova-plugin-ionic-keyboard 2.1.3, cordova-plugin-ionic-webview 4.0.1, (and 4 other plugins)
System:
NodeJS : v12.2.0 (C:\Program Files\nodejs\node.exe)
npm    : 6.9.0
OS     : Windows 10

Hi there,
Thanks for the issue. I am going to close this as this is not an Ionic issue. Ionic Framework does not deal with the webview when running in-app. This is most likely a configuration issue or an issue with the ionic webview plugin (which is a different repo).
For further help debugging this issue, feel free to post on our forum or our slack.
Thanks for using Ionic!


Can't load model in offline GPU demo

Hi @Slyne,
Thanks a lot for your work.
I followed the demo in runtime/server/x86_gpu but got an error while starting the server.
Could you offer some advice?
Log
== Triton Inference Server ==
NVIDIA Release 21.10 (build 28453983)
Copyright (c) 2018-2021, NVIDIA CORPORATION & AFFILIATES.  All rights reserved.
Various files include modifications (c) NVIDIA CORPORATION.  All rights reserved.
This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license
NOTE: Legacy NVIDIA Driver detected.  Compatibility mode ENABLED.
mkdir: cannot create directory '/ws/model_repo/attention_rescoring/1': File exists
I0104 07:43:00.874036 60 metrics.cc:298] Collecting metrics for GPU 0: Tesla V100-SXM2-16GB
I0104 07:43:02.456442 60 libtorch.cc:1092] TRITONBACKEND_Initialize: pytorch
I0104 07:43:02.456502 60 libtorch.cc:1102] Triton TRITONBACKEND API version: 1.6
I0104 07:43:02.456514 60 libtorch.cc:1108] 'pytorch' TRITONBACKEND API version: 1.6
2022-01-04 07:43:03.810494: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
I0104 07:43:04.208671 60 tensorflow.cc:2170] TRITONBACKEND_Initialize: tensorflow
I0104 07:43:04.208747 60 tensorflow.cc:2180] Triton TRITONBACKEND API version: 1.6
I0104 07:43:04.208773 60 tensorflow.cc:2186] 'tensorflow' TRITONBACKEND API version: 1.6
I0104 07:43:04.208786 60 tensorflow.cc:2210] backend configuration:
{}
I0104 07:43:04.261060 60 onnxruntime.cc:1999] TRITONBACKEND_Initialize: onnxruntime
I0104 07:43:04.262061 60 onnxruntime.cc:2009] Triton TRITONBACKEND API version: 1.6
I0104 07:43:04.262548 60 onnxruntime.cc:2015] 'onnxruntime' TRITONBACKEND API version: 1.6
I0104 07:43:04.426943 60 openvino.cc:1193] TRITONBACKEND_Initialize: openvino
I0104 07:43:04.427015 60 openvino.cc:1203] Triton TRITONBACKEND API version: 1.6
I0104 07:43:04.427036 60 openvino.cc:1209] 'openvino' TRITONBACKEND API version: 1.6
I0104 07:43:04.912607 60 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7efbbc000000' with size 268435456
I0104 07:43:04.922233 60 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864
I0104 07:43:04.945550 60 model_repository_manager.cc:1022] loading: decoder:1
I0104 07:43:05.046060 60 model_repository_manager.cc:1022] loading: feature_extractor:1
I0104 07:43:05.047111 60 onnxruntime.cc:2058] TRITONBACKEND_ModelInitialize: decoder (version 1)
I0104 07:43:05.060228 60 onnxruntime.cc:2101] TRITONBACKEND_ModelInstanceInitialize: decoder_0_0 (GPU device 0)
I0104 07:43:05.147370 60 model_repository_manager.cc:1022] loading: encoder:1
I0104 07:43:05.248009 60 model_repository_manager.cc:1022] loading: scoring:1
I0104 07:43:27.458409 60 onnxruntime.cc:2101] TRITONBACKEND_ModelInstanceInitialize: decoder_0_1 (GPU device 0)
I0104 07:43:28.442748 60 onnxruntime.cc:2058] TRITONBACKEND_ModelInitialize: encoder (version 1)
I0104 07:43:28.444939 60 model_repository_manager.cc:1183] successfully loaded 'decoder' version 1
I0104 07:43:28.445175 60 python.cc:1875] TRITONBACKEND_ModelInstanceInitialize: scoring_0_0 (CPU device 0)
Initialized Rescoring!
I0104 07:43:29.382147 60 python.cc:1875] TRITONBACKEND_ModelInstanceInitialize: feature_extractor_0_0 (GPU device 0)
I0104 07:43:52.319210 60 onnxruntime.cc:2101] TRITONBACKEND_ModelInstanceInitialize: encoder_0_0 (GPU device 0)
I0104 07:43:54.352376 60 onnxruntime.cc:2135] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I0104 07:43:54.352484 60 python.cc:1875] TRITONBACKEND_ModelInstanceInitialize: scoring_0_1 (CPU device 0)
I0104 07:43:54.352766 60 onnxruntime.cc:2081] TRITONBACKEND_ModelFinalize: delete model state
E0104 07:43:54.352834 60 model_repository_manager.cc:1186] failed to load 'encoder' version 1: Invalid argument: model 'encoder', tensor 'encoder_out': the model expects 3 dimensions (shape [-1,-1,512]) but the model configuration specifies 3 dimensions (an initial batch dimension because max_batch_size > 0 followed by the explicit tensor shape, making complete shape [-1,-1,-1])
Initialized Rescoring!
...

Resolved.
The wrong output shape is provided.
It may be caused by convert.py

Resolved.
The wrong output shape is provided. It may be caused by convert.py

@yzmyyff   Thank you. Could you provide the link to a train.yaml/config.yaml ?
I found in some model, the last dim of encoder_out is -1, which is strange. It should be the true output size. And in your case, it should be 512.
I had a comment here: https://github.com/wenet-e2e/wenet/blob/main/runtime/server/x86_gpu/scripts/convert.py#L82

Sure, here are these two files https://gist.github.com/yzmyyff/36fff2a3be9b57870034b0787e70f5ee
How about raising a warning if no shape is provided.

Sure, here are these two files https://gist.github.com/yzmyyff/36fff2a3be9b57870034b0787e70f5ee
How about raising a warning if no shape is provided.

It'll be better to extract the output shape from onnx model instead of the config file. I'll submit a pr to fix it.
Thank you again!

This bug still exists in the current main branch
how to fix it？

This bug still exists in the current main branch how to fix it？

What is your pytorch version ?  If it's above 11.1,  then please down grade it to 1.10 or follow this issue: https://github.com/pytorch/pytorch/issues/71408


Slow performance and high CPU usage  on ng serve and ng serve --aot

When run "ng serve" or "ng serve --aot" CPU usage is upper 100% and performance is so bad.
Recompiling take about 3 minutes on reload page.

environment
Angular CLI: 6.1.1
Node: 10.7.0
OS: darwin x64
Angular: 6.1.0
... animations, common, compiler, compiler-cli, core, forms
... http, language-service, platform-browser
... platform-browser-dynamic, platform-server, router

Package                           Version
-----------------------------------------------------------
@angular-devkit/architect         0.6.8
@angular-devkit/build-angular     0.6.8
@angular-devkit/build-optimizer   0.6.8
@angular-devkit/core              0.6.8
@angular-devkit/schematics        0.7.1
@angular/cli                      6.1.1
@ngtools/webpack                  6.0.8
@schematics/angular               0.7.1
@schematics/update                0.7.1
rxjs                              6.2.2
typescript                        2.7.2
webpack                           4.8.3

Most likely cause is that fsevents is not present within the project.  This should have been installed automatically.  fsevents is used to provide OS-level file watching support on MacOS.  Without it polling is used which is CPU intensive.  To confirm please run npm ls fsevents within the project.

@clydin  I've run  npm ls fsevents and get the following
MacBook-Pro-de-Esteban:apex-angular estebanrodriguez$ npm ls fsevents
<EMAIL_ADDRESS>/Users/estebanrodriguez/Documents/htdocs/newvesti/apex-angular
├─┬<EMAIL_ADDRESS>│ └─┬<EMAIL_ADDRESS>│   └──<EMAIL_ADDRESS>deduped
├──<EMAIL_ADDRESS>└─┬<EMAIL_ADDRESS>  └─┬<EMAIL_ADDRESS>    └──<EMAIL_ADDRESS>deduped

my package.json is this:
{
  "name": "apex",
  "version": "5.1.0",
  "license": "",
  "scripts": {
    "ng": "ng",
    "start": "ng serve",
    "build": "ng build",
    "test": "ng test",
    "lint": "ng lint",
    "e2e": "ng e2e"
  },
  "private": true,
  "dependencies": {
    "@agm/core": "^1.0.0-beta.3",
    "@angular/animations": "^6.0.4",
    "@angular/common": "^6.0.4",
    "@angular/compiler": "^6.0.4",
    "@angular/core": "^6.1.0",
    "@angular/forms": "^6.0.4",
    "@angular/http": "^6.0.4",
    "@angular/platform-browser": "^6.0.4",
    "@angular/platform-browser-dynamic": "^6.0.4",
    "@angular/platform-server": "^6.0.4",
    "@angular/router": "^6.0.4",
    "@ng-bootstrap/ng-bootstrap": "^2.0.0",
    "@ngrx/store": "^6.0.1",
    "@ngx-translate/core": "^10.0.2",
    "@ngx-translate/http-loader": "^3.0.1",
    "@swimlane/ngx-charts": "^8.0.2",
    "@swimlane/ngx-datatable": "^13.0.1",
    "@types/chartist": "^0.9.40",
    "@types/d3-shape": "^1.2.3",
    "@types/jquery": "^3.3.2",
    "angular-calendar": "^0.25.2",
    "bootstrap": "4.0.0",
    "chartist": "^0.11.0",
    "classlist.js": "^1.1.20150312",
    "core-js": "^2.5.7",
    "d3": "^5.4.0",
    "datatables.net": "^1.10.15",
    "fsevents": "^1.2.4",
    "intl": "^1.2.5",
    "jquery": "^3.2.1",
    "ng-chartist": "^1.1.1",
    "ng2-charts": "^1.6.0",
    "ng2-dragula": "^1.5.0",
    "ng2-file-upload": "^1.3.0",
    "ng2-img-cropper": "^0.9.0",
    "ng2-nouislider": "^1.7.11",
    "ng2-smart-table": "1.2.2",
    "ng2-validation": "^4.2.0",
    "ngx-chips": "^1.9.2",
    "ngx-quill": "^3.0.0",
    "ngx-toastr": "^8.7.3",
    "ngx-ui-switch": "^6.0.3",
    "node-sass": "^4.9.0",
    "nouislider": "^11.1.0",
    "resize-observer-polyfill": "^1.5.0",
    "rxjs": "^6.2.0",
    "screenfull": "^3.3.1",
    "sweetalert2": "^7.22.0",
    "videogular2": "^6.2.0",
    "web-animations-js": "^2.2.5",
    "zone.js": "^0.8.26"
  },
  "devDependencies": {
    "@angular-devkit/build-angular": "^0.6.8",
    "@angular/cli": "^6.1.1",
    "@angular/compiler-cli": "^6.0.4",
    "@angular/language-service": "^6.0.4",
    "@types/core-js": "^2.5.0",
    "@types/jasmine": "^2.8.8",
    "@types/node": "^10.3.1",
    "codelyzer": "^4.3.0",
    "jasmine-core": "^3.1.0",
    "jasmine-spec-reporter": "^4.2.1",
    "karma": "^2.0.2",
    "karma-chrome-launcher": "^2.2.0",
    "karma-cli": "~1.0.1",
    "karma-coverage-istanbul-reporter": "^2.0.1",
    "karma-jasmine": "^1.1.2",
    "karma-jasmine-html-reporter": "^1.1.0",
    "protractor": "^5.3.2",
    "rxjs-compat": "^6.2.0",
    "ts-node": "^6.1.0",
    "tslint": "^5.10.0",
    "typescript": "^2.7.2"
  }
}

Is the application constantly rebuilding?  Also is this reproducible with a brand new project?

I was playing with my repo https://github.com/changhuixu/angular-material-demo this afternoon. It worked until somehow I upgraded @angular/cli from v6.1.1 to v6.1.2.
ng s failed to launch app in browser. The messages in console are shown in the above image.
The local build didn't work in web server either.

I started over from scratch using @angular/cli v6.1.2 to create a new project and copy over files from my previous repo. And it works again...
The new repo is at https://github.com/changhuixu/angular-material-admin

I had the same issue with:
node 10.8.0 and npm 6.2.0 and OSX 10.13.6
I was able to temporarily resolve the CPU and node-gyp errors by running this in my local project:
npm install<EMAIL_ADDRESS>This was with

I had the same issue, even with an older version of fsevents installed (1.1.3). I had to specifically install<EMAIL_ADDRESS>as per jokecamp's comment, to fix this problem.

Well, I don't know if it is related or not but when I'm on node v10 and above it takes around 40-50 seconds to load the page and when downgrading to node v9 it takes around 20-30 seconds

I find it remarkable that the Angular build process is so crap. I regularly have to run a prod build twice on our 4gb mem server as it completely hoses the CPU and memory. A vue or react build of similar size takes 1/5 of the resources.

@jokecamp I have the same issue, I tried install fsevents 1.2.3 but it gave me following error

npm ERR! notsup Unsupported platform for<EMAIL_ADDRESS>wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

Do you have any other solution for Linux o.s


feature: improve docs

the README.md documentation has been improved and also the routes have been fixed!


[YUNIKORN-433] Extend configwatcher expiration

Travis tests have failed
Hey @kingamarton,
Please read the following log in order to understand the failure reason.
It'll be awesome if you fix what's wrong and commit the changes.
1st Build
View build log

     curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $(go env GOPATH)/bin v1.22.2

golangci/golangci-lint info checking GitHub for tag 'v1.22.2'
golangci/golangci-lint info found version: 1.22.2 for v1.22.2/linux/amd64
golangci/golangci-lint info installed /home/travis/gopath/bin/golangci-lint

     make lint

running golangci-lint
pkg/common/configs/configwatcher.go:142:3: SA4006: this value of `timer` is never used (staticcheck)
		timer = time.AfterFunc(cw.expireTime, func() {
		^
Makefile:45: recipe for target 'lint' failed
make: *** [lint] Error 1

TravisBuddy Request Identifier: 60a70bf0-04c9-11eb-a4bc-1b94c014fd22

Travis tests have failed
Hey @kingamarton,
Please read the following log in order to understand the failure reason.
It'll be awesome if you fix what's wrong and commit the changes.
1st Build
View build log

     curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $(go env GOPATH)/bin v1.22.2

golangci/golangci-lint info checking GitHub for tag 'v1.22.2'
golangci/golangci-lint info found version: 1.22.2 for v1.22.2/linux/amd64
golangci/golangci-lint info installed /home/travis/gopath/bin/golangci-lint

     make lint

running golangci-lint
pkg/common/configs/configwatcher.go:110: File is not `goimports`-ed with -local github.com/apache/incubator-yunikorn (goimports)
var timer *time.Timer
Makefile:45: recipe for target 'lint' failed
make: *** [lint] Error 1

TravisBuddy Request Identifier: 3fe92270-04e4-11eb-a4bc-1b94c014fd22

hi @kingamarton
Overall it looks good.
Looks like there is a minor lint issue, pls fix it.
Could you pls try to add UT in configwatcher_test.go? We already have something like TestConfigWatcherExpiration().


Day444

Resolves #890.

Pull Request Test Coverage Report for Build 1662

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 100.0%

Totals

Change from base Build 1660:
0.0%

Covered Lines:
5

Relevant Lines:
5

💛  - Coveralls


Gracefully handle port mismatch between Ingress and Service specs

This change addresses NullReferenceException exception during reconciliation #1736.

There is a similar problem with the name too, right?
https://github.com/microsoft/reverse-proxy/blob/b84a69f782d599fdc7e37f513c1367cfc119303b/src/Kubernetes.Controller/Converters/YarpParser.cs#L42-L43
Should we just skip the HandleIngressRulePath if we can't find the matching service/port?

@MihaZupan  If there is no matching service one line 42, I agree that there's little point continuing the checks.
I've also wondered whether ServiceData should remain a struct or if a class would be more suitable (same of the struct Endpoints).


Fix tests and be a bit lighter on mosquitto for stress tests

Just updated the test hosts to localhost so they use the local mosquitto broker.
Changed the behaviour of disconnect to still reconnect as the tests expect, when dont_reconnect = false. Figured shutdown would be used to completely stop the client.

Thanks again :)


[entropy_src/rtl] Fix observe fifo overflow reg

This PR latches the observe FIFO overflow register signal instead of only pulsing it.

This PR also contains a commit that changes a mubi4 test true signal from strict to loose.
This is done as a follow up for #21369.
Closes #21279

Added @jwnrt and @pamaury as reviewers since this might interest you.

@pamaury and @vogelpi thanks both for reviewing. Let's see what the CI tells us and then merge!


Request for small enhancement - disconnect command

This is a minor convenience.
When you are connected to the BP and you want to disconnect so you can copy a file onto the BP, the only way to do this is to (1) use your terminal program to disconnect, (2) unplug the BP, or (3) type "#" to reset the BP.
Disadvantages of Option  (1 ) - the file system remains read only. If you want to copy a file onto the BP, you have to use option 1 or 3.
Disadvantages of (2) - you have to take your hands off the keyboard.  Replugging a USB cable sometimes causes fumbling, and takes time, and if you do this several times a day this is awkward.
Disadvantages of (3) This works, but my Linux system takes a few seconds to unmount and remount the file system, and the BusPirate is reset. In  addition, my Linux system makes an alert sound when the FS disconnects and another when it remounts. Be-Bloop......... Boop-Bleep. This can get irritating the ears if heard multiple times an hour.
A command that would (a) terminate the serial connection, and (b) have the file system remounted read/write would be more convenient.

This is a duplicate of (a small part of) #110.
The current mechanism used for storage is only 97.2% safe(+).  Soon, it will not be possible to write from the host by default, unless explicitly the BP5 is switched into a host-exclusive mode.  There will be a command to switch the BP5 to that mode, and possibly also a boot method (e.g., holding the button while plugging in) will also go to that mode.
(+)
85.4% of people believe a percentage statistic when it has a decimal point.  ;-)


FS-4200-remove-html-tags-from-export-applicat-information

Ticket
https://dluhcdigital.atlassian.net/jira/software/c/projects/FS/boards/50?selectedIssue=FS-4200
Description
remove html tags from export data
How to test
unit test added
Screenshots of UI changes

@ramsharma-prog How about the other cosmetic issues with that export? The ticket was more than just removing the html tags.


gdsfactory8 issues

Issues Reproducing Routing Examples for Fiber Arrays/Pads

Recent routing changes have caused code failures when reproducing examples for fiber arrays and pads. The solution is not immediately clear. Is the documentation not yet updated or there are issues with the release?
KLive Behavior Changes on Windows
KLive now saves temporary GDS files for KLayout three levels up from the directory where the script or notebook is run. This is different from the previous behavior of saving them to the temporary folder in the home directory (e.g., AppData/Local/Tmp). The new behavior is inconvenient and confusing.

For the saving pattern, it creates a build folder in your git repository if there is one (and doesn't clean up).
If it's not a git repository, it will create a temporary file and clean it up. What different behavior would you expect for the save?
Having a build folder is kind of standard in a lot of programming languages.
@tvt173 maybe we shouldn't have added that check :smile:

Hi @sebastian-goeldi,
I'm referring to the show() function within KLive, not saving files directly. When I experiment with components in notebooks, my build folder gets cluttered with temporary GDS files. This doesn't happen in gdsfactory7, which uses the PC's temporary directory instead. It's not a crucial problem for sure, I've just noticed the difference (which is a bit annoying) between gdsfactory7 and gdsfactory8 release.

I get that. It's a problem with ipython (or rather I didn't expect this when I wrote the logic for show). Just to clarify this doesn't have anything to do with klive. klive only loads whatever it gets instructed to load with a simple HTTP request containing a JSON.
The show logic changed because gdsfactory is now using kfactory as a backend. And in the end it uses stem("file.py").oas as the name, but somehow in ipython that resolves to a number which I suspect changes with every change in the notebook.
I'll look into it in kfactory. In the meantime the easy solution is (please make absolutely sure all files you want to have tracked are in git status is to run git clean -xdf in the root folder; this will recursively delete all unknown (untracked and ignored & unknown) files for git.
Alternatively just delete the build folder some what regularly.
I will try to fix this in the show behavior of kfactory and try to get a consistent name for the oas/gds.

I can't reproduce this with a standard cpython kernel though. I can in an ipython kernel, but not the standard python.

Ah ok, but then it is working as intended. It creates one file per *.py file that is executed.
With jupyter the problem is it will create 13527282.oas and the number keeps changing

What happened to the ports?
Just upgraded from an older version (5 haha, had a lot of custom stuff), suddenly I am not able to use any tapers anymore. I keep getting stuck with:
"dwidth needs to be even to snap to grid. Got 0.155"
I've been digging through the code and it happens when the taper tries to generate its ports. Why is this function forcing me to change my taper width to an even number?

The problem is there is no good solution to this.
Assuming you are on a 1nm dbu proccess, a taper with 155nm width would have the port center at 77.5nm, therefore off-grid. Similar a wavguide with 155nm width cannot have an on-grid center, or if it is, the wavegide sides are off-grid, which is even worse.
This can cause 1nm shifts because depending how you are rounding and positioning, either you port would be assymetric wrt the center or your center is not on grid.
Sure there would be ways to calculate ports which are off-grid, but the design of it to safely avoid snapping errors require a lot of overhead which is just not worth the headache.
I would suggest to move to either 154 or 156nm width.
If you really are in a regime where it would matter foe the accuracy you probably should request your foundry to go below 1nm dbu

@sebastian-goeldi , i agree that probably the best solution in this case would be to avoid off-grid features. but if we just assume for a moment that @treep2 has a justifiable need for this, shouldn't he be able to do it with virtual cells and instances?

@sebastian-goeldi , i agree that probably the best solution in this case would be to avoid off-grid features. but if we just assume for a moment that @treep2 has a justifiable need for this, shouldn't he be able to do it with virtual cells and instances?

Correct. With virtual cells and instances it is possible to have this. I actually meant to add this in my comment but forgot ;). However, I believe gdsfactory at them moment doesn't have the virtual taper function, yet.


Index out of bounds in table code

It seems like p5.js' README is a bit of a fun stress-test for the table code. It'll make sense when you scroll to the bottom of it
https://github.com/processing/p5.js/blob/d086bae37aca14850f838ea79ae721da4e08089c/README.md

Had a look into this. Fixing the error is easy, however the bigger problem is that the tables code doesn't support images yet. I'll see if I can hack this together soonish.


Support for tailwind 1.2 [Feature Request]

Please add support for new utilities introduced with tailwind 1.2 like css grid, transform, etc..

Just digging into the source, I guess we easily could create a PR, as I guess we only have to extend these files:
https://github.com/bradlc/babel-plugin-tailwind-components/blob/master/src/static-styles.js
https://github.com/bradlc/babel-plugin-tailwind-components/blob/master/src/dynamic-styles.js

Yeah, right. Please confirm @bradlc

Alright, I'm going in.
Starting with staticStyles display section, and will go in that order and keep updating as in latest v1.2 of tw.
It will be great if more people join in and help, maybe take on dynamicStyles
Note: I'm not sure if updating only these 2 files will do the trick. Hope so.

There are features that have been introduced that require some major changes to the selector matching system in this package.
I'm releasing a new package soon that has a high degree of support for the new v1.4 features along with a few extras.
Follow for announcements

Thanks @ben-rogerson , shall wait for it then. When can this plugin be released?
It will be great if you keep the similar tw`` literal format as in this plugin.
And you mean to say tailwindcss v1.2 features, right?

I'm planning on the first release in two weeks around Feb 26th.
I've chosen to use babel-plugin-tailwind-components as a solid base so it'll include the same features, and the macro tw will definitely be kept 👍
Correct, I meant Tailwind 1.2 (I'll edit that).

Hey all, I've just released my plugin Twin 🎉

Thank you so much. @ben-rogerson

Isn't it possible to get the features from tailwindcss instead of having to keep this plugin up to date manually?

It's only possible to do so up to a certain extent. I guess, @ben-rogerson is doing that in twin.macro.

There is something in progress at https://github.com/ben-rogerson/twin.macro/pull/13