RuntimeError: Can't load tokenizer for 'gpt-cpm-large-cn'

paddlepaddle-gpu 2.2.2.post112
paddlenlp 2.2.4
python 3.8.12
from paddlenlp.transformers import AutoModel, AutoTokenizer
AutoTokenizer.from_pretrained('gpt-cpm-large-cn')

then I got
Traceback (most recent call last):
  File "/home/yangbo25/software/conda/lib/python3.8/site-packages/paddlenlp/transformers/auto/tokenizer.py", line 195, in from_pretrained
    resolved_vocab_file = get_path_from_url(community_config_path,
  File "/home/yangbo25/software/conda/lib/python3.8/site-packages/paddlenlp/utils/downloader.py", line 163, in get_path_from_url
    fullpath = _download(url, root_dir, md5sum)
  File "/home/yangbo25/software/conda/lib/python3.8/site-packages/paddlenlp/utils/downloader.py", line 199, in _download
    raise RuntimeError("Downloading from {} failed with code "
RuntimeError: Downloading from https://bj.bcebos.com/paddlenlp/models/transformers/community/gpt-cpm-large-cn/tokenizer_config.json failed with code 404!

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/yangbo25/software/conda/lib/python3.8/site-packages/paddlenlp/transformers/auto/tokenizer.py", line 199, in from_pretrained
    raise RuntimeError(
RuntimeError: Can't load tokenizer for 'gpt-cpm-large-cn'.
Please make sure that 'gpt-cpm-large-cn' is:
- a correct model-identifier of built-in pretrained models,
- or a correct model-identifier of community-contributed pretrained models,
- or the correct path to a directory containing relevant tokenizer files.

Thanks for your question abour AutoTokenizer, you can use GPTChineseTokenizer.from_pretrained('gpt-cpm-large-cn') instead. We'll upgrade AutoTokenizer to support more cases!

same problem while
AutoModel.from_pretrained('roberta-base-ft-chinanews-chn')

then I got
>>> AutoModel.from_pretrained('roberta-base-ft-chinanews-chn')
[2022-03-02 12:12:44,215] [    INFO] - Downloading model_config.json from https://bj.bcebos.com/paddlenlp/models/transformers/community/roberta-base-ft-chinanews-chn/model_config.json
[2022-03-02 12:12:44,297] [   ERROR] - Downloading from https://bj.bcebos.com/paddlenlp/models/transformers/community/roberta-base-ft-chinanews-chn/model_config.json failed with code 404!
Traceback (most recent call last):
  File "/home/yangbo25/software/conda/lib/python3.8/site-packages/paddlenlp/transformers/auto/modeling.py", line 190, in _from_pretrained
    resolved_vocab_file = get_path_from_url(community_config_path,
  File "/home/yangbo25/software/conda/lib/python3.8/site-packages/paddlenlp/utils/downloader.py", line 163, in get_path_from_url
    fullpath = _download(url, root_dir, md5sum)
  File "/home/yangbo25/software/conda/lib/python3.8/site-packages/paddlenlp/utils/downloader.py", line 199, in _download
    raise RuntimeError("Downloading from {} failed with code "
RuntimeError: Downloading from https://bj.bcebos.com/paddlenlp/models/transformers/community/roberta-base-ft-chinanews-chn/model_config.json failed with code 404!

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/yangbo25/software/conda/lib/python3.8/site-packages/paddlenlp/transformers/auto/modeling.py", line 299, in from_pretrained
    return cls._from_pretrained(pretrained_model_name_or_path, task,
  File "/home/yangbo25/software/conda/lib/python3.8/site-packages/paddlenlp/transformers/auto/modeling.py", line 194, in _from_pretrained
    raise RuntimeError(
RuntimeError: Can't load weights for 'roberta-base-ft-chinanews-chn'.
Please make sure that 'roberta-base-ft-chinanews-chn' is:
- a correct model-identifier of built-in pretrained models,
- or a correct model-identifier of community-contributed pretrained models,
- or the correct path to a directory containing relevant modeling files(model_weights and model_config).

We have updated our documents about weight names, see https://paddlenlp.readthedocs.io/zh/latest/model_zoo/transformers.html
You can use AutoModel.from_pretrained('nosaydomore/uer-roberta-base-ft-chinanews-chn') to load it.

same problem while
AutoModel.from_pretrained('roberta-base-ft-chinanews-chn')

then I got
>>> AutoModel.from_pretrained('roberta-base-ft-chinanews-chn')
[2022-03-02 12:12:44,215] [    INFO] - Downloading model_config.json from https://bj.bcebos.com/paddlenlp/models/transformers/community/roberta-base-ft-chinanews-chn/model_config.json
[2022-03-02 12:12:44,297] [   ERROR] - Downloading from https://bj.bcebos.com/paddlenlp/models/transformers/community/roberta-base-ft-chinanews-chn/model_config.json failed with code 404!
Traceback (most recent call last):
  File "/home/yangbo25/software/conda/lib/python3.8/site-packages/paddlenlp/transformers/auto/modeling.py", line 190, in _from_pretrained
    resolved_vocab_file = get_path_from_url(community_config_path,
  File "/home/yangbo25/software/conda/lib/python3.8/site-packages/paddlenlp/utils/downloader.py", line 163, in get_path_from_url
    fullpath = _download(url, root_dir, md5sum)
  File "/home/yangbo25/software/conda/lib/python3.8/site-packages/paddlenlp/utils/downloader.py", line 199, in _download
    raise RuntimeError("Downloading from {} failed with code "
RuntimeError: Downloading from https://bj.bcebos.com/paddlenlp/models/transformers/community/roberta-base-ft-chinanews-chn/model_config.json failed with code 404!

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/yangbo25/software/conda/lib/python3.8/site-packages/paddlenlp/transformers/auto/modeling.py", line 299, in from_pretrained
    return cls._from_pretrained(pretrained_model_name_or_path, task,
  File "/home/yangbo25/software/conda/lib/python3.8/site-packages/paddlenlp/transformers/auto/modeling.py", line 194, in _from_pretrained
    raise RuntimeError(
RuntimeError: Can't load weights for 'roberta-base-ft-chinanews-chn'.
Please make sure that 'roberta-base-ft-chinanews-chn' is:
- a correct model-identifier of built-in pretrained models,
- or a correct model-identifier of community-contributed pretrained models,
- or the correct path to a directory containing relevant modeling files(model_weights and model_config).

We have updated our documents about weight names, see https://paddlenlp.readthedocs.io/zh/latest/model_zoo/transformers.html You can use AutoModel.from_pretrained('nosaydomore/uer-roberta-base-ft-chinanews-chn') to load it.

work properly


Keep notification ordering and back off failed notification attempts

supersedes #263

uses a queue to keep notification ordering
starts a couple of threads to handle "retry-buckets": a token with a failed notification requests goes into the next retry-bucket until it exceeds max number of buckets and it is dropped
extensive tests also for the (rare) error cases
split "metadata" and "notification" logic into separate files

without understanding all tech: very nice to read PR <3


New(?) Device: GT-I9500

I'm using the latest version on Mac to flash a recovery to a Galaxy S4 (i9500) and it's failing (same thing happens on Linux). See below the commands I run. I've tried on several machines and with several images, it always fails. I can provide more info if needed.
Jonathans-MacBook-Pro:CF-Auto-Root-ja3g-ja3gxx-gti9500 tapicer$ heimdall flash --recovery CF-Auto-Root-ja3g-ja3gxx-gti9500.tar.md5 --verbose
Heimdall v1.4.0

Copyright (c) 2010-2013, Benjamin Dobell, Glass Echidna
http://www.glassechidna.com.au/

This software is provided free of charge. Copying and redistribution is
encouraged.

If you appreciate this software and you would like to support future
development please consider donating:
http://www.glassechidna.com.au/donate/

Initialising connection...
Detecting device...
      Manufacturer: "SAMSUNG"
           Product: "Gadget Serial"

            length: 18
      device class: 2
               S/N: 0
           VID:PID: 04E8:685D
         bcdDevice: 021B
   iMan:iProd:iSer: 1:2:0
          nb confs: 1

interface[0].altsetting[0]: num endpoints = 1
   Class.SubClass.Protocol: 02.02.01
       endpoint[0].address: 83
           max packet size: 0010
          polling interval: 09

interface[1].altsetting[0]: num endpoints = 2
   Class.SubClass.Protocol: 0A.00.00
       endpoint[0].address: 81
           max packet size: 0200
          polling interval: 00
       endpoint[1].address: 02
           max packet size: 0200
          polling interval: 00
Claiming interface...
Setting up interface...

Initialising protocol...
WARNING: Control transfer #2 failed. Result: -7
WARNING: Control transfer #3 failed. Result: -7
WARNING: Control transfer #4 failed. Result: -7
WARNING: Control transfer #5 failed. Result: -7
WARNING: Control transfer #6 failed. Result: -1
ERROR: Failed to send data: "ODIN"
Releasing device interface...

I have the same problem with a SM-T520, on 2 different linux (ubuntu 12.04 and 14.04) machines and a windows 7 machine:
my output (on a linux, ubuntu 12.04) is:
heimdall flash --RECOVERY ./recovery.img --no-reboot --verbose
Heimdall v1.4.0
Copyright (c) 2010-2013, Benjamin Dobell, Glass Echidna
http://www.glassechidna.com.au/
This software is provided free of charge. Copying and redistribution is encouraged.
If you appreciate this software and you would like to support future development please consider donating:
http://www.glassechidna.com.au/donate/

Initialising connection...
Detecting device...
  Manufacturer: "SAMSUNG"
       Product: "Gadget Serial"

        length: 18
  device class: 2
           S/N: 0
       VID:PID: 04E8:685D
     bcdDevice: 021B
iMan:iProd:iSer: 1:2:0
      nb confs: 1

interface[0].altsetting[0]: num endpoints = 1
Class.SubClass.Protocol: 02.02.01
   endpoint[0].address: 83
       max packet size: 0010
      polling interval: 09

interface[1].altsetting[0]: num endpoints = 2
Class.SubClass.Protocol: 0A.00.00
   endpoint[0].address: 81
       max packet size: 0200
      polling interval: 00
   endpoint[1].address: 02
       max packet size: 0200
      polling interval: 00
Claiming interface...
Attempt failed. Detaching driver...
Claiming interface again...
Setting up interface...

Initialising protocol...
WARNING: Control transfer #2 failed. Result: -7
WARNING: Control transfer #3 failed. Result: -7
WARNING: Control transfer #4 failed. Result: -7
WARNING: Control transfer #5 failed. Result: -7
WARNING: Control transfer #6 failed. Result: -99
ERROR: Failed to send data: "ODIN"
Releasing device interface...
Re-attaching kernel driver...

+1 for this!
I have samsung I9500 non-LTE international
sudo heimdall flash --RECOVERY ./recovery-clockwork-<IP_ADDRESS>-i9500.img --no-reboot --verbose
Heimdall v1.4.0

Copyright (c) 2010-2013, Benjamin Dobell, Glass Echidna
http://www.glassechidna.com.au/

This software is provided free of charge. Copying and redistribution is
encouraged.

If you appreciate this software and you would like to support future
development please consider donating:
http://www.glassechidna.com.au/donate/

Initialising connection...
Detecting device...
      Manufacturer: "SAMSUNG"
           Product: "Gadget Serial"

            length: 18
      device class: 2
               S/N: 0
           VID:PID: 04E8:685D
         bcdDevice: 021B
   iMan:iProd:iSer: 1:2:0
          nb confs: 1

interface[0].altsetting[0]: num endpoints = 1
   Class.SubClass.Protocol: 02.02.01
       endpoint[0].address: 83
           max packet size: 0010
          polling interval: 09

interface[1].altsetting[0]: num endpoints = 2
   Class.SubClass.Protocol: 0A.00.00
       endpoint[0].address: 81
           max packet size: 0200
          polling interval: 00
       endpoint[1].address: 02
           max packet size: 0200
          polling interval: 00
Claiming interface...
Setting up interface...

Initialising protocol...
WARNING: Control transfer #2 failed. Result: -7
WARNING: Control transfer #3 failed. Result: -7
WARNING: Control transfer #4 failed. Result: -7
WARNING: Control transfer #5 failed. Result: -7
WARNING: Control transfer #6 failed. Result: -1
ERROR: Failed to send data: "ODIN"
Releasing device interface...

same behaviour on linux and mac.

Any updates on this problem?  Galaxy Tab Pro 10.1 on Mac OSX 10.10.1 and getting this:
heimdall flash --RECOVERY recovery.img --no-reboot --verbose
Heimdall v1.4.0

Copyright (c) 2010-2013, Benjamin Dobell, Glass Echidna
http://www.glassechidna.com.au/

This software is provided free of charge. Copying and redistribution is
encouraged.

If you appreciate this software and you would like to support future
development please consider donating:
http://www.glassechidna.com.au/donate/

Initialising connection...
Detecting device...
      Manufacturer: "SAMSUNG"
           Product: "Gadget Serial"

            length: 18
      device class: 2
               S/N: 0
           VID:PID: 04E8:685D
         bcdDevice: 021B
   iMan:iProd:iSer: 1:2:0
          nb confs: 1

interface[0].altsetting[0]: num endpoints = 1
   Class.SubClass.Protocol: 02.02.01
       endpoint[0].address: 83
           max packet size: 0010
          polling interval: 09

interface[1].altsetting[0]: num endpoints = 2
   Class.SubClass.Protocol: 0A.00.00
       endpoint[0].address: 81
           max packet size: 0200
          polling interval: 00
       endpoint[1].address: 02
           max packet size: 0200
          polling interval: 00
Claiming interface...
Setting up interface...

Initialising protocol...
WARNING: Control transfer #2 failed. Result: -7
WARNING: Control transfer #3 failed. Result: -7
WARNING: Control transfer #4 failed. Result: -7
WARNING: Control transfer #5 failed. Result: -7
WARNING: Control transfer #6 failed. Result: -1
ERROR: Failed to send data: "ODIN"
Releasing device interface...

Any help is appreciated :)

Bump. Getting almost the exact same error as @j-woodlee, with the following minor difference at the end. Ubuntu 14.04.1, kernel 3.14.1. Samsung Galaxy Tab Pro 10.1, SM-T520.
Claiming interface...
Attempt failed. Detaching driver...
Claiming interface again...
Setting up interface...
Initialising protocol...
WARNING: Control transfer #2 failed. Result: -7
WARNING: Control transfer #3 failed. Result: -7
WARNING: Control transfer #4 failed. Result: -7
WARNING: Control transfer #5 failed. Result: -7
WARNING: Control transfer #6 failed. Result: -1
ERROR: Failed to receive handshake response. Retrying...
ERROR: Failed to send data: "ODIN"
Releasing device interface...
Re-attaching kernel driver...
thanks all!

Please try build the latest version from source, the issue should be resolved.

Hi Benjamin! Thanks for the tip about building from this repo. Unfortunately, it still doesn't seem to flash the device. Here's the full output from the build of heimdall as of this evening:
drew@heidi-linux:~/repositories/Heimdall/bin$ sudo ./heimdall flash --verbose --RECOVERY clockworkmodrecovery6051picassowifi.img --no-reboot
Heimdall v1.4.1
Copyright (c) 2010-2014 Benjamin Dobell, Glass Echidna
http://www.glassechidna.com.au/
This software is provided free of charge. Copying and redistribution is
encouraged.
If you appreciate this software and you would like to support future
development please consider donating:
http://www.glassechidna.com.au/donate/
Initialising connection...
Detecting device...
Manufacturer: "SAMSUNG"
Product: "Gadget Serial"
        length: 18
  device class: 2
           S/N: 0
       VID:PID: 04E8:685D
     bcdDevice: 021B

iMan:iProd:iSer: 1:2:0
nb confs: 1
interface[0].altsetting[0]: num endpoints = 1
Class.SubClass.Protocol: 02.02.01
endpoint[0].address: 83
max packet size: 0010
polling interval: 09
interface[1].altsetting[0]: num endpoints = 2
Class.SubClass.Protocol: 0A.00.00
endpoint[0].address: 81
max packet size: 0200
polling interval: 00
endpoint[1].address: 02
max packet size: 0200
polling interval: 00
Claiming interface...
Attempt failed. Detaching driver...
Claiming interface again...
Setting up interface...
Initialising protocol...
ERROR: Failed to receive handshake response. Result: -7
ERROR: Protocol initialisation failed!
Releasing device interface...
Re-attaching kernel driver...
drew@heidi-linux:~/repositories/Heimdall/bin$
Can you think of anything I might need to check that might be causing this? Or would it still be a genuine bug? I've got the ubuntu packages for libusb-1.0.0 and its -dev and -dbg versions installed, but from what I understand, the -7 error is related to libusb. I'm happy to help debug however I can!

heimdall print-pit working after building from last (today) source!
Quick instructions for the lazy ones:
git clone https://github.com/Benjamin-Dobell/Heimdall.git
cd heimdall
sudo apt-get install build-essential cmake zlib1g-dev qt5-default libusb-1.0-0-dev libgl1-mesa-glx libgl1-mesa-dev
mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=Release ..
make
cd bin
./heimdall print-pit --no-reboot
Just need to reboot the phone in download mode before issuing further heimdall commands.

@apsabelhaus  @kriswebdev Your's might be the same issue as #228. Try my fix with the udev rule.


Add env FAKER_LOCALE to easy change faker locale

You cannot call config or env methods in the framework core.


creating workflow

creating workflow

Dear @apophis-web, please do not use the pull-request functionality for personal modifications to the code and its style. Please use a fork of the repository for this purpose.


Add auto-deploy option for react-codesandboxer

This option allows a sandbox to deploy on mount. I to use this to automatically creates an iframe embed from a component.

This is the idea! https://codesandbox.io/s/215z0lwj0y

I've been making this harder to do since I didn't want you to end up with too many requests when they weren't needed

Hmm yes, that's understandable. I think that for the client sandboxes our servers will be fine with this, so no problem!

I think it would actually be cool if, as well as adding this prop, we exported a second component from this package which was the SandboxEmbed component.

Hah, I did something like that! (https://github.com/CompuIves/codesandbox-client/tree/master/packages/react-embed)
It's probably better to move it to this repo, I agree!

I’m going to merge this pr so this just gets in my next release tonight.
Would welcome the embed pr, but happy if you want to keep owning it.
Thanks again for the contribution ❤
Ps. Would be neat if you run yarn changeset on future PRs. I’m testing out a monorepo changeset management tool.


Can't start training.

I get the following error. I am not sure what I need to do to fix my tf record files.
2019-06-10 21:27:25.351566: W tensorflow/core/framework/op_kernel.cc:1431] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Feature: image/key/sha256 (data type: string) is required but could not be found.
2019-06-10 21:27:25.351650: W tensorflow/core/framework/op_kernel.cc:1431] OP_REQUIRES failed at iterator_ops.cc:988 : Invalid argument: Feature: image/key/sha256 (data type: string) is required but could not be found.
	 [[{{node ParseSingleExample/ParseSingleExample}}]]
Traceback (most recent call last):
  File "train.py", line 178, in <module>
    app.run(main)
  File "/home/shaun/tf2/env/lib/python3.6/site-packages/absl/app.py", line 300, in run
2019-06-10 21:27:25.351867: W tensorflow/core/framework/op_kernel.cc:1431] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Feature: image/key/sha256 (data type: string) is required but could not be found.
    _run_main(main, args)
  File "/home/shaun/tf2/env/lib/python3.6/site-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "train.py", line 173, in main
    validation_data=val_dataset)
  File "/home/shaun/tf2/env/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 791, in fit
    initial_epoch=initial_epoch)
  File "/home/shaun/tf2/env/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 1515, in fit_generator
    steps_name='steps_per_epoch')
  File "/home/shaun/tf2/env/lib/python3.6/site-packages/tensorflow/python/keras/engine/training_generator.py", line 213, in model_iteration
    batch_data = _get_next_batch(generator, mode)
  File "/home/shaun/tf2/env/lib/python3.6/site-packages/tensorflow/python/keras/engine/training_generator.py", line 355, in _get_next_batch
    generator_output = next(generator)
  File "/home/shaun/tf2/env/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 556, in __next__
    return self.next()
  File "/home/shaun/tf2/env/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 585, in next
    return self._next_internal()
  File "/home/shaun/tf2/env/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 577, in _next_internal
    output_shapes=self._flat_output_shapes)
  File "/home/shaun/tf2/env/lib/python3.6/site-packages/tensorflow/python/ops/gen_dataset_ops.py", line 1954, in iterator_get_next_sync
    _six.raise_from(_core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: Feature: image/key/sha256 (data type: string) is required but could not be found.
	 [[{{node ParseSingleExample/ParseSingleExample}}]] [Op:IteratorGetNextSync]

Where are you getting your tf records from? I recall I had similar issues and what I can say is that the vott pascal export works amazingly.
 Right now there are open issues regarding training. Just wondering if you managed to train successfully.

@ycelik  @rlewkowicz  I agree with ycelik, I give the tfrecord file for my custom dataset and getting errors while training

DId you solve it? I miss one error like your problem.
2019-07-26 09:28:27.838122: W tensorflow/core/framework/op_kernel.cc:1546] OP_REQUIRES failed at iterator_ops.cc:1055 : Invalid argument: Paddings must be non-negative: 0 -16
         [[{{node Pad}}]]
Traceback (most recent call last):
  File "train.py", line 177, in <module>
    app.run(main)
  File "/home/rainweic/anaconda3/envs/python35/lib/python3.5/site-packages/absl/app.py", line 300, in run
    _run_main(main, args)
  File "/home/rainweic/anaconda3/envs/python35/lib/python3.5/site-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "train.py", line 116, in main
    for batch, (images, labels) in enumerate(train_dataset):
  File "/home/rainweic/anaconda3/envs/python35/lib/python3.5/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 586, in __next__
    return self.next()
  File "/home/rainweic/anaconda3/envs/python35/lib/python3.5/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 623, in next
    return self._next_internal()
  File "/home/rainweic/anaconda3/envs/python35/lib/python3.5/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 615, in _next_internal
    output_shapes=self._flat_output_shapes)
  File "/home/rainweic/anaconda3/envs/python35/lib/python3.5/site-packages/tensorflow/python/ops/gen_dataset_ops.py", line 2120, in iterator_get_next_sync
    _six.raise_from(_core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: Paddings must be non-negative: 0 -16
         [[{{node Pad}}]] [Op:IteratorGetNextSync]
Exception ignored in: <bound method _CheckpointRestoreCoordinator.__del__ of <tensorflow.python.training.tracking.util._CheckpointRestoreCoordinator object at 0x7f0c8052abe0>>
Traceback (most recent call last):
  File "/home/rainweic/anaconda3/envs/python35/lib/python3.5/site-packages/tensorflow/python/training/tracking/util.py", line 244, in __del__
  File "/home/rainweic/anaconda3/envs/python35/lib/python3.5/site-packages/tensorflow/python/training/tracking/util.py", line 93, in node_names
  File "/home/rainweic/anaconda3/envs/python35/lib/python3.5/site-packages/tensorflow/python/training/tracking/object_identity.py", line 76, in __getitem__
KeyError: (<tensorflow.python.training.tracking.object_identity._ObjectIdentityWrapper object at 0x7f0c8049a048>,)```

I managed to resolve the issue a while back by going back and making sure filepaths were absolute and that bbox coordinates were integers. I also used a different script to convert my training data to a tfrecord. Not sure which made the difference.

I will leave this thread open for the present time because it appears others are having the same issue.
Thank you @rlewkowicz for taking the time to provide suggestions. Sorry for the late reply to this thread, I got caught up in school work.

I managed to resolve the issue a while back by going back and making sure filepaths were absolute and that bbox coordinates were integers. I also used a different script to convert my training data to a tfrecord. Not sure which made the difference.

Can you show me the script? Thanks!

Hello, I have a similar problem these days, and I could not solve it. Do you have some advice?
Epoch: [93/10] step: [186/2] time: 0.2719242572784424s, mse: 0.03679807484149933
2019-07-28 12:23:43.405599: W tensorflow/core/framework/op_kernel.cc:1431] OP_REQUIRES failed at iterator_ops.cc:988 : Invalid argument: Input shape axis 0 must equal 4, got shape [3]
[[{{node crop_to_bounding_box_1/unstack}}]]
Traceback (most recent call last):
File "train.py", line 380, in
train()
File "train.py", line 178, in train
for step, (lr_patchs, hr_patchs) in enumerate(train_ds):
File "/home/xyh/anaconda3/envs/tf2/lib/python3.7/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 556, in next
return self.next()
File "/home/xyh/anaconda3/envs/tf2/lib/python3.7/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 585, in next
return self._next_internal()
File "/home/xyh/anaconda3/envs/tf2/lib/python3.7/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 577, in _next_internal
output_shapes=self._flat_output_shapes)
File "/home/xyh/anaconda3/envs/tf2/lib/python3.7/site-packages/tensorflow/python/ops/gen_dataset_ops.py", line 1954, in iterator_get_next_sync
_six.raise_from(_core._status_to_exception(e.code, message), None)
File "", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: Input shape axis 0 must equal 4, got shape [3]
[[{{node crop_to_bounding_box_1/unstack}}]] [Op:IteratorGetNextSync]
2019-07-28 12:23:43.507127: W tensorflow/core/kernels/data/generator_dataset_op.cc:79] Error occurred when finalizing GeneratorDataset iterator: Failed precondition: Python interpreter state is not initialized. The process may be terminated.
[[{{node PyFunc}}]]

Where are you getting your tf records from? I recall I had similar issues and what I can say is that the vott pascal export works amazingly.
You'll have to make some small modifications to this script:
https://github.com/tensorflow/models/blob/master/research/object_detection/dataset_tools/create_pascal_tf_record.py
But it will generate some fully functional tf records

Hello, I have a similar problem these days, and I could not solve it. Do you have some advice?
Epoch: [93/10] step: [186/2] time: 0.2719242572784424s, mse: 0.03679807484149933
2019-07-28 12:23:43.405599: W tensorflow/core/framework/op_kernel.cc:1431] OP_REQUIRES failed at iterator_ops.cc:988 : Invalid argument: Input shape axis 0 must equal 4, got shape [3]
[[{{node crop_to_bounding_box_1/unstack}}]]
Traceback (most recent call last):
File "train.py", line 380, in
train()
File "train.py", line 178, in train
for step, (lr_patchs, hr_patchs) in enumerate(train_ds):
File "/home/xyh/anaconda3/envs/tf2/lib/python3.7/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 556, in next
return self.next()
File "/home/xyh/anaconda3/envs/tf2/lib/python3.7/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 585, in next
return self._next_internal()
File "/home/xyh/anaconda3/envs/tf2/lib/python3.7/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 577, in _next_internal
output_shapes=self._flat_output_shapes)
File "/home/xyh/anaconda3/envs/tf2/lib/python3.7/site-packages/tensorflow/python/ops/gen_dataset_ops.py", line 1954, in iterator_get_next_sync
_six.raise_from(_core._status_to_exception(e.code, message), None)
File "", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: Input shape axis 0 must equal 4, got shape [3]
[[{{node crop_to_bounding_box_1/unstack}}]] [Op:IteratorGetNextSync]
2019-07-28 12:23:43.507127: W tensorflow/core/kernels/data/generator_dataset_op.cc:79] Error occurred when finalizing GeneratorDataset iterator: Failed precondition: Python interpreter state is not initialized. The process may be terminated.
[[{{node PyFunc}}]]

Any updates? This issue still exists.
I got the same errors

I think I've figured this out.
This is an issue with the tfrecord file. The file when generated din't had the image/key/sha256 feature which is required during training.
Please refer to the tensorflow issue https://github.com/tensorflow/models/issues/6253

I think I've figured this out.
As suggested by @rlewkowicz, This is an issue with the tfrecord file. The file when generated din't had the image/key/sha256 feature which is required during training.
Please refer to the tensorflow issue tensorflow/models#6253 (comment)

That ended up being the problem for me as well. Since  then I have used Microsoft VoTT to markup data and the tool has a tfrecord export option which works.

# Script to take image data and bounding box data from multiple files,
# combine with the image to create a single tf record file
import tensorflow as tf, cv2, glob, hashlib
from absl import app, flags, logging
from absl.flags import FLAGS
from object_detection.utils import dataset_util

# Path to output single tf.record file to
flags.DEFINE_string('output_path', '/home/will/BnO/Dataset/A9_tf/A9.tfrecord', 'Path to output TFRecord')
# Paths for dataset images and teh associated label files
# Assumes following conventions:
# /dataset_image/000.jpg.....xyz.jpg
# /dataset_label/000.txt.....xyz.txt
# Where files are not > 999, txt label file corresponds to jpg file and example label file is:
# 0 0.233076923077 0.776744186047 0.210769230769 0.432558139535
# Where: Class number, normalised x min, normalised x max, normalised y min, normalised y max
flags.DEFINE_string('dataset_image', '/Your_image_directory', 'Path to dataset images')
flags.DEFINE_string('dataset_label', '/Your_label_directory', 'Path to dataset labels')

# Function for creating tensorflow record
# image height, width, filename of image, encoded image data, key metadata,
# image format (png, jg, etc),
# list of minimum x bounding box values, list of maximum x bounding box values,
# list of minimum y bounding box values, list of maximum x bounding box values,
# text string of class (cat, dog, etc)
# classes - list of classes in the image
# NB this script only supports conversion for one bounding box for one class in the dataset,
# but could be converted to support multiple
def create_tf_example(height, width, image_filename, encoded_image_data, key,
                      image_format, xmins, xmaxs, ymins, ymaxs, classes_text, classes):

    # Tensorflow record fields, note the image key
    tf_example = tf.train.Example(features=tf.train.Features(feature={
        'image/height': dataset_util.int64_feature(height),
        'image/width': dataset_util.int64_feature(width),
        'image/filename': dataset_util.bytes_feature(image_filename.encode('utf8')),
        'image/source_id': dataset_util.bytes_feature(image_filename.encode('utf8')),
        'image/key/sha256': dataset_util.bytes_feature(key.encode('utf8')),
        'image/encoded': dataset_util.bytes_feature(encoded_image_data),
        'image/format': dataset_util.bytes_feature(image_format.encode('utf8')),
        'image/object/bbox/xmin': dataset_util.float_list_feature(xmins),
        'image/object/bbox/xmax': dataset_util.float_list_feature(xmaxs),
        'image/object/bbox/ymin': dataset_util.float_list_feature(ymins),
        'image/object/bbox/ymax': dataset_util.float_list_feature(ymaxs),
        'image/object/class/text': dataset_util.bytes_list_feature([classes_text.encode('utf8')]),
        'image/object/class/label': dataset_util.int64_list_feature(classes),
    }))
    return tf_example

def main(_argv):

    # Initialise tf writer, put this inside loop for multiple files
    writer = tf.io.TFRecordWriter(FLAGS.output_path)

    # Sort images
    for image_filename in sorted(glob.glob(FLAGS.dataset_image + '/*.jpg')):

        # Name of class to be used
        classes_text = "A9"

        # Print the image filename
        print("Image", image_filename)

        # Derive the filename of the label
        split = image_filename.split("/", -1)[-1:][0]
        split = split.split(".")[0]
        label_filename = FLAGS.dataset_label + '/' + split + '.txt'

        # Read in label data
        f = open(label_filename, "r")
        a = f.read()
        a = a[:-1]
        a = a.split(" ")

        xmins = []
        xmaxs = []
        ymins = []
        ymaxs = []

        class_id = [int(a[0]) + 1]
        xmins.append(float(a[1]))
        xmaxs.append(float(a[2]))
        ymins.append(float(a[3]))
        ymaxs.append(float(a[4]))

        # Find height and width of image
        raw_image = cv2.imread(image_filename, 0)
        height, width = raw_image.shape[:2]

        # Read in image data
        image = open(image_filename, 'rb')
        encoded_image_data = image.read()

        # Image 'key' - metadata needed for learning
        key = hashlib.sha256(encoded_image_data).hexdigest()

        # Once all data is collected create the tf record
        tf_example = create_tf_example(height, width, image_filename,  encoded_image_data, key,
                                       'jpg', xmins, xmaxs, ymins, ymaxs, classes_text, class_id)

        # Write tf record to file
        writer.write(tf_example.SerializeToString())

    # Close tf writer, put this inside loop for multiple files
    writer.close()

if __name__ == '__main__':
    app.run(main)

Thought this might be useful so just going to leave it here, it's a barebones python script for transforming image and label data into a tf record, in case anyone has any problems using VOTT and wants to do this from scratch. In particular VOTT sometimes creates individual tf record files, this is much neater as it only creates one tf record file with all the info in it. Currently only handles single bounding boxes and single classes, however it could easily be expanded.

I was facing the same issue, and coming to believe it has to do with TFRecord files (more about Tensorflow tf.records . Here I am sharing the full story and the solution I found that worked for me, hope others can use.
I initially started off using the TFRecord files generated using, I guess it was Dat Tran Object Detection repo  that implemented the generate_tfrecord.py script. However, using the this TFRecord files won't work and because it is missing the image/key/sha256 feature!
Others suggested an easy solution using Microsoft VoTT to export the TFRecords. However, I have already images with annotations, and I couldn't reload/resume in Microsoft VoTT annotations not even in the desktop version so that I can easily export the TFRecords (maybe I missed how to do it, I spent one hour and couldn't figure out and I thought this feature is not available), and there was no way I would relabel from scratch!! Anyway, long story short I noticed after coming across this blog post that is possible to use create_pascal_tf_record.py within Google Object Detection. The blog post gives a fairly enough explanation about changes one needs to do before simply running the script. Either you need to modify more in the script or simply adopt the data structure proposed and the script will work. Just a few remarks on top of the blog post documentation:

train.txt/test.txt simply contains list of train/test files (images/xml) without extensions and header!
Take extra care about Widows paths. Even it says 'change the data['folder'] in line 85 into your own dataset path', in Windows I replaced the data['folder'] with e.g. r'C:\full\path\to\datasetName', notice the r in the front of the path string.
Funnily I have had in my xml generated using of these annotating tools, I have had 'annotations', where in line 181 of the create_pascal_tf_record.py script in was originally dataset_util.recursive_parse_xml_to_dict(xml)['annotation'], make sure this corresponds to your xml naming.
When you follow the directory structure proposed in the blog, you need to be careful about importing modules (from object_detection.utils import dataset_util, label_map_util), the error would be obvious and you will fix it. But if you struggle just import sys.path.append("..")  before importing the object_detection.utils, and you should be fine.

At the end, with great attention to these small details (it is not that hard!, but they matter), you can run the script (I did within Anaconda Prompt in Windows):
$ python data\create_pascal_tf_record.py --data_dir=data\ --year=datasetName --set=test --output_path=data\record\test.tfrecord --label_map_path=data\labelmap.pbtxt
The TFRecords generated will work just fine with yolov3-tf2 (this repo). Happy detecting.

@mmortazavi could you share your version of create_pascal_tf_record.py? I am trying to recreate it based on your instructions and I have made progress but I am encountering some issues I can't solve.

Sure @cole8888. What what kind of error you get though? I have the following directory tree:

And here is the create_pascal_tf_record.py, which slighlty modified to generate the TFRecords:
# Copyright 2017 The TensorFlow Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================

r"""Convert raw PASCAL dataset to TFRecord for object_detection.

Example usage:
    python object_detection/dataset_tools/create_pascal_tf_record.py \
        --data_dir=/home/user/VOCdevkit \
        --year=VOC2012 \
        --output_path=/home/user/pascal.record
"""
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import hashlib
import io
import logging
import os
import sys

sys.path.append("..")

from lxml import etree
import PIL.Image
import tensorflow as tf

from object_detection.utils import dataset_util
from object_detection.utils import label_map_util

flags = tf.app.flags
flags.DEFINE_string('data_dir', '', 'Root directory to raw PASCAL VOC dataset.')
flags.DEFINE_string('set', 'train', 'Convert training set, validation set or '
                    'merged set.')
flags.DEFINE_string('annotations_dir', 'Annotations',
                    '(Relative) path to annotations directory.')
flags.DEFINE_string('year', 'VOC2007', 'Desired challenge year.')
flags.DEFINE_string('output_path', '', 'Path to output TFRecord')
flags.DEFINE_string('label_map_path', 'data/pascal_label_map.pbtxt',
                    'Path to label map proto')
flags.DEFINE_boolean('ignore_difficult_instances', False, 'Whether to ignore '
                     'difficult instances')
FLAGS = flags.FLAGS

SETS = ['train', 'test']
YEARS = ['datasetName ']

def dict_to_tf_example(data,
                       dataset_directory,
                       label_map_dict,
                       ignore_difficult_instances=False,
                       image_subdirectory='JPEGImages'):
  """Convert XML derived dict to tf.Example proto.

  Notice that this function normalizes the bounding box coordinates provided
  by the raw data.

  Args:
    data: dict holding PASCAL XML fields for a single image (obtained by
      running dataset_util.recursive_parse_xml_to_dict)
    dataset_directory: Path to root directory holding PASCAL dataset
    label_map_dict: A map from string label names to integers ids.
    ignore_difficult_instances: Whether to skip difficult instances in the
      dataset  (default: False).
    image_subdirectory: String specifying subdirectory within the
      PASCAL dataset directory holding the actual image data.

  Returns:
    example: The converted tf.Example.

  Raises:
    ValueError: if the image pointed to by data['filename'] is not a valid JPEG
  """
  img_path = os.path.join(r'C:\tensorflow1\models\research\object_detection\data\datasetName', image_subdirectory, data['filename'])
  full_path = os.path.join(dataset_directory, img_path)
  with tf.gfile.GFile(full_path, 'rb') as fid:
    encoded_jpg = fid.read()
  encoded_jpg_io = io.BytesIO(encoded_jpg)
  image = PIL.Image.open(encoded_jpg_io)
  if image.format != 'JPEG':
    raise ValueError('Image format not JPEG')
  key = hashlib.sha256(encoded_jpg).hexdigest()

  width = int(data['size']['width'])
  height = int(data['size']['height'])

  xmin = []
  ymin = []
  xmax = []
  ymax = []
  classes = []
  classes_text = []
  truncated = []
  poses = []
  difficult_obj = []
  if 'object' in data:
    for obj in data['object']:
      difficult = bool(int(obj['difficult']))
      if ignore_difficult_instances and difficult:
        continue

      difficult_obj.append(int(difficult))

      xmin.append(float(obj['bndbox']['xmin']) / width)
      ymin.append(float(obj['bndbox']['ymin']) / height)
      xmax.append(float(obj['bndbox']['xmax']) / width)
      ymax.append(float(obj['bndbox']['ymax']) / height)
      classes_text.append(obj['name'].encode('utf8'))
      classes.append(label_map_dict[obj['name']])
      truncated.append(int(obj['truncated']))
      poses.append(obj['pose'].encode('utf8'))

  example = tf.train.Example(features=tf.train.Features(feature={
      'image/height': dataset_util.int64_feature(height),
      'image/width': dataset_util.int64_feature(width),
      'image/filename': dataset_util.bytes_feature(
          data['filename'].encode('utf8')),
      'image/source_id': dataset_util.bytes_feature(
          data['filename'].encode('utf8')),
      'image/key/sha256': dataset_util.bytes_feature(key.encode('utf8')),
      'image/encoded': dataset_util.bytes_feature(encoded_jpg),
      'image/format': dataset_util.bytes_feature('jpeg'.encode('utf8')),
      'image/object/bbox/xmin': dataset_util.float_list_feature(xmin),
      'image/object/bbox/xmax': dataset_util.float_list_feature(xmax),
      'image/object/bbox/ymin': dataset_util.float_list_feature(ymin),
      'image/object/bbox/ymax': dataset_util.float_list_feature(ymax),
      'image/object/class/text': dataset_util.bytes_list_feature(classes_text),
      'image/object/class/label': dataset_util.int64_list_feature(classes),
      'image/object/difficult': dataset_util.int64_list_feature(difficult_obj),
      'image/object/truncated': dataset_util.int64_list_feature(truncated),
      'image/object/view': dataset_util.bytes_list_feature(poses),
  }))
  return example

def main(_):
  if FLAGS.set not in SETS:
    raise ValueError('set must be in : {}'.format(SETS))
  if FLAGS.year not in YEARS:
    raise ValueError('year must be in : {}'.format(YEARS))

  data_dir = FLAGS.data_dir
  years = ['datasetName']
  if FLAGS.year != 'merged':
    years = [FLAGS.year]

  writer = tf.python_io.TFRecordWriter(FLAGS.output_path)

  label_map_dict = label_map_util.get_label_map_dict(FLAGS.label_map_path)

  for year in years:
    logging.info('Reading from PASCAL %s dataset.', year)
    examples_path = os.path.join(data_dir, year, 'ImageSets', 'Main',
                                  FLAGS.set + '.txt')
    annotations_dir = os.path.join(data_dir, year, FLAGS.annotations_dir)
    examples_list = dataset_util.read_examples_list(examples_path)
    # print(examples_list)
    for idx, example in enumerate(examples_list):
      # print('Step 0', example)
      if idx % 100 == 0:
        logging.info('On image %d of %d', idx, len(examples_list))
      path = os.path.join(annotations_dir, example + '.xml')
      print('Step 1', path)
      with tf.gfile.GFile(path, 'r') as fid:
        xml_str = fid.read()
      xml = etree.fromstring(xml_str)
      data = dataset_util.recursive_parse_xml_to_dict(xml)['annotations']

      tf_example = dict_to_tf_example(data, FLAGS.data_dir, label_map_dict,
                                      FLAGS.ignore_difficult_instances)
      writer.write(tf_example.SerializeToString())

  writer.close()

if __name__ == '__main__':
  tf.app.run()

And this is called from Anaconda Promt:
$ python data\create_pascal_tf_record.py --data_dir=data\ --year=datasetName --set=test --output_path=data\record\test.tfrecord --label_map_path=data\labelmap.pbtxt
Please take a good care of the directory structure. For me, the datasetName sits within a the data folder that contains the create_pascal_tf_record.py (see below), unless you know how to handle these things properly!
|-- data
    |--datasetName
    |--record
    |--create_pascal_tf_record.py

This should work! Although as I mentioned here #43 that even with these TFRecords I have problem training! Let me know how you proceed.

I tried your code and it gives me an error saying that tensorflow has no attribute "app".
From what I've read online Tensorflow 2.0 doesn't use this. Are you using a different version of Tensorflow?
Regardless I solved it by changing tf.app.run() to tf.compat.v1.app.run().
I got similar errors later which needed solutions such as flags = tf.app.flags to flags = tf.compat.v1.app.flags and tf.python_io to tf.compat.v1.python_io and tf.gfile.GFile to tf.io.gfile.GFile.
Now I get this error:
Epoch 1/100 2019-10-02 10:18:53.096213: W tensorflow/core/framework/op_kernel.cc:1622] OP_REQUIRES failed at sparse_xent_op.cc:90 : Invalid argument: Received a label value of -1 which is outside the valid range of [0, 80).  Label values: 0 0 0 0 0 0 0 0 ....
after a lot of zeros and a few -1s I get:
2019-10-02 10:18:53.096300: W tensorflow/core/common_runtime/base_collective_executor.cc:216] BaseCollectiveExecutor::StartAbort Invalid argument: Received a label value of -1 which is outside the valid range of [0, 80).  Label values: 0 0 0.....
This seems to be related to the other issue you mentioned.

@mmortazavi Hi Majid,
Were you able to successfully train?
I'm getting Nan's when training. Any ideas how to solve this?
@cole8888 Hey Cole,
were you able to solve your issue?

@cole8888 Oh sorry I had to mention that I was running on Tensorflow '1.14.0' executing create_pascal_tf_record.py, which I did not need to change many things in the codes.
I wonder if that is the reason the generated TFRecords are not working for training with this repo that is based on Tensorflow 2.
I just tried to create the TFRecords by executing create_pascal_tf_record.py again using Tensorflow 2, specifically '2.0.0-rc1', and I confirm that I get to change a few scripts (one like you mentioned):

tf.app.run() to tf.compat.v1.app.run()
tf.app.flags to tf.compat.v1.app.flags
tf.python_io to tf.compat.v1.python_io
tf.gfile to tf.io.gfile
Note that these changes were not only necessary in the create_pascal_tf_record.py, but also in others in utils that you get to see the error to fix then one by one! Finally running:
$ python data\create_pascal_tf_record.py --data_dir=data\ --year=datasetName --set=test --output_path=data\record\test.tfrecord --label_map_path=data\labelmap.pbtxt

Successfully generated the TFRecords again.
BTW, about that error:

Epoch 1/100 2019-10-02 10:18:53.096213: W tensorflow/core/framework/op_kernel.cc:1622] OP_REQUIRES failed at sparse_xent_op.cc:90 : Invalid argument: Received a label value of -1 which is outside the valid range of [0, 80). Label values: 0 0 0 0 0 0 0 0 ....

Yes, I think this is different. Ahh I used to get this, but unfortunately do not remember how I fixed it! I think I kept searching until I found a suggestion or simply I was missing some changes in files which was not compatible for my dataset (take a closer look)!!
However, I can not train again! The training begins for the first epoch, and after a few mins, it stops with:
[[node loss/yolo_output_2_loss/SparseSoftmaxCrossEntropyWithLogits/SparseSoftmaxCrossEntropyWithLogits (defined at C:\Users\MajidMortazavi\Anaconda3\envs\yolov3-tf2\lib\site-packages\tensorflow_core\python\framework\ops.py:1751) ]] [Op:__inference_distributed_function_40079]

Function call stack:
distributed_function

It seems this is a know issue for Tensorflow '2.0.0-rc*', found this open issue . So @AnaRhisT94 No!!!!

@mmortazavi I see, thank you. I'm still trying to be able to train, on any dataset, doesn't matter, atleast to get it working and continue from there. If you get it working please let me know

@AnaRhisT94 I was unable to resolve my issues and have started looking into other repositories which use Tensorflow 1.15. Apparently Tensorflow 2.0 does not work well with any of the object detection programs right now, you are better off using Tensorflow 1.15. If you do manage to find one that works let me know!

Agreed. Well I have a detector trained based on Google Objecxt Detection API that is working, but inference is slow and I thought I give Yolo a shot. So far I could not make a Yolo based on Tensorflow work. Meanwhile I have found this repo, looked promising, but similarity run into multiple issues and it is not even TF2!

@cole8888 @AnaRhisT94 It took a while but I have YOLOv3 working quite nicely in Tensorflow 2.0. Tfrecords are irritating as they are hard to verify and inspect. Microsoft VoTT (https://github.com/microsoft/VoTT) is the tool I use to create the tfrecords. Although it is a real pain to get working on Linux I think you can use it from the browser and I've heard it is easier to use on Windows. A nice test is to simply generate one tfrecord file and then to train from scratch on that, then test on the training file. 1000 epochs on one image of an object should get it to learn that object image. Once you've established that it's not so hard to get it working for larger datasets

@AntiDoctor I have VoTT installed on my linux enviroment and I am able to label my images but whenever I export them as Tfrecords it creates one for each image. How did you get it to work?

Does the training work for one image? If you look at the function for loading the dataset you'll see that it can handle multiple tf records, simply use dataset.load_tfrecord_dataset and pass it /your_directory/*.tfrecord as your file pattern.

@AntiDoctor using this repo (https://github.com/AntonMu/TrainYourOwnYOLO) @mmortazavi linked a few days ago I was able to train and detect with tensorflow 1.15.0 using CSV labels exported by VoTT. You should be able to get that one working also.

I think I'm facing the same issue.
I get this error when I try to train the model:
tensorflow.python.framework.errors_impl.InvalidArgumentError:  {{function_node __inference_Dataset_map_<lambda>_14279}} Paddings must be non-negative: 0 -13 	 [[{{node Pad}}]] 	 [[IteratorGetNext]] [Op:__inference_distributed_function_34604]
It seems like if the tf record has any error. I've seen that this error appears when the bounding box points are specified with integers, but I have used normalized points with values between 0 and 1 so I don't really know whats the problems.
This is the code that I use to generate each trecord:
`def create_tf_example(label_and_data_info, directory):
path = directory + "/" + label_and_data_info["name"] + ".jpg"
im = Image.open(path)
width, height = im.size
filename = label_and_data_info["name"].encode('utf8') # Filename of the image. Empty if image is not from file
image_format = b'jpeg' # b'jpeg' or b'png'
with tf.gfile.GFile(path, 'rb') as fid:
encoded_jpg = fid.read()
key = hashlib.sha256(encoded_jpg).hexdigest()
xmins = label_and_data_info["xmins"] #list of floats between 0.0 and 1.0
xmaxs = label_and_data_info["xmaxs"] #list of floats between 0.0 and 1.0
ymins = label_and_data_info["ymins"] #list of floats between 0.0 and 1.0
ymaxs = label_and_data_info["ymaxs"] #list of floats between 0.0 and 1.0
classes_text = [x.encode('utf8') for x in label_and_data_info["labels"]] #list of strings ("Car" or "Person")
classes = label_and_data_info["classes"] #list of integers (0 or 1)
tf_label_and_data = tf.train.Example(features=tf.train.Features(feature={
'image/height': dataset_util.int64_feature(height),
'image/width': dataset_util.int64_feature(width),
'image/filename': dataset_util.bytes_feature(filename),
'image/source_id': dataset_util.bytes_feature(filename),
'image/encoded': dataset_util.bytes_feature(encoded_jpg),
'image/key/sha256': dataset_util.bytes_feature(key.encode('utf8')),
'image/format': dataset_util.bytes_feature(image_format),
'image/object/bbox/xmin': dataset_util.float_list_feature(xmins),
'image/object/bbox/xmax': dataset_util.float_list_feature(xmaxs),
'image/object/bbox/ymin': dataset_util.float_list_feature(ymins),
'image/object/bbox/ymax': dataset_util.float_list_feature(ymaxs),
'image/object/class/text': dataset_util.bytes_list_feature(classes_text),
'image/object/class/label': dataset_util.int64_list_feature(classes),
}))
return tf_label_and_data`

I think I'm facing the same issue.
I get this error when I try to train the model:
tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node __inference_Dataset_map_<lambda>_14279}} Paddings must be non-negative: 0 -13 [[{{node Pad}}]] [[IteratorGetNext]] [Op:__inference_distributed_function_34604]
It seems like if the tf record has any error. I've seen that this error appears when the bounding box points are specified with integers, but I have used normalized points with values between 0 and 1 so I don't really know whats the problems.
This is the code that I use to generate each trecord:
`def create_tf_example(label_and_data_info, directory):
path = directory + "/" + label_and_data_info["name"] + ".jpg"
im = Image.open(path)
width, height = im.size
filename = label_and_data_info["name"].encode('utf8') # Filename of the image. Empty if image is not from file
image_format = b'jpeg' # b'jpeg' or b'png'
with tf.gfile.GFile(path, 'rb') as fid:
encoded_jpg = fid.read()
key = hashlib.sha256(encoded_jpg).hexdigest()
xmins = label_and_data_info["xmins"] #list of floats between 0.0 and 1.0
xmaxs = label_and_data_info["xmaxs"] #list of floats between 0.0 and 1.0
ymins = label_and_data_info["ymins"] #list of floats between 0.0 and 1.0
ymaxs = label_and_data_info["ymaxs"] #list of floats between 0.0 and 1.0
classes_text = [x.encode('utf8') for x in label_and_data_info["labels"]] #list of strings ("Car" or "Person")
classes = label_and_data_info["classes"] #list of integers (0 or 1)
tf_label_and_data = tf.train.Example(features=tf.train.Features(feature={
'image/height': dataset_util.int64_feature(height),
'image/width': dataset_util.int64_feature(width),
'image/filename': dataset_util.bytes_feature(filename),
'image/source_id': dataset_util.bytes_feature(filename),
'image/encoded': dataset_util.bytes_feature(encoded_jpg),
'image/key/sha256': dataset_util.bytes_feature(key.encode('utf8')),
'image/format': dataset_util.bytes_feature(image_format),
'image/object/bbox/xmin': dataset_util.float_list_feature(xmins),
'image/object/bbox/xmax': dataset_util.float_list_feature(xmaxs),
'image/object/bbox/ymin': dataset_util.float_list_feature(ymins),
'image/object/bbox/ymax': dataset_util.float_list_feature(ymaxs),
'image/object/class/text': dataset_util.bytes_list_feature(classes_text),
'image/object/class/label': dataset_util.int64_list_feature(classes),
}))
return tf_label_and_data`

I've been able to pass this error (now I'm having others) by defininig mun_classes parameter.

I think I'm facing the same issue.
I get this error when I try to train the model:
tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node __inference_Dataset_map_<lambda>_14279}} Paddings must be non-negative: 0 -13 [[{{node Pad}}]] [[IteratorGetNext]] [Op:__inference_distributed_function_34604]
It seems like if the tf record has any error. I've seen that this error appears when the bounding box points are specified with integers, but I have used normalized points with values between 0 and 1 so I don't really know whats the problems.
This is the code that I use to generate each trecord:
def create_tf_example(label_and_data_info, directory): path = directory + "/" + label_and_data_info["name"] + ".jpg" im = Image.open(path) width, height = im.size filename = label_and_data_info["name"].encode('utf8') # Filename of the image. Empty if image is not from file image_format = b'jpeg' # b'jpeg' or b'png' with tf.gfile.GFile(path, 'rb') as fid: encoded_jpg = fid.read() key = hashlib.sha256(encoded_jpg).hexdigest() xmins = label_and_data_info["xmins"] #list of floats between 0.0 and 1.0 xmaxs = label_and_data_info["xmaxs"] #list of floats between 0.0 and 1.0 ymins = label_and_data_info["ymins"] #list of floats between 0.0 and 1.0 ymaxs = label_and_data_info["ymaxs"] #list of floats between 0.0 and 1.0 classes_text = [x.encode('utf8') for x in label_and_data_info["labels"]] #list of strings ("Car" or "Person") classes = label_and_data_info["classes"] #list of integers (0 or 1) tf_label_and_data = tf.train.Example(features=tf.train.Features(feature={ 'image/height': dataset_util.int64_feature(height), 'image/width': dataset_util.int64_feature(width), 'image/filename': dataset_util.bytes_feature(filename), 'image/source_id': dataset_util.bytes_feature(filename), 'image/encoded': dataset_util.bytes_feature(encoded_jpg), 'image/key/sha256': dataset_util.bytes_feature(key.encode('utf8')), 'image/format': dataset_util.bytes_feature(image_format), 'image/object/bbox/xmin': dataset_util.float_list_feature(xmins), 'image/object/bbox/xmax': dataset_util.float_list_feature(xmaxs), 'image/object/bbox/ymin': dataset_util.float_list_feature(ymins), 'image/object/bbox/ymax': dataset_util.float_list_feature(ymaxs), 'image/object/class/text': dataset_util.bytes_list_feature(classes_text), 'image/object/class/label': dataset_util.int64_list_feature(classes), })) return tf_label_and_data

I've been able to pass this error (now I'm having others) by defininig mun_classes parameter.

Correction: My previous problems were caused because I was trying to train a 2 classes model using the 80 classes pretrained weights.
Now that I have chosed 'none' transfer to train from the scratch after some time training I get the same error again:
Paddings must be non-negative: 0 -13 	 [[{{node Pad}}]] 	 [[IteratorGetNext]] [Op:__inference_distributed_function_39850]

@ancoca13 If I were you I'd switch to a Tensorflow 1.X package, as of writing Tensorflow 2 is not able  to run object detection reliably on all platforms.
I've had success with https://github.com/AntonMu/TrainYourOwnYOLO if you'd like to try it out.

@ancoca13 I literally had this same issue, stop using your own script to generate the tfrecords and use VOTT. Your script is wrong. Problem solved. If you want to reverse engineer a script to create the tfrecord files after you've created them using VOTT you can, but object detection is perfectly reliable on tensorflow 2

Hello,
I found the problem. Some of the images that I was using had a lo of bounting boxes. Specifically there was one image with 113 bounding boxes the one that was failing.
I'm pretty sure that the promem is with the following line in dataset.py
paddings = [[0, 100 - tf.shape(y_train)[0]], [0, 0]]
Thats why I get "Paddings must be non-negative: 0 -13".
As I'm not sure if it would be good to increase that '100' in the code I decided to remove that image fom my tfrecord file and now it works.

I have added --yolo_max_boxes flag and introduced a complete tutorial on training please create new issue if you still have trouble with training.
https://github.com/zzh8829/yolov3-tf2/blob/master/docs/training_voc.md


Full Check

Please implement a full breach check, where "Cloudbleed" and "Have I Been Pwned" are both checked.
In the result window you already have column "Breach Name" where you show "Cloudbleed" or the name of the breach.
Very useful because it is just one action and I see if one entry is affected by multiple breaches.

Thank you for implementing this feature!


[Metrics UI] User Defined Index Patterns in Alerts

Today the Create Alert dialog in Metrics UI looks like this:

The features we implemented in Phase 1 have resonated well with the field, specifically: multiple conditions, filters and separate alert instances. While these features are useful for any metrics coming from any sources, the current implementation only works with Metricbeat data. There are several implications of this:

Alerts can only be built under the assumption that the single source of metrics is Metricbeat. By limiting the sources of metrics we lock the user in the infrastructure metrics silo.
We're contradicting our own story around Unified Alerting, because while the data is there in Elasticsearch, we don't allow the user to combine/correlate it. Users won't be able to build truly unified alerts with conditions spanning across infrastructure metrics, APM metrics, or any fields extracted from logs. E.g. user cases like "I want to be notified when the CPU usage reaches 90% AND the transaction duration for serviceA doubles" can only be build if both Metricbeat and APM data can be simultaneously used in alert conditions.

Use Case Example: New Screen Delivered in a Video Game, they track the screen time for the new screen.
As a Video Game PM, I want to be alerted if the average screen engagement time is below 5s for the last 10m for the new screen I delivered. I want to be alerted by region.
Index: ingame-metrics-*
Condition : avg-screen-time < 5s for the last 10m
Filter By : screen-id : "newscreen-id"
Alert By : region

Today that would require a PM to ask someone to write a watcher... if we had this they could be self service.
source: https://elastic.slack.com/archives/CHSSGF015/p1585860413229500
cc: @simianhacker @phillipb @Zacqary @jasonrhodes @hbharding @sgrodzicki

Alerts can only be built under the assumption that the single source of metrics is Metricbeat.

More specifically they're built under the assumption that the single source of metrics is whatever index pattern you set in your Source Configuration, which will usually be Metricbeat. I believe if you add APM index patterns to your source config, you'll be able to alert on APM metrics.
To recap something we discussed in a planning meeting today: as of 7.7, alerts will be passed a sourceId parameter when they're created, referring to whichever Source Configuration the user has the Metrics Explorer pointing to. In all likelihood the value of the sourceId will always be default, since AFAIK we haven't actually built support for multiple Source Configurations. This issue for Alerting might be a good catalyst for us to decide if we want to go all-in with Source Configurations or not; it might be more natural to select a Source Configuration for an alert rather than to specify a custom index pattern.

Thanks @Zacqary, this is useful info. I just configured Metric indices with an additional endgame-* index, and here's what I got:

Added endgame-* in the Settings

which allowed me to browse metrics from that index in Metrics Explorer.

I was able to create an Alert with the browsed fields

Would the alert rule work if I removed the engdame-* index after creating it?

Would the alert rule work if I removed the engdame-* index after creating it?

So, funny story, yes it would still work, but that's actually a bug that will be fixed when https://github.com/elastic/kibana/pull/62708 merges
We definitely do need to come up with a real UX for defining a different index pattern or source config for alerts.


Backdates API versions of Ingress and PriorityClass for 1.13 compatibility

This PR re-enables backward compatibility with k8s 1.13 (.5 tested), addressing #106

Thanks for testing @almahmoud!


docs: Add firstname.club to the showcase

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Hello @bitswarming,
thank you for adding your website to the showcase of Hugo. Hugo's showcase section should be updated within the next minutes.
Cheers,
Digitalcraftsman


chore: Rename Nat.factors to Nat.primeFactorsList

Nat.factors was a very generic name that didn't emphasize that

It is a list
It is not all factors but only the prime ones

Can you limit this PR to the renaming?

What am I doing that is not the renaming?

I should say that I am not particularly attached to the name primeFactorsList and I would be happy to hear alternatives.

Was a trivial fix
bors merge

bors merge


Example Requested: AzAPI

Hoping for an example on using newres with AzAPI. Is it capable of pulling down any resource using the resource API version?

Thanks @kevball2 for opening this issue, for now this tool doesn't support generating a specified AzAPI resource, now it only reads the resource schema and generate resource block based on that. But as we might adopt AzAPI as AVM's primary provider, I'm thinking of supporting such feature. If we want to do it we have to write a special generator to generate block based on rest api spec.
I'll discuss with my team, personally I think this feature request makes sense to me and is a high valuable one.
Anyone who would like this feature could leave a thumb up here, more thumbs we got, the sooner I can bring this feature to newres.


[tutorial] %HOMEPATH% is missing

I'm following
https://docs.zephyrproject.org/latest/develop/getting_started/index.html
but  %HOMEPATH% doesn't exist in win10 powershell.
(.venv) PS C:\Users\xxx\zephyrproject> pip install -r %HOMEPATH%\zephyrproject\zephyr\scripts\requirements.txt
ERROR: Could not open requirements file: [Errno 2] No such file or directory: '%HOMEPATH%\\zephyrproject\\zephyr\\scripts\\requirements.txt'

It does in cmd, but then I have to reactivate venv.
Would be great to simply avoid using powershell.
Anyway tutorial needs updating

Powershell is not supported, this is detailed on that page:

These instructions must be run in a cmd.exe command prompt. The required commands differ on PowerShell.

If you are using python 3.12 then you need to get python 3.11

?? only constrain mentioned was
no version constrain was specified during: choco install ninja gperf python git dtc-msys2 wget 7zip
so it installed 3.12, now you telling me it's too new?
Tutorial needs updating either way. Currently it seems impossible for someone to install Zephyr on Windows, maybe on linux it's 2lines. I'm simply following the tutorial.

I'll update the guide to document the (hopefully very temporary) limitation to Python <3.12 - I believe at this point we'd only need https://github.com/trezor/cython-hidapi/issues/160 to be fixed.
Regarding the confusion due to some mentions of Powershell, #64726 will be taking care of that.


Generalize environment injection for all mounted elements

This is a suggestion for #195 that reduces code duplication.

Thanks for this!

I don’t know much about this area so I’m leaving it up to y’all


Feature: Add some kind of network graph visualizer

A web front end app that uses lnmessage to query one of the CLN nodes with the network graph.

https://github.com/evansmj/cln-node-visualization


Too slow for low-performance PC for loading the page

Description
my PC, (win10,  11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz   2.42 GHz, 16GB RAM, with Chrome), have very serious performance issue when reading this kind of HTML page. Knowing that the workload for rendering large num of Latex expressions is costly, It would be helpful for taking more performance consideration. e.g. split the HTML page.
(Optional:) Please add any files, screenshots, or other information here.
No response
(Required) What is this issue most closely related to? Select one.
Choose One
Internal issue ID
1e370019-9d3c-4c27-9b7c-f17ded5b8ff5
Paper URL
https://arxiv.org/html/2407.10671v1
Browser
Chrome/<IP_ADDRESS>
Device Type
Desktop

Hi @Shi-Soul ,
You have reported this on a 25-page paper, which is about medium as arXiv documents come. Could you also report the time it takes to load on your machine - maybe in seconds?
https://arxiv.org/html/2407.10671v1
On high bandwidth and a modern CPU that page appears in my browser in under 1 second.
Chrome on a 2GHz CPU should give you comparable rendering speeds to a classic smartphone device ( I would guess ~2-3 secs), so I am wondering if this has to do more with bandwidth (and startup) than rendering time.

It takes me ~4 secs for loading it, with VPN used(it's necessary due to my location).
What's really annoying me is when I slide up and down, the page will stop responding for ~0.5 secs frequently. This problem also happens when I use Chrome to browse other long pages but I don't have a good solution.
Thanks for responding!

The additional lag during navigation is a good detail to add, thank you!


Fix install instructions

These are missing a lil step, and updated to the recent rails idiom

@kimadactyl thanks for contributing! I've merged another PR that changes it to db:setup.


Fillit test case error_03

This should not be in the error file directory  because this is a valid case.
Please update the checker and put the test case in valid cases directory. 😠

@Manmeet2018 You can open the Pull request yourself if you wish.

And this is not the correct repository. Please see https://github.com/anisg/fillit_checker

Dear @Manmeet2018
You are mentioning a behavior that is not intended by 42FC but fillit_checker
Please open an issue on this repository: https://github.com/anisg/fillit_checker/issues


No way to install a subset of zsh completions

Today you can install all completions using znap install ohmyzsh/ohmyzsh. I didn't see a way in the readme or znap help to install only a single completion from a repo.
For example if you want to install only the ripgrep completion from ohmyzsh, running znap install ohmyzsh/ohmyzsh plugins/ripgrep will try to clone plugins/ripgrep, instead of installing only the docker completion.
Having a way to install a selection of completions would allow mixing and matching completions from different repos and places.

running znap install ohmyzsh/ohmyzsh plugins/ripgrep will try to clone plugins/ripgrep, instead of installing only the docker completion.

The syntax for znap install is different from that of znap source. Whereas znap source allows you to specify exactly which files or modules to source from a particular repo, znap install instead lets you install multiple repos in parallel. It currently does not let you pick exactly what to install.
However, there is no harm in installing multiple completions. They won't actually be loaded unless you try to use them. 🙂


Bump Go to 1.21.1

Thanks @C0rWin , it looks like a unit test needs to be updated to match updated error messages in Go 1.21.
The Broken Link Checker failure can be ignored (it only gets run when you update the broken-link-checker.yml file.

Thanks @C0rWin , it looks like a unit test needs to be updated to match updated error messages in Go 1.21.
The Broken Link Checker failure can be ignored (it only gets run when you update the broken-link-checker.yml file).

yep, @denyeart will update unit-test

There are a number of CFT-Deliver tests that are failing in CI, while I cannot spot similar failures locally, e.g. cannot reproduce them. 🤔

/azp run

I've re-run the failed unit test check, and it keeps failing in the same place:
Summarizing 2 Failures:
  [FAIL] CFT-Deliverer the consecutive errors are unbounded and the peer is not a static leader [It] hits the maximum sleep time value in an exponential fashion and retries until exceeding the max retry duration
  /home/runner/work/fabric/fabric/internal/pkg/peer/blocksprovider/deliverer_test.go:278
  [FAIL] CFT-Deliverer the consecutive errors are unbounded and the peer is static leader [It] hits the maximum sleep time value in an exponential fashion and retries indefinitely
  /home/runner/work/fabric/fabric/internal/pkg/peer/blocksprovider/deliverer_test.go:294

Ran 28 of 28 Specs in 2.390 seconds
FAIL! -- 26 Passed | 2 Failed | 0 Pending | 0 Skipped
--- FAIL: TestBlocksProvider (2.39s)
FAIL
coverage: 77.4% of statements
FAIL	github.com/hyperledger/fabric/internal/pkg/peer/blocksprovider	2.775s

Like @C0rWin I can't reproduce when testing locally with Go 1.21.1, the tests pass on my mac.
I don't understand the failure in CI. @tock-ibm can you take a look and provide your thoughts? I think you are familiar with these tests...

I've re-run the failed unit test check, and it keeps failing in the same place:
Summarizing 2 Failures:
  [FAIL] CFT-Deliverer the consecutive errors are unbounded and the peer is not a static leader [It] hits the maximum sleep time value in an exponential fashion and retries until exceeding the max retry duration
  /home/runner/work/fabric/fabric/internal/pkg/peer/blocksprovider/deliverer_test.go:278
  [FAIL] CFT-Deliverer the consecutive errors are unbounded and the peer is static leader [It] hits the maximum sleep time value in an exponential fashion and retries indefinitely
  /home/runner/work/fabric/fabric/internal/pkg/peer/blocksprovider/deliverer_test.go:294

Ran 28 of 28 Specs in 2.390 seconds
FAIL! -- 26 Passed | 2 Failed | 0 Pending | 0 Skipped
--- FAIL: TestBlocksProvider (2.39s)
FAIL
coverage: 77.4% of statements
FAIL	github.com/hyperledger/fabric/internal/pkg/peer/blocksprovider	2.775s

Like @C0rWin I can't reproduce when testing locally with Go 1.21.1, the tests pass on my mac.
I don't understand the failure in CI. @tock-ibm can you take a look and provide your thoughts? I think you are familiar with these tests...

Both tests are fairly simple they just make sure that backoff behaves as expected and assert later the amount of sleep calls to see whenever backoff stops if the condition met or keeps trying.

Hey, @denyeart, I managed to make it pass. The tests were too slow and lacked time to complete, hence the error.


Added on to the #104 pull request.

I just made the wiki entry Ryan Bates entered a file, and added it to the symlinked list for relish. Hope it works :)

Thanks!


optimised small collections

One of the improvements made in clojure 1.8 was incorporating some work done by Zach Tellman to create 'unrolled' variants of maps and vectors. Essentially, it was a piece of clojure that generated an optimised java class for each size of vector and map up to N (I think N was five or six) so that that costs of creating and querying small vectors and maps was faster. When an operation would make them too big, they turn into the more algorithmically optimal type.
We already have an array and guidance to use it for up to ~100 items but we don't have anything for maps and sets. There is also associated code change when moving from arrays to vector / flex_vector. It would be nice to have a uniform API for this like clojure.
A reasonable first stab at it might just be a small_map that maintained an array of pairs and simply iterated through it instead of doing a hash lookup.
Obviously I haven't done any benchmarking at all and this was work done to clojure's collections which may not directly carry over to immer, but food for thought nonetheless. If there is a performance improvement to be made on small collections, it could make considerable differences in a dynamic language environment where maps are often used for argument passing.

I am aware of that optimization. Because of static typing it might be counter-productive to implement it in C++ (type-erasing the underlying impl means adding overhead that I suspect would in most cases shadow the potential gains). I'll keep this open though in case we find in the future particular workloads where this would make a difference.

And thanks for the suggestion!

I see your point with the auto-promotion, but I think a small map type may be beneficial. You know your benchmark results better than me, of course.

I see your point with the auto-promotion, but I think a small map type
may be beneficial.

Yes, you are right.  It should be relatively easy to hack on top of
immer::array and see what benchmarks show.  I leave this open for
later---next step is map/set transients.


where is generate-shell.nix to be found?

Issue description
in doc/languages-frameworks/r.section.md it is not clear where the user should find a file called generate-shell.nix used in the following:
nix-shell generate-shell.nix

This is a documentation issue.

The docs got moved from development/r-modules, where the referenced code resides, so that explains why it's confusing. Thinking about the right doc edits...

This issue has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/2022-10-22-documentation-team-meeting-notes-12-nixcon-edition/22689/1

Looks like this was documented in #139312


Better date defaults

Right now the date picker defaults to showing the entire duration of data we have, which probably isn't the ideal default. Open to opinions/ideas on this, but how does placing the slider showing a max of 2 years sound?
Related: issues created is missing the slider.

Broken issuesCreated slider into issue #46.


PermissionError

G:\data\specific\Wikipedia\PlainTextWikipedia-main>python jsonify_wikipedia.py
20210401
Traceback (most recent call last):
  File "G:\data\specific\Wikipedia\PlainTextWikipedia-main\jsonify_wikipedia.py", line 106, in <module>
    main(file)
  File "G:\data\specific\Wikipedia\PlainTextWikipedia-main\jsonify_wikipedia.py", line 83, in main
    with open(archive_dir + file, 'r', encoding='utf-8') as infile:
PermissionError: [Errno 13] Permission denied: 'G:/data/specific/Wikipedia/data/20210401'

G:/data/specific/Wikipedia/data/20210401 is existing folder. What's wrong? Running as admin tried too

Looks like you need to add ".xml" or check the filename variable.

My problem was in output folder lying inside input folder. This case processed with this code:
104 line: if 'bz2' in file or os.path.isdir(archive_dir + file):


ClipMenu Snippets

I have a ton of clipmenu snippets I've collected over the years. Clippy seems to be a newer version of Clipmenu -- but I can't import the older Snippets.xml into clippy -- seems the xml has DRASTICALLY changed. Is there a tool to convert clipmenu snippets to work with Clippy?

I am not sure about that. could you provide a simple Snippets.xml to learn its format?


Autoload expensive files

hey @weppos plz take a look on this PR. Btw, what motivates me to autoload these files is that I need to require it on Gemfile but not necessarily gonna use it in that moment (useful when testing other parts of your app).

@rainerborene are you able to provide benchmarks before/after?

Closing due to no follow up. There are mixed opinion about autoload in the Ruby community, and without benchmarks I'm not eager to merge this PR.


prezto_install rake task

Remove zpreztorc twice symlink, avoid post-install edits on zshrc

Hey @nandalopes I don't get what was the problem and hence what are you fixing. There's a comment in the code saying that is needed as a workaround, etc... Could you elaborate how with your code this is not needed anymore? Thanks!

zpreztorc is linked twice on Rakefile, one at line 256 and another at line 269
I moved zshrc file to the prezto-override folder, now it's not necessary to edit zshrc after symlink.

Thank you, Nanda. Could you confirm how you tested this? I remember this was very finnicky to test unless you completely uninstalled it and installed afresh. Could you confirm you tested it from scratch and it worked as expected? Thanks!

I've tested with Ubuntu Bionic and docker, first the master branch, and then update with prezto-install branch.
After I've tested directly prezto-install from scratch.
It's working as expected in both tests.
PS: I've merged locally a modified Dockerfile and PR-855, cause current docker-compose build dotfiles stops at vim plugin installation

I've added the modified setup for test on #858 PR

Terrific. Thanks again!


Run services with supervisor

With this change, most of the services are run via supervisor, so they'll be automatically restarted if they die. Fixes #434 and #575
Note that to run the processes with supervisor I had to use the 'run in foreground' version of their start commands, rather than the init.d script that was used before.  @tomav you should verify that I haven't lost any special configurations you had for those services.
Also, I added tests, but I'm having an error actually running the automated test suite on my machine. The problem seems unrelated to my changes. When I run make, the container mail fails to stay up. It looks like its start-mailserver.sh is exiting with this error:
Check that hostname/domainname is provided or overidden (no default docker hostname/kubernetes) [_check_hostname]
  * Setting hostname/domainname is required

Is any special environment required to run make?

@LukeAllen this is a nice step forward. Supervisor does a good job managing processes, let me know if I can help in any way.

@gegere thanks! If you could figure out what went wrong in the automated test, that would be really helpful. It looks like the TravisCI automated test is failing at a different point than on my machine, so I'm kind of confused.

Restarted build.

I reproduced the issue locally. Will try to have a look.

Note that there's no more mail.log output in foreground (accessible using docker logs), which was a good thing to debug. (this is important otherwise DMS_DEBUG has no interest)

Problem seems related to Dovecot. I tried multiple times and its not always the same container which stops. But I can always see the same thing in the logs:
Fatal: Dovecot is not running (read from /var/run/dovecot/master.pid)

The output of start-mailserver.sh goes to stdout now, so it will be visible. (I had been redirecting the output to /var/log/container-startup.log which is why it wasn't showing. Now it goes both places.)

I guess services have to be launched in background mode, and supervisor in foreground.
start-mailserver.sh probably can't get exit 0 from foreground startups and hangs.
Just my 2 cents.

Yeah people usually run supervisor in the foreground for docker containers, but it can run in the background too. I ran it in the background to stay compatible with your existing startup script.
I replaced the old init.d calls with supervisorctl start appname which does immediately return exit 0. It does return before the app has finished starting though. I'm not sure if that breaks some assumption of the tests or the startup script. I can't figure out where the test is actually failing; is there a way to make the TravisCI tests run with DMS_DEBUG=1?

You can add DMS_DEBUG=1 in the Makefile

Hi @LukeAllen, any update on this?

I fixed the dovecot problem. The issue was that in some configurations, start-mailserver.sh starts dovecot, writes some config files, then immediately calls dovecot reload. The dovecot reload broke because supervisor hadn't finished starting dovecot. I modified it to just write all config files before starting dovecot. So now all the tests run, but there are new failures.

HI @LukeAllen, it seems that a lot of tests are not passing
https://travis-ci.org/tomav/docker-mailserver/builds/250677415?utm_source=github_status&utm_medium=notification

@LukeAllen do you need help getting this done?
I think we should check whether the tasks could run in foreground mode or not. If so (notifications and errors are mapped to stdout and stderr) we should set the following to each supervisord definition:
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0

Benefit would be that we would have 1 interface for all logging informations.

@alinmear good idea; I incorporated your logging suggestion. Thanks for the help offer; yeah I could use some help. I'm having trouble understanding these tests. I've verified that the processes are actually all running, and the container can send mail, so my guess is that some of the tests might make assumptions that aren't true any more. Not sure though.

@LukeAllen will try to check this on the weekend!

Another possibility would be to use baseimage-docker by phusion which was designed specifically for this purpose (and other enhancements). Maybe they've already solved the problem and we don't need to reinvent the wheel?
I would be willing to invest some time into this. The docker image is perfect for running a mailserver, but I am unsure about dependability and resiliance against service failures...

Hi all,
I have rebased the changes from @LukeAllen  on the new master branch and did some switching of the flow.
Currently I have almost all tests working nicely and have only 8 errors (ldap/log related).
Could someone take a review of my changes on top of the @LukeAllen branch issue-631-run-services-with-supervisor and give out some ideas if some changes are needed before I wrap it up?
When done I need to make a new pull request or what is the procedure of it?

Solved all tests, see PR #676 because I could not make a PR on this PR.

Awesome thanks @johansmitsnl; I was having trouble understanding those tests. Closing this PR now since PR #676 incorporates it.


[BEAM-14113] Improve SamzaJobServerDriver extensibility

Add protected constructor for SamzaJobServerDriver so it can be extended as parent class.
Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

@xinyuiscool @dxichen Can you take a look?


Implement changes in review request

@Jiltseb

Also please add changes for: "comment/remove unnecessary variable"


Feat/more tests

Coverage increased (+19.7%) to 37.736% when pulling 491eb4e59fcc164c791cc0df8693ad676825f9e9 on feat/more-tests into 6504dc439b47942146b6e0e160d69a998867cbe3 on master.


Fix : tab reloading when changed

Thanks for the contribution!
I'd love to merge this, but could you please filter out the extra changes? I see some additional print statements and changes to things like local.properties, plus a bunch of changes to whitespace. Your git client should be able to let you select the exact chunk of code that you're contributing.
Thanks again, and sorry for the delay. Somehow the github notification didn't reach my inbox.

Yes, I have resolved all the conflicts.

Sorry for being difficult, but I still see whitespace changes in 4 files: https://github.com/filiph/hn_app/pull/9/files.


fix: brew install path and chocolatey roadmap

Signed-off-by: Moriarty<EMAIL_ADDRESS>

See my comment here: https://github.com/animo/siera/issues/235#issuecomment-1288761252
brew tap animo/siera && brew install siera work fine, you just have to remove your old animo/agent-cli tap.


[Question] Performance single big list vs key per item

Question
I have a question about performance.
I have a list of up to 50000 items.
The list is read a lot but nearly never written.
My question is:
Is there a performance difference if I save the list as one big list with one key and load it into memory before accessing it vs storing each item with a key and use the „get“ function?
I need to access the items by key only.
TLDR: how much slower are read operations if the box has a lot of keys compared to accessing the list from memory? Is it negligible?
What impacts performance of box reads in general?
Version

Platform: Android
Flutter version: 1.22.3
Hive version: 1.4.4

I 'm wondering the same thing. So 50-60 mb data can be stored per key like in 30-40 key? which means ~2400 MB?

I write a simple benchmark for testing single key / multiple key read / writes. Read/writing with single key preforms better than multiple keys.
Benchmark code

import 'dart:async';
import 'dart:math';

import 'integration.dart';

Future<int> benchmark(String what, FutureOr<void> Function() cb) async {
  var start = DateTime.now();
  await cb();
  var delta = DateTime.now().difference(start);
  print('$what took ${delta.inMilliseconds}ms');
  return delta.inMilliseconds;
}

void main() async {
  var box1 = await openBox<double>(false);
  var box2 = await openBox<List<double>>(false);
  var random = Random.secure();

  print('Generating items...');
  var count = 500000;
  var list = List<double>.generate(count, (index) => random.nextDouble());

  var bench1 = 0, bench2 = 0;
  var total1 = 0.0, total2 = 0.0;

  bench1 += await benchmark('Writing multiple keys', () => box1.addAll(list));
  bench2 += await benchmark('Writing single key', () => box2.put(1, list));

  await box1.close();
  await box2.close();

  bench1 += await benchmark('Opening w/ multiple keys', () async {
    box1 = await box1.reopen();
  });
  bench2 += await benchmark('Opening w/ single key', () async {
    box2 = await box2.reopen();
  });

  var randomKeys = List<int>.generate(
    count ~/ 1000,
    (index) => random.nextInt(count - 1),
  );

  bench1 += await benchmark('Randomly reading multiple keys', () async {
    for (var key in randomKeys) {
      total1 += (await box1.get(key)) as double;
    }
  });
  bench2 += await benchmark('Randomly reading single key', () async {
    var list = await box2.get(1);
    for (var key in randomKeys) {
      total1 += list[key] as double;
    }
  });

  bench1 += await benchmark('Reading multiple keys', () async {
    for (var i = 0; i < count; i++) {
      total1 += (await box1.get(i)) as double;
    }
  });
  bench2 += await benchmark('Reading single key', () async {
    var list = await box2.get(1);
    for (var i = 0; i < count; i++) {
      total2 += list[i] as double;
    }
  });

  print('Equals: ${total1 == total2}');
  print('Multiple keys: ${bench1}ms');
  print('Single key: ${bench2}ms');
}

Results:
Generating items...
Writing multiple keys took 1773ms
Writing single key took 30ms
Opening w/ multiple keys took 1810ms
Opening w/ single key took 29ms
Randomly reading multiple keys took 4ms
Randomly reading single key took 0ms
Reading multiple keys took 1196ms
Reading single key took 6ms
Equals: false
Multiple keys: 4783ms
Single key: 65ms


ImagesTable: add error status

Add error information for failed image builds. The error details are only displayed
for failed builds and the error highest up the error chain is displayed in the expandable
section of the image.


Add optional 'Next of Kin' fields to registration page

Resolves https://github.com/Australian-Genomics/CTRL/issues/57.

It seems a little odd to me that the next of kin phone is a required field but the subject's phone is not.

You're right. I think I thoughtlessly copied/pasted that from the personal details edit page. Fix it now...
The other points you make are good too. I raised a ticket (https://github.com/Australian-Genomics/CTRL/issues/59).

@rosiejbrown What should we do about the quirk @mathob identified? Is it even a quirk? The "Next of Kin" were required to edit the user's personal details even before I added them to the registration page. But is that what we want?

@DoxasticFox Just to clarify, does this mean that when the NOK is switched 'on' (True) it appears in the registration page and is a compulsory field. When it is 'off' (false) the NOK part doesn't appear on the registration page, but is still a compulsory component of the 'extra personal information' they are required to enter off of the dashboard?
If so I think this is correct, the NOK should probably still be compulsory on the extra personal details page.

@rosiejbrown Yep, your understanding is correct.


Added profile to AboutUs

Added profile photo and role to AboutUs. Fixes #19

Nice photo. Thanks for volunteering to take on Documentation.


Follow cartographer-project/cartographer#1750.

Signed-off-by: Wolfgang Hess<EMAIL_ADDRESS>

FYI @bochen87 @clalancette


version for requests dependency

I had a look for a history or rationale for the restriction on version <2.16 for the requests dependency in setup.py and couldn't find anything, nor could I see anything in the release history of the requests library for versions 2.17+ that would explain it.
Would it be possible to expand this to include up to <=2.19 for compatibility with huggingface datasets library?

Yes, I think we can do this.  We've had some dependency issues with some of spacy's dependencies, but I will assess if this is feasible.

@gkennos Sorry it took some time to come back with a response, but we have removed the dependency for requests from our setup.py.  It should be possible to use later versions like you mention is you use medspacy 1.0.0.
If you still have issues with doing this in medspacy 1.0.0, could you please re-open this ticket?


Can't create a 2nd VM in one cloud service

https://github.com/Azure/azure-sdk-for-python/issues/362

Lets get this into 2.0 please


Basic swap contract

Description
Contract
This adds a simple swaps contract that allows a participant to offer FA2 tokens exchange and then another participant to accept it.
The specification is in this document.
There still are some open questions, any input on them is welcome:

[ ] Whether do we want to preserve swap statuses in storage or not?
(to discuss with Jacob, by default I'll remove them)
[ ] Whether do we want to group offered/requested assets by from_ since FA2 interface does so?
I think that even if this increases performance, that will be a little save taking very expensive get_entrypoint_ops calls into account, so we'd better prefer a simpler interface.
[x] Whether do we want to forbid empty offered assets or empty requested assets?
I tend to think that it's worth forbidding since empty assets (on both sides) most probably mean a user's mistake.
And offering/requested tokens without asking/proposing anything in exchange, even if is useful in some cases, seems like functionality for another contract.
Though this also can be seen as an extra check that is better to be done off-chain where it will be free.
Answer: better not to perform this check.

Tests
As discussed, we covered the contract with cleveland tests.
As this is the first cleveland code in this repository and it is far not the main part of the repository, I aimed at using as simple infrastructure/code as possible.
There is one technical controversial point here related to how addresses and other entries are allocated, so it would be nice to decide whether "Rewrite setup type to use fixed-len lists" commit is good or we better revert it for simplicity.

[ ] Add summary of swaps contract to README
[ ] Swap Contract specification page (similar to marketplace & auction examples)

Hey my only comment is that we have been trying to merge features into develop and then merge into main from develop with tags. Maybe it would be best to open a new MR and close/reference this one (unless there is an easy way to switch the target branch that I am not aware of). Otherwise looks great.

Got it :+1:
Looks like I managed to update the PR in-place (rebased on origin/develop and edited the base branch of this PR).

@jacobarluck Could you check whether this summary is enough: updated README.
Also, do you want me to inline the specification into swap's README, or just providing the link to the spec as done now is enough?

I'm going to clean up and merge this in few hours because of upcoming deadlines, but feel free to leave any comments, I'll resolve them in a separate PR.

@jacobarluck Could you check whether this summary is enough: updated README.
Also, do you want me to inline the specification into swap's README, or just providing the link to the spec as done now is enough?

Yes please inline the specification similar into Swap's README similar to this one: https://github.com/tqtezos/minter-sdk/tree/main/contracts/ligo/src/english_auction


Counter for chip deaths

Add a counter for chip deaths in the blackboard and let it be increased in the harvest task if the crypto is dead. Return the status in the GET crypto endpoint

This will of course not be persistent across restarts but I think that is good enough.


Expected errors due to 2 layers svg

Any direct interaction with Series is now impossible
Legend' onClick , useHide are not working https://github.com/zakodium/react-plot/issues/367 (bug fixed)

Any direct interaction with Series is now impossible

Do we have this kind of interaction somewhere?

Do we have this kind of interaction somewhere?

no, but we may need to consider that in future releases


Blueprint for VPC Lattice

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

What is the outcome that you are trying to reach?
I would like to see an example of how I can provision VPC Lattice and use the API gateway controller to connect services across EKS clusters and possibly with Lambda/ECS based services.
Describe the solution you would like
A terraform pattern that provisions an architecture similar to the one described here.
Describe alternatives you have considered
Manual setup as described in the blog.
Additional context
Related: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/issues/94

@askulkarni2  I would be happy to work on this one ?

@neelaruban are you still interested in working on this?

@Tim-AWS  yes

@neelaruban I've got also something working here, can we sync on what you plan to deliver also ?

@allamand sure

Nice i'll look at it.