Styleguide won't open with error - "<User: >" needs to have a value for field "id" before this many-to-many relationship can be used.

Found a bug? Please fill out the sections below. 👍
Issue Summary
The Wagtail Styleguide does not open in my current install of Wagtail. It did work for us initially, but at some point a change in our project must have broken it (I've been scratching my head as to what did it for a while now). The issue has persisted from Wagtail version 2.3 all the way up to 2.5.1.
Steps to Fix
I was able to fix this with one line in /wagtail/contrib/styleguide/views.py, line 75:
Change user =<EMAIL_ADDRESS>to user = User(id=0).
It's hard to pin if this is a problem with our database or with Django 2.0's User class (from django.contrib.auth.models). The only way I got the error to go away was to specify an ID - simply changing the email to a user that exists in our DB did not work.
I realize my fix won't work for everyone - this code assumes you've got a user with an ID of "0" in your database, which won't work for anyone who has deleted that user. Hopefully this helps anyone out there who ran into this and needs a temporary fix for their local machine. I'm in the middle of customizing the admin template for my site and figured my time was better spent on CSS than debugging this further.

I have confirmed that this issue can be reproduced as described on a fresh Wagtail project: No

Technical details

Python version: 3.6.8
Django version: 2.0
Wagtail version: 2.5.1
Browser version: Firefox 60

Debug Output
Request Method: | GET
-- | --
http://localhost:8000/admin/styleguide/
2.0
ValueError
"<User: >" needs to have a value for field "id" before this many-to-many relationship can be used.
/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/db/models/fields/related_descriptors.py  in __init__, line 795
/Users/juan0tron/.virtualenvs/wagtail-project/bin/python
3.6.8
['/Users/juan0tron/Documents/wagtail-project',  '/Users/juan0tron/.virtualenvs/wagtail-project/lib/python36.zip',  '/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6',  '/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/lib-dynload',  '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6',  '/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages',  '/Users/juan0tron/.virtualenvs/wagtail-project/src/django-solo',  '..',  '/var/www/app']
Fri, 19 Jul 2019 19:01:20 +0000

Environment:

Request Method: GET
Request URL: http://localhost:8000/admin/styleguide/

Django Version: 2.0
Python Version: 3.6.8
Installed Applications:
['color',
 'download',
 'export',
 'logo',
 'metric',
 'home',
 'images',
 'page',
 'search',
 'wagtail.contrib.forms',
 'wagtail.contrib.modeladmin',
 'wagtail.contrib.redirects',
 'wagtail.contrib.settings',
 'wagtail.embeds',
 'wagtail.sites',
 'wagtail.users',
 'wagtail.snippets',
 'wagtail.documents',
 'wagtail.images',
 'wagtail.search',
 'wagtail.admin',
 'wagtail.core',
 'wagtail.contrib.postgres_search',
 'modelcluster',
 'taggit',
 'wagtailfontawesome',
 'django_cleanup.apps.CleanupConfig',
 'django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'storages',
 'rest_framework',
 'wagtail.contrib.styleguide',
 'bakery',
 'wagtailbakery']
Installed Middleware:
['django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware',
 'django.middleware.security.SecurityMiddleware',
 'wagtail.core.middleware.SiteMiddleware',
 'wagtail.contrib.redirects.middleware.RedirectMiddleware']

Template error:
In template /Users/juan0tron/Documents/wagtail-project/wagtail-app/templates/wagtailadmin/skeleton.html, error at line 0
   "<User: >" needs to have a value for field "id" before this many-to-many relationship can be used.
   1 : <!doctype html>
   2 : {% load static i18n %}
   3 : {% get_current_language as LANGUAGE_CODE %}
   4 : <html class="no-js" lang="{{ LANGUAGE_CODE }}">
   5 : <head>
   6 :     <meta charset="utf-8" />
   7 :     <title>Wagtail - {% block titletag %}{% endblock %}</title>
   8 :     <meta name="description" content="" />
   9 :     <meta name="viewport" content="width=device-width, initial-scale=1.0" />
   10 :     <meta name="robots" content="noindex" />

Traceback:

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/template/base.py" in _resolve_lookup
  835.                     current = current[bit]

During handling of the above exception ('User' object is not subscriptable), another exception occurred:

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/core/handlers/exception.py" in inner
  35.             response = get_response(request)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
  128.                 response = self.process_exception_by_middleware(e, request)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
  126.                 response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/views/decorators/cache.py" in _wrapped_view_func
  44.         response = view_func(request, *args, **kwargs)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/wagtail/admin/urls/__init__.py" in wrapper
  102.             return view_func(request, *args, **kwargs)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/wagtail/admin/decorators.py" in decorated_view
  34.             return view_func(request, *args, **kwargs)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/wagtail/contrib/styleguide/views.py" in index
  82.         'user': user,

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/shortcuts.py" in render
  36.     content = loader.render_to_string(template_name, context, request, using=using)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/template/loader.py" in render_to_string
  62.     return template.render(context, request)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/template/backends/django.py" in render
  61.             return self.template.render(context)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/template/base.py" in render
  175.                     return self._render(context)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/test/utils.py" in instrumented_test_render
  98.     return self.nodelist.render(context)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/template/base.py" in render
  943.                 bit = node.render_annotated(context)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/template/base.py" in render_annotated
  910.             return self.render(context)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/template/loader_tags.py" in render
  155.             return compiled_parent._render(context)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/test/utils.py" in instrumented_test_render
  98.     return self.nodelist.render(context)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/template/base.py" in render
  943.                 bit = node.render_annotated(context)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/template/base.py" in render_annotated
  910.             return self.render(context)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/template/loader_tags.py" in render
  155.             return compiled_parent._render(context)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/test/utils.py" in instrumented_test_render
  98.     return self.nodelist.render(context)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/template/base.py" in render
  943.                 bit = node.render_annotated(context)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/template/base.py" in render_annotated
  910.             return self.render(context)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/template/loader_tags.py" in render
  155.             return compiled_parent._render(context)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/test/utils.py" in instrumented_test_render
  98.     return self.nodelist.render(context)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/template/base.py" in render
  943.                 bit = node.render_annotated(context)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/template/base.py" in render_annotated
  910.             return self.render(context)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/template/loader_tags.py" in render
  155.             return compiled_parent._render(context)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/test/utils.py" in instrumented_test_render
  98.     return self.nodelist.render(context)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/template/base.py" in render
  943.                 bit = node.render_annotated(context)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/template/base.py" in render_annotated
  910.             return self.render(context)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/template/loader_tags.py" in render
  155.             return compiled_parent._render(context)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/test/utils.py" in instrumented_test_render
  98.     return self.nodelist.render(context)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/template/base.py" in render
  943.                 bit = node.render_annotated(context)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/template/base.py" in render_annotated
  910.             return self.render(context)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/template/loader_tags.py" in render
  155.             return compiled_parent._render(context)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/test/utils.py" in instrumented_test_render
  98.     return self.nodelist.render(context)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/template/base.py" in render
  943.                 bit = node.render_annotated(context)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/template/base.py" in render_annotated
  910.             return self.render(context)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/template/loader_tags.py" in render
  155.             return compiled_parent._render(context)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/test/utils.py" in instrumented_test_render
  98.     return self.nodelist.render(context)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/template/base.py" in render
  943.                 bit = node.render_annotated(context)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/template/base.py" in render_annotated
  910.             return self.render(context)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/template/loader_tags.py" in render
  67.                 result = block.nodelist.render(context)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/template/base.py" in render
  943.                 bit = node.render_annotated(context)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/template/base.py" in render_annotated
  910.             return self.render(context)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/template/loader_tags.py" in render
  67.                 result = block.nodelist.render(context)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/template/base.py" in render
  943.                 bit = node.render_annotated(context)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/template/base.py" in render_annotated
  910.             return self.render(context)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/template/defaulttags.py" in render
  161.                 values = self.sequence.resolve(context, True)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/template/base.py" in resolve
  676.                 obj = self.var.resolve(context)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/template/base.py" in resolve
  802.             value = self._resolve_lookup(context)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/template/base.py" in _resolve_lookup
  843.                         current = getattr(current, bit)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/db/models/fields/related_descriptors.py" in __get__
  498.         return self.related_manager_cls(instance)

File "/Users/juan0tron/.virtualenvs/wagtail-project/lib/python3.6/site-packages/django/db/models/fields/related_descriptors.py" in __init__
  795.                                  (instance, self.pk_field_names[self.source_field_name]))

Exception Type: ValueError at /admin/styleguide/
Exception Value: "<User: >" needs to have a value for field "id" before this many-to-many relationship can be used.

Hi @juan0tron - thanks for the report.
The line<EMAIL_ADDRESS>or User(id=0) isn't retrieving a user from the database - it's building a temporary in-memory User instance specifically for the purpose of displaying its avatar. As such, it's hard to tell why User(id=0) fixes the issue in your case - although I can see how there might be potential problems if a custom user model is in use, since it's currently hard-coding a reference to the default User model from django.contrib.auth.
Having said that, the code in question is somewhat redundant at this point... previously it would have displayed<EMAIL_ADDRESS>real avatar as registered on gravatar.com, but it seems that isn't active any more, and it's just displaying the default Gravatar icon instead - so we might as well skip that whole step and simply hard-code the default icon instead. I'll open a PR to do that now.

PR added: #5452

Fixed via https://github.com/wagtail/wagtail/commit/44a1e6f9f8906d27b82c409549744a16c604ee43


chore(deps): upgrade dependencies

Upgrades project dependencies. See details in workflow run.

Automatically created by projen via the "upgrade" workflow

AWS CodeBuild CI Report

CodeBuild project: GitHubPipelineAutoBuildProj-XK8ItpxnSow5
Commit ID: 653f65f95d2d1c9bebe15ecd15cb024e6fddc684
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository


Both Azure Core SDK and Azure Storage SDK are supposed to be portable. So we cannot assume that on every platforms with all kinds of hardware configuration, the stack size is at least 16KB.

Both Azure Core SDK and Azure Storage SDK are supposed to be portable. So we cannot assume that on every platforms with all kinds of hardware configuration, the stack size is at least 16KB.
Originally posted by @JinmingHu-MSFT in https://github.com/Azure/azure-sdk-for-cpp/pull/214

(Change assignee)
This is in storage ut code, so it should be handled by storage team.

fixed in https://github.com/Azure/azure-sdk-for-cpp/pull/213


Cache lockfile already exists

Using the train_and_evaluate method from estimator API and cache policy on filesystem, I get an error because the evaluation starts before that all cache is written on the filesystem. So when the train runs the second time, after the evaluation, I get the error because it finds the lock file.
If the cache is written before the first evaluation I don't get the error.
Here a snippet runnable on colab
import tensorflow as tf
import numpy as np
from tensorflow import keras as ks
import itertools

def batch_reshape(a, b):
    a.set_shape([None, None, 3])
    b.set_shape([None, None, 1])
    return (a, b)

def gen(n):
    a = np.array(np.random.rand(5, 5, 3).astype(np.float32))
    b = np.array(np.random.rand(5, 5, 1).astype(np.float32))
    return (a, b)

def my_input_fn_train():
    ds = tf.data.Dataset.range(100000)
    ds = ds.map(
        lambda x: tf.py_func(func=gen, inp=[x], Tout=[tf.float32, tf.float32]))
    ds = ds.map(batch_reshape)
    ds = ds.cache("/tmp/mycache_train")

    ds = ds.repeat(3)

    value = ds.make_one_shot_iterator().get_next()

    return {"input_rgb": value[0]}, {"softmax": value[1]}

def my_input_fn_eval():
    ds = tf.data.Dataset.range(100000)
    ds = ds.map(
        lambda x: tf.py_func(func=gen, inp=[x], Tout=[tf.float32, tf.float32]))
    ds = ds.map(batch_reshape)
    ds = ds.cache("/tmp/mycache_eval")

    ds = ds.repeat(3)

    value = ds.make_one_shot_iterator().get_next()

    return {"input_rgb": value[0]}, {"softmax": value[1]}

def main():
    tf.logging.set_verbosity(tf.logging.INFO)
    input_rgb = ks.layers.Input(shape=(1, 5, 5, 3), name="input_rgb")
    x = ks.layers.Dense(1, activation='relu', name="Dense_1")(input_rgb)
    x = ks.layers.Dense(1, activation='softmax', name="softmax")(x)
    model = ks.models.Model(inputs=[input_rgb], outputs=[x])
    model.compile(
        loss={'softmax': 'binary_crossentropy'},
        optimizer=tf.keras.optimizers.Adam())

    estimator = ks.estimator.model_to_estimator(keras_model=model)

    train_spec = tf.estimator.TrainSpec(input_fn=my_input_fn_train)

    eval_spec = tf.estimator.EvalSpec(
        input_fn=my_input_fn_eval, steps=5, throttle_secs=2)

    tf.estimator.train_and_evaluate(estimator, train_spec, eval_spec)

if __name__ == "__main__":
    main()

and here the output
INFO:tensorflow:Using the Keras model from memory.
INFO:tensorflow:Using default config.
WARNING:tensorflow:Using temporary folder as model directory: /tmp/tmpd1r865ry
INFO:tensorflow:Using config: {'_model_dir': '/tmp/tmpd1r865ry', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': None, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fcb48f9ecf8>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
INFO:tensorflow:Running training and evaluation locally (non-distributed).
INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after 2 secs (eval_spec.throttle_secs) or training is finished.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from /tmp/tmpd1r865ry/keras_model.ckpt
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Saving checkpoints for 1 into /tmp/tmpd1r865ry/model.ckpt.
INFO:tensorflow:loss = 9.25762, step = 1
INFO:tensorflow:global_step/sec: 612.893
INFO:tensorflow:loss = 9.348354, step = 101 (0.166 sec)
INFO:tensorflow:global_step/sec: 952.69
INFO:tensorflow:loss = 8.319871, step = 201 (0.108 sec)
INFO:tensorflow:global_step/sec: 838.883
INFO:tensorflow:loss = 7.7207212, step = 301 (0.119 sec)
INFO:tensorflow:global_step/sec: 917.55
INFO:tensorflow:loss = 8.657611, step = 401 (0.110 sec)
INFO:tensorflow:global_step/sec: 866.062
INFO:tensorflow:loss = 8.567427, step = 501 (0.118 sec)
INFO:tensorflow:global_step/sec: 862.789
INFO:tensorflow:loss = 7.4227247, step = 601 (0.110 sec)
INFO:tensorflow:global_step/sec: 847.577
INFO:tensorflow:loss = 6.676866, step = 701 (0.118 sec)
INFO:tensorflow:global_step/sec: 911.827
INFO:tensorflow:loss = 7.4915752, step = 801 (0.112 sec)
INFO:tensorflow:global_step/sec: 837.15
INFO:tensorflow:loss = 7.7332606, step = 901 (0.120 sec)
INFO:tensorflow:global_step/sec: 857.915
INFO:tensorflow:loss = 7.283838, step = 1001 (0.116 sec)
INFO:tensorflow:global_step/sec: 786.144
INFO:tensorflow:loss = 8.61713, step = 1101 (0.127 sec)
INFO:tensorflow:Saving checkpoints for 1150 into /tmp/tmpd1r865ry/model.ckpt.
INFO:tensorflow:Loss for final step: 8.417924.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2018-04-05-15:36:08
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from /tmp/tmpd1r865ry/model.ckpt-1150
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Evaluation [1/5]
INFO:tensorflow:Evaluation [2/5]
INFO:tensorflow:Evaluation [3/5]
INFO:tensorflow:Evaluation [4/5]
INFO:tensorflow:Evaluation [5/5]
INFO:tensorflow:Finished evaluation at 2018-04-05-15:36:08

INFO:tensorflow:Saving dict for global step 1150: global_step = 1150, loss = 8.261229
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from /tmp/tmpd1r865ry/model.ckpt-1150
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.

---------------------------------------------------------------------------
AlreadyExistsError                        Traceback (most recent call last)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py in _do_call(self, fn, *args)
   1360     try:
-> 1361       return fn(*args)
   1362     except errors.OpError as e:

/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py in _run_fn(session, feed_dict, fetch_list, target_list, options, run_metadata)
   1339           return tf_session.TF_Run(session, options, feed_dict, fetch_list,
-> 1340                                    target_list, status, run_metadata)
   1341

/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/errors_impl.py in __exit__(self, type_arg, value_arg, traceback_arg)
    515             compat.as_text(c_api.TF_Message(self.status.status)),
--> 516             c_api.TF_GetCode(self.status.status))
    517     # Delete the underlying status object from memory otherwise it stays alive

AlreadyExistsError: There appears to be a concurrent caching iterator running - cache lockfile already exists ('/tmp/mycache_train.lockfile'). If you are sure no other running TF computations are using this cache prefix, delete the lockfile and re-initialize the iterator. Lockfile contents: Created at:<PHONE_NUMBER>
	 [[Node: IteratorGetNext = IteratorGetNext[output_shapes=[[?,?,3], [?,?,1]], output_types=[DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](OneShotIterator)]]

During handling of the above exception, another exception occurred:

AlreadyExistsError                        Traceback (most recent call last)
<ipython-input-5-9d8332f50ddd> in <module>()
     66
     67 if __name__ == "__main__":
---> 68     main()

<ipython-input-5-9d8332f50ddd> in main()
     62         input_fn=my_input_fn_eval, steps=5, throttle_secs=2)
     63
---> 64     tf.estimator.train_and_evaluate(estimator, train_spec, eval_spec)
     65
     66

/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/training.py in train_and_evaluate(estimator, train_spec, eval_spec)
    419         '(with task id 0).  Given task id {}'.format(config.task_id))
    420
--> 421   executor.run()
    422
    423

/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/training.py in run(self)
    492         config.task_type != run_config_lib.TaskType.EVALUATOR):
    493       logging.info('Running training and evaluation locally (non-distributed).')
--> 494       self.run_local()
    495       return
    496

/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/training.py in run_local(self)
    624           input_fn=self._train_spec.input_fn,
    625           max_steps=self._train_spec.max_steps,
--> 626           hooks=train_hooks)
    627
    628       # Final export signal: For any eval result with global_step >= train

/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/estimator.py in train(self, input_fn, hooks, steps, max_steps, saving_listeners)
    350
    351     saving_listeners = _check_listeners_type(saving_listeners)
--> 352     loss = self._train_model(input_fn, hooks, saving_listeners)
    353     logging.info('Loss for final step: %s.', loss)
    354     return self

/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/estimator.py in _train_model(self, input_fn, hooks, saving_listeners)
    889         loss = None
    890         while not mon_sess.should_stop():
--> 891           _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
    892       return loss
    893

/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py in run(self, fetches, feed_dict, options, run_metadata)
    544                           feed_dict=feed_dict,
    545                           options=options,
--> 546                           run_metadata=run_metadata)
    547
    548   def run_step_fn(self, step_fn):

/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py in run(self, fetches, feed_dict, options, run_metadata)
   1020                               feed_dict=feed_dict,
   1021                               options=options,
-> 1022                               run_metadata=run_metadata)
   1023       except _PREEMPTION_ERRORS as e:
   1024         logging.info('An error was raised. This may be due to a preemption in '

/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py in run(self, *args, **kwargs)
   1111         raise six.reraise(*original_exc_info)
   1112       else:
-> 1113         raise six.reraise(*original_exc_info)
   1114
   1115

/usr/local/lib/python3.6/dist-packages/six.py in reraise(tp, value, tb)
    691             if value.__traceback__ is not tb:
    692                 raise value.with_traceback(tb)
--> 693             raise value
    694         finally:
    695             value = None

/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py in run(self, *args, **kwargs)
   1096   def run(self, *args, **kwargs):
   1097     try:
-> 1098       return self._sess.run(*args, **kwargs)
   1099     except _PREEMPTION_ERRORS:
   1100       raise

/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py in run(self, fetches, feed_dict, options, run_metadata)
   1168                                   feed_dict=feed_dict,
   1169                                   options=options,
-> 1170                                   run_metadata=run_metadata)
   1171
   1172     for hook in self._hooks:

/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py in run(self, *args, **kwargs)
    948
    949   def run(self, *args, **kwargs):
--> 950     return self._sess.run(*args, **kwargs)
    951
    952   def run_step_fn(self, step_fn, raw_session, run_with_hooks):

/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py in run(self, fetches, feed_dict, options, run_metadata)
    903     try:
    904       result = self._run(None, fetches, feed_dict, options_ptr,
--> 905                          run_metadata_ptr)
    906       if run_metadata:
    907         proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py in _run(self, handle, fetches, feed_dict, options, run_metadata)
   1135     if final_fetches or final_targets or (handle and feed_dict_tensor):
   1136       results = self._do_run(handle, final_targets, final_fetches,
-> 1137                              feed_dict_tensor, options, run_metadata)
   1138     else:
   1139       results = []

/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
   1353     if handle is None:
   1354       return self._do_call(_run_fn, self._session, feeds, fetches, targets,
-> 1355                            options, run_metadata)
   1356     else:
   1357       return self._do_call(_prun_fn, self._session, handle, feeds, fetches)

/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py in _do_call(self, fn, *args)
   1372         except KeyError:
   1373           pass
-> 1374       raise type(e)(node_def, op, message)
   1375
   1376   def _extend_graph(self):

AlreadyExistsError: There appears to be a concurrent caching iterator running - cache lockfile already exists ('/tmp/mycache_train.lockfile'). If you are sure no other running TF computations are using this cache prefix, delete the lockfile and re-initialize the iterator. Lockfile contents: Created at:<PHONE_NUMBER>
	 [[Node: IteratorGetNext = IteratorGetNext[output_shapes=[[?,?,3], [?,?,1]], output_types=[DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](OneShotIterator)]]

Caused by op 'IteratorGetNext', defined at:
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py", line 16, in <module>
    app.launch_new_instance()
  File "/usr/local/lib/python3.6/dist-packages/traitlets/config/application.py", line 658, in launch_instance
    app.start()
  File "/usr/local/lib/python3.6/dist-packages/ipykernel/kernelapp.py", line 477, in start
    ioloop.IOLoop.instance().start()
  File "/usr/local/lib/python3.6/dist-packages/zmq/eventloop/ioloop.py", line 177, in start
    super(ZMQIOLoop, self).start()
  File "/usr/local/lib/python3.6/dist-packages/tornado/ioloop.py", line 888, in start
    handler_func(fd_obj, events)
  File "/usr/local/lib/python3.6/dist-packages/tornado/stack_context.py", line 277, in null_wrapper
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/zmq/eventloop/zmqstream.py", line 440, in _handle_events
    self._handle_recv()
  File "/usr/local/lib/python3.6/dist-packages/zmq/eventloop/zmqstream.py", line 472, in _handle_recv
    self._run_callback(callback, msg)
  File "/usr/local/lib/python3.6/dist-packages/zmq/eventloop/zmqstream.py", line 414, in _run_callback
    callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tornado/stack_context.py", line 277, in null_wrapper
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/ipykernel/kernelbase.py", line 283, in dispatcher
    return self.dispatch_shell(stream, msg)
  File "/usr/local/lib/python3.6/dist-packages/ipykernel/kernelbase.py", line 235, in dispatch_shell
    handler(stream, idents, msg)
  File "/usr/local/lib/python3.6/dist-packages/ipykernel/kernelbase.py", line 399, in execute_request
    user_expressions, allow_stdin)
  File "/usr/local/lib/python3.6/dist-packages/ipykernel/ipkernel.py", line 196, in do_execute
    res = shell.run_cell(code, store_history=store_history, silent=silent)
  File "/usr/local/lib/python3.6/dist-packages/ipykernel/zmqshell.py", line 533, in run_cell
    return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py", line 2718, in run_cell
    interactivity=interactivity, compiler=compiler, result=result)
  File "/usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py", line 2822, in run_ast_nodes
    if self.run_code(code, result):
  File "/usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py", line 2882, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-5-9d8332f50ddd>", line 68, in <module>
    main()
  File "<ipython-input-5-9d8332f50ddd>", line 64, in main
    tf.estimator.train_and_evaluate(estimator, train_spec, eval_spec)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/training.py", line 421, in train_and_evaluate
    executor.run()
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/training.py", line 494, in run
    self.run_local()
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/training.py", line 626, in run_local
    hooks=train_hooks)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/estimator.py", line 352, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/estimator.py", line 809, in _train_model
    input_fn, model_fn_lib.ModeKeys.TRAIN))
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/estimator.py", line 668, in _get_features_and_labels_from_input_fn
    result = self._call_input_fn(input_fn, mode)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/estimator.py", line 760, in _call_input_fn
    return input_fn(**kwargs)
  File "<ipython-input-5-9d8332f50ddd>", line 28, in my_input_fn_train
    value = ds.make_one_shot_iterator().get_next()
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/iterator_ops.py", line 330, in get_next
    name=name)), self._output_types,
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_dataset_ops.py", line 866, in iterator_get_next
    output_shapes=output_shapes, name=name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 3271, in create_op
    op_def=op_def)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 1650, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

AlreadyExistsError (see above for traceback): There appears to be a concurrent caching iterator running - cache lockfile already exists ('/tmp/mycache_train.lockfile'). If you are sure no other running TF computations are using this cache prefix, delete the lockfile and re-initialize the iterator. Lockfile contents: Created at:<PHONE_NUMBER>
	 [[Node: IteratorGetNext = IteratorGetNext[output_shapes=[[?,?,3], [?,?,1]], output_types=[DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](OneShotIterator)]]

@saeta Could this be an instance of the same bug as the ds.cache().take(N).repeat() one that you're looking at?

@mrry I think that there is a conceptual problem with the train_and_evaluate.
Mainly you control train/eval phases with EvalSpec (with a default throttle_secs=600) or check point in RunConfig.
If the dataset is quite big you cannot evaluate until the cache is done cause a new TrainSpec call will rebuild the input/dataset pipeline and you will find a lock on the cache dir.
There could be a not so trivial workaround with RunConfig save_checkpoints_steps to block throttle until you will not have covered the epoch  but in the distributed setup I suppose that the dataset cache is working locally per node so is it still plausible?
Do you see any other solution (that I suppose still require documentation)?

/cc @isaprykin for suggestions about how to use train_and_evaluate().

If I understand the problem correctly, user defines the dataset with cache(filename). For this input_fn, as the input_fn is invoked multiple times, train_and_evalaute will error out. Is this the issue?
Unfortunately, train_and_evaluate does not specify the contract how many times train input_fn and eval input_fn will be invoked today. It will limit the implementation space we have.
@mrry can user change the code into following way?
ds = tf.data.Dataset.range(100000)
ds = ds.map(
    lambda x: tf.py_func(func=gen, inp=[x], Tout=[tf.float32, tf.float32]))
ds = ds.map(batch_reshape)
ds = ds.cache("/tmp/mycache_train")
ds = ds.repeat(3)

def my_input_fn_train():
    value = ds.make_one_shot_iterator().get_next()
   return {"input_rgb": value[0]}, {"softmax": value[1]}

@xiejw Unfortunately that won't work because ds will be bound to a graph that is different from the estimator's graph.
I'm not sure why train_and_evaluate() is calling the input function multiple times, but the right solution would seem to be allowing concurrent iterators to build a cache with the same eventual filename.

What do you think about @ispirmustafa comment?

current local run implementation is something like:
for ...:
estimator.train(...)
estimator.evaluate(...)
but making train_and_evaluate local mode to create this graphs only once is a good feature request.
for now as a workaround you can do following:
cnt = 0
def input_fn():
cnt += 1
cache_fn = ... + cnt

@ispirmustafa The workaround It will be useless cause will skip the lock but you will never match the cache. Also generally what about cache matching if the pipeline is rebuild and shuffled at every Train phase?

@bhack could you please explain more? The error message is about using same file as cache. this workaround resolves that error. what do you mean by "skip the lock and never match"?

@ispirmustafa  I.e. You have a large dataset over a network file system that needs to be cached over multiple train/evaluate rounds. With this you will workaround the lock but you will never match the cache after the first evaluation.

Also any temporary workaround is quite invalidated by limits/bugs in https://github.com/tensorflow/tensorflow/issues/17650

Nagging Assignee @saeta: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @saeta: It has been 38 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @saeta: It has been 52 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @saeta: It has been 67 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Some news at https://github.com/tensorflow/tensorflow/issues/19062#issuecomment-400129963. But I think it will not solve this.

Do you need to fix  https://github.com/tensorflow/tensorflow/issues/19062#issuecomment-401736720?

Nagging Assignee @saeta: It has been 15 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @saeta: It has been 30 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @saeta: It has been 45 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @saeta: It has been 60 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @saeta: It has been 75 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @saeta: It has been 89 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @saeta: It has been 104 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @saeta: It has been 119 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @saeta: It has been 135 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @saeta: It has been 150 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

tensorflow 2.0.0-alpha0
cache lockfile already exists.  When going for the 2nd epoch, of evaluation, the previous lock file isn't removed i can see it on the file system.  Doesnt happen with training data hence it fires this error as it cant get hold of ds.cache(file) it seems:
Code:
def get_ds_tfdata_cache(self, X_train, y_train, batch_size, load_preprocess_image, filename):
    import tensorflow as tf
    from Source.tfdata.PathConstants import Constants
    cachedir = Constants().get_tfcache_dir_path()
    ds = tf.data.Dataset.from_tensor_slices((X_train, y_train))
    ds = ds.map(load_preprocess_image)
    ds = ds.cache(filename=cachedir + "/" + filename)
    ds = ds.apply(
        tf.data.experimental.shuffle_and_repeat(buffer_size=batch_size))
    ds = ds.batch(batch_size=batch_size)
    ds = ds.prefetch(buffer_size=batch_size)
    return ds

image_ds_train = data.get_ds_tfdata_cache(X_train=X_train, y_train=y_train, batch_size=params_list.get_batch_size(),
load_preprocess_image=helpermethods.load_and_preprocess_image,
filename=params_list.get_tf_cache_train_filename())
image_ds_validate = data.get_ds_tfdata_cache(X_train=X_train, y_train=y_train,
batch_size=params_list.get_batch_size(),
load_preprocess_image=helpermethods.load_and_preprocess_image,
filename=params_list.get_tf_cache_test_filename())
early_stop_patience = 5
training_epochs = 2
history = model.fit(x=image_ds_train, validation_data=image_ds_validate, epochs=training_epochs,
steps_per_epoch=steps_per_epoch,
validation_steps=validation_steps, callbacks=[early_stop, tensorboard])
Error:
2019-05-09 11:12:53.961797: W tensorflow/core/framework/op_kernel.cc:1431] OP_REQUIRES failed at iterator_ops.cc:988 : Already exists: There appears to be a concurrent caching iterator running - cache lockfile already exists ('./tfdata_dir/tfcache_dir/cache_validate.tf-data_0.lockfile'). If you are sure no other running TF computations are using this cache prefix, delete the lockfile and re-initialize the iterator. Lockfile contents: Created at:<PHONE_NUMBER>
Traceback (most recent call last):
File "", line 53, in
File "C:\ML_VirtualEnv\tf2_alpha\lib\site-packages\tensorflow\python\keras\engine\training.py", line 791, in fit
initial_epoch=initial_epoch)
File "C:\ML_VirtualEnv\tf2_alpha\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1515, in fit_generator
steps_name='steps_per_epoch')
File "C:\ML_VirtualEnv\tf2_alpha\lib\site-packages\tensorflow\python\keras\engine\training_generator.py", line 315, in model_iteration
steps_name='validation_steps')
File "C:\ML_VirtualEnv\tf2_alpha\lib\site-packages\tensorflow\python\keras\engine\training_generator.py", line 213, in model_iteration
batch_data = _get_next_batch(generator, mode)
File "C:\ML_VirtualEnv\tf2_alpha\lib\site-packages\tensorflow\python\keras\engine\training_generator.py", line 355, in _get_next_batch
generator_output = next(generator)
File "C:\ML_VirtualEnv\tf2_alpha\lib\site-packages\tensorflow\python\data\ops\iterator_ops.py", line 556, in next
return self.next()
File "C:\ML_VirtualEnv\tf2_alpha\lib\site-packages\tensorflow\python\data\ops\iterator_ops.py", line 585, in next
return self._next_internal()
File "C:\ML_VirtualEnv\tf2_alpha\lib\site-packages\tensorflow\python\data\ops\iterator_ops.py", line 577, in _next_internal
output_shapes=self._flat_output_shapes)
File "C:\ML_VirtualEnv\tf2_alpha\lib\site-packages\tensorflow\python\ops\gen_dataset_ops.py", line 1984, in iterator_get_next_sync
_six.raise_from(_core._status_to_exception(e.code, message), None)
File "", line 3, in raise_from
tensorflow.python.framework.errors_impl.AlreadyExistsError: There appears to be a concurrent caching iterator running - cache lockfile already exists ('./tfdata_dir/tfcache_dir/cache_validate.tf-data_0.lockfile'). If you are sure no other running TF computations are using this cache prefix, delete the lockfile and re-initialize the iterator. Lockfile contents: Created at:<PHONE_NUMBER> [Op:IteratorGetNextSync]

+1 this is a bug in the design.  cache files should support concurrent usage since one can easily get there via tf.data.experimental.parallel_interleave().

I wrote a little utility function to get rid of lock files
import os
import shutil
from pathlib import Path

def delete_trailing_lock_files(cache_root):
  lock_file_dirs = {}
  for filename in Path(cache_root).glob('**/*.lock*'):
    lock_file_dirs[os.path.split(filename)[0] + '/'] = ''
  for dir in lock_file_dirs.keys():
    try:
      shutil.rmtree(dir)
    except:
      print('failed removing dir', filename)

Note that what you want to do is to create a single dataset iterator and keep using it:
val_dataset = tf.data.Dataset.range(10)
val_dataset = val_dataset.cache('/tmp/cacheset/')
val_dataset = val_dataset.repeat()

val_iterator = iterator(val_dataset) #Never do this more than once!

I also find the .cache method of tf.data.Dataset a bit odd, with the docs stating:

filename: A tf.string scalar tf.Tensor, representing the name of a directory on the filesystem to use for caching tensors in this Dataset. If a filename is not provided, the dataset will be cached in memory.

e.g. asking for filename when filename actually represents the name of a directory.
Part of the cache method problem is that the cached file is not always auto-cleaned up.
I think adding a positional argument overwrite solves this issue by allowing users to cache to a known location like:
ds_train = ds_train.cache(filename=os.path.join(cache_dir, 'train'), overwrite=True)
ds_valid = ds_valid.cache(filename=os.path.join(cache_dir, 'valid'), overwrite=True)

@lenlen, did you try the @SumNeuron's workaround. Please let us know if that solves your problem. Thanks!

@gadagashwini the overwrite kwarg does not exist, it's just a proposal: https://github.com/tensorflow/tensorflow/blob/r1.14/tensorflow/python/data/ops/dataset_ops.py#L1738-L1740
overwrite would be helpful but wouldn't solve the concurrency problem.  The op needs a do-over.

This is fixed with TF version 1.15.0 Thanks!


Considerable launch delay in 122 / SSE3 / W7x86

System Details

OS: [Windows 7x86]
Thorium Version [M122.0.6261.132 - thorium_WIN32_SSE3_mini_installer.exe]

Problem
Launching the app adds considerable amount of wait time. Had to revert back to version 119.
Additional Notes
The app launches blank for several seconds. Perhaps my old Core 2 Duo T7600 struggles with this version?
Has anything changed in regards to rendering?

@naryfa No changes in rendering. I can try the 32 bit SSE3 version on my Core 2 Duo laptop. Im using the 64 bit SSE3 version and I don't notice any difference from previous releases.
However, I don't really test the 32 bit builds, and they are provided as a convenience.

I understand. Thank you for testing on your end. 64bit OS on my old laptop would be a waste as the chipset allows only for 3.2GB usable. Every precious ounce of RAM is gold here so I'm forced to run x86.


Scroll lag after update

We have written the needed data into your clipboard because it was too large to send. Please paste.
Issue Type: Bug
I'm editing a long markdown file (~1000 lines) and after upgrade to latest insider vscode the scroll lags.

Yes, it still reproduces after vim disabled. I guess it's related to windows insider update, so I've sent a ticket to windows team.

Yes, it still reproduces after vim disabled. I guess it's related to windows insider update, so I've sent a ticket to windows team.

When people edit a huge file and face lag due when they navigate the file, it used to be related to the Bracket Pair Colorizer and Vim extension.

That's not a huge file I suppose, and every file (even small ones) will lag on scroll.

Can you please check whether you see the same with disabled extension? https://github.com/microsoft/vscode/wiki/Performance-Issues#run-with-extensions-disabled

@kieferrm It doesn't work. Here is performance debug chart:

Select one long frame

We cannot work off screen shots, can you just attach the profile file

@jrieken Attached. Thanks!
Profile-20201009T182412.zip

Hi @alexdima any update? I still experience scroll lags.

I have found a temporary fix for scrolling lag that I have been experiencing (happened even with extensions disabled), I now run vscode from terminal with code --disable-renderer-accessibility. (I'm on Windows 10 with version 1.51)

I've found the root cause: text cursor indicator. Disable the text cursor indicator in settings could solve this issue. Already tracked in windows feedback.


Compilation Error: register_types.cpp not finding include file version_generated.gen.h

OS target (Android/iOS):
OSX & iOS
Godot version:
3.1 latest source from master branch today
Issue description:
I can build OSX Godot and iOS export modules fine. But when I add this module, I get compilation error unable to find "version_generated.gen.h" file:
[ 13%] Compiling ==> modules/admob/register_types.cpp
modules/admob/register_types.cpp:1:10: fatal error: 'version_generated.gen.h' file not found
#include <version_generated.gen.h>
I have the "admob" directory in the Godot "modules" directory and have dropped Google Mobile Ads SDK in the ios/lib dir.
version_generated.gen.h exists in godot/core directory. I have a completely clean build directory pulled from git today. I'm not familiar with scons so don't know where the include directories should be specified. Any help would be appreciated. Thanks!

Could anybody confirms if it works with just:
env.Append(CPPPATH=['/core'])

If so I can update the code with this fix.

Where exactly do we put this line of code? I'm willing to try both, and I have but it doesn't seem to fix the error for me at least.

It should be in the file admob/config.py, line 13, just after
	env.Append(FRAMEWORKPATH=['modules/admob/ios/lib'])

Something like this: https://pastebin.com/SmCtKkD4

@GhostWalker562 nevermind, just updated the repository, try again with my last commit.

Works 👍

@GhostWalker562 Nice, thanks for testing!
As I mentioned before, I don't have any Apple hardware to maintain and test this module for iOS. I wrote this for a project some time ago and borrowed a mac mini to do the development.


what do I need to set to ignore a label?

Hi! I have set ignore_index = 0 in configs file, but mIoU of Class_0 is still calculated. I would like to know if it's supposed to work this way, or if I've missed some config setting.

this is my config file:
_base_ = ["../_base_/default_runtime.py"]

# misc custom setting
batch_size = 1  # bs: total bs in all gpus
mix_prob = 0.8
empty_cache = False
# enable_amp = True
enable_amp = False

# model settings
model = dict(
    type="DefaultSegmentorV2",
    num_classes=8,
    backbone_out_channels=64,
    backbone=dict(
        type="PT-v3m1",
        in_channels=3,
        order=["z", "z-trans", "hilbert", "hilbert-trans"],
        stride=(2, 2, 2, 2),
        enc_depths=(2, 2, 2, 6, 2),
        enc_channels=(32, 64, 128, 256, 512),
        enc_num_head=(2, 4, 8, 16, 32),
        enc_patch_size=(1024, 1024, 1024, 1024, 1024),
        dec_depths=(2, 2, 2, 2),
        dec_channels=(64, 64, 128, 256),
        dec_num_head=(4, 4, 8, 16),
        dec_patch_size=(1024, 1024, 1024, 1024),
        mlp_ratio=4,
        qkv_bias=True,
        qk_scale=None,
        attn_drop=0.0,
        proj_drop=0.0,
        drop_path=0.3,
        shuffle_orders=True,
        pre_norm=True,
        enable_rpe=False,
        enable_flash=True,
        upcast_attention=False,
        upcast_softmax=False,
        cls_mode=False,
        pdnorm_bn=False,
        pdnorm_ln=False,
        pdnorm_decouple=True,
        pdnorm_adaptive=False,
        pdnorm_affine=True,
        pdnorm_conditions=("nuScenes", "SemanticKITTI", "Waymo", "dales", "whu3d"),
    ),
    criteria=[
        dict(type="CrossEntropyLoss", loss_weight=1.0, ignore_index=0),
        dict(type="LovaszLoss", mode="multiclass", loss_weight=1.0, ignore_index=0),
    ],
)

# scheduler settings
epoch = 1000
eval_epoch = 100
optimizer = dict(type="AdamW", lr=0.002, weight_decay=0.005)
scheduler = dict(
    type="OneCycleLR",
    max_lr=[0.002, 0.0002],
    pct_start=0.04,
    anneal_strategy="cos",
    div_factor=10.0,
    final_div_factor=100.0,
)
param_dicts = [dict(keyword="block", lr=0.0002)]

# dataset settings
dataset_type = "WHU3DDataset"
data_root = "/home/hqu/ljc/Pointcept/data/whu3d"
ignore_index = 0
names = [
    "Other",
    "Ground",
    "Building",
    "Tree",
    "Veg.",
    "Low.Veg.",
    "Light",
    "Electric",
]

data = dict(
    num_classes=8,
    ignore_index=ignore_index,
    names=names,
    train=dict(
        type=dataset_type,
        split="train",
        data_root=data_root,
        transform=[
            # dict(type="RandomDropout", dropout_ratio=0.2, dropout_application_ratio=0.2),
            # dict(type="RandomRotateTargetAngle", angle=(1/2, 1, 3/2), center=[0, 0, 0], axis="z", p=0.75),
            dict(type="RandomRotate", angle=[-1, 1], axis="z", center=[0, 0, 0], p=0.5),
            # dict(type="RandomRotate", angle=[-1/6, 1/6], axis="x", p=0.5),
            # dict(type="RandomRotate", angle=[-1/6, 1/6], axis="y", p=0.5),
            dict(type="PointClip", point_cloud_range=(-75.2, -75.2, -4, 75.2, 75.2, 2)),
            dict(type="RandomScale", scale=[0.9, 1.1]),
            # dict(type="RandomShift", shift=[0.2, 0.2, 0.2]),
            dict(type="RandomFlip", p=0.5),
            dict(type="RandomJitter", sigma=0.005, clip=0.02),
            # dict(type="ElasticDistortion", distortion_params=[[0.2, 0.4], [0.8, 1.6]]),
            dict(
                type="GridSample",
                grid_size=0.05,
                hash_type="fnv",
                mode="train",
                keys=("coord", "segment"),
                return_grid_coord=True,
            ),
            # dict(type="SphereCrop", point_max=1000000, mode="random"),
            # dict(type="CenterShift", apply_z=False),
            dict(type="ToTensor"),
            dict(
                type="Collect",
                keys=("coord", "grid_coord", "segment"),
                feat_keys=("coord"),
            ),
        ],
        test_mode=False,
        ignore_index=ignore_index,
    ),
    val=dict(
        type=dataset_type,
        split="test",
        data_root=data_root,
        transform=[
            dict(type="PointClip", point_cloud_range=(-75.2, -75.2, -4, 75.2, 75.2, 2)),
            dict(
                type="GridSample",
                grid_size=0.05,
                hash_type="fnv",
                mode="train",
                keys=("coord", "segment"),
                return_grid_coord=True,
            ),
            dict(type="ToTensor"),
            dict(
                type="Collect",
                keys=("coord", "grid_coord", "segment"),
                feat_keys=("coord"),
            ),
        ],
        test_mode=False,
        ignore_index=ignore_index,
    ),
    test=dict(
        type=dataset_type,
        split="test",
        data_root=data_root,
        transform=[
            dict(type="PointClip", point_cloud_range=(-75.2, -75.2, -4, 75.2, 75.2, 2)),
            dict(type="Copy", keys_dict={"segment": "origin_segment"}),
            dict(
                type="GridSample",
                grid_size=0.025,
                hash_type="fnv",
                mode="train",
                keys=("coord", "segment"),
                return_inverse=True,
            ),
        ],
        test_mode=True,
        test_cfg=dict(
            voxelize=dict(
                type="GridSample",
                grid_size=0.05,
                hash_type="fnv",
                mode="test",
                return_grid_coord=True,
                keys=("coord"),
            ),
            crop=None,
            post_transform=[
                dict(type="ToTensor"),
                dict(
                    type="Collect",
                    keys=("coord", "grid_coord", "index"),
                    feat_keys=("coord"),
                ),
            ],
            aug_transform=[
                [
                    dict(
                        type="RandomRotateTargetAngle",
                        angle=[0],
                        axis="z",
                        center=[0, 0, 0],
                        p=1,
                    )
                ],
                [
                    dict(
                        type="RandomRotateTargetAngle",
                        angle=[1 / 2],
                        axis="z",
                        center=[0, 0, 0],
                        p=1,
                    )
                ],
                [
                    dict(
                        type="RandomRotateTargetAngle",
                        angle=[1],
                        axis="z",
                        center=[0, 0, 0],
                        p=1,
                    )
                ],
                [
                    dict(
                        type="RandomRotateTargetAngle",
                        angle=[3 / 2],
                        axis="z",
                        center=[0, 0, 0],
                        p=1,
                    )
                ],
            ],
        ),
        ignore_index=ignore_index,
    ),
)

Hi @gitKincses , I did the same thing. IoU for class 0 is still calculated, but the model did not produce class 0 during prediction / inference. It's just my IoU and accuracy for class 0 is always 1.0 / 1.0, no matter what models I used. I was expecting IoU and accuracy to be 0.0 for class 0 since the model will not produce class 0. But anyway, the models do the job by not producing class 0. I'm happy with that. I only need to recalculate the accuracy by removing class 0.

Yes @aldinorizaldy , I also realized that I can recalculate mIoU and mAcc. However, it seems that allAcc cannot be recalculated in the same way
:(

Why do you think you can't calculate allAcc in the same way? I would say including class 0 for accuracy calculation during training would not harm the way we monitor the training progress. Let's say we want to keep the best model from the training. Even if there is class 0, the accuracy numbers still show us in which epoch we get the best model, as long as the accuracy of class 0 is constant (either 0 or 1). The accuracy is indeed not showing us the actual accuracy without class 0, but we can still use it for monitoring the training progress. Once we done, we can calculate the accuracy on test set without class 0, with any metrics (allAcc, mIoU, F1, etc.)

Hi @gitKincses! I am not the author, but I have a suggestion.
There will always be a class_0 , because the indexing of the classes always start from 0 ( that is how the loss is calculated). If you remove class 0, you would need to map all other class to a different range, so they start from 0.
Also, using ignore_index =  0 it is not recommended, since it messes up with the training pipeline. The number 0 is always reserved for a class you are trying to predict. Ignore_index is just used to ignore noisy points during calculation of the metrics, they basically are not taken into account.
However, there are a lot more changes that you need to do. You would have to also change the number of predicted classes (yours is still 8). So your model still predicts 8 classes, and that is why you still see the 8 classes in metrics. You need to also change the list of names of classes. I would suggest mapping all points, that you want to ignore, to -1, and using this as ignore index. Also make sure that used class labels stat from 0.
Please let me any know, if you have any other questions.

Why do you think you can't calculate allAcc in the same way? I would say including class 0 for accuracy calculation during training would not harm the way we monitor the training progress. Let's say we want to keep the best model from the training. Even if there is class 0, the accuracy numbers still show us in which epoch we get the best model, as long as the accuracy of class 0 is constant (either 0 or 1). The accuracy is indeed not showing us the actual accuracy without class 0, but we can still use it for monitoring the training progress. Once we done, we can calculate the accuracy on test set without class 0, with any metrics (allAcc, mIoU, F1, etc.)

Thank you for your answer. I have solved the problem of how to ignore the index, but I am still confused about the calculation of allAcc. I think that to calculate allAcc, I would need to know the number of correct and incorrect predictions. Therefore, I am unsure if I can calculate the 7 classes allAcc using only the accuracy of each class and the 8 classes allAcc. Please correct me if I am mistaken.

Thank you for your suggestion. I will try to implement it.

Hi @gitKincses! I am not the author, but I have a suggestion.
There will always be a class_0 , because the indexing of the classes always start from 0 ( that is how the loss is calculated). If you remove class 0, you would need to map all other class to a different range, so they start from 0.
Also, using ignore_index = 0 it is not recommended, since it messes up with the training pipeline. The number 0 is always reserved for a class you are trying to predict. Ignore_index is just used to ignore noisy points during calculation of the metrics, they basically are not taken into account.
However, there are a lot more changes that you need to do. You would have to also change the number of predicted classes (yours is still 8). So your model still predicts 8 classes, and that is why you still see the 8 classes in metrics. You need to also change the list of names of classes. I would suggest mapping all points, that you want to ignore, to -1, and using this as ignore index. Also make sure that used class labels stat from 0.
Please let me any know, if you have any other questions.

@SpeedyGonzales949 Thanks for your answer.


[SPARK-33108][BUILD] Remove sbt-dependency-graph SBT plugin

What changes were proposed in this pull request?
This PR aims to remove sbt-dependency-graph SBT plugin.
Why are the changes needed?
sbt-dependency-graph officially doesn't support SBT 1.3.x and it's broken due to NoSuchMethodError. This cannot be fixed in sbt-dependency-graph side at SBT 1.3.x

https://github.com/sbt/sbt-dependency-graph

Note: Under sbt >= 1.3.x some features might currently not work as expected or not at all (like dependencyLicenses).

$ build/sbt dependencyTree
Launching sbt from build/sbt-launch-1.3.13.jar
[info] welcome to sbt 1.3.13 (AdoptOpenJDK Java 1.8.0_252)
...
[error] java.lang.NoSuchMethodError: sbt.internal.LibraryManagement$.cachedUpdate(Lsbt/librarymanagement/DependencyResolution;Lsbt/librarymanagement/ModuleDescriptor;Lsbt/util/CacheStoreFactory;Ljava/lang/String;Lsbt/librarymanagement/UpdateConfiguration;Lscala/Function1;ZZZLsbt/librarymanagement/UnresolvedWarningConfiguration;Lsbt/librarymanagement/EvictionWarningOptions;ZLsbt/internal/librarymanagement/CompatibilityWarningOptions;Lsbt/util/Logger;)Lsbt/librarymanagement/UpdateReport;

ALTERNATIVES

One alternative is coursier, but it requires coursier-based sbt launcher which is more intrusive.

https://get-coursier.io/docs/sbt-coursier.html#sbt-13x

you'll have to use the coursier-based sbt launcher, via its custom sbt-extras launcher for example.

Another alternative is moving to SBT 1.4.0 which uses sbt-dependency-graph as a built-in, but it's still new and will requires many change.

So, this PR aims to remove the broken plugin simply.
Does this PR introduce any user-facing change?
No. This is a dev-only change.
How was this patch tested?
Manual.
$ build/sbt dependencyTree
...
[error] Not a valid command: dependencyTree
[error] Not a valid project ID: dependencyTree
[error] Not a valid key: dependencyTree (similar: dependencyOverrides, sbtDependency, dependencyResolution)
[error] dependencyTree
[error]               ^

Test build #129611 has started for PR 29997 at commit 3895e0d.

cc @srowen and @HyukjinKwon .
I found this during investigating SPARK-33104 .

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34214/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34214/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34214/
Test PASSed.

cc @gemelen and @ScrapCodes FYI

Thank you, @srowen and @HyukjinKwon . Sure. I'll add an IDed TODO comment.

Test build #129617 has started for PR 29997 at commit 6cc043e.

I'll merge this because this is not tested by any CI. I checked this manually.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34221/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34221/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34221/
Test PASSed.

Test build #129611 has finished for PR 29997 at commit 3895e0d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129611/
Test PASSed.

Test build #129617 has finished for PR 29997 at commit 6cc043e.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129617/
Test FAILed.


Print errors comming from callv

Fixes #18386.
Additionally, changed a bit how space for the arguments is allocated, making it similar to MessageQueue::_call_function:
https://github.com/godotengine/godot/blob/e761af57949a81a680551159344180b561d3d528/core/message_queue.cpp#L245-L253

Thanks!


please delete testing branch, this is giving me an headache

no seriously, running pages on a non-main is diabolical

main is what via will yionk to publish for the website, testing is for us to use

Where still doing this?

You did not just ask that


⚠ Bofrost.gr has degraded performance

In 5290636, Bofrost.gr (https://www.bofrost.gr) experienced degraded performance:

HTTP code: 200
Response time: 1887 ms

Resolved: Bofrost.gr performance has improved in 16c8bf8 after 6 minutes.


[gcp_managed_relational_db] Added GCP functionality and unit tests.

@stfeng2 @gareth-ferneyhough
Added methods and unit tests for GCP managed relational db.

Last two minor comments, otherwise LGTM.
After you address these comments, I will let Gareth do the final merge (because this is his branch. ;)

@gareth-ferneyhough I've approved this. Please feel free to merge after your approval. Thanks!


Custom CA's

How do I add custom CA's into the tool?

One method would be to have your custom root CA to your engine that you are using to perform the migration on and set the USE_INSECURE_CURL environment variable appropriately.  If you don't want to set the USE_INSECURE_CURL environment variable, you could create a Dockerfile that adds your root CA's public certificate to the existing trusted root CAs.  Here is an example:
FROM docker/migrator:latest

COPY your_root_ca.crt /usr/local/share/ca-certificates/your_root_ca.crt
RUN update-ca-certificates

Either way, your root CA certificate needs to be added to wherever your engine is running (you can use the same method above if you are on Ubuntu or Debian) and then you need to restart that engine for it to recognize the changes.

I would prefer the insecure method as I have access to localhost:5000 on the v2 server. But I might need to add the CA so I can hit the  v1 server.  THanks for the help.

Just to make sure it is clear, the insecure method does still require SSL certs.  I could probably add flags to allow connections over http instead of https but in all honesty, I'd rather encourage people to make sure they are securing their registries with SSL.

Is there any way to use http only? Like I said I want to run this on the v2 host where I have access to local host:5000.

USE_INSECURE_CURL  worked thanks.


when i click on web site not opened

Describe the bug

Expected behavior

Screenshots (optional)

Additional context (optional )

Issue resolved


Inconsistent background color on dark mode [Chromium ONLY]

Package
filament/filament
Package Version
v3.2.110
Laravel Version
11.11
Livewire Version
v3.5.6
PHP Version
PHP 8.2
Problem description
The background color is inconsistent in dark mode for all chromium-based browsers including Google Chrome and Brave.

A workaround to solve the problem is to give the same background color to .fi-main
Expected behavior
I think it should be same color across the page
Steps to reproduce
switch to dark mode and go to a page with less content so your browser doesn't start showing scroller.
For me, with a resolution of 2304 * 1296, it is possible to see the issue on the demo as well
Reproduction repository (issue will be closed if this is not valid)
https://github.com/filamentphp/demo
Relevant log output
No response

Donate 💰 to fund this issue

You can donate funding to this issue. We receive the money once the issue is completed & confirmed by you.
100% of the funding will be distributed between the Filament core team to run all aspects of the project.
Thank you in advance for helping us make maintenance sustainable!

Duplicate of #7759?

@andrewdwallo yes but don't understand why closed

I guess because it's browser specific and not an issue/bug with Filament? That is, unless it's something overlooked in Filament.

Issues are closed if there aren't actions we can take to fix Filament, it's like keeping an item on your todo list that is for someone else to solve. If you have any suggestions, please share or make a PR.


Build pipeline pull req

This configuration builds the application and produces builds for both Debug and Release configurations.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: PatrickDelaney17 sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.


Adding a custom block handler

Coverage decreased (-0.3%) to 93.814% when pulling 3e9e810921f16c55e839c4c6584eed115f79b023 on govi:master into 7a2f9ed9b8007680c67a82ebbc32ae55375c8922 on globocom:master.

Hello @govi!
Could you add some tests to fix coveralls?
Thanks for your contribution! 🎉

sure thing!

Coverage increased (+0.04%) to 94.158% when pulling 3b7ea20fdab20a126bac30de166d064af1ef79ac on govi:master into 7a2f9ed9b8007680c67a82ebbc32ae55375c8922 on globocom:master.

We will create a new release asap.