ValueError: Checkpoint version should be V2

GPU: NVIDIA GEFORCE RTX 2060
CPU: 16GB RAM, 6 processor cores
TensorFlow: 2.3.1
Python: 3.8.6
CUDA: 10.1
cuDNN: 7.6
I am training a Mask R-CNN Inception ResNet V2 1024x1024 algorithm, as downloaded from the TensorFlow 2 Detection Model Zoo . I am training this algorithm on my custom dataset, which I have labeled using Label-img . When I train the model using the Anaconda command python model_main_tf2.py --model_dir=models/my_faster_rcnn --pipeline_config_path=models/my_faster_rcnn/pipeline.config, I get the following error:
Traceback (most recent call last):
  File "model_main_tf2.py", line 113, in <module>
    tf.compat.v1.app.run()
  File "C:\user\anaconda3\envs\object_detection_api\lib\site-packages\tensorflow\python\platform\app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "C:\user\anaconda3\envs\object_detection_api\lib\site-packages\absl\app.py", line 303, in run
    _run_main(main, args)
  File "C:\user\anaconda3\envs\object_detection_api\lib\site-packages\absl\app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "model_main_tf2.py", line 104, in main
    model_lib_v2.train_loop(
  File "C:\user\anaconda3\envs\object_detection_api\lib\site-packages\object_detection\model_lib_v2.py", line 564, in train_loop
    load_fine_tune_checkpoint(detection_model,
  File "C:\user\anaconda3\envs\object_detection_api\lib\site-packages\object_detection\model_lib_v2.py", line 348, in load_fine_tune_checkpoint
    raise ValueError('Checkpoint version should be V2')
ValueError: Checkpoint version should be V2

How can I resolve this error? Below are the scripts referenced in the error:
model_main_tf2.py
# Lint as: python3
# Copyright 2020 The TensorFlow Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================

r"""Creates and runs TF2 object detection models.

For local training/evaluation run:
PIPELINE_CONFIG_PATH=path/to/pipeline.config
MODEL_DIR=/tmp/model_outputs
NUM_TRAIN_STEPS=10000
SAMPLE_1_OF_N_EVAL_EXAMPLES=1
python model_main_tf2.py -- \
  --model_dir=$MODEL_DIR --num_train_steps=$NUM_TRAIN_STEPS \
  --sample_1_of_n_eval_examples=$SAMPLE_1_OF_N_EVAL_EXAMPLES \
  --pipeline_config_path=$PIPELINE_CONFIG_PATH \
  --alsologtostderr
"""
from absl import flags
import tensorflow.compat.v2 as tf
from object_detection import model_lib_v2

flags.DEFINE_string('pipeline_config_path', None, 'Path to pipeline config '
                    'file.')
flags.DEFINE_integer('num_train_steps', None, 'Number of train steps.')
flags.DEFINE_bool('eval_on_train_data', False, 'Enable evaluating on train '
                  'data (only supported in distributed training).')
flags.DEFINE_integer('sample_1_of_n_eval_examples', None, 'Will sample one of '
                     'every n eval input examples, where n is provided.')
flags.DEFINE_integer('sample_1_of_n_eval_on_train_examples', 5, 'Will sample '
                     'one of every n train input examples for evaluation, '
                     'where n is provided. This is only used if '
                     '`eval_training_data` is True.')
flags.DEFINE_string(
    'model_dir', None, 'Path to output model directory '
                       'where event and checkpoint files will be written.')
flags.DEFINE_string(
    'checkpoint_dir', None, 'Path to directory holding a checkpoint.  If '
    '`checkpoint_dir` is provided, this binary operates in eval-only mode, '
    'writing resulting metrics to `model_dir`.')

flags.DEFINE_integer('eval_timeout', 3600, 'Number of seconds to wait for an'
                     'evaluation checkpoint before exiting.')

flags.DEFINE_bool('use_tpu', False, 'Whether the job is executing on a TPU.')
flags.DEFINE_string(
    'tpu_name',
    default=None,
    help='Name of the Cloud TPU for Cluster Resolvers.')
flags.DEFINE_integer(
    'num_workers', 1, 'When num_workers > 1, training uses '
    'MultiWorkerMirroredStrategy. When num_workers = 1 it uses '
    'MirroredStrategy.')
flags.DEFINE_integer(
    'checkpoint_every_n', 1000, 'Integer defining how often we checkpoint.')
flags.DEFINE_boolean('record_summaries', True,
                     ('Whether or not to record summaries during'
                      ' training.'))

FLAGS = flags.FLAGS

def main(unused_argv):
  flags.mark_flag_as_required('model_dir')
  flags.mark_flag_as_required('pipeline_config_path')
  tf.config.set_soft_device_placement(True)

  if FLAGS.checkpoint_dir:
    model_lib_v2.eval_continuously(
        pipeline_config_path=FLAGS.pipeline_config_path,
        model_dir=FLAGS.model_dir,
        train_steps=FLAGS.num_train_steps,
        sample_1_of_n_eval_examples=FLAGS.sample_1_of_n_eval_examples,
        sample_1_of_n_eval_on_train_examples=(
            FLAGS.sample_1_of_n_eval_on_train_examples),
        checkpoint_dir=FLAGS.checkpoint_dir,
        wait_interval=300, timeout=FLAGS.eval_timeout)
  else:
    if FLAGS.use_tpu:
      # TPU is automatically inferred if tpu_name is None and
      # we are running under cloud ai-platform.
      resolver = tf.distribute.cluster_resolver.TPUClusterResolver(
          FLAGS.tpu_name)
      tf.config.experimental_connect_to_cluster(resolver)
      tf.tpu.experimental.initialize_tpu_system(resolver)
      strategy = tf.distribute.experimental.TPUStrategy(resolver)
    elif FLAGS.num_workers > 1:
      strategy = tf.distribute.experimental.MultiWorkerMirroredStrategy()
    else:
      strategy = tf.compat.v2.distribute.MirroredStrategy()

    with strategy.scope():
      model_lib_v2.train_loop(
          pipeline_config_path=FLAGS.pipeline_config_path,
          model_dir=FLAGS.model_dir,
          train_steps=FLAGS.num_train_steps,
          use_tpu=FLAGS.use_tpu,
          checkpoint_every_n=FLAGS.checkpoint_every_n,
          record_summaries=FLAGS.record_summaries)

if __name__ == '__main__':
  tf.compat.v1.app.run()

pipeline.config file:
# Mask R-CNN with Inception Resnet v2 (no atrous)
# Sync-trained on COCO (with 8 GPUs) with batch size 16 (1024x1024 resolution)
# Initialized from Imagenet classification checkpoint
# TF2-Compatible, *Not* TPU-Compatible
#
# Achieves XXX mAP on COCO

model {
  faster_rcnn {
    number_of_stages: 3
    num_classes: 1
    image_resizer {
      fixed_shape_resizer {
        height: 1024
        width: 1024
        # pad_to_max_dimension: true
      }
    }
    feature_extractor {
      type: 'faster_rcnn_inception_resnet_v2_keras'
    }
    first_stage_anchor_generator {
      grid_anchor_generator {
        scales: [0.25, 0.5, 1.0, 2.0]
        aspect_ratios: [0.5, 1.0, 2.0]
        height_stride: 16
        width_stride: 16
      }
    }
    first_stage_box_predictor_conv_hyperparams {
      op: CONV
      regularizer {
        l2_regularizer {
          weight: 0.0
        }
      }
      initializer {
        truncated_normal_initializer {
          stddev: 0.01
        }
      }
    }
    first_stage_nms_score_threshold: 0.0
    first_stage_nms_iou_threshold: 0.7
    first_stage_max_proposals: 300
    first_stage_localization_loss_weight: 2.0
    first_stage_objectness_loss_weight: 1.0
    initial_crop_size: 17
    maxpool_kernel_size: 1
    maxpool_stride: 1
    second_stage_box_predictor {
      mask_rcnn_box_predictor {
        use_dropout: false
        dropout_keep_probability: 1.0
        fc_hyperparams {
          op: FC
          regularizer {
            l2_regularizer {
              weight: 0.0
            }
          }
          initializer {
            variance_scaling_initializer {
              factor: 1.0
              uniform: true
              mode: FAN_AVG
            }
          }
        }
        mask_height: 33
        mask_width: 33
        mask_prediction_conv_depth: 0
        mask_prediction_num_conv_layers: 4
        conv_hyperparams {
          op: CONV
          regularizer {
            l2_regularizer {
              weight: 0.0
            }
          }
          initializer {
            truncated_normal_initializer {
              stddev: 0.01
            }
          }
        }
        predict_instance_masks: true
      }
    }
    second_stage_post_processing {
      batch_non_max_suppression {
        score_threshold: 0.0
        iou_threshold: 0.6
        max_detections_per_class: 100
        max_total_detections: 100
      }
      score_converter: SOFTMAX
    }
    second_stage_localization_loss_weight: 2.0
    second_stage_classification_loss_weight: 1.0
    second_stage_mask_prediction_loss_weight: 4.0
    resize_masks: false
  }
}

train_config: {
  batch_size: 1
  num_steps: 200000
  optimizer {
    momentum_optimizer: {
      learning_rate: {
        cosine_decay_learning_rate {
          learning_rate_base: 0.008
          total_steps: 200000
          warmup_learning_rate: 0.0
          warmup_steps: 5000
        }
      }
      momentum_optimizer_value: 0.9
    }
    use_moving_average: false
  }
  gradient_clipping_by_norm: 10.0
  fine_tune_checkpoint: "pre-trained-models/mask_rcnn_inception_resnet_v2_1024x1024_coco17_gpu-8/checkpoint/ckpt-0"
  data_augmentation_options {
    random_horizontal_flip {
    }
  }
}

train_input_reader: {
  tf_record_input_reader {
    input_path: "annotations/train.record"
  }
  load_instance_masks: true
  mask_type: PNG_MASKS
}

eval_config: {
  metrics_set: "coco_detection_metrics"
  metrics_set: "coco_mask_metrics"
  eval_instance_masks: true
  use_moving_averages: false
  batch_size: 1
  include_metrics_per_category: true
}

eval_input_reader: {
  label_map_path: "annotations/label_map.pbtxt"
  shuffle: false
  num_epochs: 1
  tf_record_input_reader {
    input_path: "annotations/test.record"
  }
  load_instance_masks: true
  mask_type: PNG_MASKS
}

app.py (first reference):
# Copyright 2015 The TensorFlow Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================

"""Generic entry point script."""
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import sys as _sys

from absl.app import run as _run

from tensorflow.python.platform import flags
from tensorflow.python.util.tf_export import tf_export

def _parse_flags_tolerate_undef(argv):
  """Parse args, returning any unknown flags (ABSL defaults to crashing)."""
  return flags.FLAGS(_sys.argv if argv is None else argv, known_only=True)

@tf_export(v1=['app.run'])
def run(main=None, argv=None):
  """Runs the program with an optional 'main' function and 'argv' list."""

  main = main or _sys.modules['__main__'].main

  _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)

app.py (second and third reference):
# Copyright 2017 The Abseil Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

"""Generic entry point for Abseil Python applications.

To use this module, define a 'main' function with a single 'argv' argument and
call app.run(main). For example:

    def main(argv):
      if len(argv) > 1:
        raise app.UsageError('Too many command-line arguments.')

    if __name__ == '__main__':
      app.run(main)
"""

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import collections
import errno
import os
import pdb
import sys
import traceback

from absl import command_name
from absl import flags
from absl import logging

try:
  import faulthandler
except ImportError:
  faulthandler = None

FLAGS = flags.FLAGS

flags.DEFINE_boolean('run_with_pdb', False, 'Set to true for PDB debug mode')
flags.DEFINE_boolean('pdb_post_mortem', False,
                     'Set to true to handle uncaught exceptions with PDB '
                     'post mortem.')
flags.DEFINE_alias('pdb', 'pdb_post_mortem')
flags.DEFINE_boolean('run_with_profiling', False,
                     'Set to true for profiling the script. '
                     'Execution will be slower, and the output format might '
                     'change over time.')
flags.DEFINE_string('profile_file', None,
                    'Dump profile information to a file (for python -m '
                    'pstats). Implies --run_with_profiling.')
flags.DEFINE_boolean('use_cprofile_for_profiling', True,
                     'Use cProfile instead of the profile module for '
                     'profiling. This has no effect unless '
                     '--run_with_profiling is set.')
flags.DEFINE_boolean('only_check_args', False,
                     'Set to true to validate args and exit.',
                     allow_hide_cpp=True)

# If main() exits via an abnormal exception, call into these
# handlers before exiting.
EXCEPTION_HANDLERS = []

class Error(Exception):
  pass

class UsageError(Error):
  """Exception raised when the arguments supplied by the user are invalid.

  Raise this when the arguments supplied are invalid from the point of
  view of the application. For example when two mutually exclusive
  flags have been supplied or when there are not enough non-flag
  arguments. It is distinct from flags.Error which covers the lower
  level of parsing and validating individual flags.
  """

  def __init__(self, message, exitcode=1):
    super(UsageError, self).__init__(message)
    self.exitcode = exitcode

class HelpFlag(flags.BooleanFlag):
  """Special boolean flag that displays usage and raises SystemExit."""
  NAME = 'help'
  SHORT_NAME = '?'

  def __init__(self):
    super(HelpFlag, self).__init__(
        self.NAME, False, 'show this help',
        short_name=self.SHORT_NAME, allow_hide_cpp=True)

  def parse(self, arg):
    if self._parse(arg):
      usage(shorthelp=True, writeto_stdout=True)
      # Advertise --helpfull on stdout, since usage() was on stdout.
      print()
      print('Try --helpfull to get a list of all flags.')
      sys.exit(1)

class HelpshortFlag(HelpFlag):
  """--helpshort is an alias for --help."""
  NAME = 'helpshort'
  SHORT_NAME = None

class HelpfullFlag(flags.BooleanFlag):
  """Display help for flags in the main module and all dependent modules."""

  def __init__(self):
    super(HelpfullFlag, self).__init__(
        'helpfull', False, 'show full help', allow_hide_cpp=True)

  def parse(self, arg):
    if self._parse(arg):
      usage(writeto_stdout=True)
      sys.exit(1)

class HelpXMLFlag(flags.BooleanFlag):
  """Similar to HelpfullFlag, but generates output in XML format."""

  def __init__(self):
    super(HelpXMLFlag, self).__init__(
        'helpxml', False, 'like --helpfull, but generates XML output',
        allow_hide_cpp=True)

  def parse(self, arg):
    if self._parse(arg):
      flags.FLAGS.write_help_in_xml_format(sys.stdout)
      sys.exit(1)

def parse_flags_with_usage(args):
  """Tries to parse the flags, print usage, and exit if unparseable.

  Args:
    args: [str], a non-empty list of the command line arguments including
        program name.

  Returns:
    [str], a non-empty list of remaining command line arguments after parsing
    flags, including program name.
  """
  try:
    return FLAGS(args)
  except flags.Error as error:
    sys.stderr.write('FATAL Flags parsing error: %s\n' % error)
    sys.stderr.write('Pass --helpshort or --helpfull to see help on flags.\n')
    sys.exit(1)

_define_help_flags_called = False

def define_help_flags():
  """Registers help flags. Idempotent."""
  # Use a global to ensure idempotence.
  global _define_help_flags_called

  if not _define_help_flags_called:
    flags.DEFINE_flag(HelpFlag())
    flags.DEFINE_flag(HelpshortFlag())  # alias for --help
    flags.DEFINE_flag(HelpfullFlag())
    flags.DEFINE_flag(HelpXMLFlag())
    _define_help_flags_called = True

def _register_and_parse_flags_with_usage(
    argv=None,
    flags_parser=parse_flags_with_usage,
):
  """Registers help flags, parses arguments and shows usage if appropriate.

  This also calls sys.exit(0) if flag --only_check_args is True.

  Args:
    argv: [str], a non-empty list of the command line arguments including
        program name, sys.argv is used if None.
    flags_parser: Callable[[List[Text]], Any], the function used to parse flags.
        The return value of this function is passed to `main` untouched.
        It must guarantee FLAGS is parsed after this function is called.

  Returns:
    The return value of `flags_parser`. When using the default `flags_parser`,
    it returns the following:
    [str], a non-empty list of remaining command line arguments after parsing
    flags, including program name.

  Raises:
    Error: Raised when flags_parser is called, but FLAGS is not parsed.
    SystemError: Raised when it's called more than once.
  """
  if _register_and_parse_flags_with_usage.done:
    raise SystemError('Flag registration can be done only once.')

  define_help_flags()

  original_argv = sys.argv if argv is None else argv
  args_to_main = flags_parser(original_argv)
  if not FLAGS.is_parsed():
    raise Error('FLAGS must be parsed after flags_parser is called.')

  # Exit when told so.
  if FLAGS.only_check_args:
    sys.exit(0)
  # Immediately after flags are parsed, bump verbosity to INFO if the flag has
  # not been set.
  if FLAGS['verbosity'].using_default_value:
    FLAGS.verbosity = 0
  _register_and_parse_flags_with_usage.done = True

  return args_to_main

_register_and_parse_flags_with_usage.done = False

def _run_main(main, argv):
  """Calls main, optionally with pdb or profiler."""
  if FLAGS.run_with_pdb:
    sys.exit(pdb.runcall(main, argv))
  elif FLAGS.run_with_profiling or FLAGS.profile_file:
    # Avoid import overhead since most apps (including performance-sensitive
    # ones) won't be run with profiling.
    import atexit
    if FLAGS.use_cprofile_for_profiling:
      import cProfile as profile
    else:
      import profile
    profiler = profile.Profile()
    if FLAGS.profile_file:
      atexit.register(profiler.dump_stats, FLAGS.profile_file)
    else:
      atexit.register(profiler.print_stats)
    retval = profiler.runcall(main, argv)
    sys.exit(retval)
  else:
    sys.exit(main(argv))

def _call_exception_handlers(exception):
  """Calls any installed exception handlers."""
  for handler in EXCEPTION_HANDLERS:
    try:
      if handler.wants(exception):
        handler.handle(exception)
    except:  # pylint: disable=bare-except
      try:
        # We don't want to stop for exceptions in the exception handlers but
        # we shouldn't hide them either.
        logging.error(traceback.format_exc())
      except:  # pylint: disable=bare-except
        # In case even the logging statement fails, ignore.
        pass

def run(
    main,
    argv=None,
    flags_parser=parse_flags_with_usage,
):
  """Begins executing the program.

  Args:
    main: The main function to execute. It takes an single argument "argv",
        which is a list of command line arguments with parsed flags removed.
        The return value is passed to `sys.exit`, and so for example
        a return value of 0 or None results in a successful termination, whereas
        a return value of 1 results in abnormal termination.
        For more details, see https://docs.python.org/3/library/sys#sys.exit
    argv: A non-empty list of the command line arguments including program name,
        sys.argv is used if None.
    flags_parser: Callable[[List[Text]], Any], the function used to parse flags.
        The return value of this function is passed to `main` untouched.
        It must guarantee FLAGS is parsed after this function is called.
  - Parses command line flags with the flag module.
  - If there are any errors, prints usage().
  - Calls main() with the remaining arguments.
  - If main() raises a UsageError, prints usage and the error message.
  """
  try:
    args = _run_init(
        sys.argv if argv is None else argv,
        flags_parser,
    )
    while _init_callbacks:
      callback = _init_callbacks.popleft()
      callback()
    try:
      _run_main(main, args)
    except UsageError as error:
      usage(shorthelp=True, detailed_error=error, exitcode=error.exitcode)
    except:
      exc = sys.exc_info()[1]
      # Don't try to post-mortem debug successful SystemExits, since those
      # mean there wasn't actually an error. In particular, the test framework
      # raises SystemExit(False) even if all tests passed.
      if isinstance(exc, SystemExit) and not exc.code:
        raise

      # Check the tty so that we don't hang waiting for input in an
      # non-interactive scenario.
      if FLAGS.pdb_post_mortem and sys.stdout.isatty():
        traceback.print_exc()
        print()
        print(' *** Entering post-mortem debugging ***')
        print()
        pdb.post_mortem()
      raise
  except Exception as e:
    _call_exception_handlers(e)
    raise

# Callbacks which have been deferred until after _run_init has been called.
_init_callbacks = collections.deque()

def call_after_init(callback):
  """Calls the given callback only once ABSL has finished initialization.

  If ABSL has already finished initialization when `call_after_init` is
  called then the callback is executed immediately, otherwise `callback` is
  stored to be executed after `app.run` has finished initializing (aka. just
  before the main function is called).

  If called after `app.run`, this is equivalent to calling `callback()` in the
  caller thread. If called before `app.run`, callbacks are run sequentially (in
  an undefined order) in the same thread as `app.run`.

  Args:
    callback: a callable to be called once ABSL has finished initialization.
      This may be immediate if initialization has already finished. It
      takes no arguments and returns nothing.
  """
  if _run_init.done:
    callback()
  else:
    _init_callbacks.append(callback)

def _run_init(
    argv,
    flags_parser,
):
  """Does one-time initialization and re-parses flags on rerun."""
  if _run_init.done:
    return flags_parser(argv)
  command_name.make_process_name_useful()
  # Set up absl logging handler.
  logging.use_absl_handler()
  args = _register_and_parse_flags_with_usage(
      argv=argv,
      flags_parser=flags_parser,
  )
  if faulthandler:
    try:
      faulthandler.enable()
    except Exception:  # pylint: disable=broad-except
      # Some tests verify stderr output very closely, so don't print anything.
      # Disabled faulthandler is a low-impact error.
      pass
  _run_init.done = True
  return args

_run_init.done = False

def usage(shorthelp=False, writeto_stdout=False, detailed_error=None,
          exitcode=None):
  """Writes __main__'s docstring to stderr with some help text.

  Args:
    shorthelp: bool, if True, prints only flags from the main module,
        rather than all flags.
    writeto_stdout: bool, if True, writes help message to stdout,
        rather than to stderr.
    detailed_error: str, additional detail about why usage info was presented.
    exitcode: optional integer, if set, exits with this status code after
        writing help.
  """
  if writeto_stdout:
    stdfile = sys.stdout
  else:
    stdfile = sys.stderr

  doc = sys.modules['__main__'].__doc__
  if not doc:
    doc = '\nUSAGE: %s [flags]\n' % sys.argv[0]
    doc = flags.text_wrap(doc, indent='       ', firstline_indent='')
  else:
    # Replace all '%s' with sys.argv[0], and all '%%' with '%'.
    num_specifiers = doc.count('%') - 2 * doc.count('%%')
    try:
      doc %= (sys.argv[0],) * num_specifiers
    except (OverflowError, TypeError, ValueError):
      # Just display the docstring as-is.
      pass
  if shorthelp:
    flag_str = FLAGS.main_module_help()
  else:
    flag_str = FLAGS.get_help()
  try:
    stdfile.write(doc)
    if flag_str:
      stdfile.write('\nflags:\n')
      stdfile.write(flag_str)
    stdfile.write('\n')
    if detailed_error is not None:
      stdfile.write('\n%s\n' % detailed_error)
  except IOError as e:
    # We avoid printing a huge backtrace if we get EPIPE, because
    # "foo.par --help | less" is a frequent use case.
    if e.errno != errno.EPIPE:
      raise
  if exitcode is not None:
    sys.exit(exitcode)

class ExceptionHandler(object):
  """Base exception handler from which other may inherit."""

  def wants(self, exc):
    """Returns whether this handler wants to handle the exception or not.

    This base class returns True for all exceptions by default. Override in
    subclass if it wants to be more selective.

    Args:
      exc: Exception, the current exception.
    """
    del exc  # Unused.
    return True

  def handle(self, exc):
    """Do something with the current exception.

    Args:
      exc: Exception, the current exception

    This method must be overridden.
    """
    raise NotImplementedError()

def install_exception_handler(handler):
  """Installs an exception handler.

  Args:
    handler: ExceptionHandler, the exception handler to install.

  Raises:
    TypeError: Raised when the handler was not of the correct type.

  All installed exception handlers will be called if main() exits via
  an abnormal exception, i.e. not one of SystemExit, KeyboardInterrupt,
  FlagsError or UsageError.
  """
  if not isinstance(handler, ExceptionHandler):
    raise TypeError('handler of type %s does not inherit from ExceptionHandler'
                    % type(handler))
  EXCEPTION_HANDLERS.append(handler)

model_lib_v2.py:
# Copyright 2019 The TensorFlow Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================
r"""Constructs model, inputs, and training environment."""

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import copy
import os
import time

import tensorflow.compat.v1 as tf
import tensorflow.compat.v2 as tf2

from object_detection import eval_util
from object_detection import inputs
from object_detection import model_lib
from object_detection.builders import optimizer_builder
from object_detection.core import standard_fields as fields
from object_detection.protos import train_pb2
from object_detection.utils import config_util
from object_detection.utils import label_map_util
from object_detection.utils import ops
from object_detection.utils import visualization_utils as vutils

# pylint: disable=g-import-not-at-top
try:
  from tensorflow.contrib import tpu as contrib_tpu
except ImportError:
  # TF 2.0 doesn't ship with contrib.
  pass
# pylint: enable=g-import-not-at-top

MODEL_BUILD_UTIL_MAP = model_lib.MODEL_BUILD_UTIL_MAP

RESTORE_MAP_ERROR_TEMPLATE = (
    'Since we are restoring a v2 style checkpoint'
    ' restore_map was expected to return a (str -> Model) mapping,'
    ' but we received a ({} -> {}) mapping instead.'
)

def _compute_losses_and_predictions_dicts(
    model, features, labels,
    add_regularization_loss=True):
  """Computes the losses dict and predictions dict for a model on inputs.

  Args:
    model: a DetectionModel (based on Keras).
    features: Dictionary of feature tensors from the input dataset.
      Should be in the format output by `inputs.train_input` and
      `inputs.eval_input`.
        features[fields.InputDataFields.image] is a [batch_size, H, W, C]
          float32 tensor with preprocessed images.
        features[HASH_KEY] is a [batch_size] int32 tensor representing unique
          identifiers for the images.
        features[fields.InputDataFields.true_image_shape] is a [batch_size, 3]
          int32 tensor representing the true image shapes, as preprocessed
          images could be padded.
        features[fields.InputDataFields.original_image] (optional) is a
          [batch_size, H, W, C] float32 tensor with original images.
    labels: A dictionary of groundtruth tensors post-unstacking. The original
      labels are of the form returned by `inputs.train_input` and
      `inputs.eval_input`. The shapes may have been modified by unstacking with
      `model_lib.unstack_batch`. However, the dictionary includes the following
      fields.
        labels[fields.InputDataFields.num_groundtruth_boxes] is a
          int32 tensor indicating the number of valid groundtruth boxes
          per image.
        labels[fields.InputDataFields.groundtruth_boxes] is a float32 tensor
          containing the corners of the groundtruth boxes.
        labels[fields.InputDataFields.groundtruth_classes] is a float32
          one-hot tensor of classes.
        labels[fields.InputDataFields.groundtruth_weights] is a float32 tensor
          containing groundtruth weights for the boxes.
        -- Optional --
        labels[fields.InputDataFields.groundtruth_instance_masks] is a
          float32 tensor containing only binary values, which represent
          instance masks for objects.
        labels[fields.InputDataFields.groundtruth_keypoints] is a
          float32 tensor containing keypoints for each box.
        labels[fields.InputDataFields.groundtruth_dp_num_points] is an int32
          tensor with the number of sampled DensePose points per object.
        labels[fields.InputDataFields.groundtruth_dp_part_ids] is an int32
          tensor with the DensePose part ids (0-indexed) per object.
        labels[fields.InputDataFields.groundtruth_dp_surface_coords] is a
          float32 tensor with the DensePose surface coordinates.
        labels[fields.InputDataFields.groundtruth_group_of] is a tf.bool tensor
          containing group_of annotations.
        labels[fields.InputDataFields.groundtruth_labeled_classes] is a float32
          k-hot tensor of classes.
        labels[fields.InputDataFields.groundtruth_track_ids] is a int32
          tensor of track IDs.
    add_regularization_loss: Whether or not to include the model's
      regularization loss in the losses dictionary.

  Returns:
    A tuple containing the losses dictionary (with the total loss under
    the key 'Loss/total_loss'), and the predictions dictionary produced by
    `model.predict`.

  """
  model_lib.provide_groundtruth(model, labels)
  preprocessed_images = features[fields.InputDataFields.image]

  prediction_dict = model.predict(
      preprocessed_images,
      features[fields.InputDataFields.true_image_shape],
      **model.get_side_inputs(features))
  prediction_dict = ops.bfloat16_to_float32_nested(prediction_dict)

  losses_dict = model.loss(
      prediction_dict, features[fields.InputDataFields.true_image_shape])
  losses = [loss_tensor for loss_tensor in losses_dict.values()]
  if add_regularization_loss:
    # TODO(kaftan): As we figure out mixed precision & bfloat 16, we may
    ## need to convert these regularization losses from bfloat16 to float32
    ## as well.
    regularization_losses = model.regularization_losses()
    if regularization_losses:
      regularization_losses = ops.bfloat16_to_float32_nested(
          regularization_losses)
      regularization_loss = tf.add_n(
          regularization_losses, name='regularization_loss')
      losses.append(regularization_loss)
      losses_dict['Loss/regularization_loss'] = regularization_loss

  total_loss = tf.add_n(losses, name='total_loss')
  losses_dict['Loss/total_loss'] = total_loss

  return losses_dict, prediction_dict

# TODO(kaftan): Explore removing learning_rate from this method & returning
## The full losses dict instead of just total_loss, then doing all summaries
## saving in a utility method called by the outer training loop.
# TODO(kaftan): Explore adding gradient summaries
def eager_train_step(detection_model,
                     features,
                     labels,
                     unpad_groundtruth_tensors,
                     optimizer,
                     learning_rate,
                     add_regularization_loss=True,
                     clip_gradients_value=None,
                     global_step=None,
                     num_replicas=1.0):
  """Process a single training batch.

  This method computes the loss for the model on a single training batch,
  while tracking the gradients with a gradient tape. It then updates the
  model variables with the optimizer, clipping the gradients if
  clip_gradients_value is present.

  This method can run eagerly or inside a tf.function.

  Args:
    detection_model: A DetectionModel (based on Keras) to train.
    features: Dictionary of feature tensors from the input dataset.
      Should be in the format output by `inputs.train_input.
        features[fields.InputDataFields.image] is a [batch_size, H, W, C]
          float32 tensor with preprocessed images.
        features[HASH_KEY] is a [batch_size] int32 tensor representing unique
          identifiers for the images.
        features[fields.InputDataFields.true_image_shape] is a [batch_size, 3]
          int32 tensor representing the true image shapes, as preprocessed
          images could be padded.
        features[fields.InputDataFields.original_image] (optional, not used
          during training) is a
          [batch_size, H, W, C] float32 tensor with original images.
    labels: A dictionary of groundtruth tensors. This method unstacks
      these labels using model_lib.unstack_batch. The stacked labels are of
      the form returned by `inputs.train_input` and `inputs.eval_input`.
        labels[fields.InputDataFields.num_groundtruth_boxes] is a [batch_size]
          int32 tensor indicating the number of valid groundtruth boxes
          per image.
        labels[fields.InputDataFields.groundtruth_boxes] is a
          [batch_size, num_boxes, 4] float32 tensor containing the corners of
          the groundtruth boxes.
        labels[fields.InputDataFields.groundtruth_classes] is a
          [batch_size, num_boxes, num_classes] float32 one-hot tensor of
          classes. num_classes includes the background class.
        labels[fields.InputDataFields.groundtruth_weights] is a
          [batch_size, num_boxes] float32 tensor containing groundtruth weights
          for the boxes.
        -- Optional --
        labels[fields.InputDataFields.groundtruth_instance_masks] is a
          [batch_size, num_boxes, H, W] float32 tensor containing only binary
          values, which represent instance masks for objects.
        labels[fields.InputDataFields.groundtruth_keypoints] is a
          [batch_size, num_boxes, num_keypoints, 2] float32 tensor containing
          keypoints for each box.
        labels[fields.InputDataFields.groundtruth_dp_num_points] is a
          [batch_size, num_boxes] int32 tensor with the number of DensePose
          sampled points per instance.
        labels[fields.InputDataFields.groundtruth_dp_part_ids] is a
          [batch_size, num_boxes, max_sampled_points] int32 tensor with the
          part ids (0-indexed) for each instance.
        labels[fields.InputDataFields.groundtruth_dp_surface_coords] is a
          [batch_size, num_boxes, max_sampled_points, 4] float32 tensor with the
          surface coordinates for each point. Each surface coordinate is of the
          form (y, x, v, u) where (y, x) are normalized image locations and
          (v, u) are part-relative normalized surface coordinates.
        labels[fields.InputDataFields.groundtruth_labeled_classes] is a float32
          k-hot tensor of classes.
        labels[fields.InputDataFields.groundtruth_track_ids] is a int32
          tensor of track IDs.
    unpad_groundtruth_tensors: A parameter passed to unstack_batch.
    optimizer: The training optimizer that will update the variables.
    learning_rate: The learning rate tensor for the current training step.
      This is used only for TensorBoard logging purposes, it does not affect
       model training.
    add_regularization_loss: Whether or not to include the model's
      regularization loss in the losses dictionary.
    clip_gradients_value: If this is present, clip the gradients global norm
      at this value using `tf.clip_by_global_norm`.
    global_step: The current training step. Used for TensorBoard logging
      purposes. This step is not updated by this function and must be
      incremented separately.
    num_replicas: The number of replicas in the current distribution strategy.
      This is used to scale the total loss so that training in a distribution
      strategy works correctly.

  Returns:
    The total loss observed at this training step
  """
  # """Execute a single training step in the TF v2 style loop."""
  is_training = True

  detection_model._is_training = is_training  # pylint: disable=protected-access
  tf.keras.backend.set_learning_phase(is_training)

  labels = model_lib.unstack_batch(
      labels, unpad_groundtruth_tensors=unpad_groundtruth_tensors)

  with tf.GradientTape() as tape:
    losses_dict, _ = _compute_losses_and_predictions_dicts(
        detection_model, features, labels, add_regularization_loss)

    total_loss = losses_dict['Loss/total_loss']

    # Normalize loss for num replicas
    total_loss = tf.math.divide(total_loss,
                                tf.constant(num_replicas, dtype=tf.float32))
    losses_dict['Loss/normalized_total_loss'] = total_loss

  for loss_type in losses_dict:
    tf.compat.v2.summary.scalar(
        loss_type, losses_dict[loss_type], step=global_step)

  trainable_variables = detection_model.trainable_variables

  gradients = tape.gradient(total_loss, trainable_variables)

  if clip_gradients_value:
    gradients, _ = tf.clip_by_global_norm(gradients, clip_gradients_value)
  optimizer.apply_gradients(zip(gradients, trainable_variables))
  tf.compat.v2.summary.scalar('learning_rate', learning_rate, step=global_step)
  tf.compat.v2.summary.image(
      name='train_input_images',
      step=global_step,
      data=features[fields.InputDataFields.image],
      max_outputs=3)
  return total_loss

def validate_tf_v2_checkpoint_restore_map(checkpoint_restore_map):
  """Ensure that given dict is a valid TF v2 style restore map.

  Args:
    checkpoint_restore_map: A nested dict mapping strings to
      tf.keras.Model objects.

  Raises:
    ValueError: If they keys in checkpoint_restore_map are not strings or if
      the values are not keras Model objects.

  """

  for key, value in checkpoint_restore_map.items():
    if not (isinstance(key, str) and
            (isinstance(value, tf.Module)
             or isinstance(value, tf.train.Checkpoint))):
      if isinstance(key, str) and isinstance(value, dict):
        validate_tf_v2_checkpoint_restore_map(value)
      else:
        raise TypeError(
            RESTORE_MAP_ERROR_TEMPLATE.format(key.__class__.__name__,
                                              value.__class__.__name__))

def is_object_based_checkpoint(checkpoint_path):
  """Returns true if `checkpoint_path` points to an object-based checkpoint."""
  var_names = [var[0] for var in tf.train.list_variables(checkpoint_path)]
  return '_CHECKPOINTABLE_OBJECT_GRAPH' in var_names

def load_fine_tune_checkpoint(
    model, checkpoint_path, checkpoint_type, checkpoint_version, input_dataset,
    unpad_groundtruth_tensors):
  """Load a fine tuning classification or detection checkpoint.

  To make sure the model variables are all built, this method first executes
  the model by computing a dummy loss. (Models might not have built their
  variables before their first execution)

  It then loads an object-based classification or detection checkpoint.

  This method updates the model in-place and does not return a value.

  Args:
    model: A DetectionModel (based on Keras) to load a fine-tuning
      checkpoint for.
    checkpoint_path: Directory with checkpoints file or path to checkpoint.
    checkpoint_type: Whether to restore from a full detection
      checkpoint (with compatible variable names) or to restore from a
      classification checkpoint for initialization prior to training.
      Valid values: `detection`, `classification`.
    checkpoint_version: train_pb2.CheckpointVersion.V1 or V2 enum indicating
      whether to load checkpoints in V1 style or V2 style.  In this binary
      we only support V2 style (object-based) checkpoints.
    input_dataset: The tf.data Dataset the model is being trained on. Needed
      to get the shapes for the dummy loss computation.
    unpad_groundtruth_tensors: A parameter passed to unstack_batch.

  Raises:
    IOError: if `checkpoint_path` does not point at a valid object-based
      checkpoint
    ValueError: if `checkpoint_version` is not train_pb2.CheckpointVersion.V2
  """
  if not is_object_based_checkpoint(checkpoint_path):
    raise IOError('Checkpoint is expected to be an object-based checkpoint.')
  if checkpoint_version == train_pb2.CheckpointVersion.V1:
    raise ValueError('Checkpoint version should be V2')

  features, labels = iter(input_dataset).next()

  @tf.function
  def _dummy_computation_fn(features, labels):
    model._is_training = False  # pylint: disable=protected-access
    tf.keras.backend.set_learning_phase(False)

    labels = model_lib.unstack_batch(
        labels, unpad_groundtruth_tensors=unpad_groundtruth_tensors)

    return _compute_losses_and_predictions_dicts(
        model,
        features,
        labels)

  strategy = tf.compat.v2.distribute.get_strategy()
  if hasattr(tf.distribute.Strategy, 'run'):
    strategy.run(
        _dummy_computation_fn, args=(
            features,
            labels,
        ))
  else:
    strategy.experimental_run_v2(
        _dummy_computation_fn, args=(
            features,
            labels,
        ))

  restore_from_objects_dict = model.restore_from_objects(
      fine_tune_checkpoint_type=checkpoint_type)
  validate_tf_v2_checkpoint_restore_map(restore_from_objects_dict)
  ckpt = tf.train.Checkpoint(**restore_from_objects_dict)
  ckpt.restore(checkpoint_path).assert_existing_objects_matched()

def get_filepath(strategy, filepath):
  """Get appropriate filepath for worker.

  Args:
    strategy: A tf.distribute.Strategy object.
    filepath: A path to where the Checkpoint object is stored.

  Returns:
    A temporary filepath for non-chief workers to use or the original filepath
    for the chief.
  """
  if strategy.extended.should_checkpoint:
    return filepath
  else:
    # TODO(vighneshb) Replace with the public API when TF exposes it.
    task_id = strategy.extended._task_id  # pylint:disable=protected-access
    return os.path.join(filepath, 'temp_worker_{:03d}'.format(task_id))

def clean_temporary_directories(strategy, filepath):
  """Temporary directory clean up for MultiWorker Mirrored Strategy.

  This is needed for all non-chief workers.

  Args:
    strategy: A tf.distribute.Strategy object.
    filepath: The filepath for the temporary directory.
  """
  if not strategy.extended.should_checkpoint:
    if tf.io.gfile.exists(filepath) and tf.io.gfile.isdir(filepath):
      tf.io.gfile.rmtree(filepath)

def train_loop(
    pipeline_config_path,
    model_dir,
    config_override=None,
    train_steps=None,
    use_tpu=False,
    save_final_config=False,
    checkpoint_every_n=1000,
    checkpoint_max_to_keep=7,
    record_summaries=True,
    **kwargs):
  """Trains a model using eager + functions.

  This method:
    1. Processes the pipeline configs
    2. (Optionally) saves the as-run config
    3. Builds the model & optimizer
    4. Gets the training input data
    5. Loads a fine-tuning detection or classification checkpoint if requested
    6. Loops over the train data, executing distributed training steps inside
       tf.functions.
    7. Checkpoints the model every `checkpoint_every_n` training steps.
    8. Logs the training metrics as TensorBoard summaries.

  Args:
    pipeline_config_path: A path to a pipeline config file.
    model_dir:
      The directory to save checkpoints and summaries to.
    config_override: A pipeline_pb2.TrainEvalPipelineConfig text proto to
      override the config from `pipeline_config_path`.
    train_steps: Number of training steps. If None, the number of training steps
      is set from the `TrainConfig` proto.
    use_tpu: Boolean, whether training and evaluation should run on TPU.
    save_final_config: Whether to save final config (obtained after applying
      overrides) to `model_dir`.
    checkpoint_every_n:
      Checkpoint every n training steps.
    checkpoint_max_to_keep:
      int, the number of most recent checkpoints to keep in the model directory.
    record_summaries: Boolean, whether or not to record summaries.
    **kwargs: Additional keyword arguments for configuration override.
  """
  ## Parse the configs
  get_configs_from_pipeline_file = MODEL_BUILD_UTIL_MAP[
      'get_configs_from_pipeline_file']
  merge_external_params_with_configs = MODEL_BUILD_UTIL_MAP[
      'merge_external_params_with_configs']
  create_pipeline_proto_from_configs = MODEL_BUILD_UTIL_MAP[
      'create_pipeline_proto_from_configs']

  configs = get_configs_from_pipeline_file(
      pipeline_config_path, config_override=config_override)
  kwargs.update({
      'train_steps': train_steps,
      'use_bfloat16': configs['train_config'].use_bfloat16 and use_tpu
  })
  configs = merge_external_params_with_configs(
      configs, None, kwargs_dict=kwargs)
  model_config = configs['model']
  train_config = configs['train_config']
  train_input_config = configs['train_input_config']

  unpad_groundtruth_tensors = train_config.unpad_groundtruth_tensors
  add_regularization_loss = train_config.add_regularization_loss
  clip_gradients_value = None
  if train_config.gradient_clipping_by_norm > 0:
    clip_gradients_value = train_config.gradient_clipping_by_norm

  # update train_steps from config but only when non-zero value is provided
  if train_steps is None and train_config.num_steps != 0:
    train_steps = train_config.num_steps

  if kwargs['use_bfloat16']:
    tf.compat.v2.keras.mixed_precision.experimental.set_policy('mixed_bfloat16')

  if train_config.load_all_detection_checkpoint_vars:
    raise ValueError('train_pb2.load_all_detection_checkpoint_vars '
                     'unsupported in TF2')

  config_util.update_fine_tune_checkpoint_type(train_config)
  fine_tune_checkpoint_type = train_config.fine_tune_checkpoint_type
  fine_tune_checkpoint_version = train_config.fine_tune_checkpoint_version

  # Write the as-run pipeline config to disk.
  if save_final_config:
    pipeline_config_final = create_pipeline_proto_from_configs(configs)
    config_util.save_pipeline_config(pipeline_config_final, model_dir)

  # Build the model, optimizer, and training input
  strategy = tf.compat.v2.distribute.get_strategy()
  with strategy.scope():
    detection_model = MODEL_BUILD_UTIL_MAP['detection_model_fn_base'](
        model_config=model_config, is_training=True)

    def train_dataset_fn(input_context):
      """Callable to create train input."""
      # Create the inputs.
      train_input = inputs.train_input(
          train_config=train_config,
          train_input_config=train_input_config,
          model_config=model_config,
          model=detection_model,
          input_context=input_context)
      train_input = train_input.repeat()
      return train_input

    train_input = strategy.experimental_distribute_datasets_from_function(
        train_dataset_fn)

    global_step = tf.Variable(
        0, trainable=False, dtype=tf.compat.v2.dtypes.int64, name='global_step',
        aggregation=tf.compat.v2.VariableAggregation.ONLY_FIRST_REPLICA)
    optimizer, (learning_rate,) = optimizer_builder.build(
        train_config.optimizer, global_step=global_step)

    if callable(learning_rate):
      learning_rate_fn = learning_rate
    else:
      learning_rate_fn = lambda: learning_rate

  ## Train the model
  # Get the appropriate filepath (temporary or not) based on whether the worker
  # is the chief.
  summary_writer_filepath = get_filepath(strategy,
                                         os.path.join(model_dir, 'train'))
  if record_summaries:
    summary_writer = tf.compat.v2.summary.create_file_writer(
        summary_writer_filepath)
  else:
    summary_writer = tf2.summary.create_noop_writer()

  if use_tpu:
    num_steps_per_iteration = 100
  else:
    # TODO(b/135933080) Explore setting to 100 when GPU performance issues
    # are fixed.
    num_steps_per_iteration = 1

  with summary_writer.as_default():
    with strategy.scope():
      with tf.compat.v2.summary.record_if(
          lambda: global_step % num_steps_per_iteration == 0):
        # Load a fine-tuning checkpoint.
        if train_config.fine_tune_checkpoint:
          load_fine_tune_checkpoint(detection_model,
                                    train_config.fine_tune_checkpoint,
                                    fine_tune_checkpoint_type,
                                    fine_tune_checkpoint_version,
                                    train_input,
                                    unpad_groundtruth_tensors)

        ckpt = tf.compat.v2.train.Checkpoint(
            step=global_step, model=detection_model, optimizer=optimizer)

        manager_dir = get_filepath(strategy, model_dir)
        if not strategy.extended.should_checkpoint:
          checkpoint_max_to_keep = 1
        manager = tf.compat.v2.train.CheckpointManager(
            ckpt, manager_dir, max_to_keep=checkpoint_max_to_keep)

        # We use the following instead of manager.latest_checkpoint because
        # manager_dir does not point to the model directory when we are running
        # in a worker.
        latest_checkpoint = tf.train.latest_checkpoint(model_dir)
        ckpt.restore(latest_checkpoint)

        def train_step_fn(features, labels):
          """Single train step."""
          loss = eager_train_step(
              detection_model,
              features,
              labels,
              unpad_groundtruth_tensors,
              optimizer,
              learning_rate=learning_rate_fn(),
              add_regularization_loss=add_regularization_loss,
              clip_gradients_value=clip_gradients_value,
              global_step=global_step,
              num_replicas=strategy.num_replicas_in_sync)
          global_step.assign_add(1)
          return loss

        def _sample_and_train(strategy, train_step_fn, data_iterator):
          features, labels = data_iterator.next()
          if hasattr(tf.distribute.Strategy, 'run'):
            per_replica_losses = strategy.run(
                train_step_fn, args=(features, labels))
          else:
            per_replica_losses = strategy.experimental_run_v2(
                train_step_fn, args=(features, labels))
          # TODO(anjalisridhar): explore if it is safe to remove the
          ## num_replicas scaling of the loss and switch this to a ReduceOp.Mean
          return strategy.reduce(tf.distribute.ReduceOp.SUM,
                                 per_replica_losses, axis=None)

        @tf.function
        def _dist_train_step(data_iterator):
          """A distributed train step."""

          if num_steps_per_iteration > 1:
            for _ in tf.range(num_steps_per_iteration - 1):
              # Following suggestion on yaqs/5402607292645376
              with tf.name_scope(''):
                _sample_and_train(strategy, train_step_fn, data_iterator)

          return _sample_and_train(strategy, train_step_fn, data_iterator)

        train_input_iter = iter(train_input)

        if int(global_step.value()) == 0:
          manager.save()

        checkpointed_step = int(global_step.value())
        logged_step = global_step.value()

        last_step_time = time.time()
        for _ in range(global_step.value(), train_steps,
                       num_steps_per_iteration):

          loss = _dist_train_step(train_input_iter)

          time_taken = time.time() - last_step_time
          last_step_time = time.time()

          tf.compat.v2.summary.scalar(
              'steps_per_sec', num_steps_per_iteration * 1.0 / time_taken,
              step=global_step)

          if global_step.value() - logged_step >= 100:
            tf.logging.info(
                'Step {} per-step time {:.3f}s loss={:.3f}'.format(
                    global_step.value(), time_taken / num_steps_per_iteration,
                    loss))
            logged_step = global_step.value()

          if ((int(global_step.value()) - checkpointed_step) >=
              checkpoint_every_n):
            manager.save()
            checkpointed_step = int(global_step.value())

  # Remove the checkpoint directories of the non-chief workers that
  # MultiWorkerMirroredStrategy forces us to save during sync distributed
  # training.
  clean_temporary_directories(strategy, manager_dir)
  clean_temporary_directories(strategy, summary_writer_filepath)

def eager_eval_loop(
    detection_model,
    configs,
    eval_dataset,
    use_tpu=False,
    postprocess_on_cpu=False,
    global_step=None):
  """Evaluate the model eagerly on the evaluation dataset.

  This method will compute the evaluation metrics specified in the configs on
  the entire evaluation dataset, then return the metrics. It will also log
  the metrics to TensorBoard.

  Args:
    detection_model: A DetectionModel (based on Keras) to evaluate.
    configs: Object detection configs that specify the evaluators that should
      be used, as well as whether regularization loss should be included and
      if bfloat16 should be used on TPUs.
    eval_dataset: Dataset containing evaluation data.
    use_tpu: Whether a TPU is being used to execute the model for evaluation.
    postprocess_on_cpu: Whether model postprocessing should happen on
      the CPU when using a TPU to execute the model.
    global_step: A variable containing the training step this model was trained
      to. Used for logging purposes.

  Returns:
    A dict of evaluation metrics representing the results of this evaluation.
  """
  train_config = configs['train_config']
  eval_input_config = configs['eval_input_config']
  eval_config = configs['eval_config']
  add_regularization_loss = train_config.add_regularization_loss

  is_training = False
  detection_model._is_training = is_training  # pylint: disable=protected-access
  tf.keras.backend.set_learning_phase(is_training)

  evaluator_options = eval_util.evaluator_options_from_eval_config(
      eval_config)
  batch_size = eval_config.batch_size

  class_agnostic_category_index = (
      label_map_util.create_class_agnostic_category_index())
  class_agnostic_evaluators = eval_util.get_evaluators(
      eval_config,
      list(class_agnostic_category_index.values()),
      evaluator_options)

  class_aware_evaluators = None
  if eval_input_config.label_map_path:
    class_aware_category_index = (
        label_map_util.create_category_index_from_labelmap(
            eval_input_config.label_map_path))
    class_aware_evaluators = eval_util.get_evaluators(
        eval_config,
        list(class_aware_category_index.values()),
        evaluator_options)

  evaluators = None
  loss_metrics = {}

  @tf.function
  def compute_eval_dict(features, labels):
    """Compute the evaluation result on an image."""
    # For evaling on train data, it is necessary to check whether groundtruth
    # must be unpadded.
    boxes_shape = (
        labels[fields.InputDataFields.groundtruth_boxes].get_shape().as_list())
    unpad_groundtruth_tensors = (boxes_shape[1] is not None
                                 and not use_tpu
                                 and batch_size == 1)
    labels = model_lib.unstack_batch(
        labels, unpad_groundtruth_tensors=unpad_groundtruth_tensors)

    losses_dict, prediction_dict = _compute_losses_and_predictions_dicts(
        detection_model, features, labels, add_regularization_loss)

    def postprocess_wrapper(args):
      return detection_model.postprocess(args[0], args[1])

    # TODO(kaftan): Depending on how postprocessing will work for TPUS w/
    ## TPUStrategy, may be good to move wrapping to a utility method
    if use_tpu and postprocess_on_cpu:
      detections = contrib_tpu.outside_compilation(
          postprocess_wrapper,
          (prediction_dict, features[fields.InputDataFields.true_image_shape]))
    else:
      detections = postprocess_wrapper(
          (prediction_dict, features[fields.InputDataFields.true_image_shape]))

    class_agnostic = (
        fields.DetectionResultFields.detection_classes not in detections)
    # TODO(kaftan) (or anyone): move `_prepare_groundtruth_for_eval to eval_util
    ## and call this from there.
    groundtruth = model_lib._prepare_groundtruth_for_eval(  # pylint: disable=protected-access
        detection_model, class_agnostic, eval_input_config.max_number_of_boxes)
    use_original_images = fields.InputDataFields.original_image in features
    if use_original_images:
      eval_images = features[fields.InputDataFields.original_image]
      true_image_shapes = tf.slice(
          features[fields.InputDataFields.true_image_shape], [0, 0], [-1, 3])
      original_image_spatial_shapes = features[
          fields.InputDataFields.original_image_spatial_shape]
    else:
      eval_images = features[fields.InputDataFields.image]
      true_image_shapes = None
      original_image_spatial_shapes = None

    keys = features[inputs.HASH_KEY]
    if eval_input_config.include_source_id:
      keys = features[fields.InputDataFields.source_id]
    eval_dict = eval_util.result_dict_for_batched_example(
        eval_images,
        keys,
        detections,
        groundtruth,
        class_agnostic=class_agnostic,
        scale_to_absolute=True,
        original_image_spatial_shapes=original_image_spatial_shapes,
        true_image_shapes=true_image_shapes)

    return eval_dict, losses_dict, class_agnostic

  agnostic_categories = label_map_util.create_class_agnostic_category_index()
  per_class_categories = label_map_util.create_category_index_from_labelmap(
      eval_input_config.label_map_path)
  keypoint_edges = [
      (kp.start, kp.end) for kp in eval_config.keypoint_edge]

  for i, (features, labels) in enumerate(eval_dataset):
    eval_dict, losses_dict, class_agnostic = compute_eval_dict(features, labels)

    if class_agnostic:
      category_index = agnostic_categories
    else:
      category_index = per_class_categories

    if i % 100 == 0:
      tf.logging.info('Finished eval step %d', i)

    use_original_images = fields.InputDataFields.original_image in features
    if (use_original_images and i < eval_config.num_visualizations
        and batch_size == 1):
      sbys_image_list = vutils.draw_side_by_side_evaluation_image(
          eval_dict,
          category_index=category_index,
          max_boxes_to_draw=eval_config.max_num_boxes_to_visualize,
          min_score_thresh=eval_config.min_score_threshold,
          use_normalized_coordinates=False,
          keypoint_edges=keypoint_edges or None)
      sbys_images = tf.concat(sbys_image_list, axis=0)
      tf.compat.v2.summary.image(
          name='eval_side_by_side_' + str(i),
          step=global_step,
          data=sbys_images,
          max_outputs=eval_config.num_visualizations)
      if eval_util.has_densepose(eval_dict):
        dp_image_list = vutils.draw_densepose_visualizations(
            eval_dict)
        dp_images = tf.concat(dp_image_list, axis=0)
        tf.compat.v2.summary.image(
            name='densepose_detections_' + str(i),
            step=global_step,
            data=dp_images,
            max_outputs=eval_config.num_visualizations)

    if evaluators is None:
      if class_agnostic:
        evaluators = class_agnostic_evaluators
      else:
        evaluators = class_aware_evaluators

    for evaluator in evaluators:
      evaluator.add_eval_dict(eval_dict)

    for loss_key, loss_tensor in iter(losses_dict.items()):
      if loss_key not in loss_metrics:
        loss_metrics[loss_key] = tf.keras.metrics.Mean()
      # Skip the loss with value equal or lower than 0.0 when calculating the
      # average loss since they don't usually reflect the normal loss values
      # causing spurious average loss value.
      if loss_tensor <= 0.0:
        continue
      loss_metrics[loss_key].update_state(loss_tensor)

  eval_metrics = {}

  for evaluator in evaluators:
    eval_metrics.update(evaluator.evaluate())
  for loss_key in loss_metrics:
    eval_metrics[loss_key] = loss_metrics[loss_key].result()

  eval_metrics = {str(k): v for k, v in eval_metrics.items()}
  tf.logging.info('Eval metrics at step %d', global_step)
  for k in eval_metrics:
    tf.compat.v2.summary.scalar(k, eval_metrics[k], step=global_step)
    tf.logging.info('\t+ %s: %f', k, eval_metrics[k])

  return eval_metrics

def eval_continuously(
    pipeline_config_path,
    config_override=None,
    train_steps=None,
    sample_1_of_n_eval_examples=1,
    sample_1_of_n_eval_on_train_examples=1,
    use_tpu=False,
    override_eval_num_epochs=True,
    postprocess_on_cpu=False,
    model_dir=None,
    checkpoint_dir=None,
    wait_interval=180,
    timeout=3600,
    eval_index=None,
    **kwargs):
  """Run continuous evaluation of a detection model eagerly.

  This method builds the model, and continously restores it from the most
  recent training checkpoint in the checkpoint directory & evaluates it
  on the evaluation data.

  Args:
    pipeline_config_path: A path to a pipeline config file.
    config_override: A pipeline_pb2.TrainEvalPipelineConfig text proto to
      override the config from `pipeline_config_path`.
    train_steps: Number of training steps. If None, the number of training steps
      is set from the `TrainConfig` proto.
    sample_1_of_n_eval_examples: Integer representing how often an eval example
      should be sampled. If 1, will sample all examples.
    sample_1_of_n_eval_on_train_examples: Similar to
      `sample_1_of_n_eval_examples`, except controls the sampling of training
      data for evaluation.
    use_tpu: Boolean, whether training and evaluation should run on TPU.
    override_eval_num_epochs: Whether to overwrite the number of epochs to 1 for
      eval_input.
    postprocess_on_cpu: When use_tpu and postprocess_on_cpu are true,
      postprocess is scheduled on the host cpu.
    model_dir: Directory to output resulting evaluation summaries to.
    checkpoint_dir: Directory that contains the training checkpoints.
    wait_interval: The mimmum number of seconds to wait before checking for a
      new checkpoint.
    timeout: The maximum number of seconds to wait for a checkpoint. Execution
      will terminate if no new checkpoints are found after these many seconds.
    eval_index: int, optional If give, only evaluate the dataset at the given
      index.

    **kwargs: Additional keyword arguments for configuration override.
  """
  get_configs_from_pipeline_file = MODEL_BUILD_UTIL_MAP[
      'get_configs_from_pipeline_file']
  merge_external_params_with_configs = MODEL_BUILD_UTIL_MAP[
      'merge_external_params_with_configs']

  configs = get_configs_from_pipeline_file(
      pipeline_config_path, config_override=config_override)
  kwargs.update({
      'sample_1_of_n_eval_examples': sample_1_of_n_eval_examples,
      'use_bfloat16': configs['train_config'].use_bfloat16 and use_tpu
  })
  if train_steps is not None:
    kwargs['train_steps'] = train_steps
  if override_eval_num_epochs:
    kwargs.update({'eval_num_epochs': 1})
    tf.logging.warning(
        'Forced number of epochs for all eval validations to be 1.')
  configs = merge_external_params_with_configs(
      configs, None, kwargs_dict=kwargs)
  model_config = configs['model']
  train_input_config = configs['train_input_config']
  eval_config = configs['eval_config']
  eval_input_configs = configs['eval_input_configs']
  eval_on_train_input_config = copy.deepcopy(train_input_config)
  eval_on_train_input_config.sample_1_of_n_examples = (
      sample_1_of_n_eval_on_train_examples)
  if override_eval_num_epochs and eval_on_train_input_config.num_epochs != 1:
    tf.logging.warning('Expected number of evaluation epochs is 1, but '
                       'instead encountered `eval_on_train_input_config'
                       '.num_epochs` = '
                       '{}. Overwriting `num_epochs` to 1.'.format(
                           eval_on_train_input_config.num_epochs))
    eval_on_train_input_config.num_epochs = 1

  if kwargs['use_bfloat16']:
    tf.compat.v2.keras.mixed_precision.experimental.set_policy('mixed_bfloat16')

  detection_model = MODEL_BUILD_UTIL_MAP['detection_model_fn_base'](
      model_config=model_config, is_training=True)

  # Create the inputs.
  eval_inputs = []
  for eval_input_config in eval_input_configs:
    next_eval_input = inputs.eval_input(
        eval_config=eval_config,
        eval_input_config=eval_input_config,
        model_config=model_config,
        model=detection_model)
    eval_inputs.append((eval_input_config.name, next_eval_input))

  if eval_index is not None:
    eval_inputs = [eval_inputs[eval_index]]

  global_step = tf.compat.v2.Variable(
      0, trainable=False, dtype=tf.compat.v2.dtypes.int64)

  for latest_checkpoint in tf.train.checkpoints_iterator(
      checkpoint_dir, timeout=timeout, min_interval_secs=wait_interval):
    ckpt = tf.compat.v2.train.Checkpoint(
        step=global_step, model=detection_model)

    ckpt.restore(latest_checkpoint).expect_partial()

    for eval_name, eval_input in eval_inputs:
      summary_writer = tf.compat.v2.summary.create_file_writer(
          os.path.join(model_dir, 'eval', eval_name))
      with summary_writer.as_default():
        eager_eval_loop(
            detection_model,
            configs,
            eval_input,
            use_tpu=use_tpu,
            postprocess_on_cpu=postprocess_on_cpu,
            global_step=global_step)

https://stackoverflow.com/questions/64824662/tensorflow-valueerror-checkpoint-version-should-be-v2/64826609#64826609


Uses live-common "supported fiats" with updated fiat whitelist

not to merge.
this integrate PR https://github.com/LedgerHQ/ledger-live-common/pull/328

need #1002 merged first
need a live-common bump

will be unified in #1002


Create a Data Package (datapackage.json) online (generic - no specific tool)

Create a Data Package (datapackage.json) online (generic - no specific tool)

[ ] Reinstate datapackage-init-js demo
[ ] Walkthrough of Datapackagist

Current issues with DataPackagist

[ ] UX is confusing in many ways
[ ] Very complex codebase

This issue was moved to frictionlessdata/frictionlessdata.io#90


Internal Error

Random runtime issue is observed: -
[ERR] Internal error (/var/lib/jenkins/workspace/Build_Package-Release-All/sysrepo/src/common.c:2428
Netopeer2 version : 2.1.59
Also done a make clean, But the issue persists.
Sometimes we observe the issue when updating data in yang model using sysrepocfg --edit=vim

What could be the possible cause for the same?
Is there way I can enable debug logs for netopeer2, libyang and sysrepo to provide you to assess what wrong we are doing in our application ?

Seems like a lock problem, somehow there are more unlocks than locks.
Yes, you run the server with -d -v2, it should then print information about what is going on. But you can provide the most information by describing your use-case in such a detail that I can reproduce it. Also, the issue may have already been fixed, you can try using the latest versions in the devel branches of all our projects.


(dev) Configure iptables port forward using vpp-agent

Coverage remained the same at 76.189% when pulling a337c9f316aed1289527cecac0c4aa1aaf3a8661 on rastislavszabo:dev into d10fd204236b2228b4d3fa0656c83dbcd9c637f8 on contiv:dev.


Make request & response body a Pkl viewer

Apple.released an awesome typed configuration language that is heavily inspired by nix and it is as minimal as it could be: https://pkl-lang.org/index.html
Clicking/expanding/browsing over data types and simplifying the view of these complex data/htpp body structures would be super useful.
I think this is rather a requirement than a nice to have for full adoption of this tool, and should ship with openapi-tui rather than being an addon perhaps to the TUI.
Maybe I can also contribute to this feature, let me know if you'd appreciate a PR implementing this.

I've heard about the pkl-lang, but tbh I didn't quit get the idea. could you please give me an example?
does pkl-lang support api documentations like openapi?
p.s. contribution is highly appreciated :-)

openapi yaml configuration could be represented in pkl format in more terse and readable way. You can check the pkl examples on the website link I shared ;)
Let's see what I can do, I'll see if theres a way to hack smt together(in a good way :))


[Candidate 1v1 Sacred Land ZH v1] Map hosts large rocks on passable terrain

Observed behavior
Map hosts large rocks on passable terrain.
Expected behavior
Map hosts large rocks on impassable terrain only. Passable terrain has small rocks only.
Screenshots

Newest map update has some more changes but I think they look good enough.

This issue is still present.

Can you show me some of them please, I'll look over them again.
I've lowered many of them and small enough to me.

As shown on images above. Rocks are not as tall anymore, but still tall. I tested against USA Dozer.


[WIP] DCOS-61101: top-level form structure components

These are the components that make up the structure of a form at a macro level. There are additional subtasks on the parent ticket (DCOS-61100) to create other helpful form components.
Screenshot

I'm comfortable with this merging in it's current state. Izhar and I are meeting tomorrow to collaborate on the documentation.

@GeorgiSTodorov - thanks for spotting the missing knobs! I'll fix that today


Remember to have block and line drawing characters fill their cells properly

Just a reminder for the new Windows Terminal, remember to make sure block and line drawing characters fill their cells properly at all zoom levels.
This is really important for ANSI-art and TUI apps.

See color bleeding and horizontal lines in the larger Windows Terminal tab.
Also, some characters are randomly replaced by � (<?>) characters when changing zoom level or when printing out lot of VT markup.

You know, I coulda swore @miniksa had a PR for this just last week, but I dunno what happened to it...

Some more examples, comparing mlterm (running in Ubuntu on WSL with X410) with Windows Terminal

See it exhibits not only the block and line characters alignments and joints issues, but also identical colors rendering as different colors (the whole ANSI-art is using only the 16 palettized VT colors).
You can get the test file from the following repository : https://github.com/PhMajerus/ANSI-art/blob/master/TestPattern ANSI.ans

The identical colors rendering as different colors mentioned above has been diagnosed and hopefully fixed in #677

I just tested it and it looks like the colors are now matching correctly. So, this is gonna be fixed in #688.

I did find another bug though, look at how some characters get rendered incorrectly if I didn't execute another command before showing the test pattern.
For example:
show pic -> incorrect chars
clear -> show pic -> incorrect chars
clear -> any command (even clear) -> show pic -> correct output.

@Ghosty141 thanks for fixing and testing it, that's one thing out of the way.
What you're observing are the two other issues I described in the first message.
The terminal cells and the size of the block drawing characters do not match at all zoom level, making them overlap or not fill their cell, creating visual horizontal lines.
This is easily observed if you display █▀▄░▒▓, their heights won't be consistent. Strangely, trying these at different sizes in Word with the Consolas font shows them always filling the exact same rectangle, only Conhost and Terminal seems to stretch these characters at some sizes, sometimes by as much as 2x.
The last issue with the � characters appearing a bit randomly seems like a bug in Terminal itself, and is probably in the rendering engine as it depends on the zoom level and at the same time seems to always appear at the exact same place for the exact same buffer to render.

@PhMajerus, I'm aware of the randomish U+FFFD issue. I have an outstanding task to try to debug why that's happening. I'll get to it eventually.

Interesting, is this related also to the way that F7 in cmd.exe renders itself with spaces/lines?

F7 is implemented in the COOKED_READ logic for CMD.exe and actually changes which symbols it uses for the outline based on codepage... or at least it used to.

from @vblazhkun, #2006

Environment
Windows build number: [Version 10.0.18362.239]
Windows Terminal version (if applicable): 0.2.1831.0

Any other software?

WSL Ubuntu: [1804.2019.521.0]

Steps to reproduce

Install Pragmata Pro Mono Liga 0.827 font (any other Unicode font should work as well).
Open the Windows Terminal with WSL Ubuntu tab.
Download and display the UTF-8 Demo Text.
$ curl https://www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-demo.txt

Expected behavior
No visible interline gap (the screenshot is from Cmder 180626 with WLS Ubuntu):

Actual behavior
Visible sparse interline gap:

Just checking in on this. This is Fira Code using GUI.CS

@shanselman I have just checked with the latest dev build, but the inter-line gap is still there.

Some terminals, including VTE and Konsole draw the U+2500..257F (or even up to 259F) glyphs manually, rather than taking them from the font. This prevents such problems (and also avoids ugly looking antialiasing). I haven't seen any monospace font that would just look perfect on its own.
Interestingly, independently from each other, both VTE and Konsole came up with the idea of defining the look of most of these codepoints as a 5x5 matrix (where how the given 5x5 grid is stretched to the character cell is not necessarily the same for thick lines vs. double lines).
VTE link1, link2
Update: Konsole recently changed not to use a 5x5 matrix but other means of drawing.

from @egmontkob

If you're about to implement manually drawing these glyphs, please also consider the U+1FB00 block to be added to Unicode 13 (further info via the VTE issue).

Let's note that this issue now tracks two technically independent stories:

A thin dark horizontal line between the background of every second or so text row, related to HiDPI;

Line/box drawing glyphs (their foregrounds) not perfectly aligning.

@egmontkob  And the possibility of the Terminal rendering the box and block drawing glyphs itself, so alignment will be perfect, no matter what Line Spacing or character cell widths and heights the current font is set to?

Yup, this is what I meant with my second bullet point :)

The same issue here. But I can fix it by setting "fontSize": 13. my resolution is 2560 X 1600, scale factor 150%.
when "fontSize": 12

after setting "fontSize": 13

Same issue here, HiDPI problem: 1.75 scaling factor / 4k resolution.

@egmontkob And the possibility of the Terminal rendering the box and block drawing glyphs itself, so alignment will be perfect, no matter what Line Spacing or character cell widths and heights the current font is set to?

I think the terminal should not render the glyphs itself but use the normal ones from the font that is configured otherwise it breaks with the general appearance of the configured font.
And generally asked, is there any progress on this matter? I try to imagine why it is so complicated after all. All other consoles known to me seem to have no trouble with this.

At the moment, there hasn't been any specific progress on this. It's triaged into the 1.0 milestone, so we plan on getting to it before we release a 1.0 version of the Terminal, but there's been like 4000 other things that have also been raised since this issue was filed, so we've been a little busy and just haven't gotten to this yet. Sorry about that!

Wov, I see there're already so much work is done...
Anyway guys you are doing something awesome. Scripts UI refreshment is something entire software world would be happy to explore!

Drawing the blocks correctly looks like it could be related to #900 ish

https://github.com/microsoft/terminal/issues/455#issuecomment-490249051

The gapping between the lines should be solved already by #5149

https://github.com/microsoft/terminal/issues/455#issuecomment-560280863
https://github.com/microsoft/terminal/issues/455#issuecomment-558489512
https://github.com/microsoft/terminal/issues/455#issuecomment-518455051

* Drawing the blocks correctly looks like it could be related to #900 ish
  * [#455 (comment)](https://github.com/microsoft/terminal/issues/455#issuecomment-490249051)

Nope, I believe it has to do with the vertical line padding/spacing. I'll dig.

I just installed 0.11 and, while improved (most blocks are rendering without spaces), the solid arrow characters in Cascadia Code PL 1911.210 (latest) still show a space:

Here's the good news, I've figured out a way to correct the box/line glyphs in the vertical direction:

Bad news, it's not perfect yet. The gaps are gone between the lines, but the diagonal shadings still don't line up. And I haven't actually figured out the right metrics here, this is just a proof of concept to see if I can find the knob WHERE I can adjust the heights.
Also, I haven't yet figured the little bit of horizontal gapping that occurs at certain resolutions and font sizes.

 E.g. if you want them to align perfectly, you do need to overflow to adjacent cells. Also figure out how the line should end (e.g. suddenly (rectangle shape) or with a half circle), which shape gives proper joining if they cross, or a / and \ meet each other, for all this to look "perfectly", whatever that means. Ideally without the drawing of a cell depending on other cells, as that would be pointless overengineering. There's no single perfect solution, but there are a couple of good enough ones.

Thanks, Egmont. I'll probably defer perfect diagonals until post-1.0 in a follow on issue.
Digging in further this morning, @DHowett-MSFT and I figured out that something isn't authored quite right with Cascadia Code that is contributing here. We're looking into that too.

 So now I'm going crazy in Excel finding something that I can use to adjust the baseline, the overall height, or whatnot either for the entire font or for the glyphs only in the U+2500-U+259F range (with the ascenderOffset flag inside the CustomTextLayout)

Another note: Anonymous Pro has line drawing chars but no box drawing chars, so that's a fallback and probably why it's looking extra weird.

@miniksa This is amazing compared to the previous rendering, great work!
The shading blocks are a very difficult case, and if you look again on my first screenshot of that test pattern using mlterm, you'll see they don't line up perfectly either.
The problem is the original EGA/VGA font (8x14 font in ROM on old graphic cards) had the pattern perfect for alignment, but even the CGA (8x8) was somewhat off for the dark shade block. If I remember correctly, the default font used in Win95 console needed an alternate of shaded blocks to fit properly side by side (as in, inverting the foreground and background colors and using the opposite shading block to make their joins merge correctly using an invisible checkerboard pattern).
So I wouldn't worry too much about it for v1, the result you got already seems on par with other mature terminal emulators.
I find the colors narrowing a bigger problem for ANSI-art and Text UI, especially the convertion of black background to transparent background when they were explicitly black instead of CSI 49m.
Can you include the following in your tests ?
echo -e "\e[30m\u2590\e[47;7m#\e[27;49m\u258C\e[m"
Should display silver # symbol over a black square even with acrylic blur background, but it is shown as ▐#▌ instead, the background of the # cell being converted to transparent instead of staying black like the two sides.

Can you include the following in your tests ?
echo -e "\e[30m\u2590\e[47;7m#\e[27;49m\u258C\e[m"

That might actually be #2661

@zadjii-msft
Yeah, your ColorType IsDefault flag should take care of it, but since we're getting close to v1 I'm trying to avoid something slipping through that gets noticed too late... Like #399 is still bothering me every single day in 20H1.
I just want to make sure you have a test case where the background color of a cell happens to be the exact same color as the default background color as seen by legacy conhost concepts, but needs to stay solid opaque, while CSI 49m gives acrylic transparency.
Really hope these VT stuff will all work for v1.
Thanks for all the work on fixing VT edge cases, It is looking great so far.

OK I spent all day fiddling around.
I have the math that matches the results from the states above.
Unfortunately I put the "with gap correction" to the left of "w/o gap correction"... but it goes as follows.
If there are 3 green boxes next to each other for that font... it's going to look fine in the Terminal.
If there are not, it will gap somewhere. (2nd and 3rd columns describe where it's gapping.)

The left most one is green if the designed height of the █ glyph is tall enough to fit the cell space we've given it after we've ceiling'd the baseline so it lands on a solid pixel and ceiling'd out the cell as a whole to an integer height.
(We integer-ify the baseline to ensure that the bottom of characters sits nice and crisply on a pixel. We integer-ify the cell height out to a perfect integer to enable scrolling offload.)
The middle column is green if the top of the █ glyph will touch or stretch outside the top of the cell after it's been drawn at the modified baseline.
The right most column is green if the bottom of the █ glyph will touch or stretch outside the bottom of the cell after it's been drawn at the modified baseline.
There were 4 triple-greens in the old algorithm (which aligned with the specific fonts in the test data). And there are 3 triple-greens in the baseline-adjust algorithm (also aligning with the specific fonts in the test data.)
I believe there's two steps to success here:

Given that boxes/lines don't specifically care about the rounded baseline for the solid pixel, translate the glyphs back upward to where they should have been. Match the centerpoint of the glyph to the centerpoint of the cell.
For glyphs that still wouldn't be big enough after translating them into the centerpoint of the cell, use a scale transform to fill the cell (or slightly overscan it by a few DU to guarantee it looks good).

Beyond that, I've written some code today that:

Adds an analysis step to CustomTextLayout that can identify runs of box/line drawing characters and segment them out into their own runs.
Tag all of these identified runs with a payload object that can direct custom rendering.
Added the custom rendering object to the runs so it is carried through.
Piped custom rendering objects the whole way down through CustomTextRenderer
Repurposed the unused "draw glyphs manually" method to become our box drawing method. Corrected it to draw the outlines and the fills. Set sample translate/scale transformations between geometry source and sinks. Tested this out manually and showed that it visually looks correct... as long as we have the right metrics.

My goal for Friday will be:

Take the math out of the Excel spreadsheet and code it into the analyzer so it can determine the translation and/or scale factors needed to adjust box glyphs so they look proper.
Expand the custom object to hold this data and carry it down to the renderer.
Update and verify this works against the custom text renderer by filling the geometry transform matrix.
Then refine by:
A. Don't do all this if they would already fit and look just fine. What a performance nightmare!
B. See if I can generalize the box/line drawing data into character classes that can be loaded/cached on font change instead of loaded in real time.
C. Ensure this still works for font fallback scenarios as some of these fonts implement SOME of the line/box glyphs and not others and we'd want the fallback glyphs to look decent while we're doing all this complex work anyway.

And the backup plan is:

Instead of actually determining scale/translate transform metrics for the geometries, instead implement a U+2500-U+259F-part switch case that just draws them with FillRectangle instead and ignores the glyphs in the fonts. I don't want to do this because I like the style/flavor that fonts can provide here. But it's a backup plan to all this math.

Either the main or backup plan solution should eliminate all gapping in both the X and Y dimensions for all of these glyphs no matter what font is used and no matter if that font implements all of these glyphs or not.

That’s a lot of work to work around font rendering issues.
It seems that having a lot of complicated mitigation code might make it more difficult for font designers to test their fonts, as they might be tinkering with their design to try to make something that looks good in a few test apps, and the different heuristics used by the code might appear random from the font designer point of view. I believe we need a switch to explicitly turn off all mitigation code paths so they can test their fonts against the more predictable optimized renderer and try to build a font that doesn’t rely on fallback paths. That would be both in settings.json and at the renderer XAML control level for other apps hosting the Terminal.
About using the font glyphs versus a fixed set of block elements and line drawing glyphs, I agree using the font to allow fonts to have their own identities.
For that idea, I have a suggestion that I believe would provide added value to the Windows Terminal in general and provide a fallback mechanism when needed.
Most fonts won’t provide all the glyphs that the Terminal can display, we’ll have Latin font, CJK font, Arabic font,… at this time we can select a single font, and the Terminal uses some other fallback font for missing characters. But how about letting the user specify a set of fonts by Unicode blocks? Just like MS Word allows the user to select a Latin font and another Asian text font, the Terminal could allow the user to select a main font as well as explicit alternate fonts for the CJK block, the Arabic block, the Math symbols block, and of course, the Block elements and Box drawing blocks. This would provide better control over the look of mixed scripts in the Terminal, and allow an optimized font that doesn’t need the performance hit of the fallbacks to be used for blocks and lines if the user’s favorite font doesn’t provide good ones.
This also has the added benefit of making it possible to separate things like the Powerline and Nerd glyphs into separate fonts. Instead of merging them into special versions of Cascadia Code, and missing them when using a font that didn’t merge these glyphs, the Terminal could simply use Powerline-only and Nerd-only fonts for those Unicode blocks according to the same user-defined “fonts set”.

Thanks for your response, @PhMajerus. I still believe it's up to me to fix it in a complex way because I'm the one breaking it in the first place for many of these fonts. I'm the one adding additional vertical line padding in some cases and throwing off the prescribed dimensions of the fonts. I do it in order to use the scrolling mechanisms of Present1, maintain the integer-pixel cell dimensions that most of our code and client applications rely on, and to try to preserve the crispness of the glyphs through these adjustments. I know the "just don't mess with it" solution would probably resolve this on its own, but then I'd have to give back all the performance I got from differential drawing and Present1 and I'd have a new host of problems with hit testing and line wrapping and API-mapping.
I do believe in adding a fallback mechanism including being able to specify which ranges it applies to, but that's definitely going to come after 1.0 with the complexity of the settings configuration required. Such a thing will also still require me to adjust the analysis steps to further parse and identify the ranges to apply each font to, so that would have to be done for either path.
Finally... I think adding a setting to turn it off might be advisable and possible.

 Generating draft PR now. Here's an Anonymous Pro sample. Not perfect, but pretty darned good.

OK, I've solved most of it. There's still a few circumstances with a minor vertical banding between blocks in the horizontal direction, but I've filed #5816 as a follow on for that because the PR at #5743 adds so much goodness for a majority of cases.


When window is scrolling swipe event didn't fires

Hi, when I scroll window and while it is scrolling swipe events from swipeable element didn't fires.

Hey @DrBoria was curious if you could possibly give me a little more to go off of? maybe a codesandbox.io/react-swipeable example setup with an explanation of how to replicate the issue?

closing due to lack of more info or example. please re-open if needed 😃


[zh] Translate blog/_posts/2021-12-22-kubernetes-in-kubernetes-and-pxe-bootable-server-farm.md into Chinese

Hi, I'm interested to translate the "2021-12-22-kubernetes-in-kubernetes-and-pxe-bootable-server-farm.md" page.
This is a Feature Request:
translate en/blog/_posts/2021-12-22-kubernetes-in-kubernetes-and-pxe-bootable-server-farm.md into Chinese.
What would you like to be added:
zh/blog/_posts/2021-12-22-kubernetes-in-kubernetes-and-pxe-bootable-server-farm.md
/language zh
/kind feature

/triage accepted
/help

/assign


MINOR: Update collaborators list

I did the following:

git shortlog --email --numbered --summary --since=2023-07-24 > /tmp/contributors.txt

Then I excluded all committers from the list.
Cut the list to the top 10 and for each member I found their GitHub user name. List is in order of commits.
Note I used the --email flag as stated in https://issues.apache.org/jira/browse/KAFKA-14995
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

Nice PR one problem with the used command is that any one that has same email but different name attached to the commit don't count as one person.
I tried this
git shortlog --email --numbered --summary --since=2023-07-24 | awk '
{
  # Extract email (the part between < and >)
  match($0, /<[^>]*>/);
  email = substr($0, RSTART+1, RLENGTH-2);

  # Add to the associative array
  commits[email] += $1;
}

END {
  # Print the result
  for (email in commits) {
    print commits[email], email;
  }
}
' | sort -nr

and got slightly different order.

I can give it a try and hope we don't have two people with the same name 😃

maybe we should ping those active contributors to take a look at this PR. Otherwise, they could never know that permission.

I can give it a try and hope we don't have two people with the same name
I tried the original command you shared and definitely my email is shown twice once

    27 Omnia Ibrahim<EMAIL_ADDRESS>    4	 Omnia G.H Ibrahim<EMAIL_ADDRESS>
This make my total 31 which should be right after 34	Nikolay<EMAIL_ADDRESS>

I think the tricky part would be same person two emails :D but I can't find such one in the original git shortlog --email --numbered --summary --since=2023-07-24 command yet. Hopefully we don't have this case

Same with others

another case:
20<EMAIL_ADDRESS>12<EMAIL_ADDRESS>
Those active contributors have many alias :)

How about using Github contributors page?
https://github.com/apache/kafka/graphs/contributors?from=2023-07-24&to=2024-07-24&type=c

20<EMAIL_ADDRESS>12<EMAIL_ADDRESS>
Yes this is me.
This happened because I didn't know that GitHub has private email protection before... 😥

Same with others

another case:
20<EMAIL_ADDRESS>12<EMAIL_ADDRESS>
Those active contributors have many alias :)

yup this is the tricky one I mentioned above same person different emails :D

How about using Github contributors page?

FrankYang0529
kamalcph
apoorvmittal10
lianetm
brandboat
kirktrue
nizhikov
OmniaGM
dongnuo123
frankvicky

maybe we should ping those active contributors to take a look at this PR. Otherwise, they could never know that permission.

Yes @chia7712 , I was planning on doing this once we had the changes merged to make sure the pinged people were the ones that made it to the list for sure.

I updated KAFKA-14995 to reflect this way to obtain the contributors that seems more fair than the one with emails or names.

I'll merge this PR, as the pipelines don't check this file and we don't need to waste CPU cycles on this.
Mentioning the newly added collaborators here as an FYI - @FrankYang0529 @apoorvmittal10 @lianetm @brandboat @kirktrue @nizhikov @OmniaGM @dongnuo123 @frankvicky . Welcome all! Let us know if you have any questions!
You can check out the Collaborators section on https://kafka.apache.org/contributing, or this dev mailing list thread for more details.


Valid Discover card displayed as invalid

Using valid Discover card test number: 6229 2567 3724 8989
Caishen UI displays as invalid. Simply asking Caishen to do a Luhn check on it, passes though 🤔

@DannyVancura - @quentin41500 is using the Android Braintree SDK which verifies this as a Discover card. as of typing in the first two numbers 62, Braintree says its China UnionPay; however, as of typing the first 3 numbers 622, it picks it up as Discover

looking up Discover BIN ranges, indeed the provided number would be a Discover card - i.e. 622925
Discover Card
6011
622126-622925
644-649
65
https://blog.groundlabs.com/cardholder-data-discovery-anatomy-of-a-credit-card-bin-ranges-luhn-checks


Improved performance and some minor issues

Checklist before merging

[X] I added an appropriate entry to CHANGELOG.md

Closes #377
Closes #359
Closes #369
Closes #351

Here's the loading time for the projects page for a user with 33 projects (that is, Joel's user from a previous backup) - read from firefox dev tools.
main: ~350 ms
this PR: 25 ms
The trade-off is that, as of this PR, each time we open the "info" modal for one project there are some more 20-60 ms for loading the dataset list. This looks perfectly fine to me: we are very fast in showing the project list, and may be slightly less responsive if the user wants to see the "info" for several projects.

I guess https://github.com/fractal-analytics-platform/fractal-web/pull/379/commits/b3123ec70ef51a6149f110a5747250fc199b0f53 also improves the tasks-page performance, but I'm not sure of where I should look.

main: ~350 ms
this PR: 25 ms

It turns out that this is probably true, even if it's surprising.
Starting with
alias tcurl='curl -o /dev/null -s -w '\''Total: %{time_total}s\n'\'''

we can profile API calls directly from the command-line (after a first call that is necessary to know the appropriate authentication info).
The relevant command (thanks @zonia3000) looks like
tcurl 'http://localhost:8000/auth/current-user/' -H 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:120.0) Gecko/20100101 Firefox/120.0' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8' -H 'Accept-Language: en,en-US;q=0.7,it;q=0.3' -H 'Accept-Encoding: gzip, deflate, br' -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Cookie: fief_session=-2YdettvtewOihLRmD_2kyK0r6Jzxeu01sWZEorxX10; fief_login_hint=REDACTED%40REDACTED.REDACTED; ph_phc_OcwKamvVT5QCgASAkv6Sk1vnvBCovXCDKJPukNApioW_posthog=%7B%22distinct_id%22%3A%221a660789-97cd-4ed9-9760-4423348a3958%22%2C%22%24sesid%22%3A%5B1702902836895%2C%22018c7ce6-dd7e-7879-acb6-279abeb1d275%22%2C1702902553982%5D%7D; fief_csrftoken=CoW7-8JVLamokCoS8DmWsA8VDjKiZJ0sk8gh7s1lM1Q; fief_admin_session=krkiZmQlKk0mspXIPOZh6MB6Jml-JlrU-bDoJDqPngI; fief_session_data=eyJjcmVhdGVfd29ya3NwYWNlIjoge319.ZYA8NA.zN7i9cPr8bjcJTc_0ROCv5pYm0g; fastapiusersauth=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiI2IiwiYXVkIjpbImZhc3RhcGktdXNlcnM6YXV0aCJdLCJleHAiOjE3MDMwNjk0NjR9.nCmqlCISQeazM7GvhfHEYk5CF-vbQa5UVvMOE0PxfpU' -H 'Upgrade-Insecure-Requests: 1' -H 'Sec-Fetch-Dest: document' -H 'Sec-Fetch-Mode: navigate' -H 'Sec-Fetch-Site: same-origin' -H 'If-None-Match: "16922va"'

and it can be iterated a few times to get some stats.
Here are the results
http://localhost:8000/api/v1/project/

Total: 0.016885s
Total: 0.020372s
Total: 0.020306s
Total: 0.018525s
Total: 0.019509s
Total: 0.019746s
Total: 0.019822s
Total: 0.020035s
Total: 0.021317s
Total: 0.014740s

http://localhost:8000/api/v1/dataset/

Total: 0.266370s
Total: 0.253966s
Total: 0.256037s
Total: 0.256888s
Total: 0.212706s
Total: 0.295564s
Total: 0.261209s
Total: 0.257948s
Total: 0.249718s
Total: 0.213506s

This explains why removing the /api/v1/dataset/ makes such a difference.


Sorting dates properly

This is not a bug but a feature request or help.
Currently, if we sort a column of dates, it sorts them in the alphabetical order. For example, there are three dates: January 28, 2019, April 1, 2019, and May 15, 2019, it orders them as "April 1, 2019", "January 28, 2019", "May 15, 2019". Is there any way to have alternate values for any data so that we can sort using those alternate values but display the display values?

I found a workaround. I am using a template for the datatable. So, in the columns options, I am using the raw datetime string. But in the template, I am using formatted dates to display in the rows.
I believe, alternatively, we can use representedAs if we are not using the template.