TaskMergeScheduler: Salvage or deprecate?

TaskMergeScheduler was originally created to sidestep issues with limited multithreading support on .NET Standard 1.x. However, since we have dropped support for .NET Standard 1.x, it is no longer technically a requirement. While having a merge scheduler based on TPL sounds like a brilliant idea, in its current incarnation it falls considerably short of where it needs to be to be included in the release.

It is several orders of magnitude slower than ConcurrentMergeScheduler.
The Lucene.Net.Index.TestTaskMergeScheduler::TestSubclassTaskMergeScheduler() test occasionally fails (see #269).

TaskMergeScheduler has been removed from being randomly injected into tests in the test framework because of the negative performance impact it has on test runs.
I have fixed a few bugs with it, including the fact it was failing to re-throw a background merge failure, a requirement for IndexWriter to retry.
The main performance issue stems from the fact that it uses ToArray() to copy its _mergeThreads collection when looping. An attempt was made to replace the collection with ConcurrentHashSet<T> and remove ToArray() on each loop, but at least one of the tests starts failing with that change.
So, my question is should we attempt to salvage it, or deprecate it? By deprecate, I mean it will be marked obsolete in the next beta and deleted from the project in the first release candidate.
If someone wants to take a stab at either fixing it or redesigning it, it is fair game. This could be an opportunity to add a more performant alternative to ConcurrentMergeScheduler to the project. If it performs adequately and provides the correct behavior, we will consider making it the default.
using J2N.Collections.Generic.Extensions;
using Lucene.Net.Support.Threading;
using Lucene.Net.Util;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Runtime.CompilerServices;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using Directory = Lucene.Net.Store.Directory;

namespace Lucene.Net.Index
{
    /*
     * Licensed to the Apache Software Foundation (ASF) under one or more
     * contributor license agreements.  See the NOTICE file distributed with
     * this work for additional information regarding copyright ownership.
     * The ASF licenses this file to You under the Apache License, Version 2.0
     * (the "License"); you may not use this file except in compliance with
     * the License.  You may obtain a copy of the License at
     *
     *     http://www.apache.org/licenses/LICENSE-2.0
     *
     * Unless required by applicable law or agreed to in writing, software
     * distributed under the License is distributed on an "AS IS" BASIS,
     * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
     * See the License for the specific language governing permissions and
     * limitations under the License.
     */

    ///
    /// A <see cref="MergeScheduler"/> that runs each merge using
    /// <see cref="Task"/>s on the default <see cref="TaskScheduler"/>.
    ///
    /// <para>If more than <see cref="MaxMergeCount"/> merges are
    /// requested then this class will forcefully throttle the
    /// incoming threads by pausing until one more more merges
    /// complete.</para>
    ///
    /// LUCENENET specific
    ///
    public class TaskMergeScheduler : MergeScheduler, IConcurrentMergeScheduler
    {
        public const string COMPONENT_NAME = "CMS";

        private readonly TaskScheduler _taskScheduler = TaskScheduler.Default;
        private readonly ReaderWriterLockSlim _lock = new ReaderWriterLockSlim();
        private readonly ManualResetEventSlim _manualResetEvent = new ManualResetEventSlim();
        ///
        /// List of currently active <see cref="MergeThread"/>s.
        private readonly IList<MergeThread> _mergeThreads = new List<MergeThread>();

        ///
        /// How many <see cref="MergeThread"/>s have kicked off (this is use
        /// to name them).
        ///
        private int _mergeThreadCount;

        ///
        /// <see cref="Directory"/> that holds the index.
        private Directory _directory;

        ///
        /// <see cref="IndexWriter"/> that owns this instance.
        ///
        private IndexWriter _writer;

        ///
        /// Sole constructor, with all settings set to default
        /// values.
        ///
        public TaskMergeScheduler() : base()
        {
            MaxThreadCount = _taskScheduler.MaximumConcurrencyLevel;
            MaxMergeCount = _taskScheduler.MaximumConcurrencyLevel;
        }

        ///
        /// Sets the maximum number of merge threads and simultaneous merges allowed.
        ///
        /// <param name="maxMergeCount"> The max # simultaneous merges that are allowed.
        ///       If a merge is necessary yet we already have this many
        ///       threads running, the incoming thread (that is calling
        ///       add/updateDocument) will block until a merge thread
        ///       has completed.  Note that we will only run the
        ///       smallest <paramref name="maxThreadCount"/> merges at a time. </param>
        /// <param name="maxThreadCount"> The max # simultaneous merge threads that should
        ///       be running at once.  This must be <= <paramref name="maxMergeCount"/> </param>
        public void SetMaxMergesAndThreads(int maxMergeCount, int maxThreadCount)
        {
            // This is handled by TaskScheduler.Default.MaximumConcurrencyLevel
        }

        ///
        /// Max number of merge threads allowed to be running at
        /// once.  When there are more merges then this, we
        /// forcefully pause the larger ones, letting the smaller
        /// ones run, up until <see cref="MaxMergeCount"/> merges at which point
        /// we forcefully pause incoming threads (that presumably
        /// are the ones causing so much merging).
        ///
        /// <seealso cref="SetMaxMergesAndThreads(int, int)"/>
        public int MaxThreadCount { get; private set; }

        ///
        /// Max number of merges we accept before forcefully
        /// throttling the incoming threads
        ///
        public int MaxMergeCount { get; private set; }

        ///
        /// Return the priority that merge threads run at. This is always the same.
        ///
        public int MergeThreadPriority
        {
            get
            {
#if !FEATURE_THREAD_PRIORITY
                return 2;
#else
                return (int)ThreadPriority.Normal;
#endif
            }
        }

        ///
        /// This method has no effect in <see cref="TaskMergeScheduler"/> because the
        /// <see cref="MergeThreadPriority"/> returns a constant value.
        ///
        public void SetMergeThreadPriority(int priority)
        {
        }

        ///
        /// Called whenever the running merges have changed, to pause & unpause
        /// threads. This method sorts the merge threads by their merge size in
        /// descending order and then pauses/unpauses threads from first to last --
        /// that way, smaller merges are guaranteed to run before larger ones.
        ///
        private void UpdateMergeThreads()
        {
            foreach (var merge in _mergeThreads.ToArray())
            {
                // Prune any dead threads
                if (!merge.IsAlive)
                {
                    _mergeThreads.Remove(merge);
                    merge.Dispose();
                }
            }
        }

        ///
        /// Returns <c>true</c> if verbosing is enabled. This method is usually used in
        /// conjunction with <see cref="Message(string)"/>, like that:
        ///
        ///
        /// if (Verbose) {
        ///     Message("your message");
        /// }
        ///
        ///
        protected bool Verbose => _writer != null && _writer.infoStream.IsEnabled(COMPONENT_NAME);

        ///
        /// Outputs the given message - this method assumes <see cref="Verbose"/> was
        /// called and returned <c>true</c>.
        ///
        protected virtual void Message(string message)
        {
            _writer.infoStream.Message(COMPONENT_NAME, message);
        }

        protected override void Dispose(bool disposing)
        {
            Sync();
            _manualResetEvent.Dispose();
        }

        ///
        /// Wait for any running merge threads to finish.
        /// This call is not interruptible as used by <see cref="MergeScheduler.Dispose()"/>.
        ///
        public virtual void Sync()
        {
            foreach (var merge in _mergeThreads.ToArray())
            {
                if (merge == null || !merge.IsAlive)
                {
                    continue;
                }

                try
                {
                    merge.Wait();
                }
                catch (OperationCanceledException)
                {
                    // expected when we cancel.
                }
                catch (AggregateException ae)
                {
                    ae.Handle(ex =>
                    {
                        if (!(ex is OperationCanceledException))
                        {
                            HandleMergeException(ex);
                            return true;
                        }

                        return false;
                    });
                }
            }
        }

        ///
        /// Returns the number of merge threads that are alive. Note that this number
        /// is <= <see cref="_mergeThreads"/> size.
        ///
        private int MergeThreadCount
        {
            get { return _mergeThreads.Count(x => x.IsAlive && x.CurrentMerge != null); }
        }

        [MethodImpl(MethodImplOptions.NoInlining)]
        public override void Merge(IndexWriter writer, MergeTrigger trigger, bool newMergesFound)
        {
            using (_lock.Write())
            {
                _writer = writer;
                _directory = writer.Directory;

                if (Verbose)
                {
                    Message("now merge");
                    Message("  index: " + writer.SegString());
                }

                // First, quickly run through the newly proposed merges
                // and add any orthogonal merges (ie a merge not
                // involving segments already pending to be merged) to
                // the queue.  If we are way behind on merging, many of
                // these newly proposed merges will likely already be
                // registered.

                // Iterate, pulling from the IndexWriter's queue of
                // pending merges, until it's empty:
                while (true)
                {
                    long startStallTime = 0;
                    while (writer.HasPendingMerges() && MergeThreadCount >= MaxMergeCount)
                    {
                        // this means merging has fallen too far behind: we
                        // have already created maxMergeCount threads, and
                        // now there's at least one more merge pending.
                        // Note that only maxThreadCount of
                        // those created merge threads will actually be
                        // running; the rest will be paused (see
                        // updateMergeThreads).  We stall this producer
                        // thread to prevent creation of new segments,
                        // until merging has caught up:
                        startStallTime = Environment.TickCount;
                        if (Verbose)
                        {
                            Message("    too many merges; stalling...");
                        }

                        _manualResetEvent.Reset();
                        _manualResetEvent.Wait();
                    }

                    if (Verbose)
                    {
                        if (startStallTime != 0)
                        {
                            Message("  stalled for " + (Environment.TickCount - startStallTime) + " msec");
                        }
                    }

                    MergePolicy.OneMerge merge = writer.NextMerge();
                    if (merge == null)
                    {
                        if (Verbose)
                        {
                            Message("  no more merges pending; now return");
                        }
                        return;
                    }

                    bool success = false;
                    try
                    {
                        if (Verbose)
                        {
                            Message("  consider merge " + writer.SegString(merge.Segments));
                        }

                        // OK to spawn a new merge thread to handle this
                        // merge:
                        var merger = CreateTask(writer, merge);

                        merger.MergeThreadCompleted += OnMergeThreadCompleted;

                        _mergeThreads.Add(merger);

                        if (Verbose)
                        {
                            Message("    launch new thread [" + merger.Name + "]");
                        }

                        merger.Start(_taskScheduler);

                        // Must call this after starting the thread else
                        // the new thread is removed from mergeThreads
                        // (since it's not alive yet):
                        UpdateMergeThreads();

                        success = true;
                    }
                    finally
                    {
                        if (!success)
                        {
                            writer.MergeFinish(merge);
                        }
                    }
                }
            }
        }

        ///
        /// Does the actual merge, by calling <see cref="IndexWriter.Merge(MergePolicy.OneMerge)"/>
        [MethodImpl(MethodImplOptions.NoInlining)]
        protected virtual void DoMerge(MergePolicy.OneMerge merge)
        {
            _writer.Merge(merge);
        }

        private void OnMergeThreadCompleted(object sender, EventArgs e)
        {
            var mergeThread = sender as MergeThread;

            if (mergeThread == null)
            {
                return;
            }

            mergeThread.MergeThreadCompleted -= OnMergeThreadCompleted;

            using (_lock.Write())
            {
                UpdateMergeThreads();
            }
        }

        ///
        /// Create and return a new <see cref="MergeThread"/>
        private MergeThread CreateTask(IndexWriter writer, MergePolicy.OneMerge merge)
        {
            var count = Interlocked.Increment(ref _mergeThreadCount);
            var name = string.Format("Lucene Merge Task #{0}", count);

            return new MergeThread(name, writer, merge, writer.infoStream, Verbose, _manualResetEvent, HandleMergeException, DoMerge);
        }

        ///
        /// Called when an exception is hit in a background merge
        /// thread
        ///
        protected virtual void HandleMergeException(Exception exc)
        {
            // suppressExceptions is normally only set during testing
            if (suppressExceptions)
            {
                return;
            }

//#if FEATURE_THREAD_INTERRUPT
//            try
//            {
//#endif
                // When an exception is hit during merge, IndexWriter
                // removes any partial files and then allows another
                // merge to run.  If whatever caused the error is not
                // transient then the exception will keep happening,
                // so, we sleep here to avoid saturating CPU in such
                // cases:
                Thread.Sleep(250);
//#if FEATURE_THREAD_INTERRUPT // LUCENENET NOTE: Senseless to catch and rethrow the same exception type
//            }
//            catch (ThreadInterruptedException ie)
//            {
//                throw new ThreadInterruptedException("Thread Interrupted Exception", ie);
//            }
//#endif
            throw new MergePolicy.MergeException(exc, _directory);
        }

        private bool suppressExceptions;

        ///
        /// Used for testing
        public virtual void SetSuppressExceptions()
        {
            suppressExceptions = true;
        }

        ///
        /// Used for testing
        public virtual void ClearSuppressExceptions()
        {
            suppressExceptions = false;
        }

        public override string ToString()
        {
            StringBuilder sb = new StringBuilder(this.GetType().Name + ": ");
            sb.AppendFormat("maxThreadCount={0}, ", MaxThreadCount);
            sb.AppendFormat("maxMergeCount={0}", MaxMergeCount);
            return sb.ToString();
        }

        public override object Clone()
        {
            TaskMergeScheduler clone = (TaskMergeScheduler)base.Clone();
            clone._writer = null;
            clone._directory = null;
            clone._mergeThreads.Clear();
            return clone;
        }

        ///
        /// Runs a merge thread, which may run one or more merges
        /// in sequence.
        ///
        internal class MergeThread : IDisposable
        {
            public event EventHandler MergeThreadCompleted;

            private readonly CancellationTokenSource _cancellationTokenSource;
            private readonly ReaderWriterLockSlim _lock = new ReaderWriterLockSlim();
            private readonly ManualResetEventSlim _resetEvent;
            private readonly Action<Exception> _exceptionHandler;
            private readonly Action<MergePolicy.OneMerge> _doMerge;
            private readonly InfoStream _logger;
            private readonly IndexWriter _writer;
            private readonly MergePolicy.OneMerge _startingMerge;
            private readonly bool _isLoggingEnabled;

            private Task _task;
            private MergePolicy.OneMerge _runningMerge;
            private volatile bool _isDisposed = false;
            private volatile bool _isDone;

            ///
            /// Sole constructor.
            public MergeThread(string name, IndexWriter writer, MergePolicy.OneMerge startMerge,
                InfoStream logger, bool isLoggingEnabled,
                ManualResetEventSlim resetEvent, Action<Exception> exceptionHandler, Action<MergePolicy.OneMerge> doMerge)
            {
                Name = name;
                _cancellationTokenSource = new CancellationTokenSource();
                _writer = writer;
                _startingMerge = startMerge;
                _logger = logger;
                _isLoggingEnabled = isLoggingEnabled;
                _resetEvent = resetEvent;
                _exceptionHandler = exceptionHandler;
                _doMerge = doMerge;
            }

            public string Name { get; private set; }

            public Task Instance
            {
                get
                {
                    using (_lock.Read())
                    {
                        return _task;
                    }
                }
            }

            ///
            /// Record the currently running merge.
            public virtual MergePolicy.OneMerge RunningMerge
            {
                get
                {
                    using (_lock.Read())
                    {
                        return _runningMerge;
                    }
                }
                set => Interlocked.Exchange(ref _runningMerge, value);
            }

            ///
            /// Return the current merge, or <c>null</c> if this
            /// <see cref="MergeThread"/> is done.
            ///
            public virtual MergePolicy.OneMerge CurrentMerge
            {
                get
                {
                    using (_lock.Read())
                    {
                        if (_isDone)
                        {
                            return null;
                        }

                        return _runningMerge ?? _startingMerge;
                    }
                }
            }

            public bool IsAlive
            {
                get
                {
                    if (_isDisposed || _isDone)
                    {
                        return false;
                    }

                    using (_lock.Read())
                    {
                        return _task != null
                            && (_task.Status != TaskStatus.Canceled
                            || _task.Status != TaskStatus.Faulted
                            || _task.Status != TaskStatus.RanToCompletion);
                    }
                }
            }

            public void Start(TaskScheduler taskScheduler)
            {
                using (_lock.Write())
                {
                    if (_task == null && !_cancellationTokenSource.IsCancellationRequested)
                    {
                        _task = Task.Factory.StartNew(() => Run(_cancellationTokenSource.Token), _cancellationTokenSource.Token, TaskCreationOptions.None, taskScheduler);
                    }
                }
            }

            public void Wait()
            {
                if (!IsAlive)
                {
                    return;
                }

                _task.Wait(_cancellationTokenSource.Token);
            }

            public void Cancel()
            {
                if (!IsAlive)
                {
                    return;
                }

                using (_lock.Write())
                {
                    if (!_cancellationTokenSource.IsCancellationRequested)
                    {
                        _cancellationTokenSource.Cancel();
                    }
                }
            }

            private void Run(CancellationToken cancellationToken)
            {
                // First time through the while loop we do the merge
                // that we were started with:
                MergePolicy.OneMerge merge = _startingMerge;

                try
                {
                    if (_isLoggingEnabled)
                    {
                        _logger.Message(COMPONENT_NAME, "  merge thread: start");
                    }

                    while (true && !cancellationToken.IsCancellationRequested)
                    {
                        RunningMerge = merge;
                        // LUCENENET NOTE: We MUST call DoMerge(merge) instead of
                        // _writer.Merge(merge) because the tests specifically look
                        // for the method name DoMerge in the stack trace.
                        _doMerge(merge);

                        // Subsequent times through the loop we do any new
                        // merge that writer says is necessary:
                        merge = _writer.NextMerge();

                        // Notify here in case any threads were stalled;
                        // they will notice that the pending merge has
                        // been pulled and possibly resume:
                        _resetEvent.Set();

                        if (merge != null)
                        {
                            if (_isLoggingEnabled)
                            {
                                _logger.Message(COMPONENT_NAME, "  merge thread: do another merge " + _writer.SegString(merge.Segments));
                            }
                        }
                        else
                        {
                            break;
                        }
                    }

                    if (_isLoggingEnabled)
                    {
                        _logger.Message(COMPONENT_NAME, "  merge thread: done");
                    }
                }
                catch (Exception exc)
                {
                    // Ignore the exception if it was due to abort:
                    if (!(exc is MergePolicy.MergeAbortedException))
                    {
                        //System.out.println(Thread.currentThread().getName() + ": CMS: exc");
                        //exc.printStackTrace(System.out)
                        _exceptionHandler(exc);
                    }
                }
                finally
                {
                    _isDone = true;

                    if (MergeThreadCompleted != null)
                    {
                        MergeThreadCompleted(this, EventArgs.Empty);
                    }
                }
            }

            public void Dispose()
            {
                if (_isDisposed)
                {
                    return;
                }

                _isDisposed = true;
                _lock.Dispose();
                _cancellationTokenSource.Dispose();
            }

            public override string ToString()
            {
                return _task == null
                    ? string.Format("Task[{0}], Task has not been started yet.", Name)
                    : string.Format("Task[{0}], Id[{1}], Status[{2}]", Name, _task.Id, _task.Status);
            }

            public override bool Equals(object obj)
            {
                var compared = obj as MergeThread;

                if (compared == null
                    || (Instance == null && compared.Instance != null)
                    || (Instance != null && compared.Instance == null))
                {
                    return false;
                }

                return Instance.Id == compared.Instance.Id;
            }

            public override int GetHashCode()
            {
                return Instance == null
                    ? base.GetHashCode()
                    : Instance.GetHashCode();
            }
        }
    }
}

If there's already a faster and more stable implementation with ConcurrentMergeScheduler, is there any reason to keep TaskMergeScheduler?  Or are you saying that there isn't currently a replacement for TaskMergeScheduler?

If there's already a faster and more stable implementation with ConcurrentMergeScheduler, is there any reason to keep TaskMergeScheduler? Or are you saying that there isn't currently a replacement for TaskMergeScheduler?

ConcurrentMergeScheduler is a direct port from Lucene.
TaskMergeScheduler was only required because ConcurrentMergeScheduler didn't compile on .NET Standard 1.x.

The only reason to consider keeping TaskMergeScheduler is the amount of time that has been invested in it already. That said, I don't plan to invest any more time on it myself, so if we don't have a party willing to make it production ready it will definitely be   excluded from the final release. IMO, there is no reason to hold up the release for a duplicate of something we already have.
But if someone were willing to step up and make TaskMergeScheduler reliable and more performant than ConcurrentMergeScheduler, it would not only be worth keeping but we could potentially have a new default setting.

Sounds like a challenge! I'll tweet it out and see if there's any takers :)

Given that #381 doesn't seem to solve our issue and we have had no other takers, it looks like it is time to deprecate TaskMergeScheduler.
[Obsolete("Use ConcurrentMergeScheduler instead. This class will be removed in 4.8.0 release candidate."), System.ComponentModel.EditorBrowsable(System.ComponentModel.EditorBrowsableState.Never)]
public class TaskMergeScheduler : MergeScheduler, IConcurrentMergeScheduler

As part of the deprecation, we need to tear out the LuceneTestCase.ConcurrentMergeSchedulerFactories and revert all of the tests that depend on it back to their original Java Lucene implementation. Other references to TaskMergeScheduler should also be removed from tests.
We can also remove the tests for TaskMergeScheduler altogether, which will improve testing run times.
Although it wasn't part of Lucene, we can leave the IConcurrentMergeScheduler interface in Lucene.NET as it might be in use by users already.


KeyError 'Token'

I'm getting
[08/02/2016 07:31:32] Starting new HTTP connection (1): mattermost.domain.com
[08/02/2016 07:31:32] Starting new HTTPS connection (1): mattermost.domain.com
Traceback (most recent call last):
  File "/usr/local/bin/matterbot", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/mattermost_bot/cli.py", line 17, in main
    b = bot.Bot()
  File "/usr/local/lib/python2.7/dist-packages/mattermost_bot/bot.py", line 27, in __init__
    settings.SSL_VERIFY
  File "/usr/local/lib/python2.7/dist-packages/mattermost_bot/mattermost.py", line 127, in __init__
    self.login(team, email, password)
  File "/usr/local/lib/python2.7/dist-packages/mattermost_bot/mattermost.py", line 131, in login
    self.user = self.api.login(team, email, password)
  File "/usr/local/lib/python2.7/dist-packages/mattermost_bot/mattermost.py", line 45, in login
    self.token = p.headers["Token"]
  File "/usr/lib/python2.7/dist-packages/requests/structures.py", line 54, in __getitem__
    return self._store[key.lower()][1]
KeyError: 'token'

My settings are:
BOT_URL = 'http://mattermost.domain.com/api/v3'
BOT_TEAM = 'dev'

I have tried the suggestions in #2 and #16, and tried other version values, but this has not helped.
I'm running mattermost v3.2 on Debian stable

Fixed by changing the BOT_URL to
BOT_URL = 'https://mattermost.domain.com/api/v3'

I just want to add that I had the same issue and was at a loss since none of the proposed solutions were working, I then printed out p.text and found that the error was that the bot user's email was not verified. I had enabled email verification for everyone after the bot went live and hence it was getting blocked. Just putting this in here in case it helps someone.
{"id":"api.user.login.not_verified.app_error","message":"Login failed because email address has not been verified","detailed_error":"","request_id":"xxxxxxxxxxxx","status_code":401}


Check the git commit has any sensitive data or not.

Check the git commit has any sensitive data or not.

The alpha implementation is done.


Push v2.0.1 to CocoaPods Specs repo

Correct me if I'm wrong, but it seems the latest versions (v2.0.0 and v2.0.1) have not been pushed to the CocoaPods Specs repo.
The pod page shows v1.5.3, as does the readme of this repo.

+1


There is a selection filter so that only A is selectable.
First the user selects row A => row A is selected.
Then the user attempts to select row B => Row B is not selected.
Expected behavior: row A should remain selected (?)
Current behavior: the selection is cleared.
https://github.com/FlowingCode/GridHelpers/blob/ff0409803f40932da24a2bfdbf81a965e5effad9/src/test/java/com/flowingcode/vaadin/addons/gridhelpers/it/SelectionFilterIT.java#L71-L80

IMHO it should work as expected: row A should remain selected


Handle Out of memory errors

Describe the solution you'd like
Handle gracefully Out of memory errors
Additional context
v1.4.0 implementation:

https://github.com/MobilityData/gtfs-validator/blob/v1.4.0/usecase/src/main/java/org/mobilitydata/gtfsvalidator/usecase/HandleOutOfMemoryError.java
tests: https://github.com/MobilityData/gtfs-validator/blob/v1.4.0/usecase/src/test/java/org/mobilitydata/gtfsvalidator/usecase/HandleOutOfMemoryErrorTest.java

Closed by https://github.com/MobilityData/gtfs-validator/pull/596

Closed by https://github.com/MobilityData/gtfs-validator/pull/596


can't download

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', 'https://www.youtube.com/watch?v=y_l2Ug64WY0']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.10.6 (CPython) - Linux-5.15.0-60-generic-x86_64-with-glibc2.35
[debug] exe versions: ffmpeg 4.4.2, ffprobe 4.4.2, rtmpdump 2.4
[debug] Proxy map: {}
[youtube] y_l2Ug64WY0: Downloading webpage
ERROR: Unable to extract uploader id; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/youtube_dl/YoutubeDL.py", line 815, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/youtube_dl/YoutubeDL.py", line 836, in __extract_info
    ie_result = ie.extract(url)
  File "/usr/lib/python3/dist-packages/youtube_dl/extractor/common.py", line 534, in extract
    ie_result = self._real_extract(url)
  File "/usr/lib/python3/dist-packages/youtube_dl/extractor/youtube.py", line 1794, in _real_extract
    'uploader_id': self._search_regex(r'/(?:channel|user)/([^/?&#]+)', owner_profile_url, 'uploader id') if owner_profile_url else None,
  File "/usr/lib/python3/dist-packages/youtube_dl/extractor/common.py", line 1012, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)
youtube_dl.utils.RegexNotFoundError: Unable to extract uploader id; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Checklist

[ ] I'm reporting a broken site support issue
[ ] I've verified that I'm running youtube-dl version 2021.12.17
[ ] I've checked that all provided URLs are alive and playable in a browser
[ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[ ] I've searched the bugtracker for similar bug reports including closed ones
[ ] I've read bugs section in FAQ

Verbose log

PASTE VERBOSE LOG HERE

Description

WRITE DESCRIPTION HERE

Complete the template: #30839.

I'm having this same problem - just started today!

You too!


Mark functions as fallible

Problem
Division is fallible if the denominator is 0. It would be nice to know if the operation can fail and prove if it will or won't fail. Right now the best action is to just crash or throw out an Exception. For strict mode "crashes" are definitely not valid.
Describe the solution you'd like
Mark a function as fallible with a description of the conditions under which it will fail.
let $"/"(n: Number, d: Number): Number
     where fails if d == 0;

LM


Cable older version message could be improved

It has a typo here (vesion vs. version):
  [cable ] Cable 0.2.10 (vesion older than 0.2.11) initialized in the `hera`

Fixed in 311c0599


Azure AD connect Health for AD FS is also supported on Windows Server 2019

Azure AD connect Health for AD FS is also supported on Windows Server 2019
Please add that Azure AD connect Health for AD FS is also supported on Windows Server 2019 as per
https://learn.microsoft.com/en-us/azure/active-directory/hybrid/reference-connect-health-faq#what-version-of-identity-roles-are-supported-by-azure-ad-connect-health-

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 79ab2bbb-4adc-3106-c8dc-3462cc23c587
Version Independent ID: 8fa8f32e-ceca-6604-f2e3-a994efd17ecc
Content: What is Azure AD Connect and Connect Health. - Microsoft Entra
Content Source: articles/active-directory/hybrid/whatis-azure-ad-connect.md
Service: active-directory
Sub-service: hybrid
GitHub Login: @billmath
Microsoft Alias: billmath

@AlexCesarini
Thanks for pointing this out. Required changes have been made, once approve by author it will reflect in the document itself.


Return custom Not Found error

finding an object that does not exist currently returns
{
  name: 'not_found',
  message: 'missing'
  error: true,
  status: 404
}

coming from PouchDB’s MISSING_DOC error. For hoodie, we want
{
  name: 'Not found',
  message: 'Object with id "abc4567" is missing',
  status: 404
}

"abc4567" being the object’s id passed to store.find

This issue should be fixed in https://github.com/hoodiehq/pouchdb-hoodie-api as it implements the .find() method

[x] claim this issue (@janpawellek)
[x] add a test to tests/specs/find.js, tests/specs/update.js and master/tests/specs/remove.js
[x] commit the change with test: Not Found error
[x] start a PR (https://github.com/hoodiehq/pouchdb-hoodie-api/pull/101)
[x] implement the custom Not Found error
[x] commit the change with fix: Not Found error
[x] review & merge
[x] update https://github.com/gr2m/milestones/issues/88

If you have any questions, ping us at http://hood.ie/chat/ or https://twitter.com/hoodiehq

starting to work on this issue

fixed via #101


dosc: fix link in zh/guide/basic/custom-page.mdx

https://github.com/web-infra-dev/rspress/blob/main/packages/core/src/theme-default/styles/vars.css is 404 ---->>
https://github.com/web-infra-dev/rspress/blob/main/packages/theme-default/src/styles/vars.css
Summary
Related Issue

Checklist

[ ] I have added changeset via pnpm run change.
[ ] I have updated the documentation.
[ ] I have added tests to cover my changes.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Thanks!


Bug fix in NpgsqlQuerySqlGenerator with PgFunctionExpression

Bug
PgFunction("abbrev", typeof(string), new [] { inetExpression })
// where inetExpression is something like x => x.Inet

-- before:
SELECT abbrev(x."Inet", );

-- after:
SELECT abbrev(x."Inet");

Other
Also includes general cleanup to match project style, remove unused using-statements, etc.

Thanks!


Feature/join game polish

Issue #20
Includes

error states for modals
updated overlay css
disabled wombats for occupied colors
polished designs for dropdown

This does not include the game is full alert, which i'm going to put in a separate ticket @morganedmonds

#115 created for game full error alert

@oconn I'm going to merge this for now, but I'll definitely look for opportunities to use app state over local


Keep comments in datasource block during re-introspection

In the absence of another env management approach I'd like to be able to use comments in the datasource block. But re-introspecting removes them.

Hi there,
As part of our effort to clean up our GitHub repo, we’re closing this issue to help focus on features and improvements that align with our focus areas and those of our community.
This issue was originally created for internal tracking purposes, but if you feel this is important to you, feel free to reopen the issue and get like-minded users to upvote as we will prioritize fixing issues based on community interest.
For more insight into our priorities and how we’re thinking about the future of Prisma, please see our blog post: https://www.prisma.io/blog/prisma-orm-manifesto.
Thank you for your understanding and support!


coregistration GUI does not exit cleanly

Describe the bug
If I open the coregistration GUI and it crashes for some reason (e.g., missing subjects_dir), a dangling welcome banner is left which becomes distracting the next time the GUI is opened. One can exit ipython and restart the script, but this might require redoing lengthy computation.
Steps to reproduce
$ unset SUBJECTS_DIR

then in ipython:
>>> from mne.gui import coregistration
>>> coregistration()

Expected results
The welcome banner closes
Actual results☺

Additional information
In [2]: mne.sys_info()
Platform:         Linux-3.10.0-1160.76.1.el7.x86_64-x86_64-with-glibc2.10
Python:           3.8.6 | packaged by conda-forge | (default, Oct  7 2020, 19:08:05)  [GCC 7.5.0]
Executable:       /autofs/space/meghnn_001/users/mjas/anaconda3/envs/mne/bin/python3.8
CPU:              x86_64: 64 cores
Memory:           125.4 GB
Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr..match_module_callback at 0x7f8f64346940>
Traceback (most recent call last):
File "/autofs/space/meghnn_001/users/mjas/anaconda3/envs/mne/lib/python3.8/site-packages/threadpoolctl.py", line 400, in match_module_callback
self._make_module_from_path(filepath)
File "/autofs/space/meghnn_001/users/mjas/anaconda3/envs/mne/lib/python3.8/site-packages/threadpoolctl.py", line 515, in _make_module_from_path
module = module_class(filepath, prefix, user_api, internal_api)
File "/autofs/space/meghnn_001/users/mjas/anaconda3/envs/mne/lib/python3.8/site-packages/threadpoolctl.py", line 606, in init
self.version = self.get_version()
File "/autofs/space/meghnn_001/users/mjas/anaconda3/envs/mne/lib/python3.8/site-packages/threadpoolctl.py", line 646, in get_version
config = get_config().split()
AttributeError: 'NoneType' object has no attribute 'split'
mne:              1.4.dev0
numpy:            1.23.0 {unknown linalg bindings}
scipy:            1.5.3
matplotlib:       3.3.3 {backend=Qt5Agg}
sklearn:          0.23.2
numba:            Not found
nibabel:          3.2.1
nilearn:          0.7.0
dipy:             1.3.0
openmeeg:         Not found
cupy:             Not found
pandas:           1.1.5
pyvista:          0.36.1 {OpenGL 4.5.0 NVIDIA 455.45.01 via Quadro P5000/PCIe/SSE2}
pyvistaqt:        0.9.0
ipyvtklink:       Not found
vtk:              9.0.1
qtpy:             2.0.1 {PyQt5=5.12.9}
ipympl:           Not found
/autofs/space/meghnn_001/users/mjas/anaconda3/envs/mne/lib/python3.8/site-packages/pyqtgraph/colors/palette.py:1: RuntimeWarning: PyQtGraph supports Qt version >= 5.15, but 5.12.9 detected.
from ..Qt import QtGui
pyqtgraph:        0.13.1
pooch:            v1.5.2
mne_bids:         Not found
mne_nirs:         Not found
mne_features:     Not found
mne_qt_browser:   0.4.0
mne_connectivity: Not found
mne_icalabel:     Not found

@jasmainak I transferred it, no need to recreate.

wooops, sorry too late. Feel free to close one of them. Sorry for the noise.


updated contact us

I have well ensured that contact us is updated.
a sentence is deleted
Pull Request Template

Please provide all of the details below. Pull requests missing any of the information below will be closed!
updated the Contact us section.
Deleted the unrequired lines.
Providing the issue number/url helps us identify the problem being solved and keep the issues tab clean and tidy
33571f258e

This is not a fix. This is a request to add login and signup pages to the site.
Summarize what this PR fixes:-
updated the contact us
documented the saved changes
Fixes #10
Checklist
(Add an x between the brackets to check the items)

[ ] I have summarize what this PR fixes above
[ ] I have provided the issue ID/URL above
[ ] I have not added Bootstrap or jQuery and used correct styling guide.
[ ] This PR fixes only the issue mentioned above (create separate PRs for separate issues)
[ ] I was the first one to claim the issue or I created the issue - I created the issue

updated contact us


Feature/graph analysis merging

As discussed, I'm opening this PR to the work in progress graph-analysis-merging branch to speed up the PR process.
I will add more documentation and an overview of this PR once it's fully ready 😉

@herbiebradley @rdnfn Branch is now ready for review. I'm aware that a detailed review today will likely not be possible, so don't feel pressured to. For the publicly deployed branch we can do that after the report deadline if needed.
Main new features:

NodeMap: An object to store node identifications between two GeoGraph objects. It's essentially a bipartite graph. This forms the basis for timestacking. It can be inverted easily.
GeoGraphTimeline: The main new addition. Allows to stack several geographs in time, perform node identification between them (via .timestack()) and to analyse time-series of landscape, habitat, class and patch level metrics. The node identification also allows us to determine absolute and relative growth rates for nodes as well as classifying the node-dynamics type. While the functions for node-dynamics calculations, etc. are currently written in pure python, in my experiements they always exectued within under a second on graphs with ~2000 nodes. The major performance bottleneck are still edge computations when building up habitats or for getting e.g. avg_component_isolation across time. I think several of these might be alleviated sligthly by casting the performance relevant code into cython or numba-izing what can be transformed with numba. But that's definitely something for after the report 😉
Connectivity=8 now supported: We can fix the GDAL problem via our node identification on a connectivity=4 polygonised raster image. Connectivity=8 (diagonal patch connectivity) is important becauses it is the usual definition in rasterised landscape ecology. So using connectivity=8 is required to compare our metrics to Fragstats/pylandstats. Connectivity=4 is still supported as before.

Other changes:

TimedGeoGraph: A GeoGraph with a self.time attribute. Inherits all else from GeoGraph.
Handy wrappers for adding and removing nodes from GeoGraphs
Minor bugfixes (e.g. crs handling of GeoGraph, ...)
Demo notebooks that show how to plot and perform the time-series analysis. I will still clean those up a bit today.

All done! Looks good to merge, I just made a commit adding a docstring to _add_node in geograph.py, and improving some type hints.

Thanks a lot, that's excellent!


Support Merach MR-667 (by Huawei) indoor bike

Jean Caziewski brought my attention to an indoor bike "Merach MR-667" which has an empty Service Data.
Bluetooth name: Merach-MR667-B44F
Make, model: Merach MR-667

Video of the behavior: https://youtube.com/shorts/fnBTSyygtF8
Further notes from Jean:

When I allow open searching for all low energy Bluetooth devices, it displays in the list with a play icon and an hourglass icon in the top right corner.
There appears to be no response when I tap on the icon.

So because it's a question mark it wouldn't even show up if the device filtering is turned on (which is on by default). That's not good. It doesn't advance because probably there's some crash.

Released with v235 beta


fix(doc):updated and cleaner PR template

[ ] What does this PR do? [Mandatory]
updated and cleaner PR template. Eliminating noise from the template

[ ] What issue/task does this PR references? [Mandatory]

N/A
[ ] Are the tests Included? [Mandatory]

N/A
[ ] Is the documentation Included?

[ ] Are the Release Notes included?

[ ] Is/Are the @mention(s) to reviewer(s) included

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@cd1f237). Click here to learn what that means.
The diff coverage is n/a.

@@           Coverage Diff            @@
##             master   #2589   +/-   ##
========================================
  Coverage          ?   9.54%
========================================
  Files             ?     214
  Lines             ?   13702
  Branches          ?    1914
========================================
  Hits              ?    1308
  Misses            ?   12323
  Partials          ?      71

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cd1f237...fc4fd5e. Read the comment docs.

@rgarg1 Your image is available in the registry. Run docker pull registry.devshift.net/fabric8-ui/fabric8-planner:SNAPSHOT-PR-2589 && docker run -it -p 5000:8080 registry.devshift.net/fabric8-ui/fabric8-planner:SNAPSHOT-PR-2589 and visit http://localhost:5000 to access it.


Fix implementation issue of convTranspose2d to support options.groups

@fdwr @huningxin PTAL, thanks.

Thanks @huningxin for reviewing. I've updated commit to address your comments, please take another look.


Fix TrustedKeyKeys typo

In the godoc of TrustedIDKeyHashes, it should say "Not required if TrustedIDKeys is provided." instead of "Not required if TrustedKeyKeys is provided.".

Thanks for the fix!


fix: selectors: adds size prop, fixes styles to match the latest

SUMMARY:

Updated the button CSS
Updated the Dialog Button type
Updated the Dialog CSS
Added props to selectors and selector groups
updated selector CSS

https://user-images.githubusercontent.com/99700808/181661384-0437c09b-8ef6-42b6-bcec-fff7ddb49658.mp4
https://user-images.githubusercontent.com/99700808/181661407-9629210c-f389-4d58-bd47-47ab841b4777.mp4
JIRA TASK (Eightfold Employees Only):
ENG-23846: [Button] [Theme2] Default style has the wrong color in pressed state
ENG-23847: [Button] [Theme2] Split style's divider line is not passing accessibility
ENG-23852: [Dialog] [Theme2] cancel button should be neutral version and not default version
ENG-23853: [Modal] [Theme2] Close button should be in neutral style
ENG-23858; [Info Bar] [Theme2] surrounding padding should be 20px.
ENG-25252: Add ability to disable radio and checkbox groups at parent level
ENG-25728: [Selectors] RadioButton and CheckBox need size prop
ENG-25730: [Selectors] AllowDisabledFocus prop is missing
CHANGE TYPE:

[X] Bugfix Pull Request
[ ] Feature Pull Request

TEST COVERAGE:

[ ] Tests for this change already exist
[X] I have added unittests for this change

TEST PLAN:
Pull this pr and do yarn and yarn storybook verify the changes tot he following stories:
RadioButton
CheckBox
Default Button
Split Button
Dialog
Modal (inherits from Dialog)
InfoBar

Codecov Report

:exclamation: No coverage uploaded for pull request base (main@2ec21fd). Click here to learn what that means.
The diff coverage is n/a.

:exclamation: Current head 1f67c3e differs from pull request most recent head 7c2f62b. Consider uploading reports for the commit 7c2f62b to get more accurate results

@@           Coverage Diff           @@
##             main     #275   +/-   ##
=======================================
  Coverage        ?   88.36%
=======================================
  Files           ?      384
  Lines           ?    10403
  Branches        ?     3515
=======================================
  Hits            ?     9193
  Misses          ?     1190
  Partials        ?       20

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2ec21fd...7c2f62b. Read the comment docs.


No licence provided, neither in the source tree nor in the files

As the title says, I would appreciate to copy/paste your code in my project, but I'm not sure of you would accept it or not.
Thanks for the good examples anyways. They are at least instructive.
Mt

I must be blind :-(


Issue tagging

I noticed @grahamboree started tagging some issues as bugs - which is cool because it's something I was going to suggest! Should we pick a nice common set of tags to standardise on?
There are loads of articles about issue tagging best practices:

http://programmers.stackexchange.com/questions/129714/how-to-manage-github-issues-for-priority-etc
https://docs.saltstack.com/en/latest/topics/development/labels.html
https://robinpowered.com/blog/best-practice-system-for-organizing-and-tagging-github-issues/

I don't think we need to go quite as far as they do (I don't think functional areas or the milestones feature are useful for us, for example) but it would be nice to have some of it. At least category (bug/feature/discussion etc), and perhaps a status (resolved/wontfix etc). Thoughts?

I think the real benefit of labels is just being able to see quickly what the status of everything is at a high level.  At work, we have a pretty extensive issue labeling system that works well for us (labeling sprints, components, severity, etc..) but we also have an order of magnitude more issues than recast does. :stuck_out_tongue: I'd say, let's stick with something lightweight for now until we find the need to get more fine-grained to stay organized.  I like the Robin article's list, specifically how similar concerns are color-coordinated, though I think about 1/3 of those tags are relevant to us.  Want to just go with the relevant labels from their list for now?

I like the Programmers categorization better for it has type in each of its items - something that helps a lot to keep an eye on. It's clear and can be colored just as well by category.

I've added a few labels from the robin link that seem immediately relevant.  I think we should follow that style (color coding categories of labels) in the future, but we probably don't need to get much more crazy with labeling right now.  Here's my take on their definitions:

label
meaning

bug
undesired behavior, crashes, memory leaks, etc..

duplicate
this issue has already been reported

enhancement
change that would make an existing feature better

feature
a new feature/function that would be useful to have

invalid
this is not an appropriate or meaningful discussion to have here

optimization
performance-oriented issues or proposals

question
general discussion

wontfix
this is desired behavior and not something we're going to change.

I'm going to close this issue for now.  If we feel we need anything more than this in the future, we can re-open it.

Thanks Graham; I just realised I opened this then went away for 5 days - sorry! I agree, that cut-down list looks good. I've just added 'documentation' too which the robin link doesn't mention, but I think is useful.

Awesome.  Documentation is another good one.  :smile:


CLIMATE-743 - Partially restore utils.normalize_lat_lon_values

the old longitude conversion is a correct way to handle datasets whose longitude values are between 180 and 360.

Merged build triggered. Test Failed.

Merged build started. Test Failed.

Merged build finished. Test Failed.


NIHMS packages created from End-to-end testing contains no journal metadata

here is the metadata blob:
[
   {
      "id":"common",
      "data":{
         "title":"Electrogenerated chemiluminescence of Ru(bpy)32+ at a black phosphorus quantum dot modified electrode and its sensing application",
         "container-title":"The Analyst",
         "volume":"143",
         "issue":"1",
         "ISSN":"0003-2654,1364-5528",
         "abstract":"Strong anodic electrogenerated chemiluminescence (ECL) of Ru(bpy)32+ can be obtained under neutral conditions at a black phosphorus quantum dot (BPQD) modified electrode due to the catalytic effect of BPQDs. Dopamine exhibits an apparent inhibiting effect on the ECL signal, and as a result, can be sensitively detected.",
         "authors":[
            {
               "author":"Lei Zhang"
            },
            {
               "author":"KaiJin Tian"
            },
            {
               "author":"YongPing Dong",
               "orcid":"http://orcid.org/0000-0002-0455-9982"
            },
            {
               "author":"HouCheng Ding"
            },
            {
               "author":"ChengMing Wang"
            }
         ]
      }
   },
   {
      "id":"crossref",
      "data":{
         "doi":"10.1039/c7an01617d",
         "publisher":"Royal Society of Chemistry (RSC)",
         "journal-title-short":"Analyst"
      }
   }
]

one of the files sent to NIHMS that failed.
nihms-native-2017-07_2018-06-20_20-06-25_d0ff66f3-f336-40c1-b8c1-57042c65057b.tar.gz

more blob
[
   {
      "id":"JScholarship",
      "data":{
         "embargo":"NON-EXCLUSIVE LICENSE FOR USE OF MATERIALS This non-exclusive license defines the terms for the deposit of Materials in all formats into the digital repository of materials collected, preserved and made available through the Johns Hopkins Digital Repository, JScholarship. The Contributor hereby grants to Johns Hopkins a royalty free, non-exclusive worldwide license to use, re-use, display, distribute, transmit, publish, re-publish or copy the Materials, either digitally or in print, or in any other medium, now or hereafter known, for the purpose of including the Materials hereby licensed in the collection of materials in the Johns Hopkins Digital Repository for educational use worldwide. In some cases, access to content may be restricted according to provisions established in negotiation with the copyright holder. This license shall not authorize the commercial use of the Materials by Johns Hopkins or any other person or organization, but such Materials shall be restricted to non-profit educational use. Persons may apply for commercial use by contacting the copyright holder. Copyright and any other intellectual property right in or to the Materials shall not be transferred by this agreement and shall remain with the Contributor, or the Copyright holder if different from the Contributor. Other than this limited license, the Contributor or Copyright holder retains all rights, title, copyright and other interest in the images licensed. If the submission contains material for which the Contributor does not hold copyright, the Contributor represents that s/he has obtained the permission of the Copyright owner to grant Johns Hopkins the rights required by this license, and that such third-party owned material is clearly identified and acknowledged within the text or content of the submission. If the submission is based upon work that has been sponsored or supported by an agency or organization other than Johns Hopkins, the Contributor represents that s/he has fulfilled any right of review or other obligations required by such contract or agreement. Johns Hopkins will not make any alteration, other than as allowed by this license, to your submission. This agreement embodies the entire agreement of the parties. No modification of this agreement shall be of any effect unless it is made in writing and signed by all of the parties to the agreement.",
         "agreement-to-deposit":"true"
      }
   },
   {
      "id":"common",
      "data":{
         "title":"Late-life suicide prevention strategies: current status and future directions",
         "container-title":"Current Opinion in Psychology",
         "volume":"22",
         "ISSN":"2352-250X",
         "authors":[
            {
               "author":"Kim Van Orden"
            },
            {
               "author":"Charlene Deming"
            }
         ],
         "Embargo-end-date":"2018-06-30"
      }
   },
   {
      "id":"crossref",
      "data":{
         "doi":"10.1016/j.copsyc.2017.08.033",
         "publisher":"Elsevier BV",
         "journal-title-short":"Current Opinion in Psychology"
      }
   }
]

@htpvu Ok, for testing purposes I have merged the two JSON documents to insure coverage across the different elements represented in each document, and have changed container-title to journal-title.  This is the document (or minor variations of) that deposit services will be using in its ITs and UTs:
[
  {
    "id":"common",
    "data":{
      "title":"Electrogenerated chemiluminescence of Ru(bpy)32+ at a black phosphorus quantum dot modified electrode and its sensing application",
      "journal-title":"The Analyst",
      "volume":"143",
      "issue":"1",
      "ISSN":"0003-2654,1364-5528",
      "abstract":"Strong anodic electrogenerated chemiluminescence (ECL) of Ru(bpy)32+ can be obtained under neutral conditions at a black phosphorus quantum dot (BPQD) modified electrode due to the catalytic effect of BPQDs. Dopamine exhibits an apparent inhibiting effect on the ECL signal, and as a result, can be sensitively detected.",
      "authors":[
        {
          "author":"Lei Zhang"
        },
        {
          "author":"KaiJin Tian"
        },
        {
          "author":"YongPing Dong",
          "orcid":"http://orcid.org/0000-0002-0455-9982"
        },
        {
          "author":"HouCheng Ding"
        },
        {
          "author":"ChengMing Wang"
        }
      ],
      "Embargo-end-date":"2018-06-30"
    }
  },
  {
    "id":"crossref",
    "data":{
      "doi":"10.1039/c7an01617d",
      "publisher":"Royal Society of Chemistry (RSC)",
      "journal-title-short":"Analyst"
    }
  },
  {
    "id":"JScholarship",
    "data":{
      "embargo":"NON-EXCLUSIVE LICENSE FOR USE OF MATERIALS This non-exclusive license defines the terms for the deposit of Materials in all formats into the digital repository of materials collected, preserved and made available through the Johns Hopkins Digital Repository, JScholarship. The Contributor hereby grants to Johns Hopkins a royalty free, non-exclusive worldwide license to use, re-use, display, distribute, transmit, publish, re-publish or copy the Materials, either digitally or in print, or in any other medium, now or hereafter known, for the purpose of including the Materials hereby licensed in the collection of materials in the Johns Hopkins Digital Repository for educational use worldwide. In some cases, access to content may be restricted according to provisions established in negotiation with the copyright holder. This license shall not authorize the commercial use of the Materials by Johns Hopkins or any other person or organization, but such Materials shall be restricted to non-profit educational use. Persons may apply for commercial use by contacting the copyright holder. Copyright and any other intellectual property right in or to the Materials shall not be transferred by this agreement and shall remain with the Contributor, or the Copyright holder if different from the Contributor. Other than this limited license, the Contributor or Copyright holder retains all rights, title, copyright and other interest in the images licensed. If the submission contains material for which the Contributor does not hold copyright, the Contributor represents that s/he has obtained the permission of the Copyright owner to grant Johns Hopkins the rights required by this license, and that such third-party owned material is clearly identified and acknowledged within the text or content of the submission. If the submission is based upon work that has been sponsored or supported by an agency or organization other than Johns Hopkins, the Contributor represents that s/he has fulfilled any right of review or other obligations required by such contract or agreement. Johns Hopkins will not make any alteration, other than as allowed by this license, to your submission. This agreement embodies the entire agreement of the parties. No modification of this agreement shall be of any effect unless it is made in writing and signed by all of the parties to the agreement.",
      "agreement-to-deposit":"true"
    }
  }
]

understood.

⛔ = fixed value, or provided by the target repo
✅ = identified in the metadata blob
❌ = verified that we don’t need to capture this or include it in repo-specific metadata
NIH Metadata
Manuscript Metadata:

nihms id⛔ (provided by target)
isPublisherPdf⛔(effectively fixed, not provided by blob so defaults to false)
showPublisherPdf⛔(effectively fixed, not provided by blob so defaults to false)
manuscriptUrl⛔ (Not parsed from the Submission, unused)
manuscript title✅

Article Metadata:

embargo is ignored (for NIH)❌
doi✅

Journal Metadata

journal id❌
journal id type⛔ (fixed value)
publication type❓(not in the NIH metadata, defaults to null in serializer, and never set, so probably not needed)
issn ✅
journal title✅


Create systemd unit file for portus-cronos

We recently started to use cronos to handle some background jobs. We have to ensure the cronos service is started automatically inside of the appliance.
I think the cleanest solution is to create a systemd unit file for that.
Note well: cronos must be started in foreground mode otherwise systemd will have troubles handling it.
@jordimassaguerpla: I can help you with that if you want

Also update the wiki when this systemd service file gets created: https://github.com/SUSE/Portus/wiki/Synchronizing-the-Registry-and-Portus (the TODO on the bottom of the page)

I've started some work in:
https://github.com/jordimassaguerpla/Portus/tree/add_crono_systemd
I still need to properly test it but you can see the idea

https://github.com/SUSE/Portus/compare/master...jordimassaguerpla:add_crono_systemd?expand=1

adding here the packaging guidelines as a reminder on what I have to review to make it openSUSE compliant
https://en.opensuse.org/openSUSE:Systemd_packaging_guidelines


Decide what "ephemeral routes" are.

We should list the use cases, then determine how to address each one.

Right now they are used by:

popup menus
dropdown menus
snack bars
modal bottom sheets
non-modal bottom sheets

They are not used by the following, but maybe should be:

drawers
drag-and-drop

These fall into the following categories.
Modality.

modal. you cannot interact with the routes below them while they are up. popup menus, dropdown menus, modal bottom sheets, drawers.

non-modal. you can interact with other stuff while they are up. snack bars, non-modal bottom sheets, drag-and-drop.

Pop behaviour.

autopop on all push. when any new route is pushed, these must be popped. popup menus, dropdown menus, modal bottom sheets, drawers.

autopop on page push. when a new page route is pushed, these must be popped, but they can remain while other things are pushed on top. snack bars.

tied to parent lifetime. these don't go away until they are popped explicitly or their parent route is popped. non-modal bottom sheets.

long-lived. these don't go away until they are popped explicitly, they can even survive their parent being popped unless that explicitly pops them. drag-and-drop.

modal + autopop on all push is common to: popup menus, dropdown menus, modal bottom sheets, drawers.

 They should not time out while another route (e.g. a dialog) is over them, they should reset their timeout when their parent route resumes being the current route, they should be displayed consecutively in time, etc. They probably shouldn't use routes at all. #432

Right now, we handle the drawer as a regular modal route and manually pop it when pushing another route on top. This seems reasonable for now, though we may wish to revisit it if it becomes too painful.
This leaves one group: popup menus, dropdown menus, and modal bottom sheets, which are all auto-popping-on-any-push modal routes. There's only ever one of these at a time.
I think the way to handle these is just to use regular routes, and when a new route is pushed on top, tell them, and have them automatically pop themselves off then.

Seems reasonable.

A modal bottom sheet might have a menu, but we wouldn't want to dismiss the bottom sheet when its menu is shown.
Similarly a modal bottom sheet might pop up a dialog and not expect to be dismissed first.

Good point. Ok. Let me try this again.
First, I think the earlier classification has usefully scoped the problem down to popup menus, dropdown menus, modal bottom sheets, and drawers, so I will continue to ignore snack bars (#432), non-modal bottom sheets (#431), and drag-and-drop.
New proposal: we have a PageRoute subclass of ModalRoute, and a WidgetRoute subclass of ModalRoute, and when you push a PageRoute on top of a WidgetRoute, the WidgetRoute pops itself (and defers to its previous route).

So modal bottom sheets and popup menus would be WidgetRoutes and pushing a modal PageRoute would dismiss (pop) the WidgetRoutes?  This seems right to me. I'm not sure if it's necessary to encode the "dismiss me if I'm obscured by a page route" feature with a type, maybe it's just a Route flag?

I'm wary of making Routes have too many flags, but yeah, maybe. It's certainly hard to encode all the different axes in a class hierarchy.

So I implemented this, but I'm not at all convinced it's the way to go. I think it's actually fine for us to say "if you push a page route, you have to have popped your popup routes".
So unless someone objects, I'm going to just assert that you have.


Changed Color Image Mapping to be more accurate

Hi Marek,
while using Livescan, we noticed that it uses the color_image_to_depth_camera Transformation, which doesn't handle occlusion correctly. I think this was done for performance reasons?
Our team wanted to have more accuracy, so I implemented the depth_image_to_color_camera transformation. Here are the results:
Current version, which is using the color_image_to_depth_camera transformation:

This Pull Request, which is using the depth_image_to_color_camera transformation:

There are of course downsides to this.
This mapping technique is quite a bit slower than the original transformation, mainly because it produces a larger pointcloud, I implemented a color image resizer, which scales the color image down to the same height as the depth image, while preserving the aspect ratio of the color image. This produce a much smaller pointcloud.
I could also save some processing speed on the "StoreFrame" function by avoiding the use of "Vector.push_back".
But even using these tricks, this mapping has a higher performance cost, which could leed to a decrease in FPS on older systems.
However I think this is in a reasonable range, as I tested the software on two mid-tier processors (Desktop Ryzen 2600 and Mobile 8750H) and they held a stable 30 FPS while recording.
That said, I'm also quite a newbie regarding C++, so I hope I didn't do anything catastrophically wrong. I tested the software quite a bit and didn't encounter any errors/Memory leaks.

Hey Christopher,
Thanks a lot for submitting this, it looks very interesting!
How much work would it be to make it an option, selectable through settings?
Thanks,
Marek

Hi Marek,
Thanks, yes I can do that!
I'm on vacation in the next two weeks, so I'll provide the feature after that.
Best Regards!

Hi Christopher,
I think you answered to my previous comment about making this optional, which I since deleted - sorry.
On second though I realized that it makes sense for this feature to be enabled in all cases, especially given it works at full framerate on a mid-tier CPUs.
Also, when you checked the performance, how many Kinects did you run on the same machine?
Thanks,
Marek

Hi Marek,
Thank you for looking through the code and the detailed comments! I'll try to sort them out after the vacation.
Regarding the performance:
Both of the PCs I used weren't able to run more than one Kinect at full frame rate, even with the standard Livescan( for Azure Kinect), but I don't know if this is due to processor or USB limitations. So I didn't test this pull request with two Kinects at the same machine.
I'll try to get a bit more detailed performance statistics, so that you can decide if this should be enabled by default or as a an option!

Hi Marek,
So I did a Performance Test today of both the original Azure Kinect branch and this fix... And I could not really detect any significant difference between the two. But I think my testing method is flawed. Currently I set one breakpoint at the start of the update loop in liveScanClient.cpp and then read out the time it took to update one frame (Visual studio shows this per default at the right end of the line at the breakpoint). This happens in Debug mode, so the performance is obiously distorted, compared to Release Build performance.
Do you know any way to better measure the performance, maybe even for a Release Build?

Hi Christopher,
Visual Studio has a great profiler, which allows for measuring the performance of your code. Here is the documentation:
https://docs.microsoft.com/en-us/visualstudio/profiling/?view=vs-2019
Marek

Hi Marek,
thanks for your tip! I used the tool and the results are kinda interesting for me. So here they are:

The default Azure Kinect Branch uses ~730ms/Frame while ideling and ~800ms/Frame while Recording
This Pull Request uses ~520ms/Frame while ideling and ~600ms/Frame while Recording

I expected the results to be the other way around, so that this pull request is slower. I tried to to find the reason for this, and it basically comes down to the "mfmjpegdec.dll" not being used in this pull request. I don't really know the reason for this, maybe this has something to do with the K4A-SDK? I tested the Azure Kinect Branch and this pull request both with SDK Version 1.4.1
Here are the detailed results if you are interested:
Azure Kinect Branch:

This pull request:

Regarding the performance of "INTER_LINEAR" and INTER_NEAREST": I couldn't really detect any difference between the two, so I changed the interpolation method to inter_linear.
Otherwise I fixed all the changes you requested with the latest commit.

Hi Christopher,
Thanks a lot for looking into this. Did you profile the app in Debug or Release mode? The times looks to be quite long which makes me think it might have been Debug. Would you be able to profile again in Release? Sorry I'm not able to help, but I do not have access to a Kinect at the moment.
If that looks good I am happy to accept the PR.
Marek

Hello Marek,
yes you were right, I did profile in Debug mode, my bad! I didn't realize you could also profile in Release mode.
I did profile in Release mode again, and here the numbers look a lot more coherent with what I'd expect:

The default Azure Kinect Branch uses ~27-29ms/Frame while ideling and 28-30~ms/Frame while Recording

This Pull Request uses ~31-33ms/Frame while ideling and ~35-40ms/Frame while Recording

So there is a performance penalty while using this pull request.
There seems to be some performance overhead while profiling though, because running the build without it, the FPS-Counter in the Client Window never drops under 29.72 FPS while recording (for this pull request) . So I'm not really sure which measure is to trust.

Thanks for running the test Christopher. I think these numbers are fine, I'll now accept the PR.
Best regards,
Marek

Thanks a lot Marek for taking the time to review this PR!


Embed not working in mobile Chrome

I have implemented the package and it works great in all desktop browsers, however, most of the time it give the following error "The video is either unavailable or not supported in this browser (Error #400)" in mobile Chrome on both tablets and phones. I have attached a screenshot of one of the websites I am using this on. www.impactfestival.ca Mobile Safari and Firefox work fine on my iPad.
I looked into the error code and it seems to be a CORS issue related to the parent attribute on the iframe. I have added a parent property for the domain that the embed is hosted on, but I still see this issue. Any ideas?
For reference, this is my component implementation
import React from "react";
// @ts-ignore
import ReactTwitchEmbedVideo from "react-twitch-embed-video";

import styles from "./index.module.scss";

const TwitchChannel = ({ channel }: { channel: string }) => {
  const [showChat, setShowChat] = React.useState(false);

  return (

        <button
          className={styles.ChatButton}
          onClick={() => setShowChat(!showChat)}
        >
          {showChat ? "Hide" : "Show"} Chat
        </button>

      <ReactTwitchEmbedVideo
        layout={showChat ? "video-with-chat" : "video"}
        chat={window.innerWidth > 1025 ? "default" : "mobile"}
        theme="dark"
        width="100%"
        height={window.innerWidth > 1025 || showChat ? "900px" : "300px"}
        channel={channel}
        parent={["impactfestival.ca"]}
      />

  );
};

export default TwitchChannel;

This seems to be an issue with the stream itself. Unsure if it's when we are offline, or if there stream is corrupted somehow for mobile chrome, but a live channel works fine. Perhaps not an issue with this package, but I am curious if anyone else landing here has found this issue.

I haven seen any reports on this issue but its worth keeping this open to see if other reports come in, might be something to report to the Twitch Dev team if more poeple see the issue.


Positional params completion

Add Bash completion of positional parameters based on the position in the COMP_WORDS completion array.
Fixed issue: #468

Thanks for the PR! Let me take a look tomorrow.

One issue I found is that when I run the AutoCompleteTest, the nestedSubcommands test hangs and eventually dies with an OOM (stack trace below). This is probably because the default index range is 0..* and PositionalParamSpec.index().max is Integer.MAX_VALUE.
java.lang.OutOfMemoryError
	at java.lang.AbstractStringBuilder.hugeCapacity(AbstractStringBuilder.java:161)
	at java.lang.AbstractStringBuilder.newCapacity(AbstractStringBuilder.java:155)
	at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:125)
	at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
	at java.lang.StringBuilder.append(StringBuilder.java:136)
	at picocli.AutoComplete.addPosParamIndexRef(AutoComplete.java:548)
	at picocli.AutoComplete.generateNonEnumPositionParamCompletionCandidates(AutoComplete.java:543)
	at picocli.AutoComplete.generateFunctionForCommand(AutoComplete.java:500)
	at picocli.AutoComplete.bash(AutoComplete.java:349)
	at picocli.AutoCompleteTest.nestedSubcommands(AutoCompleteTest.java:134)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)

Would you mind adding a test with multiple positional parameters that showcases the new completion logic?

Hm, I thought Integer.MAX_VALUE would be just inserted as it is into the Bash script.... Why should it cause the OOM.
Alright, will add as soon as I get to my laptop :D

Ah, right. I got it...  Will fix.

@remkop I thought I fixed but the tests started failing for some reason... I'll try to fix it today/tomorrow...

Great, thanks! Please put a comment after you pushed so I get a notification.

Codecov Report

Merging #505 into master will increase coverage by 0.01%.
The diff coverage is 94.73%.

@@             Coverage Diff              @@
##             master     #505      +/-   ##
============================================
+ Coverage     92.86%   92.88%   +0.01%
- Complexity      393      404      +11
============================================
  Files             2        2
  Lines          5834     5871      +37
  Branches       1527     1537      +10
============================================
+ Hits           5418     5453      +35
  Misses          203      203
- Partials        213      215       +2

Impacted Files
Coverage Δ
Complexity Δ

src/main/java/picocli/AutoComplete.java
96.92% <94.73%> (-0.27%)
131 <8> (+11)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5ad0b3d...851ecd9. Read the comment docs.

@remkop Okay, so I've fixed everything I could but the last issue from CodeClimate about duplicated code is not clear to me...
Also added a test subcommand for more positional params variations + added support for multi-values.

Great, thank you!
I’ll take a detailed look in the next few days.


[retirement] Migrate /retirements pages - dev implementation

Related design task
jabby to scope out requirements/tasks here based on design...

duplicated by #1089


Update comments: s/UnsafeUnbounded/UnsafeIndefiniteSafeZone/g

UnsafeUnbounded was renamed to UnsafeIndefiniteSafeZone at some point, but
some comments still mentioned the former. Update them.

bors merge


DNS Preview Event Viewer Error - Event 4502, HealthService,

I am getting this error in the event viewer and logs are not going up to Azure Sentinel DNS preview.  This is working on 2 of the 3 DNS servers but not this server.
A module of type "Microsoft.EnterpriseManagement.Mom.Modules.DDIAnalytics.DnsCollector.DnsZonePerformanceDataSource" reported an exception System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: index
at System.ThrowHelper.ThrowArgumentOutOfRangeException(ExceptionArgument argument, ExceptionResource resource)
at System.Collections.Generic.List1.get_Item(Int32 index)    at Microsoft.EnterpriseManagement.Mom.Modules.DDIAnalytics.DnsCollector.DnsZoneDataCollector.GetDnsZoneQPS(List1 zoneInfoList, Int32 uploadTimeinterval, Dictionary2 prevState, Dictionary2& currentState) which was running as part of rule "Microsoft.IntelligencePacks.DDIAnalytics.DnsZonePerformanceCollector" running for instance "" with id:"{3AAF8287-8CA5-723B-6217-CF8C70D4D7AE}" in management group "AOI-5a039ff1-a96b-42c8-8689-42233f192a6a".

wrong place to post.

@seedosrun Thanks for the update. Just FYI which you might already be aware by now - In future, if you share any feedback under this azure-docs repo then please mention Azure documentation link / URL to which feedback is related to. This helps us to provide you better assistance. Cheers!


Ensure backend uses external Postgres database

Our backend needs a database to maintain the state of the service catalogue.

[x] Create Postgress database through pipeline
[x] Get Backstage to work with a Postgres DB - @SEQUOIIA
[x] Integrate database with backend config - @SEQUOIIA
[X] Pipeline connection string variable
[X] Connecting pipeline connection string variable to docker image?
[x] Populate catalogue
[x] Make it so Kubernetes secrets are created during pipeline run and not manually

https://github.com/dfds/backstage/issues/22

Duplicate of #22. Closing the older issue.

Create Postgress database through pipeline

Pipeline manifest can be found here: https://github.com/dfds/backstage-infrastructure/blob/master/db/azure-pipelines.yaml

Get Backstage to work with a Postgres DB

Done in this branch: users/sequoiia/database-change-sqlite-to-postgres

Integrate database with backend config

Done in this branch: users/sequoiia/database-change-sqlite-to-postgres

Pipeline connection string variable

Connecting pipeline connection string variable to docker image?

Didn't end up being necessary.

Populate catalogue

Has been done. For future reference on how to do this in a bulk fashion, see https://github.com/dfds/backstage-component-manifests-research/blob/master/apply-all.sh. Until that endpoint is protected by auth, it's rather easy..

Make it so Kubernetes secrets are created during pipeline run and not manually

Context: Secrets were created manually during development of this. It is now created during a regular pipeline run with the 'qetza.replacetokens.replacetokens-task.replacetokens@3' task.
See
https://github.com/dfds/backstage/blob/dfds-release/azure-pipelines.yml
and
https://github.com/dfds/backstage/blob/dfds-release/install-dfds/kubernetes/secrets.yaml


Fix REI compat: Display recipes for items not in primary output

Overview:
Fixed an issue where recipes with the searched item as a secondary (or later) output were not displayed in REI.
Changes:

Modified CreateDisplay to account for recipes with multiple outputs.
Improved REI integration to show recipes even if the searched item isn't the primary output.

Impact:
Players can now search for recipes where the requested item is a 2nd, 3rd, 4th, ... output, enhancing the recipe discovery experience.
Testing:
Test with "Redstone Dust", which is a secondary output of the Bulk Washing "Crushed Raw Iron" recipe. Before the fix, this recipe wouldn't appear when searching for recipes of "Redstone Dust".

Fixes https://github.com/Fabricators-of-Create/Create/issues/466

REI compat is overdue for a rewrite but this works as a stopgap


Update to Spezi Networking 2.0.1

Update to Spezi Networking 2.0.1
:recycle: Current situation & Problem

Several elements have moved in to Spezi Networking and the package got a 2.0 update.

:gear: Release Notes

Update to Spezi Networking 2.0.1

:pencil: Code of Conduct & Contributing Guidelines
By submitting creating this pull request, you agree to follow our Code of Conduct and Contributing Guidelines:

[x] I agree to follow the Code of Conduct and Contributing Guidelines.

Haha, my bad! Thank you @Supereg!


FancyZones does not activate

FancyZones does not activate. Just that.
First, I installed PowerToys on my PC (Windows 10 64x) and it worked just fine. I've set PowerToys to run at startup. And, after a restart, I saw the icon in the traybar - Image resizer and Shortcut guide work just right... But, FancyZones does not work anymore. I have to close the app and start it again, and again, and after 2 times of restarting it, FancyZones work again. That occurs every day (as I shut down my computer at night). So, I think is a bug in the app.

Hi @ticobye
please read this https://github.com/microsoft/PowerToys/issues/2563#issuecomment-622462804
It may solve your problem.

Let's see!

Going to mark this as a dup of #2563.  We believe this would would have been fixed in 0.18 release time frame.  If this did not resolve the issue, please reopen


[3.x] Fix description of OccluderShapePolygon.two_way

Despite the property name, the current description sounds like the opposite way, with true being one way only:
https://github.com/godotengine/godot/blob/2dd545b512a19e1c1c6a8dfddff62083e96bcb49/doc/classes/OccluderShapePolygon.xml#L39-L41

Ah yes that is clearer. :+1:

Thanks!


llcppg:close pipe writer

Without this close, cause the program to hang.

ready for review. @xushiwei


Houdini: Move Houdini Save Current File to beginning of ExtractorOrder

Brief description
In the spirit of this PR https://github.com/pypeclub/OpenPype/pull/2744 with this commit moving the order for the Save Scene plug-ins to beginning of ExtractorOrder this makes Houdini align with that too.
As discussed in the comments this seemed like the more sensible moment to save the current workfile.
Additional note
In Houdini there seemed to be two plug-ins involved with saving - one operating at a different target for different families.
We could technically add both in a single file and do something like:
class SaveCurrentScene(pyblish.api.ContextPlugin):
    """Save current scene"""

    label = "Save current file"
    order = pyblish.api.ExtractorOrder - 0.49
    hosts = ["houdini"]
    targets = ["deadline"]

    def process(self, context):
        ...

# Inherit from SaveCurrentScene to always save for all families for target `deadline`
class SaveCurrentSceneDeadline(SaveCurrentScene):
    hosts = ["houdini"]
    targets = ["deadline"]

That way the logic could be less duplicated and the Save plug-ins close together.

Shouldn't we add more families there, or maybe use it for all of them?

Shouldn't we add more families there, or maybe use it for all of them?

Quickly checked with our Houdini artist - he had no issue with scene being saved on all families by default. We can always tone that down if we notice issues with that approach.

or maybe use it for all of them?

Implemented with https://github.com/pypeclub/OpenPype/pull/2747/commits/7de42ee923dabba0c534774d6367fc44df67ecf6


coral is just cross domain MMD distance

Codecov Report
Attention: Patch coverage is 0% with 40 lines in your changes missing coverage. Please review.

Project coverage is 94.53%. Comparing base (88fd530) to head (f9bdbb1).

Files with missing lines
Patch %
Lines

domainlab/algos/trainers/train_coral.py
0.00%
40 Missing :warning:

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #875      +/-   ##
==========================================
- Coverage   95.23%   94.53%   -0.71%
==========================================
  Files         131      132       +1
  Lines        5229     5269      +40
==========================================
+ Hits         4980     4981       +1
- Misses        249      288      +39

Flag
Coverage Δ

unittests
94.53% <0.00%> (-0.71%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Error when attempting to restore

I'm trying to restore a backup file. The restore succeeds, but TDS seems to be failing to handle the response:
** (EXIT from #PID<0.73.0>) an exception was raised:
    ** (DBConnection.ConnectionError) client #PID<0.73.0> stopped: ** (FunctionClauseError) no function clause matching in Keyword.delete_key/3
    (elixir) lib/keyword.ex:498: Keyword.delete_key([[%{class: 0, length: 204, line_number: 2, msg_text: "Processed 105008 pages for database 'mydb', file 'mydb_data' on file 1.", number: 4035, proc_name: "", server_name: "589042304e43", state: 1}]], :done, false)
    (elixir) lib/keyword.ex:493: Keyword.put/3
    lib/tds/tokens.ex:381: Tds.Tokens.decode_token/2
    lib/tds/tokens.ex:49: Tds.Tokens.decode_tokens/2
    lib/tds/messages.ex:99: Tds.Messages.parse/4
    lib/tds/protocol.ex:326: Tds.Protocol.new_data/2
    lib/tds/protocol.ex:505: Tds.Protocol.send_param_query/3
    lib/db_connection.ex:958: DBConnection.handle/4
    lib/db_connection.ex:1078: DBConnection.describe_run/5
    lib/db_connection.ex:1199: DBConnection.run_begin/3
    lib/db_connection.ex:1133: DBConnection.run_meter/3
    lib/db_connection.ex:584: DBConnection.prepare_execute/4
    lib/tds.ex:24: Tds.query!/4

This should be fixed since v2, decoder is rewritten and logs info messages if debug level is turned on


VoiceData not being parsed correctly?

Looking at the hex dumps of the data portion of VoiceData packets, it seems that they're not being parsed properly (as portions of obviously not voice data are showing up).
Relevant excerpt from the hexdump of a VoiceData packet:
000000f0  33 04 00 04 a1 01 00 00  09 00 00 00 2d 62 61 63  |3...........-bac|
00000100  6b 20 20 20 00 04 a1 01  00 00 0a 00 00 00 20 2b  |k   .......... +|
00000110  66 6f 72 77 61 72 64 00  04 a1 01 00 00 20 00 00  |forward...... ..|
00000120  00 20 61 6c 69 61 73 20  6e 6d 6f 76 5f 72 65 66  |. alias nmov_ref|
00000130  6f 72 77 61 72 64 20 2b  66 6f 72 77 61 72 64 20  |orward +forward |
00000140  00 05 a2 01 00 00 8d 05  00 00 16 00 00 00 1b 0b  |................|

Code I'm using to grab the data:
for (const packet of analyser.getPackets()) {
  if (packet.packetType === "voiceData") {
      var buffer = packet.data.readArrayBuffer(packet.length);
      fs.writeFileSync(..., buffer);
  }
}

Any ideas to wha'ts going on?

voicedata is not something I've put a lot of effort in since it's useless for my use case but I'll see if I can find some time to double check it.
That dump looks like it's from a pov demo, if so, can you check if the same problem occurs for stv demos

Thanks for the response!
I ran the same segment on an STV demo and got this from running strings on one of the packets
,1qUc@
2#!p
r="(
^=C}
<``zMM
E'5H
Yd{$
<~=I-
})m.
2L=@;
wXHr]QH
j+H,(.<
;mxu
<IP_ADDRESS>:27020
D!w2?i
SdZV
        C]~

So it looks like it affects STV demos as well.


Add Profile SAS 9.4 (SASPy)

Summary
Add profile SAS 9.4 (SASPy) which uses Python and SASPy for those users who have python/saspy but not ITC in the target machine.
In order to use this connection type, you'll need to have Python and SASPy installed locally (the machine VS Code is installed on) or remotely (if you are using VScode remote development). SASPy fully supports connection methods of STDIO (Unix only), SSH, IOM (Remote and Local) and HTTP, and partially supports connection method of COM. You can visit the following link to find out how to install SASPy and how to configure it.
This work is mainly based on the ITC profile or mimic it.
Testing
Tested under

SASPy and local SAS 9.4 in Windows
Remote SASPy and SAS 9.4 in Windows server vis VScode remote SSH development
SASpy and remote SAS 9.4 (windows server) via IOM

@ShuguangSun thank you for this contribution! The maintainers team discussed this morning, and some questions come to mind. Overall, what is the motivation behind contributing this PR? Are there missing connection methods that you would like to see in what we currently offer in the extension?
Overall, it seems like the main benefit of this contribution, would be that python offers an operating system agnostic way to use the connection providers that saspy supports (except for COM), but wanted to see if there were additional motivations? We already support many of the connection methods that saspy does, the main difference is that our IOM and COM (COM is really IOM without java) implementation has a current requirement on ITC, which binds us to windows support only.
In this approach, this would require users to install python, configure it, and then install saspy, and setup a configuration profile for it, in addition to setting up the extension's saspy connection profile.  Saspy is mainly targeted at programmers that want an api level of direct access to execute SAS code. I'm concerned that requiring end users to successfully configure saspy as an additional setup would introduce un-necessary friction, that could otherwise be mitigated, if we were to directly add support for additional connection providers directly to the extension, versus going through an additional python adapter.
For the initial IOM research, we created a prototype that uses java and grpc to broker requests between the IOM jars and the extension itself, where the extension had a compact grpc client that could call the "remote" functions on the local server. While it offered an operating system agnostic way to interact with SAS on the client side, it introduced an external requirement on java, in addition to having a technical requirement that the IOM jars are present on a classpath when the tiny grpc server spins up.
Apologies for the long-winded comment. User feedback is important to us. We'd like to iterate on this discussion to hone in on details before deciding to accept or reject it.

Hi, thank you for the amazing works provided in the sas-extension and saspy ! I'm adding our feedback here regarding SAS 9.4 profiles.
We've a python-first linux dev environment based on a remote vscode that we access through a web url. We use saspy to query SAS Data through IOM, but we want to go further by using vscode as a SAS IDE.
Currently no connection type works for us, maybe this saspy based connection or the prototype that you mentioned would solve this problem as they are operating system agnostic.
Looking forward for it !

I have a situation that not have the priviliege to install ITC on my machine, but I can use python/saspy which is similar to @takikadiri . This is the trigger of my idea of SASPy Profile.
SASPy is more mature than the sas extensition and have more functions. Fore example, it provides a run script mode (batch mode), and like SAS EG ans studio, if we want to extend the extenstion a background run function, it can be easyly implemented in SASPy profile.
Python and SASpy provide easier way to handle log and output as they have done the dirty work. For example, to fetch the results, IOM/COM profile needs to identify local or remote directiory and the , howeer, saspy can just need to return the ll['LOG'] and writhe ll['LST'] to file. It might work for download data (as csv) as well.
Python and SASPy provide some modeling functions which looks like those task and applicable procedure in SAS EG and Studio. It might be not in our extension's cope. However if someone else want to provide some 3rd party exentension, the modeling things in SASPy can be used.
By the way, it should be a litte faster than IOM/COM in windows box especially when there a large chunck of flush or coding convert things.

Hi @ShuguangSun , @takikadiri , trying to better understand your use case. A lot of functionality in the extension has been added, but not all that functionality is supported for all connection types. Being able to maintain all configuration and connection definition information in a single location is very important, as was already mentioned earlier in this comment thread.
Instead of adding even more connection types, we'd rather enhance capabilities for existing connection types. This to keep development effort and maintenance to a minimum when more capabilities are added in the future .
I saw mention of not having privileges to install SAS Integration Technologies Client (ITC) software on the local machine. Could you elaborate on why that is? SAS Integration Technologies Client is officially supported software from SAS Institute that is installed on (almost) every Windows machine that has SAS software on it. And we have large amounts of users using this software.
I also saw mention of remote development. Am I correct in assuming that this is a setup as described in https://code.visualstudio.com/docs/remote/remote-overview? If so, then I have a number of follow up question:

Are you saying connection type SSH  in the SAS Extension does not work in this setup?
Or are you perhaps asking for enhancing the list of capabilities supported by the SSH connection type, so it includes more of the capabilities that are already supported for connection type "iom" (remote-IOM) and/or connection type "rest" (SAS Viya)? If so, which specific capabilities? We can then create issues of type feature request for those specific capabilities.
On which OS does the VS Code client run in your setup?

Hi @snlwih , our SAS setup doesn't allow SSH, we don't have user management at OS level. Moreover, we're using code-server as a web editor in a Linux environment. Currently i see no SAS 9.4 connection type that let us connect to a remote SAS server from a Linux environment.
The @ShuguangSun contribution seems to bring an OS agnostic connection type to SAS 9.4 (with a saspy dependency)

I also saw mention of remote development. Am I correct in assuming that this is a setup as described in https://code.visualstudio.com/docs/remote/remote-overview? If so, then I have a number of follow up question:
Yes.

* Are you saying connection type SSH  in the SAS Extension does not work in this setup?

I'm working with a SAS server on Windows which doesn't support -nodm. In other words, the current SSH profile only works for linux/unix kind server. For SAS server on Windows, it seems requiring meta server. As our IT can't setup the meta server well, I tried the local (COM) and IOM in vscode remote-ssh enviroment and failed.

* Or are you perhaps asking for enhancing the list of capabilities supported by the SSH connection type, so it includes more of the capabilities that are already supported for connection type "iom" (remote-IOM) and/or connection type "rest" (SAS Viya)? If so, which specific capabilities? We can then create issues of type feature request for those specific capabilities.

* On which OS does the VS Code client run in your setup?

Local Window to remote Windows.