IMAP server disconnected after 3 hours

All appeared fine in testing this morning - it processed a couple of emails sitting in the inbox, then  a few more i sent over the next 10 mintes. However when I sent another email almost 3 hours later I got the exception seen below. I have included the last successful read and move at 11:53
2024-08-07 11:53:21.5670 [DEBUG] Caa.Api.Email.MailMonitor: Incoming testing 4
2024-08-07 11:53:21.9330 [DEBUG] Caa.Api.Email.MailMonitor: After arrival move return uniqueId: 62
2024-08-07 11:53:26.6315 [ INFO] MailKitSimplified.Receive: imap.gmail.com<EMAIL_ADDRESS>INBOX message #68 departure processed.
2024-08-07 14:43:47.8126 [ INFO] MailKitSimplified.Receive: IMAP protocol exception, checking connection. MailKit.Net.Imap.ImapProtocolException: The IMAP server has unexpectedly disconnected.
   at MailKit.Net.Imap.ImapStream.ReadAheadAsync(Int32 atleast, CancellationToken cancellationToken)
   at MailKit.Net.Imap.ImapStream.ReadTokenAsync(String specials, CancellationToken cancellationToken)
   at MailKit.Net.Imap.ImapCommand.StepAsync()
   at MailKit.Net.Imap.ImapEngine.IterateAsync()
   at MailKit.Net.Imap.ImapEngine.RunAsync(ImapCommand ic)
   at MailKit.Net.Imap.ImapClient.NoOpAsync(CancellationToken cancellationToken)
   at MailKitSimplified.Receiver.Services.MailFolderMonitor.ProcessMessagesArrivedAsync(Boolean firstConnection, CancellationToken cancellationToken)
   at MailKitSimplified.Receiver.Services.MailFolderMonitor.WaitForNewMessagesAsync(CancellationToken cancellationToken)
2024-08-07 14:43:48.4505 [DEBUG] MailKitSimplified.Receive: imap.gmail.com<EMAIL_ADDRESS>INBOX message #69 arrived.
2024-08-07 14:43:48.4928 [DEBUG] Caa.Api.Email.MailMonitor: Incoming test 5
2024-08-07 14:43:48.6119 [ERROR] MailKitSimplified.Receive: Error occurred processing arrival queue item. imap.gmail.com<EMAIL_ADDRESS>INBOX #69. MailKit.ServiceNotConnectedException: The ImapClient is not connected.
   at MailKit.Net.Imap.ImapFolder.CheckState(Boolean open, Boolean rw)
   at MailKit.Net.Imap.ImapFolder.GetEnumerator()+MoveNext()
   at Microsoft.Extensions.Logging.LogValuesFormatter.TryFormatArgumentIfNullOrEnumerable[T](T value, Object& stringValue)
   at Microsoft.Extensions.Logging.LogValuesFormatter.Format(Object[] values)
   at NLog.Extensions.Logging.NLogLogger.CreateLogEventInfo[TState](LogLevel nLogLogLevel, EventId eventId, TState state, Exception exception, Func`3 formatter)
   at NLog.Extensions.Logging.NLogLogger.Log[TState](LogLevel logLevel, EventId eventId, TState state, Exception exception, Func`3 formatter)
   at Microsoft.Extensions.Logging.Logger.<Log>g__LoggerLog|14_0[TState](LogLevel logLevel, EventId eventId, ILogger logger, Exception exception, Func`3 formatter, List`1& exceptions, TState& state)

Originally posted by @alasdair-richardson in https://github.com/danzuep/MailKitSimplified/issues/70#issuecomment-2273531711

This may or may not be helpful (hopefully it is), but IMAP servers can drop client connections at any time for any reason.
It is possible that this is not a bug in your (existing) code but rather just a normal (but still "exceptional") circumstance that your library could potentially handle, much like HTTP "retry" libraries do.
I've been pondering trying to figure out a way to handle random server disconnects in MailKit itself, but it's "complicated".
(and gets even more complicated once you factor in OAuth2 access tokens which may have expired since the most recent successful login)

Thanks Jeffrey! It's an honour to have your input here as the one who made all of this possible. Kudos.

By the way @jstedfast , if you want to host this project on your site I'd be more than happy to transfer it accross as long as I can stay on as a maintainer.

The catch statement that logged the exception already reconnects which is why the next email that arrived was added to the queue, and the message is added back to the queue right after that error message, so I'll change it to just be a warning.

@danzuep As you can see from the log messages i posted I got the original Imap protocol exception,  then less than a second later it started processing the message (message #69 arrived then my code said subject was 'test 5'). It was after thta it fot the ServiceNotConnected exception. So I processed the email but it did not get moved.
When I tried just now to send another email, I got the Imap protocol exception again, then i saw your code receiving message #70, but my code was no longed called, so no attmept was made to process or move the email. log below
2024-08-07 17:26:55.1450 [ INFO] MailKitSimplified.Receive: IMAP protocol exception, checking connection. MailKit.Net.Imap.ImapProtocolException: The IMAP server has unexpectedly disconnected.
   at MailKit.Net.Imap.ImapStream.ReadAheadAsync(Int32 atleast, CancellationToken cancellationToken)
   at MailKit.Net.Imap.ImapStream.ReadTokenAsync(String specials, CancellationToken cancellationToken)
   at MailKit.Net.Imap.ImapCommand.StepAsync()
   at MailKit.Net.Imap.ImapEngine.IterateAsync()
   at MailKit.Net.Imap.ImapEngine.RunAsync(ImapCommand ic)
   at MailKit.Net.Imap.ImapClient.NoOpAsync(CancellationToken cancellationToken)
   at MailKitSimplified.Receiver.Services.MailFolderMonitor.ProcessMessagesArrivedAsync(Boolean firstConnection, CancellationToken cancellationToken)
   at MailKitSimplified.Receiver.Services.MailFolderMonitor.WaitForNewMessagesAsync(CancellationToken cancellationToken)
2024-08-07 17:26:55.9056 [DEBUG] MailKitSimplified.Receive: imap.gmail.com<EMAIL_ADDRESS>INBOX message #70 arrived.

Sending a further message gets it to your code without a problem but my code is not being called
2024-08-07 17:38:51.9108 [DEBUG] MailKitSimplified.Receive: imap.gmail.com<EMAIL_ADDRESS>INBOX message #71 arrived.

FWIW, since you seem to be dealing with GMail:
https://developers.google.com/gmail/imap/imap-smtp

Session length limits

Gmail POP sessions are limited to about 7 days. Gmail IMAP sessions are limited to about 24 hours. If the session was authenticated using OAuth credentials, it's limited to about the validity period of the access token used (usually 1 hour). A session in this context is one continuous TCP connection.

When the time elapses and the session expires, Gmail closes the connection with a message saying that the session expired. After that, the client can reconnect, authenticate again, and continue. If using OAuth, make sure that the access token used is valid (if you try to use an access token older than 1 hour, it might be invalid).

(bold text added to draw attention)

At the moment I am using Imap with user name and password.

I added a retry to the arrival queue so you should be fine now. I've run one of these for over a year before, but I had a local try-catch so I didn't need the service retry. Seems like it'll be useful for other people too though. Let me know next time you see that message, it should just come through as a warning now and carry on.

I have updated and restarted this morning. I will send more emails during the day, and let you know what happens,

@danzuep all appears to be working in terms of that reconnection. I had to put in a bit of extra logic to check if I had already stored the email - my process was store, move and if the move fails and I get the email again i don't want to store it again. This morning  found the following exception in the log and the monitor has stopped
2024-08-10 06:43:46.4584 [ INFO] MailKitSimplified.Receive: IMAP I/O exception, reconnecting.
2024-08-10 06:43:46.4694 [ERROR] MailKitSimplified.Receive: Idle client failed. System.Net.Sockets.SocketException (11001): No such host is known.
   at System.Net.NameResolutionPal.ProcessResult(SocketError errorCode, GetAddrInfoExContext* context)
   at System.Net.NameResolutionPal.GetAddressInfoExCallback(Int32 error, Int32 bytes, NativeOverlapped* overlapped)
--- End of stack trace from previous location ---
   at MailKit.Net.SocketUtils.ConnectAsync(String host, Int32 port, IPEndPoint localEndPoint, CancellationToken cancellationToken)
   at MailKit.Net.SocketUtils.ConnectAsync(String host, Int32 port, IPEndPoint localEndPoint, Int32 timeout, CancellationToken cancellationToken)
   at MailKit.MailService.ConnectNetworkAsync(String host, Int32 port, CancellationToken cancellationToken)
   at MailKit.Net.Imap.ImapClient.ConnectAsync(String host, Int32 port, SecureSocketOptions options, CancellationToken cancellationToken)
   at MailKitSimplified.Receiver.Services.ImapReceiver.ConnectImapClientAsync(CancellationToken cancellationToken)
   at MailKitSimplified.Receiver.Services.ImapReceiver.ConnectAuthenticatedImapClientAsync(CancellationToken cancellationToken)
   at MailKitSimplified.Receiver.Services.MailFolderMonitor.ReconnectAsync(CancellationToken cancellationToken)
   at MailKitSimplified.Receiver.Services.MailFolderMonitor.WaitForNewMessagesAsync(CancellationToken cancellationToken)
   at MailKitSimplified.Receiver.Services.MailFolderMonitor.IdleStartAsync(CancellationToken cancellationToken)

I assume I need to put a try catch block round the the Monitor function, though I am not sure if the exception propagated up ... Something like
while (true)
{
    try
    {
        await MonitorAsync();
    }
    catch (Exception e)
    {
        // Log it ...
    }
}

But there are a couple of questions.
1 Is this correct i.e. is there an exception to catch?
2 What the heck is No Such Host Known (imap.gmail.com)?
3 Should I catch any exception or just particular one(s)?
4 Should I also include the line in the try creating the DestinationFolder mailFolderClient.GetFolderAsync

Or maybe I had some network/internet connection glitch this morning, and this is unlikely to happen once we start running this in Azure? But some kind of back-off/retry logic should be possible, e.g. put in a wait in the catch block above?

Weirdly when I reconnected expecting to see the email I sent this morning get processed, it didn't. When I  look at the gmail inbox the email I sent this morning does not appear in the inbox or the processed folder - but I see it on the All Mail tab. It does not appear anywhere else (including Bin).

I've added a catch for the SocketException so you don't have to, but I don't expect it to be very common. See also.

Thank you @danzuep once again. Unfortunately we are gong away for a few days and I need to test this at least overnight and preferably for longer. We will be back Friday and I will test over the weekend and let you know the results.

Just getting round to starting testing. I saw 2 versions in nuget - 2.11.10-tags-v2-11-10-rc0001 (which is what I already installed) and 2.11.10-tags-v2-11-10-r-0001  (r-0001 rather the=an rc0001) which did seem to be slightly newer so I downloaded that.

Well things seem to be fine now. Not sure what the different disconnects are (see below), and I don't know if any of them was caused by the 'No such host' exception I saw before, but it seems to be reconnecting fine. I think you can probably close this. And thanks again
2024-08-19 21:50:25.6619 [ INFO] MailKitSimplified.Receive: Session expired, please login again. Reconnecting and trying again.
2024-08-19 23:20:38.9285 [ INFO] MailKitSimplified.Receive: The IMAP server has unexpectedly disconnected. Reconnecting and trying again.
2024-08-19 23:30:45.7210 [ INFO] MailKitSimplified.Receive: The IMAP server has unexpectedly disconnected. Reconnecting and trying again.
2024-08-20 00:58:39.3936 [ INFO] MailKitSimplified.Receive: The IMAP server has unexpectedly disconnected. Reconnecting and trying again.
2024-08-20 12:01:11.9890 [ INFO] MailKitSimplified.Receive: System error Reconnecting and trying again.
2024-08-20 12:53:59.4123 [ INFO] MailKitSimplified.Receive: System error Reconnecting and trying again.
2024-08-20 13:59:35.4224 [ INFO] MailKitSimplified.Receive: The IMAP server has unexpectedly disconnected. Reconnecting and trying again.

I've changed that message to DEBUG and removed the cancellation from Move so it doesn't end up in between folders again like last time.
https://www.nuget.org/packages/MailKitSimplified.Receiver/2.11.10

I'm afraid the no such host seems to have reared its head again. This morning I see the following in the log. I noticed when sending an email this afternoon and the monitor is not picking it up at all, seems not to be listening any more. No more messages in log from MailkitSimplified after 8.05m though the server is still running. On restarting the server it p[icked up the new emails ok.
2024-08-24 08:05:05.9133 [ INFO] MailKitSimplified.Receive: IMAP I/O exception, reconnecting.
2024-08-24 08:05:06.0098 [ INFO] MailKitSimplified.Receive: IMAP socket exception during connection attempt #1. imap.gmail.com<EMAIL_ADDRESS>INBOX.
2024-08-24 08:05:06.0207 [ INFO] MailKitSimplified.Receive: IMAP socket exception during connection attempt #2, backing off for 100ms. imap.gmail.com<EMAIL_ADDRESS>INBOX.
2024-08-24 08:05:06.1281 [ERROR] MailKitSimplified.Receive: IMAP socket exception during connection attempt #3, backing off for 100ms. imap.gmail.com<EMAIL_ADDRESS>INBOX. System.Net.Sockets.SocketException (11001): No such host is known.
   at System.Net.NameResolutionPal.ProcessResult(SocketError errorCode, GetAddrInfoExContext* context)
   at System.Net.NameResolutionPal.GetAddressInfoExCallback(Int32 error, Int32 bytes, NativeOverlapped* overlapped)
--- End of stack trace from previous location ---
   at MailKit.Net.SocketUtils.ConnectAsync(String host, Int32 port, IPEndPoint localEndPoint, CancellationToken cancellationToken)
   at MailKit.Net.SocketUtils.ConnectAsync(String host, Int32 port, IPEndPoint localEndPoint, Int32 timeout, CancellationToken cancellationToken)
   at MailKit.MailService.ConnectNetworkAsync(String host, Int32 port, CancellationToken cancellationToken)
   at MailKit.Net.Imap.ImapClient.ConnectAsync(String host, Int32 port, SecureSocketOptions options, CancellationToken cancellationToken)
   at MailKitSimplified.Receiver.Services.ImapReceiver.ConnectImapClientAsync(CancellationToken cancellationToken)
   at MailKitSimplified.Receiver.Services.ImapReceiver.ConnectAuthenticatedImapClientAsync(CancellationToken cancellationToken)
   at MailKitSimplified.Receiver.Services.MailFolderMonitor.ReconnectAsync(CancellationToken cancellationToken)
2024-08-24 08:05:06.2480 [ INFO] MailKitSimplified.Receive: IMAP service not connected, reconnecting.
2024-08-24 08:05:06.2577 [ INFO] MailKitSimplified.Receive: IMAP socket exception during connection attempt #1. imap.gmail.com<EMAIL_ADDRESS>INBOX.
2024-08-24 08:05:06.2577 [ INFO] MailKitSimplified.Receive: IMAP socket exception during connection attempt #2, backing off for 100ms. imap.gmail.com<EMAIL_ADDRESS>INBOX.
2024-08-24 08:05:06.3815 [ERROR] MailKitSimplified.Receive: IMAP socket exception during connection attempt #3, backing off for 100ms. imap.gmail.com<EMAIL_ADDRESS>INBOX. System.Net.Sockets.SocketException (11001): No such host is known.
   at System.Net.NameResolutionPal.ProcessResult(SocketError errorCode, GetAddrInfoExContext* context)
   at System.Net.NameResolutionPal.GetAddressInfoExCallback(Int32 error, Int32 bytes, NativeOverlapped* overlapped)
--- End of stack trace from previous location ---
   at MailKit.Net.SocketUtils.ConnectAsync(String host, Int32 port, IPEndPoint localEndPoint, CancellationToken cancellationToken)
   at MailKit.Net.SocketUtils.ConnectAsync(String host, Int32 port, IPEndPoint localEndPoint, Int32 timeout, CancellationToken cancellationToken)
   at MailKit.MailService.ConnectNetworkAsync(String host, Int32 port, CancellationToken cancellationToken)
   at MailKit.Net.Imap.ImapClient.ConnectAsync(String host, Int32 port, SecureSocketOptions options, CancellationToken cancellationToken)
   at MailKitSimplified.Receiver.Services.ImapReceiver.ConnectImapClientAsync(CancellationToken cancellationToken)
   at MailKitSimplified.Receiver.Services.ImapReceiver.ConnectAuthenticatedImapClientAsync(CancellationToken cancellationToken)
   at MailKitSimplified.Receiver.Services.MailFolderMonitor.ReconnectAsync(CancellationToken cancellationToken)
2024-08-24 08:05:06.4968 [ INFO] MailKitSimplified.Receive: IMAP service not connected, reconnecting.
2024-08-24 08:05:06.5041 [ INFO] MailKitSimplified.Receive: IMAP socket exception during connection attempt #1. imap.gmail.com<EMAIL_ADDRESS>INBOX.
2024-08-24 08:05:06.5041 [ INFO] MailKitSimplified.Receive: IMAP socket exception during connection attempt #2, backing off for 100ms. imap.gmail.com<EMAIL_ADDRESS>INBOX.
2024-08-24 08:05:06.6320 [ERROR] MailKitSimplified.Receive: IMAP socket exception during connection attempt #3, backing off for 100ms. imap.gmail.com<EMAIL_ADDRESS>INBOX. System.Net.Sockets.SocketException (11001): No such host is known.
   at System.Net.NameResolutionPal.ProcessResult(SocketError errorCode, GetAddrInfoExContext* context)
   at System.Net.NameResolutionPal.GetAddressInfoExCallback(Int32 error, Int32 bytes, NativeOverlapped* overlapped)
--- End of stack trace from previous location ---
   at MailKit.Net.SocketUtils.ConnectAsync(String host, Int32 port, IPEndPoint localEndPoint, CancellationToken cancellationToken)
   at MailKit.Net.SocketUtils.ConnectAsync(String host, Int32 port, IPEndPoint localEndPoint, Int32 timeout, CancellationToken cancellationToken)
   at MailKit.MailService.ConnectNetworkAsync(String host, Int32 port, CancellationToken cancellationToken)
   at MailKit.Net.Imap.ImapClient.ConnectAsync(String host, Int32 port, SecureSocketOptions options, CancellationToken cancellationToken)
   at MailKitSimplified.Receiver.Services.ImapReceiver.ConnectImapClientAsync(CancellationToken cancellationToken)
   at MailKitSimplified.Receiver.Services.ImapReceiver.ConnectAuthenticatedImapClientAsync(CancellationToken cancellationToken)
   at MailKitSimplified.Receiver.Services.MailFolderMonitor.ReconnectAsync(CancellationToken cancellationToken)
``

.

If I got a notification in my code I could back off for longer periods, perhaps extending the period etc

I've added another pre-release that allows you to disable some of the built in idle client exception handling with a Boolean so that you can back off for longer periods or do whatever else you find works. I might change the usage before the next release though to make it fluent.

Thank you. I assume that this only affects the disconnects happening on the listening socket, and the code to reconnect the Processed folder when moving the processed email is separate? It would help me if you could just point me at the actual reconnection code you have allowed us to disable so I can use that as a template for my replacement code.

This works:
await _monitorFolder
    .SetMessageSummaryItems()
    .SetIgnoreExistingMailOnConnect()
    .OnMessageArrival(ProcessMessage)
    .IdleAsync(cancellationToken)
    .ContinueWith(t => _logger.LogError(t.Exception?.GetBaseException(),
        "Idle client failed."), TaskContinuationOptions.OnlyOnFaulted); // add retry error handling here

Alternatively, to catch all the exceptions mentioned in the error messages you posted above:
                try
                {
                // idle here
                }
                catch (IOException ex)
                {
                    _logger.LogError(ex, $"{_imapReceiver} IMAP I/O exception, reconnecting.");
                }
                catch (SocketException ex)
                {
                    _logger.LogError(ex, ($"{_imapReceiver} IMAP socket exception, reconnecting.");
                }
                catch (ServiceNotConnectedException ex)
                {
                    _logger.LogError(ex, ($"{_imapReceiver} IMAP service not connected, reconnecting.");
                }

I have implemented some fallback/retry logic using a library called Polly. A bit of overkill for this purpose but it looks to be useful for other things too. Unfortunately the arrived messages are not being passed to my code. I think this may be due to the following code in MailFolderMonitor.IdleAsync
if (!handleExceptions)
        {
          taskArray = new Task[4]
          {
            mailFolderMonitor.IdleStartAsync(mailFolderMonitor._cancel.Token),
            mailFolderMonitor.ProcessDepartureQueueAsync(mailFolderMonitor._messageArrivalMethod, mailFolderMonitor._cancel.Token),
            mailFolderMonitor.ProcessDepartureQueueAsync(mailFolderMonitor._messageDepartureMethod, mailFolderMonitor._cancel.Token),
            mailFolderMonitor.ProcessFlagChangeQueueAsync(mailFolderMonitor._messageFlagsChangedMethod, mailFolderMonitor._cancel.Token)
          };
        }

Note that ProcessDepartureQueueAsync is being added twice rather than ProcessArrivalQueueAsync which is being added in the else section.

Try https://www.nuget.org/packages/MailKitSimplified.Receiver/2.11.13-tags-v2-11-13-b0001

Well it is calling my handling code now. I will let you know how it behaves over the next few days.

Finally got an error which backed out into my error handling. Unfortunately it did not work - I suspect I have done something  wrong - see exceptions below. Overnight this is what happened
2024-09-03 00:22:27.9247 [ INFO] ervices.MailFolderMonitor: The IMAP server has unexpectedly disconnected. IMAP protocol exception, reconnecting and trying again.
2024-09-03 03:25:00.1697 [ INFO] ervices.MailFolderMonitor: imap.gmail.com<EMAIL_ADDRESS>INBOX IMAP I/O exception, reconnecting.
2024-09-03 03:25:00.1793 [ INFO] ervices.MailFolderMonitor: imap.gmail.com<EMAIL_ADDRESS>INBOX IMAP socket exception during connection attempt #1.
2024-09-03 03:25:00.1793 [ INFO] ervices.MailFolderMonitor: imap.gmail.com<EMAIL_ADDRESS>INBOX IMAP socket exception during connection attempt #2, backing off for 501ms.
2024-09-03 03:25:00.7711 [DEBUG] Caa.Api.Email.MailMonitor: Exception(SocketException): No such host is known. ... delaying for 2000ms
2024-09-03 03:25:02.8458 [DEBUG] Caa.Api.Email.MailMonitor: Exception(SocketException): No such host is known. ... delaying for 4000ms
2024-09-03 03:25:06.9153 [DEBUG] Caa.Api.Email.MailMonitor: Exception(SocketException): No such host is known. ... delaying for 8000ms
2024-09-03 03:25:14.9691 [DEBUG] Caa.Api.Email.MailMonitor: Exception(SocketException): No such host is known. ... delaying for 16000ms
2024-09-03 03:25:31.0363 [DEBUG] Caa.Api.Email.MailMonitor: Exception(SocketException): No such host is known. ... delaying for 32000ms
2024-09-03 03:26:14.1735 [DEBUG] Caa.Api.Email.MailMonitor: Exception(SocketException): No such host is known. ... delaying for 64000ms
2024-09-03 03:27:18.8094 [DEBUG] Caa.Api.Email.MailMonitor: Exception(ImapProtocolException): The IMAP server has unexpectedly disconnected. ... delaying for 128000ms
2024-09-03 03:29:27.4969 [ INFO] ervices.MailFolderMonitor: imap.gmail.com<EMAIL_ADDRESS>INBOX (0) idle monitor started, fetching existing emails.
2024-09-03 03:29:27.4969 [ERROR] ervices.MailFolderMonitor: imap.gmail.com<EMAIL_ADDRESS>INBOX IMAP client is being accessed by multiple threads. System.ObjectDisposedException: The CancellationTokenSource has been disposed.
   at System.Threading.CancellationTokenSource.get_Token()
   at MailKitSimplified.Receiver.Services.MailFolderMonitor.WaitForNewMessagesAsync(CancellationToken cancellationToken)
2024-09-03 03:29:27.5032 [ INFO] ervices.MailFolderMonitor: imap.gmail.com<EMAIL_ADDRESS>INBOX monitoring complete.
2024-09-03 03:29:28.6271 [ INFO] ervices.MailFolderMonitor: imap.gmail.com<EMAIL_ADDRESS>INBOX (0) idle monitor started, fetching existing emails.
2024-09-03 03:29:28.6286 [ERROR] ervices.MailFolderMonitor: imap.gmail.com<EMAIL_ADDRESS>INBOX IMAP client is being accessed by multiple threads. System.ObjectDisposedException: The CancellationTokenSource has been disposed.
   at System.Threading.CancellationTokenSource.get_Token()
   at MailKitSimplified.Receiver.Services.MailFolderMonitor.WaitForNewMessagesAsync(CancellationToken cancellationToken)
2024-09-03 03:29:28.6286 [ INFO] ervices.MailFolderMonitor: imap.gmail.com<EMAIL_ADDRESS>INBOX monitoring complete.

Those last 4 lines in the log are now repeating every second or so and seem to indicate that my code and the base code perhaps both accesing the imap client in deifferent threads or somehting (no idea really). I will add the code from my retry logic below

Here is the code for my MonitorWithRetryAsync method. As I said before I am using the Polly library to do the retries, but that should not be causing the problem.
    private async Task MonitorWithRetryAsync(CancellationToken ct)
    {
        var strategy = new ResiliencePipelineBuilder().AddRetry(new()
        {
            ShouldHandle = new PredicateBuilder().Handle<Exception>(),
            MaxRetryAttempts = int.MaxValue,
            BackoffType = DelayBackoffType.Exponential,
            MaxDelay = new TimeSpan(0, 5, 0),
            OnRetry = args =>
            {
                var exception = args.Outcome.Exception!;
                var name = exception.GetType().Name;
                logger.LogDebug("Exception({Name}): {Message} ... delaying for {Time}ms", name,
                    exception.Message, args.RetryDelay.TotalMilliseconds);
                return default;
            }
        }).Build();

        var totalRequests = 0;
        while (!ct.IsCancellationRequested)
        {
            totalRequests++;
            try
            {
                await strategy.ExecuteAsync(async token => await MonitorAsync(token), ct);
            }
            catch (Exception e)
            {
                logger.LogError("Request {No} eventually failed with: {Message}", totalRequests, e.Message);
            }

            await Task.Delay(TimeSpan.FromSeconds(1), ct);
        }
    }

Note that as i have told the strategy to retry forever (but limited the maximum delay to 5 minutes) the code in the catch section above wshouldnever be called - I just left it in from the example in case ... I guess the main question is shoudl I do somethig in the OnRetry code in the strategy which might prevent the access from different threads problem? Particularly if the exception is IMAP server has unexpectedly disconnected.

This looks like it might be more related to Gmail than anything else sorry. I've hopefully fixed the cancellation token disposed excpetion in the latest pre-release, sorry for the delay in getting back to you.

Sorry I have not got back to you - I have not been well I am afraid. I will get back to testing this next week with your latest version. If I get the same problem again i will try to work out in more detail how it could have happened...

Hope you're feeling better now Alsdair. I added a forced reconnection after an error but haven't had a connection failure for it to be tested with yet, let me know how it goes.

Hi @danzuep I've been following this thread because I had a similar issue with Outlook 365.
Last week I decided to use your latest pre-release version 2.11.13-tags-v2-11-13-e0001 and added some code to ensure the IMAP client was still connected before trying to move emails.
if (!_imapReceiverClient.ImapClient.IsConnected) {     _logger.LogWarning("Reconnecting the IMAP client.");     await _imapReceiverClient.ConnectAuthenticatedImapClientAsync(); }
Everything seems to be working well regarding keeping the connection alive, so thanks!
However, there is an issue that is separate to this, so perhaps another thread can be started: Most of the emails that get process are then moved to a sub folder successfully but some emails still remain in the inbox despite no errors being thrown when I do the following:
await _imapReceiverClient.MailFolderClient.MoveToAsync(messageSummary, destinationFolder, _cancel.Token);
Is there a way to see why some emails are not moving with this call?

Good to hear it's working now, thanks for the update Mark. I'll close this topic then once I've done a release, so please create a new thread for the MoveToAsync issue. Please share the Debug logs there too.

OK, well I'm getting the sub folders. Here is my code block:
if (!_imapReceiverClient.ImapClient.IsConnected)                 {                     _logger.LogWarning("Reconnecting the IMAP client.");                     await _imapReceiverClient.ConnectAuthenticatedImapClientAsync();                 }                 await messageSummary.AddFlagsAsync(MessageFlags.Seen);                 var destinationFolder = await _imapReceiverClient.ImapClient.Inbox.GetSubfolderAsync(destFolderName, _cancel.Token);                 await _imapReceiverClient.MailFolderClient.MoveToAsync(messageSummary, destinationFolder, _cancel.Token);
I added the first check at the same time as updating my package to your new version at the same time. Are you saying I don't need to check it's connected before checking sub folders? I see from your link that it reconnects for the GetFolderAsync extension method but will this work too?

The release is done so I'll close this topic as complete. Please open a new issue for ongoing items.


modbus update

Hello,
is there a plan to update pymodus to 2.5.3 ? I ask because I can't use different pymodbus versions for every tool whats using pymodbus.
Best regards

Hi @CannonRS, you're welcome to contribute to this repo.

I've added #36 to upgrade to the latest version of pymodbus.


[SPARK-17495] [SQL] Support Decimal type in Hive-hash

What changes were proposed in this pull request?
Hive hash to support Decimal datatype. Hive internally normalises decimals and I have ported that logic as-is to HiveHash.
Generated code (in case any reviewer wants to examine):
/* 031 */   protected void processNext() throws java.io.IOException {
/* 032 */     while (inputadapter_input.hasNext() && !stopEarly()) {
/* 033 */       InternalRow inputadapter_row = (InternalRow) inputadapter_input.next();
/* 034 */       project_value = 0;
/* 035 */
/* 036 */       boolean inputadapter_isNull = inputadapter_row.isNullAt(0);
/* 037 */       Decimal inputadapter_value = inputadapter_isNull ? null : (inputadapter_row.getDecimal(0, 38, 0));
/* 038 */       if (!inputadapter_isNull) {
/* 039 */         project_childHash = org.apache.spark.sql.catalyst.expressions.HiveHashFunction.normalizeDecimal(
/* 040 */           inputadapter_value.toJavaBigDecimal(), true).hashCode();
/* 041 */       }
/* 042 */       project_value = (31 * project_value) + project_childHash;
/* 043 */       project_childHash = 0;
/* 044 */       project_rowWriter.write(0, project_value);
/* 045 */       append(project_result);
/* 046 */       if (shouldStop()) return;
/* 047 */     }
/* 048 */   }

How was this patch tested?
Added unit tests

ok to test

Test build #73410 has started for PR 17056 at commit a378b3e.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73410/
Test FAILed.

Jenkins retest this please
last failure was weird termination of jenkins:
[info] NaiveBayesClusterSuite:
Traceback (most recent call last):
  File "./dev/run-tests-jenkins.py", line 226, in <module>
    main()
  File "./dev/run-tests-jenkins.py", line 213, in main
    test_result_code, test_result_note = run_tests(tests_timeout)
  File "./dev/run-tests-jenkins.py", line 140, in run_tests
    test_result_note = ' * This patch **fails %s**.' % failure_note_by_errcode[test_result_code]
KeyError: -9
[error] running /home/jenkins/workspace/SparkPullRequestBuilder@2/build/sbt -Phadoop-2.6 -Phive-thriftserver -Phive -Dtest.exclude.tags=org.apache.spark.tags.ExtendedHiveTest,org.apache.spark.tags.ExtendedYarnTest hive-thriftserver/test mllib/test hive/test examples/test sql/test sql-kafka-0-10/test catalyst/test ; process was terminated by signal 9

retest this please

Test build #73460 has started for PR 17056 at commit 8595305.

Test build #73460 has finished for PR 17056 at commit 8595305.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73460/
Test PASSed.

Merged build finished. Test PASSed.

@cloud-fan @gatorsmile : can you please review this PR ?

Jenkins test this please

Test build #73595 has started for PR 17056 at commit 2387515.

Test build #73596 has started for PR 17056 at commit 2387515.

Test build #73595 has finished for PR 17056 at commit 2387515.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73595/
Test PASSed.

Merged build finished. Test PASSed.

cc @cloud-fan @gatorsmile

Test build #73596 has finished for PR 17056 at commit 2387515.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73596/
Test PASSed.

Test build #73610 has started for PR 17056 at commit 428a9a4.

Test build #73610 has finished for PR 17056 at commit 428a9a4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73610/
Test PASSed.

Let me manually check whether the results are consistent with Hive.

Test build #73670 has started for PR 17056 at commit c0c8390.

@gatorsmile : I really appreciate your help in reviewing this PR to the extent that you are manually checking the hashes over Hive. If you haven't already embarked on that, here is the set of hive queries corresponding to the test case in the PR which you can easily copy paste:
SELECT HASH(CAST(18BD AS DECIMAL(38, 0)));
SELECT HASH(CAST(-18BD AS DECIMAL(38, 0)));
SELECT HASH(CAST(-18BD AS DECIMAL(38, 12)));
SELECT HASH(CAST(18446744073709001000BD AS DECIMAL(38, 19)));
SELECT HASH(CAST(-18446744073709001000BD AS DECIMAL(38, 22)));
SELECT HASH(CAST(-18446744073709001000BD AS DECIMAL(38, 3)));
SELECT HASH(CAST(18446744073709001000BD AS DECIMAL(38, 4)));
SELECT HASH(CAST(9223372036854775807BD AS DECIMAL(38, 4)));
SELECT HASH(CAST(-9223372036854775807BD AS DECIMAL(38, 5)));
SELECT HASH(CAST(00000.00000000000BD AS DECIMAL(38, 34)));
SELECT HASH(CAST(-00000.00000000000BD AS DECIMAL(38, 11)));
SELECT HASH(CAST(123456.1234567890BD AS DECIMAL(38, 2)));
SELECT HASH(CAST(123456.1234567890BD AS DECIMAL(38, 20)));
SELECT HASH(CAST(123456.1234567890BD AS DECIMAL(38, 10)));
SELECT HASH(CAST(-123456.1234567890BD AS DECIMAL(38, 10)));
SELECT HASH(CAST(123456.1234567890BD AS DECIMAL(38, 0)));
SELECT HASH(CAST(-123456.1234567890BD AS DECIMAL(38, 0)));
SELECT HASH(CAST(123456.1234567890BD AS DECIMAL(38, 20)));
SELECT HASH(CAST(-123456.1234567890BD AS DECIMAL(38, 20)));
SELECT HASH(CAST(123456.123456789012345678901234567890BD AS DECIMAL(38, 0)));
SELECT HASH(CAST(-123456.123456789012345678901234567890BD AS DECIMAL(38, 0)));
SELECT HASH(CAST(123456.123456789012345678901234567890BD AS DECIMAL(38, 10)));
SELECT HASH(CAST(-123456.123456789012345678901234567890BD AS DECIMAL(38, 10)));
SELECT HASH(CAST(123456.123456789012345678901234567890BD AS DECIMAL(38, 20)));
SELECT HASH(CAST(-123456.123456789012345678901234567890BD AS DECIMAL(38, 20)));
SELECT HASH(CAST(123456.123456789012345678901234567890BD AS DECIMAL(38, 30)));
SELECT HASH(CAST(-123456.123456789012345678901234567890BD AS DECIMAL(38, 30)));
SELECT HASH(CAST(123456.123456789012345678901234567890BD AS DECIMAL(38, 31)));

Thank you! I checked Hive 2.1. It has the exactly same hash values.

LGTM. cc @cloud-fan for final signing-off

Test build #73670 has finished for PR 17056 at commit c0c8390.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73670/
Test PASSed.

Merged build finished. Test PASSed.

@cloud-fan ping !!

Test build #73945 has started for PR 17056 at commit 65a09e9.

Test build #73945 has finished for PR 17056 at commit 65a09e9.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73945/
Test FAILed.

LGTM(if tests pass)

Test build #74018 has started for PR 17056 at commit 7c0b6c8.

Test build #74018 has finished for PR 17056 at commit 7c0b6c8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74018/
Test PASSed.

thanks, merging to master!


Allow all editor modes to select nodes in the viewport

Closes godotengine/godot-proposals#8563
All behavior once a node is selected is the same, this only allows you to select a node with any editor mode when no node is selected.

This breaks Ruler Mode. Since it doesn't interact with nodes, it should not be capable of selecting. Same with Pan Mode.

This breaks Ruler Mode. Since it doesn't interact with nodes, it should not be capable of selecting. Same with Pan Mode.

Good catch, it's fixed. As well I allowed for the move tool to allow dragging in the same click as selecting, just like the select tool.


Look over the API and check to see if any variables that an interview needs to define should be added to the wizard

interview_short_title

See also all of the steps here to see what can be automated: https://trello.com/c/unTtUH4b

We have been slowly adding new required variables to the project. We should set those to defaults in the wizard so it's easy to see and edit.

We already fixed interview_short_title. Let's address this when we have more specific ideas.


Wiki create process test

Noticing that there are no test cases for the "PerformEditsProcess" class in the wiki extension, I try to create a simple test case to test the constructor of "PerformEditsProcess" by calling the "createProcess" method in PerformWikibaseEditsOperation. I plan to further test the process with threading simulation if this approach is acceptable.
Changes proposed in this pull request:
add "testCreateProcess" method in PerformWikibaseEditsOperationTest.java

Hi @chupengrocky, thank you very much for this pull request!
First, you can safely ignore the prepare_ui_test_matrix failing check, it is unrelated (we are working on it).
Concerning your new test, it tackles an area that is indeed not so easy to test (especially the run method)!
I wonder if it could be possible to add some assertions on the Process instance returned by createProcess? For instance, you could check the value of some fields (since they are protected they should be visible from this test class).

Hi @chupengrocky, do you still intend to work on this?

Yes, I am still working on this. Sorry about the late update:
https://github.com/OpenRefine/OpenRefine/pull/4221


mavsdk_server is not installed in Docker image

When I install MAVSDK-Python through pip in a Docker image, the mavsdk_server binary is not downloaded, even though I believe it should be (sys.platform.startswith('linux') and platform.machine() == "x86_64" from setup.py evaluates to True). Manually downloading the binary from https://github.com/mavlink/MAVSDK/releases/download/v0.27.0/mavsdk_server_manylinux2010-x64 to /usr/local/lib/python3.6/dist-packages/mavsdk/bin/mavsdk_server, then setting it executable using chmod +x works perfectly and as if the binary had been installed by pip.
I am working off of the px4io/px4-dev-simulation-bionic Docker image. A minimal Dockerfile to reproduce this issue looks as follows:
FROM px4io/px4-dev-simulation-bionic
RUN pip3 install mavsdk

I have no idea why exactly this happens. But I'm assuming it should be an easy fix as it can be done manually no problem at all. Maintainability of the manual version is kind of bad, though.

I just tried in an ubuntu:20.04 with docker, and it seems to work. Would you mind trying that, too? I did the following:
docker run --rm -it ubuntu:20.04
apt update
apt install python3
apt install python3-pip
pip3 install mavsdk

I got the following output, confirming that it installed the manylinux2010 wheel:
root@901a74d4b60f:/# pip3 install mavsdk
Collecting mavsdk
  Downloading mavsdk-0.9.1-py3-none-manylinux2010_x86_64.whl (7.1 MB)
     |████████████████████████████████| 7.1 MB 5.1 MB/s

And I could run the embedded mavsdk_server manually:
root@901a74d4b60f:/# /usr/local/lib/python3.8/dist-packages/mavsdk/bin/mavsdk_server
[12:23:02|Info ] MAVSDK version: 0.27.0 (mavsdk_impl.cpp:26)
[12:23:02|Debug] New: System ID: 0 Comp ID: 0 (mavsdk_impl.cpp:372)
[12:23:02|Info ] Server started (grpc_server.cpp:43)
[12:23:02|Info ] Server set to listen on <IP_ADDRESS>:44155 (grpc_server.cpp:44)
[12:23:02|Info ] Waiting to discover system on udp://:14540... (connection_initiator.h:22)

Could you confirm that it works for an ubuntu:20.04 container?

I can confirm that it does indeed work for an ubuntu:20.04 container.
I also tried the exact same based on an ubuntu:bionic container (which the px4io/px4-dev-simulation-bioniccontainer is based on). In this case it does not work.
A few things to note:

With ubuntu:bionic I get Python 3.6.9 (as opposed to 3.8.2)
pip3 install mavsdk runs for the better part of 15 minutes in the ubuntu:bionic container and remains stuck on Running setup.py bdist_wheel for grpcio .... I have actually seen this happen before as well.
The following output seems to confirm that the wrong wheel is being installed:

Collecting mavsdk
  Downloading https://files.pythonhosted.org/packages/25/7f/d4f8a506fc6dc0660543a49024ef7bd7d63d3825d800c0722dcac056d918/mavsdk-0.9.1-py3-none-any.whl (192kB)

Right. I don't know why, then. Note that setup.py is there to create the wheel, but it is not the one deciding which wheel is installed on a platform. This is done through the name: mavsdk-0.9.1-py3-none-manylinux2010_x86_64.whl says that it is a manylinux2010_x86_64 wheel, and then I believe that pip is the one deciding which one it wants.
It's supposed to take the more specific one first (in this case manylinux2010_x86_64). So apparently, with bionic, pip seems to believe that the platform does not support manylinux2010_x86_64 and it therefore falls back to none, which is the pure python package (i.e. the one that does not embed mavsdk_server).
I believe that it works (i.e. it downloads the manylinux2010 one) on Ubuntu 18.04 (without docker), but I'm not completely sure. If somebody with an Ubuntu 18.04 could test that, it would be informative.
Some ideas regarding your notes:

With ubuntu:bionic I get Python 3.6.9 (as opposed to 3.8.2)

This should be fine, we need 3.6+

remains stuck on Running setup.py bdist_wheel for grpcio ...

I believe that if pip doesn't find binaries for the platform, it will try to build gRPC from sources. Which takes a while. This does not happen for mavsdk because we don't instruct pip on how to build mavsdk_server. Either it finds the binary, or it takes the pure wheel. But gRPC does. That's typically what happens when installing on an RPi: pip install mavsdk will end up building gRPC from sources.

The following output seems to confirm that the wrong wheel is being installed

Yes, it chose the generic, "pure python" wheel.

Closing as it is should now be easy to install using pip install mavsdk.


[Fabric Bot] Service Contacts for Digital Twins

Summary
The focus of these changes is to update the service contacts for Digital Twins, incorporating the changes made in #2111.

/check-enforcer override


Schema explorer

Insomnia Version: ___Version 6.0.1 (<IP_ADDRESS>8)
Operating System: ___ macOs High Sierra

Details
Would be nice to have a schema explorer that show different mutations/queries available for a particular service.
Schema will include the different input/output types expected and help in writing the different queries/mutations in the client.

Closing this as it's a duplicate of #451


Content in a new row is not aligned

Hi
I'm trying to display images, but the result is not as expected
the fifth picture should appear on the bottom left, and be aligned with the content above.
The HTML file is attached to this message
thank you in advance for your help
Unaligned.zip

for how-to/troubleshooting questions, please use stackoverflow or similar. see https://github.com/twbs/bootstrap/blob/master/CONTRIBUTING.md#using-the-issue-tracker


Why do I need FileReader at all?

I need to post without using multipart
I can do this
$http.post("https://api.parse.com/1/files/" + files[0].name, files[0], {
		            withCredentials: false,
		            headers: {
		            		'Content-Type': files[0].type
		            },
		           transformRequest: angular.identity
		    });

It works totally fine. But when I try to switch to this library $uploader.http() method based on https://github.com/danialfarid/ng-file-upload/issues/88 I have to use a FileReader.
What is going on? Why do I suddenly need FileReader?

You are right it is not needed.
I have updated the README and gonna update the demo page.
Thanks

Fixed in version 3.3.0


Pure and zero concentrated adaptive composition

Adaptive composition for pure and approximate DP
Linked issue #23

See #29


User specific repoID or xorm builder conditions for issue search (#19475)

Backport #19475

:rocket:


upgrade to python3.12.4

Also upgrade celery. The 5.3 release that caused us trouble (5.3.3) was ultimately yanked so we'll accept the latest kombu/celery in 5.3 series as part of this upgrade.

Closes: https://github.com/pypi/warehouse/issues/13757


[core-http] New Pipeline Design RFC

Redesigning Pipelines
Goals
Today it is not very easy to customize pipelines in core-http as they are presented as a difficult-to-inspect array of policy factories. The new design seeks to:

Make it easy to compose pipelines by expressing dependencies between policies
Make it easy to remove policies that are unnecessary (so as to replace them with new custom policies)
Avoid unnecessary allocations after creating a policy, such that new isn't required to be called when executing a particular policy per operation.

Design
Policies
A PipelinePolicy is an object that implements the following interface:
type SendRequest = (request: Request) => Promise<Response>;

interface PipelinePolicy {
    name: string;
    sendRequest(request: Request, next: SendRequest): Promise<Response>;
}

It is given a string name (policies defined in core will use strings from an exported string enum) so that consumers can express dependencies relative to existing policies. The sendRequest method uses a middleware pattern, which is familiar to JS devs.
Open Question: Is there a way to type responses here for deserialization policies?
Pipelines
A Pipeline composes several policies and provides the ability to make a request that will flow through the policies in a specific order determined by their requirements.
interface AddPipelineOptions  {
    beforePolicy?: string | string[];
    afterPolicy?: string | string[];
    beforePhase?: string | string[];
    afterPhase?: string | string[];
    duringPhase?: string | string[];
}

interface Pipeline {
    addPolicy(policy: PipelinePolicy, options?: AddPipelineOptions);
    removePolicy(options: { name?: string; phase?: string });
    sendRequest(httpClient: HttpClient, request: Request): Promise<Response>;
    getOrderedPolicies(): PipelinePolicy[];
    clone(): Pipeline;
}

Dependencies expressed when calling addPolicy are not sensitive to order. The following code snippets have identical effects:
pipeline.addPolicy(barPolicy);
pipeline.addPolicy(fooPolicy, { afterPolicy: 'barPolicy' });

// The below is the same as the above.
// Policies don't have to be part of the pipeline in order to be
// referenced by before/afterPolicy.
pipeline.addPolicy(fooPolicy, { afterPolicy: 'barPolicy' });
pipeline.addPolicy(barPolicy);

Phases are predefined buckets of activity that pre-defined policies inside core will fall into. They are currently a fixed set and not customizable by pipeline consumers.
Current proposed phases:

Serialize
Retry

Known phase names will be exported as a constant string enum.
Champion Scenarios
Inserting a new policy at a specific place
interface TracingPolicyOptions {
    tracingEnabled: boolean;
}

function tracingPolicy(options: TracingPolicyOptions): PipelinePolicy {
    async function sendRequest(request: Request, next: SendRequest): Promise<Response> {
        if (options.tracingEnabled) {
            // add some headers or something!
        }
        return next(request);
    }
    return {
        name: "TracingPolicy",
        sendRequest
    }
}

// Extend an existing Pipeline
pipeline.addPolicy(
    tracingPolicy({tracingEnabled: true}),
    {beforePhase: PipelinePhases.Retry}
);

Replacing retry policies
pipeline.removePolicy({phase: PipelinePhases.Retry);
pipeline.addPolicy(customRetryPolicy, {duringPhase: PipelinePhases.Retry})

Other Considerations

How to create/customize default pipelines in service clients
How to customize pipelines per client operation

/cc @chradek @bterlson @ramya-rao-a @daviwil @jeremymeng

Simplified phases down to just Retry and Serialize.


Can't get https running

There's an .env file with following content in the root directory of the react app:
HTTPS=true
SSL_CRT_FILE=/ssl/certificate.crt
SSL_KEY_FILE=/ssl/certificate.key

 /app/myProjectPath:
  total used in directory 484 available<PHONE_NUMBER>
  drwxrwxrwx  1 user Domänen-Benutzer   8192 06-16 19:17 ..
  drwxrwxrwx  1 user Domänen-Benutzer   4096 06-21 10:59 .
  -rw-rw-rw-  1 user Domänen-Benutzer     82 06-22 17:18 .env
  drwxrwxrwx  1 user Domänen-Benutzer   4096 06-21 10:59 .git
  -rw-rw-rw-  1 user Domänen-Benutzer    324 2020-06-20  .gitignore
  -rw-rw-rw-  1 user Domänen-Benutzer   1067 06-21 10:56 node_modules
  -rw-rw-rw-  1 user Domänen-Benutzer    759 06-21 10:59 package.json
  -rw-rw-rw-  1 user Domänen-Benutzer    753 06-16 19:28 package.json.bkp
  -rw-rw-rw-  1 user Domänen-Benutzer    176 2020-11-12  post.sh
  drwxrwxrwx  1 user Domänen-Benutzer   4096 06-16 19:17 public
  -rw-rw-rw-  1 user Domänen-Benutzer   9830 06-16 19:17 README.md
  drwxrwxrwx  1 user Domänen-Benutzer   4096 06-16 19:17 src
  -rw-rw-rw-  1 user Domänen-Benutzer<PHONE_NUMBER>-06-20  yarn.lock

When the server starts, I've got the output:
><EMAIL_ADDRESS>start /app/myProjectPath

> react-scripts start

certificate.crt file exists.

certificate.key file exists.

Starting the development server...

Compiled successfully!

You can now view react-template in the browser.

https://localhost:3000/

Note that the development build is not optimized.

To create a production build, use yarn build.

To ensure that both the certificate and key file can be read by node.js, I've added some debug code inside start.js:
if(fs.existsSync('/ssl/certificate.crt')) {
	          console.log("certificate.crt file exists.");
} else {
                  console.log('certificate.crt file does not exist.');
}

if(fs.existsSync('/ssl/certificate.key')) {
	          console.log("certificate.key file exists.");
} else {
                  console.log('certificate.key file does not exist.');
}

As you can see in the output above, both files could be read at startup.
If I call https://localhost:3000 I've got different browser error messages:
Firefox 89: SEC_ERROR_INADEQUATE_KEY_USAGE
Chrome 88: NET::ERR_CERT_AUTHORITY_INVALID
The same key and certificate files are working well in a simple Apache https config, so I assume the certificate and its key are issued correctly.
So what's wrong with node.js https development server here?

Any fixes ?


UI: download and display s3 vault data

Task Description
When a vault run has no normal users, the vault will upload the results to s3. The UI then needs to grab and unpack this for the user
Task Notes

download the resource based on the runid - this may not be available yet while it upload
the open local results button should be download results if its a vault only run
unpack the results as if there were normal results eg COINSTAC_HOME/consortiumID/runID/{results}
the results page may need some work to not crash if the results are not available (before download)

Will need to add state to the run document. "has-downloadable-files", "file-upload-complete"

other things to think about

wiring up a subcription on the run doc to update the button state (results uploading to ready to download?)
the download button should change to the normal 'view files' button if
) 1 its not a vault only run (so normal)
) 2 it is but there are no files (read; run doc says so)
) 3 there are files but they are there locally (you could check the file system orrrr save something in local state, up to you,
the later may run afoul if the user deletes the files etc)


No PRs opened for potential updates in requirements.txt

Hi there,
I hope this is the right place to open this issue. I configured dependabot to keep track of two python packages. See config here.
For example, the requirements.txt (see here) demands cluster_tools>=1.18 , but version 1.29 already exists. Why doesn't dependabot open a PR to bump this version? Explicitly clicking "bump now" does not help either.
Any help is very appreciated!

Dependabot won't update that requirement because it already supports the latest version. If you instead pin it (i.e., cluster_tools==1.18) then Dependabot will create PRs to update it.

Thanks for your fast response! However, I followed your suggestion to use cluster_tools==1.18 and dependabot still doesn't open a new PR to bump to 1.27?

Hmmm, it looks like Dependabot is normalising the name to cluster-tools and that's causing problems. Let me take a loook.

This will be fixed by https://github.com/dependabot/dependabot-core/commit/2aef0dc5aa837c173d698687d9dc302e40376472, but it's going to take me 15 mins to get that deployed. Hand tight, and thanks for reporting this!

Fixed!

Awesome, thank you!


feat: remove deprecations

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (5a7856a) 59.75% compared to head (7ce276b) 60.56%.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1078      +/-   ##
==========================================
+ Coverage   59.75%   60.56%   +0.81%
==========================================
  Files          19       19
  Lines        2609     2574      -35
  Branches      920      917       -3
==========================================
  Hits         1559     1559
+ Misses        710      676      -34
+ Partials      340      339       -1

Files
Coverage Δ

src/glog/logging.h
73.01% <ø> (+7.30%)
:arrow_up:

src/logging.cc
68.47% <ø> (+1.41%)
:arrow_up:


Error when attempting to create an IoT Hub device-id

This is autogenerated. Please review and update as needed.
Describe the bug
Command Name
az iot hub device-identity create Extension Name: azure-cli-iot-ext. Version: 0.8.7.
Errors:
The command failed with an unexpected error. Here is the traceback:
No module named 'azure.mgmt.iothub.iot_hub_client'
Traceback (most recent call last):
  File "/opt/az/lib/python3.10/site-packages/knack/cli.py", line 231, in invoke
    cmd_result = self.invocation.execute(args)
  File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 663, in execute
    raise ex
  File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 726, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
  File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 697, in _run_job
    result = cmd_copy(params)
  File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 333, in __call__
    return self.handler(*args, **kwargs)
  File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
    return op(**command_args)
  File "/home/eustace/.azure/cliextensions/azure-cli-iot-ext/azext_iot/operations/hub.py", line 87, in iot_device_create
    target = get_iot_hub_connection_string(cmd, hub_name, resource_group_name, login=login)
  File "/home/eustace/.azure/cliextensions/azure-cli-iot-ext/azext_iot/common/_azure.py", line 112, in get_iot_hub_connection_string
    client = iot_hub_service_factory(cmd.cli_ctx)
  File "/home/eustace/.azure/cliextensions/azure-cli-iot-ext/azext_iot/_factory.py", line 28, in iot_hub_service_factory
    from azure.mgmt.iothub.iot_hub_client import IotHubClient
ModuleNotFoundError: No module named 'azure.mgmt.iothub.iot_hub_client'

To Reproduce:
Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

Put any pre-requisite steps here...
Following this QSG: https://docs.microsoft.com/en-us/azure/iot-edge/quickstart-linux?view=iotedge-2020-11
Running Az CLI from Edge browser.
Create a resource group in EAST US
Create an IoT Hub in S1 sku, then attempt command below with a new device device name for the created IoT Hub
az iot hub device-identity create --device-id {} --edge-enabled --hub-name {}

Expected Behavior
Returns error above.
Environment Summary
Linux-5.4.0-1080-azure-x86_64-with-glibc2.28 (Cloud Shell), Common Base Linux Delridge (quinault)
Python 3.10.4
Installer: DEB

azure-cli 2.37.0

Extensions:
azure-cli-iot-ext 0.8.7
azure-iot 0.14.1
ai-examples 0.2.5
ssh 1.1.1

Dependencies:
msal 1.18.0b1
azure-mgmt-resource 21.1.0b1

Additional Context

route to CXP team

Issue resolved.
In addition to the 'azure-iot' Azure CLI extension installed, I also had the older 'azure-cli-iot-ext' still installed.  Removing the older version solved the problem.
Closing as resolved.


Update ext-ibmmqqueue/definition.yml

Relevant information
Added a TTL tag to the ext-ibmmqqueue/definition.yml file of P1W. This should remove any tags that aren't present within a week solving the issue of older/historical tags remaining in the NR UI.
Checklist

[X] I've read the guidelines and understand the acceptance criteria.
[X] The value of the attribute marked as identifier will be unique and valid.
[X] I've confirmed that my entity type wasn't already defined. If it is I'm providing an
explanation above.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

From: https://newrelic.slack.com/archives/C01CFRKGPMK/p1654183948714279
Seems the lib we use doesn't like P1W so i would sugest to use P1D
Other than that & the cla signed LGTM


how to use this bot

We are aware that there are mass bans of different magnitudes being experienced by a lot of the bot users with symptoms such as:

Bot being able to login, and catch pokemon but NOT spin forts
Bot being able to login, NOT catch pokemon (vanishing) but able to spin forts
Some accounts not being able to login at all

We are also aware of the getting stuck between 2 forts and getting caught in an infinite loop
If any issues are created relating to the above KNOWN issues they will be closed without reason
Stop!  Use http://jsonlint.com/ to check.
Have you searched to see if there are other issues for the same issue? If so, comment on that issue instead.
Are you running master? We work on the dev branch and then add that functionality to master. Your issue may be fixed on dev and there is no need for this issue, just wait and it will eventually be merged to master.

Expected Behavior
Actual Behavior
Your config.json (remove your credentials and any other private info)
your config here

Steps to Reproduce
Other Information
OS:
Branch: dev or master
Git Commit: (run 'git log -n 1 --pretty=format:"%H"' and paste it here)
Python Version: (run 'python -V' and paste it here)

#3682 - META Server busy or offline issues
#3308 - META Stuck between two forts / Moving to 2 forts at the same time / Constant Loop
#3700 - META Soft bans - Try again spins / Pokemon vanishing
#3746 - META Banwaves - Permanent Bans


Minimum requirements

I would upgrade matplotlib to minimum 1.4.2 (I could not even install the oldest version from pypi on my system). 1.4.2 is already the minimum in the requirements.txt

Actually can we used matplotlib 2.0 actually. I got into trouble to find a proper colormap which is nice.
It required numpy 1.7.1 so it should not be an issue, isn't it?

Another requirement will be to make work the command %matplotlib ipympl for interactive plot which will work in both notebook and jupyter-lab which requires matplotlib 2.0:
https://github.com/matplotlib/jupyter-matplotlib#installation

I was not even aware of that. But we don't want to rely on that and just use inline, right?

it depends if you want the interactive plot by increasing the depth with the widget                                                                                Sent from my phone - sorry to be brief and potential misspell.


xcodebuild support

I've read the readme about Indexing while building technique, so I've got the point. And also I'm aware that swift itself has a limited support for Xcode projects by purpose.
Anyway, is there a way to index [use Xcode index somehow] the code by xcodebuild, to make LSP work with it at least in a less limited way?

Tracked in Apple’s issue tracker as rdar://108104659

@konrad1977 to you for sharing that hack! Didn't knew about that. Could you please point me out to the line in your plug-in that passes target to the lsp server?
Ps: unfortunately I'm not a emacs under, but I guess I would be able to understand the logic behind the code.

-Xswiftc "-sdk" -Xswiftc "xcrun --sdk iphonesimulator --show-sdk-path"
-Xswiftc "-target" -Xswiftc "arm_64-apple-ios16.2-simulator"
This thread is a good starting point:
https://forums.swift.org/t/how-to-use-sourcekit-lsp-with-ios-projects/28273

@konrad1977 Thank you so much for that link. There I was found xcode-build-server utility. Though it's manual it fits my needs.
I would not close this issue though, since I do not consider that utility as a comprehensive solution.

Hey, @konrad1977 sorry to bother you after such a long time.
I've tried just now your approach to make sourcekit-lsp working, and it works great at my side. But there's just one issue I've faced: it's unable to fetch data from any dependency package that I have.
I've looked at a glance into your command that is copying xcodebuild derived data into a SPM .build folder and as far as I can tell there's no dependencies handling here rsync -ua --include='*/' --delete --include='*.swiftmodule' --exclude '*'.
Do I missing something or is it made such by purpose?
Worth to say I'm having no clue where these dependencies are stored within derived data, so for me this is research yet to be done.

SourceKit-LSP exposes its build system integration to external build systems via the build server protocol now, as described in Implementing a BSP server.md. Please file feedback at https://feedbackassistant.apple.com/ for Xcode-specific feature requests, like integration with SourceKit-LSP via BSP.


Release/74

Hey @chuwy - this PR gives a good idea of the various things required to get a release ready...


Risk Tracker: Connect to CalData Portfolio Doc

Goal is to pull in the Risk Tracker from the Portfolio Doc into the DSE Coda doc here https://coda.io/d/Data-Services-Engineering_dY5Oul8-jdi/Risk-Tracking_sughj0ah#_lun1aKPA

Next step on this is to do a test to ensure 2-way sync is working in both directions on the tables. After testing I will close this issue

Tested and confirmed that two-way sync is working. One caveat is if you create a new record in the child document (the DSE team page in this example) you must immediately choose the Project to auto-populate the team name. Otherwise as soon as the sync happens it "hides" the new record due to the filter on Team. I added a callout above the table in the DSE team doc to remind us of that.


Fix for adding subscriptions through the django admin

the admin.py for subscriber tries to do a url reverse for subscriber_callback, but the primary key doesn't exist for new subscriber objects, so it errors out.  these commits just have it return None if the pk doesn't exist.  this shouldn't mess up any existing code, and callback_url isn't called for anything significant until after the object is created, so returning None shouldn't have any side effects.  all tests pass.  if you think this is all good, please merge and push to pypi at your convenience, thanks!

Can you add a regression test?


TypeError: (0 , serverless_express_1.configure) is not a function

Using version @vendia/serverless-express: 4.5.2
Trying to import the "serverlessExpress" module and do something similar as described here:
https://github.com/vendia/serverless-express/blob/mainline/examples/basic-starter-api-gateway-v2/src/lambda-async-setup.js
but, in a TypeScript project.
import logger from 'src/logger';
import ConfigurationProvider from 'src/ConfigurationProvider';
import { configure as serverlessExpress } from '@vendia/serverless-express';
import { APIGatewayEvent, Context } from 'aws-lambda';

let serverlessInstance;

/**
 * See https://github.com/vendia/serverless-express/blob/5c1e0a59e9fdce03cbba3912ba33de8dd9773348/README.md#async-setup-lambda-handler
 */
async function setup(event: APIGatewayEvent, context: Context, configProvider = new ConfigurationProvider()) {
  // Loads the configuration from SSM.
  await configProvider.loadAppConfig();

  const app = (await import('src/app')).default;

  serverlessInstance = serverlessExpress({ app: app() });
  serverlessInstance(event, context);
}

/**
 */
export function handler(event: APIGatewayEvent, context: Context) {
  if (serverlessInstance) {
    return serverlessInstance(event, context);
  }

  return setup(event, context).catch(err => {
    logger.error('Could not start app!', err);
  });
}

This is my tsconfig.json:
{
  "compilerOptions": {
    "allowSyntheticDefaultImports": true,
    "module": "commonjs",
    "outDir": "./dist",
    "noImplicitAny": false,
    "sourceMap": true,
    "skipLibCheck": true,
    "resolveJsonModule": true,
    "jsx": "react",
    "lib": ["es2020", "dom"],
    "target": "es2019",
    "baseUrl": ".",
    "plugins": [
      { "transform": "ts-transformer-imports" }
    ]
  }

Tried to import both with:
import serverlessExpress from '@vendia/serverless-express';

And
import { configure as serverlessExpress } from '@vendia/serverless-express';

Can't get it to work. Always get the following error:
TypeError: (0 , serverless_express_1.configure) is not a function

Or:
TypeError: (0 , serverless_express_1.default) is not a function

I was able to fix it by adding "esModuleInterop": true, to my tsconfig.json. Maybe this helps you too?

Adding "esModuleInterop": true to my tsconfig.json did not work for me. Is there another issue here?


gnucash: Update manifest

version: fix correct version

bin: use bin\gnucash-cli.exe

persist: persist etc\gnucash\environment.local

checkver: find correct version

autoupdate.hash.regex: remove it because scoop already has built-in regex and I have already tested it with auto-pr.ps1

[x] I have read the Contributing Guide.

/verify

/verify


Crash FocusState with two TextFields

If you use focus state with two Textfields, app crashes.
If you replace VStack with Form/List it works, but I dont want use a Form.
import SwiftUI
import SwiftUIBackports

struct FocusState: View {
    var body: some View {
        NavigationLink {
            Demo()
                .backport.navigationTitle("Sign In")
        } label: {
            Text("Focus State")
        }
    }
}

private struct Demo: View {
    enum Field: Hashable {
        case username
        case password
    }

    @State private var username = ""
    @State private var password = ""
    @Backport.FocusState private var focusedField: Field?

    var body: some View {
        VStack {
            TextField("Username", text: $username)
                .backport.focused($focusedField, equals: .username)
                .backport.submitLabel(.next)
                .backport.onSubmit {
                    if !username.isEmpty {
                        focusedField = .password
                    }
                }

            TextField("Password", text: $password)
                .backport.focused($focusedField, equals: .password)
                .backport.submitLabel(.done)
                .backport.onSubmit {
                    signIn()
                }

            Button("Sign In") {
                signIn()

            }
        }
        .padding()

        .onAppear {
            focusedField = .username
        }
    }

    private func signIn() {
        if username.isEmpty {
            focusedField = .username
        } else if password.isEmpty {
            focusedField = .password
        } else {
            focusedField = nil
            print(username, password)
        }
    }
}

Stacktrace
Thread 0 Crashed::  Dispatch queue: com.apple.main-thread
0   libobjc.A.dylib               	       0x105b1ee3c bool objc::DenseMapBase<objc::DenseMap<DisguisedPtr<objc_object const>, unsigned long, (anonymous namespace)::RefcountMapValuePurgeable, objc::DenseMapInfo<DisguisedPtr<objc_object const>>, objc::detail::DenseMapPair<DisguisedPtr<objc_object const>, unsigned long>>, DisguisedPtr<objc_object const>, unsigned long, (anonymous namespace)::RefcountMapValuePurgeable, objc::DenseMapInfo<DisguisedPtr<objc_object const>>, objc::detail::DenseMapPair<DisguisedPtr<objc_object const>, unsigned long>>::LookupBucketFor<DisguisedPtr<objc_object const>>(DisguisedPtr<objc_object const> const&, objc::detail::DenseMapPair<DisguisedPtr<objc_object const>, unsigned long>*&) + 4
1   libobjc.A.dylib               	       0x105b1f9c7 objc::DenseMapBase<objc::DenseMap<DisguisedPtr<objc_object const>, unsigned long, (anonymous namespace)::RefcountMapValuePurgeable, objc::DenseMapInfo<DisguisedPtr<objc_object const>>, objc::detail::DenseMapPair<DisguisedPtr<objc_object const>, unsigned long>>, DisguisedPtr<objc_object const>, unsigned long, (anonymous namespace)::RefcountMapValuePurgeable, objc::DenseMapInfo<DisguisedPtr<objc_object const>>, objc::detail::DenseMapPair<DisguisedPtr<objc_object const>, unsigned long>>::operator[](DisguisedPtr<objc_object const>&&) + 35
2   libobjc.A.dylib               	       0x105b1f97d objc_object::sidetable_retain(bool) + 89
3   BackportsDemo                 	       0x10500b164 Coordinator.responds(to:) + 84
4   BackportsDemo                 	       0x10500b2c7 @objc Coordinator.responds(to:) + 39
5   BackportsDemo                 	       0x10500b221 Coordinator.responds(to:) + 273
6   BackportsDemo                 	       0x10500b2c7 @objc Coordinator.responds(to:) + 39
7   BackportsDemo                 	       0x10500b221 Coordinator.responds(to:) + 273
8   BackportsDemo                 	       0x10500b2c7 @objc Coordinator.responds(to:) + 39
9   BackportsDemo                 	       0x10500b221 Coordinator.responds(to:) + 273
10  BackportsDemo                 	       0x10500b2c7 @objc Coordinator.responds(to:) + 39
11  BackportsDemo                 	       0x10500b221 Coordinator.responds(to:) + 273
12  BackportsDemo                 	       0x10500b2c7 @objc Coordinator.responds(to:) + 39
13  BackportsDemo                 	       0x10500b221 Coordinator.responds(to:) + 273
....
SwiftUIBackportsDemo-main.zip

Interesting, I think this indicates an underlying issue (similar to #62) because they have almost identical configurations for the delegate-dance in order to prevent introducing issues into native SwiftUI code, as well as allow multiple modifiers to co-exist.
@roland-schmitz-ocu worth you taking a look at this as well, I think its related and perhaps even proves my points I made on the other issue.
I'm fairly confident this all worked previously, so I wonder if something was introduced since iOS 16 that just wasn't uncovered until now since a lot of users of this library are here for "backports" therefore not iOS the latest iOS versions.

Thanks @shaps80 for the pointer to this issue. I could reproduce it issue on iOS 15, 16 and 17 and yes you are right, the issue is very similar to #62. Also here the delegate dance happened multiple times which created a cycle in the delegate chain. I could fix it in the same way as in the other two cases by not touching the delegate when it is already been setup. I added a third commit to my PR #63 which 100% fixes also this issue on iOS 15, iOS 16 and iOS 17.

Thanks @shaps80 for the pointer to this issue. I could reproduce it on iOS 15, 16 and 17 and yes you are right, the issue is very similar to #62. Also here the delegate dance happened multiple times which created a cycle in the delegate chain. I could fix it in the same way as in the other two cases by not touching the delegate when it has already been setup. I added a third commit to my PR #63 which 100% fixes also this issue on iOS 15, iOS 16 and iOS 17.

Perfect, I merged already :)


Allow more than 1 search input for monitor

 Issue by agone1
Wednesday Nov 11, 2020 at 16:16 GMT
Originally opened as https://github.com/opendistro-for-elasticsearch/alerting/issues/297

Is your feature request related to a problem? Please describe.
There is a need to have a monitor making two or more search requests.
But when user tries to create a monitor with more than 1 search input (using POST _opendistro/_alerting/monitors API call), he gets the following error message:
"Monitors can only have 1 search input."
Describe the solution you'd like
I assume API limitation is originated from UI limitation, that does not imply more than one search request as monitor input.
I think it would not cause much harm for user experience if this limitation for API would be gone.

 Comment by skkosuri-amzn
Saturday Nov 28, 2020 at 23:18 GMT

Could you please provide more details and use case for this enhancement.

 Comment by agone1
Monday Nov 30, 2020 at 09:40 GMT

The simple use case is when one needs two or more sources of data in order to determine if alert should be triggered on correlated conditions. The description is below.
First use case:
1st index - daily monitoring data.
2nd index - aggregated over more than 3 years data, which contains various stats: average values, median, std_deviation, etc.
Simple threshold value is not enough to determine if alert should be triggered. I need to compare interesting value with statistics for same day of week, month, etc.
Second use case:
1st index - daily monitoring data.
2nd index - all triggered alerts.
Let's imagine than one runs monitor every 5 minutes with 12 hours deep search on first index. If monitor conditions are met the action should be executed. Since monitor runs every 5 minutes there would be 144 actions on the same set of data. In order to suppress excessive actions it would be convinient to check 2nd index, which keeps tracks of triggered alerts.
Both use cases are possible using xpack watcher. It would be great if opendistro monitor provides same functionality.

 Comment by rayliutoronto
Tuesday Jun 01, 2021 at 18:49 GMT

This is useful. Multi-search input result could be placed in ctx.result[].


[v2] Disconnect packet is never/not properly sent

When disconnecting from a server, the server does not get notified of the disconnect

Used the wrong method. The disconnect method of the channel should be used instead of the close method


Cleanup quick project search

What does this PR do and why?
Improvements to the quick project search menu:

debounce search input. wait 250 millis after typing to run search
cleanup timeouts
better cmd + k shortcut performance using "keydown" event
add a "no results" message to empty search results state

Screenshots or screen recordings
If applicable, provide screenshots or screen recordings to demonstrate the changes.
Acceptance checklist

[ ] I have evaluated the Approval Guidelines for this PR.
[ ] I have tested this PR in all supported browsers.
[ ] I have tested this PR in dark mode and light mode (if applicable).

@wraeth-eth i saw that you opened another similar PR. does this accomplish the same thing?

it does - shit sorry I merged the other one 😳
@peripheralist

Current dependencies on/for this PR:

main

PR #2653   👈

This comment was auto-generated by Graphite.


UI Improvements ["YakYak 2.0"]

Hi, fellow Yak-ers!
I'd say it's high time to iterate on the UI design for YakYak (not to mention update a few UX interactions that are highly counter-intuitive).
I opened this thread as a starting point for myself (and others) to share some new design concepts. This won't look like a major overhaul but will begin to develop a new design language for YakYak that moves beyond just "imitating" Hangouts.
A basic overview:

New squared style for message bubbles, unique to YakYak
Improved color distinction between "myself" and the other members of a conversation
New workflow for creating new conversations and managing existing conversations
Color tones that focus on the content (the messages), rather than the elements of the UI
More intuitive conversation view (new messages create their own blocks)
Minor UI and typeface tweaks to polish and improve the overall experience
A new launch screen that makes the loading experience feel "snappier"

I'll be posting mockups in this thread soon (over the next couple of days). Maybe some of the incredible past collaborators (who are more proficient technical developers than myself) can make this come to life once again!
Some requested new features, that could possibly be released along with this update

The option to reply directly to notifications
The ability to add new members to an existing conversation
Link previews (view linked images, videos, and GIFs within the app; website previews)
Image zoom within the YakYak window

looking forward for the mockups and discussion!! :D

Hey again! Here are a few screens that show the updated UI. Many of these updates are not far from how things are in the current build, yet they would make for massive improvements. Take a look and leave some feedback below!
Launch Screen
See this article for an explanation of why we should do this. The example is a bit dated, but the concept holds true. Showing the UI elements will help the launch time to feel quicker because parts of the applications have already loaded.

Conversation View – Rich Links
Here you can see an overview of the updated UI, as well as a concept of link previews.

Conversation View – Alerts
This screen shows an improved view for alerts (such as "this person is calling you" or "you ended a call with this person").

Start New Conversation
This demonstrates the UI when creating a new conversation, either with one person, or with an entire group.

@ariu any desire to take a stab at this one? 😉

@austin-guevara you are good.

launch screen sounds awesome and should be "easy" to implement as the previous iteration of yakyak's start-up was drawing an empty interface and waiting for the connection to hangouts
current "connecting" and "loading contact information" is there just to hide that barebone interface... changing it to have some mockup contacts could replace that with meaningful information


Extension specification of python/system dependencies

DeepForge extensions (such as https://github.com/deepforge-dev/deepforge-keras) should be able to specify dependencies so the deepforge deployment can be updated accordingly on installation.
This is related to #1402 and #1392

As the current plan is to use miniconda (#1402) and there are 3 different types of dependencies in deepforge (server, worker, and job-specific - the former 2 are the most relevant for this comment), extensions could specify their dependencies in conda environment files such as:

environment.server.yml for server dependencies
environment.worker.yml for worker dependencies
alternatively/optionally, environment.yml could be specified for common dependencies across the worker and server


avoid use Array.get/set (JNI) to boost the performance

use java cast instead of JNI(Array.set/.get) to boost the performance in serialize

Thanks!.  Do u have some numbers (microbench) you can show how much perf is gained with your approach?

Merged.  Thanks @wuhongzhi


[BUG] Agent registration not working when overriding a robot uid

Hi!
It looks like @register_agent decorator is not working correctly when override=True. If the uid is already in REGISTERED_AGENTS and override=True we should not return before adding the new agent spec to REGISTERED_AGENTS. We are missing a tab on line 35 in maniskill/agents/registration.py:
Current code:
def _register_agent(agent_cls: type[BaseAgent]):
        if agent_cls.uid in REGISTERED_AGENTS:
            if override:
                logger.warn(f"Overriding registered agent {agent_cls.uid}")
                REGISTERED_AGENTS.pop(agent_cls.uid)
            else:
                logger.warn(
                    f"Agent {agent_cls.uid} is already registered. Skip registration."
                )
            return agent_cls

        REGISTERED_AGENTS[agent_cls.uid] = AgentSpec(
            agent_cls=agent_cls, asset_download_ids=asset_download_ids
        )
        assets.DATA_GROUPS[agent_cls.uid] = asset_download_ids
        return agent_cls

Suggested update (return early only if uid already registered and override=False):
def _register_agent(agent_cls: type[BaseAgent]):
        if agent_cls.uid in REGISTERED_AGENTS:
            if override:
                logger.warn(f"Overriding registered agent {agent_cls.uid}")
                REGISTERED_AGENTS.pop(agent_cls.uid)
            else:
                logger.warn(
                    f"Agent {agent_cls.uid} is already registered. Skip registration."
                )
                return agent_cls

        REGISTERED_AGENTS[agent_cls.uid] = AgentSpec(
            agent_cls=agent_cls, asset_download_ids=asset_download_ids
        )
        assets.DATA_GROUPS[agent_cls.uid] = asset_download_ids
        return agent_cls

Can create a pull request if all is ok

How I came across this: my goal is to override the stiffness and damping of the Panda robot. This requires me to create a custom robot and register it with override=True as below:
import sapien.physx as physx
from mani_skill.agents.controllers import *
from mani_skill.agents.registration import register_agent
from mani_skill.agents.robots import Panda

@register_agent(override=True)
class PandaCustom(Panda):
    # Keep the same id as original Panda to override it
    uid = "panda"

    # Unchanged yet
    arm_stiffness = 1e3
    arm_damping = 1e2
    arm_force_limit = 100

    # Gripper stiffness changed from 1e3
    gripper_stiffness = 500
    # Gripper stiffness changed from 1e2
    gripper_damping = 50
    gripper_force_limit = 100

However, I get RuntimeError: Agent panda not found in the dict of registered agents. If the id is not a typo then make sure to apply the @register_agent() decorator because the panda robot uid is removed from REGISTERED_AGENTS but not added with the new spec.

Thanks for the bug find, could you make a PR for this fix? I'll merge it asap


where is the MarketPlace

I jus make my installation on my server but the marketplace wont load, does someone knows why this coul be, i also have an instal in my MAMP localhost and it loads the marketplace perfectly

could you give us some more information about your server? is it unix based? ms based? what does "wont load" mean? I currently cant test it on my machines as I am migrating them


added mongoc_client_read_command_with_opts

to be able to run collStats and other commands

Looks good! Can you rebase so there's no redundant merge commit?

I did a rebase, but I think I made it worse.  let me know if there's a simple command to this easily.  thank you.

good?


Update twitter python demo for 1.0.0

Update the twitter-sentiment-processor python demo to use runtime 1.0.0 and python-sdk 1.0.0

@wcs1only - Thanks so much for this!


Feature Request: Add options for controlling max swipe count with scroll/swipe direction, and setting scroll speed in Selenide Appium

The problem
Feature 01: Pass MaxSwipeCount and PointHeightPercentage together with scroll direction
In current implementation of Selenide Appium scroll/swipe gestures, we have the option to combine the scroll direction with either maxSwipeCount or the scroll Coordinates but not both. For example, see below the scroll down code:
$x("xpathLocator").scroll(AppiumScrollOptions.down());
$x("xpathLocator").scroll(AppiumScrollOptions.down(float topPointHeightPercent, float bottomPointHeightPercent));
$x("xpathLocator").scroll(AppiumScrollOptions.down(int maxSwipeCounts));
$x("xpathLocator").scroll(AppiumScrollOptions.with(ScrollDirection.DOWN, int maxSwipeCounts));
$x("xpathLocator").scroll(AppiumScrollOptions.with(ScrollDirection.DOWN, float topPointHeightPercent, float bottomPointHeightPercent));

I would like to have the ability to pass maxSwipeCount also as an additional parameter together with scroll direction, and scrollCoordinates. This should look like this with current implementation:-
$x("xpathLocator").scroll(AppiumScrollOptions.with(ScrollDirection scrollDirection, float topPointHeightPercent, float bottomPointHeightPercent, int maxSwipeCounts));

In absence of this feature, selenide keeps on scrolling for DEFAULT_MAX_SWIPE_COUNTS i.e. 30 which is not required in most cases.
Feature 02: Control over pointer move speed while scrolling to swipe/scroll with custom set speed
Additionally, it would rather be more beneficial to allow user to pass the pointer move duration also as an optional parameter so the speed of scroll/swipe can be controlled by user. Currently it is hard-coded to 200ms which may be too fast or slow for some users:
https://github.com/selenide/selenide/blob/main/modules/appium/src/main/java/com/codeborne/selenide/appium/commands/AppiumScrollTo.java#L120
An additional parameter with an overloaded method should look like this:
$x("xpathLocator").scroll(AppiumScrollOptions.with(ScrollDirection direction, float topPointHeightPercent, float bottomPointHeightPercent, int maxSwipeCounts, Duration pointerMoveDuration));

This will allow user to scroll fast or slow with defined pointer speed depending on the nature of their application. Pointermove duration should also be able to pass only with swipe or max swipe count to have as much flexibility for the user.
Details
This allows the user to have more control and flexibility over scrolling. Ofcourse, we can have custom command implementation but i feel these two should be part of core selenide appium to have as much flexibility for the user while working with selenide ready-to-use scroll commands.
Tell us about your environment

Selenide-Appium Version: 7.1.0

@itkhanz - Good suggestion. Do you want to create a PR to implement this feature?


limit_req block the correct operation of Blazor WebAssembly

Blazor WebAssembly fetch from the server more than 50 files at loading. If the limit_req  zone=one burst=10 nodelay; is set, like the example in this doc, will block the correct operation of Blazor WebAssembly, since Nginx will not deliver the files to the browser. This limit should be 60 at least (tested with 3.2.0-rc1.20223.4).

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: dcdd2c35-bf2b-2673-d150-303a8aaa54d2
Version Independent ID: 815fe034-c9fc-aee3-a8b5-f2e860b30302
Content: Host ASP.NET Core on Linux with Nginx
Content Source: aspnetcore/host-and-deploy/linux-nginx.md
Product: aspnet-core
Technology: aspnetcore-hostdeploy
GitHub Login: @Rick-Anderson
Microsoft Alias: riande

Hello @OrihuelaConde ... Stand-by. These issues are usually triaged on Fridays. We'll find out if we should provide that information in the topic. ...... and perhaps in the Blazor Host and Deploy node, too, at ...
https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/blazor/webassembly?view=aspnetcore-3.1#nginx

@guardrex, the issue is not in the "Standalone deployment", like you mentioned. It is in "Hosted deployment with ASP.NET Core", because it guide you to "Host and deploy ASP.NET Core" and then to the specific doc for Nginx.

Thanks ... I've updated my note :point_up:.


feat: Zoom to select area

Implements #24
Upon selecting an area, we're not just passing the areaId but the whole Feature; in the area-select action we're extracting the area ID and bounding box and put both in store.
Other changes

Adds @turf/bbox to calculated the bounding box from the geometry
Add effect hook to globe to set the bounding box on the map

Refactoring
I cleaned up the xstate related code:

Changed the naming of actions and events so events and actions are a bit more differentiable
Moved the context, events, and actions types to its on module to make the machine module a bit more readable
Removed anything related to loading the country geojson

Fit area to bounds
I kept the area GeoJSON code because we cannot rely on the geometry returned by queryRenderedFeature to get the area bounding box (bbox). If the vector tile doesn't contain the full geometry, it will zoom to the tile bounds:
I suggest keeping the current implementation or open a ticket to track this until we have an API route to query the area bounds.

But we’ve never used the GeoJSON in the actual app, which is why I’ve removed it. It feels wrong to keep all this data in memory just so we can query the geometry occasionally.
I tried to solve this but ran into my limits of my understanding how the state machine with xstate should be configured.
Here’s how I approached it:

I extended the query in area/[areaId]/page so it returns the geometry bbox for the area.
From the page I then want to trigger an xstate event that contains the bbox, and the subsequent action would update the state.

There will be two states related to this scenario:

Country is selected
Map bounding box is set

Now “Country is selected” is a state that we can have with or without the map bounding box. Likewise the map bounding box can be set when now country is selected; for example, when a route or node is selected. I feels like we can have several states in parallel, which is contrary to how I understand state machines. In a state machine, there should one state at a time. So how can we model this in the state machine?

Highlighted area by filtering the id
I made a change to use the area ID instead of the queried feature to draw the area outline. Due to the same issue mentioned above, the highlighted area would be the tile bbox if the area is not fully visible. By using the area ID, the full feature is outlined.
Highlighted area render performance
I thought we had solved this issue in #37, but it still persists. I don't think this is a blocker for this PR, so we can open a ticket for it.

The issue regressed because of the update you’ve made. Filtering across the whole area dataset is too expensive and should be avoided, which is why I have removed it based on the initial review.  I agree, this is a bug that we need to fix, but it’s a separate issue unrelated to the work we’re discussing here. I’ve opened a ticket.

XState conventions
I often rely on the machine diagram to make changes, it is much easier that editing the machine definition. When the machine gets larger, it becomes difficult to modify it by just looking at the code, but relatively easy to do through the XState editor, which also has a VSCode extension.
My preference is to keep the more verbose convention as it looks more readable in the diagram than the concise version.

From the labels you suggested, I can never tell whether I’m looking at an action or an event or a state. I couldn’t get the visual editor to work, which is why I made the change. The naming I proposed, is a lot clearer as it follows a simple convention: type:target:change so you can quickly process what is happening and what part of the state is affected. If we introduce more actions/events it will be harder to make sense of the naming written in verbose prose. I don’t agree that he naming in prose is easier to process in the visual editor once you understand my convention. To me, there’s no difference between Area is selected and area:selected for the states.
Also, I’d argue there’s a long-standing convention in software development to use unambiguous names for identifiers instead of conversational English.

XState VSCode Extension
To my surprise, this extension is not working. There are a couple of open issues about this on their GitHub repo (statelyai/xstate-tools#325 and statelyai/xstate-tools#468). In our case, it’s not working because the machine is under a folder with parentheses in its path (src/app/(home)/globe/state/index.tsx).
Editing the machine without the diagram editor is quite unpleasant. I'm wondering if we could move it to a dedicated folder like src/app/machines/globe. We could also consider moving the components of src/app/(home) back to the root level (src/app/index.tsx), since we now know the app won't have subpages that don't include the globe.

We can make this change, but it’s not related to the ticket we’re discussing here. I think, we should tackle that in a separate PR.

@oliverroick, your arguments make sense. We should use a more structured convention for naming the elements in the machine.
I found a solution for the hovering issue; it should now be snappy and highlight the full geometry. It involved using map.setFeatureState, as shown in this example from the Mapbox docs, and adding an integer ID to the MVT features to make the map query more performant:

I also added the following changes:

Renamed the globe component to map and moved it to the components folder, fixing the diagram viewer issue.
Moved event and action types to separate files for easier access and maintainability
Changed the mousemove event handler to trigger events in XState. This is a pattern I used in Pearl to keep the map component as 'dumb' as possible, making it easier to handle map effects at different states of the app

Next steps should be:

Add a new route that returns the area and its bounding box. This is required to avoid using the vector tile feature bbox, which is not always fully loaded
Apply the correct bounding box on click
Apply the correct bounding box when accessing the route /area/:id

@wrynearson, as discussed, I'll add changes to this while Oliver is out and will ask for your review when it’s ready.

@wrynearson this is ready for review, here is a screen capture:

How to test:

Start the server and visit http://localhost:3000
Move the mouse point over the map, the hovered area should have a different style and a tooltip is shown
Click on a area, the map should zoom to it
Click on another area, the map should zoom to it

Know issues: the side panel is not displayed yet and it is not possible yet to visit an area URL to get an area view. This is a change we should address on a separated PR.

@wrynearson thanks for the review! I increased the padding to give more breathing room to the selected area. Let's revisit this later when there are more elements on the page, as we'll probably need to adjust further.


Is this repo still in maintenance?

It's a surprise as I find that my favorite RPG in DOTA2 uses C# 😄

No, original maintainer died, and we migrated to using different language and framework (Python/FastAPI)

Sorry to hear that. What a pity!


[FEATURE] Experiment Logs Do Not Contain Logs from Chatting

The Experiment logs complete once the experiment finishes.  This means issues that occur when chatting with the finished model will not be captured in the logs.
The chat logs should be appended to the experiment logs zip file.

We can maybe add an expandable log view to the chat tab that shows the same information as in the terminal.
Not sure if we really want to store all past chats in a persistent manner.

Which kinds of issues are we expecting to log here?
I would not add any chats to the fine-tune training logs of LLM by default. Are you looking for an additional feature for exporting chat logs?

Closing as not relevant for fine-tuning


Stop the wp-signup.php MU

Hi there, is it possible prevent the wp-signup.php default redirect with the MU Wordpress? I just want the singlesite registration. Thank u :)

Having the same issue! i found the following code that prevents it. From my understanding the first function will get the url of the blog the second will return you to the correct page.
add_filter( 'wp_signup_location', 'mbdr_current_blog_home' );
function mbdr_current_blog_home($location) {
    return home_url('/register/');
}

add_action('before_signup_header', 'mbdr_redirect_signup_home');
function mbdr_redirect_signup_home() {
    wp_redirect( home_url('/register/') );
    exit();
}

The issue i'm facing now is that the errors and validation for signing up doesn't work.
Also found this plugin instead. Probably this can be partly forked into existing code here. https://wordpress.org/plugins/network-subsite-user-registration/


Validate content on flutter.io is accurate and up-to-date

Tracking item for work we plan to do for M2.
We want to sweep through the pages on flutter.io and double check that they are still accurate w.r.t. snippets, concepts, and explanations. We hope to catch major inaccuracies.

Perhaps a plan here is to enumerate each article, assign a technical reviewer. We'll ask a technical reviewer to vet the technical concepts and explanations. It's not a copy edit, but a technical verification. Based on feedback, we'll triage: tweak, rewrite, hide.

Important but doens't need to be tracked separately as a metabug.