Subscription intermittently stops delivering messages

Hi, similar issue to #234, but with slightly different behaviour, so I'm raising a new issue.  One of the subscriptions on a specific channel causes all of the others to stall and results in back pressure on the publication.
Here is AeronStat output for stream -1762877036.  Note registrationId 1167 is show a sub-pos much smaller than the others.
AeronStat
983:               65,184 - pub-lmt:<PHONE_NUMBER>94 -1762877036 aeron:udp?group=<IP_ADDRESS>:10000|interface=<IP_ADDRESS>
984:               32,416 - snd-pos:<PHONE_NUMBER>94 -1762877036 aeron:udp?group=<IP_ADDRESS>:10000|interface=<IP_ADDRESS>
1099:               29,920 - sub-pos: 1428 880428294 -1762877036 aeron:udp?group=<IP_ADDRESS>:10000|interface=<IP_ADDRESS>
1134:               29,920 - sub-pos:<PHONE_NUMBER>94 -1762877036 aeron:udp?group=<IP_ADDRESS>:10000|interface=<IP_ADDRESS>
1135:               29,920 - sub-pos:<PHONE_NUMBER>94 -1762877036 aeron:udp?group=<IP_ADDRESS>:10000|interface=<IP_ADDRESS>
1136:               29,920 - sub-pos:<PHONE_NUMBER>94 -1762877036 aeron:udp?group=<IP_ADDRESS>:10000|interface=<IP_ADDRESS>
1137:               29,920 - sub-pos:<PHONE_NUMBER>94 -1762877036 aeron:udp?group=<IP_ADDRESS>:10000|interface=<IP_ADDRESS>
1140:               29,920 - sub-pos:<PHONE_NUMBER>94 -1762877036 aeron:udp?group=<IP_ADDRESS>:10000|interface=<IP_ADDRESS>
1141:               29,920 - sub-pos: 76 880428294 -1762877036 aeron:udp?group=<IP_ADDRESS>:10000|interface=<IP_ADDRESS>
1227:               29,920 - sub-pos: 1332 880428294 -1762877036 aeron:udp?group=<IP_ADDRESS>:10000|interface=<IP_ADDRESS>
1435:               29,920 - sub-pos:<PHONE_NUMBER>94 -1762877036 aeron:udp?group=<IP_ADDRESS>:10000|interface=<IP_ADDRESS>
1436:               29,920 - sub-pos:<PHONE_NUMBER>94 -1762877036 aeron:udp?group=<IP_ADDRESS>:10000|interface=<IP_ADDRESS>
1437:                  384 - sub-pos: 1167 880428294 -1762877036 aeron:udp?group=<IP_ADDRESS>:10000|interface=<IP_ADDRESS>
1438:               32,416 - rcv-hwm: 1270 880428294 -1762877036 aeron:udp?group=<IP_ADDRESS>:10000|interface=<IP_ADDRESS>
1464:               29,920 - sub-pos: 1293 880428294 -1762877036 aeron:udp?group=<IP_ADDRESS>:10000|interface=<IP_ADDRESS>
1712:               29,920 - sub-pos: 1962 880428294 -1762877036 aeron:udp?group=<IP_ADDRESS>:10000|interface=<IP_ADDRESS>
1761:               29,920 - sub-pos: 1671 880428294 -1762877036 aeron:udp?group=<IP_ADDRESS>:10000|interface=<IP_ADDRESS>
1973:               29,920 - sub-pos: 1836 880428294 -1762877036 aeron:udp?group=<IP_ADDRESS>:10000|interface=<IP_ADDRESS>
2039:               29,920 - sub-pos: 1928 880428294 -1762877036 aeron:udp?group=<IP_ADDRESS>:10000|interface=<IP_ADDRESS>
2060:               29,920 - sub-pos: 2038 880428294 -1762877036 aeron:udp?group=<IP_ADDRESS>:10000|interface=<IP_ADDRESS>

I have some custom monitoring of my handler so that I can track if this subscription is still being polled.  You can see the streamId and registrationId matches the problematic subscription and pollCount is increasing, which means I'm confident our polling code hasn't crashed.
$>bean com.lmax:name=tfx-group__InstrumentNotificationServicePublic,service=cfdx,type=AeronReceiver
#bean is set to com.lmax:name=tfx-group__InstrumentNotificationServicePublic,service=cfdx,type=AeronReceiver
$>get *
#mbean = com.lmax:name=tfx-group__InstrumentNotificationServicePublic,service=cfdx,type=AeronReceiver:
RegistrationId = 1167;
TopicName = tfx-group::InstrumentNotificationServicePublic;
StreamId = -1762877036;
PollCount = 441575;
Uri = aeron:udp?group=<IP_ADDRESS>:10000|interface=<IP_ADDRESS>;

$>get *
#mbean = com.lmax:name=tfx-group__InstrumentNotificationServicePublic,service=cfdx,type=AeronReceiver:
RegistrationId = 1167;
TopicName = tfx-group::InstrumentNotificationServicePublic;
StreamId = -1762877036;
PollCount = 691272;
Uri = aeron:udp?group=<IP_ADDRESS>:10000|interface=<IP_ADDRESS>;

Polling code:
    public int poll()
    {
        final int poll = subscription.poll(handler, 20);
        ++pollCount;
        return poll;
    }

Looking at the log inspect for the appropriate subscription image show that aeron thinks the term is unused and dirty.  I.e. the first show no messages, but has not been cleaned the remaining 2 terms are all zeros.  The LogInspector suggests that there are no messages in that term.
 ======================================================================
 Fri Jun 17 03:38:17 UTC 2016 Inspection dump for /dev/shm/aeron-tradex/images/UDP-ac1d0101-0-efc10001-10000-347A4506-96ECA194-4F6.logbuffer
 ======================================================================
 Time of last SM: Thu Jan 01 00:00:00 UTC 1970
 Initial term id: -1970406852
    Active index: 0
     Term length: 65536
      MTU length: 4096

 default Data Header{frame_length=0 version=0 flags=11000000 type=1 frame_length=0 term_offset=0 session_id=880428294 stream_id=-1762877036 term_id=-1970406852 reserved_value=0}

 Index 0 Term Meta Data status=CLEAN termOffset=0 termId=0
 Index 1 Term Meta Data status=CLEAN termOffset=0 termId=0
 Index 2 Term Meta Data status=CLEAN termOffset=0 termId=0
 %n======================================================================
 Index 0 Term Data

 Data Header{frame_length=0 version=0 flags=00000000 type=0 frame_length=0 term_offset=0 session_id=0 stream_id=0 term_id=0 reserved_value=0}
<PHONE_NUMBER>0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
<PHONE_NUMBER>0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
<PHONE_NUMBER>0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
<PHONE_NUMBER>0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
<PHONE_NUMBER>0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
 00002C00000000C001008001000006457A3494A1EC963CFA8D8A0000000000000000000000080000000000000000000000000000000000000000000000000000000020000000
 00C00100C001000006457A3494A1EC963CFA8D8A00000000000000002C00000000C00100E001000006457A3494A1EC963CFA8D8A000000000000000000000008000000000000
 ...

It looks like the log is in its initial term and has been half filled by the position counters. However the termOffset suggests the tail has not been incremented which is just wrong. I'd like a closer look at that log buffer. Can you send it?

@mikeb01 Are the client subscribers in the same process or different processes?

The min of the sub-pos will be the position that is set at 384. That will be the consumption position in the SMs and from which the window will begin.
With the other sub-pos values and if that Subscription is being polled, the only way I can see that position not incrementing is if there is loss that isn't being repaired right at the point and the other subscriptions started right after the loss point.
The internal contiguous position, rebuildPosition, isn't tracked by a counter. This makes me think it might be good to make it a counter along with the rcv-hwm.
@mjpt777 Images are a little different than publication buffers. The tail isn't really used. Instead, the term reading is totally controlled by the presence of unfilled framelength values (or reaching capacity of the term while reading). The nature of Images might make the log inspector output seem strange. In any case, a closer look at the Image logbuffer might give us an idea of what has happened.

@tmontgomery You are right. I forgot that Images do not have the tail updated. They don't need it.
+1 I think it would be useful to have a rcv-pos counter that is the rebuild position for monitoring.

I'll add the rcv-pos counter. It would be quite useful.
Nice thing is with the image logbuffer and log inspector and the stats, we could look at what is likely to have occurred. Or at least what might have occurred.

I've attached the publication and the image data (aeron-data.zip) for the specified stream.  The subscriptions and publication are in separate processes.  The media driver is also in separate process too.

Is there any other information that would be useful for debugging this issue.  I can do some re-testing on Friday.

I think it would be useful to record the joining position in the log buffer for images for later tracking.

@mikeb01 Do you have a screen grab of the system counters from this time? I see you have some from the positions.

Let me add some more stats for tracking so that when run again, there is more information. Will look at that later today.

Having the Aeron data files sit around after shut down is very useful...
20:33:27 - Aeron Stat
=========================
  0:           88,291,136 - Bytes Sent
  1:          899,057,984 - Bytes received
  2:                    0 - Failed offers to ReceiverProxy
  3:                    0 - Failed offers to SenderProxy
  4:                    0 - Failed offers to DriverConductorProxy
  5:              266,102 - NAKs sent
  6:              260,676 - NAKs received
  7:           12,875,602 - Status Messages sent
  8:           12,805,048 - Status Messages received
  9:           25,018,505 - Heartbeats sent
 10:           24,203,284 - Heartbeats received
 11:              100,482 - Retransmits sent
 12:                4,978 - Flow control under runs
 13:               55,875 - Flow control over runs
 14:                    0 - Invalid packets
 15:                    0 - Errors
 16:                    0 - Short sends
 17:              493,189 - Client keep-alives
 18:                  439 - Sender flow control limits applied
 19:                    0 - Unblocked Publications
 20:                    0 - Unblocked Control Commands
 21:                    0 - Possible TTL Asymmetry
 22:                    1 - send-channel: aeron:udp?group=<IP_ADDRESS>:10000|interface=<IP_ADDRESS>

Thanks Mike. As an observation that machine is seeing a lot of loss. It might be worth increasing the size of the socket buffers to see if it comes more under control by compensating for the congestion once this gets resolved.

@tmontgomery Does that ratio of NAKs to retransmits look fishy to you?

Possibly. It definitely means that less than 40% of NAKs are being answered. It could be because the retransmit is not found when scanning, though.

@mikeb01 The beginning of the term being zeros is very odd. If it happens again can you grab another image of the publication or the source publication to see if they are the same?

I plan to do some more testing tomorrow, so will do.  I can replicate this fairly easily.  Around half of the ~60 application instances in CI run into this problem.

@mikeb01 Can you post the config options you have applied?

Our setup is below.  The OS buffer memory max size is probably less than is specified.  As an aside, I wonder if it would be worthwhile having the configuration options stored as counters so they could be retrieved via AeronStat.
        System.setProperty("aeron.event.log", "CMD_IN_ADD_PUBLICATION,CMD_IN_ADD_SUBSCRIPTION");
        System.setProperty("aeron.socket.so_sndbuf", "2097152");
        System.setProperty("aeron.socket.so_rcvbuf", "2097152");
        System.setProperty("aeron.rcv.buffer.length", "16384");
        System.setProperty("agrona.disable.bounds.checks", "true");
        System.setProperty("aeron.term.buffer.sparse.file", "true");

        final int termBufferLength = 64 * 1024;

        final MediaDriver.Context ctx = new MediaDriver.Context()
                .threadingMode(ThreadingMode.SHARED)
                .sharedIdleStrategy(SlowIdleStrategy.INSTANCE)
                .imageLivenessTimeoutNs(10_000_000_000L)
                .publicationTermBufferLength(termBufferLength)
                .initialWindowLength(termBufferLength / 2);

I think it makes sense to have the properties stored in the C'n'C file as a section that can be read with a tool. They are not really counters.

I've run the 0.9.9-SNAPSHOT version through our CI system and I get the same sort of failure.
AeronStat:
02:35:40 - Aeron Stat
=========================
  0:           94,747,936 - Bytes sent
  1:           97,622,944 - Bytes received
  2:                    0 - Failed offers to ReceiverProxy
  3:                    0 - Failed offers to SenderProxy
  4:                    0 - Failed offers to DriverConductorProxy
  5:               37,100 - NAKs sent
  6:               37,100 - NAKs received
  7:            1,393,407 - Status Messages sent
  8:            1,393,634 - Status Messages received
  9:            2,696,965 - Heartbeats sent
 10:            2,472,784 - Heartbeats received
 11:               11,251 - Retransmits sent
 12:                1,158 - Flow control under runs
 13:                    0 - Flow control over runs
 14:                    0 - Invalid packets
 15:                    0 - Errors
 16:                    0 - Short sends
 17:               55,230 - Client keep-alives
 18:                  433 - Sender flow control limits applied
 19:                    0 - Unblocked Publications
 20:                    0 - Unblocked Control Commands
 21:                    0 - Possible TTL Asymmetry

1306:               65,792 - pub-lmt:<PHONE_NUMBER>93 -1762877036 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS>
 1307:               33,024 - snd-pos:<PHONE_NUMBER>93 -1762877036 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS>
 1480:               33,024 - sub-pos: 76 926135193 -1762877036 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS> @576
 1481:               33,024 - sub-pos:<PHONE_NUMBER>93 -1762877036 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS> @576
 1482:               33,024 - sub-pos:<PHONE_NUMBER>93 -1762877036 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS> @576
 1483:               33,024 - sub-pos:<PHONE_NUMBER>93 -1762877036 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS> @576
 1484:               33,024 - sub-pos:<PHONE_NUMBER>93 -1762877036 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS> @576
 1485:               33,024 - sub-pos:<PHONE_NUMBER>93 -1762877036 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS> @576
 1486:               33,024 - sub-pos:<PHONE_NUMBER>93 -1762877036 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS> @576
 1487:               33,024 - sub-pos:<PHONE_NUMBER>93 -1762877036 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS> @576
 1488:                  576 - sub-pos: 1136 926135193 -1762877036 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS> @576
 1489:               33,024 - rcv-hwm: 1182 926135193 -1762877036 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS>
 1490:               33,024 - rcv-pos: 1182 926135193 -1762877036 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS>
 1513:               33,024 - sub-pos: 1207 926135193 -1762877036 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS> @24,160
 1578:               33,024 - sub-pos: 1261 926135193 -1762877036 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS> @24,256
 1906:               33,024 - sub-pos: 1492 926135193 -1762877036 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS> @33,024
 2090:               33,024 - sub-pos: 1639 926135193 -1762877036 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS> @33,024
 2217:               33,024 - sub-pos: 1745 926135193 -1762877036 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS> @33,024
 2332:               33,024 - sub-pos: 1848 926135193 -1762877036 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS> @33,024
 2360:               33,024 - sub-pos: 1875 926135193 -1762877036 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS> @33,024
 2391:               33,024 - sub-pos: 1913 926135193 -1762877036 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS> @33,024

The log buffer data (aeron-buffer.zip) is also available.
Is there any other data that would be useful to capture?

Thanks, Mike.
Looks like the everything is normal, actually. The joining position is 576. Which is where registrationId 1136 is located. Others have this same joining position and moved on just fine. The rcv-pos is equal to rcv-hwm is equal to snd-pos. So, there is no loss detected from 576 onward. Which means that delivery should have continued just fine.
It would be good to know if that subscription has continued to poll.
I'll take a look at extending LogInspector to handle logbuffers that start with zero as this one will as the lowest joining position is 576.
Something interesting is that the joining position of several subscriptions is the current snd-pos, etc. Nothing wrong with that, really. Just interesting.
As the lowest join position is 576, this means that there is/was a separate media driver for this stream or that a previous run on this media driver used this stream and moved it from 0 to 576.

I've added an option to LogInspector to scan over zeroes to take a look at the Image file.
There is a frame at offset 576 in Term index 0. Which is the active index. So, there is data at the position.
So, given that there is data at the position, the only ways I can see poll not delivering are:

poll is not called. Unlikely given what you have monitored.
this image isn't the one being polled somehow. Maybe weirdness of mapping?

One thing that might be useful would be the sequence of commands that show Available/Unavailable images. To help with the later above.
Something I noticed is that some messages at the Publication are 0 length. While not an issue, I wanted to mention it in case it isn't intentional.

It would be good to know the sequence of pub/sub adds/removes here as well as the available/unavailable images. It looks from the registrationIds and join positions that there is a good bit of add/remove of subs and pubs around this streamId going on.

@mikeb01 are you checking the return value from Subscription::poll?

Thanks @tmontgomery.  The subscription should still be polled.  I forgot to capture it on the latest run, but our code hasn't changed since the previous so I would expect it to be.
I'm assuming that the information around pub/sub adds/removes as well as the available/unavailable images is available via logging.  I'll have a look at getting that switched on and do another run.

I'll look to capture the following:
CMD_IN_ADD_PUBLICATION
CMD_IN_REMOVE_PUBLICATION
CMD_IN_ADD_SUBSCRIPTION
CMD_IN_REMOVE_SUBSCRIPTION
CMD_OUT_PUBLICATION_READY
CMD_OUT_AVAILABLE_IMAGE
CMD_OUT_ON_UNAVAILABLE_IMAGE

Thanks, @mikeb01 Those should work fine. Remember that the logging now uses the aeron-agent if you haven't used it in a while.

Just look at that now.  Is there a reason why you configure the agent using a system property?  The typical approach for configuring the java would be:
-javaagent:agent.jar=aeron.event.log=admin,FRAME_IN

Simply didn't get around to doing it.

I have a suspicion that something might be corrupting the log. I've pushed a new feature that allows images to be mapped read-only which should catch this. It is now the default and can be changed from the client context.

@mikeb01 is it possible that an exception is being seen on the fragment handler for that stream?
Also, are the 0 length messages intentional?

I'll do some more digging on Friday.  I think the zero length messages are intentional, but I will need to double check.  I'll also run through the code to make sure there are no silent exceptions occurring.

The only other thing that is suspicious is that the joining positions are non-0. This can only really happen if there is another Media Driver and subscription that is listening and/or a different Flow Control strategy or channel endpoint.

Are the joining positions, the values after the @ in AeronStat?

Yes

I don't think that is too big a deal.  With out system it is possible that a subscription could occur after publication has been created and messages have already been sent.  In that case our late join capability should kick in to provide redeliver messages where enabled and in other cases the message loss is inconsequential.
I'm assuming that for a multicast channel, a subscription can join a stream at any point and receive messages from that point onward.  Messages delivered prior to joining won't be seen by the new subscription, but for messages sent after the new subscription will be included in the set of Subscriptions that can create back-pressure.

@mikeb01 correct. Joining in is fine. And gives the semantics you mention.
But to get into that state, another media driver must be around... or flow control strategy, etc.

I'll have to compare to what I see locally.  I am running ~60 instances of the application each with a separate media driver on separate virtual machines.  Each is using a dummy interface to keep all of the multicast traffic local to that instance.  However they will be sharing multicast addresses so I'll make sure that multicast traffic is not leaking across somehow.

I don't believe this should be a problem. Just something I have noticed. While trying to simulate what you are seeing, I have been trying to make things happen, etc.

@mikeb01 is there any exceptions in the Aeron client error handler for any of the clients? or any evidence of FD or VM exhaustion in the applications?
@mjpt777 and I are talking about possibly the mappings being 0 due to VM or FD exhaustion somehow.

Checking my acronyms, FD == file descriptor and VM == virtual memory?  I'll double check the file descriptor ulimits, but they should be okay.  As for memory, we have 16GB allocated to /dev/shm and during a run have used <1GB of that.  I'll grab some more data when I do some more runs through tomorrow.

correct on acronyms, yes.

Another run through today.  File descriptors and memory looks okay
[barkerm@lddev-build-sas018 ~]$ sysctl fs.file-nr
fs.file-nr = 22560	0	3265313
[barkerm@lddev-build-sas018 ~]$ pgrep -f name=aeron-media-driver
22694
[barkerm@lddev-build-sas018 ~]$ lsof -a -p 22694 |  wc -l
5
[barkerm@lddev-build-sas018 ~]$ cat /proc/meminfo
MemTotal:       32878504 kB
MemFree:        16942308 kB
Buffers:          329116 kB
Cached:          3647380 kB
...

For this run it was stream -215825341 that failed.
AeronStat:
01:35:57 - Aeron Stat
=========================
  0:          101,125,472 - Bytes sent
  1:          103,091,488 - Bytes received
  2:                    0 - Failed offers to ReceiverProxy
  3:                    0 - Failed offers to SenderProxy
  4:                    0 - Failed offers to DriverConductorProxy
  5:               14,199 - NAKs sent
  6:               14,199 - NAKs received
  7:            1,475,304 - Status Messages sent
  8:            1,475,030 - Status Messages received
  9:            2,870,423 - Heartbeats sent
 10:            2,781,648 - Heartbeats received
 11:                4,989 - Retransmits sent
 12:                  831 - Flow control under runs
 13:                    0 - Flow control over runs
 14:                    0 - Invalid packets
 15:                    0 - Errors
 16:                    0 - Short sends
 17:               58,353 - Client keep-alives
 18:                  433 - Sender flow control limits applied
 19:                    0 - Unblocked Publications
 20:                    0 - Unblocked Control Commands
 21:                    0 - Possible TTL Asymmetry

78:               64,928 - pub-lmt: 48<PHONE_NUMBER> -215825341 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS>
79:               32,160 - snd-pos: 48<PHONE_NUMBER> -215825341 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS>
1285:               32,160 - rcv-hwm: 1104<PHONE_NUMBER> -215825341 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS>
1286:               32,160 - rcv-pos: 1104<PHONE_NUMBER> -215825341 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS>
1289:               32,160 - sub-pos: 1017<PHONE_NUMBER> -215825341 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS> @0
1327:                  288 - sub-pos: 1127<PHONE_NUMBER> -215825341 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS> @288
1873:               32,160 - sub-pos: 1374<PHONE_NUMBER> -215825341 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS> @6,848
2115:               32,160 - sub-pos: 1583<PHONE_NUMBER> -215825341 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS> @32,160
2211:               32,160 - sub-pos: 1658<PHONE_NUMBER> -215825341 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS> @32,160

I've included the publication and image files, along with the event log in
aeron-data.zip.
Interestingly the image does go unavailable during the run, then becomes available again.
[3225040.590694] CMD_OUT_AVAILABLE_IMAGE [150/150]:<PHONE_NUMBER>:-215825341 [0:1470:1017] "<IP_ADDRESS>:32933" [1018]
[3225050.909230] CMD_OUT_ON_UNAVAILABLE_IMAGE [73/73]: aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS> -215825341 [1018]
[3225051.761203] CMD_OUT_AVAILABLE_IMAGE [150/150]:<PHONE_NUMBER>:-215825341 [0:1289:1017] "<IP_ADDRESS>:32933" [1104]
[3225054.078744] CMD_OUT_AVAILABLE_IMAGE [150/150]:<PHONE_NUMBER>:-215825341 [0:1329:1127] "<IP_ADDRESS>:32933" [1018]
[3225054.078805] CMD_OUT_AVAILABLE_IMAGE [150/150]:<PHONE_NUMBER>:-215825341 [0:1327:1127] "<IP_ADDRESS>:32933" [1104]
[3225079.567818] CMD_OUT_AVAILABLE_IMAGE [150/150]:<PHONE_NUMBER>:-215825341 [0:1873:1374] "<IP_ADDRESS>:32933" [1104]
[3225114.814036] CMD_OUT_AVAILABLE_IMAGE [150/150]:<PHONE_NUMBER>:-215825341 [0:2115:1583] "<IP_ADDRESS>:32933" [1104]
[3225123.918616] CMD_OUT_AVAILABLE_IMAGE [150/150]:<PHONE_NUMBER>:-215825341 [0:2211:1658] "<IP_ADDRESS>:32933" [1104]

I've also added a bunch of additional exception handling on the subscriber side, with logging.  The logs are quiet, so it doesn't seem to be an exception in the handler breaking it.
I also tracked down the zero length messages.  The run that I've included the data from still has those messages, but I've had subsequent runs without the zero length messages and I still see the same behaviour.

Retesting with 0.9.10-SNAPSHOT, still seeing the same issue:
15:44:09 - Aeron Stat
=========================
  0:           95,839,872 - Bytes sent
  1:           97,988,320 - Bytes received
  2:                    0 - Failed offers to ReceiverProxy
  3:                    0 - Failed offers to SenderProxy
  4:                    0 - Failed offers to DriverConductorProxy
  5:               14,815 - NAKs sent
  6:               14,815 - NAKs received
  7:            1,434,072 - Status Messages sent
  8:            1,434,250 - Status Messages received
  9:            2,785,669 - Heartbeats sent
 10:            2,695,995 - Heartbeats received
 11:                5,692 - Retransmits sent
 12:                  239 - Flow control under runs
 13:                    0 - Flow control over runs
 14:                    0 - Invalid packets
 15:                    0 - Errors
 16:                    0 - Short sends
 17:               57,004 - Client keep-alives
 18:                  423 - Sender flow control limits applied
 19:                    0 - Unblocked Publications
 20:                    0 - Unblocked Control Commands
 21:                    0 - Possible TTL Asymmetry
 22:                    1 - send-channel: aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS>

721:               64,800 - pub-lmt:<PHONE_NUMBER>97 -1762877036 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS>
722:               32,032 - snd-pos:<PHONE_NUMBER>97 -1762877036 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS>
858:               32,032 - sub-pos: 1153 -747111497 -1762877036 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS> @64
1323:               32,032 - sub-pos: 73 -747111497 -1762877036 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS> @0
1324:               32,032 - sub-pos:<PHONE_NUMBER>97 -1762877036 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS> @0
1325:               32,032 - sub-pos:<PHONE_NUMBER>97 -1762877036 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS> @0
1326:               32,032 - sub-pos:<PHONE_NUMBER>97 -1762877036 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS> @0
1327:               32,032 - sub-pos:<PHONE_NUMBER>97 -1762877036 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS> @0
1328:               32,032 - sub-pos:<PHONE_NUMBER>97 -1762877036 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS> @0
1329:               32,032 - sub-pos:<PHONE_NUMBER>97 -1762877036 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS> @0
1330:               32,032 - sub-pos:<PHONE_NUMBER>97 -1762877036 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS> @0
1331:               32,032 - rcv-hwm: 1009 -747111497 -1762877036 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS>
1332:               32,032 - rcv-pos: 1009 -747111497 -1762877036 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS>
1459:                    0 - sub-pos: 1099 -747111497 -1762877036 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS> @0
1553:               32,032 - sub-pos: 1213 -747111497 -1762877036 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS> @8,320
1794:               32,032 - sub-pos: 1358 -747111497 -1762877036 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS> @32,032
2073:               32,032 - sub-pos: 1575 -747111497 -1762877036 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS> @32,032
2211:               32,032 - sub-pos: 1685 -747111497 -1762877036 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS> @32,032
2320:               32,032 - sub-pos: 1786 -747111497 -1762877036 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS> @32,032
2350:               32,032 - sub-pos: 1812 -747111497 -1762877036 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS> @32,032
2375:               32,032 - sub-pos: 1848 -747111497 -1762877036 aeron:udp?endpoint=<IP_ADDRESS>:10000|interface=<IP_ADDRESS> @32,032

Increased imageLivenessTimeout to 30 seconds and the system appears to work.

@mikeb01 I assume the image unavailability also stops with upping the liveness?

@mikeb01 back a while ago, you identified that an unavailable image was causing data to stop. At the time we tracked that down to SETUP not being used when data was not sent causing things to stop. I wonder, though, if somehow the unavailable image is not being handled appropriately.
You are calling Subscription.poll and not Image.poll directly, correct?
As images go unavailable and come back, they are new images. Polling on old ones will be fruitless. Subscription manages this. Or should. Which I have looked at a number of times in the last few weeks. Nothing I see should be perpetually out of whack.

@tmontgomery I unfortunately didn't get the logs on the run that managed to start, so I'm not sure if the unavailable image events went away or not.  I'll do some more runs and capture that information.
We are using Subscription.poll.  Let me know if there is any additional logging I can include to try and track down what it going on.  It is pretty easy for me to repeat the failure.

@mikeb01 I tracked CPU usage to a busy spinning setup thread when configuring a large number of streams. It can take a number of milliseconds to setup each new publication or subscription in creating log buffers and sockets. The client thread now sleeps awaiting the operation to finish and thus frees the CPU resource to carry out that task when in a contended environment such as CI.

I'll give a run next week.

@mikeb01 Also given you are using the shared threading mode in the driver for CI then the move to incremental cleaning should make things more responsive.

Should also add a means to have the BroadcastReceiver on the client track the number of missed messages and expose it for clients to be able to see. For debugging, that is.

If the client cannot keep up with the broadcast buffer it gets an IllegalStateException. I assume this has not been observed in the client?

@mikeb01 The exception will propagate up the call stack if it occurs during operation to add or release a subscription/publication, or to the error handler if it occurs during the duty cycle of the client conductor.

The fact that a change in image liveness timeout makes me think it is worth investigating the state machine for image cleanup for when many subscriptions exist.

Ah. The CopyBroadcastReceiver has the check. Was just looking at the BroadcastReceiver. Seems odd to place that behavior there to me at the moment. But that is probably just initial reaction since I wrote the C++ version a while ago (and it has the check also).
@mikeb01 do you have logs and logbuffers for the last run you had that demonstrated the issue? (i.e. not the clean run, but the bad one with the stats above). If not, that is OK. I just want more data to stare at.
Let me work through some logic...
The state machine has almost no dependency to subscriptions aside from event purposes. And, the logs look normal from an image management perspective. Unless I am missing something. And, the unavailable image event has nothing to do with the subscription position. It simply informs the client that the image is not available. Then the client matches the image correlation Id per Subscription. Thus, only a single event per image is needed. If state management of the image were a problem in the driver, the log would show something odd. If state management of the image were a problem in the client, then it would be fairly obvious since the logic is quite simple and is totally driven by events.
The thread starvation causing an image to go unavailable is a temporary thing and the system should eventually resume. It could be pathological, though. 5 seconds of no DATA or SETUP is a long time. And loss indications point to a lot of congestion and/or starvation of the Receiver thread. But the stats indicate there is data received and delivered to all but one subscriber. Which means the data does make it into the logbuffer. And the logbuffer files indicate that also.

Hmmm. I think I might see something. Another event log might help to confirm.
addImage looks for sessionId. But I see on an addSubscription that the same sessionId is being added twice. Which is fine for situations where there is an image going away and being re-instated. Which is this case. But it is on a new subscription.
This might be a timing thing related to adding a subscription while one is in linger. Somewhat like you mention, @mjpt777 , but actually, not. :)

The anomaly in the log indicates some potential issues. It may not be the issue, though. But it could explain the behavior. Would rather have some more event logs and stats of bad runs to solidify.
What I am thinking is that relying on sessionId for addImage is the wrong thing to do on the client. It needs to be correlationId instead.
That is one thing, but there is another.
Another is that sending unavailable image on going into linger is fine, but then it is imperative to not send available image after that when doing an addSubscription. This must be coordinated in the driver between Receiver and Conductor threads correctly. Or waiting for unavailable image to just before cleanup.

I don't have any to hand, but I can produce some later this week.

I've done 6f2d1f14d02ce0fce825ba6eb1b560f9eb0b3d92 to try and avoid issuing an availableImage when doing an addSubscription on an image that is not ACTIVE. This should help avoid the case seen in the one event log that could explain things.
I don't know if this also addresses anything else seen, though.

Convinced myself to go ahead and make the addImage filtering be correlationId based. bbaf75dba0d6f42435d1ca9bfc9628559e85c9ed

Which version do you want me to gather data from?  Latest release or latest snapshot?

Just trying out latest snapshot should say whether things are working better or not. Gathering any logbuffers and event logs and stats from it should help.

I think this is now fixed.  3 runs and the problem is yet to recur.  Not sure if it is due to less aggressive spinning (b8d83be5baeb4eb731a6cd632d5039204b173fb3) or using correlationIds in addImage (bbaf75dba0d6f42435d1ca9bfc9628559e85c9ed).

Thanks @mikeb01 all the feedback has been really helpful.

Thanks, @mikeb01 !


[REQUEST] Allow Commit No. Series for Checkout

At the moment, git plus: checkout only gives me a list of names branches. Just wonder if it's possible to type in historical commit no. series, so users can quickly jump among various code versions. Thanks.
P.S. git time-machine does provide the request feature, but it only realises it on a file basis, instead of the entire commit version.
P.S.S. BTW, is it possible to have a merge and push option (as well as its shortcut) in git plus. I personally find it quite useful. Thanks again.

This isn't a feature I would want in this package. At most it would be a really low priority. I think this is something that fits better in the time machine package.
For the merge and push option: I don't want to introduce these combo commands anymore because they aren't always common for workflows. I want the commands to be the simple and common git commands we use at the command line. A feature I'm looking into introducing is for flexible customization which would include allowing users to specify their own commands and/or chaining existing commands together so you can have merge and push.


Support sqlite3_open_v2

Perhaps take inspiration from this pull request in the direct-sqlite repo.

That PR maps the open function flags to explicit integer values. Perhaps it would be better to use the #{const ...} stuff in Types.hsc to refer to the flag names.

The fourth parameter to sqlite3_open_v2() is the name of the sqlite3_vfs object that defines the operating system interface that the new database connection should use. If the fourth parameter is a NULL pointer then the default sqlite3_vfs object is used.

Think about how to encode that NULL in the high-level API.

Added in 221e6ba0141620549eb6f568029358738806461f.


⚠ Bofrost.de has degraded performance

In 4f8d417, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 1009 ms

Resolved: Bofrost.de performance has improved in a6ec2cb after 35 minutes.


Plugin should support increments even if version ends with alpha chars

Is your feature request related to a problem? Please describe.
According to Semantic Versioning 2.0.0 the below version patterns are valid.

3.0.0-rc1
3.0.0-rc.1
3.0.0-beta1
3.0.1-beta

The  above versioning should be supported.
Expected Results

Current
After Execution

3.0.0
3.0.1

3.0.0-rc1
3.0.0-rc2

3.0.0-rc.1
3.0.0-rc.2

3.0.0-beta1
3.0.0-beta2

3.0.0-beta
3.0.1-beta

Hi @VenomVendor, thanks for your feedback!
It's a good idea and I think projects will benefit from this suffix.
I started talking about it with @xvelx and I suggested to save the suffix as part of the created version tag as well.
We can add an extra option to the versioning { } block like tagSuffix where a developer can put a static string or a custom function that generates the expected value.
What do you think?

We should try to resolve without additional input from users.
Describe the solution you'd like

Split existing version by .
Incase of major/minor update, we can directly increment.
If patch, then split by digit \D+

reverse loop
increment first digit
break

Pseudo Code
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import re

# Uncommment for testing different possible versioning
# actual_version = '0.0.29'# --> '0.0.30'
actual_version = '2.0.0-rc.1'# --> '2.0.0-rc.2'
# actual_version = '0.0.22beta'# --> '0.0.23beta'
# actual_version = '0.0.22rc-1'# --> '0.0.22rc-2'
# actual_version = '0.0.22d1201'# --> '0.0.22d1202'

stage_indicator = ''
stage_version = -1

semantic = re.split("\.", actual_version, 2)
major = semantic[0]
minor = semantic[1]

increment_patch = True # hardcoded runtime value.

if(len(semantic) > 2 and increment_patch):
    remaining = re.split('(\D+)', semantic[2])[::-1]

    for i, val in enumerate(remaining):
        if val.isdigit():
            remaining[i] = int(val) + 1
            break

    remaining = remaining[::-1]
    incremented_patch = ''.join(map(str,remaining))

    print "{}.{}.{}".format(major, minor, incremented_patch)


process remains running after exiting automatic1111

the python process launched for comfyui  is not properly terminated and remains running after exiting automatic1111 which causes issues and errors if trying to relaunch automatic1111 while the original process is still running

Interresting, what is your OS?

Interesting, what is your OS?

Windows 10 Home
Version	22H2
OS build	19045.3693
Experience	Windows Feature Experience Pack 1000.19053.1000.0
Python 3.11.4

traceback
Traceback (most recent call last):   File "C:\Users\throwromans\stable-diffusion-webui\extensions\sd-webui-comfyui\lib_comfyui\comfyui\pre_main.py", line 94, in <module>     main()   File "C:\Users\throwromans\stable-diffusion-webui\extensions\sd-webui-comfyui\lib_comfyui\ipc\__init__.py", line 41, in wrapper     return function(*args, **kwargs)            ^^^^^^^^^^^^^^^^^^^^^^^^^   File "C:\Users\throwromans\stable-diffusion-webui\extensions\sd-webui-comfyui\lib_comfyui\comfyui\pre_main.py", line 41, in main     patch_comfyui()   File "C:\Users\throwromans\stable-diffusion-webui\extensions\sd-webui-comfyui\lib_comfyui\ipc\__init__.py", line 41, in wrapper     return function(*args, **kwargs)            ^^^^^^^^^^^^^^^^^^^^^^^^^   File "C:\Users\throwromans\stable-diffusion-webui\extensions\sd-webui-comfyui\lib_comfyui\comfyui\pre_main.py", line 80, in patch_comfyui     paths.share_webui_folder_paths()   File "C:\Users\throwromans\stable-diffusion-webui\extensions\sd-webui-comfyui\lib_comfyui\ipc\__init__.py", line 41, in wrapper     return function(*args, **kwargs)            ^^^^^^^^^^^^^^^^^^^^^^^^^   File "C:\Users\throwromans\stable-diffusion-webui\extensions\sd-webui-comfyui\lib_comfyui\webui\paths.py", line 10, in share_webui_folder_paths     webui_folder_paths = get_webui_folder_paths()                          ^^^^^^^^^^^^^^^^^^^^^^^^   File "C:\Users\throwromans\stable-diffusion-webui\extensions\sd-webui-comfyui\lib_comfyui\ipc\__init__.py", line 20, in wrapper     res = current_callback_proxies[process_id].get(args=(function.__module__, function.__qualname__, args, kwargs))           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^   File "C:\Users\throwromans\stable-diffusion-webui\extensions\sd-webui-comfyui\lib_comfyui\ipc\callback.py", line 70, in get     self._args_sender.send((args if args is not None else (), kwargs if kwargs is not None else {}))   File "C:\Users\throwromans\stable-diffusion-webui\extensions\sd-webui-comfyui\lib_comfyui\ipc\payload.py", line 45, in send     self._strategy.set_data(lock_file, pickle.dumps(value))   File "C:\Users\throwromans\stable-diffusion-webui\extensions\sd-webui-comfyui\lib_comfyui\ipc\strategies.py", line 98, in set_data Model loaded in 6.6s (load weights from disk: 1.7s, create model: 0.5s, apply weights to model: 3.6s, calculate empty prompt: 0.7s).     self._shm = SharedMemory(name=self._shm_name, create=True, size=data_len)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^   File "C:\Python311\Lib\multiprocessing\shared_memory.py", line 143, in __init__     raise FileExistsError( FileExistsError: [WinError 183] File exists: 'ipc_payload_args_webui' [ComfyUI] [sd-webui-comfyui] Stopped callback listeners for process comfyui

One workaround we could try is preventing from running the webui code that replaces the sigint handler with a direct python exit.
I think this could work because it should let the atexit callbacks run.
I don't like this solution because if another extension does the same thing, there could be a difference in what the sigint handler actually does in each extension. I'll ask what the webui maintainers think and see if there could be instead a solution on their side.

This went over my head a little bit. A possible fix is to have the comfyui server continually ping the webui for status and then let it kill itself when it gets no reply. This is all happening from a local machine so no reliability method should be needed.


Issue with terminal in LM20(Cinnamon)

After multiple issues involving separate logical partitions(one for Home and one for Root), I have finally been able to get a result from typing Fetchord into gnome-terminal. However, after the following output:
Connecting RPC connection successful.
The terminal process does not run in the background. I cannot find a way to 'disown' the 'job', so in order to keep the program running I always need to have a terminal window open. I've tried to work around this by switching to a different desktop, but I don't think that it's the best solution. Is fetchcord supposed to act like this, or is it a problem with my system? I have followed the installation and startup instruction just as shown on the homepage, besides reinstalling<PHONE_NUMBER>4030847539484 times since my system is rather problematic due to my partitioning system. Otherwise, fetchcord works perfectly. I do have 2 requests but I'm not going to put them here as I don't know how difficult it will be to heed them.

ok that was nerdy

for the other 2 requests, just make another issue or 2 whatever and we'll see what we can do about them.

One is already being addressed.

can you try againn?

This issue will be closed in 1 day.


Worldwide Data Automation

Hello @jvfe,
Nice work with the automation with the datahub.io data!
I believe we can use as reference:
Reference URL : https://datahub.io/core/covid-19
File name in archive (P7793) : r/countries-aggregated.csv
Retrieve in : <today's date>
(This is the download URL https://datahub.io/core/covid-19/r/countries-aggregated.csv)
Also, nice that you blacklisted pages that are being manually updated!
What we can do is check which of the items in the list had an update in the past 20 hours and
only update those that didn't.
This in addition to the ones you detected as manually curated.
What do you think?
I'll make those changes to the code now, but we can discuss further.

Sure, I think the change is great! I actually totally forgot about this script until about an hour ago and I just updated the items before I saw your comment.
But the changes are welcome, I think it's not hard to implement. Thanks!

Nice,
I have implemented the code to do the screening and get the nonupdated items.
I think your update did not work for all of the items, see the Angola page, for exemple.
If you could merge your changes to the files I pushed here, it would be awesome, so we do not have editing conflicts.
Cool, it is working!

Oh yeah it didn't, don't know why. I actually didn't change anything in the code from last time. Just ran it and updated, so there's nothing to merge, thanks anyway for asking.
I believe the main thing for automation both in this script and Brazil data would be integrating with the QS api or registering a bot for it (still don't get how that works in wikidata) but I couldn't get the API to work in any way, shape or form. It keeps saying I don't have permission due to not having used QS in batch mode (I must've used it more than ten times).
Anyway thanks for the update!

I get exactly the same problem whenever I try to use the QS API! I think
that is some bug on their side.
So, registering for a bot is basically just going to this page
https://www.wikidata.org/wiki/Wikidata:Bot_requests and following the
instructions there, but this does not have to limit you. I mean, for Brazil
at least the edits are not that many.
I am working in parallel with the CellosaurusBot (for cell line info) :
https://github.com/lubianat/cellosaurus-wikidata-bot.
I could run it from my personal, non-bot account. It uses WikidataIntegrator
https://github.com/SuLab/WikidataIntegrator to run updates to Wikidata
via python, I believe it may be useful.
On Wed, Apr 29, 2020 at 5:29 PM João Vitor<EMAIL_ADDRESS>wrote:

Oh yeah it didn't, don't know why. I actually didn't change anything in
the code from last time. Just ran it and updated, so there's nothing to
merge, thanks anyway for asking.
I believe the main thing for automation both in this script and Brazil
data would be integrating with the QS api or registering a bot for it
(still don't get how that works in wikidata) but I couldn't get the API to
work in any way, shape or form. It keeps saying I don't have permission due
to not having used QS in batch mode (I must've used it more than ten times).
Anyway thanks for the update!
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/lubianat/wikidata_covid19/issues/2#issuecomment-621445758,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AB4NC76NJUKKK4KW6QZIEYDRPCE33ANCNFSM4MT7WPCA
.

Hey @jvfe,
I will make a Bot Request for the worldwide data, then!
It will run every day for country items that were not updated in the past 23 hours and that is not on the items you curated as "manually updated".
The point is: I have never done a bot request before.
I will keep you updated here of whatever happens there! There would be a bot account that I was thinking of running automatically from a server (my pc, most likely), but we could divide the account too. Actually, that would be super welcome.
Best,
Tiago

Sure, I just updated the list of countries I saw being manually updated as the previous list was from more than a week ago.
I think we can split the account too, it's your choice, to me it's all good!


AddDoesNotPerformAssertionToNonAssertingTestRector incorrectly adds @doesNotPerformAssertions annotation

Consider test method and its related methods:
public function testSomething(): void
{
    $this->mockSomething(); // This makes assertion internally by using `MockObject::expects()`
    $this->runSomething(true);
}

private function mockSomething(): void
{
    $this->something // This is mock prepared in setUp()
        ->expects(self::once())
        ->method('handle');
}

Currently AddDoesNotPerformAssertionToNonAssertingTestRector adds @doesNotPerformAssertions annotation to such tests, which is not true. Method itself does not have assertion calls, but other method(s) called within test makes assertions with MockObject::expects().

Fixed in https://github.com/rectorphp/rector-phpunit/pull/211


chore: refactor body of resolveDimensions

This is part of the Gatsby fp-ts refactor.

Current dependencies on/for this PR:

next

PR #190

This comment was auto-generated by Graphite.


Allow to not export threads related variables

Some programs don't use threads efficiently (e.g.: GATK).
On cluster (e.g.: kubernetes) deployment, the number of threads determines how many jobs can be started on each node.
Currently, by setting the number of threads, multiple environment variables are set by default (e.g.: OMP_NUM_THREADS) besides the 'threads' snakemake variable which can be passed in the commandline.
https://github.com/snakemake/snakemake/blob/225e68cb05ecbf8fe001a884a416da0239cfd4d1/snakemake/shell.py#L218
If we could disable this behaviour, the user would be able to describe how many threads should be requested from the cluster (aka, how many tasks per node) and how much to overcommit (e.g.: by using a function and runnig 2X as many threads)

I also think the setting of threading-related environment variables should be user-configurable. I would like to prepare a PR. @johanneskoester What do you think where this configuration should happen? I suggest to add it to the rule options, e.g.:
rule NAME:
    input: "path/to/inputfile", "path/to/other/inputfile"
    output: "path/to/outputfile", "path/to/another/outputfile"
    threads: 8
    threads_env: False  # <== defaults to True
    shell: "somecommand --threads {threads} {input} {output}"

My current workaround:
rule NAME:
    params:
        threads_calc = lambda wildcards, threads: threads*2,
    shell:
        """
            export OMP_NUM_THREADS={{params.threads_calc}};
            export GOTO_NUM_THREADS={{params.threads_calc}};
            export OPENBLAS_NUM_THREADS={{params.threads_calc}};
            export MKL_NUM_THREADS={{params.threads_calc}};
            export VECLIB_MAXIMUM_THREADS={{params.threads_calc}};
            export NUMEXPR_NUM_THREADS={{params.threads_calc}};

            whatever_command.py --threads {{params.threads_calc}}
        """


Alternative Sampling Methods

This PR primarily implements alternative methods such as Top-P-X and Top-A from RWKV. I'll evaluate whether they perform better than the baseline, and if they do, merge this PR or perhaps add even more options to the inference code. Once we're confident that our sampling performs well, we can take a stab at alternative data distributions (#5, #9) and automated evaluation (#21).

https://github.com/HomebrewNLP/HomebrewNLP-Jax/pull/42/commits/a67d2e0784498b0fff46c18d86d0be88ad4c57db works fine

..and so does https://github.com/HomebrewNLP/HomebrewNLP-Jax/pull/42/commits/4fdc94b2c0f686a8928ac7781618a3ea41e2bb6e


NixOS switches to a terminal from X11 on switch

Issue description
As discussed in mailing list, NixOS sometimes suddenly switches to a text terminal from X11 on switch. When working with multiple monitors, it's more bizzare: one monitor switches and other continues to show X11 but mouse is inaccessible (you can go back to X11 with C-M-F1 C-M-F7). For me, also, after some time if this occured a sudden graceful reboot happens.
Steps to reproduce

You should be in X11;
You should switch from a different enough revision;
Run nixos-rebuild swtich and wait till it comletes;
Observe terminal.

Technical details

Nixpkgs revision: f19c961b4e461da045f2e72e73701059e5117be0

Restarted services:
stopping the following units: acpid.service, alsa-store.service, audit.service, autofs.service, avahi-daemon.service, bumblebeed.service, chronyd.service, colord.service, cups-browsed.service, cups.service, dbus-org.bluez.service, gpm.service, idmapd.service, kmod-static-nodes.service, logmein-hamachi.service, nat.service, network-local-commands.service, network-manager.service, network-setup.service, nix-daemon.service, nix-gc.timer, nscd.service, postgresql.service, rngd.service, rpcbind.service, rtkit-daemon.service, statd.service, systemd-journald.service, systemd-modules-load.service, systemd-sysctl.service, systemd-tmpfiles-clean.timer, systemd-tmpfiles-setup-dev.service, systemd-udev-trigger.service, systemd-udevd.service, thermald.service, tlp.service, upower.service, wpa_supplicant.service
Warning: Stopping nix-daemon.service, but it can still be activated by:
  nix-daemon.socket
Warning: Stopping systemd-journald.service, but it can still be activated by:
  systemd-journald-dev-log.socket
  systemd-journald-audit.socket
  systemd-journald.socket
Warning: Stopping systemd-udevd.service, but it can still be activated by:
  systemd-udevd-kernel.socket
  systemd-udevd-control.socket
NOT restarting the following changed units: display-manager.service<EMAIL_ADDRESS>systemd-backlight@backlight:intel_backlight.service, systemd-backlight@leds:tpacpi::kbd_backlight.service, systemd-journal-flush.service, systemd-random-seed.service, systemd-remount-fs.service, systemd-tmpfiles-setup.service, systemd-update-utmp.service, systemd-user-sessions.service<EMAIL_ADDRESS><EMAIL_ADDRESS>activating the configuration...
setting up /etc...
restarting systemd...
reloading the following units: dbus.service, dev-hugepages.mount, dev-mqueue.mount, sys-fs-fuse-connections.mount, sys-kernel-debug.mount
restarting the following units: polkit.service, sshd.service, systemd-logind.service
starting the following units: acpid.service, alsa-store.service, audit.service, autofs.service, avahi-daemon.service, bumblebeed.service, chronyd.service, colord.service, cups-browsed.service, cups.service, dbus-org.bluez.service, gpm.service, idmapd.service, kmod-static-nodes.service, logmein-hamachi.service, nat.service, network-local-commands.service, network-manager.service, network-setup.service, nix-daemon.service, nix-gc.timer, nscd.service, postgresql.service, rngd.service, rpcbind.service, rtkit-daemon.service, statd.service, systemd-journald.service, systemd-modules-load.service, systemd-sysctl.service, systemd-tmpfiles-clean.timer, systemd-tmpfiles-setup-dev.service, systemd-udev-trigger.service, systemd-udevd.service, thermald.service, tlp.service, upower.service, wpa_supplicant.service
the following new units were started: NetworkManager-dispatcher.service, systemd-hostnamed.service

Subscribing here as I reported this issue on the ML and I'm also experiencing this.

I was getting dbus.service reload failure but it didn't seem to cause any problems.

I have kind of a related issue. When a nixos-rebuild switch upgrades systemd, pressing Alt and cursor keys left or right in X will switch to another tty and also propagate that key event to X.
This could possibly be related to https://github.com/systemd/systemd/issues/3842 as systemctl daemon-reexec is be called to upgrade systemd. The linked Debian bug report has more details on the tty issues that seem to match our symptoms: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=834367
I'll apply the patch and test if that fixes the problem.

for anyone who wants a quick fix, just run (as root)
dumpkeys | sed "s/Incr_Console/Right/g; s/Decr_Console/Left/g" | loadkeys

@taktoa what does this do?

sorry, I should have been more specific in saying that it fixes the issue mentioned above by @fpletz with the alt+cursor keys.

Related to #17560?


Could Redis-Android supports RadisGraph?

I want to use RedisGraph on Android based on Redis. Official document says that Redis load RedisGraph during startup by adding the following to redis.conf file:
loadmodule /path/to/module/src/redisgraph.so
Could redis-android supports add .so file or uses other methods?

Um, I've never tried it.

@zhuguocheng29 did you manage to run the module on android?

@gkorland No, it doesn't work


handling my own url

I would like your opinion: I'm not sure what method to use to manage your own url manager. Currently when I click on a URL it attempts to open on my browser. I would like to modify my egui interface based on this url.
I thought it was handled by egui itself, I tried that, but it doesn't return anything:
    fn update(&mut self, ctx: &egui::Context, _frame: &mut eframe::Frame) {

        ctx.output_mut(|o| {
            if let Some(openurl) = &o.open_url {
                println!("{}",openurl.url);
            }
        });
...

How to do ? Thank you!

Chatgpt hallucinated an idea like
CommonMarkViewer::new()
        .link_action(|url| {
            handle_link(url);
            LinkAction::None // or Open, or OpenInTab, or...
        })
        .show(ctx, markdown_text);

I also need to do this. Did you ever figure it out?

There are two methods.
// egui_commonmark stuff here
ctx.output_mut(|o| {
   // open_url must be converted to None so that egui does not open the url afterwards
   if let Some(openurl) = &o.open_url.take() {
          println!("{}", openurl.url);
    }
});

It should work, but it has to be done after the egui_commonmark calls.
This is the easiest method, but it has an annoyance that if you do the following:
// This used to be the default, and is the most sane option IMO. You should always be able
// to see where an external link takes you.
ui.style_mut().url_in_tooltip = true;

This will show the url on hover.  For normal urls this is good, but it is annoying when the link is supposed to be handled by the app. To avoid this egui_commonmark can be informed about the link before hand so that it can create it as a link instead of a hyperlink.
For this purpose link hooks can be used. In the example they are processed before the egui_commonmark call, but that is just due to that it is directly linked to what the example renders. It should be possible to call it afterwards too.


Incorrect example code in 'Create a dataset' docs

Describe the bug
On this page, the example code for loading in images and audio is incorrect.
Currently, examples are:
from datasets import ImageFolder
dataset = load_dataset("imagefolder", data_dir="/path/to/pokemon")

and
from datasets import AudioFolder
dataset = load_dataset("audiofolder", data_dir="/path/to/folder")

I'm pretty sure the imports are wrong and should be:
from datasets import load_dataset
dataset = load_dataset("audiofolder", data_dir="/path/to/folder")

I am happy to update this if this is right but just wanted to check before making any changes.
Steps to reproduce the bug
Go to https://huggingface.co/docs/datasets/create_dataset
Expected behavior
N/A
Environment info
N/A

This was fixed in https://github.com/huggingface/datasets/pull/6247. You can find the fix in the main version of the docs

Ah great, thanks :)


Place "Attach Images" button to the top

It would be great to have attach images button at the top of slideshow field because when you upload many images you have to scroll down each time to attach an additional one.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Support for functions?

Are there any plans to add the new functions feature?

Are there any plans to add the new functions feature?

Hello @Wizzel1 👋
Unfortunately, I don't have much spare time so the project development is stopped for now...
Meanwhile, you could check out this really cool package that has - basically - everything almost implemented!
Enjoy 🚀

@Azzeccagarbugli okay thanks for the quick response, all the best


Hi @luftuq,

#506               Hi @luftuq,
Thank you for reporting this issue. To help us understand and resolve it more efficiently, could you please provide a minimal reproduction example? This will allow us to replicate the problem on our end and investigate it further.
Originally posted by @gauravsaini04 in https://github.com/devcontainers/spec/issues/476#issuecomment-2404435859

Looks like this may not have the actionable info mentioned in the original comment. If you have that info, or a specific issue related to dev containers, please let us know. Thank you!


Fix figures

Most of the changes are visually indistinguishable. Apparently I used a different random seed for the single-species SDMs when I re-ran everything, so a few of those results jiggled around a small amount.
I had forgotten to push the addition of "A" and "B" to the code for the observer figure, so that's now added.

this is where the swipe feature really shines

They asked for a color legend for Fig. 2. I think it's fine as is. I also got a lab mate opinion and they thought the colors were fine as well. But it was suggested to improve this sentence in the Fig 2 legend.
In this example most models are made more robust to observer turnover by including an observer model.

Thansk @sdtaylor. Like this? https://github.com/weecology/bbs-forecasting/pull/211/commits/4dfdedf2cadb1c91d69be47534b6b5fffa7e1d0d

LGTM.

:+1:


Correction of the language name in README.md

crates.io does the syntax coloration only when the word rust is written in lowercase.
For example https://crates.io/crates/debug_print:
in the README.md rust is written in lowercase and then in crates.io the syntax coloration appears.
but in your README.md Rust is written an with uppercase R and syntax coloration doesn't appear in your crates.io.

Nice catch, thanks!


Paste option sometimes does not show up properly.

There are quite a few moments where the paste option is not showing up. I am not sure if it's only paste option or it is for the first option in the context menu shown.

Do you have any suggestions on how to reproduce this, even if it's inconsistent?

What I noticed was to continuously try to repeat the tap on the where the carat will be (or the line that the carat is at). This is significantly more consistent on a simulator, not sure if that makes a difference in the fix.
https://github.com/stevengharris/MarkupEditor/assets/4402214/fd5e0c70-7597-448c-ba0f-839fdc8dc449

In debugging some other paste issues, it seems like the pasteboard is sometimes not being populated when copying from within the MarkupEditor. Do you think this is what you are observing? If the pasteboard is empty, then the paste option won't show up. I can see this in Mac Catalyst as the paste menu item being disabled.


Add SSZ container: SignedVoluntaryExit

Depends on: #66
Follow these instructions: https://github.com/lambdaclass/lambda_ethereum_consensus/blob/main/native/ssz_nif/README.md
SignedVoluntaryExit
class SignedVoluntaryExit(Container):
    message: VoluntaryExit
    signature: BLSSignature

can I take this ser?

Go ahead! 😄


Remember undo history on hot-exit

This will make restarts/crashes more and more seamless.

:+1: would definitely be useful, currently the hot exit code doesn't touch this stuff yet so not sure how much effort it would take to persist the undo stacks.

I think it would be nice to persist any undo stack across all editors. Needs https://github.com/Microsoft/vscode/issues/7536

Could we pretty please have this? With sugar on top.

sublime has this, this would be great

Will you kindly share the status of this issue now?  It’s the end of 2023 now.  There's an option "restore undo stack", but it doesn't seem to work.

@yllekz you can configure that behavior with the terminal.integrated.confirmOnExit setting. If you're just reloading the window it should reconnect the terminals as well.

I will a bit heavy regarding RAM consumption, but It would be great if VSCode has a feature as "Local History" as Jetbrains IDEs


Runtime tetragon

Signed-off-by: pxp928<EMAIL_ADDRESS>Fixes Issue #212 - Adding runtime visibility and enforcement into frsca
Future updates will need to sign and attest the runtime output based on the Pipelines/taskruns being run. This would utilize witness.
The first way is to observe the raw json output from the stdout container log:

kubectl logs -n kube-system -l app.kubernetes.io/name=tetragon -c export-stdout -f

REQUIREMENT:
The base kernel should support BTF or the BTF file should be placed where Tetragon can read it. Tetragon repository provides a Vagrantfile that can be use to install a vagrant box for running Tetragon with BTF requirement.
The testing of this was done on GKE. Other cloud provider kubernetes offering would also work.

Is it supposed to do something right now?
I ran make quickstart, and I do see it being deployed and starting (see snippet bellow), but after that I am not sure how to check that it does what it is supposed to do.
Setting up Tetragon...
+ helm upgrade --install tetragon /Users/rgreinhofer/projects/buildsec/ssf/platform/vendor/tetragon/chart --values /Users/rgreinhofer/projects/buildsec/ssf/platform/components/tetragon/values.yaml --namespace kube-system --wait
Release "tetragon" does not exist. Installing it now.
NAME: tetragon
LAST DEPLOYED: Wed Jul  6 12:21:20 2022
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
+ kubectl rollout status -n kube-system daemonset/tetragon
Waiting for daemon set "tetragon" rollout to finish: 0 of 1 updated pods are available...
daemon set "tetragon" successfully rolled out
+ echo -e '\033[32mEnable tracing policies for Tetragon...\033[0m'
Enable tracing policies for Tetragon...
+ kubectl apply -f /Users/rgreinhofer/projects/buildsec/ssf/platform/vendor/tetragon/tracingpolicy/v0.8.0/sys_write_follow_fd_prefix.yaml
tracingpolicy.cilium.io/sys-read-follow-prefix created
+ kubectl apply -f /Users/rgreinhofer/projects/buildsec/ssf/platform/vendor/tetragon/tracingpolicy/v0.8.0/tcp-connect.yaml
tracingpolicy.cilium.io/connect created

The log command you provided does not return anything.
╭ ssf on  pr/249 via 🐹 v1.18.3 ⌛ 8s
╰❯ kubectl logs -n kube-system ds/tetragon -c export-stdout

Is it supposed to do something right now?
I ran make quickstart, and I do see it being deployed and starting (see snippet bellow), but after that I am not sure how to check that it does what it is supposed to do.
Setting up Tetragon...
+ helm upgrade --install tetragon /Users/rgreinhofer/projects/buildsec/ssf/platform/vendor/tetragon/chart --values /Users/rgreinhofer/projects/buildsec/ssf/platform/components/tetragon/values.yaml --namespace kube-system --wait
Release "tetragon" does not exist. Installing it now.
NAME: tetragon
LAST DEPLOYED: Wed Jul  6 12:21:20 2022
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
+ kubectl rollout status -n kube-system daemonset/tetragon
Waiting for daemon set "tetragon" rollout to finish: 0 of 1 updated pods are available...
daemon set "tetragon" successfully rolled out
+ echo -e '\033[32mEnable tracing policies for Tetragon...\033[0m'
Enable tracing policies for Tetragon...
+ kubectl apply -f /Users/rgreinhofer/projects/buildsec/ssf/platform/vendor/tetragon/tracingpolicy/v0.8.0/sys_write_follow_fd_prefix.yaml
tracingpolicy.cilium.io/sys-read-follow-prefix created
+ kubectl apply -f /Users/rgreinhofer/projects/buildsec/ssf/platform/vendor/tetragon/tracingpolicy/v0.8.0/tcp-connect.yaml
tracingpolicy.cilium.io/connect created

The log command you provided does not return anything.
╭ ssf on  pr/249 via 🐹 v1.18.3 ⌛ 8s
╰❯ kubectl logs -n kube-system ds/tetragon -c export-stdout

try this command. I updated the above to match
kubectl logs -n kube-system -l app.kubernetes.io/name=tetragon -c export-stdout -f

Same thing:
╭ ssf on  pr/249 via 🐹 v1.18.3
╰❯ kubectl logs -n kube-system -l app.kubernetes.io/name=tetragon -c export-stdout

╭ ssf on  pr/249 via 🐹 v1.18.3
╰❯

Even if I run another example in another tab, I do not see any logs.

eBPF won't work on a Mac, so we will need someone with a linux host to test it.

REQUIREMENT:
The base kernel should support BTF or the BTF file should be placed where Tetragon can read it. Tetragon repository provides a Vagrantfile that can be use to install a vagrant box for running Tetragon with BTF requirement.
The testing of this was done on GKE. Other cloud provider kubernetes offering would also work.

So it runs in my environment and I definitely get logs. What I'm looking for is either a test of something it prevents from running, or a malicious build that it detects. Just looking at the container stdout, I don't think anyone could sort out if there's something wrong from that, it's far too much of a firehose of raw logs.

So it runs in my environment and I definitely get logs. What I'm looking for is either a test of something it prevents from running, or a malicious build that it detects. Just looking at the container stdout, I don't think anyone could sort out if there's something wrong from that, it's far too much of a firehose of raw logs.

I agree. I can add in a test in there that shows it either blocking or detecting malicious builds

@pxp928 We can look at my old "rust" compiler build where a malicious compiler injects a binary at runtime into memory and runs it. https://github.com/mlieberman85/supply-chain-examples/tree/main/build_with_ebpf

@pxp928 We can look at my old "rust" compiler build where a malicious compiler injects a binary at runtime into memory and runs it. https://github.com/mlieberman85/supply-chain-examples/tree/main/build_with_ebpf

Yup that's what I was thinking. Will do.

Should we had a check not to deploy Tetragon if the platform is not linux? Just asking because the pod keeps on crashing in the background and there is no specific reason for me to have it running since I am using a mac M1.

hmmm. For some reason the new pipeline I created is hanging. When I tested it on GKE. It completes very quickly. @sudo-bmitch can you try running the make example-trufflehog locally on your machine.

@pxp928 I'm getting an ImagePullBackoff:
Back-off pulling image "madhuakula/k8s-goat-build-code"


[feat][m] Google Cloud Platform integration for external storage

Google Cloud Platform integration, using lib google-cloud-storage
Relates to #16

@rufuspollock @sglavoie Initial version ready for review; we can discuss in the future (or now):

Better method signatures
Better error handling

@sglavoie  I opened an issue for better integration tests see #20 . I also improved the README file as you suggested 😀

@hannelita i think this looks good to merge - do you want to self merge or for me to merge for you?


Line-list for Colombia

Hi - Many thanks for the estimates and projections at subnational level in Colombia. I noticed you mention that line-list data to inform a Colombia specific estimate of the reporting delay was not available.
Just wanted to point out that the line-list with dates of onset of symptoms (FIS), date of notification, date of diagnosis is made available every day from the official site. I wonder if those can be used to infer delays.
https://www.ins.gov.co/Noticias/Paginas/Coronavirus.aspx
or a direct link here:
https://www.datos.gov.co/Salud-y-Protecci-n-Social/Casos-positivos-de-COVID-19-en-Colombia/gt2j-8ykr/data
Also, the dataset contains information about health state. Notice that "Asymptomatic" category has increased over time as active searches are increasing, which may led to bias as well.
Thank you!

Hi Dr. Zulma, I am trying to do the same for Mexico. We have the same similar reports released by the Health Secretary(Secretaria de Salud) every day, so we need to make nowcasting to know what is the real situation of COVID in the country. As soon as I have something, I will tell you so we can test with the data of Colombia. Greetings

Hello,
We have started working in Mexico as well, currently using a shared linelist. However we are in the process of updating to a custom one:
We haven't sent our issues so far, but you can check our results here:
https://datoscovid.mx/

Hello Zulma, I hope you are well! :)
Apologies for the slow reply, we've been trying to sort out techinical problems that have been preventing us from updating the site.
I think the way the site currently works is that everywhere uses the same delay distribution, which is helpful for places where there is no data but makes less sense for places like Colombia like you point out. The approach that we've agreed on is that our site will be as general as possible but that we are keen to make the EpiNow package as user-friendly as possible so that people can re-run the analysis on their own data for countries that they are specifically interested in. Otherwise we'd be spending a lot of time trying to make the analysis for each country unique and we don't have the labour power for that.
If any of you need any help getting any of the code working on your own data for Colombia or Mexico just comment on here or send me an email at<EMAIL_ADDRESS>


Move final build result to private repo for license reasons

[x] move prod-page to private repo
[x] move staging-oage
[x] make static repo private

See #210


How can I contact you ?

ERROR: Reference sequence length is not match GRCh38, GRCh37 or hg19.
If you need to use DeepMEI for other species or non-GRCh38, GRCh37 and hg19 reference genomes, please contact us.

My email is<EMAIL_ADDRESS>and wechat is xuxiaooo.

My email is<EMAIL_ADDRESS>and wechat is xuxiaooo.

thanks


Update to PRT 2.2 (CityEngine 2020.0)

CityEngine 2020.0 along with PRT 2.2 has been released early June: https://github.com/Esri/cityengine-sdk/releases
Tasks:

Update included PRT to version 2.2 from CityEngine 2020.0
Make sure basic input of array attribute works (add conversion functions for Houdini arrays, use the existing mapping of Houdini data types to PRT data types etc)
Full name of CE has changed from "Esri CityEngine" to "ArcGIS CityEngine" - we should just use CityEngine wherever possible.

(internal ticket CE-8046)


RxSwift support?

Is there any push to create an RxSwift compatible library for this?  It would be very helpful to be able to perform queries in a reactive manor

I'd love to adopt reactive patterns, especially for error handling. Some of them were discussed in #569, and subsequently implemented in GRDB.swift, so maybe check there. @groue, how did it turn out in the end?

I know there is a RxGRDB repo: (https://github.com/RxSwiftCommunity/RxGRDB).  I even tried to dig around in there to see how it was being done, to see if I could glean anything to layer over this repo.  Obviously to get it done a PublishSubject would need to be added somewhere, and that's not too hard, but what I'm having trouble with, is how we'd know when the table is updated properly.
Looking at RxGRDB I think he's using the SQLite Pre-Update hooks to do so, but I'm not 100% on that at all

Hello @jberkel, @cody1024d

what I'm having trouble with, is how we'd know when the table is updated properly.

This does not happen in RxGRDB, which is just a thin layer. The meat belongs to GRDB which has a long history of database observation features.
When you mix SQLite Compile-Time Authorization Callbacks, Commit And Rollback Notification Callbacks and Data Change Notification Callbacks, you can:

track transactions and the savepoint stack
know which columns of which tables are modified by a statement
know about all row insertions, updates, and deletions

All those information together give you a precise enough idea of the changes eventually committed by a transaction. You can package this in the public TransactionObserver protocol, the mother of all database observations features.
Compile-Time Authorization Callbacks also tell which columns of which tables are read by a statement. You can thus write a TransactionObserver that fuels a reactive sequence when a transaction has changed the tracked tables and columns: that's the idea of RxGRDB.
SQLite.swift already has updateHook, commitHook and rollbackHook: that's the place to start.

@jberkel, I think @cody1024d wasn't talking about using reactive patterns to consume the results of a single request, but to be notified of changes in the results in a request.
#569 was about the fact that Swift Sequence does not quite fit the consumption of the results of a request, and that a cursor whose next method can throw is more suitable. That's why GRDB generally provides three fetching methods, depending on the way you want to consume the results of a request:
let request = Player.all()
// Lazy, does not consume much memory
try request.fetchCursor(db)  // DatabaseCursor<Player>
// Contains copies of database values, may consume a lot of memory
try request.fetchAll(db)     // [Player]
// Consumes a single row
try request.fetchOne(db)     // Player?

RxGRDB does not bring any reactive APIs on top of cursors. It's just a couple lines of code away, though:
// Turns DatabaseCursor into RxSwift Observable
extension DatabaseCursor : ObservableType {
    public typealias E = Element
    public func subscribe<O: ObserverType>(_ observer: O) -> Disposable where O.E == E {
        do {
            while let element = try next() {
                observer.onNext(element)
            }
            observer.onCompleted()
        } catch {
            observer.onError(error)
        }
        return Disposables.create()
    }
}

// Consume a cursor in a reactive way:
try dbQueue.inDatabase { db in
    let cursor = try Player.fetchCursor(db)
    cursor.subscribe(
        onNext: { player in print(player) },
        onError: { error in print(error) }
    )
}

What RxGRDB brings is request observation, that is to say observable sequences of changes in the results of a request:
// Observe a single request:
Player.all().rx
    .fetchAll(in: dbQueue)
    .subscribe(onNext: { players: [Player] in
        print("Players have changed: \(players)")
    })

// Observe some request(s) and fetch consistent results from other request(s):
dbQueue.rx
    .changeTokens(in: [Player.all()])
    .mapFetch { (db: Database) -> ([Player], Int) in
        let players = try Player
            .order(Player.Columns.score.desc)
            .limit(10)
            .fetchAll(db)
        let count = try Player.fetchCount(db)
        return (players, count)
    }
    .subscribe(onNext: { (players, count) in
        print("Best players out of \(count): \(players)")
    })

I think this is what @cody1024d is after.
Best thing is that RxGRDB eats all GRDB requests, which means that you can observe records, raw database rows, or values fetched from the query builder or even raw SQL:
// Observe maximum player score:
let request = Player
    .select(max(Player.Columns.score))
    .asRequest(of: Double.self)
request.rx
    .fetchOne(in: dbQueue)
    .subscribe(onNext: { maxScore: Double? in
        print("Maximum score: \(maxScore ?? 0)")
    })

// Observe minimum and maximum player scores:
let request = Player
    .select(
        min(Player.Columns.score).aliased("min"),
        max(Player.Columns.score).aliased("max"))
    .asRequest(of: Row.self)
request.rx
    .fetchOne(in: dbQueue)
    .subscribe(onNext: { row: Row? in
        let minScore: Double? = row!.value(named: "min")
        let maxScore: Double? = row!.value(named: "max")
        print("Score range: \(minScore ?? 0)-\(maxScore ?? 0)")
    })

// Observe players fetched from the occasionally complex SQL query:
let request = SQLRequest("SELECT ...")
    .asRequest(of: Player.self)
request.rx
    .fetchAll(in: dbQueue)
    .subscribe(onNext: { player: [Player] in
        print("Players have changed: \(players)")
    })

Thank you @groue for the hardcore breakdown!  You are right, that that is what I'm after; the ability to subscribe to, essentially a table, and be notified of updates to said table.  I apologize that my initial comment was not well worded.  Looking back on it now, even I was confused by what I was asking.
I think I probably could string something together leveraging the hooks built into SQLite.Swift, at least enough so that I could get the end result I wanted.  As long as I have the table that was updated in the commitHook, I could hook up a PublishSubject to it correctly, and store the initial query when it's performed, and then in the commitHook just perform that same query again, and push it out over the PublishSubject.
Or, I guess, I could switch to GRDB; however my only concern with that is it leans more towards the Active Record approach, and I much prefer a DAO/Repository approach :/.  I guess I could query with raw SQLite within the repositories, but would love to avoid that too.  Lol, I want the best of both worlds I guess

@cody1024d It's good for both you and SQLite.swift that you start experimenting on the subject.
GRDB is just mature enough now so that you can use it and focus of your application domain. I'm not sure one can say it leans towards Active Record, even though it does indeed makes the assumption that application use models more than raw rows, and that forcing the user to explicitly turn raw rows into models is a burden that can be avoided. Player.fetchAll(db) is simply easier to the eye than db.prepare(...).map { ... } (and I don't talk about possible optimizations such as direct access to SQLite statements for better performance, or about error handling troubles behind #569). You'll maybe give GRDB a try in a future project ;-)

@groue I'm going to start an issue over at GRDB about the syntax stuff; as I think it's an interesting subject, but don't want to clog this issue up any longer.
@jberkel I'm going to do some looking into getting RX up and running for SQLite.Swift; albeit I'm not sure how clean it will be; but it may be a jumping off point.

@cody1024d ok, great! don't worry too much, we can clean it up, it will be good to have a starting point

@jberkel
Here are my thoughts so far:
-The RxConnection will wrap around the standard connection
    -It will register itself as the update hook to it's inner connection
    -It has a prepare method (and all the methods that the connection normally has)
        -This will allow me to store the query type, etc. to perform when a change is found
The prepare method psuedo code:
        Save the passed in query, associated with a specific table (dictionary)
            -TableName to some object that holds a PublishSubject, and the passed in query.
-[String: TableListener] ? (TableListener contains a PublishSubject and the query)
        Execute the query and pass it back through the publish subject that was created before
        Return the publish subject that was just created
On the Update Hook:
Look up in the dictionary to see if there is an ObservableListener
        If there is, execute the stored query (that's bound to that ObservableListener), and publish it out through the Publish Subject
I haven't implemented any of this yet, but that's kind of my thinking on how I will go about it.  Thoughts?

Ok so I am finally getting around to implementing this.
So the above is trivial for insert/delete (atleast in the simplest cases).  The interesting part is knowing what columns have changed, given only the ability to use the Connection API (as it stands now).  Right now I'm thinking I may have to essentially "rip" the column names out of the Update struct that's passed into the Connection, before it's run.
@jberkel Any insight into a better way of potentially capturing what columns are going to be updated in the context of this library?  --This would allow me to give Rx-based observations at the column granularity, as opposed to just at the table; which would be inefficient for updates.  For now, my code will return the query results; but for every action performed on the entirety of the table, even if the query and the update are on unrelated columns.

@cody1024d great, i'll have a closer look at this soon, travelling right now with limited internet

@jberkel No problem; here is the code I'm using right now.  It is working for Inserts/Deletes based on table; although not at the column granularity I wanted.  Also you'll notice the hack to get the table name out of the query that I very much don't like :P
It could a bit more encapsulation work, to make the implementation a bit cleaner, that's my goal later today, just wanted to get a working concept together first.
class RxConnection {
    let rawConnection: Connection
    //For every table (and column set eventually) we will have a map of query to subject.  This way, we only perform the query once, even if there are multiple observers for the same query
    //One thing to think about is how do we handle someone make changes when they see a change?  This would end in an endless loop I think
    private var watchedQueries = [ChangesDefinition: [WatchableQuery: [PublishSubject<AnySequence<Row>>]]]()

    init(db: Connection) {
        self.rawConnection = db
        db.updateHook { (operation, db, tableName, rowId) in
            let changesDefinition = ChangesDefinition(tableName: tableName)
            if let queryWatchers = self.watchedQueries[changesDefinition] {
                queryWatchers.forEach { (watchableQuery, subjects) in
                    do {
                        let result = try self.rawConnection.prepare(watchableQuery.query)
                        subjects.forEach { $0.onNext(result) }
                    } catch {
                        subjects.forEach { $0.onError(error) }
                    }

                }
            }
            /**
                TODO--When an update has been done successfully
                If Update:
                    -Need to know which column was to be updated?  --> Need to store which column via the update query?
                    -How do we handle concurrency, two threads call update on this instance,
            **/
        }
    }

    public func scalar<V : Value>(_ query: ScalarQuery<V>) throws -> V {
        return try rawConnection.scalar(query)
    }

    @discardableResult public func run(_ query: Insert) throws -> Int64 {
        return try rawConnection.run(query)
    }

    @discardableResult public func run(_ query: Delete) throws -> Int {
        return try rawConnection.run(query)
    }

    @discardableResult public func run(_ query: Update) throws -> Int {
        return try rawConnection.run(query)
    }

    //TODO--Implement all methods on the Connection, and pass through to it

    public func prepare(_ query: QueryType) -> Observable<AnySequence<Row>> {
        //Does not currently handle the granularity of columns, as the update hook does not know which columns were updated
        let tableName: String = query.expression.template.slice(from: "FROM \"", to: "\"")!
        let changesDefinition = ChangesDefinition(tableName: tableName)
        if !watchedQueries.keys.contains(changesDefinition) {
            watchedQueries[changesDefinition] = [:]
        }
        let watchableQuery = WatchableQuery(query: query)
        if !(watchedQueries[changesDefinition]?.keys.contains(watchableQuery) ?? false) {
            watchedQueries[changesDefinition]?[watchableQuery] = []
        }
        let subject = PublishSubject<AnySequence<Row>>()
        watchedQueries[changesDefinition]?[watchableQuery]?.append(subject)

        let originalObs = Observable<AnySequence<Row>>.create { observer in
            do {
                observer.onNext(try self.rawConnection.prepare(query))
            } catch {
                observer.onError(error)
            }
            return Disposables.create {
                self.rawConnection.interrupt()
            }
        }

        return Observable.merge(subject, originalObs)
    }
}

struct WatchableQuery: Hashable {
    let query: QueryType

    public var hashValue: Int {
        return query.asSQL().hashValue
    }

    public static func ==(lhs: WatchableQuery, rhs: WatchableQuery) -> Bool {
        return lhs.query.asSQL() == rhs.query.asSQL()
    }
}

struct ChangesDefinition: Hashable {
    let tableName: String

    public var hashValue: Int {
        return tableName.hashValue
    }

    public static func ==(lhs: ChangesDefinition, rhs: ChangesDefinition) -> Bool {
        return lhs.tableName == rhs.tableName
    }

}

@jberkel Any input on this?  I'm using it and it has worked well for me.  Definitely can be improved, but I'd like to contribute if I can

@cody1024d how would this be integrated? as a subspec with RxSwift dependency?

Yeah, I would say so.  This way users can choose if they want the RX support or not.  We could either add it as a subspec to this library, or create another github project for it.
I think the only decision there would be or you (we) would want to support it; if we'd want issues to be lumped together and all, or not.

I'd say lets keep it in one repo, since everything is already set up there

Ok cool.  I'm coming down to the wire on the project that it's used in.  But I should have time to clean it up, and do a formal PR on it soon(ish).  I may have time after my 9-5 to do it as well, but we'll see.  We can use a formal PR to code review, etc. it, I think would be best?  I'll close this for the time being, and open the PR when I can :)

Has there been any progress on adding reactive functionality? This would just be the icing on the cake for SQLite.swift

bump

bump

@norbdev I used https://github.com/groue/GRDB.swift eventually :P


Use analyzer without default entities

In the documentation, it's mentioned that when the AnalyzerEngine is loaded, it loads all available recognizers including the NlpEngine. I was wondering if it's possible to run the analyzer without the NLP entities being detected.

Hi @hummingbird1989 you can use the ad_hoc_recognizers parameter to set which recognizers to use

Hi @hummingbird1989, if you want to create an AnalyzerEngine with no recognizers, you can instantiate the RecognizerRegistry this way:
from presidio_analyzer import AnalyzerEngine, RecognizerRegistry

registry = RecognizerRegistry()
analyzer = AnalyzerEngine(registry=registry)

This would start Presidio with no loaded recognizers. Then, you could add the ones you're interested in:
analyzer.registry.add_recognizer(my_recognizer)

Thanks, @omri374 !

@omri374 I tried the method shown above to add a custom recognizer that I built similarly to the CreditCardRecognizer()
Here's the definition for my class
class IPRecognizer(PatternRecognizer):
    def __init__(
        self,
        patterns: Optional[List[Pattern]] = None,
        context: Optional[List[Pattern]] = None,
        supported_language: str = "en",
        supported_entity: str = "IP_ADDRESS",
    ):
        patterns = patterns if patterns else self.PATTERNS
        context = (context,)
        super().__init__(
            supported_entity=supported_entity,
            patterns=patterns,
            context=context,
            supported_language=supported_language,
        )

When I try to add it to the registry, I get the following error
Traceback (most recent call last):
  File "bigcode_pii_presidio.py", line 8, in <module>
    registry.add_recognizer(IPRecognizer)
  File "C:\Anaconda2\envs\pii_detection\lib\site-packages\presidio_analyzer\recognizer_registry\recognizer_registry.py", line 193, in add_recognizer
    raise ValueError("Input is not of type EntityRecognizer")
ValueError: Input is not of type EntityRecognizer

What should I change?


Awesome books: plain JavaScript with objects.

In this PR Me and My partner we :

Build a basic website that allows users to add/remove books from a list.
Create a collection that keeps a list of books.
-Create a function to add a new book to the collection.
Create a function to add a new book to the collection.
Display all books saved in the collection at the top part of the page.

Hi @Hisoka37 and @EnixWord,
While you made a great effort in this project, unfortunately, I cannot proceed to review your code.
Invalid Code Review Request
You have submitted a project that does not have a descriptive readme file.
Your Code Review Request will be marked as invalid in your Dashboard, so please submit a new one once you are ready 🙏
Cheers and Happy coding!👏👏👏
Feel free to leave any questions or comments in the PR thread if something is not 100% clear.
Please, do not open a new Pull Request for re-reviews. You should use the same Pull Request submitted for the first review, either valid or invalid unless it is requested otherwise.

Invalid Code Review Request does not count into the code reviews limit.


Problem with background on iOS

Hi, I use the GeoLocator's position stream and send the updates to the foreground task receiveport which I listen to. On Android it works fine but on iOS it stops after about 10-15 seconds after returning to the home screen. Any ideas how to fix this?
Geolocator.getPositionStream(intervalDuration: Duration(seconds: 1)).listen((event) {
      print(event);
      sendPort!.send(event);
    });

_receivePort = receivePort;
      _receivePort!.listen((message) {
        print("RECIEVE PORT");
        print(message);
      });

Thanks anyway :)

@nicop2000
As explained by README, the iOS platform has limitations on background processing.Therefore, the task stops after 15-30 seconds in the background.
At my current level, there is no way to solve it. If there is a way to solve the problem, please provide a PR, I would appreciate it.

Did you find solution for IOS foreground.


Paginate the commit history page with query params

Motivation:
It could paginate the commit history page but a page index or a page size is not exposed in query parameters. Because these values were managed only as internal states of Javascript, sharing specific history as a URL was impossible.
Modifications:

Update Breadcrumbs to specify query parameters
Increase the width of CompareButton
Add an option to render a custom component when data is empty in DynamicDataTable
Add an option to hide a go-to button in PaginationBar.
Expose a browse button when if the path is a directory in HistoryList
Refactored HistoryListPage to use query parameters to specify revision ranges.
Use a form tag to submit the revision range with an enter key in ChangesViewPage
Link to a history page instead of a commit page in a FileEditor

Commits for a specific file must be retrieved sequentially from the HEAD revision, the Central Dogma server has a limit of 1000 if maxCommits is not specified. This means a commit for a specific file is not indexed and may not be found in the recent 1000 commits.
As we can't scan the entire history, it would be better to show the history and guide the user to find a commit rather than failing to find a commit for the file.

Result:
You can now share a specific range of commit history as a URL.


Address the high overall complexity of addon/components/sl-menu.js

https://codeclimate.com/github/softlayer/sl-ember-components/issues/categories/complexity

Resolved by https://github.com/softlayer/sl-ember-components/pull/499


Streamline updateSummary

Streamline Biographer, et al.
Streamline Biographer on updateSummary and similar commands for other bots. Aim is to remove two-step instructions to biographer intelligence, and inject the summary into the LLM by force. Talk about fine-tuning! Might be just the trick...

[ ] Incorporate real-time summary in either conversation message thread as a separate message or just include in the content the member has contrived
[ ] Update LLM instructions
[ ] Test consequences

from mylife-avatar.mjs

if(itemId){
    // @todo - check if item exists in memory, fewer pings and inclusions overall
    const { summary, } = await factory.item(itemId)
    if(summary?.length){
        message = `possible **update-summary-request**: itemId=${ itemId }\n`
        + `**current-summary-in-database**:\n`
        + summary
        + `\n**member-update-request**:\n`
        + message
        console.log('chat::update-summary-request', message)
    }
}