dns: add query cache

If dnsc_query is used on a server its possible to add a local dns resolver for caching (like unbound). But on desktop or mobile  this is not always possible. Some nameservers/resolvers (e.g. cheap routers) can't handle many dns requests. This PR introduces a query caching (takes into account the lowest resource record TTL).

Something for discussion: Should cached resource records rotate (round robin dns)? Or is it enough to wait until they expire?

does it also handle negative replies, i.e. if a record does not exist does it delete the cache entry ?
some DNS servers return a random order of multiple entries, and some return a fixed order.
it is probably good enough to wait for the entries to expire, it will rebuild the cache next time.

Negative replies are cached by SOA TTL like described here: https://datatracker.ietf.org/doc/html/rfc2308
	/**
	 * Don't cache empty RR answer if authority is also empty.
	 * Otherwise negative answer is cached with respecting the SOA TTL.
	 */
	if ((!dq.hdr.nans && !dq.hdr.nauth)) {
		mem_deref(q);
		goto out;
	}

I read a bit more about SOA TTL handling and added a better RFC 2308 check:
	/* Cache negative answer with SOA negative TTL value. (RFC 2308) */
	if (!dq.hdr.nans && dq.hdr.nauth) {
		struct dnsrr *rr = list_ledata(list_head(&q->rrlv[1]));

		if (!rr || rr->type != DNS_TYPE_SOA) {
			mem_deref(q);
			goto out;
		}

		if (rr->rdata.soa.ttlmin < CACHE_TTL_MAX)
			ttl = rr->rdata.soa.ttlmin;
	}

Is there any intention to make baresip use the caching, i.e., have possibility configure dnsc_cache_max time?

The caching is enabled by default. I can add a config option in baresip.

Sebastian Reimers writes:

The caching is enabled by default. I can add a config option in
baresip.

OK, then fine.  I thought that default is 0 and didn't find any
dnsc_cache_max() call in baresip. Now when I checked, it turned out that
default is 1800.  So I don't need it to be configurable.
-- Juha

I suspect that in my Android baresip app DNS caching is sometimes causing about 30 sec delay before INVITE request is sent out.  I don't have any proof, but I have not seen the delay after I turned caching off:
err = baresip_init(conf_config());
...
dnsc_cache_max(net_dnsc(baresip_network()), 0);

The account is not using any medianat protocol that could cause the delay.

Can you increase the DNS client debug mode?:
--- a/src/dns/client.c
+++ b/src/dns/client.c
@@ -20,7 +20,7 @@

 #define DEBUG_MODULE "dnsc"
-#define DEBUG_LEVEL 5
+#define DEBUG_LEVEL 6
 #include <re_dbg.h>

You should see all DNS queries and answers, for cached answers "ANSWER SECTION (CACHED)" is printed.

If its the cache, I can only think of a tmr related problem. Which polling method is your app using?
The value of next timer would be also interesting if this happens: https://github.com/baresip/re/blob/195ef617575638284b7b070656f5ab19ee52ccd6/src/main/main.c#L698

Sebastian Reimers writes:

If its the cache, I can only think of a tmr related problem. Which
polling method is your app using?

I don't know. I just start baresip with these calls:
runLoggingThread();
err = libre_init();
baresip_init(conf_config());
err = ua_init("baresip v" BARESIP_VERSION " (" ARCH "/" OS ")",
true, true, true);
uag_set_exit_handler(ua_exit_handler, NULL);
err = uag_event_register(ua_event_handler, NULL);
err = message_listen(baresip_message(), message_handler, NULL);
err = conf_modules();
err = mqueue_alloc(&mq, mqueue_handler, NULL);
err = re_main(signal_handler);

I set debug level to 6 and made call to<EMAIL_ADDRESS> It produced:
06-05 18:35:50.790 29862 29862 D Baresip Lib: ua event (CALL_OUTGOING<EMAIL_ADDRESS>06-05 18:35:50.880 29862 30043 D Baresip Lib: dnsc: --- QUESTION SECTION id: 33619 ---
06-05 18:35:50.880 29862 30043 D Baresip Lib: dnsc: box.tutpro.com.	IN	A
06-05 18:35:50.880 29862 30043 D Baresip Lib: dnsc: --- QUESTION SECTION id: 20131 ---
06-05 18:35:50.880 29862 30043 D Baresip Lib: dnsc: box.tutpro.com.	IN	AAAA
06-05 18:35:50.880 29862 29862 D Baresip Lib: ua event (CALL_LOCAL_SDP) offer
********** after 51 second delay INVITE was sent out
06-05 18:36:41.629 29862 30043 D Baresip Lib: dnsc: --- ANSWER SECTION (CACHED) id: 33619 ---
06-05 18:36:41.629 29862 30043 D Baresip Lib: dnsc: box.tutpro.com.                 3179 IN   A       <IP_ADDRESS>
06-05 18:36:41.629 29862 30043 D Baresip Lib: dnsc: --- ANSWER SECTION (CACHED) id: 20131 ---
06-05 18:36:41.629 29862 30043 D Baresip Lib: dnsc: box.tutpro.com.                 3179 IN   AAAA    2001:67c:224:666::26
06-05 18:36:41.651 29862 30043 D Baresip Lib: dnsc: ttl cache delete (id: 14063)
06-05 18:36:41.654 29862 30043 D Baresip Lib: proxy.sipthor.net.	 IN AAAA
******** and baresip got response back
06-05 18:36:41.715 29862 30043 D Baresip Lib: call: SIP Progress: 100 Trying (/)

Before the above call the calling UA registers and then I see these messages in log:
06-05 18:35:41.772 29862 30043 D Baresip Lib: dnsc: --- QUESTION SECTION id: 2407 ---
06-05 18:35:41.772 29862 30043 D Baresip Lib: dnsc: box.tutpro.com.	IN	A
06-05 18:35:41.772 29862 30043 D Baresip Lib: dnsc: --- QUESTION SECTION id: 7416 ---
06-05 18:35:41.772 29862 30043 D Baresip Lib: dnsc: box.tutpro.com.	IN	AAAA
06-05 18:35:41.772 29862 30043 D Baresip Lib: dnsc: --- ANSWER SECTION (CACHED) id: 2407 ---
06-05 18:35:41.772 29862 30043 D Baresip Lib: dnsc: box.tutpro.com.
06-05 18:35:41.772 29862 30043 D Baresip Lib:     3179 IN  A     <IP_ADDRESS>
06-05 18:35:41.772 29862 30043 D Baresip Lib: dnsc: --- ANSWER SECTION (CACHED) id:
06-05 18:35:41.772 29862 30043 D Baresip Lib: 7416 ---
06-05 18:35:41.772 29862 30043 D Baresip Lib: dnsc: box.tutpro.com.
06-05 18:35:41.772 29862 30043 D Baresip Lib:   3179 IN
06-05 18:35:41.772 29862 30043 D Baresip Lib: AAAA
06-05 18:35:41.772 29862 30043 D Baresip Lib: 2001:67c:224:666::26

So during register, DNS answer comes immediately, but takes long time during invite.  Register domain is the same as invite request uri domain (tutpro.com) and UA's outbout is set to host box.tutpro.com, which means that both register and invite is sent to the same destination.

So if I understand the log messages correctly, if after 51 delay the answer still comes from cache, why doesn't it come from there immediately?

why doesn't it come from there immediately?

I would guess the tmr is called lately because DEBUG_INFO("next timer: %llu ms\n", to);  is wrong, or if this is right the poll wait does not respect the timeout correctly. You can test if a 10ms tmr works better:
--- a/src/dns/client.c
+++ b/src/dns/client.c
@@ -759,7 +759,7 @@ static bool query_cache_handler(struct dns_query *q)
        hash_unlink(&q->le);
        list_append(&q->dnsc->hdl_cache, &q->le_hdl, q);

-       tmr_start(&q->dnsc->hdl_tmr, 0, hdl_tmr_cache, &q->dnsc->hdl_cache);
+       tmr_start(&q->dnsc->hdl_tmr, 10, hdl_tmr_cache, &q->dnsc->hdl_cache);

        return true;
 }

If its the cache, I can only think of a tmr related problem. Which polling method is your app using?
I don't know. I just start baresip with these calls:

You can call re_debug() for getting the current poll method name.

I tried with
tmr_start(&q->dnsc->hdl_tmr, 10, hdl_tmr_cache, &q->dnsc->hdl_cache);

and the delay is still there.
I called re_debug() at CALL_OUTGOING event and got:
06-05 21:48:54.341   593   593 D Baresip Lib: re main loop:
06-05 21:48:54.341   593   593 D Baresip Lib:   maxfds:  1024
06-05 21:48:54.341   593   593 D Baresip Lib:   nfds:    107
06-05 21:48:54.341   593   593 D Baresip Lib:   method:  3 (epoll)

I don't understand what timers could have to do with getting DNS query answer from the cache.

I don't understand what timers could have to do with getting DNS query answer from the cache.

The cached answer has to be async and comes from a tmr instead a network callback. So it looks like the tmr is called lately. Timer callbacks are called from the main loop and can only be late, if:

The next tmr timeout is wrong
Or something blocks the main loop

Can you reproduce this on linux or only on android?

It seems to work OK on Linux.

But never mind, I can turn the cache off.

My Android app does not make any re tmr related calls. But could threads have something to do with this issue?
Android app user interface runs on UI thread.  When user makes a call, the app calls native (C) function
JNIEXPORT jlong JNICALL
Java_com_tutpro_baresip_Api_ua_1call_1alloc(JNIEnv *env, jobject thiz, jlong ua, jlong xCall, jint vidMode)
{
    struct call *call = NULL;
    int err;
    re_thread_enter();
    err = ua_call_alloc(&call, (struct ua *)ua, (enum vidmode)vidMode, NULL, (struct call *)xCall,
            call_localuri((struct call *)xCall), true);
    re_thread_leave();
    if (err)
        LOGW("call allocation for ua %ld failed with error %d\n", (long)ua, err);
    return (jlong)call;
}

As shown, it calls re_thread_enter() before calling API function ua_call_alloc() so that the API function is executed on re thread instead of UI thread.

But never mind, I can turn the cache off.

Sure, but I'm a little worried about the fact, that such a simple timer callback is broken (works not reliable) on android.
Can you try these patches:
--- a/src/main/main.c
+++ b/src/main/main.c
@@ -695,7 +695,7 @@ static int fd_poll(struct re *re)
        fd_set rfds, wfds, efds;
 #endif

-       DEBUG_INFO("next timer: %llu ms\n", to);
+       DEBUG_NOTICE("next timer: %llu ms\n", to);

        /* Wait for I/O */
        switch (re->method) {

--- a/src/dns/client.c
+++ b/src/dns/client.c
@@ -20,7 +20,7 @@

 #define DEBUG_MODULE "dnsc"
-#define DEBUG_LEVEL 5
+#define DEBUG_LEVEL 6
 #include <re_dbg.h>

@@ -761,6 +761,8 @@ static bool query_cache_handler(struct dns_query *q)
        tmr_start(&q->dnsc->hdl_tmr, 0, hdl_tmr_cache, &q->dnsc->hdl_cache);

+       DEBUG_NOTICE("%H\n", tmr_debug);
+
        return true;
 }

As shown, it calls re_thread_enter() before calling API function ua_call_alloc() so that the API function is executed on re thread instead of UI thread.

Should normally be fine, since tmr_poll should be executed after you call re_thread_leave, but maybe there is a race condition I don't see yet.

Here is test call logging with the above debugs.
06-06 09:31:40.564  5471  5471 D Baresip Lib: ua event (CALL_OUTGOING<EMAIL_ADDRESS>06-06 09:31:40.620  5471  5647 D Baresip Lib: dnsc: --- QUESTION SECTION id: 33977 ---
06-06 09:31:40.620  5471  5647 D Baresip Lib: dnsc: box.tutpro.com.	IN	A
06-06 09:31:40.620  5471  5647 D Baresip Lib: dnsc: Timers (21):
06-06 09:31:40.620  5471  5647 D Baresip Lib:   0x78fe55f028: th=0x7
06-06 09:31:40.620  5471  5647 D Baresip Lib: 89e1c7098 expire=0ms file=src/dns/client.c:762
06-06 09:31:40.620  5471  5647 D Baresip Lib:   0x78a0c93c38: th=0x
06-06 09:31:40.620  5471  5647 D Baresip Lib: 789e188f34 expire=42ms file=src/metric.c
06-06 09:31:40.620  5471  5647 D Baresip Lib: :65
06-06 09:31:40.620  5471  5647 D Baresip Lib:
06-06 09:31:40.620  5471  5647 D Baresip Lib: 0x78a0c93d20: th=0x789e188f34
06-06 09:31:40.620  5471  5647 D Baresip Lib:  expire=42ms file=src/metric.c:
06-06 09:31:40.620  5471  5647 D Baresip Lib: 65
06-06 09:31:40.620  5471  5647 D Baresip Lib:   0x78fc621a98
06-06 09:31:40.620  5471  5647 D Baresip Lib: : th=0x789e1a8814 expire=58269
06-06 09:31:40.620  5471  5647 D Baresip Lib: ms file=src/sip/transp.c:309
06-06 09:31:40.620  5471  5647 D Baresip Lib:   0x78fc621798: th=0x
06-06 09:31:40.620  5471  5647 D Baresip Lib: 789e1a8814 expire=60414ms file=src/sip/transp.c
06-06 09:31:40.620  5471  5647 D Baresip Lib: :309
06-06 09:31:40.620  5471  5647 D Baresip Lib:
06-06 09:31:40.620  5471  5647 D Baresip Lib: 0x78fc621918: th=0x
06-06 09:31:40.620  5471  5647 D Baresip Lib: 789e1a8814 expire=71247ms file=src/sip/transp.c
06-06 09:31:40.620  5471  5647 D Baresip Lib: :309
06-06 09:31:40.620  5471  5647 D Baresip Lib:   0x
06-06 09:31:40.620  5471  5647 D Baresip Lib: 78fe5b0078: th=0x789e1ab7c8 expire=
06-06 09:31:40.620  5471  5647 D Baresip Lib: 390670ms file=src/sipevent/subscribe.c:
06-06 09:31:40.620  5471  5647 D Baresip Lib: 141
06-06 09:31:40.620  5471  5647 D Baresip Lib:   0x
06-06 09:31:40.620  5471  5647 D Baresip Lib: 78fe5b07f8: th=0x789e1ab7c8 expire=391043ms file=src/sipevent/subscribe.c:141
06-06 09:31:40.620  5471  5647 D Baresip Lib:   0x78fc6d7a38: th=
06-06 09:31:40.620  5471  5647 D Baresip Lib: 0x789e1ad0b8 expire=570638ms file=
06-06 09:31:40.620  5471  5647 D Baresip Lib: src/sipreg/reg.c:263
06-06 09:31:40.620  5471  5647 D Baresip Lib:
06-06 09:31:40.620  5471  5647 D Baresip Lib: 0x78fc6d7c38: th=0x
06-06 09:31:40.620  5471  5647 D Baresip Lib: 789e1ad0b8 expire=570895ms file=
06-06 09:31:40.620  5471  5647 D Baresip Lib: src/sipreg/reg.c:263
06-06 09:31:40.620  5471  5647 D Baresip Lib:   0x
06-06 09:31:40.620  5471  5647 D Baresip Lib: 78fc6d7d38: th=0x789e1ad0b8
06-06 09:31:40.620  5471  5647 D Baresip Lib:  expire=571046ms file=src/sipreg/reg.c:
06-06 09:31:40.620  5471  5647 D Baresip Lib: 263
06-06 09:31:40.620  5471  5647 D Baresip Lib:   0x
06-06 09:31:40.620  5471  5647 D Baresip Lib: 7895be5a38: th=0x789e1ad0b8
06-06 09:31:40.620  5471  5647 D Baresip Lib:  expire=631034ms file=src/sipreg/reg.c:
06-06 09:31:40.620  5471  5647 D Baresip Lib: 263
06-06 09:31:40.620  5471  5647 D Baresip Lib:   0x
06-06 09:31:40.620  5471  5647 D Baresip Lib: 78fe5fe950: th=0x789e1a90f8 expire=
06-06 09:31:40.620  5471  5647 D Baresip Lib: 751046ms file=src/sip/transp.c:526
06-06 09:31:40.620  5471  5647 D Baresip Lib:
06-06 09:31:40.620  5471  5647 D Baresip Lib:   0x78fc621a50: th=
06-06 09:31:40.620  5471  5647 D Baresip Lib: 0x789e1a90f8 expire=849043
06-06 09:31:40.620  5471  5647 D Baresip Lib: ms file=src/sip/transp.c:474
06-06 09:31:40.620  5471  5647 D Baresip Lib:   0x78fc6218d0: th=0x
06-06 09:31:40.620  5471  5647 D Baresip Lib: 789e1a90f8 expire=856956ms file=src/sip/transp.c
06-06 09:31:40.620  5471  5647 D Baresip Lib: :474
06-06 09:31:40.620  5471  5647 D Baresip Lib:
06-06 09:31:40.620  5471  5647 D Baresip Lib: 0x78fc6afd10: th=0x789e1c7c08
06-06 09:31:40.620  5471  5647 D Baresip Lib:  expire=859583ms file=src/dns/client.c:350
06-06 09:31:40.620  5471  5647 D Baresip Lib:
06-06 09:31:40.620  5471  5647 D Baresip Lib:   0x78fc6afed0
06-06 09:31:40.620  5471  5647 D Baresip Lib: : th=0x789e1c7c08 expire=
06-06 09:31:40.620  5471  5647 D Baresip Lib: 859587ms file=src/dns/client.c:350
06-06 09:31:40.620  5471  5647 D Baresip Lib:
06-06 09:31:40.620  5471  5647 D Baresip Lib:   0x78fc621750
06-06 09:31:40.620  5471  5647 D Baresip Lib: : th=0x789e1a90f8 expire=864336
06-06 09:31:40.620  5471  5647 D Baresip Lib: ms file=src/sip/transp.c:474
06-06 09:31:40.620  5471  5647 D Baresip Lib:   0x78fc6af990: th=
06-06 09:31:40.620  5471  5647 D Baresip Lib: 0x789e1c7c08 expire=880580
06-06 09:31:40.621  5471  5647 D Baresip Lib: ms file=src/dns/client.c:350
06-06 09:31:40.621  5471  5647 D Baresip Lib:   0x78fc6b0090: th=
06-06 09:31:40.621  5471  5647 D Baresip Lib: 0x789e1c7c08 expire=880587ms file=src/dns/client.c:
06-06 09:31:40.621  5471  5647 D Baresip Lib: 350
06-06 09:31:40.621  5471  5647 D Baresip Lib:   0x78fc6b0250
06-06 09:31:40.621  5471  5647 D Baresip Lib: : th=0x789e1c7c08 expire=
06-06 09:31:40.621  5471  5647 D Baresip Lib: 880591ms file=src/dns/client.c:350
06-06 09:31:40.621  5471  5647 D Baresip Lib:
06-06 09:31:40.621  5471  5647 D Baresip Lib:
06-06 09:31:40.621  5471  5647 D Baresip Lib: dnsc: --- QUESTION SECTION id: 34226 ---
06-06 09:31:40.621  5471  5647 D Baresip Lib: dnsc: box.tutpro.com.	IN
06-06 09:31:40.621  5471  5647 D Baresip Lib: 	AAAA
06-06 09:31:40.621  5471  5647 D Baresip Lib:
06-06 09:31:40.621  5471  5647 D Baresip Lib:
06-06 09:31:40.621  5471  5647 D Baresip Lib: dnsc: Timers (21
06-06 09:31:40.621  5471  5647 D Baresip Lib: ):
06-06 09:31:40.621  5471  5647 D Baresip Lib:   0x78fe55f028
06-06 09:31:40.621  5471  5647 D Baresip Lib: : th=0x789e1c7098 expire=
06-06 09:31:40.621  5471  5647 D Baresip Lib: 0ms file=src/dns/client.c:762
06-06 09:31:40.621  5471  5647 D Baresip Lib:
06-06 09:31:40.621  5471  5647 D Baresip Lib:   0x78a0c93c38
06-06 09:31:40.621  5471  5647 D Baresip Lib: : th=0x789e188f34 expire=41
06-06 09:31:40.621  5471  5647 D Baresip Lib: ms file=src/metric.c:65
06-06 09:31:40.621  5471  5647 D Baresip Lib:   0x78a0c93d20: th=0x
06-06 09:31:40.621  5471  5647 D Baresip Lib: 789e188f34 expire=41ms file=
06-06 09:31:40.621  5471  5647 D Baresip Lib: src/metric.c:65
06-06 09:31:40.621  5471  5647 D Baresip Lib:   0x78fc621a98: th=0x
06-06 09:31:40.621  5471  5647 D Baresip Lib: 789e1a8814 expire=58268ms file=src/sip/transp.c
06-06 09:31:40.621  5471  5647 D Baresip Lib: :309
06-06 09:31:40.621  5471  5647 D Baresip Lib:   0x
06-06 09:31:40.621  5471  5647 D Baresip Lib: 78fc621798: th=0x789e1a8814 expire=60413
06-06 09:31:40.621  5471  5647 D Baresip Lib: ms file=src/sip/transp.c:309
06-06 09:31:40.621  5471  5647 D Baresip Lib:   0x78fc621918: th=0x
06-06 09:31:40.621  5471  5647 D Baresip Lib: 789e1a8814 expire=71246ms file=
06-06 09:31:40.621  5471  5647 D Baresip Lib: src/sip/transp.c:309
06-06 09:31:40.621  5471  5647 D Baresip Lib:
06-06 09:31:40.621  5471  5647 D Baresip Lib: 0x78fe5b0078: th=0x
06-06 09:31:40.621  5471  5647 D Baresip Lib: 789e1ab7c8 expire=390669ms file=src/sipevent/subscribe.c
06-06 09:31:40.621  5471  5647 D Baresip Lib: :141
06-06 09:31:40.621  5471  5647 D Baresip Lib:   0x78fe5b07f8: th=
06-06 09:31:40.621  5471  5647 D Baresip Lib: 0x789e1ab7c8 expire=391042ms file=
06-06 09:31:40.621  5471  5647 D Baresip Lib: src/sipevent/subscribe.c:141
06-06 09:31:40.621  5471  5647 D Baresip Lib:
06-06 09:31:40.621  5471  5647 D Baresip Lib: 0x78fc6d7a38: th=0x
06-06 09:31:40.621  5471  5647 D Baresip Lib: 789e1ad0b8 expire=570637ms file=src/sipreg/reg.c
06-06 09:31:40.621  5471  5647 D Baresip Lib: :263
06-06 09:31:40.621  5471  5647 D Baresip Lib:
06-06 09:31:40.621  5471  5647 D Baresip Lib: 0x78fc6d7c38: th=0x789e1ad0b8
06-06 09:31:40.621  5471  5647 D Baresip Lib:  expire=570894ms file=src/sipreg/reg.c:263
06-06 09:31:40.621  5471  5647 D Baresip Lib:
06-06 09:31:40.621  5471  5647 D Baresip Lib:   0x78fc6d7d38: th=
06-06 09:31:40.621  5471  5647 D Baresip Lib: 0x789e1ad0b8 expire=571045ms file=
06-06 09:31:40.621  5471  5647 D Baresip Lib: src/sipreg/reg.c:263
06-06 09:31:40.621  5471  5647 D Baresip Lib:   0x7895be5a38: th=0x
06-06 09:31:40.621  5471  5647 D Baresip Lib: 789e1ad0b8 expire=631033ms file=
06-06 09:31:40.621  5471  5647 D Baresip Lib: src/sipreg/reg.c:263
06-06 09:31:40.621  5471  5647 D Baresip Lib:   0x78fe5fe950: th=0x
06-06 09:31:40.621  5471  5647 D Baresip Lib: 789e1a90f8 expire=751045ms file=
06-06 09:31:40.621  5471  5647 D Baresip Lib: src/sip/transp.c:526
06-06 09:31:40.621  5471  5647 D Baresip Lib:
06-06 09:31:40.621  5471  5647 D Baresip Lib: 0x78fc621a50: th=0x789e1a90f8
06-06 09:31:40.621  5471  5647 D Baresip Lib:  expire=849042ms file=src/sip/transp.c
06-06 09:31:40.621  5471  5647 D Baresip Lib: :474
06-06 09:31:40.621  5471  5647 D Baresip Lib:   0x
06-06 09:31:40.621  5471  5647 D Baresip Lib: 78fc6218d0: th=0x789e1a90f8 expire=
06-06 09:31:40.621  5471  5647 D Baresip Lib: 856955ms file=src/sip/transp.c:474
06-06 09:31:40.621  5471  5647 D Baresip Lib:
06-06 09:31:40.621  5471  5647 D Baresip Lib:   0x78fc6afd10: th=
06-06 09:31:40.621  5471  5647 D Baresip Lib: 0x789e1c7c08 expire=859582
06-06 09:31:40.621  5471  5647 D Baresip Lib: ms file=src/dns/client.c:350
06-06 09:31:40.621  5471  5647 D Baresip Lib:
06-06 09:31:40.621  5471  5647 D Baresip Lib:   0x78fc6afed0: th=
06-06 09:31:40.621  5471  5647 D Baresip Lib: 0x789e1c7c08 expire=859586
06-06 09:31:40.621  5471  5647 D Baresip Lib: ms file=src/dns/client.c:350
06-06 09:31:40.621  5471  5647 D Baresip Lib:   0x78fc621750: th=0x
06-06 09:31:40.621  5471  5647 D Baresip Lib: 789e1a90f8 expire=864335ms file=src/sip/transp.c
06-06 09:31:40.621  5471  5647 D Baresip Lib: :474
06-06 09:31:40.621  5471  5647 D Baresip Lib:
06-06 09:31:40.621  5471  5647 D Baresip Lib: 0x78fc6af990: th=0x789e1c7c08
06-06 09:31:40.621  5471  5647 D Baresip Lib:  expire=880579ms file=src/dns/client.c:
06-06 09:31:40.621  5471  5647 D Baresip Lib: 350
06-06 09:31:40.621  5471  5647 D Baresip Lib:   0x
06-06 09:31:40.621  5471  5647 D Baresip Lib: 78fc6b0090: th=0x789e1c7c08 expire=
06-06 09:31:40.621  5471  5647 D Baresip Lib: 880586ms file=src/dns/client.c:350
06-06 09:31:40.621  5471  5647 D Baresip Lib:
06-06 09:31:40.621  5471  5647 D Baresip Lib:   0x78fc6b0250
06-06 09:31:40.621  5471  5647 D Baresip Lib: : th=0x789e1c7c08 expire=880590
06-06 09:31:40.621  5471  5647 D Baresip Lib: ms file=src/dns/client.c:350
06-06 09:31:40.621  5471  5647 D Baresip Lib:
06-06 09:31:40.621  5471  5471 D Baresip Lib: ua event (CALL_LOCAL_SDP) offer
06-06 09:32:38.921  5471  5647 D Baresip Lib: dnsc: --- ANSWER SECTION (CACHED) id: 33977 ---
06-06 09:32:38.921  5471  5647 D Baresip Lib: dnsc: box.tutpro.com.                 1009 IN   A       192.26.11
06-06 09:32:38.921  5471  5647 D Baresip Lib: 1.38
06-06 09:32:38.921  5471  5647 D Baresip Lib: dnsc: --- ANSWER SECTION (CACHED) id: 34226 ---
06-06 09:32:38.921  5471  5647 D Baresip Lib: dnsc: box.tutpro.com.                 1009 IN   AAAA
06-06 09:32:38.921  5471  5647 D Baresip Lib: 2001:67c:224:666::26
06-06 09:32:38.921  5471  5647 D Baresip Lib:
06-06 09:32:38.933  5471  5647 D Baresip Lib:
06-06 09:32:38.935  5471  5647 D Baresip Lib: main: next timer:
06-06 09:32:38.935  5471  5647 D Baresip Lib: 2101
06-06 09:32:38.936  5471  5647 D Baresip Lib:  ms
06-06 09:32:38.937  5471  5647 D Baresip Lib:
06-06 09:32:39.034  5471  5647 D Baresip Lib: call: SIP Progress: 100 Trying (/)

Regarding re_thread_enter/leave calls, I double checked and every re_thread_enter() call is matched by re_thread_leave() call.

Ok thanks for your patience, maybe its another tmr that causes the delay, can you try this patch:
--- a/src/tmr/tmr.c
+++ b/src/tmr/tmr.c
@@ -27,12 +27,13 @@

 #define DEBUG_MODULE "tmr"
 #define DEBUG_LEVEL 5
+#define TMR_DEBUG 1
 #include <re_dbg.h>

 /** Timer values */
 enum {
-       MAX_BLOCKING = 500   /**< Maximum time spent in handler [ms] */
+       MAX_BLOCKING = 100   /**< Maximum time spent in handler [ms] */
 };

 extern struct list *tmrl_get(void);
@@ -57,7 +58,7 @@ static bool inspos_handler_0(struct le *le, void *arg)

 #if TMR_DEBUG
-static void call_handler(tmr_h *th, void *arg)
+static void call_handler(struct tmr *tmr, tmr_h *th, void *arg)
 {
        const uint64_t tick = tmr_jiffies();
        uint32_t diff;
@@ -68,8 +69,9 @@ static void call_handler(tmr_h *th, void *arg)
        diff = (uint32_t)(tmr_jiffies() - tick);

        if (diff > MAX_BLOCKING) {
-               DEBUG_WARNING("long async blocking: %u>%u ms (h=%p arg=%p)\n",
-                             diff, MAX_BLOCKING, th, arg);
+               DEBUG_WARNING(
+                       "long async blocking: %u>%u ms %s:%d (h=%p arg=%p)\n",
+                       diff, MAX_BLOCKING, tmr->file, tmr->line, th, arg);
        }
 }
 #endif
@@ -106,7 +108,7 @@ void tmr_poll(struct list *tmrl)
                        continue;

 #if TMR_DEBUG
-               call_handler(th, th_arg);
+               call_handler(tmr, th, th_arg);
 #else
                th(th_arg);
 #endif

I tried with the above patch and no warnings were produced (below) during the call nor before that from the start of the app.   Since next timer (from fd_poll) is logged after the cached answer, it looks like it got stuck in tmr_poll, but if there is no long async blocking logged, I'm clueless at the moment. I have to think about this.

hi,
please note that if you call tmr_start from a different thread and the thread running re_main,
it will not "wake up" the sleeping re_main (for example epoll).
in order to test this you can add a new dummy timer that is restarted
e.g. every 1 second. If the dns works with this and the delay is around 1 second,
then you will know that that is the case.
another test is to add a call to mqueue_send at the same time as the timer is started,
this will force the re_main to "wake up".
There is a patch somewhere that checks from which thread the fd_listen and tmr_start
is called, perhaps find it and apply it for local testing ?

Hope that I didn't miss anything in the patches.  Below is diff that I have been using.
diff --git a/src/dns/client.c b/src/dns/client.c
index 88343a8..9959234 100644
--- a/src/dns/client.c
+++ b/src/dns/client.c
@@ -20,7 +20,7 @@

 #define DEBUG_MODULE "dnsc"
-#define DEBUG_LEVEL 5
+#define DEBUG_LEVEL 6
 #include <re_dbg.h>

@@ -761,6 +761,8 @@ static bool query_cache_handler(struct dns_query *q)

 	tmr_start(&q->dnsc->hdl_tmr, 0, hdl_tmr_cache, &q->dnsc->hdl_cache);

+	DEBUG_NOTICE("%H\n", tmr_debug);
+
 	return true;
 }

diff --git a/src/main/main.c b/src/main/main.c
index 023759e..f15fd01 100644
--- a/src/main/main.c
+++ b/src/main/main.c
@@ -695,7 +695,7 @@ static int fd_poll(struct re *re)
 	fd_set rfds, wfds, efds;
 #endif

-	DEBUG_INFO("next timer: %llu ms\n", to);
+	DEBUG_NOTICE("next timer: %llu ms\n", to);

 	/* Wait for I/O */
 	switch (re->method) {
diff --git a/src/tmr/tmr.c b/src/tmr/tmr.c
index 3ee28b9..2b4be50 100644
--- a/src/tmr/tmr.c
+++ b/src/tmr/tmr.c
@@ -27,12 +27,13 @@

 #define DEBUG_MODULE "tmr"
 #define DEBUG_LEVEL 5
+#define TMR_DEBUG 1
 #include <re_dbg.h>

 /** Timer values */
 enum {
-	MAX_BLOCKING = 500   /**< Maximum time spent in handler [ms] */
+	MAX_BLOCKING = 100   /**< Maximum time spent in handler [ms] */
 };

 extern struct list *tmrl_get(void);
@@ -57,7 +58,7 @@ static bool inspos_handler_0(struct le *le, void *arg)

 #if TMR_DEBUG
-static void call_handler(tmr_h *th, void *arg)
+static void call_handler(struct tmr *tmr, tmr_h *th, void *arg)
 {
 	const uint64_t tick = tmr_jiffies();
 	uint32_t diff;
@@ -68,8 +69,8 @@ static void call_handler(tmr_h *th, void *arg)
 	diff = (uint32_t)(tmr_jiffies() - tick);

 	if (diff > MAX_BLOCKING) {
-		DEBUG_WARNING("long async blocking: %u>%u ms (h=%p arg=%p)\n",
-			      diff, MAX_BLOCKING, th, arg);
+		DEBUG_WARNING("long async blocking: %u>%u ms %s:%d (h=%p arg=%p)\n",
+			      diff, MAX_BLOCKING, tmr->file, tmr->line, th, arg);
 	}
 }
 #endif
@@ -106,7 +107,7 @@ void tmr_poll(struct list *tmrl)
 			continue;

 #if TMR_DEBUG
-		call_handler(th, th_arg);
+		call_handler(tmr, th, th_arg);
 #else
 		th(th_arg);
 #endif

Just one comment regarding timers and wake ups. On mobile devices low
battery consumption is VERY important.  So please avoid all absolutely
not necessary timer wake ups and everything else unnecessary CPU usage.
So far baresip has done well compared to "competition" and I would like
to keep it that way.

@juha-h can you verify if this fixes your DNS Cache timer problems: https://github.com/baresip/re/commit/75001eda3e83f24247b4310588a107d83d02dd7a

No it didn't.  Invite delay is still very long if I have DNS name in my outbound proxy URI.

Forgot to mention, you need to initialize (re_thread_async_init) the new thread worker first, like here:
https://github.com/baresip/baresip/commit/d743e912796bf15fcd72f960e3e671e1ae016a8a

Looks like it helped in some cases, for example, calls via my outbound proxy now work fast, but when I try to call e.g. sip2sip.info directly w/o outbound proxy, the call times out.

there will probably be some functions in the future that depend on it (currently only async gettaddrinfo).

I made more tests with dns cache on and noticed that there is INVITE delay to sip2sip.info only when tls is used.  So the delay is not DNS query related.

Sebastian Reimers writes:

I don't understand what timers could have to do with getting DNS
query answer from the cache.

The cached answer has to be async and comes from a tmr instead a
network callback. So it looks like the tmr is called lately. Timer
callbacks are called from the main loop and can only be late, if:

The next tmr timeout is wrong
Or something blocks the main loop

Can you reproduce this on linux or only on android?

It seems to work OK on Linux.
Regarding timers on Android, re-register interval has never worked on
Android. I have had make this patch to make baresip register at 12
minuted interval when reg_int is 3600:
diff --git a/src/sipreg/reg.c b/src/sipreg/reg.c
index f787624..b929ce9 100644
--- a/src/sipreg/reg.c
+++ b/src/sipreg/reg.c
@@ -375,7 +375,7 @@ static int vsipreg_alloc(struct sipreg **regp, struct sip *sip,
reg->sip     = mem_ref(sip);
reg->expires = expires;

reg->rwait   = 90;

reg->rwait   = 20;
reg->resph   = resph ? resph : dummy_handler;
reg->arg     = arg;
reg->regid   = regid;

But this has nothing to do with the DNS query cache. Right?
Why should we set 20% as default? 90% should be okay.
Our testers found that in some situations (TLS) the expiry parameter in the Contact header of the response to the REGISTER is not used correctly. I suggest that the parsing of the contact header does not work correctly. But had no time so far to verify this.

My comment was done long ago via email and now appeared (like many others). No, nothing to do with DNS query cache and recently I removed the above hack.  It is not needed anymore (don't know why).  I have ;regint=900 and have been following registrations in the registrar syslog.  I have not found any misses.
BUT one thing I did notice from the logs: my registrar removes registration if tcp or tls connection to the UA gets closed.  When baresip then reconnects, it does not re-REGISTER automatically after that, but waits for regint to expire.

Is it possible via some API function to dump contents of DNS cache?

Is it possible via some API function to dump contents of DNS cache?

At the moment not. If I have some spare time I can add one, in the meantime maybe dnsc debug mode helps:

--- a/src/dns/client.c
+++ b/src/dns/client.c
@@ -20,7 +20,7 @@

 #define DEBUG_MODULE "dnsc"
-#define DEBUG_LEVEL 5
+#define DEBUG_LEVEL 6
 #include <re_dbg.h>

You should see all DNS queries and answers, for cached answers "ANSWER SECTION (CACHED)" is printed.

I noticed in Android docs, that Android automatically caches DNS query
results on behalf of all apps.

But only if you use, dns_getaddrinfo, otherwise baresip directly use the configured nameserver.

I'll turn on dns_getaddrinfo in my Android app for testing what the effect is (if any).


Unknown property,   source: 'css'

VSCode Version: Code 1.14.2 (cb82febafda0c8c199b9201ad274e25d9a76874e, 2017-07-19T23:34:09.706Z)
OS Version: Windows_NT ia32 6.1.7601
Extensions:

Extension
Author (truncated)
Version

aessoft-class-autocomplete
AES
0.1.0

open-html-in-browser
cod
0.1.21

githistory
don
0.2.2

vscode-npm-script
eg2
0.2.0

beautify
Hoo
1.1.1

autofilename
Jer
0.1.3

LiveServer
rit
1.6.7

vscode-icons
rob
7.12.0

Steps to Reproduce:
text-decoration-color  -  write message: 'Unknown property.'  source: 'css'

Reproduces without extensions: Yes/No

@Andrei079 pls provide steps that allow us to reproduce the problem.

I edited the file( .CSS) in the webstorm (EAP version). There were no mistakes. When I opened an error message in the VS. Thank you.

@Andrei079 thanks! This is fixed in the latest insider builds and will be in the next stable.


[CodeCompletion] Fast completion inside function builder function

PreCheckFunctionBuilderRequest applies PreCheckExpression to the expressions inside the function body. Previously it used to receive only AnyFunctionRef (FunctionDecl or ClosureExpr) as the parameter. However, when fast-completion kicks-in, it replaces the body of the function, then tries to call PreCheckFunctionBuilderRequest again, with the same function decl as before. It used to return cached "Success" result, but it didn't actually apply PreCheckExpression. So any UnresolvedDeclRefExpr remained unresolved.
In this patch, make PreCheckFunctionBuilderRequest receive "body" of the function as well, so it doesn't return the cached result for the previous body.
rdar://problem/65692922

@swift-ci Please smoke test

@CodaFi Is there a better way to differentiate request evaluator keys, for example, manually creating the key from the arguments?

When we need to create custom keys, I've taken to using the Descriptor pattern, which will let you define your own custom operator== and hashing primitives. IRGenDescriptor is a good example of going the other way and defining these in terms of less data than you're given. You just want that pattern but with more data.


No base quality data in extracted subreads?

After extracting subreads from a PB bamfile (latest Sequel chemistry) using bam2fastq, I've ended up with a fastq file where all phred scores are "!" [!=0].
Is this an error, or is there actually no base quality data stored in the subreads.bam?

You are correct, there are no base-level qualities in the raw subreads.


⚠ Bofrost.fr has degraded performance

In fefaeb6, Bofrost.fr (https://www.bofrost.fr) experienced degraded performance:

HTTP code: 200
Response time: 919 ms

Resolved: Bofrost.fr performance has improved in 3e58c80.


Fundamental API design flaw -- with respect to encoding and serialization

Expected behavior
The stated purpose of the library behaves as documented and expected -- i.e. users should not need to be aware of implementation and it should "just work".   One stated goal/priority for reddison is that users can use existing or new java classes 'As Is' and simply use reddison as a transparent native object database.   This is a very compelling feature (which has been attempted many times over the last 30+ years).  Considering the maturity and feature rich nature of the product I nievely assumed that this would 'just work' as intended (of course accepting some edge cases that are problematic.)
In particular, with the variety of codecs, including Jackson -- which is if not the single most used JSON data mapper/serialization library, atleast in the top 3, that if a class could round-trip serialize with Jackson that class would be equally usable with Reddison -- this is explicitly stated (with some caveots).
By leveraging the work of existing encoders (like jackson) it is natural to assume that this is not magic -- it would work 'as good as' whatever encoder was configured, so if one has domain knowlege in that encoder and has created classes which behave nicely with that encoder then Reddison would indeed be a transparent 'Drop In' -- and be able to use the the other really awesome features provided.
Actual behavior
It simply does not work this way, and can not work as stated with the existing APIs.
I believe this is due to a fundamental difference in how various encoders/codecs work, and not fully understanding the subtleties.   Some encoders work by always storing the type information with data so that deserialization is possible from the data alone with no external context or special per-instance configuration.    Other encoders (such as Jackson) prioritize round trip to user controllable serialization formats.   The concept being that the codec can be configured such that the output (JSON in this case) is produced in a form that either pre-exists (must be matched exactly) or 'user friendly' (intentionally undefined -- but again, assumed to exist and must be matched).
To achieve this, the default mode, and in fact almost all features of Jackson are designed to NOT embed type into the JSON data itself, but rather to make use of the serialization or deserialization dynamic context (and other out-of-band configurations) to be able to parse 'arbitrary' JSON 'as is' into the expect type.   This is fundamentally different approach and goals then codecs which produce 'black box' output and instead focus on minimal or no out-of-band context needed to round trip Java objects.
The side effect is that in order for Jackson to function properly (i.e. in the use cases it was primarily designed for), type information MUST be provided out-of-band in order to deserialize.
The exception, which is not commonly used, and which recently has been shown to be a severe security problem,  is the special mode of embedding type information with each JSON element.
These modes are not interchangeable and have non-compatible requirements, and in general are 'all or nothing' -- you must code for one or the other mode explicitly and consistently.
The current Reddison APIs cannot handle either mode correctly in many cases.  For example, RBucket, without extreme hacks, cannot round trip most non-primitive objects -- even basic Java Bean or POJO.
This is NOT solvable by simply overriding the codec and tweaking it somehow -- because internally, in intentionally non-exposed implementation code, the necessary usage of the core Jackson APIs are incorrect and inconsistent in at least a few places.   The default Jackson codec uses a non-standard configuration of the ObjectMapper which is not able to round-trip consistently given the implementation of the deserializer (which in many places pasess a literal Object.class as the target type)
I have found that it is possible to hack this on a case by case basis by 'creative' (i.e. probably will break many other things) mangling of the codecs -- on a per use basis -- It still requires every API call to use a distinct codec taylored for the type,  and I have no idea what will happen with nested RObjects -- or other internal code -- What this demonstrates, in my opinion, is that the API itself needs to be enhanced slightly to accommodate the behaviour of codecs which need out of bound type information,
not by making this a burden on the users -- particularly since this relies on understanding many details of the private implementation of Reddison to work -- with no assurance that ones understanding is fully correct or that the undocumented internal implementation will not change.
This essentially makes Reddison unsuitable for large classes of its intended use cases.  That is sad, as I was looking forward to making use of the AWESOME features it does provide -- to be blocked by a few fundamental API flaws.
Recomendation:
Solving this problem is not complex, but it cannot be done cleanly by end users, it needs to be implemented internally and exposed through the APIs.
There are various other products which do this exact thing that can be used as a model for what is needed.
Essentially -- every API which deserializes a user defined object MUST have some mechanism of providing the target class explicitly.   For example:  Jackson itself has a Class argument for every method which does deserialization.   This is not just an 'advanced' feature, its necessary for all but a few special cases.
cache2k https://cache2k.org/  solves the problem by making use of a rarely known, subtle feature of Java which provides a way to prevent type erasure of generics.   E.g.
  Cache<String,String> cache = new Cache2kBuilder<String, String>() {}
This subtle 'trick' of creating a anonymous type explicitly (note the "{}" at the end -- critical), allows the cache to access the generic parameter types and use them later as needed for deserialization.
Using the first suggestion, I believe all it would take to make RBucket.get() to work properly is to add the class argument to get  (   RBucket.get( Type.class ) )
Yes this is ugly and yes this breaks some of the simplicity of implementing interfaces that do not have this signature -- but at least it works reliably and doesn't require the end user to have deep domain knowledge of both the codecs and reddison to handle basic simple java classes as the framework is designed to do.   This would have to be done likewise to all the methods which deserialize top level objects.   Once the top level class is properly deserialized, all its internal references will work fine (if you allow Jackson to do it for you)  -- Jackson will use the declared type of the parents reference variable to deserialize the children.   All the subtleties of how to get this to work are then defered to jackson and whatever customization one needs to do via annotations and configurations are all Jackson problems.
That way you could truly state that "If a class round trips through Jackson using the provided ObjectMapper then Reddison will work correctly for that class without changes or custom configurations needed"
I do not know the extent this affects internal code -- this is why I am not attempting to implement this as a PR -- I suspect that I would need to fully understand all the intricacies of Reddison to make sure I got all the cases correct -- I believe the author is the best source of this domain knowledge.
Note that these suggested approaches are NOT simply having the user customizing the codec -- the codec is not the right place to store per-type per-use information, that needs to come from the callsite directly, either in the creation of the R or in its methods -- One could still store the necessary class information in the codec as is done in the JsonJacksonMapCodec -- but 'hide' this from the user -- that is an implementation choice.
For JVM Languages which handle generics 'better' , such as Kotlin, one could write a wrapper around Reddison to provide the simpilier APIs and manage the explicit class arguments in the wrapper.
This is what the Jackson Kotlin module does
https://github.com/FasterXML/jackson-module-kotlin
Most of this module is a single source file that provides 'extension functions' which convert the generic template types into explicit class types to pass to Jackson.
E.g.
inline fun <reified T> ObjectMapper.readValue(src: File): T = readValue(src, jacksonTypeRef<T>())
This extension function exposes the intended user experience of
    val myclass = mapper.readValue<MyClass>(File("file.json"))
and under-the-hood converts this to roughly
   mapper.readValue( File , MyClass.class )
This is not possible in pure Java, which is why cache2k uses its subtle 'trick' during the cache creation to extract the generics class types.
Note: that the problem exists in Reddison even without user supplied classes begin generic/templates --
this is an issue with even basic POJO type classes with no inheritance, generics, or other 'advanced' features -- Reddison simply does not supply the API's necessary to deserialize these properly using codecs like Jackson which require out of band (or context provided) type information.
You just cant get there from here ...
Note: I was very surprised that such a simple use case failed -- until I checked the test cases in the examples and all of them which would have triggered these problems were using basic types (String, Int, Long) not custom classes.  Those types do not need any out of band type information so they do work just fine.   Similarly, the Reference class handles this problem in its own way (and may in fact be unnecessary if the above suggestions are implemented -- not sure)
Steps to reproduce or test case
Redis version
Redisson version
Redisson configuration

FSTCodec is used by default now. It solves such problem.

The largest mistake made by the writers of Redisson is try to implement java.util Map, List, ... interfaces rather than supply their own. There is plenty of methods that are supposed to work in a certain way according to java.lang such as the compute method on ConcurrentHashMap but is not properly implemented in Redisson.
Also, the iterators of a Map, for instance map.getEntries() simply uses the value and makes no round trip to redis server. This is in contract with how map.get(key) works, meaning the iterator is not concurrently safe, and should not be used but rather you should iterate keys and get the value yourself.
There are tons of similar usecases, as well as poor documentation overall on how things are supposed to work and what you can expect.
Main issue is trying to match the java api for all kinds of collections and objects rather than provide 100% bullet proof and limited ones themselves.
Also, having to name everything is not particularly great. If you have a map that contains keys to other Rmap or Rlist, you need to provide the key to the new values as well. What happens when you remove the key from first map? No body knows, but likely the value will continue to linger and pollute the redis database having no reference at all to it. So you need to do those things as well.
I would not recommend this library to anyone, or Redis for that matter. Serialization defaults to using Serializable required jboss. If you use Jackson, it can not handle cyclic references, while google GSON can. About to write a codec but that interface is massive. Not a fun job. Also everything is interface based and this makes little sense and makes everything harder to navigate.  No body will ever write another implementation of this API as the problem is the API itself.

Thank you for your feedback!

map.getEntries() simply uses the value and makes no round trip to redis server

I'm afraid you're mistaken, it uses Redis HSCAN command and not any local state of RMap object.

how things are supposed to work and what you can expect.

All Redisson objects excluding Local Cached Map are stateless and store state in Redis.

What happens when you remove the key from first map? No body knows, but likely the value will continue to linger and pollute the redis database having no reference at all to it. So you need to do those things as well.

If you remove key mapped to value as reference to other RMap or RList object then you need to delete these objects separtely, since cascade deleting is not supported in this case.

java.io.NotSerializableException:
java.io.NotSerializableException: org.redisson.RedissonMap
org.redisson.RedissonAtomicLong
wether with Jackson, Gson or Jboss.
I can't include these things in a wrapper class or instance? Jeezus. The limitations are tremendous. at least I would have figured I could serialize these things. Is it because of the ReddisonClient?
How are we supposed to create logic containing various things? Only singular values? I am lost as to how one should use this library.
I figured you could use an RList with an RMap for values, I don't see why I can't use them in a class. Am I required to setup a bunch of serialization logic for this to work?

RMap, RAtomicLong are wrappers around structures already stored in Redis.  You can't serialize them since you don't access to RedissonClient.

I can think of ways of circumventing that limitations. Several ways. ThreadLocal being one way. Another is through getters and setters and forcing the extending of RMap to get the client. But threadlocal is the way.
In fact, I decided a few hours (shortly after this message) ago to move on to Jedis and already implemented a serializer for a map and others, and what not that can do exactly this, and without having to have an instance of a client.
I figured this kind of things would be a given in such a giant library. Like declaring instance variables. I think you started with the wrong notion of trying to match the java util package and it backfired down the line. You also focus on given all kinds of options, such as the ability to serialize in 100 ways, rather than just rely on ONE option that sets the standard for how things should work even if that would have meant writing your own serializer. The serializer could have for instance provided the redisclient or through threadlocal.
Anyhow, I don't see how I can use this if my understanding of it are correct. I have not tried liveobjects, which is also a way, but I don't see why you would need the annotation of @REntity or @RClient on your own RMap, RList and what not. When the serializer encounters a RMap, it should replace it with a reference and when it reads it from redis it should fetch it from redis and create a copy of the previous RMap. I think you do that with live object surely, but why not living up your own collections? Would involve a rewrite? Too much work.
I am sure I am complaining too much, but I've spent a week on this and just keep hitting wall after wall, limitation after limitation.
Jedis gives me the oppurtonity to be in full control and your Redisson library while i think you are a genius indeed grew the scope of the project way too much. You should have focused on prodiving a copy of hibernate for redis. Meaning everything is managed and not neccessarily be limited of the various serializers that "automatically" will do stuff for you, while at the same time limiting you of that you can do, including initiating the classes and settting their property.
I am sure I am wrong but I am pretty sure I give up now. And that at the very end of getting the queue working, and I could just be able to get around the limitation by moving things around, but I don't see why I should continue build on this. For what I am trying to do it has already grown way too complex and without the feel of control of what is happening.

I can think of ways of circumventing that limitation. Several ways. ThreadLocal being one way. Another is through getters and setters and forcing the extending of RMap to get the client. But threadlocal is the way.

Sorry, which ThreadLocal object do you mean?

I have not tried liveobjects, which is also a way, but I don't see why you would need the annotation of @rentity or @RClient on your own RMap, RList and what not.

I'm afraid you missed LiveObject conception - it allows to bind entity objects to Redis hash and store field value as hash entry. If you look at Usage section you will find that @REntity annotation applied to POJO entity and not to RMap, RList object.

You should have focused on prodiving a copy of hibernate for redis

I think you missed the idea behind the project. I tried to provide convinent API for Java Developer to Redis. More other I implemented other useful Redis based objects which not avaialble in Redis itself.
Here is the feature comparision vs Jedis lib https://redisson.org/feature-comparison-redisson-vs-jedis.html
With Jedis you'll need to re-invent features offered by Redisson, maintain connection pool, care of thread safety and so on...

I am saying you could have used the concept of live object yourself on your own RMap, RList to avoid that annoying limitation. And you own implementation you would no have need to annotate them at all.

I just wanted to echo what @momomo and @DALDEI have already said in this thread. @mrniko you're definitely a brilliant engineer creating a library of this complexity, but there are also some gaps that need to be addressed. For my project, we're using RLiveObjectService to handle a wide variety of different Java-based entities. I think the general expectation of a service like this that as a developer I can throw in any Java POJO and "it just works™". Unfortunately, in reality there are a lot of "gotchas":

The default codec FST doesn't handle any type of changes to POJOs. So if you're changing your data model, but your Redis database still has data in the previous format, expect to see exceptions on read. Moreover, FST doesn't seem to support Java 17 at the moment, blocking our Java 17 upgrade.
Codecs that rely on Jackson can handle model changes, but the default config provided by Redisson only works for POJOs that use basic Java primitives. If you want to use java.time types like LocalDateTime or ZonedDateTime (which IMO is a very common use-case) you'll quickly end up rolling your own JsonJacksonCodec and you carefully have to setup its internals in order to correctly roundtrip data. Adding JavaTimeModule is not enough, for instance ZonedDateTime will deserialize but without the original timezone. Ouch. Moreover, using a type like this on the "root" object (the one annotated with REntity) will lead to a ClassCastException. This is the fundamental design flaw @DALDEI talks about, and makes sense considering how Redisson uses the ObjectMapper in this case: https://github.com/redisson/redisson/blob/master/redisson/src/main/java/org/redisson/codec/JsonJacksonCodec.java#L99 telling Jackson to deserialize to Object without providing any other details, meaning there's no way for Jackson to know it's dealing with a ZonedDateTime.
Behavioral differences between the root-entity and any nested classes within that root-entity. The "live" behavior only applies to the getters and setters on the root-entity, but doesn't propagate down to nested classes. Unless, it just happens to be a List or Map that automagically gets turned into something else by Redisson, which comes with its own rules (e.g. properly handling cascading deletes when the root-entity gets deleted).

I think, rather than trying to support every (de)serialization mechanism out there, I would focus on providing one (or a few) that are preconfigured to work well with Redisson and support a wider set of uses cases:

Support for Java primitives, and commonly used types from the JDK, such as java.time. With support I mean developers should be able to use these types out-of-the-box and without any issues.
Support for non-breaking changes to POJOS (e.g. additive changes).
Adequate tradeoff between storage-footprint and performance. I know I can wrap a JsonJacksonCodec in a SnappyV2Codec, but why not provide one by default? Or make it a configuration setting to use compression and optionally specify a compression method.

Again, Redisson is a great library and you've done excellent work in creating this. I hope these suggestions help you to improve your project.

@MartinDevillers
Thank you for your feedback.

FST doesn't seem to support Java 17 at the moment, blocking our Java 17 upgrade.

FstCodec is deprecated since it's too buggy. Default codec has been changed to MarshallingCodec since 3.13.0 version. Also you can try Kryo5Codec. Anyway you can always implement own codec.

Moreover, using a type like this on the "root" object (the one annotated with REntity) will lead to a ClassCastException.

The "live" behavior only applies to the getters and setters on the root-entity, but doesn't propagate down to nested classes.

Can you create an issues for these tasks?

I know I can wrap a JsonJacksonCodec in a SnappyV2Codec, but why not provide one by default?

Since for some data can't be compressed then it cause only excessive CPU consumption. It depends on use case and only developer can decide if compression is really necessary. JsonJacksonCodec is not the only codec. There are 9 codecs which developer should have ability to choose from.

@nachobot
MarshallingCodec was deprecated due to this issue https://github.com/redisson/redisson/issues/4634. Kryo5Codec has been a default codec since 3.19.0 version.

 I realized that within 1 day of using it. I ended up writing my own library. 100X more powerful. Will see if I come to publishing it soon.


Apply labels related to the test plan items.

This is to apply missing labels pointing to test case from the test plan.

/cc @nikita133


tf2ss() returns incorrect result

I checked the code. It seems that the problem relates to the function td04ad in slycot but I am not familiar with the language.

Why do you think the result is incorrect? Both state space systems are equivalent, just a different representation.

Why do you think the result is incorrect? Both state space systems are equivalent, just a different representation.

Is there any

If you don't have Slycot, it would be a call to scipy.signal.tf2ss()
https://github.com/python-control/python-control/blob/6b24bb42a8da5b15cb79699e74fb33b30ba7935a/control/statesp.py#L984
With Slycot it is a call to TD04AD()
https://github.com/python-control/python-control/blob/6b24bb42a8da5b15cb79699e74fb33b30ba7935a/control/statesp.py#L959-L960
Which is documented in the source:
https://github.com/python-control/Slycot/blob/c8d3a2143c6e0091db4801f2ee68f0d256537f2f/slycot/src/TD04AD.f#L175-L219

If you don't have Slycot, it would be a call to scipy.signal.tf2ss()
https://github.com/python-control/python-control/blob/6b24bb42a8da5b15cb79699e74fb33b30ba7935a/control/statesp.py#L984
With Slycot it is a call to TD04AD()
https://github.com/python-control/python-control/blob/6b24bb42a8da5b15cb79699e74fb33b30ba7935a/control/statesp.py#L959-L960
Which is documented in the source:
https://github.com/python-control/Slycot/blob/c8d3a2143c6e0091db4801f2ee68f0d256537f2f/slycot/src/TD04AD.f#L175-L219

Thank you for your generous help! I will close the issue now.


65 auth implementation

What changed

Auth layer
Quota usage tracking

Why do we need it

Access to the core logic using Rest API.
for v0.0.7, auth of webclient's users

Codecov Report

Merging #68 (109c07b) into master (a46a504) will increase coverage by 2.41%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master      #68      +/-   ##
==========================================
+ Coverage   90.69%   93.11%   +2.41%
==========================================
  Files          15        2      -13
  Lines        1204      276     -928
==========================================
- Hits         1092      257     -835
+ Misses         90       13      -77
+ Partials       22        6      -16

Flag
Coverage Δ

cmd-httpserver
?

pkg-gcpsecretsmanager
?

pkg-httpclient
100.00% <ø> (ø)

pkg-openai
90.95% <ø> (ø)

pkg-postgres
?

root
?

Flags with carried forward coverage won't be shown. Click here to find out more.
see 13 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


delete one row

Can I use this to delete just one specific row from my spreadsheet?

I'd suggest gspread (https://github.com/burnash/gspread) if you want to make detailed changes like that. This tool is aimed at syncing a table of data to a google worksheet.


eventio.simtel.simtelfile issues while loading a file and parsing

Hi,
I run some sim_telarray simulation, first with the sim_telarray options :
MIN_PHOTONS = 0
MIN_PHOTOELECTRONS = 0
then convert this simtel files into npz numpy format file via importing SimTelFile from eventio.simtel.simtelfile, as:
from eventio.simtel.simtelfile import SimTelFile
This works fine.
then I run another simulation changing the sim_telarray options :
MIN_PHOTONS = 0
MIN_PHOTOELECTRONS = 45
or also another with :
MIN_PHOTONS = 150
MIN_PHOTOELECTRONS = 45
when I tried to use the same method described above, I got the following error :
/work/se1/unige/software/anaconda3/envs/lst-dev-our_MC/lib/python3.7/site-packages/corsikaio/subblocks/dtypes.py:20: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  return np.dtype(dict(**dt))
Traceback (most recent call last):
  File "/work/se1/unige/software/prog/simtel_chain/validation/convert_simtel_to_npz/read_simtel_all.py", line 116, in <module>
    entry(files, output, max_event_id)
  File "/work/se1/unige/software/prog/simtel_chain/validation/convert_simtel_to_npz/read_simtel_all.py", line 57, in entry
    with SimTelFile(file) as f:
  File "/work/se1/unige/software/anaconda3/envs/lst-dev-our_MC/lib/python3.7/site-packages/eventio/simtel/simtelfile.py", line 111, in __init__
    self.next_low_level()
  File "/work/se1/unige/software/anaconda3/envs/lst-dev-our_MC/lib/python3.7/site-packages/eventio/simtel/simtelfile.py", line 168, in next_low_level
    self.telescope_descriptions[o.telescope_id][key] = o.parse()
  File "/work/se1/unige/software/anaconda3/envs/lst-dev-our_MC/lib/python3.7/site-packages/eventio/simtel/objects.py", line 417, in parse
    HV_disabled = read_array(byte_stream, count=n_trig_disabled, dtype='i4')
  File "/work/se1/unige/software/anaconda3/envs/lst-dev-our_MC/lib/python3.7/site-packages/eventio/tools.py", line 41, in read_array
    return np.frombuffer(f.read(count * dt.itemsize), count=count, dtype=dt)
ValueError: buffer is smaller than requested size

it seems that the simtel files created under values different than 0 for MIN_PHOTOELECTRONS and MIN_PHOTONS can't be opened (or open correctly), in addition the parsing is also not working.
By looking into the documentation I saw I could also used EventIOFile to open the file via :
from eventio import EventIOFile
when using this method the simtel files open correctly.
I am using currently running on :
Scientific Linux release 6.10 (Carbon)
eventio 1.4.1
python  3.7.6
To create the simtel files I used :
corsika7.7_simtelarray_2020-06-29
sim_telarray built with prod5-lp-sc-baseline option (allowing up to 14000 pixels cameras)
Thank you

EventIO files is more low-level that does not try to parse any data until you ask it to do so.
Can you provide the files somehow? E.g. via dropbox/google drive / nextcloud?

Dear Maximilian,
You can find both files in the link below :
https://www.dropbox.com/sh/0ybe200330avkn1/AAAT-begxtENZgtudl2o6BCJa?dl=0
corsika_min_ph_pe_non_zero.simtel.gz -> this is the failing-to-load file
corsika_min_ph_pe_zero.simtel.gz -> this is successful-to-load file
The both differ in the amount of events, the non_zero has more events than the zero one, therefore that difference in size

Just to add that :
MCEvent and MCShower instances can be parsed successfully
eventio_print_object_information /work/se3/davidsudm/lst_sipm_task/simtel/lst-sipm-pmma/output/min_photon_proton_nsbx1/corsika_run102.simtel.gz
 Type | Version | Level | #Objects | eventio-class
------------------------------------------------------------
   70 |       1 |     0 |        1 | simtel.objects.History
   71 |       1 |     1 |        1 | simtel.objects.HistoryCommandLine
   72 |       1 |     1 |      147 | simtel.objects.HistoryConfig
  100 |       2 |     0 |        1 | histograms.Histograms
 1204 |       0 |     0 |     1396 | iact.objects.TelescopeData
 1208 |       2 |     1 |     1396 | iact.objects.PhotoElectrons
 1212 |       0 |     0 |        1 | iact.objects.InputCard
 1216 |       1 |     0 |        1 | iact.objects.AtmosphericProfile
 2000 |       2 |     0 |        1 | simtel.objects.RunHeader
 2001 |       4 |     0 |        1 | simtel.objects.MCRunHeader
 2002 |       5 |     0 |        1 | simtel.objects.CameraSettings
 2003 |       1 |     0 |        1 | simtel.objects.CameraOrganization
 2004 |       3 |     0 |        1 | simtel.objects.PixelSettings
 2005 |       0 |     0 |        1 | simtel.objects.DisabledPixels
 2006 |       0 |     0 |        1 | simtel.objects.CameraSoftwareSettings
 2007 |       0 |     0 |        1 | simtel.objects.PointingCorrection
 2008 |       0 |     0 |        1 | simtel.objects.DriveSettings
 2009 |       3 |     1 |      224 | simtel.objects.TriggerInformation
 2010 |       0 |     0 |      224 | simtel.objects.ArrayEvent
 2011 |       3 |     2 |      224 | simtel.objects.TelescopeEventHeader
 2013 |       3 |     2 |      224 | simtel.objects.ADCSamples
 2014 |       5 |     2 |      195 | simtel.objects.ImageParameters
 2016 |       1 |     2 |      224 | simtel.objects.PixelTiming
 2020 |       1 |     0 |   100000 | simtel.objects.MCShower
 2021 |       1 |     0 |  1000000 | simtel.objects.MCEvent
 2022 |       0 |     0 |        1 | simtel.objects.CameraMonitoring
 2023 |       3 |     0 |        1 | simtel.objects.LaserCalibration
 2026 |       2 |     0 |      224 | simtel.objects.MCPhotoelectronSum
 2027 |       0 |     2 |      195 | simtel.objects.PixelList
 2100 |       0 |     1 |      224 | simtel.objects.TrackingPosition
 2200 |       1 |     1 |      224 | simtel.objects.TelescopeEvent
------------------------------------------------------------

I identified the problem, a wrong variable is used for the number of pixels at that point.
Can you produce a similar file with less events? So that the size is maybe < 5 MB?

    416         n_HV_disabled = read_int(byte_stream)
--> 417         HV_disabled = read_array(byte_stream, count=n_trig_disabled, dtype='i4')

This should be n_HV_disabled instead of n_trig_disabled in line 417. Probably a copy/paste error.

@davidsudm never mind the new file, I was able to cut the file you send appropriately to include it in the test resources.

Ah, and just as a side note, I think this has nothing to do with the min photons options.
It failed because the number of disabled trigger pixels was not the same as the number of pixels with disabled HV.

Fixed in #219, will release 1.4.2 as soon as it is merged

great, thanks! We fixed in in our private installation meanwhile and will update to 1.4.2. as soon as it will be available on PyPi

Thank you, @maxnoe , it works!

release 1.4.2 is made and should be on pypi in a couple of minutes.

Ah, and just as a side note, I think this has nothing to do with the min photons options.
It failed because the number of disabled trigger pixels was not the same as the number of pixels with disabled HV.

Something it is not clear yet for me @mahnen. How does it explain that it worked for the min_pe and min_photons equals to 0 but failed for non 0 values?
does the min_pe option actually can change the number of pixels by enabling or disabling them?

I don't know. Maybe it was just chance? How do you simulate the disabled pixels / non disabled pixels?
I am not really an expert on the inner workings of simtel, only the files that come out.

Between the 2 files I sent, the only main difference is the value of min_photoelectrons and min_photons, beside the assumed threshold that might differ. However, the rest of the configuration of the telescope is the same.
We select our trigger conditon via the analogsum parameter for trigger plus a camera_config file that has the required geometry of our telescope and the pixels it needs to sum (sectors).
We will ask to Konrad about it since he must know if the min_pe option actually disables pixels!
Thank you

Looking at set_disabled_pixels in reconstruct.c, there is an option to have random pixels broken.
Do you have broken_pixels_fraction > 0?

Fro what I see in sim_telarray documentationm, this parameter is passed via :
• DEAD_PIXELS
(type: Int, items: MAX_DEAD_PIXELS, default: "all: -1") List of broken pixels. Value of ’-1’ terminates the list.
in my telescope simtel config file as well as in the log outputs I got :
    DEAD_PIXELS       Int      20                          all: -1
                      Current:   all: -1
    DEAD_PIXEL_PROBability
                      Double   1        0        1         0        0
    DEAD_BOARD_PROBability
                      Double   1        0        1         0        0
    DEAD_MODULE_PROBability
                      Double   1        0        1         0        0

So they are all set to 0 and all pixels are unbroken


Google login broken

Logging into the opendc.org website using Google no longer works.
I believe I submitted a ticket about this - which was resolved - but the ticket is now also gone.
I have tried both Chrome and Firefox, have cleared cookies and tried again, etc. No luck.
Be great to get this resolved. :-)

Hi @lukemunn,
There was some issue with the web server instance running opendc.org. I have tried to resolve the issue.
Could you verify whether it works for you again?

Yes logging in now works. Thanks for resolving.


Fixes #472. Avoid segfault when wasm invokes httpCall to cluster whos…

…e DNS name cannot be resolved.
Signed-off-by: Robert Panzer<EMAIL_ADDRESS>For an explanation of how to fill out the fields, please see the relevant section
in PULL_REQUESTS.md
Description: Bugfix for #472. Change order of initializing properties of AsyncClientHandler so that these are initialized in case the request is answered synchronously (by Envoy)
Risk Level: low
Testing: Currently only manual for the specific bug, the code base itself is tested. Unfortunately I am not capable right now to setup such a test. (I would happily learn to do that though if somebody tells me how. Looking at wasm_filter_test.cc I couldn't figure out how to mock the cluster manager to generate the required config
Docs Changes: None
Release Notes:
Fixes #472. Avoid segfault when invoking httpCall for a cluster whose DNS name cannot be resolved.

@jplevyak could you take a look?

Looks like another solution is in progress


SQL errors silently fail

Hi there,
I'm using the ExecuteStoredProcedure extension method to run a stored proc.  It's all working fine, but when the underlying stored procedure failed due to a key constraint violation, no exception was thrown, so it failed silently.
Is there anyway to ensure that any exceptions that occur at database level are bubbled and thrown?
Thanks

Hello @daveb84 ,
Do you think you could provide a project sample with this issue?
It will help my developer investigate the issue more efficiently.
From what I know, we don't make any silence try/catch.
We now always ask for a project sample even if the issue is easy to reproduce. As a free product, we must find some way to save time to offer an overall better experience for everyone and release fixes faster.
You can send it to<EMAIL_ADDRESS>if you need to keep the source private
Best Regards,
Jonathan

Performance Libraries
context.BulkInsert(list, options => options.BatchSize = 1000);
Entity Framework Extensions • Entity Framework Classic • Bulk Operations •  Dapper Plus
Runtime Evaluation
Eval.Execute("x + y", new {x = 1, y = 2}); // return 3
C# Eval Function • SQL Eval Function


rkt: inherit environment

All of the applications should inherit the environment by default. This is because we should assume that the user is running rkt under a tool like an init system where the environment is significant. To turn off environment inheritance we should have something like:
rkt run --inherit-environment=false example.com/foo:v2.0.4
/cc @robszumski

What takes precedence when the app has its own explicit environment settings?
i.e. if the app's image.json sets PATH=/foo/bar, do we really want to inherit the user's PATH all the way down to the app's environment and override /foo/bar with "/usr/local/bin:/usr/bin..." as the default behavior?
This doesn't strike me as particularly well thought out.  When implementing rkt enter I had originally wanted to support a UX like FOO=bar rkt enter UUID /usr/bin/foobar but there's no distinguishing the variables set presently via the bash commandline and the rest of the user's environment.  I would have liked to set FOO= in the entered context since it's clearly explicit, and leave behind the rest in favor of the app's environment.  It looked like the best way to approach the problem was to add a commandline option for rkt to specify environment variables explicitly.

@vcaputo raises a good point, this was something I stumbled upon while working on this. Unfortunately there is no way to distinguish between intentionally and unintentionally set env vars.

It also doesn't seem like a very wise default choice to automatically inherit all environment variables down to a container's execution environment, there's potentially significant information leaking into something the user has opted to execute as a container, that generally implies a desire for isolation.

Here is how I think of the inheritance:

rkt execution environment
image environment
pod manifest environment

I am fine with having --inherit-environment be false by default but I can see it as being a very nice feature that I would use.
We could also add a --environment flag that would essentially belong to the "pod manifest" and explicitly set environment variables that would clobber anything else. But, that is a different feature.

Anyone have thoughts on this order of operations? Seems straightforward and easy to understand. The primary use case will be things like REDIS_HOST=foobar MAILSERVER=baz types of stuff so I think it is OK in most cases.
cc @vcaputo

@philips the inheritance makes sense. I'll prefer to leave --inherit-environment disabled by default as it can expose sensitive data but also change the app behavior in unexpected ways.
I also think that a per app --environment flag that will add env vars to the pod manifest (permitting to add new vars and override the app's and the inherited ones) will be very useful.

Fine with me provided wholesale inheritance is disabled by default.
Just to confirm my understanding, when --inherit-environment is specified everything in the user's environment will make it into the environment of all the apps.  Even environment variables explicitly set by the app's image will be overridden by the inherited values of same name.

@vcaputo Yes, it will be in all apps. The environment of the app takes precedence. In order of evaluation I see:

rkt execution environment with inherit-environment
image environment
pod manifest environment

If FOO=bar in step 1 and FOO=baz in step 3 then FOO=baz will be the final value.

Ok, the utility seems fairly limited when it doesn't influence variables defaulted by the app, as those variables are likely impactful ones you'd like to alter via FOO_THING=bar rkt run foo.aci.  But if we cover that with --environment then I guess it's fine.
In an rkt prepare and rkt run-prepared world, where is --inherit-environment honored?  From the implementation's perspective, it's a natural fit for rkt prepare as that's the phase which currently produces the environment files for stage[12].  Should rkt run-prepared support it additionally and amend the environment state prior to running?

That does seem kind of limited. Wouldn't you want it:

image environment
pod manifest environment
rkt execution environment with inherit-environment

Another semi-related question: is there a way in a systemd unit to have the namespace/whatever inherit only env vars specified in the unit but not from anywhere else?

@robszumski The problem is that there are important environment variables already in the environment like PATH, TERM, HOME, etc. Making some sort of blacklist gets really tricky pretty fast.

cc @jedsmith


add navigator clipboard, ClipboardItem

This implements navigator.clipboard.write and navigator.clipboard.read, which both work with ClipboardItems. It also adds a re-implementation of ClipboardItem which supports Promises, adhering to the spec.
write, per the spec takes an array of ClipboardItems and writes to the clipboard. We're limited in what we can do here, so we extract the text/plain content of the first one and call writeText. This is a pretty reasonable default behaviour.
read, per the spec should return an array of ClipboardItems representing the user's current clipboard. We're limited what we can do here, so we call readText and convert it into a ClipboardItem with just text/plain.
ClipboardItem has a presentationStyle attribute (defaulting to 'unspecified' unless it's passed as an option), and getType() always returns Blobs. In Chrome's buggy implementation, getType('text/plain') will return a String. We've accounted for both possibilities in navigator.clipboard.write just in case the polyfill isn't applied (e.g. Chrome adds promises but keeps their buggy getType() behaviour).
This means await navigator.clipboard.write(await navigator.clipboard.read()) should be a no-op (as long as the user only has plain-text data in their clipboard) which feels like a sensible thing to do.
Customer Support Data
Here are some tables representing the % of browsers we see that support (or don't support) these APIs.
Clipboard Write

Browser
Supported Since
Latest Version
% Supported
% Unsupported

chrome
66
101
73.565
0.007

edge
79
100
6.691
0.001

firefox
Not Supported
100
0.000
12.669

firefox_android
Not Supported
100
0.000
0.128

ie
Not Supported
11
0.000
0.078

opera
63
86
1.267
0.000

safari
13.1
15.4
4.630
0.013

safari_ios
13.4
15.4
0.505
0.006

samsunginternet_android
12.0
16.0
0.020
0.000

webview_android
66
101
0.002
0.008

Total:

86.68
12.91

(Note: "Other" browsers make up 0.408%)

This is great, thank you @keithamus for putting this together 🎉
I've read through it and I have no concerns, just a couple of questions:

Is this going to provide Firefox support too?
Will this be added to index.ts, so the apply method brings it to dotcom? I'm just trying to clarify the road ahead: once we merge this and upgrade this lib in dotcom, we'd got https://github.com/github/repos/issues/1073 fixed 🙏

Tested today on Firefox and confirmed that this fixes the Copy Remote functionality in the blob page for Firefox, see details on https://github.com/github/repos/issues/1073#issuecomment-1226971804.


appcompat-v7:23.+ 和butterknife:8.8.1 错误

appcompat-v7:23.+ 和butterknife:8.8.1这什么错误 , 该如何解决 (appcompat-v7:26.+ 则没有报错)
    compile 'com.android.support:appcompat-v7:23.+'
    compile 'com.jakewharton:butterknife:8.8.1'
    annotationProcessor 'com.jakewharton:butterknife-compiler:8.8.1'

Warning:Exception while processing task java.io.IOException: Can't write [E:\Application\MyApplication\app\build\intermediates\transforms\proguard\debug\jars\3\1f\main.jar] (Can't read [C:\Users\Admin.android\build-cache\afefd4741cd56a370b6dd313befcc4640dcfbd5a\output\jars\libs\internal_impl-23.4.0.jar(;;;;;;**.class)] (Duplicate zip entry [android/support/v4/h/m.class == internal_impl-23.4.0.jar:android/support/v4/view/LayoutInflaterFactory.class]))

This is not a problem with the library. You are including two copies of a class in your build system. You can use the dependencies task to try and narrow where the duplicates are coming from.


Function Call with arguments

Hello, I need to use portal_allow function.
Portal_allow needs arguments, can you explain how I can pass it to the function using $data?
Thank you

I'm not sure how I missed this issue :(
Looking at the code I see no such method..can you point me to what you are referring to?


Azure Cli Command Problem

The bug is that when you try some commands on the cli it does not respond

There is not enough information in this report to do anything about. Please provide the required minimum information when posting an issue.


Use cobra to parse CLI

Use Cobra to parse CLI.
This fixes issue #226.
Use cobra to parse CLI

Thanks for taking care of this.
This PR closes #226

That's OK. I have fixed these nit issues in this PR. They are easy to fix :)

/lgtm
/approve


 You can speed run this game . This game increases your reflexes and thinking capacity.
How to play your game?

Press WASD to move,
Go over an enemy to kill it

Code
Check off the items that are true.

[x] The game was made using the Sprig editor.
[x] The game is placed in the in the /games directory.
[x] The code is significantly different from all other games in the Sprig gallery (except for games labeled "demo").
[x] The game runs without errors.
[x] The name of the file/game contains only alphanumeric characters, -s, or _s.
[x] The game name is not the same as the others from gallery

Image (If an image is used)

[ ] The image is in the /games/img directory.
[ ] The name of the image matches the name of your file. Example: sokoban_plus.js -> sokoban_plus.png.

Great game! could you add more levels?

Great sprites in this game!
It seems you need to make a few more changes before merging it in. Because it's been a little while I'm going to close the PR for now but feel free to reopen at a later point if you want to continue working.


Master -> Live

Docs Build status updates of commit f80fb89:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

mixed-reality-docs/mr-dev-days.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Some Microsoft Office apps do not report complete version numbers

Currently, some Office apps (I have tested microsoftexcel, microsoftoutlook, microsoftpowerpoint, microsoftword) only report a truncated version number for appNewVersion. This started causing a problem recently because we use IGNORE_APP_STORE_APPS=yes on our standard update policies in Jamf (we have currently have a mix of installs but have decided to standardize on the non-App Store versions) which bypasses the updateTool...but this is only a portion of the problem. Here is what I am seeing:
For example, as of today (14 Feb 2023) using:
downloadURL="https://go.microsoft.com/fwlink/?linkid=525134"
appNewVersion=$(curl -fsIL "$downloadURL" | grep -i location: | grep -o "/Microsoft_.*pkg" | cut -d "_" -f 3 | cut -d "." -f 1-2)

$appNewVersion returns 16.69
appversion=$(defaults read $installedAppPath/Contents/Info.plist $versionKey)
##defaults read /Applications/Microsoft\ Word.app/Contents/Info.plist CFBundleShortVersionString

$appversion returns 16.69.1 which is the currently the most recent version available
When Installomator runs, it finds that $appNewVersion and $appversion do not match, so it attempts to install the most recent version it knows about (16.69). The installer downloads but silently fails to install because a newer version (16.69.1) is already installed. An example log of what that looks like:
2023-02-06 10:30:09 : INFO  : microsoftword : App(s) found: /Applications/Microsoft Word.app
2023-02-06 10:30:09 : INFO  : microsoftword : found app at /Applications/Microsoft Word.app, version 16.69.1, on versionKey CFBundleShortVersionString
2023-02-06 10:30:09 : INFO  : microsoftword : appversion: 16.69.1
2023-02-06 10:30:09 : INFO  : microsoftword : Label is not of type “updateronly”, and it’s set to use force to install or ignoring app store apps, so not using updateTool.
2023-02-06 10:30:09 : INFO  : microsoftword : Latest version of Microsoft Word is 16.69
2023-02-06 10:30:09 : REQ   : microsoftword : Downloading https://go.microsoft.com/fwlink/?linkid=525134 to Microsoft Word.pkg
2023-02-06 10:31:03 : REQ   : microsoftword : no more blocking processes, continue with update
2023-02-06 10:31:03 : REQ   : microsoftword : Installing Microsoft Word
2023-02-06 10:31:03 : INFO  : microsoftword : Verifying: Microsoft Word.pkg
2023-02-06 10:31:03 : INFO  : microsoftword : Team ID: UBF8T346G9 (expected: UBF8T346G9 )
2023-02-06 10:31:03 : INFO  : microsoftword : Installing Microsoft Word.pkg to /
2023-02-06 10:31:31 : INFO  : microsoftword : Finishing...
2023-02-06 10:31:34 : INFO  : microsoftword : App(s) found: /Applications/Microsoft Word.app
2023-02-06 10:31:34 : INFO  : microsoftword : found app at /Applications/Microsoft Word.app, version 16.69.1, on versionKey CFBundleShortVersionString
2023-02-06 10:31:35 : REQ   : microsoftword : Installed Microsoft Word, version 16.69.1
2023-02-06 10:31:35 : INFO  : microsoftword : notifying
2023-02-06 10:31:35.127 Management Action[18876:786775] NSNotificationCenter connection invalid
2023-02-06 10:31:40 : INFO  : microsoftword : App not closed, so no reopen.
2023-02-06 10:31:40 : REQ   : microsoftword : All done!

Initially, I thought getting the full version from $downloadURL and comparing it to CFBundleVersion might be a quick fix:
downloadURL="https://go.microsoft.com/fwlink/?linkid=525134"
appNewVersion=$(curl -fsIL "$downloadURL" | grep -i location: | grep -o "/Microsoft_.*pkg" | cut -d "_" -f 3 | cut -d "." -f 1-3)

$appNewVersion returns 16.69.23010700
versionKey="CFBundleVersion"
appversion=$(defaults read $installedAppPath/Contents/Info.plist $versionKey)
##defaults read /Applications/Microsoft\ Word.app/Contents/Info.plist CFBundleVersion

$appversion returns 16.69.23011600
It seems some updates are released as only update packages instead of full installers...and only full installers are available from the downloadURL. So much for a quick fix.
Not wanting to be beat by Microsoft, I kept digging. I found I could pull the most recent version number and updater URL using:
curl -s https://officecdn.microsoft.com/pr/C1297A47-86C4-4C1F-97FA-950631F94777/MacAutoupdate/0409MSWD2019.xml | xmllint --format --xpath "//*/array/dict[1]" - | awk -F'<key>FullUpdaterLocation</key>' '{ print $2 }' | awk -F'</string>' '{ print $1 }' | sed 's/<string>//'
This works and seems to be consistent across all the apps (by replacing MSWD2019 with the desired app code)...but, just to keep things complicated, these packages can only be used to update a currently installed version so they cannot directly replace the downloadURL.
I came up with a messy way to compare the updater version with the full version and some dodgy logic to decide which package to use but I was not happy with it. I was not able find a good way to do it without rewriting significant parts of Installomator outside of the microsoftword label...which I wanted to avoid. Ideally, I see the logic working like this (in pseudo code):
GET availableFullVersion, availableUpdaterVersion
IF isInstalled THEN GET installedVersion
IF ! isInstalled || (isInstalled && INSTALL=force) || (isInstalled && isAppStore && IGNORE_APP_STORE_APPS=yes) THEN install availableFullVersion and availableUpdaterVersion if required
IF isInstalled && ! isAppStore && installedVersion < availableUpdaterVersion THEN install availableUpdaterVersion

Maybe I am approaching this the wrong way or simply making a mountain out of a molehill here. It could be this is a fairly isolated circumstance...looking at https://macadmins.software there don't appear to have been many update-only situations in the past but only time will tell.
Outside of this specific issue, it still makes sense to use the most accurate version numbers available especially considering the version number/build number method Microsoft has employed for versioning its apps.

This is an issue still.
Looking at Microsoft Company portal and currently getting v5.2310.5 from the download URL in Installomator:
https://go.microsoft.com/fwlink/?linkid=869655
The file name format is:
CompanyPortal_5.2310.5-Installer.pkg
This URL gives you a full installer with v5.2404.0:
https://go.microsoft.com/fwlink/?linkid=853070
But the format of the file is:
CompanyPortal-Installer.pkg
So appNewVersion does not have a version number in the file name to fetch
Inspired by your post, this gets the new version:
curl -s https://officecdn.microsoft.com/pr/C1297A47-86C4-4C1F-97FA-950631F94777/MacAutoupdate/0409IMCP01.xml | xmllint --format --xpath "//*/array/dict[1]" - | awk -F'<key>Title</key>' '{ print $2 }' | awk -F'</string>' '{ print $1 }' | sed 's/<string>//' | rev | cut -d' ' -f1 | rev | tr -d "\n"

So conceivably, this would work:
microsoftcompanyportal)
    name="Company Portal"
    type="pkg"
    #downloadURL="https://go.microsoft.com/fwlink/?linkid=869655"
    downloadURL="https://go.microsoft.com/fwlink/?linkid=853070"
    #appNewVersion=$(curl -fs https://macadmins.software/latest.xml | xpath '//latest/package[id="com.microsoft.intunecompanyportal.standalone"]/cfbundleshortversionstring' 2>/dev/null | sed -E 's/<cfbundleshortversionstring>([0-9.]*)<.*/\1/')
    #appNewVersion=$(curl -fsIL "$downloadURL" | grep -i location: | grep -o "/CompanyPortal_.*pkg" | cut -d "_" -f 2 | cut -d "-" -f 1)
    appNewVersion=$(curl -s https://officecdn.microsoft.com/pr/C1297A47-86C4-4C1F-97FA-950631F94777/MacAutoupdate/0409IMCP01.xml | xmllint --format --xpath "//*/array/dict[1]" - | awk -F'<key>Title</key>' '{ print $2 }' | awk -F'</string>' '{ print $1 }' | sed 's/<string>//' | rev | cut -d' ' -f1 | rev | tr -d "\n")
    expectedTeamID="UBF8T346G9"
    if [[ -x "/Library/Application Support/Microsoft/MAU2.0/Microsoft AutoUpdate.app/Contents/MacOS/msupdate" && $INSTALL != "force" && $DEBUG -eq 0 ]]; then
        printlog "Running msupdate --list"
        "/Library/Application Support/Microsoft/MAU2.0/Microsoft AutoUpdate.app/Contents/MacOS/msupdate" --list
    fi
    updateTool="/Library/Application Support/Microsoft/MAU2.0/Microsoft AutoUpdate.app/Contents/MacOS/msupdate"
    updateToolArguments=( --install --apps IMCP01 )
    ;;

Local tests look good running assemble.sh to build a new version of the script after modifying microsoftcompanyportal.sh with this code

Nice. I've made some more headway with this issue and some other related problems. Have a look at my pull request #1490
Maybe it will give you some additional inspiration.


MenuEntrySwapper: Nex Portal Private instance

Is your feature request related to a problem? Please describe.
To avoid joining in the Nex portal accidently, we can add another option in the menu entry swapper plugin. This can be by waiting for people to join in the party, miss clicking the bank, or trying to go into the private instance.
Describe the solution you'd like
A clear and concise description of what you want to happen.
By having the menus swapped between private and normal entry, we can avoid since the private instances ask if you want to join an instance or start your on.
Note: A very similar feature is the Corrupted Gauntlet menu entry.
Describe alternatives you've considered
It could also be resolved by hiding the left click to enter. You are forced to right-click the portal for the entry options

This can now be done with the changes in 92aac53f4273e7944c3da767637e083847c03808


It's a valid syntax for less and shouldn't break anything. Pretty sure the LESS guidelines state the two forward slashes should be used over the 'multi line' comment for a single line.

@danieltj27 You're right, it's valid for less, I thought it was invalid for my postcss processor doesn't recognise it. Other comments in the file are written as multi line style, should we follow the style for compatibility?


Bug: git phlow issues --mine lists all issues in repo

On my version 3.1.0, running
git phlow issues --mine lists all issues, not just those assigned to me.

That feature got accidentally butchered during the remake of the command with support for Jira. I am not sure we will bring back --mine but instead replace it with a filter function so you have higher customizability of your issue search queries

Stale issue


Scan cursor mutex

We wanted to get some performance gains around scan and scroll, and needed to add a couple of mutexes to make ScanCursor.Next() be more thread safe.
The change doesn't impact performance when only one thread is calling Next(), but they helped get us a ~40% performance improvement by letting multiple threads call Next() at the same time.

Ah nuts. sorry, I think I have a bad merge here?


bbox intersection check

b.utils.are_bboxes_intersecting and b.utils.are_bboxes_intersecting_many added. These functions are jax jittable and implement the separating axis theorem algorithm. They will be useful to check for collisions or any general checking for intersections between bbox pairs and their respective poses.
check test/test_bbox_intersect.py on how to use it. b.utils.are_bboxes_intersecting_many simply vmaps over the second pose of an object.

can you test that these functions are jittable?

can you test that these functions are jittable?

@nishadgothoskar The test file uses a jitted version of it. Is that what you meant?


Fix call to geometric_verification in reconstruction.py

After the changes from 541e93ce19d76d68828207536d3cd166dfa6b22f, the previous function geometric_verification is now called estimation_and_geometric_verification.

Good catch, thanks! We definitely need a CI with unit tests here...


Support hashicorp vault aws auth

It would be convenient if kubecost supports vault aws auth machinery (https://www.vaultproject.io/docs/auth/aws). In that case not secrets (service-key.json) needed to assume IAM Role with permissions for access Cost Usage Report.

Hi @Valiev we can look into this. I'm a little unclear on what's required here-- do any of the auth solutions we support here: https://github.com/kubecost/docs/blob/master/aws-cloud-integrations.md look like they can be made to work with vault?

Hi @AjayTripathy! The main idea is the following:

you have k8s and vault integration, so containers could authorise in vault easily
also, there is vault and aws integration, so vault provide you access (allow to assume certain roles) to AWS resources

In that setup you no longer need to create AWS User and store it's key in secrets.

This issue is being closed because it may not be relevant to the OpenCost project and appears stale. If you feel this was closed in error, please open a new OpenCost issue with updated details or if it is still relevant for Kubecost please open an issue with Kubecost Support.


Add managed ds to Vaults

Description
This adds the ability to add manage datasources into a vault.
We have a single modal for all managed datasources as in the designs.
I have not implemented the "select all visible" button of the modal, since I think it's an improvement that can be tackled later.

Currently deploying to front-edge.
Then I need to update the patch endpoint to remove a datasource view if the content is empty, cause if you unselect everything in a Notion it will keep the Notion line and if we click it says "no content".
Risk
Internal only, can be rolled back.
Deploy Plan
Deploy front.

It's availble on front-edge


FR: use static instead of dynamic CALLs

https://github.com/contextfreecode/cobol-explore/blob/d24968c407c485c02f1ff5b7b131265b7ba27822/game-resource.cpy#L21
You very likely want to request every CALL 'literal' to be generated as static call - to do so just pass -static on the cobc command line (which then also drops the need for the "do_not_call_me" function) and drops a bit of the CALL overhead, too.
Otherwise explicit specify a so called CALL-CONVENTION, possibly the embedded one call static 'SDL_INIT using ...`.
BTW: I've enjoyed your video on this game a lot, thanks for sharing your experience!
... and just to note: the reason why you needed that function when using dynamic calls is that the linker removes the "unused" libraries for you (you likely see that with ldd game, too) so they just weren't there at the run time (you could "preload" them, but in your case you really want those in).
You can tell the linker to not to this (and tell cobc to pass this request to the linker) by cobc--A "-Wl,--no-as-needed" -lSDL2 ... (but the linker won't get the idea to optimize those out if your calls to the library are static what this issue is about, so "just to let you know").

Thanks for the tips! I'll try to look into this soon.


[ISSUE #1561]Support specify max reconsume times in message level

What is the purpose of the change
setting MAX_RECONSUME_TIMES in message has no effect to control max reconsume times, this pr is intending to fix this issue
Brief changelog
When user need to control max reconsume times for a specific message, user may set MAX_RECONSUME_TIMES in the message property, it should be effected. But currently, it is not, only max reconsume times in consumer level will be effected.
    consumer.registerMessageListener(new MessageListenerConcurrently() {

        @Override
        public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs,
            ConsumeConcurrentlyContext context) {
            System.out.printf("%s Receive New Messages: %s %n", Thread.currentThread().getName(), msgs);

            for (MessageExt msg : msgs) {
                if (msg.getTopic().equals("TOPICA")) {
                    MessageAccessor.setMaxReconsumeTimes(msg, "5");//reconsume to 5 times if consome message from topicA fail
                }
            }

            //do consume now
            if (consumefail){
                return ConsumeConcurrentlyStatus.RECONSUME_LATER;
            }

            return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
        }
    });

Coverage decreased (-0.09%) to 50.473% when pulling 9f2ee75e80b260633a56cbad42b88ee1f1baa7e2 on Jaskey:dev-support-maxreconsumetimes-in-msg into 6a6f2e7d59ae8f0d307ef5b16bd0d3d4931523d9 on apache:develop.

你好，你这个修改，只修改了并行消费模式的最大重试次数。。我觉得既然是个消息级别的设置，那么应该在各种消费模式下都能起作用。
我尝试写了一版功能，你们帮我Review一下吗？PR1570

@YuShanMuGong  Yes, this is only effective for concurrently service, since it is most popular. For orderly consume service, it may be different since it does not redeliever so it is not included in this pr

@duhenglucky  would you please help to review this pr?

Codecov Report

Merging #1562 into develop will increase coverage by 0.04%.
The diff coverage is 0.00%.

@@              Coverage Diff              @@
##             develop    #1562      +/-   ##
=============================================
+ Coverage      45.42%   45.47%   +0.04%
- Complexity      4242     4247       +5
=============================================
  Files            545      545
  Lines          35843    35850       +7
  Branches        4762     4764       +2
=============================================
+ Hits           16283    16304      +21
+ Misses         17511    17494      -17
- Partials        2049     2052       +3

Impacted Files
Coverage Δ
Complexity Δ

...lient/impl/consumer/DefaultMQPushConsumerImpl.java
8.72% <0.00%> (-0.11%)
11.00 <0.00> (ø)

...ain/java/org/apache/rocketmq/test/util/MQWait.java
43.24% <0.00%> (-10.82%)
3.00% <0.00%> (-1.00%)

...in/java/org/apache/rocketmq/test/util/MQAdmin.java
38.88% <0.00%> (-5.56%)
7.00% <0.00%> (-1.00%)

...ava/org/apache/rocketmq/test/util/VerifyUtils.java
46.26% <0.00%> (-2.99%)
14.00% <0.00%> (-1.00%)

...a/org/apache/rocketmq/filter/util/BloomFilter.java
59.13% <0.00%> (-2.16%)
25.00% <0.00%> (-1.00%)

.../rocketmq/broker/filter/ConsumerFilterManager.java
72.19% <0.00%> (-0.90%)
33.00% <0.00%> (ø%)

...rocketmq/client/impl/factory/MQClientInstance.java
46.82% <0.00%> (-0.60%)
82.00% <0.00%> (ø%)

.../apache/rocketmq/logging/inner/LoggingBuilder.java
63.60% <0.00%> (-0.32%)
3.00% <0.00%> (ø%)

...g/apache/rocketmq/client/impl/MQClientAPIImpl.java
12.63% <0.00%> (-0.22%)
25.00% <0.00%> (-1.00%)

...main/java/org/apache/rocketmq/store/CommitLog.java
67.22% <0.00%> (-0.11%)
80.00% <0.00%> (ø%)

... and 12 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 583f665...6c019c4. Read the comment docs.


Bug 365 compiled query with constant fails

fix 2 two inconsistent filenames
remove unused SetParameter(object query, NpgsqlParameter parameter)

ps: still 2 regression tests failing: Lifecycle: Regression, 21 successful, 2 failed
ps2: the faster tests are great!

Got it, thank you Tim.


UI Improvement: Issues with the navbar buttons and index sidebar

Code of Conduct

[X] I have read and agree to the GitHub Docs project's Code of Conduct

What article on docs.github.com is affected?

What changes are you suggesting?
Hi noticed some issues while going through this page.

[ ] The text should still be visible after hover
[ ] The text color of the index sidebar of the article is too light and hence difficult to read

Can I pick this up if it's okay?
Additional information
I'm using brave browser

@adarshajit Thanks so much for opening an issue! I'll triage this for the team to take a look :eyes:

Hi @adarshajit - We're aware of the issue, and are looking into it. Thank you for reporting. This seems to be happening to a very small percentage of users whose OS color mode preference may be set different to GitHub. For e.g. we've seen this when the OS is set to dark and GitHub is set to light. Let me know if that is not the case!

Awesome thanks for the confirmation. Since we are aware and looking through this, I'll close this issue for now. If anything else pops up please let us know! Thank you again!