Local DNS resolution doeesn't work on macOS with MagicDNS enabled

Bug description
When connected to a Headscale server where MagicDNS is enabled, local name resolution doesn't work anymore on macOS.
For example without Tailscale enabled my machine receives DNS configuration via DHCP from my home router.
The home router runs its own DNS server and resolves <router_name> to its own IP address.
Once connected to the Headscale Tailnet, <router_name> can no longer be resolved.
With the official Tailscale coordination server on the other hand there is an explicit option whether local DNS resolution should be overriden or not. If the toggle to override local DNS resolution is turned off, the router name can still be resolved.
So this seems like a bug in how Headscale configures the Tailscale client.
Context info

Version of headscale 0.15.0
Version of tailscale 1.16.1
OS macOS 12.4
config.yaml

dns_config:
  base_domain: 'example.com'
  domains: []
  magic_dns: true
  nameservers:
  - <IP_ADDRESS>

Output of scutil --dns
The output of scutil --dns looks different when connected to Tailscale vs Headscale.
Tailscale
DNS configuration

resolver #1
  search domain[0] : internal.example.com (note: probably because I configured scutil --set HostName to be <machine_name>.internal.example.com)
  search domain[1] : <my-email-address>.beta.tailscale.net
  search domain[2] : ts.net
  search domain[3] : <router DHCP domain_name>
  nameserver[0] : <router ipv4>
  nameserver[1] : <router ipv6>
  if_index : 13 (en7)
  flags    : Request A records, Request AAAA records
  reach    : 0x00020002 (Reachable,Directly Reachable Address)

resolver #2 - #65
  domain   : <64-127>.100.in-addr.arpa.
  nameserver[0] : <IP_ADDRESS>
  if_index : 28 (utun3)
  flags    : Supplemental, Request A records, Request AAAA records
  reach    : 0x00000003 (Reachable,Transient Connection)
  order    : 103002 - 103065

resolver #66
  domain   : 0.e.1.a.c.5.1.1.a.7.d.f.ip6.arpa.
  nameserver[0] : <IP_ADDRESS>
  if_index : 28 (utun3)
  flags    : Supplemental, Request A records, Request AAAA records
  reach    : 0x00000003 (Reachable,Transient Connection)
  order    : 103001

resolver #67
  domain   : <my email address>.beta.tailscale.net.
  nameserver[0] : <IP_ADDRESS>
  if_index : 28 (utun3)
  flags    : Supplemental, Request A records, Request AAAA records
  reach    : 0x00000003 (Reachable,Transient Connection)
  order    : 103000

resolver #68
  domain   : ts.net.
  nameserver[0] : <IP_ADDRESS>
  if_index : 28 (utun3)
  flags    : Supplemental, Request A records, Request AAAA records
  reach    : 0x00000003 (Reachable,Transient Connection)
  order    : 103066

resolver #69
  domain   : internal.example.com (note: this resolver comes from corporate IPSec VPN that I am connected to as well. this corporate VPN does split-tunneling. it takes over DNS for this subdomain)
  nameserver[0] : <corporate DNS 1>
  nameserver[1] : <corporate DNS 2>
  nameserver[2] : <corporate DNS 3>
  if_index : 27 (utun4)
  flags    : Supplemental, Request A records, Request AAAA records
  reach    : 0x00000003 (Reachable,Transient Connection)
  order    : 101200

resolver #70
  domain   : local
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300000

resolver #71
  domain   : 254.169.in-addr.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300200

resolver #72
  domain   : 8.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300400

resolver #73
  domain   : 9.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300600

resolver #74
  domain   : a.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300800

resolver #75
  domain   : b.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 301000

DNS configuration (for scoped queries)

resolver #1
  search domain[0] : <router DHCP domain_name>
  nameserver[0] : <router ipv4>
  nameserver[1] : <router ipv6>
  if_index : 13 (en7)
  flags    : Scoped, Request A records, Request AAAA records
  reach    : 0x00020002 (Reachable,Directly Reachable Address)

resolver #2
  search domain[0] : <router DHCP domain_name>
  nameserver[0] : <router ipv4>
  nameserver[1] : <router ipv6>
  if_index : 15 (en0)
  flags    : Scoped, Request A records, Request AAAA records
  reach    : 0x00020002 (Reachable,Directly Reachable Address)

resolver #3
  search domain[0] : internal.example.com
  nameserver[0] : <corporate DNS 1>
  nameserver[1] : <corporate DNS 2>
  nameserver[2] : <corporate DNS 3>
  if_index : 27 (utun4)
  flags    : Scoped, Request A records
  reach    : 0x00000003 (Reachable,Transient Connection)

resolver #4
  search domain[0] : <my_email_address>.beta.tailscale.net
  search domain[1] : 0.e.1.a.c.5.1.1.a.7.d.f.ip6.arpa
  search domain[2] : 100.100.in-addr.arpa
  search domain[3] : 101.100.in-addr.arpa
  search domain[4] : 102.100.in-addr.arpa
  search domain[5] : 103.100.in-addr.arpa
  search domain[6] : 104.100.in-addr.arpa
  search domain[7] : 105.100.in-addr.arpa
  search domain[8] : 106.100.in-addr.arpa
  search domain[9] : 107.100.in-addr.arpa
  search domain[10] : 108.100.in-addr.arpa
  search domain[11] : 109.100.in-addr.arpa
  search domain[12] : 110.100.in-addr.arpa
  search domain[13] : 111.100.in-addr.arpa
  search domain[14] : 112.100.in-addr.arpa
  search domain[15] : 113.100.in-addr.arpa
  search domain[16] : 114.100.in-addr.arpa
  search domain[17] : 115.100.in-addr.arpa
  search domain[18] : 116.100.in-addr.arpa
  search domain[19] : 117.100.in-addr.arpa
  search domain[20] : 118.100.in-addr.arpa
  search domain[21] : 119.100.in-addr.arpa
  search domain[22] : 120.100.in-addr.arpa
  search domain[23] : 121.100.in-addr.arpa
  search domain[24] : 122.100.in-addr.arpa
  search domain[25] : 123.100.in-addr.arpa
  search domain[26] : 124.100.in-addr.arpa
  search domain[27] : 125.100.in-addr.arpa
  search domain[28] : 126.100.in-addr.arpa
  search domain[29] : 127.100.in-addr.arpa
  search domain[30] : 64.100.in-addr.arpa
  search domain[31] : 65.100.in-addr.arpa
  search domain[32] : 66.100.in-addr.arpa
  search domain[33] : 67.100.in-addr.arpa
  search domain[34] : 68.100.in-addr.arpa
  search domain[35] : 69.100.in-addr.arpa
  search domain[36] : 70.100.in-addr.arpa
  search domain[37] : 71.100.in-addr.arpa
  search domain[38] : 72.100.in-addr.arpa
  search domain[39] : 73.100.in-addr.arpa
  search domain[40] : 74.100.in-addr.arpa
  search domain[41] : 75.100.in-addr.arpa
  search domain[42] : 76.100.in-addr.arpa
  search domain[43] : 77.100.in-addr.arpa
  search domain[44] : 78.100.in-addr.arpa
  search domain[45] : 79.100.in-addr.arpa
  search domain[46] : 80.100.in-addr.arpa
  search domain[47] : 81.100.in-addr.arpa
  search domain[48] : 82.100.in-addr.arpa
  search domain[49] : 83.100.in-addr.arpa
  search domain[50] : 84.100.in-addr.arpa
  search domain[51] : 85.100.in-addr.arpa
  search domain[52] : 86.100.in-addr.arpa
  search domain[53] : 87.100.in-addr.arpa
  search domain[54] : 88.100.in-addr.arpa
  search domain[55] : 89.100.in-addr.arpa
  search domain[56] : 90.100.in-addr.arpa
  search domain[57] : 91.100.in-addr.arpa
  search domain[58] : 92.100.in-addr.arpa
  search domain[59] : 93.100.in-addr.arpa
  search domain[60] : 94.100.in-addr.arpa
  search domain[61] : 95.100.in-addr.arpa
  search domain[62] : 96.100.in-addr.arpa
  search domain[63] : 97.100.in-addr.arpa
  search domain[64] : 98.100.in-addr.arpa
  search domain[65] : 99.100.in-addr.arpa
  search domain[66] : ts.net
  nameserver[0] : <IP_ADDRESS>
  if_index : 28 (utun3)
  flags    : Scoped, Request A records, Request AAAA records
  reach    : 0x00000003 (Reachable,Transient Connection)

Headscale
DNS configuration

resolver #1
  search domain[0] : internal.example.com
  search domain[1] : <headscale namespace>.example.com
  search domain[2] : <router DHCP domain_name>
  nameserver[0] : <IP_ADDRESS>
  if_index : 27 (utun4)
  flags    : Supplemental, Request A records, Request AAAA records
  reach    : 0x00000003 (Reachable,Transient Connection)
  order    : 103400

resolver #2
  nameserver[0] : <router ipv4>
  nameserver[1] : <router ipv6>
  if_index : 13 (en7)
  flags    : Request A records, Request AAAA records
  reach    : 0x00020002 (Reachable,Directly Reachable Address)
  order    : 200000

resolver #3
  domain   : <headscale namespace>.example.com.
  nameserver[0] : <IP_ADDRESS>
  if_index : 27 (utun4)
  flags    : Supplemental, Request A records, Request AAAA records
  reach    : 0x00000003 (Reachable,Transient Connection)
  order    : 103401

resolver #4
  domain   : internal.example.com
  nameserver[0] : <corporate DNS 1>
  nameserver[1] : <corporate DNS 2>
  nameserver[2] : <corporate DNS 3>
  if_index : 28 (utun3)
  flags    : Supplemental, Request A records, Request AAAA records
  reach    : 0x00000003 (Reachable,Transient Connection)
  order    : 101000

resolver #5
  domain   : local
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300000

resolver #6
  domain   : 254.169.in-addr.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300200

resolver #7
  domain   : 8.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300400

resolver #8
  domain   : 9.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300600

resolver #9
  domain   : a.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300800

resolver #10
  domain   : b.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 301000

DNS configuration (for scoped queries)

resolver #1
  search domain[0] : <router DHCP domain_name>
  nameserver[0] : <router ipv4>
  nameserver[1] : <router ipv6>
  if_index : 13 (en7)
  flags    : Scoped, Request A records, Request AAAA records
  reach    : 0x00020002 (Reachable,Directly Reachable Address)

resolver #2
  search domain[0] : <router DHCP domain_name>
  nameserver[0] : <router ipv4>
  nameserver[1] : <router ipv6>
  if_index : 15 (en0)
  flags    : Scoped, Request A records, Request AAAA records
  reach    : 0x00020002 (Reachable,Directly Reachable Address)

resolver #3
  search domain[0] : internal.example.com
  nameserver[0] : <corporate DNS 1>
  nameserver[1] : <corporate DNS 2>
  nameserver[2] : <corporate DNS 3>
  if_index : 28 (utun3)
  flags    : Scoped, Request A records
  reach    : 0x00000003 (Reachable,Transient Connection)

resolver #4
  search domain[0] : <headscale namespace>.example.com
  nameserver[0] : <IP_ADDRESS>
  if_index : 27 (utun4)
  flags    : Scoped, Request A records
  reach    : 0x00000003 (Reachable,Transient Connection)

=> So it seems with Tailscale my resolver number 1 stays my router, even for the Tailscale namespaces, while with Headscale resolver number 1 is <IP_ADDRESS>

I encountered some dns issue in Linux and OpenBSD machine, have not dive into these issues, but I think it may be same as yours.

After read some code, I think DefaultResolvers should be set to nil and a Route map item { '.' : '$resolver_set_in_config'} shoule be added into Routes.
I do not have a official tailscale account, maybe someone with official tailscale account could look into the log see how dns config looks like(keyword: dns: Set:)

Oh cool, that you have a hunch so quickly! @huskyii
I just checked on a different Linux machine, that should work as well, right?
In there I have the following line in the log:
tailscaled[32149]: router: dns: Set: {Nameservers:[<IP_ADDRESS>] Domains:[<email>.beta.tailscale.net] PerDomain:false Proxied:true

Is that what you are looking for?

Oh cool, that you have a hunch so quickly! @huskyii
I just checked on a different Linux machine, that should work as well, right?
In there I have the following line in the log:
tailscaled[32149]: router: dns: Set: {Nameservers:[<IP_ADDRESS>] Domains:[<email>.beta.tailscale.net] PerDomain:false Proxied:true

Is that what you are looking for?

Hi @felixscheinost which version of tailscale r u using?
The log is different from mine. I'm using tailscale 1.26.1
Here's my log, note: no router: before dns: Set:
tailscaled[51443]: dns: Set: {DefaultResolvers:[] Routes:{} SearchDomains:[] Hosts:0}

Isn't this related to #280 ? My understanding is that an option for deciding if you want to override the DNS servers, or just use "split-dns" as in adding an extra DNS server to your current network configuration is missing on the headscale side vs tailscale.
In some cases you might want to force the DNS servers in the configuration to be the only ones (i.e if you are using another node as an exit node and the node has a DNS server configured and you don't want to leak DNS requests to the outside world), and in other cases you only want to add an extra DNS server to your configuration so that you can resolve the internal tailscale/headscale names.
I am not sure of which mode it is the default right now, for example in one of my Ubuntu systems I see:
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver <IP_ADDRESS>
nameserver <IP_ADDRESS>
nameserver fd0c:989d:1024::1
search home.i.my-private.domain local

This system was joined with the following command:
tailscale up --exit-node=<IP_ADDRESS> --exit-node-allow-lan-access --login-server=https://vpn.my-private.domain
I guess the parameters you use to join the overlay network do also effect the resulting DNS servers in your system, as someone new to tailscale/headscale it would be great to get more details on how this is supposed to work in the documentation.

@huskyii I think I was using 1.24.2 at the time.
@vquicksilver Yeah, its probably a duplicate of that issue.
I tried working around the issuee right now by using the Go tailscaled and not the version from the macOS AppStore. But then I need to configure <IP_ADDRESS> manually. I couldn't find a way to do this without also overriding the DHCP DNS servers.

Okay, I just ran tailscaled manually on Linux against official Tailscale server.

  # Same
  Hosts:7
}
dns: Resolvercfg: {
  # Difference, when override local DNS is OFF these contains the DHCP resolvers
  Routes:{.:[<IP_ADDRESS> <IP_ADDRESS> 2606:4700:4700::1111 2606:4700:4700::1001]}
  # Same
  Hosts:7
  # Difference
  LocalDomains:[felix-scheinost.googlemail.com.beta.tailscale.net. beta.tailscale.net. ts.net.]+65arpa
}
dns: OScfg: {
  # Same
  Nameservers:[<IP_ADDRESS>]
  # Same
  SearchDomains:[felix-scheinost.googlemail.com.beta.tailscale.net.]
  # Same
  MatchDomains:[]
}

# Turn override local DNS OFF
wgengine: Reconfig: configuring router
wgengine: Reconfig: configuring DNS
dns: Set: {
  DefaultResolvers:[]
  Routes:{beta.tailscale.net.:[] felix-scheinost.googlemail.com.beta.tailscale.net.:[] ts.net.:[]}+65arpa
  SearchDomains:[felix-scheinost.googlemail.com.beta.tailscale.net.]
  Hosts:7
}
dns: Resolvercfg: {
  Routes:{.:[<IP_ADDRESS> fd00::de39:6fff:fe31:7433]}
  Hosts:7
  LocalDomains:[felix-scheinost.googlemail.com.beta.tailscale.net. beta.tailscale.net. ts.net.]+65arpa
}
dns: OScfg: {
  Nameservers:[<IP_ADDRESS>]
  SearchDomains:[felix-scheinost.googlemail.com.beta.tailscale.net. fritz.box.]
  MatchDomains:[]
}

I don't think headscale has a override_local_DNS option now, maybe we should add one, what confused me is that offical Tailscale KB for maigic DNS stated that

Your network must have at least one DNS nameserver enabled in the admin console. These nameservers will receive all DNS queries not handled by MagicDNS. This restriction will be relaxed in the future.

I think it's basically says that to use magic DNS, you need override local DNS.
@felixscheinost does magic DNS work if you turn off override local DNS ?
And could you please also take a look at /etc/resolv.conf and /etc/resolv.pre-tailscale-backup.conf

Your network must have at least one DNS nameserver enabled in the admin console. These nameservers will receive all DNS queries not handled by MagicDNS. This restriction will be relaxed in the future.

I think it's basically says that to use magic DNS, you need override local DNS.

This is incorrect, at least when using resolved.
MagicDNS sets in-addr.arpa records for all the IP address space used by tailscale.
Override local DNS enabled sets +DefaultRoute for the tailscale interface (meaning this should be the default DNS unless a more specific is found) and the global DNS domain "~."
The two settings are independent - however I have no idea how this works in systems pre-resolved.

@mlincett @huskyii @felixscheinost
I think I have added support for this correctly in https://github.com/juanfont/headscale/pull/905, can you please help me test it?

Sorry, I just got around to try out the new version.
BTW: Awesome that you have a flake.nix!
@kradalby If I set dns_config.override_local_dns = false then I can again use local resolvers but I can no longer resolve MagicDNS names

My config looks like this
db_path: /var/lib/headscale/db.sqlite
db_type: sqlite3
derp:
  auto_update_enable: false
  paths: []
  server:
    enabled: 'true'
    region_code: headscale
    region_id: '999'
    region_name: Headscale Embedded DERP
    stun_listen_addr: <IP_ADDRESS>:3478
  update_frequency: 24h
  urls: []
disable_check_updates: true
dns_config:
  base_domain: example.com
  domains: []
  magic_dns: true
  nameservers:
  - <IP_ADDRESS>
ephemeral_node_inactivity_timeout: 30m
listen_addr: <IP_ADDRESS>:443
log_level: info
noise:
  private_key_path: /var/lib/headscale/noise_private.key
oidc:
  client_id: ''
  domain_map: {}
  issuer: ''
private_key_path: /var/lib/headscale/private.key
server_url: https://hz-wireguard-gw1.symentis.com:443
tls_letsencrypt_cache_dir: /var/lib/headscale/.cache
tls_letsencrypt_challenge_type: HTTP-01
tls_letsencrypt_hostname: hz-wireguard-gw1.symentis.com
tls_letsencrypt_listen: :http
unix_socket: /run/headscale/headscale.sock

Output of scutil --dns
DNS configuration

resolver #1
  <my local router>
  if_index : 10 (en6)
  flags    : Request A records, Request AAAA records
  reach    : 0x00020002 (Reachable,Directly Reachable Address)

resolver #2
  <other VPN>
  if_index : 34 (utun5)
  flags    : Supplemental, Request A records, Request AAAA records
  reach    : 0x00000003 (Reachable,Transient Connection)
  order    : 100400

resolver #3
  domain   : local
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300000

resolver #4
  domain   : 254.169.in-addr.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300200

resolver #5
  domain   : 8.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300400

resolver #6
  domain   : 9.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300600

resolver #7
  domain   : a.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300800

resolver #8
  domain   : b.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 301000

DNS configuration (for scoped queries)

resolver #1
  <my local router>
  if_index : 10 (en6)
  flags    : Scoped, Request A records, Request AAAA records
  reach    : 0x00020002 (Reachable,Directly Reachable Address)

resolver #2
  <my local router>
  if_index : 14 (en0)
  flags    : Scoped, Request A records, Request AAAA records
  reach    : 0x00020002 (Reachable,Directly Reachable Address)

resolver #3
  <internal other VPN>
  if_index : 34 (utun5)
  flags    : Scoped, Request A records
  reach    : 0x00000003 (Reachable,Transient Connection)

So now it seems as if the Tailscale client isn't configuring any DNS settings at all.

same problem here

I'm trying out headscale this week and just ran into this problem myself.  It seems like the only working options using the official macOS Tailscale client are:

Run all DNS through Tailscale/Headscale
Run no DNS through Tailscale/Headscale

I guess maybe most users are intentionally routing DNS through the overlay network for added security so this is an uncommon issue?  My long-term plans for this do involve DNS so if there's not a workaround it's probably fine but was wondering if anyone had discovered anything new on this issue?

I am currently having the issue that MagicDNS breaks when override_local_dns is set to false (on Linux, Tailscale client v1.48.1). I am not sure if this is exactly the same issue, but at least it seems to be somehow related (and I did not want to open yet another issue for this).
MagicDNS only works with override_local_dns=true, but then all DNS queries are obviously sent through the resolvers configured in Headscale, which might not be what you want.
Someone also seems to have the same issue since #905 (https://github.com/juanfont/headscale/pull/905#issuecomment-1377224539).

Same here. Might have to stop using MagicDNS.

I am currently having the issue that MagicDNS breaks when override_local_dns is set to false (on Linux, Tailscale client v1.48.1). I am not sure if this is exactly the same issue, but at least it seems to be somehow related (and I did not want to open yet another issue for this).
MagicDNS only works with override_local_dns=true, but then all DNS queries are obviously sent through the resolvers configured in Headscale, which might not be what you want. When override_local_dns=false, all DNS queries are sent through the client's resolver (whichever that is), including queries designated for MagicDNS, making the resolution fail. I verified both observations using tcpdump.
A manual dig host1.magicdns.example.com @<IP_ADDRESS> works.
Someone also seems to have the same issue since #905 (#905 (comment)).

I am not currently using headscale but I guess it could be helpful if you could say something more about your DNS configuration.  In case you are using systemd-resolved you may want to attach the output of resolvectl status.

Just tried this and I can also confirm that apparently you really have only two choices with MacOS official tailscale client + headscale currently:
You either run ALL your DNS through there or none of it. override_local_dns doesn't seem to be respected, resolver will always try to route through headscale first if you're connected.


Failing to Parse Valid JSON

The library fails to parse seemingly-valid JSON
What is the issue you have?

This code snippet:
std::string jsonStr = R"({ "key": [ { "id": "this_is_a_string", "color": { "r": 125, "g": 23, "b": 78 } } ] })";
auto json = nlohmann::json::parse(jsonStr);

Fails with:
libc++abi.dylib: terminating with uncaught exception of type nlohmann::detail::parse_error: [json.exception.parse_error.101] parse error at 62: syntax error - unexpected string literal; expected '}'

What is the expected behavior?
No error to be thrown; the JSON is apparently valid:
https://jsonlint.com/?json={ "key": [ { "id": "this_is_a_string", "color": { "r": 125, "g": 23, "b": 78 } }%20]%20}
And what is the actual behavior instead?
A crash, seemingly when parsing the integers -- it works when the integers are quoted.

Which compiler and operating system are you using?

OSX, with
Apple clang version 11.0.3 (clang-1<IP_ADDRESS>)
Target: x86_64-apple-darwin19.5.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

Which version of the library did you use?

[ ] latest release version 3.7.3
[ ] other release - please state the version: ___
[x] the develop branch

If you experience a compilation error: can you compile and run the unit tests?

[ ] yes
[ ] no - please copy/paste the error message below

I cannot reproduce the issue with the same compiler.
Apple clang version 11.0.3 (clang-1<IP_ADDRESS>)
Target: x86_64-apple-darwin19.5.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

Full code for reference (using develop branch https://github.com/nlohmann/json/commit/29ad2178c666ad6eea7b653e44454bb54607a998):
#include <iostream>
#include "json.hpp"

using json = nlohmann::json;

int main() {
    std::string jsonStr = R"({ "key": [ { "id": "this_is_a_string", "color": { "r": 125, "g": 23, "b": 78 } } ] })";
    auto j = json::parse(jsonStr);
    std::cout << j << std::endl;
}

Output:
{"key":[{"color":{"b":78,"g":23,"r":125},"id":"this_is_a_string"}]}

Can you please double check your code?

I ran that exact code and got the same error... very strange!

Can you try to execute the unit tests?

My "develop" was old. Works as expected with latest version. Thank you :)

Thanks for checking back!

I ran that exact code and got the same error... very strange! Let me try with that specific branch.
I get the same error on version 3.7.3 ...
The error occurs occasionally, when I call a lot of json::parse func.

macOS 10.14.6

The library will accept any valid JSON as long as it’s UTF-8 encoded. Can you copy the exact input and the code you use to parse it?

Sorry, I can't provide the specific data, because I chunked a large file and called JSON:: parse to serialize it. I found this error in the log file, and I found that the error is not deterministic, even if the same input data. I will continue to test it with other third-party JSON libraries, and I will reply in time whether there is an error in my program itself or the cause of JSON:: parse.

If it’s nondeterministic, maybe you pass an invalid input buffer. Try running it with ASAN.

I will

It's my mistake , I mistakenly used the network library, resulting in a message error.
sorry to bother you.


Trim input strings in database_controller

#53  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @acco and the rest of your teammates on  Graphite


Cannot GET /token_pairs

file : http/v3.md
curl https://api.0x.org/token_pairs?page=3&perPage=20
returns
<!DOCTYPE html>

<meta charset="utf-8">
<title>Error</title>

Cannot GET /token_pairs

This should be https://api.0x.org/sra/asset_pairs?page=1&perPage=20 and any example code referencing token_pairs is not correct. It is only used in an Link Header example but should be updated to be asset_pairs.


Issues with <?php tag already included in my original file

A limitation is that can not be <?php in the beginning of $sData.
Say I load a php file with file_get_contents() which have the php opening tag <?php.
Then it will be an error, if I run Obfuscator.
To cope with this I can do this workaround:
$sData = file_get_contents('my.php');
$sData = substr($Data, 5);

Hi @halojoy
I hope I understand your problem.. If you get the contents directly from a PHP file, you can strip <?php easily like below to make sure it will work without any issues.
So, change your code to
$sData = file_get_contents('my.php');
$sData = str_replace(array('<?php', '<?'), '', $sData); // Strip PHP open/close tags

@halojoy Hi again! 😃
I updated README for giving you an even clearer example
https://github.com/pH-7/Obfuscator-Class#example-3
Ref: https://github.com/pH-7/Obfuscator-Class/commit/ac1c28d4691d2b8db8b90179a430000f42e66db8

I will close it for now. If you have another problem, please reopen the ticket.

FYI, I tested my example and works well 👍


Support Swift Package Manager for Xcode 11

👋
With Xcode 11 supporting SPM natively (and really well), Cocoapods will quickly die off, so it's a great idea to start thinking about SPM asap.
I'm not sure how this would work with PrettyTimestamp since it's in ObjC, perhaps there can be an exposed Swift wrapper around it, or better yet the whole thing gets re-written in Swift? Now that we have ABI stability the Swift code should be as future proofed as the ObjC code in terms of not needing to be updated too often.

Definitely keen and would be great to rewrite the interface some more. I'll see if I can find some time over the next few weeks once I'm back 👍


Spelling issue in test manufacturer 1343

Manufacturer with ID 1343 has a typo:
Is: Zhezhiang Orient Gene Biotech Co., Ltd
Should be: Zhejiang Orient Gene Biotech Co.,Ltd.
See also https://covid-19-diagnostics.jrc.ec.europa.eu/devices/detail/1343

This refers to

https://github.com/ehn-dcc-development/ehn-dcc-valuesets/blob/fdff116a58df3066116604865f96982fcf04947e/test-manf.json#L615

However, the RATs are not maintained in this repository anymore. The Gateway is pulling them directly from the database of JRC.
To avoid any confusion I would suggest taking the test-manf.json file completely out.
@ryanbnl @dslmeinte ?

@skounis Agreed, but we should also provide clear instructions on how, and preferably a script to obtain the test-manf.json from the Gateway.

Add to the agenda for Friday. The options are:

Leave as-is.
Rename test-manf-snapshot-yyyy-dd-mm.json so it's clear that this file is a snapshot!
Remove completely.
Remove but provide an example containing a small number of fictional examples, using a different but similar file name to avoid confusion.
Use github actions to automatically update the file daily.

I vote for option #4: replace test-manf.json with an example file with a different name.

Resolved


Fixes #32 - fixing heartbeats and potential disconnects during publishing

This change also suppresses warnings when writing to sockets, in favor of the exceptions that are thrown during application code error checking.
Fixes Issue #32

Not an active user of PHP+NSQ right now - but happy to merge if still relevant.

Awesome. Yeah, our usage doesn't work without this patch.


Use Array.prototype instead of creating an empty array

It's bad practice to create an empty array when we should be using Array.prototype for accessing the Array methods

Thanks, good catch!


logit lostash filters: interpret various more accurate timestamp fields

What
From a trawl of a month's worth of logs, these appear to be the fields already present that contain earlier, more accurate, host-generated timestamps. Some are ISO timestamps, some are unix epoch floats.
The ISO timestamps seem to be interpreted at some point after the logstash filters (?) (though not set to be the @timestamp value). The unix epoch values are ingested as strings, so not sortable.
How to review
Think we need to run make logit-filters and use the result to populate one of our logit stacks logstash configurations and testing it out.

🚨⚠ Please do not merge this pull request via the GitHub UI ⚠🚨

Hmmmmmmm this still doesn't appear to be interpreting unix epoch timestamps, will need to tinker further.

Will squash


Re-add TRCL Tests

This PR replicates #42 with master as the target branch.
Part of a series of PRs resolving #60

Thanks for resubmitting this @pshriwise - and to @YeChengKSU for originally authoring it a year ago!


TensorFlow code in Tesseract

All the files under src/lstm that stars with tf.
It is useless. There is no way to train a model in TF that Tesseract can use for inference.
Can we finally remove the TF code from Tesseract?

IIRC, the code in Tesseract uses some internal TF 1.x functionality instead of a stable API.

I think I can look in old releases how an OCR engine was added, so we can indeed remove the TensorFlow code now.


Content/beta

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.1 out of 2 committers have signed the CLA.:white_check_mark: nikolasburk:x: kuldarYou have signed the CLA already but the status is still pending? Let us recheck it.

This is amazing, thanks a lot @kuldar 🙏 seems like you still need to sign the CLA before we can merge though!


Add pre-commit hooks to container

As specified here
Run git hook installation setup either just inside Devcontainer or in Dockerfile as well.

As per instructions you can't actually force Git repository to use git hooks without user's opt-in. Which is a good thing, I guess. This means, that instructions need to be updated on that topic. Inside dev container I should think installation of Git hooks can be mandatory, so they'll work from inside Codespaces.

Agree. Dockerfile install was removed because otherwise papermill parameterisation tags were cleaned from the notebooks. Now this should not be an issue

The template should also be used dockerized by default, whether or not with code spaces


Add meta.description field to Document

meta field of documents currently only contains name and version, should also contain string field: description

Document.meta contains description


URL credentials are not used for downloading packages

Platform: amd64-darwin
UV version: 0.1.23

I'm using UV with an --index-url that contains basic auth credentials (e.g. for the form https://username@password:hostname/simple. It appears that UV is using the URL correctly to read the index, but on accessing individual packages it does not.
$ uv pip install --index-url https://[REDACTED]:[REDACTED]@[REDACTED]/artifactory/api/pypi/python-all/simple requests
error: Failed to download: requests==2.31.0
  Caused by: HTTP status client error (403 Forbidden) for url (https://[REDACTED]/artifactory/api/pypi/python-all/packages/packages/70/8e/0e2d847013cb52cd35b38c009bb167a1a26b2ce6cd6965bf26b47bc0bf44/requests-2.31.0-py3-none-any.whl#sha256=58cd2187c01e70e6e26505bca751777aa9f2ee0b7f4300988b709f44e013003f)

A similar request works without issues using Pip:
$ pip install --index-url https://[REDACTED]:[REDACTED]@[REDACTED]/artifactory/api/pypi/python-all/simple flask
Defaulting to user installation because normal site-packages is not writeable
Looking in indexes: https://[REDACTED]:****@[REDACTED]/artifactory/api/pypi/python-all/simple
Collecting flask
...

Thanks for the report! Can you reproduce this on v0.1.22 as well? We just changed some things here.
Could you also share the -v verbose logs with your password omitted?
Thanks!

Ah sorry this is actually fixed by https://github.com/astral-sh/uv/pull/2592 already it's just not released in 0.1.23 like I thought.
We'll have this released soon. You can see the pull request for details. Let me know if you have a chance to build from source to test it!


Notebook Editor doesn't seem to update as quickly as the interactive window

Run this code in both:
import time
from random import randint
from yaspin import yaspin

with yaspin(text="Loading", color="yellow") as spinner:
    time.sleep(2)  # time consuming code

    success = randint(0, 1)
    if success:
        spinner.ok("✅ ")
    else:
        spinner.fail("💥 ")

In the interactive window a nice looping effect occurs. In the notebook editor it skips frames for some reason.
This same thing can be seen with other looping output.

Not for me, not on a mac.
See below:

This same thing happens when outputting a ton of plots. Use this code here:
import matplotlib.pyplot as plt
import numpy as np

for i in range(100):
    plt.plot()
    N = 50
    x = np.random.rand(N)
    y = np.random.rand(N)
    colors = np.random.rand(N)
    area = (30 * np.random.rand(N)) ** 2  # 0 to 15 point radii
    plt.title(f"plot #{i + 1}")
    plt.scatter(x, y, s=area, c=colors, alpha=0.5)
    plt.show()

Not to fix in December, but see if it's still an issue after redux addtion.

Hard to tell if this repros or not. Spinners work, but a tensorflow situation is really slow. Opening a new issue for tensorflow issue.
https://github.com/microsoft/vscode-python/issues/8652


Adding ${GO_GCFLAGS} to the go test

Signed-off-by: Sudeesh John<EMAIL_ADDRESS>This patch is to make use all the build flags for testing too. This will help to catch the issues like   https://github.com/containerd/containerd/issues/3005 well ahead.

Looks like there's an issue with your DCO sign-off;
0.05s$ DCO_VERBOSITY=-q ../project/script/validate/dco
/home/travis/gopath/bin/git-validation
 d544204 - FAIL - does not have a valid DCO
1 commits to fix
The command "DCO_VERBOSITY=-q ../project/script/validate/dco" exited with 1.

Can you change
Signed-off-by: Sudeesh John<EMAIL_ADDRESS>
to
Signed-off-by: Sudeesh John<EMAIL_ADDRESS>
(adding the pointy-brackets)?
If you have configured your Git correctly (user.name and user.email are set in your git config), you can also use the -s (git commit -s) flag to sign-off commits

After the DCO issue, the next issue (which I brought up on our -dev slack channel yesterday when doing this myself) is that you will end up with this failure for any architecture/platform which enables race detection:
+ test
-buildmode=pie not supported when -race is enabled
Makefile:162: recipe for target 'test' failed
make: *** [test] Error 1

So it isn't as simple as just adding the flags to all test invocations. We will have to think of another way to do this.

Looks like @mkumatag has verified that running make integration runs against a -buildmode=pie built binary (because containerd is os.Exec'ed in daemon_test.go) and reproduces the Golang regression.
That seems to be a viable alternative to adding -buildmode=pie to the other test targets in Makefile and then having to deal with the fallout of race detector vs. PIE/PIC enablement.

how about adding the -buildmode=pie to the TESTFLAGS
diff --git a/Makefile.linux b/Makefile.linux
index d9dda903..59f194cd 100644
--- a/Makefile.linux
+++ b/Makefile.linux
@@ -21,6 +21,7 @@ COMMANDS += containerd-shim containerd-shim-runc-v1
 # check GOOS for cross compile builds
 ifeq ($(GOOS),linux)
        GO_GCFLAGS += -buildmode=pie
+       TESTFLAGS  += -buildmode=pie
 endif

 # amd64 supports go test -race

@sudeeshjohn the problem with adding it to TESTFLAGS is the same issue; the -race detector is incompatible with -buildflags=pie so we can't add it to any flags without turning off race detection, which is not good either.
As discussed offline, a better alternative is to make sure that separate CI runs use make integration target to get coverage on the -buildmode=pie-built binary, as running make integration will cause the containerd binary to be os.Exec()'ed (via the root package test's use of daemon_test.go). I think making sure other architectures run make integration is the best case at the moment without adding another run of unit tests without race detection, but with -buildmode=pie.


message_edit: Use disabled instead of readonly to disable controls.

Currently when a user does not have the permission to edit the topic/content of a message, the edit UI/view source UI correctly shows a greyed out topic/message-content input field, however these fields incorrectly have a click behavior, so to fix this we now would want to use disabled prop instead of readonly attribute as readonly controls can still function and are still focusable whereas disabled controls can not receive focus and are unclickable.
Fixes #22565.

Fixes: #22565

Screenshots and screen captures:
Self-review checklist

[x] Self-reviewed the changes for clarity and maintainability
(variable names, code reuse, readability, etc.).

Communicate decisions, questions, and potential concerns.

[x] Explains differences from previous plans (e.g., issue description).
[x] Highlights technical choices and bugs encountered.
[x] Calls out remaining decisions and concerns.
[x] Automated tests verify logic where appropriate.

Individual commits are ready for review (see commit discipline).

[x] Each commit is a coherent idea.
[x] Commit message(s) explain reasoning and motivation for changes.

Completed manual review and testing of the following:

[x] Visual appearance of the changes.
[x] Responsiveness and internationalization.
[x] Strings and tooltips.
[x] End-to-end functionality of buttons, interactions and flows.
[x] Corner cases, error conditions, and easily imagined bugs.

The plan for this is TBD pending discussion in https://chat.zulip.org/#narrow/stream/101-design/topic/move.20.2F.20edit.20message.20UI

I think we're pretty likely to want close this in favor of #23139.

Closing as the issue was fixed by https://github.com/zulip/zulip/pull/23139. Thanks @jai2201!


Empty github url crashes grader

Empty string for the github url crashes the grader. It would be nice if it printed out a message like "missing github for ___ student." And then continued on.

Fixed in 1.1.18


feat(curriculum): adding content for form maps and sets review page

Checklist:

[x] I have read and followed the contribution guidelines.
[x] I have read and followed the how to open a pull request guide.
[x] My pull request targets the main branch of freeCodeCamp.
[x] I have tested these changes either locally on my machine, or GitPod.

Also, as a sidenote, I believe we updated the review page dashed name to maps and sets instead of maps, sets and JSON since the JSON content was moved to an earlier module.
So I would look in the previous commits from the past day or two to see if Tom's module name PR was merged in. If so, then we will need to merge the latest changes from main into this PR

I think I only renamed the title on the main page:

These others still need to be fixed. Also, perhaps we could rename the folders and dashed names as well.

@jdwilkin4 @moT01,
I hope these changes are good to go:


Changelog doesn't contain 1.17.0 release notes

Hey @jcgruenhage,
with the last release we moved to publish the changelog in Github Releases. Therefore it is "expected".

It might be intentional, but as a user it's certainly not expected. Could you at least as a note about this in the changelog file?
I'd still prefer if there was some sort of central place containing the changes for all versions, butt that's your call of course.


Rule Suggestion: Hafnium / Microsoft Exchange Server RCEs

Motivation
https://www.microsoft.com/security/blog/2021/03/02/hafnium-targeting-exchange-servers/
https://techcrunch.com/2021/03/02/microsoft-says-china-backed-hackers-are-exploiting-exchange-zero-days/
https://www.volexity.com/blog/2021/03/02/active-exploitation-of-microsoft-exchange-zero-day-vulnerabilities/
I just ran into this and thought it might be an interesting use case for ModSecurity, especially considering the considerable number of enterprises using Exchange. From the limited reading I've done, it looks like the initial exploit vector is web-based and can be detected at the HTTP level.
Proposed solution

Alternatives

Additional context

Hello @ossie-git, when looking into it, I thought the information not comprehensive enough to do a rule that really prevents this (vs. a false sense of security with a limited rule).
Can you pinpoint the exact information in the docs you quoted?

There isn't too much information but I think we can get a general rule in that can be tweaked as needed. From the Veloxity article:

In order to exploit this vulnerability, the attacker must also identify the fully qualified domain name (FQDN) of the internal Exchange server(s). Using a series of requests, Volexity determined that this information could be extracted by an attacker with only initial knowledge of the external IP address or domain name of a publicly accessible Exchange server. After this information is obtained, the attacker can generate and send a specially crafted HTTP POST request to the Exchange server with an XML SOAP payload to the Exchange Web Services (EWS) API endpoint. This SOAP request, using specially crafted cookies, bypasses authentication and ultimately executes the underlying request specified in the XML, allowing an attacker to perform any operation on the users’ mailbox.
Volexity has observed this attack conducted via OWA. The exploit involved specially crafted POST requests being sent to a valid static resources that does not require authentication. Specifically, Volexity has observed POST requests targeting files found on the following web directory:
/owa/auth/Current/themes/resources/
This folder contains image, font, and cascading style sheet files. Using any of these files for the POST request appears to allow the exploit to proceed. If a file such as /owa/auth/logon.aspx or simply a folder such as /owa/auth/ were to be used, the exploit will not work.

For the record: Snort has released some rules for subscribed users: https://snort.org/advisories/talos-rules-2021-03-02
I am not subscribed, though, so I do not know.
Can we prevent POST requests to said path and Exchange will continue to work? And is not that folder but one of several options where auth can be bypassed?
I do not want to push an incomplete prevention layer giving people a false sense of security and they no longer patch their Exchange.

Hi,
I've created a rule for this CVE based on the public report of the vulnerability and some research that is being published.
https://raw.githubusercontent.com/s-ribeiro/ModsecurityRules/main/CVE-2021-27065.rule
Because there is no public exploit (to my knowledge) I could not validate Its accuracy.
So the rule has the action "pass" and my suggestion to anyone that wants to use it is to test it first and then If no false positives detected, change modify it to the wished action.
Any suggestion or feedback is welcome.


Performance optimizitations round 1

Description
This service is taking way more time than expected when dealing with our production data. These "small" optimisations aim to improve its performance  by reducing to a minimum the amount of work done. The idea is to decide if there is a need to introduce concurrency  AFTER verifying the improvements in production.
Type of change

Breaking change (fix or feature that would cause existing functionality to not work as expected)
Refactor (refactoring code, removing useless files)

Testing steps

Updated existing UTs and validated new behaviour through previously tested scenarios
make bdd_tests passes
Each commit of this PR has been tested and benchmarked separately to see if it optimises the processing or not.

Checklist

[x] make before_commit passes
[ ] updated documentation wherever necessary
[x] added or modified tests if necessary
[ ] updated schemas and validators in insights-data-schemas in case of input/output change

It looks ok!

I'll push it all the way to prod and monitor the first runs today and tomorrow.


soundcloud: ERROR: Unable to download JSON metadata ...

youtube-dl --verbose --prefer-insecure -g -f"best" http://api.soundcloud.com/tracks/241023603
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'--verbose', u'--prefer-insecure', u'-g', u'-fbest', u'http://api.soundcloud.com/tracks/241023603']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2015.12.18
[debug] Python version 2.7.3 - Linux-3.2.0-64-virtual-x86_64-with-Ubuntu-12.04-precise
[debug] exe versions: avconv v12_dev0-355-g28396d1, avprobe v12_dev0-355-g28396d1, ffmpeg 0.8.17-4, ffprobe 0.8.17-4, rtmpdump 2.4
[debug] Proxy map: {}
ERROR: Unable to download JSON metadata: HTTP Error 404: Not Found (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/common.py", line 329, in _request_webpage
return self._downloader.urlopen(url_or_request)
File "/usr/local/lib/python2.7/dist-packages/youtube_dl/YoutubeDL.py", line 1884, in urlopen
return self._opener.open(req, timeout=self._socket_timeout)
File "/usr/lib/python2.7/urllib2.py", line 406, in open
response = meth(req, response)
File "/usr/lib/python2.7/urllib2.py", line 519, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python2.7/urllib2.py", line 444, in error
return self._call_chain(*args)
File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 527, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)

Sorry, track was not available, no Issue


Refresh caches + no internet = normal progress

From mail:

disable all internet connections
open a list, click menu, refresh (or do it with a single cache?)
the progress dialog appears, no warning, that there's no internet

Confirmed....
It is just looking like normal (ver fast) progress.
It is "progressing" faster than when high speed internet access is availabe.

Yes, this is expected, because the loading fails very fast :)
When should an error be signalled? When the cache description cannot be refreshed? When the logs cannot be refreshed? Either? Both?

I think the absence of a connection should be detected whenever it happens (can be at any point in time during the refresh) and signaled to the user. The the process should stop.

What I'm saying is that we have at least three possible causes of failure:

Internet might be down
geocaching.com might be unreachable or useless (because Internet is down or geocaching.com is down or in maintenance)
a site hosting an image might be down (because Internet is down or the site is down or in maintenance)

I'd prefer to detect the second event (geocaching.com being unusable) rather than the first one. Internet might be up and the refresh process might fail anyway because of the second one.

I think we should display a short report at the end (as a toast) how many caches were refreshed correctly, how many errors and maybe why.

Being able to distinguish between 'no network' and 'gc.com not responding' could be beneficial anyway, because we can inform the user accordingly. Telling the user 'Connectivity (3G/WLAN) is switched off' or 'No network in reach' or gc.com is not responding, perhaps overload or maintenance' is much more helpful and professional then just 'could not reach gc.com. Check network or try again later' or something.
But this is something that not only relates to cache refreshing but should be baked into c:geo in all areas were an online interaction might happen. And the login state and its cause (no credentials, login failed, token invalid, gc.com behaving badly, ...) are just another facette of the same problem IMHO.


Use le_felt for comparison

This goes for asserts too. We need to use the _felt suffix versions of the assert functions to ensure they work with values in the full range up to 251 bits and not just up to 2**128 (range check bound)

thought more about this and had a change of opinion. We do two types of number comparisons: ones on uint256s (voting power, quorum etc), and ones on timestamps.  Timestamps will always be smaller than the range check bound and therefore its more efficient to use the comparison functions only defined over this range.


Query Augmentations: Issue :from randaugment import RandAugment

Hi thanks for your continuous support, I have now reproduced results till 29.9 but I don't know where are we adding query augmentations,  I tried to use randaugment to import RandAugment but it's giving me an error. Any hint as i has spent lot of time googling it to fix this error.
Thanks again for all your time and response

Please follow the details in the paper for implementing the query augmentations.  We currently don't have this feature in this repo.

ok,Thank you so much for your support and guidance, i really cant thank you much for your response and guidance.
Thanks again


Add semantic types from Talend documentation

Talend ETL supports detection of semantic types https://help.talend.com/r/BfI632~wDyp37GnDv5xBlQ/wCacKIctt8IUkTeuFRJJ5Q
These types should be analyzed and added to the Metacrafter registry.

Add following:

[ ] SEDOL - Stock exchange daily official list
[ ] SE Social Security Number - Swedish person number
[ ] Name suffix - Name suffix (Inherited, academic, professional or military)
[ ] MX Estado Code - Mexican state code
[ ] MX Estado - Mexican state
[ ] Military suffix
[ ] French social security number
[ ] French municipality
[ ] French department
[ ] French region
[ ] Formal title
[ ] German postal code
[ ] Continent name
[ ] Continent code
[ ] Color hexadecimal code
[ ] Civility
[ ] Canadian province code
[ ] Canadian province
[ ] Bulgarian VAT number
[ ] Belgian postal code
[ ] Austrian VAT number
[ ] Animal
[ ] Academic and professional title


Make the title case style guide configurable

This works for the title func and the other places where Hugo makes title case.


Unable to scroll horizontal, PC & WEB, ListView & SingleChildScrollView

Steps to Reproduce

flutter run

when i use scroll vertical, it's correct.

Container(
      height: 200,
      width: 600,
      color: const Color.fromARGB(255, 227, 223, 223),
      child: SingleChildScrollView(
        scrollDirection: Axis.vertical,
        padding: const EdgeInsets.symmetric(horizontal: 10),
        child: Column(
          children: children,
        ),
      ),
    );

but When I changed to horizontal, it won't work anymore. unable to scroll, whether using the mouse or scroll wheel

Container(
      height: 200,
      width: 600,
      color: const Color.fromARGB(255, 227, 223, 223),
      child: SingleChildScrollView(
        scrollDirection: Axis.horizontal,
        padding: const EdgeInsets.symmetric(horizontal: 10),
        child: Row(
          children: children,
        ),
      ),
    );

Even if I use the official example, it is still incorrect:
https://docs.flutter.dev/cookbook/lists/horizontal-list
or this: https://flutter.cn/docs/cookbook/lists/horizontal-list , this page builtin a dartpad.

full test code: https://gist.github.com/alexsunday/13738cafd5a3608e1ce0afb76a449e39
Expected results:
A horizontal scroll bar be visible,  or it can be use a mouse or gesture to scroll
Actual results:
cannot scroll child elements.

Code sample

import 'package:flutter/material.dart';

class TestSingleScroll extends StatelessWidget {
  const TestSingleScroll({Key? key}) : super(key: key);

  @override
  Widget build(BuildContext context) {
    const n = 10;
    final List<Widget> children = [];
    for (int i = 0; i != n; i++) {
      children.add(Container(
          width: 100,
          height: 100,
          decoration: BoxDecoration(
            border: Border.all(color: Colors.green, width: 1),
          ),
          child: Center(
            child: Text("$i"),
          )));
    }
    return Container(
      height: 200,
      width: 600,
      color: const Color.fromARGB(255, 227, 223, 223),
      child: SingleChildScrollView(
        scrollDirection: Axis.vertical,
        padding: const EdgeInsets.symmetric(horizontal: 10),
        child: Column(
          children: children,
        ),
      ),
    );
  }
}

void main() {
  runApp(const MaterialApp(
    debugShowCheckedModeBanner: false,
    home: Scaffold(
      body: TestSingleScroll(),
    ),
  ));
}

Logs

Flutter assets will be downloaded from https://storage.flutter-io.cn. Make sure you trust this source!
[√] Flutter (Channel stable, 3.7.6, on Microsoft Windows [版本 10.0.19045.2728], locale zh-CN)
    • Flutter version 3.7.6 on channel stable at F:\flutter\flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 12cb4eb7a0 (5 weeks ago), 2023-03-01 10:29:26 -0800
    • Engine revision ada363ee93
    • Dart version 2.19.3
    • DevTools version 2.20.1
    • Pub download mirror https://pub.flutter-io.cn
    • Flutter download mirror https://storage.flutter-io.cn

[X] Windows Version (Unable to confirm if installed Windows version is 10 or greater)

[!] Android toolchain - develop for Android devices (Android SDK version 30.0.3)
    • Android SDK at d:\Android\sdk
    X cmdline-tools component is missing
      Run `path/to/sdkmanager --install "cmdline-tools;latest"`
      See https://developer.android.com/studio/command-line for more details.
    X Android license status unknown.
      Run `flutter doctor --android-licenses` to accept the SDK licenses.
      See https://flutter.dev/docs/get-started/install/windows#android-setup for more details.

[X] Chrome - develop for the web (Cannot find Chrome executable at .\Google\Chrome\Application\chrome.exe)
    ! Cannot find Chrome. Try setting CHROME_EXECUTABLE to a Chrome executable.

[√] Visual Studio - develop for Windows (Visual Studio Community 2022 17.4.4)
    • Visual Studio at D:\Program Files\Microsoft Visual Studio\2022\Community
    • Visual Studio Community 2022 version 17.4.33213.308
    • Windows 10 SDK version 10.0.22000.0

[√] Android Studio (version 4.0)
    • Android Studio at D:\Program Files\Android\Android Studio
    • Flutter plugin version 51.0.1
    • Dart plugin version 193.7547
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b01)

[√] Connected device (2 available)
    • Windows (desktop) • windows • windows-x64    • Microsoft Windows [版本 10.0.19045.2728]
    • Edge (web)        • edge    • web-javascript • Microsoft Edge 111.0.1661.62

[√] HTTP Host Availability
    • All required HTTP hosts are available

! Doctor found issues in 3 categories.

I can scroll horizontally using Shift + mouse wheel. You have to use shift as that is the keybinding for horizontal scrolling.
I do not get a scrollbar when scrolling horizontally, while I do get one when scrolling vertically.
Not sure if not having the horizontal scrollbar is the bug you are referring to?
Code sample:
import 'package:flutter/material.dart';

void main() => runApp(MyApp());

class MyApp extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Flutter Demo',
      debugShowCheckedModeBanner: false,
      theme: ThemeData(
        primarySwatch: Colors.blue,
      ),
      home: const MyHomePage(title: 'Flutter Demo Home Page'),
    );
  }
}

class MyHomePage extends StatefulWidget {
  final String title;

  const MyHomePage({
    Key? key,
    required this.title,
  }) : super(key: key);

  @override
  State<MyHomePage> createState() => _MyHomePageState();
}

class _MyHomePageState extends State<MyHomePage> {
  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: Text(widget.title),
      ),
      body: Center(
        child: Container(
          height: 200,
          width: 600,
          color: const Color.fromARGB(255, 227, 223, 223),
          child: SingleChildScrollView(
            scrollDirection: Axis.horizontal,
            padding: const EdgeInsets.symmetric(horizontal: 10),
            child: Row(
              children: [
                Container(
                  width: 200,
                  height: 100,
                  color: Colors.green,
                ),
                Container(
                  width: 200,
                  height: 100,
                  color: Colors.yellow,
                ),
                Container(
                  width: 200,
                  height: 100,
                  color: Colors.red,
                ),
                Container(
                  width: 200,
                  height: 100,
                  color: Colors.blue,
                ),
                Container(
                  width: 200,
                  height: 100,
                  color: Colors.green,
                ),
                Container(
                  width: 200,
                  height: 100,
                  color: Colors.yellow,
                ),
              ],
            ),
          ),
        ),
      ),
    );
  }
}

Hello @alexsunday. Try scrolling while holding shift to scroll horiontally

Hello @alexsunday. Try scrolling while holding shift to scroll horizontally

Microsoft Windows [VERSION 10.0.19045.2728] 64bit
both use
Shift + mouse wheel
shift + mouse click/gesture
unable scroll horizontally.

I believe this is a duplicate of https://github.com/flutter/flutter/issues/105095. While in that issue, it seems to be specific to the trackpad, I believe it might be the same issue you are experiencing. Therefore, I will close this issue in favor of that one. You can upvote the original comment of that issue, provide details in the thread and follow up with that issue. Thank you.


Please Update the Game!

Description

Steps to Reproduce

[First Step]
[Second Step]
[and so on...]

Expected behavior

Actual behavior

Reproduction frequency

Versions

Mods

Additional Information

There is a new update to terraria and i was wanting to play the full version with my friend and i cant because it says the mod loader isn't the same version as the regular game. Just in case you are wondering, i dont have any mods active but it still wont work


Recover from broken pipe

After scheduled disconnects I see endless broken pipe errors. How I can configure client to get rid of this worst behaviour? It can't reconnect for hours until application restarted.

Is there any way to reproduce behaviour you described?
Bad connections are automatically removed from the pool and client opens new connection instead of the bad one - https://github.com/go-redis/redis/blob/master/redis.go#L23-L37. The only exception to that rule is PubSub, which does automatic reconnection only if you use ReceiveMessage.

I don't use pubsub, I just created client:
func NewRedisClient(cfg *Config) *RedisClient {
	client := redis.NewClient(&redis.Options{
    	Addr:     cfg.Endpoint,
   	    Password: cfg.Password,
		DB:       cfg.Database,
    	PoolSize: cfg.PoolSize,
	})
    return &RedisClient{client: client}
}

and using it for multi and regular commands. Just try to create client, stop redis instance and run it again.
2015/11/13 02:03:52 write tcp [::1]:51737->[::1]:6379: write: broken pipe
2015/11/13 02:03:52 redis: Unwatch failed: write tcp [::1]:51737->[::1]:6379: write: broken pipe
2015/11/13 02:03:57 write tcp [::1]:51737->[::1]:6379: write: broken pipe
2015/11/13 02:03:57 redis: Unwatch failed: write tcp [::1]:51737->[::1]:6379: write: broken pipe

This is how you can reproduce it on your side:
package main

import (
	"gopkg.in/redis.v3"
	"time"
	"log"
)

var client *redis.Client

func main() {
	client = redis.NewClient(&redis.Options{
		Addr: "localhost:6379",
	})
	for {
		go func() {
			tx := client.Multi()
			defer tx.Close()

			res, err := tx.Exec(func() error {
				tx.HGet("foo", "bar")
				return nil
			})
			log.Println(res, err)
		}()

		time.Sleep(3000000000)
	}
}

It works as expected if there is no goroutine, but if i use goroutine, it ill never recover from lost connection. Also, it seems problem is only with multi.

Tested this code on v2 and it works ok.

Multi is special because it has to restore watches after reconnection. And since we don't have that functionality yet reconnection is disabled for multi...

Did you run my example in this ticket? There is infinite loop with multi, next calls should perform successfully, but there is endless broken pipe errors even if redis instance is available again. You wanna say that this is expected behaviour?

I wanted to say that it is expected behaviour, not desired.

Thanks, I was confused because it works in v2.


Feature/currency

Have built and a few of my suggestions:

On MacOS it is called ZCashUI in the top left corner, so maybe a search for that and replacing with ZelCashUI or ZelCashSwingUI would be great.

Then versioning is this V5.0.0? Currently it is set to V4.0.0

Would be nice to get a good screenshot.

Thoughts on the current Messaging feature? I seem to find it a bit annoying as it forces a setup when you click on it the first time and personally have only used it theoretically. But it is reasonable and nice to have... Maybe disabling and then replacing once we have a good messenger will be a good option?

Could we look at also replacing these icons, they look a little dated (old) compared to the new ones.

Grammar fix

It is suggested to restart your wallet after you change your settings.

Feature, maybe in the next version providing some default themes? So the user can pick some good combinations that work together without researching and editing. Under the same settings option just with additional options.

Great job @Cabecinha84


javascript and typescript: "conditions" configuration setting for conditional export resolution in package.json

I want vscode to resolve a "vscode" condition for conditional exports in my package.json. My use case is to resolve my TypeScript module, so that I can use "Go to" actions without having to bundle my code/ generate a source map.
Alternatively, vscode could add a setting for user conditions, like node.js does: https://nodejs.org/api/packages.html#resolving-user-conditions.

I want vscode to resolve a "vscode" condition for conditional exports in my package.json. My use case is to resolve my TypeScript module, so that I can use "Go to" actions without having to bundle my code/ generate a source map

Sorry but I don't fully understand what this means. Do you want conditional resolving of the vscode-module, e.g so that its import resolves to something else under some conditions?

I want vscode to resolve a "vscode" condition for conditional exports in my package.json. My use case is to resolve my TypeScript module, so that I can use "Go to" actions without having to bundle my code/ generate a source map

Sorry but I don't fully understand what this means. Do you want conditional resolving of the vscode-module, e.g so that its import resolves to something else under some conditions?

No, I want to VSCode to resolve JavaScript modules using a "vscode" conditional export in package.json:
https://nodejs.org/api/packages.html#conditional-exports
Actually now that I think about it, it would be even better if I could fully configure which conditions VSCode uses to resolve conditional exports, like node.js does using the --conditions flag: https://nodejs.org/api/cli.html#-c-condition---conditionscondition

Hey @jrieken, this issue might need further attention.
@vveisard, you can help us out by closing this issue if the problem no longer exists, or adding more information.

Hey, just mentioning this problem still exists.

Hi,
has anyone found a solution or workaround for this issue?
To give some more context, if one has referenced modules in a workspace where the package.json contains anything like:
{
  "name": "@workspace/module",
  ...
  "exports": {
      ".": {
          "dev": "./index.js",
          "import": "./dist/index.js"
      }
  }
  ...
}

(Notice the dev key, defining a conditional export.)
Suppose you have some file in the same workspace, someFile.js importing that module:
import something from '@workspace/module';

The module resolution will change when node is invoked with the conditions flag:

node --conditions=dev index.js will resolve @workspace/module from the ./index.js file
node index.js will resolve @workspace/module from the ./dist/index.js file

But the /dist/ files are only generated when the module is actually built (that is, only in production), so VSCode doesn't seem to be able to autocomplete anything from @workspace/module (the files it looks for don't exist).
It is possible to make auto-completion work by adding a hacky default to the end of exports in package.json, which would look like so:
{
  "name": "@workspace/module",
  ...
  "exports": {
      ".": {
          "dev": "./index.js",
          "import": "./dist/index.js",
+         "default": "./index.js"
      }
  }
  ...
}

But that solution is not ideal: not only does it pollute the package.json files for the sake of the specific editor being used, it can also create other errors if the package is used without being very careful about this export caveat.
Any help or pointers to resolve this are highly appreciated.

That default export hack only works when the output file does not exist.
I still haven't found any way around this issue.

Just stumbled on this issue myself and am wondering how it's not getting more attention haha

I just came across this issue and am a little disappointed it's not supported yet. The --conditions CLI option is still experimental, so I have no complaints!
It looks like the only workaround for the exports field is to generate source maps, but this is not as flexible as conditions.
For the imports, it's easier since you can map the desired paths by using the compilerOptions.paths in tsconfig.json.
I hope this feature gets more attention, though.

@romanstetsyk the beauty of --conditions is it lets you remove all paths configs from tsconfig files. Then use #whatever/foo for imports and lean on tsconfig references for resolution of other @bar/smth packages in a monorepo.
Then exports with conditions obs means you don't have have to delete dist every time you build.
Removing paths and all the associated compatibility packages for jest/vitest etc was a very satisfying thing to do :D

@dpnova, you're absolutely correct. Having it in vscode would simplify things a lot!

Note that TypeScript already supports this via the customConditions option.
It's a little bit cumbersome but doable manually, just add it to tsconfig.json/jsconfig.json, and create another tsconfig.build.json without this option for building (maybe not required if the build process can also reference the source file).
It would be interesting if vscode can automatically merge additional customConditions from its settings.


Extend connect API C++ to handle Viewer's later start or reconnect

Currently, the workflow of the logger-viewer implies that the viewer is running before logging starts, otherwise, it will lead to data accumulation (see #6933 ) and never connect. In practice, it is important to have the ability to know whether connect to the viewer was successful or not and monitor connectivity status somehow.
Describe the solution you'd like

Extend C++ API to indicate whether connect is successful or not - this would allow users to implement logic which attempts to connect.
Extend C++ API to indicate that the Viewer has been disconnected - this would allow users to handle such cases and initiate reconnect.

New API interfaces would allow users to handle these cases:

Viewer starts after logger: Logger could make periodical checks/attempts to connect and once it is connected, it will start logging.
Connected Viewer is closed or restarted: Logger should somehow monitor connection status and switch to periodical checks if the Viewer was disconnected.

I don't think this applies to the C++ sdk in particular and is rather an issue in all APIs


Updates example to use Spring MVC + Thymeleaf

Code copied from https://github.com/okta/samples-java-spring/tree/master/okta-hosted-login and updated to use Bootstrap 5

cc: @arvindkrishnakumar-okta

If you're copying from that repo, you might want to implement logout as well. https://github.com/okta-samples/okta-spring-boot-sample/issues/39

@bdemers It looks like logout kinda works. It logs you out from Spring Boot, but it does not log you out from Okta.


[Feature] Read Error Log files

This is new feature request.
I need to read error log for current reporting services on disk. I am not a powershell expert. what do you think about it ? i uploaded which i using script right now.
I am combining all errorlog and inserting datawarehouse db. After that analysis services powerbi etc. visualizing jobs. i think it would be a great feature.

Sounds like a nice feature to add, you don't have to be an expert to contribute, feel free to start with something small and build on top of that


Mac8 is failing multiple tests

Multiple infra failures happen on mac8 TOT with error:
Infra Failure
Infra Failure: Step('Dismiss dialogs.Run app to dismiss dialogs') (retcode: 65)

We will start with manually quarantining the device.

Deleted the dismiss dialog manually and restarted the device, that step is now passing. @keyonghan  did we implement removing old applications from phones as a recovery step?

Moving as P3 as the test bed is now passing correctly.

Deleted the dismiss dialog manually and restarted the device, that step is now passing. @keyonghan did we implement removing old applications from phones as a recovery step?

Yes. The recovery logic is there. Let me double check.

A device os typo is updated in the flutter ios recovery.

Thanks, closing this bug.


fix (AoT - Lazy load) : lazy modules chunks creation on Server bundle

Commented code that prevented lazy modules chunks from getting created on PlatformServer.
This change worked fine in Dev. and Production build in my project with lazy loading feature.

Updated to latest npm packages (Webpack 3, aspnet prerendering 3 etc.,).
Added bundle analyzer to visualize & investigate the modules client/browser bundle

@davidsekar are you sure resolve: { mainFields: ['main'] }, is the way to ensure server size bundles are created properly with Lazy loading? What about module-map-ngfactory-loader  see this https://github.com/angular/universal/issues/741#issuecomment-325508716

@naveedahmed1 You can see the comments provided in the same thread.
https://github.com/angular/universal/issues/741#issuecomment-324160888

there are many ways to solve this. One as you mentioned is replacing NgModuleFactoryLoader but if you're using webpack you can simply include angular into your webpack bundle which allows webpack to replace the System.import symbol.

It works well with this PR, since we are using webpack... I can confirm that
 as expected.

And what about the bundle size? the main-server is ~13mb rite?  Which is huge and will defiantly have performance issues when deployed on server especially in case of heavy traffic websites.

Server bundle is really a non-optimized pack of all required files
e.g.,

It is not minified and has lot of comments

(function(e, a) { for(var i in a) e[i] = a[i]; }(exports, /******/ (function(modules) { // webpackBootstrap
/******/ 	// The module cache
/******/ 	var installedModules = {};
/******/
/******/ 	// object to store loaded chunks
/******/ 	// "0" means "already loaded"
/******/ 	var installedChunks = {
/******/ 		1: 0
/******/ 	};
/******/
/******/ 	// The require function
/******/ 	function __webpack_require__(moduleId) {
....
/******/ 		};

Based on this repo, a production server bundle has complete 'ngx-bootstrap', 'angular', 'bootstrap.css'. in non-minified state. So it is expected to appear huge where as, when compiled to native code by node- V8 engine (comments and spaces doesn't matter any more).

Size really does matter in the network transfers on the client/browser side.

This is great David thank you! Webpack 3.x and such were definitely much needed :)
I'll give this a test spin just to make sure it's all working!

Thank you so much @davidsekar for the explanation. Can you please confirm if tree shaking is applied on the server bundle? Another thing, can we use @angular-devkit/build-optimizer to further optimize bundles both client and server side bundles? If we compare the size of the client side bundles generated through this repo with the bundles generated through Angular CLI, there is a huge difference, in my case bundles generated through Angular CLI are ~4 Mb whereas through this repo, the generated bundles for client side are over 7 mb.

Size of the server bundle doesn't mean anything as it will be compiled to native code.
But as per the request, to avoid confusion I've enabled webpack.optimize.UglifyJsPlugin() on server bundle too. This has reduced the bundle size

@davidsekar sounds great! But these haven't merged yet, correct? can you please share where can I view these changes?

Minifying server bundle broke preboot init(). still checking...

For me its working perfectly, I don't have ngx-bootstrap but material2 instead.

I have disabled mangling and compression (unsafe) for the server bundle, as it causes issue for preboot to create replay events scripts.
Now the bundle size is approx 2.9 MB, but this is fine considering everything works as expected.

I already have implemented a way to add Hash values to lazy chunks in my project based on this repo (only on production builds).
But I don't want to bundle everything(those changes too) in this PR, as it causes issues for the repo maintainers to validate.

Can we merge this one in @davidsekar ? Sorry I've been away, unbelievably busy the past week+

@davidsekar ssr in my project is also inspired from this repo (latest version) but I don't see any issue with compression. There must be some external library most probably bootstrap lib that would be breaking.

@naveedahmed1 When mangling or compression is ON, I see following error

So I think it is bit more safe this way. Thanks for your feedback and time checking this PR :)
@MarkPieszak All features along with Signalr chat system is working fine.
I just see following warning, which is related to unresolved parameters that may become an error in Angular 5

Warning: Can't resolve all parameters for SignalR in D:/OSP/aspnetcore-angular2-universal-fork/node_modules/ng2
-signalr/src/services/signalr.d.ts:

Excellent merged it for now, let's get latest and make sure everything works now just incase 🍻
Great work @davidsekar ! 👍

@davidsekar will this project not benefit from Webpack 3 scope hoisting?
https://medium.com/webpack/webpack-3-official-release-15fd2dd8f07b
I didn't see it in the webpack config.
module.exports = {
  plugins: [
    new webpack.optimize.ModuleConcatenationPlugin()
  ]
};

@davidsekar preboot doesn't work for me as well with enabled compress...

@KhalipskiSiarhei  Please use the webpack config from master.
Unsafe Compression and mangling are disabled to fix isue with preboot.
https://github.com/MarkPieszak/aspnetcore-angular2-universal/blob/ec03396a97aa6ff6f583678bca03057b369864a1/webpack.config.js#L83

@jrmcdona
This PR was to fix the lazy load issue...
Yes, scope hoisting is awesome & should be incorporated through another PR


create a manual garbage collection utility

Project:M36 currently uses a very simple Binary dump to store each transaction. However, if each commit grows the database, we can see O(n^2) growth in overall database size. As a short-term counter-measure, we could implement a garbage collection utility which would take an existing database and trim it to only its latest transactions, deleting all history before that.
Another possible way to reduce database size would be to implement some compression scheme.

C.f. #146

How is this GC supposed to know that a transaction is not used anymore in the snaphsot at a given time ? As far as I understand certain transactions can be removed without changing the current state and some of them are needed for the current state, perhaps some measure of information containement could be tied to the transactions when they are commited ?

This utility is sort on the backburner.
The reason garbage collection would be useful now is because our file format is overly simplistic (but correct). Each transaction is written out via Binary as a completely self-sufficient transaction without data dependencies on the previous transaction. Obviously, this is very wasteful in terms of IO and storage, so we have a new design coming down the pipeline.
I'm happy to discuss the proposed design in greater detail.

The new design should obviate or reduce the need for garbage collection though I could imagine a collection system which would aggregate old transactions into one when one no longer cares about history older than X.

@agentm Do you mean that with the previous design the entire database history was/is rewritten at every transaction ? I'm not sure I understand

That's almost right. The current, suboptimal design is that the DatabaseContext's record items are serialized via Binary to disk without referencing previous transactions. That means that every relation variables is re-serialized to disk on each commit. The benefit of this approach is mere simplicity and in validating correctness.
So, the entire database history is not rewritten, rather each new transaction's state is serialized to disk without consideration for what was already previously serialized.
I am currently working on the very obvious optimizations to reduce IO here.

Hi @agentm,
Do you have an idea when the new design might be implemented? Is there a ticket for it specifically?
From my perspective the current persistence strategy is the biggest issue with this project at the moment, as it not only requires excessive storage space, but also has massive performance implications. I would love to use project-m36 for an upcoming product, but I'm pretty sure that longer term performance will be unacceptable (say, with a state in the order of 100MB - 1GB).

@matchwood, I definitely empathize with you here. The current storage mechanism is a very basic implementation for durable writes and there is much low-hanging fruit to be eliminated. I am working on a storage backend using cborg, but cborg has its own set of problems.
I wish I could offer a better answer than "a few months away", but I have to contend with my daily non-Project:M36 tasks.
Please stay tuned- I think we will have some unique features in the storage space that other databases won't be able to match (such as O(1) commits), so I am excited to get them into the project.

I have started work on my own manual garbage collection utility. My use-case is for a long-running process that logs every few minutes and I was able to reduce the size from 8 MB for a few hundred entries to 80 KB.
This utility has a couple drawbacks in its current state:

It is not thread-safe. I still need to figure out how each thread accesses the transaction history and at what points they will create branches. Is branch creation completely manual or is there an automatic detection built-in to project-m36 somewhere?
All history is removed except for the most recent branch. I would like to provide some way to provide a custom function for determining which transactions to remove or keep.

I guess the third drawback of my utility would be

It works directly on the database files and not alongside the server.

I would like to get it to work alongside the server. In fact, my use case would benefit from that. I just need to do some more digging around to figure out how to do that.
I'm using the 0.6 release of project:M36 that was released on January 9th. I believe it does have the zip compression.
I read the documentation of the new storage architecture and I am intrigued. I would like to know if there is anything I could contribute to that effort. I figure I could start out with this little utility to help myself understand how the current design works and go from there.

Shoot, sorry that the compression didn't offer a temporary solution.
If you want to integrate it into the backend directly, take a look at the STM monad functions such as commitLock_. That block holds the FS lock while operating on the serialized transaction graph.
The future architecture will feature serialized graph diff thunks with simple cache management with all requests equally likely to end up the cache (thus, no fundamental "relation"-based storage).

This should no longer be an issue as of Project:M36 0.8 with the new backend rewrite. Read more about the new data independent architecture here.
The naive persistence layer is gone.


HTTPS still works in production build even though vite-plugin-mkcert is a dev dependency

Describe the bug
HTTPS still works in production build even though vite-plugin-mkcert is installed as a dev dependency. This plugin should only affect the development environment, but it appears to influence the production build as well.
If this is not an intended behavior, it might be a potential security concern as development certificates should not be used in production environments.
Reproduction
Steps to reproduce:

Install vite-plugin-mkcert as a dev dependency

npm install -D vite-plugin-mkcert

Configure vite.config.js

import { defineConfig } from 'vite';
import react from '@vitejs/plugin-react-swc';
import mkcert from 'vite-plugin-mkcert';

// https://vite.dev/config/
export default defineConfig({
  plugins: [react(), mkcert()],
});

Build and run the production version

npm run build
npm run preview

Expected behavior: HTTPS should only work in development environment
Actual behavior: HTTPS continues to work in production build
System Info
Output of npx envinfo --system --npmPackages vite,vite-plugin-mkcert --binaries --browsers:
  System:
    OS: Windows 11 10.0.22631
    CPU: (12) x64 AMD Ryzen 5 5600X 6-Core Processor
    Memory: 4.82 GB / 15.91 GB
  Binaries:
    Node: 20.3.1 - C:\Program Files\nodejs\node.EXE
    npm: 9.6.7 - C:\Program Files\nodejs\npm.CMD
  Browsers:
    Edge: Chromium (129.0.2792.52)
    Internet Explorer: 11.0.22621.3527
  npmPackages:
    vite: ^5.2.10 => 5.4.11
    vite-plugin-mkcert: ^1.17.6 => 1.17.6

Used package manager: npm

export default defineConfig(({ command, mode, isSsrBuild, isPreview }) => {
  return {
    plugins: [react(), !isPreview&&mkcert()],
  }
})


Final loss when you train your model

Hi I am trying to train a SSD using xxyy and KL loss. May I ask the final loss when you train your model using xxyy and KL loss? It seems location_loss can be a negative number, is that normal?

The loss can be a negative number. BTW, please have a look at our CVPR version code: https://github.com/yihui-he/KL-Loss
The performance is improved.

have a look at YOLOv3 + KL-loss https://github.com/yihui-he/KL-Loss#pytorch-re-implementations


http://<<DNS_NAME>>/rules?is_active=true works only when local ZZ server is running

We are working together with Ricardo on our custom rules creation / addition to the Zalando-Zally framework.
In preparation to Production deployment I’ve installed all per-requisites and Z-Z framework on the Linux RedHat 7.4 server which is the per-production PAT environment. ./build-and-run.sh script runs perfect and creates three Docker images as it should be. Code was pulled out from the ZZ Git, so it seems there is no cert problem between the Linux host and ZZ Git. We also created DNS name accessible from the corporate network or VPN. Everything works fine, but the ‘Rules’ link (http://<<DNS_NAME>>/rules?is_active=true) brings the error:
ERROR
Ooops not able to load supported rules!
Apparently the default error message ‘Rules.DEFAULT_ERROR_MESSAGE = 'Ooops not able to load supported rules!'’ is thrown from the rules.jsx class due to some issue with this.set.State(…………..) method, but !!!! ……
When each of us starts our local Z-Z instance (server) the ‘Rules’ link (http://<<DNS_NAME>>/rules?is_active=true)  starts to work redirecting to the Git repo where the local build pointed to. For example, when I am starting my local build the http://<<DNS_NAME>>/rules?is_active=true link redirect to my local Git repo while the build on the server points to the Z-Z restful API guidelines.
The situation persists across different servers: each of us can invoke the ‘Rules’ link (http://<<VLAD_IP>>/rules?is_active=true), e.g., on another laptop only when the local version runs. Otherwise, it brings the same exception. When Ricardo starts his instance the link (http://<<VLAD_IP>>/rules?is_active=true) brings rules set from Ricardo’s Git not from the Git my instance pointed to.
Finally, when I installed Chrome on the Linux RedHat server and directly start browser on Linux, the link ((http://<<DNS_NAME>>/rules?is_active=true) works proper.  It displays the rules from the Git the build is pointing to (as it should be) matching the exact behavior of my local Debian Linux instance on Windows.
We are kind of puzzled by this cross-servers dependencies. What should be done in order to run this link from ANY laptop (browser) WITHOUT Zalando-Zally server running locally? The issue persists and identical for IE, Chrome, and FireFox browsers.
Any help would be highly appreciated.
Thanks,
@vivanov19

Please disregard this issue. It was an issue on our end and it is fixed. This issue should be closed


Correction for raw affiliation King's University College (Western University)

Correction needed for raw affiliation King's University College (Western University)
raw_affiliation_name: King's University College (Western University)
new_rors: 02grkyz14
previous_rors: 048f5b183;05cgtjz78;0257kt353
works_examples: W4384706336;W4360993502
contact: a91b6d766798a7e9963257edd3fa10b5:96f02c75ce288817 @ uwo.ca

This issue was accepted and ingested by the OpenAlex team on 2024-12-21. The new affiliations should be visible within the next 7 days.


⚠ Bofrost.at has degraded performance

In e28dbcc, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 2868 ms

Resolved: Bofrost.at performance has improved in 83b48a7.


added proxy support

You can now define http_proxy variable which will be used to automatically configure Alertmanager to use http proxy.

Thanks! :tada:


Update syntax.md

Hi @yangky11!
The changes need to be made in the original Lean files. The markdown files are generated automatically.
For more info, read https://github.com/arthurpaulino/lean4-metaprogramming-book#contributing

I fixed that list issue in this PR: https://github.com/arthurpaulino/lean4-metaprogramming-book/pull/87/files#diff-6abc89e33f9cc7c0f7eb28848918984968986f78fdbef61061e760dd1ec043e9.


Add Camera or Photo Library action

This adds a new $media action called $media.cameraLibrary that presents an UIAlertView to choose between using the camera and the photo library in one view.

w00t

Thanks for the PR @hoffmabc ! One question, couldn't we achieve the same effect purely by composing existing actions? For example:
{
  "type": "$util.picker",
  "options": {
    "items": [
      {
        "text": "Camera",
        "action": {
          "type": "$media.camera"
        }
      },
      {
        "text": "Choose from Library",
        "action": {
          "type": "$media.picker"
        }
      }
    ]
  }
}

I didn't think about that, but I suppose it could be. Is there still a reason to include this option? If not, it would probably be useful to call out this capability if it's not already in the documentation as it's very useful.

The component that Brian is proposing is a more standard way of selecting to either take a picture or choose an image. The current $util.picker looks a bit odd for this use-case.

The component that Brian is proposing is a more standard way of selecting to either take a picture or choose an image. The current $util.picker looks a bit odd for this use-case.

Maybe we can solve this problem by switching the $util.picker itself to use UIAlertController?
It's always good to use the native Cocoa UI elements instead of a 3rd party library anyway (the current $util.picker uses a 3rd party library, which is why it looks different).
Also I don't think it's a good idea to create a separate action for something that can be recreated by composing other existing actions.
Here's what the new $util.picker would look like as UIAlertController:

I guess the main difference would be that by default all texts are center aligned (Apple doesn't expose a public API to align them left). What do you think?

Try this branch https://github.com/Jasonette/JASONETTE-iOS/tree/util_alert_update for the UIAlertController based $util.picker. I tested it and it's working great for me (and I feel cleaner about getting rid of another 3rd party library dependency) so plan on merging this into develop soon.
This new one will have additional parameters: title and description to add to the header of the UIAlertController. Here's what it looks like:

Let me know if you think I'm missing something. Otherwise I'll merge soon

ACK (pretentious Bitcoin lingo for +1)
Is there a max number of items that can fit in there, or is it scrollable? Looks like it is scrollable, which is great.
@hoffmabc

Yup it's scrollable. gonna merge this in soon.

Yes! Thanks!

Merged the $util.picker update at https://github.com/Jasonette/JASONETTE-iOS/pull/225
You can try it with the following code: https://jasonbase.com/things/wLYD
{
  "$jason": {
    "head": {
      "actions": {
        "$load": {
          "trigger": "pick"
        },
        "$foreground": {
          "trigger": "pick"
        },
        "pick": {
          "type": "$util.picker",
          "options": {
            "title": "Choose Image",
            "items": [
              {
                "text": "Camera",
                "action": {
                  "type": "$media.camera"
                }
              },
              {
                "text": "Choose from Library",
                "action": {
                  "type": "$media.picker"
                }
              }
            ]
          }
        }
      }
    }
  }
}


Revert "Switch from freezegun to time-machine."

Reverts jazzband/django-model-utils#510

#593 fixes it so a revert isn't needed.


github-status: allow PRs from Fujitsu

Allow to build pull requests from the Fujitsu team in crowbar/crowbar-openstack and crowbar/crowbar-monitoring.

thanks, will rework this PR then

this PR contains PR #1693 until it is merged

Not needed currently and would need a rebase anyhow.
So closing now.


update/package dependencies to support laravel 9

updates the package illuminate/support to also allow Version 9
adds the composer.json to allow PHP 8.1

Thanks so much for the PR! I've used #9 instead as I wanted to tweak the .gitignore file too.


Enable Configuration of Storage Defaults

What should the feature do:

Enable Configuration of Defaults for ResizeThresholdAnnotation, ResizeInodesThresholdAnnotation, ResizeIncreaseAnnotation and StorageLimitAnnotation
Allow PVC autoresizing without annotations on PVC. I believe when PVC does not have StorageLimitAnnotation, PvcStorageLimit function returns 0, after which isTargetPVC function returns false, resulting in PVC being skipped in pvcAutoresizer.reconcile function

What is use case behind this feature:
We are using this awesome project to enable resizing of Spark PVCs dynamically. Currently, feature of adding annotations to PVCs via Spark Confs is only available in Spark v4 (Spark Conf Reference)
Additionally, currently, we do not have the requirement to configure individual PVCs, thus would like to configure the defaults at pvc-autoresizer level.
Draft Pull Request #287

similar to #82, #86

We've encountered similar issues before and felt that implementing them would add unnecessary complexity, so we decided against it. However, it might be time to revisit this decision.
By the way, are you familiar with the downstream project https://github.com/kubesphere/pvc-autoresizer? It allows you to set a default storage limit for each StorageClass, which might be a solution for your situation.


Add agent info to host metadata + fix install script

What does this PR do?
Fixes PR #5333 by adding missing sudo.
Adds a new sub-structure to the host metadata to send metadata about agent installation, coming from a metadata file created by the agent installation method.
Updates the install-script to create install_info (+kitchen tests)
Motivation
Allow users to track their installation methods
Allows Datadog to make stats on installation methods

It's been tested with and without sudo :D.


XIONE-13188: Fix deviceid query for Realtek platform

Reason for change: Use MAC as deviceid for Realtek soc (Same with generic linux implementation) Test Procedure: After device booting into UI, run the following console commands:
curl -s -d '{"method":"DeviceIdentification.deviceidentification","id":"42","jsonrpc":"2.0"}' http://<IP_ADDRESS>:9998/jsonrpc
Priority: P1
Risks: low
Cherry-pick to personal github for testing: https://github.com/maruku/rdkservices/commit/8ee348425e8e7d548489fb862d4ce181ec6e503e
Change for testing changes: https://gerrit.teamccp.com/857241
Jenkins custom image for testing: https://jenkins.ccp.xcal.tv/jenkins/job/SKYXIONE-SKY-Yocto-Pipeline/496/
Cherry-Pick: 746925ec4e90e0bdc346cd6bbfe9ad5981192a20
Change-Id: I799b094431f4662f693ec692d968c123d0ee279d

https://gerrit.teamccp.com/#/c/857241/
Verification build success:
Primary builds completed:
https://rdkportal.ccp.xcal.tv/builds/job/ELEMENT-TEONE-Yocto-Pipeline/1117 - default - SUCCESS
https://rdkportal.ccp.xcal.tv/builds/job/SERCOMMXIONE-XOE-Kirkstone-Yocto-Pipeline/307 - default - SUCCESS
https://rdkportal.ccp.xcal.tv/builds/job/ES1-RTK-Yocto-Pipeline/630 - default - SUCCESS
APACHE-2K-Yocto-Pipeline/581" target="_blank" rel="nofollow">https://rdkportal.ccp.xcal.tv/builds/job/APACHE-2K-Yocto-Pipeline/581 - default - SUCCESS
https://rdkportal.ccp.xcal.tv/builds/job/Pioneer-UHD-Yocto-Pipeline/561 - default - SUCCESS
https://rdkportal.ccp.xcal.tv/builds/job/RaspberryPi-CTH-Yocto-Pipeline/261 - default - SUCCESS
APACHE-4K-Yocto-Pipeline/586" target="_blank" rel="nofollow">https://rdkportal.ccp.xcal.tv/builds/job/APACHE-4K-Yocto-Pipeline/586 - default - SUCCESS
https://rdkportal.ccp.xcal.tv/builds/job/SKYXIONE-SKY-Yocto-Pipeline/499 - default - SUCCESS
https://rdkportal.ccp.xcal.tv/builds/job/HISENSE-V2-Yocto-Pipeline/516 - default - SUCCESS
https://rdkportal.ccp.xcal.tv/builds/job/SERXHB1-Yocto-Pipeline/220 - default - SUCCESS
https://rdkportal.ccp.xcal.tv/builds/job/SKY-LLAMA-PANEL-Kirkstone-Yocto-Pipeline/645 - default - SUCCESS
https://rdkportal.ccp.xcal.tv/builds/job/SKYXIONE-US-FLEX2-Yocto-Pipeline/417 - default - SUCCESS
https://rdkportal.ccp.xcal.tv/builds/job/SKY-LLAMA-PANEL-G2-Kirkstone-Yocto-Pipeline/374 - default - SUCCESS
https://rdkportal.ccp.xcal.tv/builds/job/SERXHC3-Yocto-Pipeline/235 - default - SUCCESS
V2-2K-Yocto-Pipeline/569" target="_blank" rel="nofollow">https://rdkportal.ccp.xcal.tv/builds/job/PIONEER-V2-2K-Yocto-Pipeline/569 - default - SUCCESS
https://rdkportal.ccp.xcal.tv/builds/job/HISENSE-A6GP-Yocto-Pipeline/510 - default - SUCCESS
HISENSE-50A6GX-Yocto-Pipeline/494" target="_blank" rel="nofollow">https://rdkportal.ccp.xcal.tv/builds/job/HISENSE-50A6GX-Yocto-Pipeline/494 - default - SUCCESS