Memory leaks lead to OOM on VM

Do you want to request a feature or report a bug?
Bug
What did you do?
Thanks for the awesome app, folks! Here's a description of the issue we're grappling with in our prod env.
Two m5.large ubuntu 20 VMs have traefik 2.3.2 up and running. Both instances experience intermittent OOMs. Traefik is the only prod service running on the boxes.
What did you expect to see?
Traefik consuming a reasonable amount of memory and not being OOM killed
What did you see instead?

How often?
# dmesg | grep "Out of memory"
[28224.365945] Out of memory: Killed process 1280 (traefik) total-vm:12695484kB, anon-rss:7520220kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:15796kB oom_score_adj:0
[28375.784145] Out of memory: Killed process 5460 (traefik) total-vm:15401052kB, anon-rss:7528288kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:15060kB oom_score_adj:0
[28399.713629] Out of memory: Killed process 5488 (traefik) total-vm:22978332kB, anon-rss:7528016kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:15156kB oom_score_adj:0
[94915.294507] Out of memory: Killed process 5511 (traefik) total-vm:15942780kB, anon-rss:7517724kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:15852kB oom_score_adj:0
[94931.163620] Out of memory: Killed process 18783 (traefik) total-vm:16483484kB, anon-rss:7525320kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:15072kB oom_score_adj:0
[95110.762503] Out of memory: Killed process 18816 (traefik) total-vm:37592188kB, anon-rss:7520028kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:15280kB oom_score_adj:0
[96053.725702] Out of memory: Killed process 18860 (traefik) total-vm:19189820kB, anon-rss:7514768kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:15828kB oom_score_adj:0
[96163.568843] Out of memory: Killed process 19117 (traefik) total-vm:9988892kB, anon-rss:7525112kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:14980kB oom_score_adj:0
[96176.746939] Out of memory: Killed process 19223 (traefik) total-vm:13236188kB, anon-rss:7524372kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:15028kB oom_score_adj:0
[96185.127062] Out of memory: Killed process 19255 (traefik) total-vm:13236444kB, anon-rss:7525556kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:15032kB oom_score_adj:0
[96196.946749] Out of memory: Killed process 19272 (traefik) total-vm:20272252kB, anon-rss:7525456kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:15112kB oom_score_adj:0
[97019.171968] Out of memory: Killed process 19299 (traefik) total-vm:13777660kB, anon-rss:7521416kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:15044kB oom_score_adj:0
[97036.865327] Out of memory: Killed process 19454 (traefik) total-vm:12153756kB, anon-rss:7526084kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:15020kB oom_score_adj:0
[98081.669969] Out of memory: Killed process 19484 (traefik) total-vm:20813724kB, anon-rss:7521324kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:20640kB oom_score_adj:0
[98098.685799] Out of memory: Killed process 19686 (traefik) total-vm:13236444kB, anon-rss:7526004kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:15036kB oom_score_adj:0
[98116.974612] Out of memory: Killed process 19716 (traefik) total-vm:19189820kB, anon-rss:7525448kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:15096kB oom_score_adj:0
[98304.334338] Out of memory: Killed process 19742 (traefik) total-vm:63570876kB, anon-rss:7522032kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:15616kB oom_score_adj:0
[99171.381963] Out of memory: Killed process 19804 (traefik) total-vm:15942780kB, anon-rss:7523484kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:15836kB oom_score_adj:0
[99188.626082] Out of memory: Killed process 19983 (traefik) total-vm:15401308kB, anon-rss:7524364kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:15060kB oom_score_adj:0
[107617.802530] Out of memory: Killed process 20011 (traefik) total-vm:16483996kB, anon-rss:7520760kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:15832kB oom_score_adj:0
[107665.695211] Out of memory: Killed process 21601 (traefik) total-vm:14318876kB, anon-rss:7524800kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:15044kB oom_score_adj:0
[107746.275737] Out of memory: Killed process 21632 (traefik) total-vm:19189820kB, anon-rss:7524808kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:15092kB oom_score_adj:0
[107851.344719] Out of memory: Killed process 21691 (traefik) total-vm:14318876kB, anon-rss:7524104kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:15048kB oom_score_adj:0
[107871.133680] Out of memory: Killed process 21734 (traefik) total-vm:19731036kB, anon-rss:7523616kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:15092kB oom_score_adj:0
[107886.951917] Out of memory: Killed process 21758 (traefik) total-vm:18107388kB, anon-rss:7521492kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:15076kB oom_score_adj:0
[108566.098364] Out of memory: Killed process 21777 (traefik) total-vm:13236444kB, anon-rss:7525464kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:15032kB oom_score_adj:0
[108768.086118] Out of memory: Killed process 21911 (traefik) total-vm:15401308kB, anon-rss:7524392kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:15060kB oom_score_adj:0
[108781.174664] Out of memory: Killed process 21986 (traefik) total-vm:17024956kB, anon-rss:7524792kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:15068kB oom_score_adj:0
[108909.047272] Out of memory: Killed process 22011 (traefik) total-vm:22437116kB, anon-rss:7522064kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:15108kB oom_score_adj:0

dmesg output:
[245970.625639] gmain invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[245970.625642] CPU: 1 PID: 459 Comm: gmain Not tainted 5.4.0-1029-aws #30-Ubuntu
[245970.625644] Hardware name: Amazon EC2 m5.large/, BIOS 1.0 10/16/2017
[245970.625644] Call Trace:
[245970.625651]  dump_stack+0x6d/0x9a
[245970.625655]  dump_header+0x4f/0x1eb
[245970.625657]  oom_kill_process.cold+0xb/0x10
[245970.625659]  out_of_memory.part.0+0x1df/0x3d0
[245970.625660]  out_of_memory+0x6d/0xd0
[245970.625663]  __alloc_pages_slowpath+0xd5e/0xe50
[245970.625666]  ? __switch_to_asm+0x40/0x70
[245970.625668]  __alloc_pages_nodemask+0x2d0/0x320
[245970.625672]  alloc_pages_current+0x87/0xe0
[245970.625675]  __page_cache_alloc+0x72/0x90
[245970.625677]  pagecache_get_page+0xbf/0x300
[245970.625678]  filemap_fault+0x6b2/0xa50
[245970.625681]  ? unlock_page_memcg+0x12/0x20
[245970.625683]  ? page_add_file_rmap+0xff/0x1a0
[245970.625686]  ? xas_load+0xd/0x80
[245970.625687]  ? xas_find+0x17f/0x1c0
[245970.625689]  ? filemap_map_pages+0x24c/0x380
[245970.625692]  ext4_filemap_fault+0x32/0x46
[245970.625695]  __do_fault+0x3c/0x130
[245970.625697]  do_fault+0x24b/0x640
[245970.625699]  __handle_mm_fault+0x4c5/0x7a0
[245970.625701]  handle_mm_fault+0xca/0x200
[245970.625704]  do_user_addr_fault+0x1f9/0x450
[245970.625706]  __do_page_fault+0x58/0x90
[245970.625708]  do_page_fault+0x2c/0xe0
[245970.625710]  do_async_page_fault+0x39/0x70
[245970.625712]  async_page_fault+0x34/0x40
[245970.625714] RIP: 0033:0x7f026fcdf18e
[245970.625718] Code: Bad RIP value.
[245970.625719] RSP: 002b:00007f026eec0bd0 EFLAGS: 00010206
[245970.625721] RAX:<PHONE_NUMBER>f40d64 RBX: 00007f0268000c10 RCX:<PHONE_NUMBER>000000
[245970.625722] RDX:<PHONE_NUMBER>000000 RSI:<PHONE_NUMBER>000001 RDI: 000056270cd5ce90
[245970.625723] RBP: 000056270cd96740 R08:<PHONE_NUMBER>000000 R09:<PHONE_NUMBER>000001
[245970.625723] R10:<PHONE_NUMBER>000005 R11:<PHONE_NUMBER>000000 R12: 000056270cd96860
[245970.625724] R13: 00007f026fcdf150 R14: 00007f026fb35280 R15: 000056270cd96740
[245970.625726] Mem-Info:
[245970.625730] active_anon:1925548 inactive_anon:36 isolated_anon:0
                 active_file:20 inactive_file:0 isolated_file:0
                 unevictable:4615 dirty:0 writeback:0 unstable:0
                 slab_reclaimable:6189 slab_unreclaimable:11631
                 mapped:2098 shmem:190 pagetables:4324 bounce:0
                 free:25561 free_pcp:0 free_cma:0
[245970.625733] Node 0 active_anon:7702192kB inactive_anon:144kB active_file:80kB inactive_file:0kB unevictable:18460kB isolated(anon):0kB isolated(file):0kB mapped:8392kB dirty:0kB writeback:0kB shmem:760kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 6889472kB writeback_tmp:0kB unstable:0kB all_unreclaimable? yes
[245970.625734] Node 0 DMA free:15908kB min:132kB low:164kB high:196kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[245970.625737] lowmem_reserve[]: 0 2991 7726 7726 7726
[245970.625739] Node 0 DMA32 free:45048kB min:26116kB low:32644kB high:39172kB active_anon:3009360kB inactive_anon:0kB active_file:8kB inactive_file:508kB unevictable:0kB writepending:0kB present:3129320kB managed:3063784kB mlocked:0kB kernel_stack:0kB pagetables:5660kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[245970.625742] lowmem_reserve[]: 0 0 4734 4734 4734
[245970.625744] Node 0 Normal free:41288kB min:41328kB low:51660kB high:61992kB active_anon:4692832kB inactive_anon:144kB active_file:224kB inactive_file:80kB unevictable:18460kB writepending:0kB present:5023744kB managed:4856828kB mlocked:18460kB kernel_stack:2768kB pagetables:11636kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[245970.625747] lowmem_reserve[]: 0 0 0 0 0
[245970.625749] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15908kB
[245970.625756] Node 0 DMA32: 30*4kB (UM) 15*8kB (UME) 15*16kB (UME) 14*32kB (UME) 10*64kB (UME) 10*128kB (UME) 5*256kB (UE) 5*512kB (UME) 2*1024kB (U) 0*2048kB 9*4096kB (UM) = 45600kB
[245970.625764] Node 0 Normal: 1166*4kB (UME) 549*8kB (UE) 455*16kB (UME) 183*32kB (UME) 121*64kB (UE) 42*128kB (UE) 15*256kB (UME) 5*512kB (ME) 0*1024kB 0*2048kB 0*4096kB = 41712kB
[245970.625773] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[245970.625774] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[245970.625775] 2297 total pagecache pages
[245970.625775] 0 pages in swap cache
[245970.625776] Swap cache stats: add 0, delete 0, find 0/0
[245970.625777] Free swap  = 0kB
[245970.625777] Total swap = 0kB
[245970.625778] 2042264 pages RAM
[245970.625778] 0 pages HighMem/MovableOnly
[245970.625779] 58134 pages reserved
[245970.625779] 0 pages cma reserved
[245970.625780] 0 pages hwpoisoned
[245970.625781] Tasks state (memory values in pages):
[245970.625781] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[245970.625786] [    162]     0   162    22535      857   172032        0          -250 systemd-journal
[245970.625788] [    192]     0   192     4718      864    61440        0         -1000 systemd-udevd
[245970.625790] [    291]     0   291    70033     4484    86016        0         -1000 multipathd
[245970.625792] [    339]   102   339    22599      840    77824        0             0 systemd-timesyn
[245970.625794] [    405]   100   405     6719      544    73728        0             0 systemd-network
[245970.625796] [    407]   101   407     6080     1653    90112        0             0 systemd-resolve
[245970.625798] [    450]     0   450    60374     1039   106496        0             0 accounts-daemon
[245970.625800] [    451]     0   451      635      184    45056        0             0 acpid
[245970.625802] [    456]     0   456     2134      553    57344        0             0 cron
[245970.625804] [    457]   103   457     1901      627    49152        0          -900 dbus-daemon
[245970.625805] [    464]     0   464   175966     1473   217088        0             0 filebeat
[245970.625807] [    467]     0   467    20455      358    57344        0             0 irqbalance
[245970.625809] [    471]     0   471     7316     2699    90112        0             0 networkd-dispat
[245970.625811] [    476]   104   476    56076      520    86016        0             0 rsyslogd
[245970.625812] [    477]   997   477   279362     4775   364544        0             0 sensu-client
[245970.625814] [    486]     0   486   235845     3678   253952        0          -900 snapd
[245970.625816] [    487]     0   487     4237      648    73728        0             0 systemd-logind
[245970.625818] [    499]     0   499      948      548    45056        0             0 atd
[245970.625819] [    610]     0   610     1838      441    53248        0             0 agetty
[245970.625821] [    621]     0   621     1457      370    49152        0             0 agetty
[245970.625823] [    623]     0   623    27019     2696   102400        0             0 unattended-upgr
[245970.625824] [    652]     0   652    59337      920   102400        0             0 polkitd
[245970.625826] [    772]     0   772     3042      745    61440        0         -1000 sshd
[245970.625828] [   3910]     0  3910   218340     3003   229376        0             0 amazon-ssm-agen
[245970.625830] [  36059]     0 36059     3042      768    61440        0             0 sshd
[245970.625832] [  36060]   109 36060     3071      640    61440        0             0 sshd
[245970.625834] [  36073]     0 36073  5203303  1902614 15654912        0             0 traefik
[245970.625835] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/traefik.service,task=traefik,pid=36073,uid=0
[245970.625862] Out of memory: Killed process 36073 (traefik) total-vm:20813212kB, anon-rss:7610456kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:15288kB oom_score_adj:0
[245970.679722] oom_reaper: reaped process 36073 (traefik), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

ALB target group serving traffic to both instances:

VMs CPU metrics:

Output of traefik version: (What version of Traefik are you using?)
# traefik version
Version:      2.3.2
Codename:     picodon
Go version:   go1.15.3
Built:        2020-10-19T18:36:22Z
OS/Arch:      linux/amd64

What is your environment & configuration (arguments, toml, provider, platform, ...)?
/usr/local/bin/traefik --configFile=/etc/traefik.toml
[global]
  checkNewVersion = true
  sendAnonymousUsage = true

[entryPoints]
  [entryPoints.http]
    address = ":80" # does all the job
  [entryPoints.https]
    address = ":443" # not in use. Let's encrypt WIP
  [entryPoints.redirect_https]
    address = ":8308" # returns 308 Permanent redirect to https

[log]
  level = "INFO"
  filePath = "/var/log/traefik.log"
  format = "json"

[accessLog]
  filePath = "/var/log/access.log"
  format = "json"

  [accessLog.fields]
    defaultMode = "keep"

    [accessLog.fields.names]
      "ClientUsername" = "drop"
      "ClientAddr" = "drop"
      "ClientPort" = "drop"
      "DownstreamContentSize" = "drop"
      "ForceQuery" = "drop"
      "Overhead" = "drop"
      "RequestAddr" = "drop"
      "RequestContentSize" = "drop"
      "RequestCount" = "drop"
      "RequestPort" = "drop"
      "RetryAttempts" = "drop"
      "RawPath" = "drop"
      "StartLocal" = "drop"
      "ServiceURL" = "drop"

      "RouterName" = "keep"
      "ServiceName" = "keep"

    [accessLog.fields.headers]
      defaultMode = "drop"

      [accessLog.fields.headers.names]
        # "User-Agent" = "keep"
        # "Content-Type" = "keep"
        "X-Real-IP" = "keep"
        "X-Forwarded-For" = "keep"
        "X-Request-ID" = "keep"

[api]
  insecure = true
  dashboard = true
  debug = true

[providers]
  [providers.file]
    filename = "/etc/traefik-routes.toml"
    watch = true

routes
[http]
  [http.services]

  [http.services.web]
    [http.services.web.loadBalancer]
      [[http.services.web.loadBalancer.servers]]
        url = "http://FOO/"

    [http.services.api]
      [http.services.api.loadBalancer]
        [[http.services.api.loadBalancer.servers]]
          url = "http://FOO/api"

    [http.services.stag]
      [http.services.stag.loadBalancer]
        [[http.services.stag.loadBalancer.servers]]
          url = "https://BAR/api"

    [http.services.mirrored-api]
      [http.services.mirrored-api.mirroring]
        service = "api"
        maxBodySize = 500000000 # 500MB as we have in nginx
        [[http.services.mirrored-api.mirroring.mirrors]]
          name = "stag"
          percent = 10

  [http.routers]

    [http.routers.api]
      entryPoints = ["https", "http"]
      rule = "Path(`/api/`) || PathPrefix(`/api`)"
      service = "mirrored-api"

    [http.routers.web]
      entryPoints = ["https", "http"]
      rule = "Path(`/`) || PathPrefix(`/`)"
      service = "web"

    [http.routers.redirect]
      entryPoints = ["redirect_https"]
      service = "web"
      rule = "PathPrefix(`/`)"
      middlewares = ["https"]

    [http.middlewares]
      [http.middlewares.https.redirectScheme]
        permanent = true
        scheme = "https"

We're going to play with the config and try to find more breadcrumbs during the day.
Could be related to 7508 or even v1.7 7191

An update on the issue:
We adjusted config on both boxes. BoxA has logging settings removed .BoxB has logging and monitoring removed.
BoxA crashed once yesterday during the day.
BoxB didn't crash yet and has cpu utilization dropper from 15-20% to less than 1%.
Keeping an eye on both

Just an update on the state of our setup.
We disabled mirroring and customized logging on Nov 10 and didn't have any crashes since then.
Going to reenable mirroring after the holidays season to see how it's going with only mirroring enabled (and no access logging). Will report on how it's going.

At a first glance the issue seems to be that Traefik pass the reference of its internal reusableRequest object to the spawned routines taking care of mirroring the request, which then has its body copied in full to memory as a byte slice for each mirror. That alone should not be an issue if the object got GCed after, but it isn't for some reason as we can see in the heaps it survives and is still referenced by its upper caller(s) long after the requests are gone.
Including a stripped down dynamic config that can easily reproduce this with a whoami service and vegeta:
http:
  routers:
    whoami-mirrored:
      entryPoints:
      - web
      service: mirrored-whoami
      rule: Host(`whoami.docker.localhost`)

    whoami-1:
      entrypoints:
      - web
      service: whoami-1
      rule: Host:whoami1.docker.localhost

    whoami-2:
      entrypoints:
      - web
      service: whoami-1
      rule: Host(`whoami2.docker.localhost`)

    dashboard:
      entryPoints:
      - web
      rule: Host(`traefik.docker.localhost`)
      service: api@internal

  services:
    whoami-1:
      loadBalancer:
        servers:
        - url: http://<IP_ADDRESS>:8888

    whoami-2:
      loadBalancer:
        servers:
        - url: http://<IP_ADDRESS>:8889

    mirrored-whoami:
      mirroring:
        service: whoami-1
        #maxBodySize: 1024
        mirrors:
        - name: whoami-2
          percent: 100

Generate a big body message for the request:
base64 /dev/urandom | head -c 10000000 > /tmp/full_body.txt

Send multiple requests to Traefik:
echo "GET http://localhost" | vegeta attack -header "Host:whoami.docker.localhost" -body /tmp/full_body.txt -duration 10s | vegeta report

Also including the heap files generated before and after requests.
heaps.zip

At a first glance the issue seems to be that Traefik pass the reference of its internal reusableRequest object to the spawned routines taking care of mirroring the request, which then has its body copied in full to memory as a byte slice for each mirror. That alone should not be an issue if the object got GCed after, but it isn't for some reason as we can see in the heaps it survives and is still referenced by its upper caller(s) long after the requests are gone.
Including a stripped down dynamic config that can easily reproduce this with a whoami service and vegeta:
http:
  routers:
    whoami-mirrored:
      entryPoints:
      - web
      service: mirrored-whoami
      rule: Host(`whoami.docker.localhost`)

    whoami-1:
      entrypoints:
      - web
      service: whoami-1
      rule: Host:whoami1.docker.localhost

    whoami-2:
      entrypoints:
      - web
      service: whoami-1
      rule: Host(`whoami2.docker.localhost`)

    dashboard:
      entryPoints:
      - web
      rule: Host(`traefik.docker.localhost`)
      service: api@internal

  services:
    whoami-1:
      loadBalancer:
        servers:
        - url: http://<IP_ADDRESS>:8888

    whoami-2:
      loadBalancer:
        servers:
        - url: http://<IP_ADDRESS>:8889

    mirrored-whoami:
      mirroring:
        service: whoami-1
        #maxBodySize: 1024
        mirrors:
        - name: whoami-2
          percent: 100

Generate a big body message for the request:
base64 /dev/urandom | head -c 10000000 > /tmp/full_body.txt

Send multiple requests to Traefik:
echo "GET http://localhost" | vegeta attack -header "Host:whoami.docker.localhost" -body /tmp/full_body.txt -duration 10s | vegeta report

Also including the heap files generated before and after requests.
heaps.zip

@alex-roman
I doubt your issue is the same as the one we reproduced (and we think is #7964 ) , but if you want you could always try the work around that is already included in the v2.5 experimental image, to see if that helps.
( https://hub.docker.com/layers/traefik/traefik/experimental-v2.5/images/sha256-73242275482292255a04ebfb1f4d62034d8119983721e72e7175150b1bda7387?context=explore , docker pull traefik/traefik:experimental-v2.5 )

Hiya! We just merged [PR 8706] (https://github.com/traefik/traefik/pull/8706), which we think fixed this in v2. Would you be willing to try it out and give us feedback?


Add framework clarification question tests

https://trello.com/b/mNNzL8hA
Made a few changes in doing these:

Tried to further standardize where we output informational messages, especially outputting "Notify ids" of found emails and in many places printing the actual "Notify ref" that we're going looking for. This is helpful in debugging.
Record complete "found" emails as @email rather than just the body as @email_text. The following steps checking for this can perfectly well pull out email.body themselves and this also allows us to dig into various properties of the message through MAYBE_VAR.
Add basic ability to negate checks for emails.
Add ability to require that opportunistically-grabbed framework also has clarification questions either open or closed. Cycling one of our instances through these settings and re-running tests can allow us to test both paths, or just whichever one we're preparing to bring to production soon.

Juuuust checking the whole suite still passes before I merge...


feat(e2e): add a11y checks

Regarding the modal a11y issues

It seems like aria-modal="true" is only valid for role="dialog". I had read in the bootstrap github that role="document" is sometimes preferable., but it seems like it's not compatible with aria-modal. Let's just revert to using role="dialog" as the default. This can be changed here in modal.js:

Let me know if you'd prefer that I open a PR instead,

For the contrast error, #666 on top of #fff does not seem like a big contrast problem in my opinion, but it looks like it is required for AAA compliance:

https://dequeuniversity.com/rules/axe/2.6/color-contrast
I'm inclined to ignore that error but maybe we should check with design and let them know that this is being reported as an issue?


How to run designs with multiple lefs

I have a design with multiple lefs.  How to use XPlace using this command:  python main.py --custom_path lef:data/raw/toy_input.lef,def:data/raw/toy_input.def,design_name:toy,benchmark:test --load_from_raw True --detail_placement True

How many lefs do you need？ If only need tech_lef and cell_lef, you just need to replace lef:... by ...,tech_lef:...,cell_lef:...,.

I also have three macro lefs.

I also have three macro LEFs.

Hi Zhiang, maybe you can try pasting the LEFs in a single file?

I have supported multiple lefs. Please refer to examples.json and run Xplace with python main.py --custom_json examples/examples.json.
Thank you!


Updating the deploy-apps script

Problem

Currently, the deploy-apps.sh script has no way of selectively choosing which apps to deploy. This means that users will have to wait a while for all the apps to build. Further, running all of these apps is resource-intensive.

Changes

Edited the deploy-apps.sh script to allow the user to choose which apps to deploy
updated the readme to include an example of how to use the script

Issues

Fixes #26

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Hi there! Thanks for creating such a cool project. This is my first time contributing to something open source so please forgive any blatant mistakes I made 😄

Thanks so much for your feedback! Never heard of strict mode before -- it caught some strange behavior in findDeploymentIndexFromInteger.
Btw I'm not sure if I am meant to mark the conversations as resolved or if I leave that you.
Also, should I pull in the linter changes from your repo into my fork and run it? Or will Github warn me automatically?
Thanks again!

Thanks so much for your feedback! Never heard of strict mode before -- it caught some strange behavior in findDeploymentIndexFromInteger.

Btw I'm not sure if I am meant to mark the conversations as resolved or if I leave that you.

Yes, I think you can resolve them

Also, should I pull in the linter changes from your repo into my fork and run it? Or will Github warn me automatically?

If you want to get a head start, you can run shellcheck deploy-apps.sh locally: https://github.com/koalaman/shellcheck#installing
It's mostly this one that may give you trouble.

@QuinceyJames Could you please rebase? Now we have some CI checks, I'm good to merge when the CI is green https://github.com/parca-dev/parca-demo/pull/31/files


PR Labeler

PR Labeler listens to updates to pull requests and based on the files associated with each one it applies zero or more labels. It uses the CODEOWNERS file to drive this automation.

I have #753 up that will allow us to create the office bot configuration task.

Yep, I'm going to abandon this branch. Thanks for the quick solution Alex!


Use port 443 for https in Server.checkRequest

Currently, even if the parsed url.protocol is https:,
Server.checkRequest was matching against port 80. This defaults that
check to use 443 if the protocol is https:

@rase- is this still an issue?

I went ahead and pushed this myself because it needed a rebase. Thanks @evanlucas

I apologize for asking this here, but any reason this didn't end up in any release yet?
Was bitten my this bug, came here to learn it's already fixed but not available in any release (i.e. 1.3.5 does not contain it).
Thank you


Remember selected input device / short tap for toggle

Hi,
not sure if you would really want to merge this PR, but these are 2 small patches I made for my personal use of this app.
It's the first time I have worked with Swift so things might be not very good.
First is to remember the selected device by using the device ID and storing it like the selected hotkey is stored.
Second is to use short taps for toggling the mute status instead of double tapping.

Hi I would like to thanks for the contribution. I have less time to look into this at the moment. Once I have more time, I'll look into that.


[Authentication] Flash a message if the user's email does not exists.

When the user entered a wrong email, a message will flash on the login page.
You may need to communicate with #77  about the HTML file.

i'll give this a try... on it :)


Buttons disappear in Spanish version

When switching Expunge.io into Spanish, the buttons disappear. This causes an issue when the user hovers over the text because the text becomes white and it can't be seen on the background. Here is an example:

fixed


issue with css import

Ok, closing, it is issue with my webpack config, all pure css files are badly parsed

Ok, closing, it is issue with my webpack config


Is this thing on?

List all the types of science.

Ticket implementation is in progress. See pull request: https://github.com/twin-digital/coco-toolbox/pull/2

Ticket implementation is in progress. See pull request: https://github.com/twin-digital/coco-toolbox/pull/2

Ticket implementation is in progress. See pull request: https://github.com/twin-digital/coco-toolbox/pull/4

Restarting work: Issue was reassigned to me, and I don't know how to reconcile changes with existing work..

Deleted branch issue_1

Ticket implementation is in progress. See pull request: https://github.com/twin-digital/coco-toolbox/pull/6

Restarting work: Issue was edited, and I don't know how to reconcile changes with existing work..

Closed pull request 6
Deleted branch issue_1

I have opened a pull request to resolve this issue. See it here: https://github.com/twin-digital/coco-toolbox/pull/7

Restarting work: Issue was edited, and I don't know how to reconcile changes with existing work..

Closed pull request 7
Deleted branch issue_1

I have opened a pull request to resolve this issue. See it here: https://github.com/twin-digital/coco-toolbox/pull/8

Restarting work: Issue was edited, and I don't know how to reconcile changes with existing work..

Closed pull request 8
Deleted branch issue_1

I have opened a pull request to resolve this issue. See it here: https://github.com/twin-digital/coco-toolbox/pull/9

Issue Status

in-progress
Branch:
PR #:

Recent Activity
None


Fix unique generation of ActorNames

On very large datasets the current implementation of the actor naming uuid tends to be not unique anymore. Therefore we need to find an alternative for the unique name generation.
Possible approaches:

remove substring(0,8) (might clutter logs...)
find another method to generate unique identifiers (maybe hash values?

Affected line:

The system participants are built with
    def simonaActorOf(props: Props, actorId: String): ActorRef =
      refFactory.actorOf(props, actorName(props, actorId))

where the actorId is used. Find out what we want here.


story build

Fix webpack config for scroll story.

The bug seems to be related to scroll magic. Webpack outputs a bundle with no errors. The issue is that all of the scene elements render except their opacity is almost at 0. The fadein class applies and the transition from opacity: 0 to opacity: 100% happens and yet the scene text is still nearly transparent.


Nightly features CI

This adds support for testing features that require a nightly Rust build.
Seems like a reasonable way to do this, for now. I don't imagine that we will have a lot of Parsec features in the future that will require something like this, but this should do the trick for the peer credentials authenticator stuff for now. 😄

We'll have to wait until (if?) the Unix peer credentials stuff is in the nightly compiler for this to pass CI.

Closing.
This PR is not needed any more. We have cherry-picked the peer credentials code from the Rust standard library into Parsec so that we can use the functionality 'early' -- the relevant commit is in this PR: https://github.com/parallaxsecond/parsec/pull/214
We have also removed the unix-peer-credentials-authenticator feature flag.


restore Mac OS X support

SEP doesn't build on OS X because make doesn't accept the -D flag on OS X.

From Slack:
The SEP C library is tested on Mac. But, the make install command is not tested, and there appears to be something wrong on mac: I guess install doesn't have a -D option there? On linux, it means
-D                  create all leading components of DEST except the last,
                    or all components of --target-directory,
                    then copy SOURCE to DEST

Is there an equivalent option on mac?

I don't think so:
$ make --help
Usage: make [options] [target] ...
Options:
  -b, -m                      Ignored for compatibility.
  -B, --always-make           Unconditionally make all targets.
  -C DIRECTORY, --directory=DIRECTORY
                              Change to DIRECTORY before doing anything.
  -d                          Print lots of debugging information.
  --debug[=FLAGS]             Print various types of debugging information.
  -e, --environment-overrides
                              Environment variables override makefiles.
  -f FILE, --file=FILE, --makefile=FILE
                              Read FILE as a makefile.
  -h, --help                  Print this message and exit.
  -i, --ignore-errors         Ignore errors from commands.
  -I DIRECTORY, --include-dir=DIRECTORY
                              Search DIRECTORY for included makefiles.
  -j [N], --jobs[=N]          Allow N jobs at once; infinite jobs with no arg.
  -k, --keep-going            Keep going when some targets can't be made.
  -l [N], --load-average[=N], --max-load[=N]
                              Don't start multiple jobs unless load is below N.
  -L, --check-symlink-times   Use the latest mtime between symlinks and target.
  -n, --just-print, --dry-run, --recon
                              Don't actually run any commands; just print them.
  -o FILE, --old-file=FILE, --assume-old=FILE
                              Consider FILE to be very old and don't remake it.
  -p, --print-data-base       Print make's internal database.
  -q, --question              Run no commands; exit status says if up to date.
  -r, --no-builtin-rules      Disable the built-in implicit rules.
  -R, --no-builtin-variables  Disable the built-in variable settings.
  -s, --silent, --quiet       Don't echo commands.
  -S, --no-keep-going, --stop
                              Turns off -k.
  -t, --touch                 Touch targets instead of remaking them.
  -v, --version               Print the version number of make and exit.
  -w, --print-directory       Print the current directory.
  --no-print-directory        Turn off -w, even if it was turned on implicitly.
  -W FILE, --what-if=FILE, --new-file=FILE, --assume-new=FILE
                              Consider FILE to be infinitely new.
  --warn-undefined-variables  Warn when an undefined variable is referenced.
  -N OPTION, --NeXT-option=OPTION
                              Turn on value of NeXT OPTION.

This program built for i386-apple-darwin11.3.0
Report bugs to<EMAIL_ADDRESS>

Does mkpath in Julia do the same thing as make -D?

I'll just change the sep makefile to use mkdirs -p followed by install (with no -D option).
(Doing it in Julia in the build script in Celeste is an option as well, but I should fix the sep makefile anyway).

Oh, also I just noticed that you answered the question for make -D, but I was asking about options to the install command. What does install --help say on OS X?

     The options are as follows:

     -B suffix
             Use suffix as the backup suffix if -b is given.

     -b      Back up any existing files before overwriting them by renaming them to file.old.  See -B for spec-
             ifying a different backup suffix.

     -C      Copy the file.  If the target file already exists and the files are the same, then don't change
             the modification time of the target.

     -c      Copy the file.  This is actually the default.  The -c option is only included for backwards com-
             patibility.

     -d      Create directories.  Missing parent directories are created as required.

     -f      Specify the target's file flags; see chflags(1) for a list of possible flags and their meanings.

     -g      Specify a group.  A numeric GID is allowed.

     -M      Disable all use of mmap(2).

     -m      Specify an alternate mode.  The default mode is set to rwxr-xr-x (0755).  The specified mode may
             be either an octal or symbolic value; see chmod(1) for a description of possible mode values.

     -o      Specify an owner.  A numeric UID is allowed.

     -p      Preserve the modification time.  Copy the file, as if the -C (compare and copy) option is speci-
             fied, except if the target file doesn't already exist or is different, then preserve the modifica-
             tion time of the file.

     -S      Safe copy.  Normally, install unlinks an existing target before installing the new file.  With the
             -S flag a temporary file is used and then renamed to be the target.  The reason this is safer is
             that if the copy or rename fails, the existing target is left untouched.

     -s      install exec's the command strip(1) to strip binaries so that install can be portable over a large
             number of systems and binary types.

     -v      Causes install to show when -C actually installs something.

 test


ucvm_setup.py user message hidden in log

the ucvm_setup.py script prints a user message to screen during the long build process. This is difficult for the user to see this message. It should be moved to a location where the user will be sure to see it.
This information must be copied to user in another way

Libraries have been installed in:
   /home/scec-00/maechlin/ucvm17.1/model/cencal/lib
If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the -LLIBDIR' flag during linking and do at least one of the following:    - add LIBDIR to the LD_LIBRARY_PATH' environment variable
     during execution
   - add LIBDIR to the LD_RUN_PATH' environment variable      during linking    - use the -Wl,-rpath -Wl,LIBDIR' linker flag
   - have your system administrator add LIBDIR to `/etc/ld.so.conf'
See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.

I added this text and a explanation that it can be ignored by most users to the installation instructions on the wiki


Bump axios

Version 1.2.0 is broekn. See: https://github.com/api3dao/airnode/pull/1553

The latest axios supports data-URIs, hence this test. Thanks for the comment :)


[v4] Feat/Discussion: clear cache on total reset?

I've had cache bugs more than a few times.  Just noticed a new one today:
https://github.com/fuse-box/fuse-box/issues/1750
I think it would make sense for caching to be cleared between fuse runs, unless specified.
Something like:
{ cache: true } === { cache: { enabled: true; clearOnFuseStart: true; } }

This would make it so the traditional method of "turn it off, then back on again" would solve any corrupt cache issues.  It would still give users the option to not clear cache on fuse start, if that's what they prefer.

@SephReed I've been thinking of getting rid of the FS cache concept completely. Since we're going to have such a fast compiler, we don't really need it. On top of that, I was planning to introduce threading in the compiler that would further speed things up and compile a monstrous project in less than 5s, which is tolerable.

If the caching is difficult to support, I think that might be very logical.  But if it's all the same, I think every bit helps.  Someone might be running this on a Raspberry pi.
Maybe caching could be false by default?  My worry is about how confusing it is when suddenly nothing works and reseting doesn't help.

I also think cache should be cleared between runs and only be used and reused when in dev mode.
Basically delete the cache when you start the dev server.

The new caching interface
export type ICacheStrategy = 'fs' | 'memory';
export interface ICacheProps {
  enabled?: boolean;
  root?: string;
  strategy?: ICacheStrategy;
}

Now we can choose memory, however, the module has been re-written from scratch and fully tested. So the fs caching should be very reliable.
It will be available for testing very soon ;-)

Closed due to inactivity.


feat: #843 stepping progress bar component

step primevue component implemented

https://app.zenhub.com/workspaces/fsa-fingerprint-61f04f08304d3e001a4f2578/issues/gh/bcgov/nr-forests-access-management/843
note: some styles on the primevue step component is in common repo

Just wondering in the case that we put the user id and then select the role, but then remove the user id,  shall we remove the checkmark at the 2nd step in the progress bar?

Just wondering in the case that we put the user id and then select the role, but then remove the user id, the user role field will be cleaned up as well automatically, shall we remove the checkmark at the 2nd step in the progress bar?

I think that @aliramoss might be the best person to answer that.

Hi Jota, please ignore this comment, and see my latest comment, thanks!

I think that this is right, in the scenario where the user assigned an user id, then a role, then he realize that the user id is wrong, then he removes and add again, maybe the role was right. Then we might leave the field filled, and since the field is already filled, it's right to have the gray checkmark.

I have a design question to @OlgaLiber2 and @aliramoss, it's just my personal feeling when first access to our grant access page.

I feel normally we have the progress bar to walk user through the steps which are not on the same page. But here all of our form fields are on the same page, I was a little bit confused at the beginning and was trying to find a "next" button to go to the next step. My confusion was solved when I started typing and I saw the progress indicator, so just a little bit concern if other people will have the same confusion as me.

I'm also wondering if we really need a progress bar here, cause our form is quite easy and the information is quite straightforward, not sure if the progress bar will increase the complexity for the user?  Especially in the error case, maybe we could think of a better way to handle it in the progress bar.
In this case as shown in the screenshot, the user failed to add some forest client numbers but also already added a valid one, he's able to submit the application, the "submit application" button is enabled. But the progress bar shows the step 3 failed, which makes me feel the form is not finished, and based on the progress bar indicator, he shouldn't be able to submit? It might be confused.
Maybe we don't need to display any error message in the progress bar, and as long as there is a valid forest client number, we can mark it as complete?

My last question is that, we'll disable the dropdown for role selection when the user id field is empty? It confused my a little bit, cause I was trying to select a role but it's disabled, is that possible the case that the user wants to see the role options before enter a user id? And if we disable the role selection, should the forest client number field also be disabled or got cleaned up?

Hi Jota, so in the case we enter a userId and select the fom_submitter role, but later on we changed mind and cleanup the userID field. The organization information section got cleaned up, but the progress bar still shows 3 steps. Let's cleanup the 3red as well in the case we make change to the userId field. Thanks!!

Discussed within the team, we feel our grant access form is simple enough for the user to understand. We don't need to add the progress bar at this point, as it might cause more complexity to the user. Close this pr.


Off by one genomic position in Variants track in gui

Running Mango master branch using the in repo example data:
./example-files/run-example.sh

I select a region to view in Mango GUI:
chr17: 7,501,828-7,501,840

The data source for the variants track in this view is:
/mango/example-files/ALL.chr17.7500000-7515000.phase3_shapeit2_mvncall_integrated_v5a.20130502.genotypes.vcf

and in that file the first columns of VCF for the example  variant here are
17      7501832 rs368364378     T       C       100     PASS

The API service GET endpoint returns:
curl 'http://localhost:8080/variants/ALL_chr17_7500000-7515000_phase3_shapeit2_mvncall_integrated_v5a_20130502_genotypes_vcf/chr17?end=7501900&start=7501759&binning=1'

[
  "{\"variant\":{\"contig\":\"chr17\",\"position\":7501831,\"end\":7501832,\"ref\":\"T\",\"alt\":\"C\"},\"sampleIds\":[]}",
  "{\"variant\":{\"contig\":\"chr17\",\"position\":7501886,\"end\":7501887,\"ref\":\"A\",\"alt\":\"G\"},\"sampleIds\":[]}"
]

The dbSNP record for this SNP is at:
https://www.ncbi.nlm.nih.gov/projects/SNP/snp_ref.cgi?rs=368364378
and reports this flanking sequence around the SNP:
ATGGCT[C/T]CTCAGTA

The SNP can be viewed at NCBI sviewer on GRCh37 here:
https://goo.gl/oo4NlG

So, NCBI view agree with the VCF and report the variant as occurring in the second of the two consecutive "T" in "GGCTT"
The Mango view above puts the red bar for the variant under the first of these two "T", and appears to be shifted one base position less than it should.  The text pop up in the GUI reports zero-based coordinates:
{"contig":"chr17","position":7501831,"end":7501832,"ref":"T","alt":"C"}

which are likely not what user would expect, as they would expect VCF style 1-based coordinates.
These issues are related to 0 vs 1-based positioning systems, but most likely we want the view in Mango to be one-based, and the reference sequence Mango shows appears to be one-based, resulting in the the off-by-one above positioning of the red bar.
I'm not certain what the position vs end is intended to be in  defining a SNP in the pileup API service, but I assume end is non inclusive base after the variant, and position is the inclusive start position, and both appear to be shifted one base down, assuming the api json response here in one-based coordinates

I'll have a PR with a fix later today.

Given that GA4GH is zero based as @heuermh mentioned:
https://github.com/ga4gh/ga4gh-schemas/blob/master/src/main/proto/ga4gh/variants.proto#L168
perhaps this issue is best fixed in the mango client - and the server continues to serve 0-based

@akmorrow13  - can you comment on:

Do you  agree  that this off-by-one issue exists in current Mango?

Do you think we should fix in on the server side or the client?

Although my original PR seemed to fix the genome browser view in Mango,
https://github.com/bigdatagenomics/mango/pull/291
I see now that it causes existing tests to fail, and also would require updates to make queries 1-based as well.  Given goal of using GA4GH API between client and the server, and GA4GH  is zero based anyhow, I prefer changing the client, and it looks like modifying our fork of pileup.js, to accept a zero-based GA4GH input as a variant data source.
I'm happy to work on doing that, just want to check first if you agree with this approach or if you suggest something different.

I agree this off by one issue exists. I experienced this same issue while taking some snapshots for reads and vcf a could days ago.

This is a Mango issue. If you load the vcf directly into pileup, the positioning is correct. See here:

The positioning is probably too hard coded into Mango right now, so it would be good to remove these anywhere. If GA4GH is indeed zero-based compared to vcf then it should be changed in the client.

The positioning is probably too hard coded into Mango right now, so it would be good to remove these anywhere.

Could you re-state what you mean by the sentence above- I don't quite follow.
And when you say change in client - you are saying change in our fork of pileup ?

This will change will go into https://github.com/akmorrow13/pileup.js

closed by https://github.com/bigdatagenomics/mango/pull/299


Please give me some advice

Hi Erich! I admire your professionalism and would like to better understand your code. I know three js well as a library, but have no idea how a shader program works. The rendering topic is very interesting to me, since I myself am a cgi artist. Could you please suggest me a roadmap? What do I need to know to write my own shader program? What topics of mathematics, geometry or physics do I need to work with? Many thanks!

Hi @vinkovsky !
Great question! I'll try to be as helpful as I can, given the limits of this text message board.
About 7 years ago, I was in the same 'boat' as you.  I knew a fair amount about three.js, and having done 3d game programming with opengl and C on windows 98 back in the late 1990's/early 2000's , I knew just enough math to be dangerous (ha ha). More precisely, i knew through trial and error what math algos or math routines I needed at various points in my simple games to get them working and looking somewhat correct. I spent several years getting to know vectors and vector operations like the dot product, and matrices, which I did not comprehend, but knew what I needed to use and in what situation (which I just copied and pasted from other math libraries out there at the time). Now I'm thankful we have great libraries like three.js and Babylon.js and glMatrix.js, which have everything you could possibly need to do 3d graphics/games/even basic physics simulation, right at our fingertips after simply downloading!
Everything was going fine for me using the stock opengl/webgl libraries to do my simple 3d games and projects, until 1 day I when working on my 3D Asteroid Patrol game with three.js, I needed a billboard sprite explosion effect when your ship was destroyed in the game. I knew the math to correctly keep the sprite billboard always facing the camera, but how do you make fire or a ball of fire out of three.js primitives?   You need a shader effect for special effects like that, and thus began my epic journey into the world of shaders!
I first visited glslsandbox.com and Shadertoy.com  and just poked around until I found a decent explosion, sun with flares, or fire ball effect,  and then literally copied and pasted the glsl shader code into my project and attached it to a three.js plane as a three.js shader material,  and voila! All of a sudden I had a cool explosion fireball animation with alpha transparency. The next problem came when I needed a different color of explosion and a different rate of animation of the fireball expansion, depending on the situation during game play.  I went in and started changing some of the numbers in this mysterious new language, glsl, and before long, I stumbled on to where the color was set and where the animation speed was set. But so far as the effect itself, there were all these crazy math and perlin noise functions going on. It was basically a magic black box that I could safely tinker with, as long as I didn't get too deep, because then the shader would crash or the effect would go away and turn into something else, which I also did not understand, lol!
Fast forward a couple of years and I came across this Shadertoy, which converged on a realistic looking image:
I was intrigued - I later found out that this shader was inspired by the path tracer 'smallpt' by Kevin Beason. I researched his code for months and found out he got most of his routines from Peter Shirley in a book called Realistic Ray Tracing (now out of print,  but I have since bought a used copy). Peter Shirley is quite famous in the rendering world for this book, and more recently, his gteat Ray Tracing in One Weekend series of 3 digital books, and now he works for the cutting edge RTX division of NVIDIA.  I started my own project here using snippets of that Shadertoy demo, conversions from Kevin Beason's C++ code from smallpt, and my basic familiarity with three.js, and then filling in the gaps and improving it little by little for over 6 years now!  Ray Tracing can be a never ending rabbit hole,.lol!
However, if you don't want go down that path of ray tracing like I did just yet, and merely want to get acquainted with how shaders work and how to do the basics with their language, glsl, then I highly recommend watching this great video from the Art of Code channel on YouTube:
Martin is a naturally gifted instructor who explains shaders in a unique and sometimes humorous way,  all the while coding every single line on screen in real time so you can immediately see the results.
Once you watch this video, I would highly recommend going to Shadertoy.com or glslsandbox.com and just starting from the stock default template- try out some random numbers, don't be afraid to break the shader - just mess around with something different (someone else's shader that they already wrote) each time you visit the shader site, and pretty soon you'll realize what variables do what. Try stuff like changing the background color, changing the spacing of the effect, make everything smaller, everything bigger, speed up time, slow down time, stuff like that... until you feel you know what your small changes are doing to the screen output.
After that, there's a lifetime of shaders on both of these sites to keep you busy learning  (and sometimes in awe and wondering how they did that! - some of them are pure magic, I have no idea how they were implemented!).
One final thought about shaders in general that tripped me up for months back when I started this journey: programming shaders for the GPU is not like CPU programming in Javascript or three.js or C, or whatever.  Yes, i suppose that it is iterative programming like C (as opposed to object oriented programming in C++ or functional programming in Haskell),  but how the shader operates as an entire screen output unit is nothing like normal single threaded CPU programming.
My mental model of shader (pixel fragment shader to be precise) programming is the following: imagine if you had at your disposal a tiny dust-grain sized desktop computer dedicated to each and every pixel on the screen. That's thousands to millions of little bitty desktops, each assigned to its own pixel.  The advantages are: you only have to write 1 program, then behind the scenes, three.js passes around your master source code to each and every tiny computer on every pixel of your screen. And amazingly, all the tiny computers execute their own copy of your master shader code - at the same time, 60 times a second! A truly parallel way of thinking. A famous quote from the 1970s goes something like, "Ray Tracing is an embarrassingly parallel operation- a dream ray tracer would have tiny little processors for each and every pixel, to be executed in lockstep and in unison." Well, that's why many people, including myself, have gone to the GPU in modern times for tracing scenes faster and faster. NVIDIA'S RTX is almost there, with thousands of processors running in parallel on their GPUs- one day I'm hoping for the historical quote to become reality- millions, even billions of tiny processors all for the same screen output.
Now for the disadvantages of this architecture: Since you have all these tiny computers operating on their own pixels (so to speak, in reality it's 'warps' on the GPU which work on blocks of pixels at a time, but that's a digression),  you would think that you could have pixels remember what they just computed on the previous frame, or look over and see what their neighbor pixels are working on - but such is not the case. It's as if they run through your shader blazingly fast, and then all data is lost after it shows on screen. You have to use tricks like ping pong buffers for pixel 'memory' storage and built in screen space glsl derivative functions (dFdx,dFdy) in order for a.pixel to peek over at its neighbor pixel horizontally and vertically to see what's happening (I currently use this technique for edge detection in my de-noiser, to keep edges of objects and corners sharp as possible with minimal blurring and blending).
A final disadvantage, and the main one in my opinion, is that since in reality there's 'warps' operating on 2x2 blocks of pixels at the same time, too many conditional statements like 'if'statements and branching can potentially cripple your performance. In CPU land, you can have hundreds or thousands of if statements and not notice any detriment to performance, but 1 too many if statements on a GPU, and it could crash or not even compile!  GPUs like doing the same sort of computation or at least roughly the same amount of work inside the tiny 2x2 block of pixels. So even if you have a complex Mandelbrot fractal running, it will go at 60 fps all day even on mobile, because every processor is doing roughly the same famous fractal algorithm. If however you give the GPU a path tracer, which I have crazily(ha!), then one pixel's ray might bounce off of a glass surface, but its immediate neighbor inside the same 'warp' might pass on through to get the transparent effect, and then GPU diversion starts showing up. Over the years I have found a delicate balance between robustness of the path tracer (able to randomly sample any kind of scene and surfaces), vs. giving each tiny processor roughly the same amount of work to chew on so they're all done at the same time - no stragglers holding up the line for everybody else!
Sorry I got into the weeds there a little. But I am really passionate about graphics, shaders, and ray/path tracing - it is a seemingly endless journey!  Hope I could help, or at least point you in the right direction. Please feel free to clarify or ask questions related to your shader journey here on this thread. I'll be happy to assist if I am able!
-Erich

WOW! Thanks so much Erich, very detailed and consistent answer! I agree with you that the best way to learn something is to plunge headlong into an unknown topic. At one time, this is how I learned three js, practically without looking at the documentation, copied examples and changed some parameters. I really appreciate your work and will definitely study the books you mentioned.
I have a few questions about your project. I've seen several examples where you can change the material on an object, but can they be combined with each other? For example, use a diffuse texture with a normal map texture or with a metalness map.
P.S. Sorry if I made mistakes, I am not a native speaker

Hi Erich! Thank you for such a visual explanation! This is what I want to achieve. I already started to study the code from your example of an animated bvh model, of course it looks very complicated to me. I should probably start with something simpler, as you said to pay attention to the pool table. At least, the development vector is clear to me and I thank you again for your support

@vinkovsky
No problem!  Glad to be of assistance.  I just remembered you had asked what kind of math and CG skills you needed in order to make realistic graphics with shaders.  I have 2 older great online resources for learning the math and theory behind traditional ray tracing and path tracing (Monte Carlo style, like on this repo).
The first resource is Scratchapixel - in particular, if you scroll down the webpage you'll see the section called Volume 1: Foundations of 3D Rendering.  This would be an excellent place to start learning the theory behind rendering.  This Scratchapixel website is not complete (there are missing chapters everywhere, and I don't know if the author will ever complete them), but this Volume 1: Foundations of 3D Rendering is the only fully complete set, so that's why I recommend that first.  If you are curious about Monte Carlo numerical methods and why we have to use that for path tracing sampling, then you'll also find some fairly complete sections on that under the Mathematics and Physics for Computer Graphics section, which is not complete, but complete enough for our purposes!  What I like about this online learning website is that the author goes through all the details of the theory behind all CG problems, then proceeds to put everything into actual code that you could conceivably run yourself.  I say 'conceivably', because he uses C/C++, so you would need a development environment set up like Microsoft Visual Studio Community (free) to be able to compile and execute the C/C++ example code.  I'm actually ok he chose C/C++ because by the time you get to glsl, it looks a lot like C, which  has strong variable typing, upfront declarations, gets fully compiled before execution, and no garbage collection.  So you can actually use some of the math routines/algos as-is with minimal changes to get them to work in your future shaders.
The other resource is directly related to what we do here on this repo, which was inspired by Kevin Beason's brilliant old non-realtime path tracer in just 100 lines of C/C++ code  called 'smallpt' (stands for smallPathTracer).   Now when looking at his compact 100 lines which is a complete Monte Carlo path tracer, it would be hard to decipher what's going on, especially if you have not written ray tracers before, but a CG professor by the name of David Cline took Kevin's 100 C/C++ lines, unpacked them, and then made a complete Google slide presentation which takes each line, 1 by 1, and first analyzes the mathematical theory behind what Kevin is doing, then analyzes how he got that theory into actual code that runs efficiently on a computer.  Here's a link to the presentation and here's a link to Kevin's project page which has other cool older projects as well - Kevin Beason's website By the way, although Kevin magically got this complete path tracing renderer code down to just 100 lines, the algos and math he uses are directly from the older book I told you about a couple of posts ago - it is Peter Shirley's Realistic Ray Tracing: 2nd Edition, which is now out of print but you can still find used copies on vintage online bookstores.   There are many gold nuggets and gems inside this old classic book that I have directly used here on my project as well.
Best of luck on your rendering  journey!  Let me know if you have any other questions or concerns.
-Erich

Hi Erich! I keep looking into your rendering engine and here is my progress. I was able to pass variables to the shader to disable the environment while still keeping the reflections. The next goal is to be able to use multiple textures on an object like emissive texture, roughness, metalness, etc. Thanks a lot for your tips and detailed explanations!
https://user-images.githubusercontent.com/66293881/161245452-51deb4b8-e9f2-404c-aad2-aa8f13e1765a.mp4

Now I'm learning how your volumetric light example works and maybe I can create an image similar to my avatar that was created in redshift renderer :D

I think I created the lighting incorrectly, I got reflections from a quadlight on the model but lost the light from the sun

Here is my code
HDRI_Environment_Fragment.zip

Hello @vinkovsky !
Yes I could see the problem once I checked the (if hitType == COAT) section of your code.  You were accidentally overwriting the dirToLight variable when calling the different sampling functions.  In Monte Carlo-style path tracing, we are only allowed to choose one random outcome on each iteration.  So, you can either sample the Sun (HDRI environment map), OR the Quad light, OR the Sphere light - on each bounce pass.  Since we are only picking 1, but there are 3 total light options which would make the scene brighter, we must up-weight the light source that we DID end up randomly choosing.  So, in code:
weight *= 3.0; // 3.0 = number of Lights
Here is the corrected and commented glsl fragment shader file:
HDRI_Environment_Fragment.glsl
And here are a couple of screenshots of the more correct results (hopefully what you were looking for):

Hope this helps! :-)
-Erich

@vinkovsky
Oh and about the volumetric avatar rendering (pink/purple beams of light through fog), that would be really cool!  Just a word of warning though - my current Volumetric rendering glsl file is set up to find glass sphere caustics for my demo.  If you render something like the pic you posted earlier, you won't need any of my 'caustic hunting' code.   Also, Volumetric rendering is an advanced topic - it took me a while to wrap my head around how you light all the particles in the fog.  Plus, the 'equiAngular' sampling function is shader code I lifted from a fairly recent research paper by the team at Arnold renderer (I believe) - I'm not entirely sure how it works, LOL.  Best of luck though!  If you have any problems, just post here and I'll try to help the best that I can. :-)

Hello @erichlof! Thanks for your corrections, I guessed it was a random choice of light. When I looked at the geometry example, I saw this line lightChoice =spheres[int(rand() * N_LIGHTS)];
I was able to add volumetric lighting to my hdri experiment demo and had a few questions. Is there a way to use fog on top of the environment map? I think i need to somehow pass the color of the environment in the line accumCol += Fog_color * vHitEmission * geomTerm * trans/pdf;
Again, I'm not sure if I've correctly distributed the particles around the rectangular light source. I used part of the code from the sampleQuadLight function.
ParticlePos = cameraRayOrigin + xx * cameraRayDirection;
float randParticleChoice = rng();
vec3 randPointOnLight.x;
randPointOnLight.x = mix(quads[0].v0.x, quads[0].v2.x, clamp(rng(), 0.1, 0.9));
randPointOnLight.y = quads[0].v0.y;
randPointOnLight.z = mix(quads[0].v0.z, quads[0].v2.z, clamp(rng(), 0.1, 0.9));
if (randParticleChoice < 0.5) {
lightVec = randPointOnLight - particlePos;
} yet {
lightVec = sphere[0].position - particlePos;
}
d = length(lightVec);
I think using SceneIntersect twice slowed down rendering. Can it be accelerated?
And another off-topic question. What ide are you using to format the code?

My monkey code
HDRI_Environment_Fragment.glsl.zip

@vinkovsky
Those screenshots are starting to look really cool! That's a neat scene idea that haven't thought of before.
Yes, that line from the geometry showcase demo that picks a random light is basically doing the same math as the one I showed in my last post for your multiple light sources demo. The only reason it is able to be squashed into a 1-liner like that is because the geometry demo has 3 very similar type of lights, all spheres with just different colors. If the lights are very different though like in your scene, you have to do it the longer way like I showed you recently. But it all basically boils down to flipping a coin or rolling dice to decide which light source to sample for each ray bounce loop iteration.
Yes it should be possible to have an HDRI in the background, and also to sample the HDRI light source (like the Sun in the symmetrical garden outdoor HDRI) -  all while having fog on top of that.  I'll have to take a moment and look over your code and my old fog particles sampling code (haven't touched it in a while),  but hopefully I can come up with a solution that will give you multiple shafts of light, or shafts of shadows from each light source, assuming that's the effect you're wanting for this scene.
About my IDE, I really like Visual Studio Code now and exclusively use that. It has many handy extensions like code formatting, GLSL syntax highlighting, JS-linting, etc. Plus it's entirely free to download.
Will be back soon hopefully with some solutions to the fog and multiple different types of lights! :-)

@vinkovsky
Success!  :-D

The first image shows just a black background.  The colored fog (colored blue and magenta by the 2 lights) fades into black as it gets farther away - very cool in my opinion!
The second image correctly shows the HDRI in the background, partly obscured by the pink and bluish fog.  Although it could be useful in some instances I guess, I don't really think it has the same dramatic effect on the viewer as the first image with no HDRI (just fading to black).  The shafts of light vs shafts of colored light are not as pronounced with a bright HDRI like this.
I wrote some instructions in the shader for you, showing what to comment out in order to switch between HDRI background and plain black background (resulting in either of the 2 screenshots above).
With some minor modifications, I was able to sample both area lights through the fog.  The fog density, L1 and L2 colors for the lights make a big difference in this situation, so you might have to play around with these values quite a bit, in order to get the look that you're wanting (like the David statue with pink and purple lasers in fog that you shared earlier).
Also, I noticed a bug in my original volume rendering code - eHitType and vHitType were declared but not initialized.  This is why the background HDRI was not showing up for you.  Took me a while to figure out what was wrong, ha!  Updated your example and my old Volumetric Demo here on my repo as well - with the new fixed lines as well as comments warning future me (lol).
Here's a link to the new HDRI_Environment Fragment Shader
Hope you can get a little closer to your target image now!  Happy volume rendering!  ;-)

If I use a double sided rectangle I get this

@vinkovsky
Hi!  Sorry, I forgot to mention about the rectangle (Quad in this scene), that the first parameter in the struct is the normal to its own surface.  Right now it is set to vec3(0,-1,0), which points straight down.  Take a look at:
Line 715
This is work so long as you don't change the vertices.  If you do however, a new normal must be given (or computed).  Sorry that this is not automatic, I intended these simple quads to be aligned with the walls, floor, and mainly ceilings of box-shaped rooms.  Therefore, the normals are easy to imagine:
on ceiling: vec3(0,-1,0) / on floor: vec3(0,1,0)
on right wall: vec3(-1,0,0) / on left wall: vec3(1,0,0)
on front wall: vec3(0,0,-1) / on back wall: vec3(0,0,1)
I didn't intend for this simple quad to be arbitrarily rotated.  Now if you need to be able to spin it around in any random direction, I will need to replace the simple Quad with the more robust Rectangle struct:
https://github.com/erichlof/THREE.js-PathTracing-Renderer/blob/gh-pages/shaders/Kajiya_TheRenderingEquation_Fragment.glsl#L25
https://github.com/erichlof/THREE.js-PathTracing-Renderer/blob/gh-pages/shaders/Kajiya_TheRenderingEquation_Fragment.glsl#L38
https://github.com/erichlof/THREE.js-PathTracing-Renderer/blob/gh-pages/shaders/Kajiya_TheRenderingEquation_Fragment.glsl#L354
I will start trying to add this, as I feel it would be helpful in this situation.
Will return soon! :)

Thanks for the clarification, I just didn't fully understand the meaning of this line :D It would be cool if the rectangular light had a target point

@vinkovsky
Alright!

I removed the Quad geometry and light, and replaced it with Rectangle geometry and light, allowing you to point it in any direction by specifying a target direction (same as its own normal).  Also you can specify U and V measurements of the horizontal and vertical dimensions of the rectangle (when you define it in the SetupScene function.
Make sure you normalize the target direction (normal) of your rectangle, otherwise the intersection and sampling calcs might be incorrect.
As mentioned previously, the fogDensity, L1 and L2 values make a big difference in the dramatic effect.  My screenshot shows a slightly less aggressive, softer light setup, but if you want full-on shafts of dark shadows coming from the dragon, you might have to make the lights much brighter and the fog a little less dense (so that the shafts last longer all the way to the bottom of the scene).
Here's the new HDRI_Environment_Fragment.glsl  file.  Hopefully you can just copy and past this one into your project, replacing the older .glsl file.

Thank you so much, Erich! Looks very cool!

And another question. How to rotate it ?

Sure!  With some minor changes, the rectangle area light now is double-sided:
Here's the updated HDRI_Environment_Fragment.glsl file.

Ooh, that's starting to look cool!
Rotation of a rectangular area light is possible, but that will require more plumbing.  I'll see if I can throw something together.

Ok I'm back - I had to make several changes to the Rectangle code to get this working.
updated HDRI_Environment_Fragment.glsl file.
The new code allows you to rotate the rectangle light along the X axis, Y axis, and Z axis.  Since we are now rotating it in this manner, the older 'point-to-target' code doesn't really work anymore.  I briefly tried it, but the pointing direction messes up the later user-specified rotation calculations - matrix math, doh!  The new way is to specify the amounts, if any, you would like to rotate the rectangle in any or all of the axes.
I included a lot of comments and notes so hopefully you can just start using this.
One heads-up though: I had to make the rectangle's vec3 position a global variable, so that it may be accessed by the sampling and intersection functions as well as the rotation code parts.  If you now look at the bottom of the file in SetupScene(), where Rectangle is defined, it has a position of vec3(0,0,0), or the origin.  This is intended and correct, because its real actual position is now a global shader variable called rectanglePosition.  To move the light around in your scene, change this global variable rectanglePosition and not anything else.
Have fun!

@erichlof
Awesome!  Thank you very much, I will post my final render soon 😀

Hello Erich! How difficult is it to create the bloom effect in this scene?

Also my model consists of triangular polygons, how can I make it smoother?

@vinkovsky
Looking good!  About bloom, typically that is a post-process effect that doesn't actually physically occur, but it is an artifact of camera over-exposure.  My best advice (if you don't have access to Photoshop after-effects and such) is to increase the fogDensity to something thicker like 0.01 (it was at 0.00001 previously).  Then make the rectangle light thin and very tall so that the top and bottom edges are outside of the image frame.  The effect will be much like a light saber at night in the Star Wars films, or a bright neon light at night in a polluted futuristic city (I'm reminded of a 'Punk-Noir' Blade Runner type of atmosphere).  Here's the best that I can do, while keeping everything physically plausible and not cheating with Photoshop (ha):

I've created a new fragment shader file that has these changes.  Be careful not to just copy and paste everything, as I have isolated the pink laser light and am not sampling the blue light at all, just for demo purposes.  But maybe you can get some settings from the file and apply them and mess around with them.
It does take some tweaking! :)

About the triangulated look of your model, that is directly related to the vertex normals supplied by the model creator (and their model creation software).  If it is appearing faceted, the vertex normals might not be smooth enough and instead what are seeing is pure face normals (which you don't want because it will look like a low-poly retro style).  If your model doesn't have smooth enough vertex normals, I believe there is a built in three.js function helper to create these for you at model loading time (at the startup of the webpage).  I'll take a look and see if I can find it for you... Be back soon

Also wanted to ask. Is it possible to use perspective camera three.js and orbit control in your project? If so, what should be paid attention to in the adaptation process?

Sorry, i ask too many questions in this thread

@vinkovsky
Don't worry about asking questions!  I also learn from trying to help you with your issues/desires in your rendering.
Yes it is possible to have an orbit style camera, but at the moment the way I have it set up is more of a first person camera.  Yes it is already using perspective camera, but with custom rotation and flying controls. Three.js has 2 main types of cameras: Orthographic and Perspective.  I've never needed Orthographic, so I don't plan on adding that anytime soon.   But Perspective is the one you and I are using now with this path tracing project codebase, so no need to change anything there.
Let me see if I can hack something together to get you very close to a Three.js OrbitControls-type of camera.  I think it just extends the camera's position along the +Z axis, but not letting the WASD keys move the position - instead the act of rotating with an extended Z (imagine a boom camera pole) that is always connected to either the target (like your model), or to the scene origin ( Vector3(0,0,0) ), will move the camera position in an arc automatically.  I did this for my AntiGravity pool game when lining up a billiards shot, so I think it should work without too much trouble.
Sorry to hear about the model normals fix not working for you - is it possible to send me a sharing link here to your David model itself, so I can download it?  Where did you get it from originally, if I may ask?

Ok I implemented an OrbitControls camera for you.  Here's the new setup js file:
HDRI_Environment.js
Note: It loads the Stanford Bunny model, so you will have to change that line of code to your David model path on your computer.
You will also need to copy the InitCommon.js file and replace your old InitCommon.js file - as I have found a better way to offer this OrbitControls option and have updated the file on my repo here. (See, I'm learning and improving too, ha):
InitCommon.js
Using an OrbitControls camera now, the mouse wheel serves a different function - it 'dollies' the entire camera forward and backward along the camera's Z axis.  If you need a different FOV, you will need to simply type it in inside the InitSceneData() function located in the HDRI_Environment.js file.  The camera aperture and focus distance still works as normal, with the 4 arrow keys.

Hello Erich! I purchased David model for 3d max form 3ddd site. You can download converted gltf here.
New way to translate camera around the object is gorgeous! The other thing I would like to know how to disable cursor blocking while translation? Also threejs provides two ways to pan camera around the object, .screenSpacePanning must-have future for me.
Hi Erich! I bought David's model for 3d max on the 3ddd website. You can download the converted gltf here.
The new way of moving the camera around the object is awesome! I would also like to know how to disable the cursor lock when translating? Also threejs provides two ways to pan the camera around an object,  .screenSpacePanning is important to me.

Thanks for the David model!  I will place it in my test scene now so we can both be on sort of the same page.  I'll let you know if I find any solutions to the vertex normals issue.
Glad you like my brand of OrbitControls!  Yes, I'm pretty sure I can make the camera pan, but it might be easier to use the WASD keys.  I already have listeners for those set up.  I think three.js' Orbit Controls use the right mouse button drag to pan.  I could eventually implement that too, but it would require adding more listeners and a drag action-detector algo, which I don't have currently in the engine.
Speaking of drag, if you want to avoid the PointerLock (and show the mouse cursor), I can remove those lines of code for you, but then the same drag issue will arise when doing the normal camera orbiting.  How will the app know that you are wanting to orbit or pan, if the PointerLock is not engaged and you are just moving the mouse around?  The correct way would be to have the cursor always visible, and the camera does orbiting/panning only if you click and drag the mouse (left button/right button).  But since I use first person style cameras for all the demos, I just rely on key presses, since the mouse pointer is invisible and locked.
I'll see if I can throw something together that satisfies your needs, but at the same time not requiring too much refactoring and plumbing work on my end, ha.
Talk to you soon!

Sorry for the delay. It being Easter weekend and all, I haven't had as much time as usual to work on this. I successfully got the David model in my scene now and I also implemented panning with the orbit controls camera. Now I'm working on adding a couple of GUI sliders so that we can control the fog density as well as the 2 laser light brightnesses individually. This will help in faster iteration time when trying to match your old avatar renderering.
More updates coming soon!

@vinkovsky
Here's as close as I can get by just eyeballing it:

I believe I have the camera angles and the lasers very close to your old avatar pic, but the thing that I can't seem to match is the dramatic, crisp light and shadow on David's face.  On my renderer, due to the particles of fog, it seems to wash his face features out and over-expose the body of the statue toward the white end of the spectrum.  Your old rendering seems to keep the blue-purple darkness of the statue.  I've messed around with all the new settings, but I just can't seem to hit on the right combo.  More troubling still, is that there may not be a right combo at all - due to how my renderer works vs. how the old one that you used worked under the hood (so far as ray tracing is concerned).
But in any case, I have completely overhauled the js and the glsl files, so that now the GUI more finely controls the lights and the fog of the scene.  Also, WASD now pans the camera left and right, up and down, while the mouse wheel dollies the camera position back and forth, more like a physical camera on a movie set.
Here is the new JS file,
and the new Fragment Shader file.
As previously discussed, the only change I could not do is unlocking the cursor while manipulating the orbit camera, because that would have required a lot more internal plumbing overhaul.  But hopefully, the new Orbit Camera that I implemented will be intuitive enough to use will searching for that perfect frame/shot camera angle.
Hope this all helps!

Looks perfect for my new avatar :D

Render on my iphone 11
https://user-images.githubusercontent.com/66293881/164182165-a8cfd7da-e368-440e-8125-8b6abf81c433.MOV

Wow! Those renderings are looking amazing!  I'm so glad you were able to dial in the appropriate settings to bring back the contrast and shadow details to David's face.  It's so much better now, congrats!
And equally impressive is how well it runs on your iPhone 11!  That image and video you posted is why I started this three.js path tracing project in the first place - I had the dream of being able to experience real time progressive rendering on all devices, even my phone when I'm on the go!
Yes about the lights, (thanks for posting those images of Corona renderer), I agree that one-sided panel lights will work better for your scene.  At the moment, I 'm not sure how to narrow or widen the angle of total illumination from the rectangle (like in your first Corona pic), but that has inspired me to try and see if I can get something similar in the near future for my project.  It probably works similar to how a spot light controls its output angle - should be doable. Anyway, for now I will make the rectangle lights one-sided for you (don't worry, it doesn't require many changes at all).
Good to hear that you like the new orbit-style controls!  Yes, if you look at three.js' source code for their Orbit Controls, you should be able to see how they implement mouse dragging for panning (remember, you would have to disable pointer lock), and for your iPhone 11, you could also add a 2-finger swipe detection to control panning.  I may eventually add all of this to a 2nd camera (Orbit Camera) option for my path tracing renderer here, but I'm currently working on other parts of the codebase.
About the normals and triangulation issues you were having, but that are now gone (yay!), that is actually somewhat of a mystery.  When I loaded in your David model that you shared, I really couldn't see any triangulation issues on my system.  However, with each new version of the Gists (source files) that I'm sharing with you, I have been removing some of the unused code from my outdoor Dragon HDRI environment demo.  So maybe something left over from that was messing up the current fog and laser light environment lighting that we have now.  That's one of those happy bug-fixes that just magically take care of themselves, ha!
Will return soon with the 1-sided rectangle lights adjustment...

Whoo-hoo!

Here's the new Fragment shader file that has 1-sided rectangles.
By the way, I didn't know if you knew or not, but you can change the length and width of the laser rectangle lights in the shader.  Just  find their definitions inside the SetupScene function towards the bottom of the file.  You can see some of my changes in order to achieve this most recent render:
list of changes
I just didn't have time to add those rectangle U and V lengths as sliders to the GUI.  But maybe you can see now how I hook up certain sliders/parameters inside the JS setup file and matching Fragment shader GLSL file.  For instance, you can look at lightPower as a good example of how to add your own sliders.

Thanks! I will try to implement it.
Why don't you use MRT rendering in your PT?
https://threejs.org/examples/?q=webgl2_multiple_rendertargets#webgl2_multiple_rendertargets

Speaking of which, an orthographic camera would be a cool tool for this project. It would allow render images like this one

Hello!
About MRT, I do use render targets, but as more of an internal function.  In a nutshell, three.js (and all WebGL-based software) are rasterization APIs (as opposed to NVIDIA's RTX hardware-supported ray tracing API).  Forgive me if you already know this, but in the typical WebGL pipeline, 3D Triangles are supplied to the scene, then they are projected to the screen as 2D triangles, and then any pixels on the screen that happen to cover these flattened, projected 2D triangles are sent to the fragment shader for any color calculations that we want done on them.  Since I am forced to use this basic rasterization system of at least providing 1 triangle (or 2), I chose to provide 2 very large triangles back-to-back, that form a full screen quad (our vertex shader is almost non-existent!).  Since every pixel on the screen will cover 1 of these 2 huge triangles, every pixel gets sent to the fragment shader (or pixel shader), where all the PT magic happens!
Getting back to render targets, I use the path tracing fragment shader to perform ray tracing/path tracing on every pixel in parallel (well, almost, because of how GPUs are designed), and then save these results of the 1st frame of animation (samples=1) to a Render Target.  Then I have another 'copy' shader which pulls from this render target (as a screen-size texture, basically), and copies it into another Render Target.  This copied Render Target fed back through to the first path tracing shader, again as a large texture.  On the 2nd frame of animation (samples=2), the path tracing shader now has its own color history available for each pixel.  It then blends the history color with the fresh new color that it just got done calculating (through PT), and keeps doing this over and over.  Pretty soon, the image will magically converge!  So I do rely on multiple render targets, but maybe not the way you're asking about.
The ortho camera would be a cool addition indeed!  Thanks for the suggestion - I feel like I could add something like that without too much trouble.  It's all in the ray generation part of the  function inside the large PathTracingCommon.js file.  If I could make the camera rays go straight out (instead of fanning out like in the Perspective camera), it should do the trick.  I'll play around with it.  Thanks for the ortho pic - quite cool!

I successfully got the Orthographic camera working!

It still needs polishing - certain areas of the scene get unintentionally clipped, just because of the way the Ortho camera position and its view clip planes work.  The neat thing is that it only required 3 lines of code, ha.  That's the beauty of Ray Tracing vs. traditional Rasterization.  Something that would have required a whole new file and dozens to hundreds of lines (to implement Orthographic view matrices and such) in traditional Rasterization (WebGL, OpenGL, DirectX, etc), can be done with our Ray Tracing system in a couple of lines. ;-)
Haven't figured out how to expose this new feature yet - maybe as part of the setup options on each demo's JS file.  Default would be Perspective camera of course, but with a simple flag, it would turn into an Orthographic camera.  What's interesting about this whole Ortho thing is that it fools around with your eyes - ha.    The image above as well as all the other scenes I've tried it out on, both looks incredibly real and unreal(surreal?) at the same time!

Hi Erich! Thanks for the explanation of MRT, I simply looked through the InitCommon file and noticed several lines of code with RenderTarget. Now I understand it better.
Your ortho camera is awesome! Could you please share the code, maybe I will try to make a picture as I dropped above?

Ok I just implemented the Orthographic camera option for the whole repo! (Well, currently for the static scenes, not the dynamic moving scenes - coming soon!)
Here's the list of changes to path tracing  that is used by all static-scene demos.  I made changes to only 3 lines (marked with a red color on left side of page):  changes to shader main function
This is one of those beautiful Ry Tracing cases where you get awesome functionality for only a handful of lines of code - the other being the blurring effect of the camera's Depth of Field feature - like 5 lines of code, ha! 😁
A couple of notes on using this new feature:

You need to replace your InitCommon.js file with my new one I just posted (contains code for switching camera modes by pressing O for Ortho and P for Perspective)
You need to replace your PathTracingCommon.js file with my new one (has the small shader changes discussed above)

It will work across all demos that are static.  The reason it doesn't yet function on the dynamic moving scenery demos is because they do not use path tracing  shader function which I just updated with the new ortho code.  Rather, they use a custom  shader function that is tailor-made for each dynamic situation.  I will have to go and manually change all those, one by one.  But it won't be too much work - it's just a couple of lines for each demo.
Also, in the case of your David demo scene, we disabled useGenericInput so you could operate the OrbitControls using your custom key presses.  If you want to use Orthographic camera mode with a demo like yours that has non-generic custom input handler, then all you need to do is add the following line to your initSceneData() function inside your demo's accompanying JS setup file:
changeToOrthographicCamera = true;
It's as simple as that!  Doing this will begin your non-generic scene in Ortho mode.  If ever you want to have Perspective mode, simply remove this line of code - because the default for all demos is Perspective mode.
Have fun!

@vinkovsky
Quick Update: I just now changed the InitCommon.js file again.  Make sure to get the latest version.  I added the Ortho option to mobile platforms also, so now you can use the GUI checkbox on your phone or tablet to easily toggle Ortho/Perspective modes.
Enjoy!

Hello Erich! Amazing work! Exactly what i need. I'll try to create an isometric image soon. At the same time I will study how texturing works. Thank you!


[SPARK-46694][SQL][TESTS] Drop the assumptions of 'hive version < 2.0' in Hive version related tests

What changes were proposed in this pull request?

Code clean-up following SPARK-45328, this PR drops the assumptions and workarounds in Hive version-related tests， which work for hive version < 2.0
Why are the changes needed?

clean dead codes
Does this PR introduce any user-facing change?

no
How was this patch tested?

SPARK_TEST_HIVE_CLIENT_VERSIONS=3.1 ./build/sbt  -Phive -Phive-thriftserver "testOnly HiveClientSuite"
Was this patch authored or co-authored using generative AI tooling?

no

cc @HyukjinKwon @dongjoon-hyun thanks

this assumption is always true too, could u please update it as well?
https://github.com/apache/spark/blob/b7e976696f51fc4dbb13e222adb6bb705eb8324d/sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveVersionSuite.scala#L35-L36

this assumption is always true too, could u please update it as well?

I left it AS-IS on purpose

Thank you all, merged to master


"Align to column by ," doesn't align correctly

Example:
Before:

INSERT INTO address
(id              , address1                           , address2                      ) VALUES
(@club_address+1 ,'56 passage de la Coupole'          , '92400 La Défense 6 Courbevoie')         ,
(@club_address+2 ,'9 Esplanade du Général de Gaulle', '92800 Puteaux'                 )         ;

after: apply this function to all lines except first:

INSERT INTO address
(id                 , address1                           , address2                      ) VALUES
(@club_address+1    , '56 passage de la Coupole'         , '92400 La Défense 6 Courbevoie')            ,
(@club_address+2    , '9 Esplanade du Général de Gaulle' , '92800 Puteaux'                 )         ;

Please fix.

works fine for me:
INSERT INTO address
(id              , address1                           , address2 ) VALUES
(@club_address+1 , '56 passage de la Coupole'         , '92400 La Défense 6 Courbevoie') ,
(@club_address+2 , '9 Esplanade du Général de Gaulle' , '92800 Puteaux' ) ;

hm.. It doesn't work as expected when file has sql extension...

maybe something in tab/space code style config?  I aligned many sql files by it....

Maybe because of this https://youtrack.jetbrains.com/issue/DBE-4050

Can't reproduce anymore


d_a_tag_etc work

Did some cleanup based on the Feedback. I was surprised how many of my Java Habits/Styles came in, I'll have to make sure to keep an eye on that.
Only comments I have,
I'm still not able to get rangeCheck to match, not sure what I'm missing there at this point.
I'm not a fan of otherCheck's look due to the double casting of the Pointer to Medli, but I was unsure if I was allowed to modify the header to check the param type to daNpc_Md_c, so I went with the "It matches the diff" approach.
The inlining stuff still throws me off, I was using VSCode for the edits, but I'm going to try out a JetBrains' IDE on it sometime after Christmas, I've found their refactor tooling to be good at finding stuff like that in Java. Otherwise I'll just make a cheat sheet while I learn them all.

Did some cleanup based on the Feedback. I was surprised how many of my Java Habits/Styles came in, I'll have to make sure to keep an eye on that.
Only comments I have, I'm still not able to get rangeCheck to match, not sure what I'm missing there at this point.

It doesn't look like you pushed any new commits, so I can't check what changed there.

I'm not a fan of otherCheck's look due to the double casting of the Pointer to Medli, but I was unsure if I was allowed to modify the header to check the param type to daNpc_Md_c, so I went with the "It matches the diff" approach.

You shouldn't modify the function argument types, those are extracted directly from the symbol maps.
What do you mean "double casting"? You should have to cast fopAc_ac_c* to daNpc_Md_c* once, but I don't think a second cast is necessary unless I'm missing something.

The inlining stuff still throws me off, I was using VSCode for the edits, but I'm going to try out a JetBrains' IDE on it sometime after Christmas, I've found their refactor tooling to be good at finding stuff like that in Java. Otherwise I'll just make a cheat sheet while I learn them all.

I'm not sure what the IDE has to do with inlines, I use VSCode and get inlines in the autocomplete suggestions just fine if that's what you mean.
Are you looking at tag_etc's kiosk demo debug map? Because it should tell you exactly which inlines were actually used in this REL without needing to rely on the editor's guesses based off of names.
The only problem to watch out for is that each function only appears once in each map, the first time it was used (based on link order). So if the same inline gets used in two separate functions it will only show up in the tree underneath one of those functions, not both.

It doesn't look like you pushed any new commits, so I can't check what changed there.
Fixed. I get an occasional bug where I have to use git in the CLI or it doesn't push. I don't tend to do that, so sometimes it slips past me.

You shouldn't modify the function argument types
Noted

I'm not sure what the IDE has to do with inlines, I use VSCode and get inlines in the autocomplete suggestions just fine if that's what you mean
I've never really touched C++ with VSCode before, so it's a matter of VSCode is really good with Configuration, but can be hard to know all the configuration required. I have access to CLion, so I was going to try that out and see if the out of the box solution can catch those faster, since I don't like dealing with plugin management.

Are you looking at tag_etc's kiosk demo debug map?
I'm now positive that I haven't. I double checked the .map/.asm files the Randomizer can output just to do a sanity check that I wasn't missing something.
I'm assuming the Kiosk Map is a Linker Map from either the JP Version (Which I haven't extracted yet), or from somewhere else. I'd love a link to that resource though.

I'm now positive that I haven't. I double checked the .map/.asm files the Randomizer can output just to do a sanity check that I wasn't missing something.
I'm assuming the Kiosk Map is a Linker Map from either the JP Version (Which I haven't extracted yet), or from somewhere else. I'd love a link to that resource though.

The kiosk demo is a demo of WW from japanese stores before the game was officially released. The actual binaries are very similar to the japanese release version and not super useful for our purposes (there's no debug binaries like there are for TP). But they accidentally left debug symbol maps in the files, which tell us all of the inlines they used making decomp much easier.
If you check the pins in discord fig posted the demo maps.


Gulpfile eslint

Adjusted to AirBnB coding styles, and added to npm run lint command. Running eslint . takes much longer than npm run lint, I haven't found a good way to whitelist folders via eslintignore (rather than trawl through all of them and then decide which ones to use)

@sminnee Wanna reciprocate and merge my eslint fix as well? ;)


Fix broken link in Markdown

Coverage remained the same at 100.0% when pulling 73e589b8223826f41b85c07edeca31947efa5029 on dragon788:patch-2 into c9db3fb9bf492e34db1237b7c44e5542369ef1ed on mefellows:master.

Thanks!


GPII-3590: new users 2

This resurrects https://github.com/gpii-ops/gpii-infra/pull/274, which was merged and then reverted in https://github.com/gpii-ops/gpii-infra/pull/291, and fixes a very silly bug (that Google did not help me find due to a very silly error message). Thanks Stepan!

LGTM

LGTM