[bug] blank page cached on context canceled

I have a recurring error where the homepage of the website is blank.
When I check the key value in redis, it looks like this:
HTTP/0.0 200 OK
Date: Fri, 21 Apr 2023 18:44:19 GMT
X-Souin-Stored-Ttl: 24h0m0s
Content-Length: 0

What strikes to me is the key name, which is GET-https-www.website.com-/, which all the other keys have {-VARY-} appended to them. If I delete the key, and refresh the website, the new key that appears is GET-https-www.website.com-/{-VARY-}Accept-Encoding:gzip, deflate, br, and now the page isn't blank anymore.
What would cause that?

For reference, a working key contains this:
HTTP/0.0 200 OK
Cache-Control: public, max-age=86400, must-revalidate
Content-Encoding: gzip
Content-Type: text/html; charset=UTF-8
Date: Fri, 21 Apr 2023 18:58:53 GMT
Server: Caddy
Server: Google Frontend
Vary: Accept-Encoding
X-Cloud-Trace-Context: 63177d3330a11c9b0a44a650162499a1

Meaning it really comes from the backend server.

Unrelated: why the HTTP/0.0 ?

Hey @frederichoule that's weird. Do you have a reproducible repository/example please?
To explain how the vary works under the hood, it will list all keys starting with (in your case) GET-https-www.website.com-/($|\{VARY\}), either the key ends by the trailing or has the vary separator.
If the first case is the first key to be returned, the key doesn't have any vary headers to validate, so it will match the client expectations and returns the content to the user. It will loop over the other keys and try to validate the varied headers.
What I think at the moment is on the first time a user tried to load the page but the backend returned nothing with an HTTP 200 OK status code.
Btw I don't know why your server returns an HTTP/0.0 protocol version, I don't deal with it and don't try to override it.
Can you post at least your caddy configuration?

1- I'll try to reproduce the issue locally and let you know.
2- The output is clearly not from the backend, since there's no Server: Google Frontend or X-Cloud-Trace-Context.
3- Backend returns HTTP/1.1, Caddy returns HTTP/1.1, but stored key is HTTP/0.0 - This is a non-problem, I was just wondering.

Here is the Caddyfile:
{
	order cache before handle
	email<EMAIL_ADDRESS>
	admin off

	on_demand_tls {
		ask http://localhost:5555/__ask
		interval 2m
		burst 5
	}

	cache {
		cache_name Tarmac
		default_cache_control public, max-age=86400, must-revalidate
		allowed_http_verbs GET
		ttl 86400s
		redis {
			url <IP_ADDRESS>:6379
		}
		key {
			hide
		}
	}

}

(cache_regular) {
	cache {
		key {
			disable_body
			disable_query
		}
	}
}

(cache_qs) {
	cache {
		key {
			disable_body
		}
	}
}

(upstream) {

	request_header -Cache-Control

	tls {
		on_demand
	}

	header {
		-X-Cloud-Trace-Context
		Server Tarmac
	}

	@notwww {
		not host localhost
		not header_regexp www Host ^www\.[a-zA-Z0-9-]+\.[a-zA-Z]{2,}$
	}

	redir @notwww https://www.{host}{uri} permanent

	@get {
		method GET
	}

	@hits {
		path /__hits
		query id=*
		method GET
	}

	@redis {
		path /__redis
		method POST
	}

	@query_caliente query caliente=1

	@query_search {
		path /search
		query q=*
	}

	@query_after query after=*

	handle @query_caliente {
		import cache_qs
	}

	handle @query_search {
		import cache_qs
	}

	handle @query_after {
		import cache_qs
	}

	handle @hits {
		reverse_proxy http://<IP_ADDRESS>:5555
	}

	handle @redis {
		reverse_proxy http://<IP_ADDRESS>:5555
	}

	handle @get {
		import cache_regular
	}

	handle {
		abort
	}

	reverse_proxy backend.com {
		header_down -Cache-Control
		header_up -Cache-Control
		header_up Host {upstream_hostport}
		header_up X-Backend-Hostname {host}
	}

}

localhost {
	tls internal
	import upstream
}

https:// {
	import upstream
}

http:// {
	redir https://{host}{uri} permanent
}

Unable to reproduce the problem locally for now. I am closing this for now.

That could be something like cache poisoning (I wonder if @jenaye could help us about that). It could explain the
Server: Google Frontend
X-Cloud-Trace-Context: 63177d3330a11c9b0a44a650162499a1

It's a brand new website, not really public yet. I think it might have to do with robot scanning the website, generating incorrect cached page. I'll investigate it more thoroughly soon.

I was able to pinpoint when the blank page bug happens.
Whenever I have a blank page in cache, it comes from a request that contains this in the logs:
"error": "context canceled"

Any idea where it's coming from? I'll investigate after lunch, but if you have a hint for me, let me know!

Nice, that means I don't handle correctly the client disconnections. It should be quite easy to reproduce if that's this case. Let's reopen this issue then 🙂

Have you been able to reproduce the issue yet @darkweak ? I'll give it a try tomorrow if you didn't have time yet.

To keep you updated, I'll work on that tomorrow.

Amazing, thanks! Haven't had time to take a look yet, sorry about that.

It should work using @frederichoule
xcaddy build --with github.com/darkweak/souin/plugins/caddy@47ea558146d978d6be0ec6a42804f679200d6d70 --with github.com/darkweak/souin@47ea558146d978d6be0ec6a42804f679200d6d70

Amazing! Will try it ASAP.

use the commit 2cffe5ac9c989e963b3e28c327e6b6c63eab371d instead of the previous one. I pushed a fix on the uncached responses.

Testing right away.

2023/05/25 19:42:38.124	ERROR	http.log.error	context canceled	{...}, "duration": 0.628182}
panic: Header called after Handler finished

goroutine 56 [running]:
net/http.(*http2responseWriter).Header(0x40003584c0?)
	net/http/h2_bundle.go:6569 +0x80
github.com/darkweak/souin/pkg/middleware.(*CustomWriter).Header(...)
	github.com/darkweak/souin@v1.6.39-0.20230524064538-2cffe5ac9c98/pkg/middleware/writer.go:46
github.com/darkweak/souin/pkg/middleware.(*SouinBaseHandler).Upstream(0x4000990640, 0x40003584c0, 0x4000656c00, 0x40005948e0, 0x4000287c80?, {0x4000144660, 0x22})
	github.com/darkweak/souin@v1.6.39-0.20230524064538-2cffe5ac9c98/pkg/middleware/middleware.go:274 +0x404
github.com/darkweak/souin/pkg/middleware.(*SouinBaseHandler).ServeHTTP.func2()
	github.com/darkweak/souin@v1.6.39-0.20230524064538-2cffe5ac9c98/pkg/middleware/middleware.go:478 +0x40
created by github.com/darkweak/souin/pkg/middleware.(*SouinBaseHandler).ServeHTTP
	github.com/darkweak/souin@v1.6.39-0.20230524064538-2cffe5ac9c98/pkg/middleware/middleware.go:477 +0x1324

I hope that this commit fix your issue f9f8ab6da7e98e2e7898d10122766865e0b4884c

Seems OK locally. I just pushed to prod for a small segment of our network and will let you know if we still get blank pages. Thanks!

I still have blank pages. I'm unable to debug right now as my internet connection is pretty weak, but I'll give you more informations later.

Reopened again 😅

So I do have the same "context canceled" in the logs. I'm not sure if I can provide more informations than that.
{"level":"debug","ts":1685364382.2595735,"logger":"http.handlers.cache","msg":"Incomming request &{Method:GET URL:/education-et-comportement Proto:HTTP/2.0 ProtoMajor:2 ProtoMinor:0 Header:map[...] Body:0xc0008b0e10 GetBody:<nil> ContentLength:0 TransferEncoding:[] Close:false Host:www.website.com Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr:X.X.X.X:XXX RequestURI:/url TLS:0xc0000d4fd0 Cancel:<nil> Response:<nil> ctx:0xc0008b1020}"}

{"level":"debug","ts":1685364382.8981557,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"upstream.com:80","duration":0.637657811,"request":{...},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"www.website.com"}},"error":"context canceled"}

Please wait before checking anything. I just redeployed everything with cache disabled on the docker build, I think I had some cached steps.

Let me know if something weird happens.

So far everything works perfectly. Let's close this! Thanks @darkweak

Caddy crashed completely twice in the last 2 days. The last line in the log each time is related to context canceled. I don't know how to reproduce the crash locally yet. I'll find a way to make Caddy log the crash output somewhere.
{"level":"error","ts":1685643137.1032906,"logger":"http.log.error","msg":"context canceled","request":{},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"www.website.com"}},"duration":4.829884199}

{"level":"debug","ts":1685643137.1034744,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"upstream.com:80","duration":4.251552925,"request":{},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"www.website.com"}},"error":"context canceled"}

Do you have some debug logs please ? Because these are not really explicit. I will write the patch to add more debug/info logs during the request process in the next days.

This is all I got from Caddy. I'll read the doc to see what i can do to get more.

When you're running caddy you can set the debug global directive in your caddy file and Souin will run as debug mode and give you the debug logs.

It is already on debug, those logs in the last comments are what I get from the debug mode, yet I can't see what makes Caddy crash. Here I was able to see it since it happened locally: https://github.com/darkweak/souin/issues/337#issuecomment-1563420820

What are the previous logs before the crash ?

Only regular requests logs - but I had to reboot the server and didn't take a copy of the logs beforehand. I will do it at the next crash (so far looks like it's once per day... my TTL is 24 hours.. maybe related to stale?)

Crashed again, almost exactly 24h after (which is my TTL) - again, nothing in the log, but Caddy is not accepting requests anymore. Can't see what make it crash in the logs.
Will lower the TTL and try to debug that.

Got it.
fatal error: concurrent map writes

goroutine 234338 [running]:
github.com/caddyserver/caddy/v2.(*Replacer).Set(...)
	github.com/caddyserver/caddy/v2@v2.6.4/replacer.go:67
github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).ServeHTTP.func1()
	github.com/caddyserver/caddy/v2@v2.6.4/modules/caddyhttp/reverseproxy/reverseproxy.go:490 +0x70
github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).ServeHTTP(0xc000898380, {0x24fd6a0, 0xc00002c980}, 0xc000d47400, {0x24f4de0, 0x2254038})
	github.com/caddyserver/caddy/v2@v2.6.4/modules/caddyhttp/reverseproxy/reverseproxy.go:512 +0x45b
github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1({0x24fd6a0?, 0xc00002c980?}, 0x24f4de0?)
	github.com/caddyserver/caddy/v2@v2.6.4/modules/caddyhttp/routes.go:290 +0x42
github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP(0x24f4de0?, {0x24fd6a0?, 0xc00002c980?}, 0x40dea8?)
	github.com/caddyserver/caddy/v2@v2.6.4/modules/caddyhttp/caddyhttp.go:58 +0x2f
github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapRoute.func1.1({0x24fd6a0, 0xc00002c980}, 0xc000d47400)
	github.com/caddyserver/caddy/v2@v2.6.4/modules/caddyhttp/routes.go:259 +0x3a8
github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP(0x1d16de0?, {0x24fd6a0?, 0xc00002c980?}, 0x7?)
	github.com/caddyserver/caddy/v2@v2.6.4/modules/caddyhttp/caddyhttp.go:58 +0x2f
github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapRoute.func1.1({0x24fd6a0, 0xc00002c980}, 0xc000d47400)
	github.com/caddyserver/caddy/v2@v2.6.4/modules/caddyhttp/routes.go:238 +0x219
github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP(0x40be65?, {0x24fd6a0?, 0xc00002c980?}, 0xc000a8ef00?)
	github.com/caddyserver/caddy/v2@v2.6.4/modules/caddyhttp/caddyhttp.go:58 +0x2f
github.com/darkweak/souin/plugins/caddy.(*SouinCaddyMiddleware).ServeHTTP.func1({0x24fd6a0?, 0xc00002c980?}, 0x203dbc4?)
	github.com/darkweak/souin/plugins/caddy@v0.0.0-20230525203934-f9f8ab6da7e9/httpcache.go:83 +0x39
github.com/darkweak/souin/pkg/middleware.(*SouinBaseHandler).Upstream(0xc00013b400, 0xc00002c980, 0xc000d47b00, 0xc00021c820, 0x443145?, {0xc0010c6330, 0x2a})
	github.com/darkweak/souin@v1.6.39-0.20230525203934-f9f8ab6da7e9/pkg/middleware/middleware.go:258 +0x1a6
github.com/darkweak/souin/pkg/middleware.(*SouinBaseHandler).ServeHTTP.func3()
	github.com/darkweak/souin@v1.6.39-0.20230525203934-f9f8ab6da7e9/pkg/middleware/middleware.go:484 +0x3e
created by github.com/darkweak/souin/pkg/middleware.(*SouinBaseHandler).ServeHTTP
	github.com/darkweak/souin@v1.6.39-0.20230525203934-f9f8ab6da7e9/pkg/middleware/middleware.go:483 +0x19b3

Nice, now we know where, I have to check why and how to reproduce.

Hey sorry for the delay. I tried many time with in memory storages but that doesn't occur. I wonder if that could come from the reconnection in redis/etcd/olric storages

I haven't had time to dig deeper on that so far. We use Redis as storage, so maybe we can investigate on that side.

Hey, I'm not able to reproduce your issue, even if I try to cut the connection to redis and let Souin reconnect itself. I think it comes from the replacer caddy code.

Let's close this for now. I had to move to another project for now so I didn't have time to dive deeper on that. Thanks for your help @darkweak

Hey
I have the same issue, and I used the default storage. (Badger)
Souin sometimes caches blank pages.
Here my setup:
Caddy build with:
xcaddy build --with github.com/caddyserver/cache-handler

Caddyfile (cache part)
cache {
  mode bypass_request
  ttl 86400s
  key {
    hide
  }
  default_cache_control no-store
  badger {
    configuration {
      Dir /home/caddy/badger-cache
      ValueDir /home/caddy/badger-cache
    }
  }
}

I believe this is the relevant log:
Dec 20 03:52:09 ingress.example.com caddy[718]: {"level":"info","ts":1703044329.3248205,"msg":"using provided configuration","config_file":"/home/caddy/ingress/Caddyfile","config_adapter":""}
Dec 20 03:52:09 ingress.example.com caddy[718]: {"level":"info","ts":1703044329.3338363,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//<IP_ADDRESS>:2019"]}
Dec 20 03:52:09 ingress.example.com caddy[718]: {"level":"info","ts":1703044329.3360136,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
Dec 20 03:52:09 ingress.example.com caddy[718]: {"level":"info","ts":1703044329.336098,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Dec 20 03:52:09 ingress.example.com caddy[718]: {"level":"warn","ts":1703044329.3361273,"logger":"http.auto_https","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv1","http_port":80}
Dec 20 03:52:09 ingress.example.com caddy[718]: {"level":"info","ts":1703044329.33631,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000466f00"}
Dec 20 03:52:09 ingress.example.com caddy[718]: {"level":"info","ts":1703044329.3727183,"logger":"http.handlers.cache","msg":"Set nextTxnTs to 0"}
Dec 20 03:52:09 ingress.example.com caddy[718]: {"level":"info","ts":1703044329.3728356,"logger":"http.handlers.cache","msg":"Set backend timeout to 10s"}
Dec 20 03:52:09 ingress.example.com caddy[718]: {"level":"info","ts":1703044329.3728437,"logger":"http.handlers.cache","msg":"Set cache timeout to 10s"}
Dec 20 03:52:09 ingress.example.com caddy[718]: {"level":"info","ts":1703044329.3728487,"logger":"http.handlers.cache","msg":"Souin configuration is now loaded."}
Dec 20 03:52:09 ingress.example.com caddy[718]: {"level":"info","ts":1703044329.3753047,"logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]}
Dec 20 03:52:09 ingress.example.com caddy[718]: {"level":"info","ts":1703044329.375385,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
Dec 20 03:52:09 ingress.example.com caddy[718]: {"level":"info","ts":1703044329.3775697,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
Dec 20 03:52:09 ingress.example.com caddy[718]: {"level":"info","ts":1703044329.3775969,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["souin.example.com"]}
Dec 20 03:52:11 ingress.example.com caddy[718]: {"level":"info","ts":1703044331.5345001,"msg":"autosaved config (load with --resume flag)","file":"/home/caddy/.config/caddy/autosave.json"}
Dec 20 03:52:11 ingress.example.com caddy[718]: {"level":"info","ts":1703044331.5346777,"msg":"serving initial configuration"}
Dec 20 03:52:11 ingress.example.com systemd[1]: Started Caddy.
Dec 20 03:52:12 ingress.example.com caddy[718]: {"level":"info","ts":1703044332.9625828,"logger":"tls","msg":"finished cleaning storage units"}
Dec 20 04:01:51 ingress.example.com caddy[718]: {"level":"error","ts":1703044911.1341724,"logger":"http.log.error","msg":"dial tcp [::1]:30701: connect: connection refused","request":{"remote_ip":"2410:9008:b641:474e:8173:c051:5ba0:6f3","remote_port":"45754","client_ip":"2410:9008:b641:474e:8173:c051:5ba0:6f3","proto":"HTTP/2.0","method":"GET","host":"souin.example.com","uri":"/dashboard","headers":{"Cache-Control":["max-age=0"],"Sec-Ch-Ua-Mobile":["?0"],"Accept-Language":["en-IN,en;q=0.8"],"Sec-Fetch-Dest":["document"],"Cookie":[],"Accept-Encoding":["gzip, deflate, br"],"Sec-Ch-Ua-Platform":["\"Linux\""],"Upgrade-Insecure-Requests":["1"],"Sec-Gpc":["1"],"Sec-Fetch-Site":["same-origin"],"Sec-Fetch-User":["?1"],"Referer":["https://souin.example.com/projects/452862/dashboard"],"Sec-Ch-Ua":["\"Not_A Brand\";v=\"8\", \"Chromium\";v=\"120\", \"Brave\";v=\"120\""],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36"],"Sec-Fetch-Mode":["navigate"],"If-None-Match":["W/\"64dcd516918653ed2d1c773c476c87bb\""]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"souin.example.com"}},"duration":3.001623215,"status":502,"err_id":"bak0babmz","err_trace":"reverseproxy.statusError (reverseproxy.go:1267)"}
Dec 20 04:01:54 ingress.example.com caddy[718]: {"level":"error","ts":1703044914.460583,"logger":"http.log.error","msg":"dial tcp [::1]:30701: connect: connection refused","request":{"remote_ip":"2410:9008:b641:474e:8173:c051:5ba0:6f3","remote_port":"45754","client_ip":"2410:9008:b641:474e:8173:c051:5ba0:6f3","proto":"HTTP/2.0","method":"GET","host":"souin.example.com","uri":"/dashboard","headers":{"Sec-Fetch-User":["?1"],"Referer":["https://souin.example.com/projects/452862/dashboard"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-Dest":["document"],"Cache-Control":["max-age=0"],"Sec-Ch-Ua":["\"Not_A Brand\";v=\"8\", \"Chromium\";v=\"120\", \"Brave\";v=\"120\""],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Ch-Ua-Platform":["\"Linux\""],"Upgrade-Insecure-Requests":["1"],"Accept-Language":["en-IN,en;q=0.8"],"Accept-Encoding":["gzip, deflate, br"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8"],"Sec-Gpc":["1"],"Sec-Fetch-Site":["same-origin"],"Cookie":[]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"souin.example.com"}},"duration":0.002028886,"status":502,"err_id":"9ic3d5q9t","err_trace":"reverseproxy.statusError (reverseproxy.go:1267)"}
Dec 20 04:01:55 ingress.example.com caddy[718]: {"level":"error","ts":1703044915.3554115,"logger":"http.log.error","msg":"dial tcp [::1]:30701: connect: connection refused","request":{"remote_ip":"2410:9008:b641:474e:8173:c051:5ba0:6f3","remote_port":"45754","client_ip":"2410:9008:b641:474e:8173:c051:5ba0:6f3","proto":"HTTP/2.0","method":"GET","host":"souin.example.com","uri":"/dashboard","headers":{"Sec-Ch-Ua":["\"Not_A Brand\";v=\"8\", \"Chromium\";v=\"120\", \"Brave\";v=\"120\""],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8"],"Sec-Gpc":["1"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Ch-Ua-Mobile":["?0"],"Cookie":[],"Cache-Control":["max-age=0"],"Sec-Fetch-User":["?1"],"Referer":["https://souin.example.com/projects/452862/dashboard"],"Sec-Fetch-Dest":["document"],"Sec-Ch-Ua-Platform":["\"Linux\""],"Upgrade-Insecure-Requests":["1"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36"],"Accept-Language":["en-IN,en;q=0.8"],"Sec-Fetch-Site":["same-origin"],"Sec-Fetch-Mode":["navigate"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"souin.example.com"}},"duration":0.000899877,"status":502,"err_id":"0xm433tg8","err_trace":"reverseproxy.statusError (reverseproxy.go:1267)"}
Dec 20 04:01:56 ingress.example.com caddy[718]: {"level":"error","ts":1703044916.040958,"logger":"http.log.error","msg":"dial tcp [::1]:30701: connect: connection refused","request":{"remote_ip":"2410:9008:b641:474e:8173:c051:5ba0:6f3","remote_port":"45754","client_ip":"2410:9008:b641:474e:8173:c051:5ba0:6f3","proto":"HTTP/2.0","method":"GET","host":"souin.example.com","uri":"/dashboard","headers":{"Cache-Control":["max-age=0"],"Sec-Ch-Ua-Platform":["\"Linux\""],"Upgrade-Insecure-Requests":["1"],"Accept-Language":["en-IN,en;q=0.8"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-User":["?1"],"Accept-Encoding":["gzip, deflate, br"],"Referer":["https://souin.example.com/projects/452862/dashboard"],"Sec-Ch-Ua":["\"Not_A Brand\";v=\"8\", \"Chromium\";v=\"120\", \"Brave\";v=\"120\""],"Sec-Ch-Ua-Mobile":["?0"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8"],"Sec-Fetch-Site":["same-origin"],"Sec-Gpc":["1"],"Sec-Fetch-Mode":["navigate"],"Cookie":[]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"souin.example.com"}},"duration":0.000626723,"status":502,"err_id":"x4uy8ekzi","err_trace":"reverseproxy.statusError (reverseproxy.go:1267)"}
Dec 20 08:39:08 ingress.example.com systemd[1]: Reloading Caddy...

More specifically it starts with this error:

Dec 20 04:01:51 ingress.example.com caddy[718]: {"level":"error","ts":1703044911.1341724,"logger":"http.log.error","msg":"dial tcp [::1]:30701: connect: connection refused","request":{"remote_ip":"2410:9008:b641:474e:8173:c051:5ba0:6f3","remote_port":"45754","client_ip":"2410:9008:b641:474e:8173:c051:5ba0:6f3","proto":"HTTP/2.0","method":"GET","host":"souin.example.com","uri":"/dashboard","headers":{"Cache-Control":["max-age=0"],"Sec-Ch-Ua-Mobile":["?0"],"Accept-Language":["en-IN,en;q=0.8"],"Sec-Fetch-Dest":["document"],"Cookie":[],"Accept-Encoding":["gzip, deflate, br"],"Sec-Ch-Ua-Platform":["\"Linux\""],"Upgrade-Insecure-Requests":["1"],"Sec-Gpc":["1"],"Sec-Fetch-Site":["same-origin"],"Sec-Fetch-User":["?1"],"Referer":["https://souin.example.com/projects/452862/dashboard"],"Sec-Ch-Ua":["\"Not_A Brand\";v=\"8\", \"Chromium\";v=\"120\", \"Brave\";v=\"120\""],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36"],"Sec-Fetch-Mode":["navigate"],"If-None-Match":["W/\"64dcd516918653ed2d1c773c476c87bb\""]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"souin.example.com"}},"duration":3.001623215,"status":502,"err_id":"bak0babmz","err_trace":"reverseproxy.statusError (reverseproxy.go:1267)"}

Going deeper into the log, I also found an interesting error before the one above.

{"level":"error","ts":1703014208.7550611,"logger":"http.log.error","msg":"context canceled","request":{"remote_ip":"2410:9008:b641:474e:604e:765e:d8c1:dd26","remote_port":"33296","client_ip":"2410:9008:b641:474e:604e:765e:d8c1:dd26","proto":"HTTP/2.0","method":"GET","host":"souin.example.com","uri":"/__/auth/iframe.js","headers":{"Sec-Gpc":["1"],"Accept-Language":["en-IN,en;q=0.9"],"Referer":["https://souin.example.com/__/auth/iframe?apiKey=AIzaSyDhTIykU2R5ovNZrRmPlN_c_0IS8-YNbrU&appName=%5BDEFAULT%5D&v=10.5.2&eid=p&usegapi=1&jsh=m%3B%2F_%2Fscs%2Fabc-static%2F_%2Fjs%2Fk%3Dgapi.lb.en.y0xCMa4KeeI.O%2Fd%3D1%2Frs%3DAHpOoo8-3MGCaatZB3kdS5TpZdd-gOSBHg%2Fm%3D__features__"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Ch-Ua":["\"Not_A Brand\";v=\"8\", \"Chromium\";v=\"120\", \"Brave\";v=\"120\""],"Sec-Ch-Ua-Mobile":["?1"],"Accept":["*/*"],"Sec-Fetch-Mode":["no-cors"],"User-Agent":["Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Mobile Safari/537.36"],"Sec-Ch-Ua-Platform":["\"Android\""],"Sec-Fetch-Dest":["script"],"Sec-Fetch-Site":["same-origin"],"Cookie":[]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"souin.example.com"}},"duration":0.191336176}

Interestingly, it has "context canceled" message, the same as OP.

To summarize the common factor
msg: "context canceled"
proxying: something from Google frontend. For me, I have to do this: https://firebase.google.com/docs/auth/web/redirect-best-practices#proxy-requests (Option 3).
I deduced that this is rare because most people use Souin for cache HTTP/1 upstream.
Google frontend, on the other hand, utilizes also h2, h3 protocols.
I can share my upstream server with you so you can test
https://slime-systems-ab8f083fce.firebaseapp.com
Please note that the error seems random because I have yet to see the exact pattern.
It might take a lot of tries.

Hello @midnight-wonderer thank you for your feedback.
I think the workflow to reproduce is
-> User sends a request to the proxy
-> The proxy tries to reach the upstream
X-> The user cancel his request
<- The upstream cancel the request too and returns a blank page
<- The proxy gets a blank page
<- The proxy stores the blank page

I will try to make a reproducible example in the E2E tests but that's quite hard to make it reliable.

I forgot to tell crucial information:
In my case, the blank page was triggered by Google upstream with path /__/auth/iframe, but the blank page was cached at the path /projects/452862/dashboard, which has a normal HTTP/1 upstream.
This means that regardless of what triggers context canceled, the blank page can be cached to the adjacent requests.

Reproduced that again.
This time, I noticed that it happened when I clicked on a link in mobile browsers.
But I don't just click a link normally; my finger shudders a bit, causing a rapid double click.
I believe that this time, no Google front-end requests were directly involved in the rapid clicks.
I am not sure if it is the one, but I found someone is able to reproduce the issue here:
https://github.com/golang/go/issues/52183
If we assume it is a bug in the request library, can Souin prevent the problematic response from being cached?

I looked at the HAR file saved from the cached blank page and have a question.

Saved HAR file
{
  "log": {
    "version": "1.2",
    "creator": {
      "name": "WebInspector",
      "version": "537.36"
    },
    "pages": [],
    "entries": [
      {
        "_initiator": {
          "type": "other"
        },
        "_priority": "VeryHigh",
        "_resourceType": "document",
        "cache": {},
        "connection": "6031",
        "request": {
          "method": "GET",
          "url": "https://souin.example.com/projects/452860/dashboard",
          "httpVersion": "http/2.0",
          "headers": [
            {
              "name": ":authority",
              "value": "souin.example.com"
            },
            {
              "name": ":method",
              "value": "GET"
            },
            {
              "name": ":path",
              "value": "/projects/452860/dashboard"
            },
            {
              "name": ":scheme",
              "value": "https"
            },
            {
              "name": "accept",
              "value": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8"
            },
            {
              "name": "accept-encoding",
              "value": "gzip, deflate, br"
            },
            {
              "name": "accept-language",
              "value": "en-IN,en;q=0.7"
            },
            {
              "name": "cache-control",
              "value": "max-age=0"
            },
            {
              "name": "cookie",
              "value": "_document_session=IbSV8KRfMWf%2krZ5MYpZww4%2Fy4TSoe2qxshLVwEGyL6BxAyLwysVI4lEAN4nCOjUrsGom7qHrUWXrU5KXavO5c3w0ig5L1WuKcppCx5dXbEomCuw4MGJG4S%2BfwY4Nu3qLOweRMk5IlYH%2FPNubW9RAIKrxqGGksoNiGzmXCLFaj%2FG9940SnBIX%2B4VZB1DOzAdwd23gw3Q%2FSaZTlB%2BMKpbbmBNFwhvaeCbg7KOZb0dISSnCtQ5y0C92mjUFQI1%2Fwoblj1uTLK%2FFQBeKkz2wbRnjUzy3UDfWny1X5e9H7TFcJ2993MHq9OqhA5o7RxiM5HrdFe%2Fc%2F4ikwoB7b7ZKsUVkeGoJTinw2aM81fki86NEhDJp7Lxfs0yYwd8GlZqI36zUV2%2Fhm2fuhQRBSvVrBnu0azCaDw8fVm5vWzfHRyaiJb93YjzrlPBYQxeXMfqaxrkrmO%2BfGVYezXifx--FoCU8E2Lfmi4R%2Bun--DK7FYC5C9DC%2Fhug2bwVP3Q%3D%3D"
            },
            {
              "name": "referer",
              "value": "https://souin.example.com/dashboard"
            },
            {
              "name": "sec-ch-ua",
              "value": "\"Not_A Brand\";v=\"8\", \"Chromium\";v=\"120\", \"Brave\";v=\"120\""
            },
            {
              "name": "sec-ch-ua-mobile",
              "value": "?1"
            },
            {
              "name": "sec-ch-ua-platform",
              "value": "\"Android\""
            },
            {
              "name": "sec-fetch-dest",
              "value": "document"
            },
            {
              "name": "sec-fetch-mode",
              "value": "navigate"
            },
            {
              "name": "sec-fetch-site",
              "value": "same-origin"
            },
            {
              "name": "sec-fetch-user",
              "value": "?1"
            },
            {
              "name": "sec-gpc",
              "value": "1"
            },
            {
              "name": "upgrade-insecure-requests",
              "value": "1"
            },
            {
              "name": "user-agent",
              "value": "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Mobile Safari/537.36"
            }
          ],
          "queryString": [],
          "cookies": [
            {
              "name": "_thai_document_session",
              "value": "IbSV8KRfMWf%2BEZHVoIGz2cx7eSYwkrZ5MYpZww4%2F4TSoe2qxsIzvLVwEGyL6BxAyLwysVI4lEAN4nCOjUrsGom7qX%2FB%2BrU5KXavO5c3w0ig5L1WuKcB5FppCx5dXbEomCuw4MGJG4S%2BfwY4Nu3qLOweRMk5IlYH%2FPNubW9RAIKrxqGGksoNiGzmXCLFaj%2FG9940SnBIX%2B4VZB1DOzAdwd23gw3Q%2FSaZTlB%2BMKpbbmBNFwhvaeCbg7KOZb0dISSnCtQ5y0C92mjUFQI1%2Fwoblj1uTLK%2FFQBeKkz2wbRnjUzy3UDfWny1X5e9H7TFcJ2993MHq9OqhA5o7RxiM5HrdFe%2Fc%2F4ikwoB7b7ZKsUVkeGoJTinw2aM81fki86NEhDJp7Lxfs0yYwd8GlZqI36zUV2%2Fhm2fuhQRBSvVrBnu0azCaDw8fVm5vWzHRyaiJb93YjzrlPBYQxeXMfqaxrwN8D5jI3krmO%2BfGVYezXifx--FoCU8E2Lfmi4R%2Bun--DK7FYC5C9DC%2Fhug2bwVP3Q%3D%3D",
              "path": "/",
              "domain": "souin.example.com",
              "expires": "1969-12-31T23:59:59.000Z",
              "httpOnly": true,
              "secure": false,
              "sameSite": "Lax"
            }
          ],
          "headersSize": -1,
          "bodySize": 0
        },
        "response": {
          "status": 200,
          "statusText": "",
          "httpVersion": "http/2.0",
          "headers": [
            {
              "name": "age",
              "value": "2264"
            },
            {
              "name": "alt-svc",
              "value": "h3=\":443\"; ma=2592000"
            },
            {
              "name": "cache-status",
              "value": "Souin; hit; ttl=84136; key="
            },
            {
              "name": "content-length",
              "value": "0"
            },
            {
              "name": "date",
              "value": "Thu, 21 Dec 2023 15:19:34 GMT"
            },
            {
              "name": "server",
              "value": "Caddy"
            }
          ],
          "cookies": [],
          "content": {
            "size": 0,
            "mimeType": "text/plain"
          },
          "redirectURL": "",
          "headersSize": -1,
          "bodySize": -1,
          "_transferSize": 108,
          "_error": null
        },
        "serverIPAddress": "[2a02:4f8:c3c:1393:5755:95a8:b7cd:21f7]",
        "startedDateTime": "2023-12-21T15:57:15.862Z",
        "time":<PHONE_NUMBER>99191,
        "timings": {
          "blocked": 26.87999999921309,
          "dns": 148.13,
          "ssl":<PHONE_NUMBER>9999995,
          "connect":<PHONE_NUMBER>000001,
          "send": 1.9639999999999418,
          "wait":<PHONE_NUMBER>0042872,
          "receive": 2.4299999995491817,
          "_blocked_queueing": 22.69099999921309
        }
      }
    ]
  }
}

If I set default_cache_control to no-store, how come Souin caches content without a Cache-Control header present?
P.S. Sorry if my comments are too rapid; I typed what is in my mind at the moment. Please let me know if you mind, and I will try to delay them.

@darkweak is this resolved or not? I am curious since issue is closed, yet other people are reporting issues.

@midnight-wonderer @fliespl the possible fix would be to not store blank responses in cache, idk. Anyway, I'm reopening this issue.

@midnight-wonderer thank you for pointing the go pkg issue 👍

I have been struggling with a similar problem as @frederichoule had: panic: Header called after Handler finished
The problem is that there is not check in CustomWriter.Header() if the request was cancelled, which is allowed in HTTP2.
I worked on a way to reproduce it, and I have this python script
import threading
import socket
from h2.connection import H2Connection
from h2.events import RequestReceived, StreamReset
from h2.config import H2Configuration
import ssl
from h2.errors import ErrorCodes

def root_function(url='localhost'):
    while True:
        try:
            # Create a TCP connection
            sock = socket.create_connection((url, 4433))
            print(f"Create a TCP connection")
            # Wrap the socket for TLS
            ctx = ssl.create_default_context()
            ctx.check_hostname = False
            ctx.verify_mode = ssl.CERT_NONE
            ctx.set_alpn_protocols(['h2'])
            sock = ctx.wrap_socket(sock, server_hostname=url)
            print(f"Wrap socket for TLS")
            # Make sure we're using HTTP/2
            assert sock.selected_alpn_protocol() == 'h2'

            # Create HTTP/2 connection
            config = H2Configuration(client_side=True)
            conn = H2Connection(config=config)
            conn.initiate_connection()
            sock.sendall(conn.data_to_send())

            # # Create a new stream
            # stream_id = conn.get_next_available_stream_id()
            # conn.send_headers(
            #     stream_id,
            #     [(':method', 'GET'), (':authority', url), (':path', '/'), (':scheme', 'https')],
            # )
            # sock.sendall(conn.data_to_send())

            # Read some data
            while True:
                stream_id = conn.get_next_available_stream_id()
                conn.send_headers(
                    stream_id,
                    [(':method', 'GET'), (':authority', url), (':path', '/mycachepath'), (':scheme', 'https')],
                )
                sock.sendall(conn.data_to_send())
                #data = sock.recv(65535)
                #if not data:
                    #break
                conn.reset_stream(stream_id, error_code=ErrorCodes.CANCEL)
                events = conn.receive_data(data)
                for event in events:
                     if isinstance(event, RequestReceived):
                        # Cancel the stream with error code for CANCEL
                         conn.reset_stream(event.stream_id, error_code=ErrorCodes.CANCEL)
                     elif isinstance(event, StreamReset):
                        print(f"Stream {event.stream_id} cancelled.")

                sock.sendall(conn.data_to_send())
        except Exception as e:
            print(f"An error occurred: {e}")

# Create 50 threads running root_function
threads = []
for i in range(1):
    thread = threading.Thread(target=root_function)
    thread.start()
    threads.append(thread)
    print(f"Created thread {i}")

# Keep the main thread alive
for thread in threads:
    thread.join()%

The caddy file is
{
    debug
    order cache before reverse_proxy
    cache {
        log_level debug
    }
}
localhost:4433 {
    tls internal
    encode gzip
    reverse_proxy /mycachepath https://[YOURBACKENDHOST] {
        header_up Host [YOURBACKENDHOST]
        header_up User-Real-IP {remote_host}
        header_up User-Real-Proto {scheme}
        header_up X-Forwarded-For {remote_host}
        header_up X-Forwarded-Port {server_port}
        header_up X-Forwarded-Proto {scheme}
        header_up X-Forwarded-Host {host}
        health_timeout 5s
        # header_down -Cache-Control
    }
    @matchCachePath path /mycachepath
    header /mycachepath >Cache-Control "must-revalidate, no-cache, no-store, private"
    cache @matchCachePath {
        ttl 3s
        stale 3s
        # log_level debug
        mode bypass
        key {
            disable_body
            disable_host
            disable_method
        }
    }
}

The git diff for the fix is this:
diff --git a/pkg/middleware/writer.go b/pkg/middleware/writer.go
index a786c8f..c108c98 100644
--- a/pkg/middleware/writer.go
+++ b/pkg/middleware/writer.go
@@ -43,6 +43,10 @@ type CustomWriter struct {
 func (r *CustomWriter) Header() http.Header {
        r.mutex.Lock()
        defer r.mutex.Unlock()
+
+       if (r.Req.Context().Err() != nil) {
+               return http.Header{}
+       }
        if r.headersSent {
                return http.Header{}
        }

The problem is that now I get another error
fatal error: concurrent map writes

goroutine 216 [running]:
github.com/caddyserver/caddy/v2.(*Replacer).Set(...)
	github.com/caddyserver/caddy/v2@v2.7.6/replacer.go:68
github.com/caddyserver/caddy/v2/modules/caddyhttp.(*HTTPErrorConfig).WithError(0x106887c28, 0x140157a9700, {0x106887c28?, 0x107bdcee0?})
	github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/server.go:656 +0x1a4
github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Server).ServeHTTP(0x140006d2580, {0x10689a450, 0x1400008a8c0}, 0x14015826500)
	github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/server.go:368 +0x1038
net/http.serverHandler.ServeHTTP({0x0?}, {0x10689a450?, 0x1400008a8c0?}, 0x104266a40?)
	net/http/server.go:2938 +0xbc
net/http.initALPNRequest.ServeHTTP({{0x1068a0230?, 0x140157a7f80?}, 0x140157d2000?, {0x140004f64b0?}}, {0x10689a450, 0x1400008a8c0}, 0x14015826500)
	net/http/server.go:3546 +0x1b4
golang.org/x/net/http2.(*serverConn).runHandler(0x107c5b4c0?, 0x0?, 0x0?, 0x1068789e8?)
	golang.org/x/net@v0.19.0/http2/server.go:2368 +0xc0
created by golang.org/x/net/http2.(*serverConn).scheduleHandler in goroutine 232
	golang.org/x/net@v0.19.0/http2/server.go:2303 +0x208

And it is always in replacer.go:68
Hope this helps you

@rzv-me what is your running version? Did you try with the 5a88d84c26128d43e37c614c83c3d3e53da2afd1 commit?
xcaddy build --with github.com/darkweak/souin/plugins/caddy@5a88d84c26128d43e37c614c83c3d3e53da2afd1 --with github.com/darkweak/souin@5a88d84c26128d43e37c614c83c3d3e53da2afd1


UDP port forwarding corrupts DNS packets

When using OVN port forward as configured through LXD, DNS traffic is corrupted. Here's the corruption as observed over UDPv4:
# <IP_ADDRESS> is the IP assigned to u1
+ lxc network forward port add ovn-virtual-network <IP_ADDRESS> udp 53 <IP_ADDRESS>
+ dig a @<IP_ADDRESS> u1.lxd
;; Warning: Message parser reports malformed message packet.

; <<>> DiG 9.18.12-0ubuntu<IP_ADDRESS>-Ubuntu <<>> a @<IP_ADDRESS> u1.lxd
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 53495
;; flags: qr rd ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available
;; WARNING: Message has 22 extra bytes at end

;; QUESTION SECTION:
;u1.lxd.								IN	AAAA

;; ANSWER SECTION:
.			0	CLASS1232 OPT	10 8 Tj9/xCfbVaY=

;; Query time: 0 msec
;; SERVER: 2001:db8:1:2::1#53(2001:db8:1:2::1) (UDP)
;; WHEN: Wed Sep 20 00:43:16 UTC 2023
;; MSG SIZE  rcvd: 81

In both cases, the extra bytes is always 22 for UDPv4 34 for UDPv6. Our environment uses OVN 23.06.1 which isn't the latest but I couldn't find any relevant commit in recent history.
I couldn't easily extract reproducing steps using just OVN commands but will happily try to if a dev could guide me into extracting more debug info. Thanks!

I think the potential fix for https://github.com/ovn-org/ovn/issues/228#issuecomment-1878869615 will also address the issue reported here.

I'll try and get that tested soon, thanks!

@simondeziel Thanks!  I also posted the formal patch on the dev mailing list:
<EMAIL_ADDRESS>

@dceara I've yet to test your patch but with OVN v23.09.1, I'm no longer observing garbled DNS replies. I'll let you know how it goes with your patch.

@simondeziel I wonder if that's not because of https://github.com/ovn-org/ovn/commit/4b10571aa89b226c13a8c5551ceb7208d782b580 (which is in v23.09.1) but that introduces other issues, e.g. problems with EDNS: https://github.com/ovn-org/ovn/issues/228


Direct coverage when a file with the same name exists in the archive.

When adding a file to compression, a file with the same name exists in the archive. an override option should be provided.

Will include this feature in the next release

将在下一版本中包含此功能

If it is a folder, it is best to cover it as well. Thank you for your work.

Compression will crash when a file with the same name exists in the zip!

Hey @isMeiying

Compression will crash when a file with the same name exists in the zip!

Could you please provide your test code and test files? This exception seems not likely caused by same name files. More detail information could help to find out what's going on here.

ZipParameters.overrideExistingFilesInZip feature added in v2.1.3 released today.
Regarding the ZipException, I was unable to reproduce that error both on windows and a mac. As LeeYoung pointed out, please provide more info (code, zip sample, etc) for us to investigate, and please open a new issue if you have those details. Thanks.


Great Work

Wanted to tell you that this is a really cool project!
Awesome work, you can close this whenever you want!
Have a great day!

Thank you very much! You made my day. I'm super happy to know that this tool is useful to people.
I think I'll keep the issue open for a while 🙂

I had been using https://registry.hub.docker.com/r/nvidia/dcgm for awhile, but thought that SMI had so much more information that could be extracted. I probably would never have gotten around to it, so thanks!


Update version pinned dependencies

Lets update these to their latest versions.

Did you read these release notes?

Looked to be bug fixes from what I could tell.

@mlissner does this look good to merge?

Hm, I'm fine upgrading most of these, but selenium tends to have a lot of broken pieces. We can give it a shot, but I suspect it'll break things.

We can give it a shot, but I suspect it'll break things.

I guess we need a new release here as well? Or are there some additional tests that we should run before hand?

Tests in courtlistener pass with updated selenium as well, just needed to make this tweak.

I think it should be fine. I think we use Selenium in three ways:

tests, here
tests in CL
to scrape, using a headless scraper defined in k8s as a pod (or perhaps a deployment)

So monkeying with it here should affect anything and I don't think it even needs a release.

I don't think it even needs a release.

Well if we don't have a release we have to use a git dependency in courtlistener like {git = "https://github.com/freelawproject/juriscraper.git", branch="main"} in order to use the newer selenium version.

I don't think so. CL sets a variable that means it uses a remote selenium driver instead of the one that's part of Juriscraper. Check out WEBDRIVER_CONN.

Check out WEBDRIVER_CONN.

Oh, that's for using a remote webdriver server I think, I'm just referring to updating the python client library version.

Hm, right, OK. @albertisfu could you do a release of Juriscraper when you get a sec, please?

Sure, Juriscraper 2.4.8 release is ready.


Expose the fd in grpc_fd and grpc_tcp.

Also, fixes a bug where grpc_fd::released was not initialized.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

I signed the CLA.

Can you update test/core/iomgr/tcp_posix_test.c to include tests for the new functions here?

(otherwise LGTM)

tcp_posix_test.c updated.

There's a heap use after free in the test:
https://grpc-testing.appspot.com/view/Pull requests/job/gRPC_pull_requests/5017/config=asan,language=c,platform=linux/testReport/junit/(root)/tests/bins_asan_tcp_posix_test/

I had grpc_tcp_fd() after destroy because I didn't realize it was "release and destroy" and not just "release". Fixed now.


Remove GGML_CUDA_FORCE_MMQ=on from build

Removing this build flag seems to cause mmq to be decided at runtime vs always forcing it on. This stops some cuda calls from failing on V100 and other CC 7.0 cards. This may also help with RDNA3 cards
Fixes https://github.com/ollama/ollama/issues/5571
Changes to mmq functionality related to this bug: https://github.com/ggerganov/llama.cpp/compare/7c26775...a8db2a9c#diff-b2fe862fcd5119199ae59ea13d1b6a46e0d23e41e727e39d90913f828a5ff66bR78

I traced this back to the origin and we added this flag to support older CC 5.2 cards - 681a91499   I believe the implication is at least in the short term this change would fix modern cards but regress older cards.  This is probably the right short-term fix, but once I get #5049 over the finish line, then we can adjust the flags differently for the modern vs. older runners.


Bind Monitor with Context

Change construction of Monitor so that context is created along with it. This commit fixes issue #3
as monitor and context are now naturally kept together. In addition to that because MonitorSocket is
now the only owner of context (and is not cloneable) it can be safely send between threads.
Disadvantage of such approach is that application using not only monitor will implicitly create many
contexts.

Well, does not compile on rustc 1.14 because it does not yet have "?". I will fix it after code review.

Closing in favor of #13.

Closing in favor of #13.


Update ingest.py

I've added try-except blocks to handle exceptions and display error messages when an error occurs during script execution. This will help provide more meaningful error feedback to users. You can customize the error messages and handling as needed.

What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)

This PR introduces code quality improvement using Error handling.

Why was this change needed? (You can also link to an open issue here)

As Error Handling is needed during code debugging, using try-catch blocks in our code can help us to handle errors.

Other information:

Good PR, can I please ask you to add some tests for it.

If no activity here, I suggest closing this as it does not significantly improve anything like functionality, and it even a bit makes readability worse.


Code review

"@types/websocket": "^1.0.4", will need to be moved to a depenendency.
For future reference, you can check whether it will build once included by core by:

Delete node_modules and dist
Run yarn --prod
If it didnt error, and it recreated the dist folder, then you are good

your package.json is missing some bugs and repository urls. They arent essential, but do allow us to more easily direct users to this repository eg: https://github.com/bitfocus/companion-module-behringer-xair/commit/1f4963d57f5342182bea59ac6bcbe7d54b5fcd2a
https://github.com/bitfocus/companion-module-ventuz-director/blob/0fb62417dc5931d1e7e5958e06d736302fdfaf06/src/index.ts#L107
https://github.com/bitfocus/companion-module-ventuz-director/blob/0fb62417dc5931d1e7e5958e06d736302fdfaf06/src/index.ts#L92
It would be better for you to track the feedbacks that these calls are interested in yourself (inside subscribe and unsubscribe), as this.getAllFeedbacks() is not a cheap call (it iterates through every feedback on every button on every page to build a list of ones for your module). In fact, you already are creating your own array of them https://github.com/bitfocus/companion-module-ventuz-director/blob/0fb62417dc5931d1e7e5958e06d736302fdfaf06/src/feedbacksProvider.ts#L23
https://github.com/bitfocus/companion-module-ventuz-director/blob/0fb62417dc5931d1e7e5958e06d736302fdfaf06/src/index.ts#L95
You should not be modifying anything on the feedback objects here, companion is currently passing a reference to the raw object that it stored internally, and writes into the db. In 2.3/3.0 this line will probably break, and that property might already be forgotten at times (when the user edits the feedback). Instead, you could store a Map of feedbackId and this value
https://github.com/bitfocus/companion-module-ventuz-director/blob/0fb62417dc5931d1e7e5958e06d736302fdfaf06/src/actionsProvider.ts#L723
Completely optional to change, but I personally would have made this.drModuleInstance.drCompanionInfos be a {[id: string]: Soemthing} or Map<string, Something> using ${info.page}-${info.bank} as the id. Just to avoid doing large array iteration and risk of duplicates in the array. (there are 99x32=3168 buttons available currently and that number will increase at some point)
Again completely optional, but I would recommend considering boolean feedbacks rather than advanced feedbacks. Even though it means that there will need to be 2 feedbacks per button instead of one, it means they can be used in triggers, and in the future as parts of other 'logical expressions/checks'

"@types/websocket": "^1.0.4", will need to be moved to a depenendency.

Done

For future reference, you can check whether it will build once included by core by:

Delete node_modules and dist
Run yarn --prod
If it didnt error, and it recreated the dist folder, then you are good

It runs without problems, dist folder with the according files is also crated.

your package.json is missing some bugs and repository urls. They arent essential, but do allow us to more easily direct users to this repository eg: https://github.com/bitfocus/companion-module-behringer-xair/commit/1f4963d57f5342182bea59ac6bcbe7d54b5fcd2a

Done
About the code recommendations, we have to review them, change the code and retest the module, but for the moment it seems to work stably like this (we have tested it internally).

Hi @Julusian I guess we can close this issue now, with the Companion V3 Upgrade all the mentioned optimizations were handled.


Improvement idea: Just using numerical variables for graph deviation scoring?

Hello, thanks again for publishing this cool project!
If I understand the code right, you use the 'mean squared error' to calculate the loss for both: categorial and numerical variables. As far as I know: state of the art is creating an 'output branch' for every categorial variable and then using a crossentropy loss function for each of those. Since there are quite a lot categorial features in both SWAT and WADI:
Do you think it would be an improvement if you would just use the predictions of the numerical variables for the graph deviation scoring?

Thanks for your interest. I think it is an interesting question, and it could probably be divided into two sub-questions:

Would it be better to use some different loss or specialize some other operations for categorical variables?
Should we just use the predictions of the numerical variables for the scoring?

For Q1, I think it would be yes if more detailed consideration is taken in the dealing with different types of variables/data.
For Q2, I am not quite sure about it, as in some real-world cases, there could exist some abnormal case that only happens in some categorical variables, only using the predictions of the numerical variables could incur False Negatives.


💡 Syntax highlight support

There seems to be two ways to do this:

Implement a complete online editor: https://github.com/quilljs/quill or https://github.com/ajaxorg/ace or  https://github.com/facebook/draft-js or https://github.com/codemirror/codemirror.next/
Use a library: https://github.com/highlightjs/highlight.js

Don't know how to implement them. I'm sorry, I'm not familiar with react yet.

sure will look into it or you can implement this and raise a pr I will merge it

Hey, I am a complete newbie. Can I please try my hand at this?

@alceil

Alright. It's done. I've summited the pull request.

I can't see any syntax highlighting features. I think he is only changed the text editor. Requesting to reopen this issue

Im reopening this issue. Anyone can take up this issue and solve it

Im reopening this issue. Anyone can take up this issue and solve it

May I ?


fix: incorrect context help message

Related issue(s)
Resolves #282

Hey @boyney123 @Souvikns @magicmatatjahu, can you please review my PR.

We also have to update it here https://github.com/asyncapi/cli/blob/a6ba7e092620a90d2871a619516d230387efb535/src/commands/config/context/add.ts#L22

/rtm


Fix assertion to remove spec flakiness

This came up from the failure in https://travis-ci.org/studentinsights/studentinsights/jobs/608894197, which is a strange one since a) I can't reproduce locally on a few partial test runs with the same test seed and b) this code has been like this for a long time and never failed this way.  It's possible that with enough full runs and the same seed this would come up again, but not sure and that would take too long to test.
Regardless, the failure isn't meaningful anyway - these assertions should be checking set membership not the order of a list so this PR changes that.

selfie


Version is not showing in the webUI

The version is not showing at the bottom left of the page.  It shows:
ARTEMIS v.'latest@HEAD'
Not sure if this a bug.  I thought it should say something like: ARTEMIS v.2.0.0

The version you see is normal. You set the version to "latest" (first part of the version) and the "HEAD" is stemming from git itself since you downloaded the latest git code. So the string you see is correct. Resolving.


Re-logging multiple times causes server crash

I used OTclientv8 and it has auto-relog enabled. I discovered that re-logging twice will cause the server to crash and produce this error:
[12:34:48] [Debug] Microsoft.EntityFrameworkCore.ChangeTracking: Context '"OpenTibiaInMemoryDatabaseContext"' started tracking 'System.ArgumentNullException: Value cannot be null. (Parameter 'value')    at OpenTibia.Common.Utilities.Validate.ThrowIfNull(Object o, String paramName) in C:\Users\Josh\Documents\GitHub\fibula-mmo\OpenTibia.Common.Utilities\Validate.cs:line 44    at OpenTibia.Server.Thing.set_ParentCylinder(ICylinder value) in C:\Users\Josh\Documents\GitHub\fibula-mmo\OpenTibia.Server\Thing.cs:line 74    at OpenTibia.Server.PlayerInventory.Finalize() in C:\Users\Josh\Documents\GitHub\fibula-mmo\OpenTibia.Server\PlayerInventory.cs:line 52
I tried the same thing using the real tibia client and it did the same thing but produced this slightly different error instead:
[12:36:23] [Debug] OpenTibia.Server.Game: Placed Player One at [32369, 32241, 7]. System.ArgumentNullException: Value cannot be null. (Parameter 'value')    at OpenTibia.Common.Utilities.Validate.ThrowIfNull(Object o, String paramName) in C:\Users\Josh\Documents\GitHub\fibula-mmo\OpenTibia.Common.Utilities\Validate.cs:line 44    at OpenTibia.Server.Thing.set_ParentCylinder(ICylinder value) in C:\Users\Josh\Documents\GitHub\fibula-mmo\OpenTibia.Server\Thing.cs:line 74    at OpenTibia.Server.PlayerInventory.Finalize() in C:\Users\Josh\Documents\GitHub\fibula-mmo\OpenTibia.Server\PlayerInventory.cs:line 52

Should be addressed with #62.
Please reopen if you find it’s not the case, and thanks for testing!

Huh, I just reproed again, but I know what I missed. Will send out a PR in a bit.

K, now fully addressed in #63


ImportError: DLL load failed: The specified procedure could not be found.

C:\Users\Wissotsky\Desktop\butterflow-0.2.4.alpha2-win64>butterflow.exe
Traceback (most recent call last):
File "C:/msys64/mingw64/lib/python2.7/site-packages/cx_Freeze/initscripts/startup.py", line 14, in run
File "C:/msys64/mingw64/lib/python2.7/site-packages/cx_Freeze/initscripts/Console.py", line 26, in run
File "butterflow/main.py", line 5, in
File "butterflow/cli.py", line 10, in
ImportError: DLL load failed: The specified procedure could not be found.

I'm getting the same error. I am able to call -h and -prb and -d successfully.

Probing video info

PS F:\butterflow-0.2.4.alpha2-win64> .\butterflow.exe -prb F:\sky.mp4
Video information:
  Streams available     : video
  Resolution            : 1920x1080, SAR 1:1 DAR 16:9
  Rate                  : 23.976 fps
  Duration              : 00:00:13 (13.18s)
  Num of frames         : 316

Getting devices

PS F:\butterflow-0.2.4.alpha2-win64> .\butterflow.exe -d
OpenCL devices:
  Platform              : NVIDIA CUDA
  Platform Vendor       : NVIDIA Corporation
  Platform Version      : OpenCL 1.2 CUDA 10.1.120
*  Device 0             : GeForce GTX 1060 6GB
      Version           : OpenCL 1.2
      Version           : 431.36
      Work Sizes        : 1024, 1024x1024x64
      Compatible        : Yes

Processing file

PS F:\butterflow-0.2.4.alpha2-win64> .\butterflow.exe -o f:\skyslow.mp4 F:\sky.mp4
Traceback (most recent call last):
  File "C:/msys64/mingw64/lib/python2.7/site-packages/cx_Freeze/initscripts/__startup__.py", line 14, in run
  File "C:/msys64/mingw64/lib/python2.7/site-packages/cx_Freeze/initscripts/Console.py", line 26, in run
  File "butterflow/__main__.py", line 8, in <module>
  File "butterflow/cli.py", line 288, in main
RuntimeError: opencl call failed

I am actually unable to do anything

So it's failing at import cv2 and again at if not args.sw and not ocl.compat_ocl_device_available():. Probably a packaging thing (a missing DLL in the package or PATHS weren't set right).
I didn't change much code so I figured I didn't have to test this latest version, but thanks for the report I'll try to get it fixed sometime today. Use the latest "stable" version (0.2.3) for now. If you're getting device detection errors look at https://github.com/dthpham/butterflow/issues/103#issuecomment-497534023 for a fix.

Just pushed a fix (I hope it works). You can download 0.2.4a3 from the Latest Releases page and test it out. I don't have a second Windows machine to test on a clean environment so I mostly depend on user reports.


Easun SMW 11KW

hi,
i'm using a k868 a6 board that have the rs232 port, i have connected to the inverter with his original cable.
the firmware that i'm using is esphome with this configuration:
uart:
  - id: uart_0
    baud_rate: 9600
    debug:
    tx_pin: GPIO16
    rx_pin: GPIO17

modbus:
  - id: modbus0
    uart_id: uart_0
    send_wait_time: 200ms

modbus_controller:
  - id: smg0
    address: 0x01
    modbus_id: modbus0
    command_throttle: 200ms
    update_interval: 10s

and my output is this:
[11:17:05][D][uart_debug:114]: >>> 01:03:01:4B:00:07:75:E2
[11:17:05][D][uart_debug:114]: >>> 01:03:01:4B:00:07:75:E2
[11:17:05][D][uart_debug:114]: >>> 01:03:01:4B:00:07:75:E2
[11:17:05][D][uart_debug:114]: >>> 01:03:01:4B:00:07:75:E2
[11:17:05][D][modbus_controller:040]: Modbus command to device=1 register=0x14B countdown=0 no response received - removed from send queue
[11:17:05][D][uart_debug:114]: >>> 01:03:01:96:00:01:65:DA
[11:17:06][D][uart_debug:114]: >>> 01:03:01:96:00:01:65:DA
[11:17:06][D][uart_debug:114]: >>> 01:03:01:96:00:01:65:DA
[11:17:06][D][uart_debug:114]: >>> 01:03:01:96:00:01:65:DA
[11:17:06][D][uart_debug:114]: >>> 01:03:01:96:00:01:65:DA
[11:17:06][D][modbus_controller:040]: Modbus command to device=1 register=0x196 countdown=0 no response received - removed from send queue
[11:17:07][D][uart_debug:114]: >>> 01:03:00:64:00:02:85:D4
[11:17:07][D][uart_debug:114]: >>> 01:03:00:64:00:02:85:D4
[11:17:07][D][uart_debug:114]: >>> 01:03:00:64:00:02:85:D4
[11:17:07][D][uart_debug:114]: >>> 01:03:00:64:00:02:85:D4
[11:17:07][D][uart_debug:114]: >>> 01:03:00:64:00:02:85:D4
[11:17:08][D][modbus_controller:040]: Modbus command to device=1 register=0x64 countdown=0 no response received - removed from send queue
[11:17:08][D][uart_debug:114]: >>> 01:03:00:6C:00:02:04:16
[11:17:08][D][uart_debug:114]: >>> 01:03:00:6C:00:02:04:16
[11:17:08][D][uart_debug:114]: >>> 01:03:00:6C:00:02:04:16
[11:17:08][D][uart_debug:114]: >>> 01:03:00:6C:00:02:04:16
[11:17:09][D][uart_debug:114]: >>> 01:03:00:6C:00:02:04:16
[11:17:09][D][modbus_controller:040]: Modbus command to device=1 register=0x6C countdown=0 no response received - removed from send queue
[11:17:09][D][uart_debug:114]: >>> 01:03:00:C9:00:11:55:F8
[11:17:09][W][modbus_controller:183]: Duplicate modbus command found: type=0x3 address=201 count=17
[11:17:09][W][modbus_controller:183]: Duplicate modbus command found: type=0x3 address=219 count=2
[11:17:09][W][modbus_controller:183]: Duplicate modbus command found: type=0x3 address=223 count=5
[11:17:09][W][modbus_controller:183]: Duplicate modbus command found: type=0x3 address=229 count=1
[11:17:09][W][modbus_controller:183]: Duplicate modbus command found: type=0x3 address=232 count=3
[11:17:09][W][modbus_controller:183]: Duplicate modbus command found: type=0x3 address=300 count=4
[11:17:09][W][modbus_controller:183]: Duplicate modbus command found: type=0x3 address=420 count=1
[11:17:09][W][modbus_controller:183]: Duplicate modbus command found: type=0x3 address=643 count=1
[11:17:09][D][uart_debug:114]: >>> 01:03:00:C9:00:11:55:F8
[11:17:09][D][uart_debug:114]: >>> 01:03:00:C9:00:11:55:F8
[11:17:09][D][uart_debug:114]: >>> 01:03:00:C9:00:11:55:F8
[11:17:10][D][uart_debug:114]: >>> 01:03:00:C9:00:11:55:F8
[11:17:10][D][modbus_controller:040]: Modbus command to device=1 register=0xC9 countdown=0 no response received - removed from send queue
[11:17:10][D][uart_debug:114]: >>> 01:03:00:DB:00:02:B4:30
[11:17:10][D][uart_debug:114]: >>> 01:03:00:DB:00:02:B4:30
[11:17:10][D][uart_debug:114]: >>> 01:03:00:DB:00:02:B4:30
[11:17:11][D][uart_debug:114]: >>> 01:03:00:DB:00:02:B4:30
[11:17:11][D][uart_debug:114]: >>> 01:03:00:DB:00:02:B4:30
[11:17:11][D][modbus_controller:040]: Modbus command to device=1 register=0xDB countdown=0 no response received - removed from send queue
[11:17:11][D][uart_debug:114]: >>> 01:03:00:DF:00:05:B4:33
[11:17:11][D][uart_debug:114]: >>> 01:03:00:DF:00:05:B4:33
[11:17:11][D][uart_debug:114]: >>> 01:03:00:DF:00:05:B4:33
[11:17:12][D][uart_debug:114]: >>> 01:03:00:DF:00:05:B4:33
[11:17:12][D][uart_debug:114]: >>> 01:03:00:DF:00:05:B4:33
[11:17:12][D][modbus_controller:040]: Modbus command to device=1 register=0xDF countdown=0 no response received - removed from send queue
[11:17:12][D][uart_debug:114]: >>> 01:03:00:E5:00:01:95:FD
[11:17:12][D][uart_debug:114]: >>> 01:03:00:E5:00:01:95:FD
[11:17:13][D][uart_debug:114]: >>> 01:03:00:E5:00:01:95:FD
[11:17:13][D][uart_debug:114]: >>> 01:03:00:E5:00:01:95:FD
[11:17:13][D][uart_debug:114]: >>> 01:03:00:E5:00:01:95:FD
[11:17:13][D][modbus_controller:040]: Modbus command to device=1 register=0xE5 countdown=0 no response received - removed from send queue
[11:17:13][D][uart_debug:114]: >>> 01:03:00:E8:00:03:85:FF
[11:17:13][D][uart_debug:114]: >>> 01:03:00:E8:00:03:85:FF
[11:17:14][D][uart_debug:114]: >>> 01:03:00:E8:00:03:85:FF
[11:17:14][D][uart_debug:114]: >>> 01:03:00:E8:00:03:85:FF
[11:17:14][D][uart_debug:114]: >>> 01:03:00:E8:00:03:85:FF
[11:17:14][D][modbus_controller:040]: Modbus command to device=1 register=0xE8 countdown=0 no response received - removed from send queue
[11:17:14][D][uart_debug:114]: >>> 01:03:01:2C:00:04:84:3C
[11:17:14][D][uart_debug:114]: >>> 01:03:01:2C:00:04:84:3C
[11:17:15][D][uart_debug:114]: >>> 01:03:01:2C:00:04:84:3C
[11:17:15][D][uart_debug:114]: >>> 01:03:01:2C:00:04:84:3C
[11:17:15][D][uart_debug:114]: >>> 01:03:01:2C:00:04:84:3C
[11:17:15][D][modbus_controller:040]: Modbus command to device=1 register=0x12C countdown=0 no response received - removed from send queue
[11:17:15][D][uart_debug:114]: >>> 01:03:01:A4:00:01:C4:15
[11:17:16][D][uart_debug:114]: >>> 01:03:01:A4:00:01:C4:15
[11:17:16][D][uart_debug:114]: >>> 01:03:01:A4:00:01:C4:15
[11:17:16][D][uart_debug:114]: >>> 01:03:01:A4:00:01:C4:15
[11:17:16][D][uart_debug:114]: >>> 01:03:01:A4:00:01:C4:15
[11:17:16][D][modbus_controller:040]: Modbus command to device=1 register=0x1A4 countdown=0 no response received - removed from send queue
[11:17:16][D][uart_debug:114]: >>> 01:03:02:83:00:01:74:5A
[11:17:17][D][uart_debug:114]: >>> 01:03:02:83:00:01:74:5A
[11:17:17][D][uart_debug:114]: >>> 01:03:02:83:00:01:74:5A
[11:17:17][D][uart_debug:114]: >>> 01:03:02:83:00:01:74:5A
[11:17:17][D][uart_debug:114]: >>> 01:03:02:83:00:01:74:5A

but on home assistant i don't read anithyng.

This prefix (>>>) indicates outgoing traffic / requests. Your log doesn't show any responses (<<<). In other words: Your inverter does respond. Please double check your wiring and RS232 converter.

Do you know anything about the supported protocols of your inverter?

I think that the cable shouldn't have problem.
I have asked the seller if he send me the modbus protocol

Why is a gender changer required if you use the original cable? Could you provide photos of the complete setup (Inverter<->RS232 converter<->ESP).

This is the board

This is the original cable

The board has natively rs232 port

I've checked the description of the  k868 a6 board:

Using GPIO16 and GPIO17 looks good but I guess it doesn't work because RX & TX are swapped because of the wrong gender of the connector.

In fact I use this board for this reason. I have checked before to use. I have documented.
I have try also to exchange on esphome the gpio but the result is the same.
I will check the cable with the multimeter to understand the pin.

Swapping the GPIOs at the YAML won't work because the GPIOs are attached to the MAX3232 which has dedicated input and output pins. Do you remember the 0-modem cable from the 90s? ;-) This adapter did cross the RX/TX lines of a serial cable. Connecting two PCs using a RS232 cable + a gender changer won't work too.

In fact initially I was looking to use a old serial cable but some days ago I wasted all old cable 🙂↕
I can change the rj45 cable and crimp newer exchanging the cable 1 and 2?

I can change the rj45 cable and crimp newer exchanging the cable 1 and 2?

Please measure the lines first but I guess this would solve the issue.

I  have tried this

But I don't see anything

Could you try the PIP protocol as next step? F.e. using this YAML: https://github.com/syssi/esphome-pipsolar/blob/main/tests/esp8266-test-protocols.yaml
Do you have another ESP + RS232 converter so we could verify the setup is working in general?

Seems that the inverter doesn't respond

Did you try to swap the GPIOs another time?

Could you bridge the TX to the RX pin of the DSUB connector and test using this loopback? Do you see the outgoing traffic as incoming traffic?

Yes, I have tried.
Now I tried also to connect the rs485 of the inverter to the board but the result is the same

This is the log with the bridge and the dummy receiver and log both directions

I have done the bridge in this way

There is still no incoming traffic. Something is wrong with your setup. Did you try to swap the GPIOs assigned to the UART another time?

Feel free to ask more questions if something isn't clear yet.

I try to swap this morning without any good result.
This evening searching at home I find 3 serial cable. Tomorrow I will try :
Check the serial communication from the inverter, I will connect to the pc and open a serial terminal and look if it send any value
Check the rs232 of the board with an original cable. I think that the wires inside the db9 pinout doesn't make good contact

I try to make some findings explicit to provide some guidance. Your ESP32+RS232 board doesn't work as expected at the moment. If you bridge TX and RX we should receive an echo (<<<) of every outgoing frame (>>>). Some ideas what needs to be checked:

Please test the continuity between the DB9 pins (TXD (p2), RXD (p3), GND?  (p5)) and the MAX3232 (R1OUT(p12)T1IN` (p11)).
Test the continuity of the logic level signals of the MAX3232 (R1IN (p13), T1OUT (p14)) and GPIO16 + GPIO17.
Power the k868 a6 board and measure the voltage between the DB9 pin2 (TXD) and pin5 (GND).

I have check with the multimeter.
If on the 3232 chip I bridge 13 and 14 pin I see on the log some code received.
If I bridge the 11 an 12, I don't see anything in the log

Could you show your log? Your observation seems not logic.

With this bridge

I have this log

With this bridge

I have this log

Okay. This makes more sense. In other words: If we bridge GPIO16 and GPIO17 directly (at logical level = 3.3V) the loopback works fine. The uart of your ESP32 works in general. The MAX3232 transforms the signal vom logical level 3.3V to the RS232 voltage level of -12/+12V. If we bridge the RS232 TXD/RXD pins the loopback doesn't work.
As next step we have to make sure your MAX3232 is properly powered. Could you measure the voltage between pin 15 and 16? And could you measure the voltage between the RS232 TXD/RXD pins and GND (p13<->GND, p14<->GND)?

These are the voltages

I have found an old rs232 module that I have used when I was very young.

I connect it and this is the log

It works.
Then the problem is the rs232 module of the board

Connecting this module

I have this output

This garbage happens most of the time if GND isn't connected properly.

Ok now I check better

This looks like garbage only. Please share plaintext logs instead of screenshots!
What are you doing exactly at the moment?

Hi,
I haven't had any times previously, for this reason I haven't put any news.
But during this time I ordered the new device.
An esp32 with Lan the wt32, I connect a rs232 converter and yesterday I started to read some information from the inverter.
Now the question is that in the log I see the value but on home assistant no.
I add the log
INFO ESPHome 2024.12.2
INFO Reading configuration /config/esphome/inverter1.yaml...
WARNING GPIO5 is a strapping PIN and should only be used for I/O with care.
Attaching external pullup/down resistors to strapping pins can cause unexpected failures.
See https://esphome.io/guides/faq.html#why-am-i-getting-a-warning-about-strapping-pins
INFO Starting log output from <IP_ADDRESS> using esphome API
INFO Successfully connected to inverter1 @ <IP_ADDRESS> in 0.262s
INFO Successful handshake with inverter1 @ <IP_ADDRESS> in 0.082s
[02:00:41][I][app:100]: ESPHome version 2024.12.2 compiled on Jan  1 2025, 01:54:19
[02:00:41][C][wifi:600]: WiFi:
[02:00:41][C][wifi:428]:   Local MAC: 88:13:BF:C8:02:D8
[02:00:41][C][wifi:433]:   SSID: [redacted]
[02:00:41][C][wifi:436]:   IP Address: <IP_ADDRESS>
[02:00:41][C][wifi:440]:   BSSID: [redacted]
[02:00:41][C][wifi:441]:   Hostname: 'inverter1'
[02:00:41][C][wifi:443]:   Signal strength: -56 dB ▂▄▆█
[02:00:41][C][wifi:447]:   Channel: 9
[02:00:41][C][wifi:448]:   Subnet: <IP_ADDRESS>
[02:00:41][C][wifi:449]:   Gateway: <IP_ADDRESS>
[02:00:41][C][wifi:450]:   DNS1: <IP_ADDRESS>
[02:00:41][C][wifi:451]:   DNS2: <IP_ADDRESS>
[02:00:41][C][logger:185]: Logger:
[02:00:41][C][logger:186]:   Level: DEBUG
[02:00:41][C][logger:188]:   Log Baud Rate: 115200
[02:00:42][C][logger:189]:   Hardware UART: UART0
[02:00:42][C][uart.arduino_esp32:151]: UART Bus 1:
[02:00:42][C][uart.arduino_esp32:152]:   TX Pin: GPIO17
[02:00:42][C][uart.arduino_esp32:153]:   RX Pin: GPIO5
[02:00:42][C][uart.arduino_esp32:155]:   RX Buffer Size: 256
[02:00:42][C][uart.arduino_esp32:157]:   Baud Rate: 2400 baud
[02:00:42][C][uart.arduino_esp32:158]:   Data Bits: 8
[02:00:42][C][uart.arduino_esp32:159]:   Parity: NONE
[02:00:42][C][uart.arduino_esp32:160]:   Stop bits: 1
[02:00:42][D][uart_debug:158]: >>> "QMODI\xC1\r"
[02:00:42][D][pipsolar:842]: Sending polling command : QMOD with length 4
[02:00:42][C][pipsolar:864]: Pipsolar:
[02:00:42][C][pipsolar:865]: used commands:
[02:00:42][C][pipsolar:868]: QPIRI
[02:00:42][C][pipsolar:868]: QPIGS
[02:00:42][C][pipsolar:868]: QMOD
[02:00:42][C][pipsolar:868]: QFLAG
[02:00:42][C][pipsolar.switch:068]: Pipsolar Switch 'smw-ii output_source_priority_utility'
[02:00:42][C][pipsolar.switch:070]:   Icon: 'mdi:power'
[02:00:42][C][pipsolar.switch:091]:   Restore Mode: always OFF
[02:00:42][D][uart_debug:158]: <<< "(B\xE7\xC9\r"
[02:00:42][C][pipsolar.switch:068]: Pipsolar Switch 'smw-ii output_source_priority_solar'
[02:00:42][C][pipsolar.switch:070]:   Icon: 'mdi:power'
[02:00:42][C][pipsolar.switch:091]:   Restore Mode: always OFF
[02:00:42][D][pipsolar:776]: checking crc on incoming message
[02:00:42][D][pipsolar:779]: CRC OK
[02:00:42][C][pipsolar.switch:068]: Pipsolar Switch 'smw-ii output_source_priority_battery'
[02:00:42][C][pipsolar.switch:070]:   Icon: 'mdi:power'
[02:00:42][C][pipsolar.switch:091]:   Restore Mode: always OFF
[02:00:42][D][pipsolar:475]: Decode QMOD
[02:00:42][D][text_sensor:064]: 'smw-ii last_qmod': Sending state '(B'
[02:00:42][C][pipsolar.switch:068]: Pipsolar Switch 'smw-ii input_voltage_range'
[02:00:42][C][pipsolar.switch:070]:   Icon: 'mdi:power'
[02:00:42][C][pipsolar.switch:091]:   Restore Mode: always OFF
[02:00:42][D][text_sensor:064]: 'smw-ii device_mode': Sending state 'Battery mode'
[02:00:42][C][pipsolar.switch:068]: Pipsolar Switch 'smw-ii pv_ok_condition_for_parallel'
[02:00:42][C][pipsolar.switch:070]:   Icon: 'mdi:power'
[02:00:42][C][pipsolar.switch:091]:   Restore Mode: always OFF
[02:00:42][C][pipsolar.switch:068]: Pipsolar Switch 'smw-ii pv_power_balance'
[02:00:42][C][pipsolar.switch:070]:   Icon: 'mdi:power'
[02:00:42][C][pipsolar.switch:091]:   Restore Mode: always OFF
[02:00:42][C][captive_portal:089]: Captive Portal:
[02:00:42][C][mdns:116]: mDNS:
[02:00:42][C][mdns:117]:   Hostname: inverter1
[02:00:42][C][esphome.ota:073]: Over-The-Air updates:
[02:00:42][C][esphome.ota:074]:   Address: inverter1.local:3232
[02:00:42][C][esphome.ota:075]:   Version: 2
[02:00:42][C][esphome.ota:078]:   Password configured
[02:00:42][C][safe_mode:018]: Safe Mode:
[02:00:42][C][safe_mode:020]:   Boot considered successful after 60 seconds
[02:00:42][C][safe_mode:021]:   Invoke after 10 boot attempts
[02:00:42][C][safe_mode:023]:   Remain in safe mode for 300 seconds
[02:00:42][C][api:140]: API Server:
[02:00:42][C][api:141]:   Address: inverter1.local:6053
[02:00:42][C][api:143]:   Using noise encryption: YES
[02:00:43][D][uart_debug:158]: >>> "QFLAG\x98t\r"
[02:00:43][D][pipsolar:842]: Sending polling command : QFLAG with length 5
[02:00:43][D][uart_debug:158]: <<< "(EbxzDadjklnuvyI7\r"
[02:00:43][D][pipsolar:776]: checking crc on incoming message
[02:00:43][D][pipsolar:779]: CRC OK
[02:00:43][D][pipsolar:483]: Decode QFLAG
[02:00:43][D][text_sensor:064]: 'smw-ii last_qflag': Sending state '(EbxzDadjklnuvy'
[02:00:44][D][uart_debug:158]: >>> "QPIRI\xF8T\r"
[02:00:44][D][pipsolar:842]: Sending polling command : QPIRI with length 5
[02:00:44][D][uart_debug:158]: <<< "(230.0 47.8 230.0 50.0 47.8 11000 11000 48.0 47.0 46.4 56.9 56.9 3 040 150 1 2 3 9 01 0 0 52.0 0 1 480 0 000\xE1S\r"
[02:00:44][D][pipsolar:776]: checking crc on incoming message
[02:00:44][D][pipsolar:786]: CRC NOK expected: E1 29 but got: 0 0
[02:00:45][D][uart_debug:158]: >>> "QPIGS\xB7\xA9\r"
[02:00:45][D][pipsolar:842]: Sending polling command : QPIGS with length 5
[02:00:45][D][uart_debug:158]: <<< "(226.1 50.0 230.1 50.0 01149 01070 011 351 50.40<PHONE_NUMBER> 00.0 000.0 00.00 00024 00010000 00 00 00000 010 0 00 0000\xA3\xA4\r"
[02:00:45][D][pipsolar:776]: checking crc on incoming message
[02:00:45][D][pipsolar:786]: CRC NOK expected: 0 0 but got: 30 A4
[02:00:46][D][uart_debug:158]: >>> "QMODI\xC1\r"
[02:00:46][D][pipsolar:842]: Sending polling command : QMOD with length 4
[02:00:46][D][uart_debug:158]: <<< "(B\xE7\xC9\r"
[02:00:46][D][pipsolar:776]: checking crc on incoming message
[02:00:46][D][pipsolar:779]: CRC OK
[02:00:46][D][pipsolar:475]: Decode QMOD
[02:00:46][D][text_sensor:064]: 'smw-ii last_qmod': Sending state '(B'
[02:00:46][D][text_sensor:064]: 'smw-ii device_mode': Sending state 'Battery mode'
[02:00:47][D][uart_debug:158]: >>> "QFLAG\x98t\r"
[02:00:47][D][pipsolar:842]: Sending polling command : QFLAG with length 5
[02:00:47][D][uart_debug:158]: <<< "(EbxzDadjklnuvyI7\r"
[02:00:47][D][pipsolar:776]: checking crc on incoming message
[02:00:47][D][pipsolar:779]: CRC OK
[02:00:47][D][pipsolar:483]: Decode QFLAG
[02:00:47][D][text_sensor:064]: 'smw-ii last_qflag': Sending state '(EbxzDadjklnuvy'
[02:00:48][D][uart_debug:158]: >>> "QPIRI\xF8T\r"
[02:00:48][D][pipsolar:842]: Sending polling command : QPIRI with length 5
[02:00:48][D][uart_debug:158]: <<< "(230.0 47.8 230.0 50.0 47.8 11000 11000 48.0 47.0 46.4 56.9 56.9 3 040 150 1 2 3 9 01 0 0 52.0 0 1 480 0 000\xE1S\r"
[02:00:48][D][pipsolar:776]: checking crc on incoming message
[02:00:48][D][pipsolar:786]: CRC NOK expected: 8C 37 but got: 0 0
[02:00:49][D][uart_debug:158]: >>> "QPIGS\xB7\xA9\r"
[02:00:49][D][pipsolar:842]: Sending polling command : QPIGS with length 5
[02:00:49][D][uart_debug:158]: <<< "(226.3 50.0 230.0 50.0 01150 01068 011 351 50.40<PHONE_NUMBER> 00.0 000.0 00.00 00023 00010000 00 00 00000 010 0 00 0000\x8F\x9A\r"
[02:00:49][D][pipsolar:776]: checking crc on incoming message
[02:00:49][D][pipsolar:786]: CRC NOK expected: 0 0 but got: 30 9A
[02:00:50][D][uart_debug:158]: >>> "QMODI\xC1\r"
[02:00:50][D][pipsolar:842]: Sending polling command : QMOD with length 4
[02:00:50][D][uart_debug:158]: <<< "(B\xE7\xC9\r"
[02:00:50][D][pipsolar:776]: checking crc on incoming message
[02:00:50][D][pipsolar:779]: CRC OK
[02:00:50][D][pipsolar:475]: Decode QMOD
[02:00:50][D][text_sensor:064]: 'smw-ii last_qmod': Sending state '(B'
[02:00:50][D][text_sensor:064]: 'smw-ii device_mode': Sending state 'Battery mode'
[02:00:51][D][uart_debug:158]: >>> "QFLAG\x98t\r"
[02:00:51][D][pipsolar:842]: Sending polling command : QFLAG with length 5
[02:00:51][D][uart_debug:158]: <<< "(EbxzDadjklnuvyI7\r"
[02:00:51][D][pipsolar:776]: checking crc on incoming message
[02:00:51][D][pipsolar:779]: CRC OK
[02:00:51][D][pipsolar:483]: Decode QFLAG
[02:00:51][D][text_sensor:064]: 'smw-ii last_qflag': Sending state '(EbxzDadjklnuvy'
[02:00:52][D][uart_debug:158]: >>> "QPIRI\xF8T\r"
[02:00:52][D][pipsolar:842]: Sending polling command : QPIRI with length 5
[02:00:52][D][uart_debug:158]: <<< "(230.0 47.8 230.0 50.0 47.8 11000 11000 48.0 47.0 46.4 56.9 56.9 3 040 150 1 2 3 9 01 0 0 52.0 0 1 480 0 000\xE1S\r"
[02:00:52][D][pipsolar:776]: checking crc on incoming message
[02:00:52][D][pipsolar:786]: CRC NOK expected: 62 8C but got: 0 0
[02:00:53][D][uart_debug:158]: >>> "QPIGS\xB7\xA9\r"
[02:00:53][D][pipsolar:842]: Sending polling command : QPIGS with length 5
[02:00:53][D][uart_debug:158]: <<< "(225.9 50.0 230.2 50.0 01128 01044 011 351 50.40<PHONE_NUMBER> 00.0 000.0 00.00 00023 00010000 00 00 00000 010 0 00 0000\xEB>\r"
[02:00:58][D][pipsolar:759]: timeout command to poll: QPIGS
[02:00:58][D][uart_debug:158]: >>> "QMODI\xC1\r"
[02:00:58][D][pipsolar:842]: Sending polling command : QMOD with length 4
[02:00:58][D][uart_debug:158]: <<< "(B\xE7\xC9\r"
[02:00:58][D][pipsolar:776]: checking crc on incoming message
[02:00:58][D][pipsolar:779]: CRC OK
[02:00:58][D][pipsolar:475]: Decode QMOD
[02:00:58][D][text_sensor:064]: 'smw-ii last_qmod': Sending state '(B'
[02:00:58][D][text_sensor:064]: 'smw-ii device_mode': Sending state 'Battery mode'
[02:00:59][D][uart_debug:158]: >>> "QFLAG\x98t\r"
[02:00:59][D][pipsolar:842]: Sending polling command : QFLAG with length 5
[02:00:59][D][uart_debug:158]: <<< "(EbxzDadjklnuvyI7\r"
[02:00:59][D][pipsolar:776]: checking crc on incoming message
[02:00:59][D][pipsolar:779]: CRC OK
[02:00:59][D][pipsolar:483]: Decode QFLAG
[02:00:59][D][text_sensor:064]: 'smw-ii last_qflag': Sending state '(EbxzDadjklnuvy'
[02:01:00][D][uart_debug:158]: >>> "QPIRI\xF8T\r"
[02:01:00][D][pipsolar:842]: Sending polling command : QPIRI with length 5
[02:01:00][D][uart_debug:158]: <<< "(230.0 47.8 230.0 50.0 47.8 11000 11000 48.0 47.0 46.4 56.9 56.9 3 040 150 1 2 3 9 01 0 0 52.0 0 1 480 0 000\xE1S\r"
[02:01:00][D][pipsolar:776]: checking crc on incoming message
[02:01:00][D][pipsolar:786]: CRC NOK expected: 54 4E but got: 0 0
[02:01:01][D][uart_debug:158]: >>> "QPIGS\xB7\xA9\r"
[02:01:01][D][pipsolar:842]: Sending polling command : QPIGS with length 5
[02:01:02][D][uart_debug:158]: <<< "(226.9 49.9 230.0 50.0 01150 01072 011 351 50.40<PHONE_NUMBER> 00.0 000.0 00.00 00024 00010000 00 00 00000 010 0 00 0000\xE4\xF6\r"
[02:01:02][D][pipsolar:776]: checking crc on incoming message
[02:01:02][D][pipsolar:786]: CRC NOK expected: 0 0 but got: 30 F6
[02:01:02][D][uart_debug:158]: >>> "QMODI\xC1\r"
[02:01:02][D][pipsolar:842]: Sending polling command : QMOD with length 4
[02:01:02][D][uart_debug:158]: <<< "(B\xE7\xC9\r"
[02:01:02][D][pipsolar:776]: checking crc on incoming message
[02:01:02][D][pipsolar:779]: CRC OK
[02:01:02][D][pipsolar:475]: Decode QMOD
[02:01:02][D][text_sensor:064]: 'smw-ii last_qmod': Sending state '(B'
[02:01:02][D][text_sensor:064]: 'smw-ii device_mode': Sending state 'Battery mode'
[02:01:03][D][uart_debug:158]: >>> "QFLAG\x98t\r"
[02:01:03][D][pipsolar:842]: Sending polling command : QFLAG with length 5
[02:01:03][D][uart_debug:158]: <<< "(EbxzDadjklnuvyI7\r"
[02:01:03][D][pipsolar:776]: checking crc on incoming message
[02:01:03][D][pipsolar:779]: CRC OK
[02:01:03][D][pipsolar:483]: Decode QFLAG
[02:01:03][D][text_sensor:064]: 'smw-ii last_qflag': Sending state '(EbxzDadjklnuvy'
[02:01:04][D][uart_debug:158]: >>> "QPIRI\xF8T\r"
[02:01:04][D][pipsolar:842]: Sending polling command : QPIRI with length 5
[02:01:05][D][uart_debug:158]: <<< "(230.0 47.8 230.0 50.0 47.8 11000 11000 48.0 47.0 46.4 56.9 56.9 3 040 150 1 2 3 9 01 0 0 52.0 0 1 480 0 000\xE1S\r"
[02:01:05][D][pipsolar:776]: checking crc on incoming message
[02:01:05][D][pipsolar:786]: CRC NOK expected: E0 D0 but got: 0 0
[02:01:05][D][uart_debug:158]: >>> "QPIGS\xB7\xA9\r"
[02:01:05][D][pipsolar:842]: Sending polling command : QPIGS with length 5
[02:01:06][D][uart_debug:158]: <<< "(226.9 50.0 230.2 49.9 01150 01063 011 351 50.40<PHONE_NUMBER> 00.0 000.0 00.00 00024 00010000 00 00 00000 010 0 00 0000r\x02\r"
[02:01:10][D][pipsolar:759]: timeout command to poll: QPIGS
[02:01:10][D][uart_debug:158]: >>> "QMODI\xC1\r"
[02:01:10][D][pipsolar:842]: Sending polling command : QMOD with length 4
[02:01:10][D][uart_debug:158]: <<< "(B\xE7\xC9\r"
[02:01:10][D][pipsolar:776]: checking crc on incoming message
[02:01:10][D][pipsolar:779]: CRC OK
[02:01:10][D][pipsolar:475]: Decode QMOD
[02:01:10][D][text_sensor:064]: 'smw-ii last_qmod': Sending state '(B'
[02:01:10][D][text_sensor:064]: 'smw-ii device_mode': Sending state 'Battery mode'
[02:01:11][D][uart_debug:158]: >>> "QFLAG\x98t\r"


Repository for LIM (fortifydocker/lim) is not present in docker hub

I have tried to install the helm charts for LIM and it doesn't work because we can't pull the image from docker repo.
When I try the command docker pull docker.io/fortifydocker/lim:23.2.ubi.8 I have the following response:
Error response from daemon: pull access denied for fortifydocker/lim, repository does not exist ... requested access to the resource is denied
I have check on docker hub and the repository for LIM is not present. How can I resolve this problem?

Hi, LIM image belongs to a private repository. Please contact support to obtain access.


Resolved date migration from JIRA to ADO

Hi Team,
From JIRA 2 ADO migration we tried to migrate resolved date but we are facing issues while migrating into any system field or any  custom field.
Please let us know how can we do it

@surendra-github please share your field mapping.
Does the resolved date appear in the .json files that the tool creates? Or is the problem on the ADO import side?

On closer inspection, I think that the Microsoft.VSTS.Common date fields should be set automatically as soon as your work item enters the "Resolved" state, right? So Maybe you have omitted Resolved from your state mapping? Is that why Microsoft.VSTS.Common.ResolvedDate is never set? Because I'm pretty sure it's supposed to be set under the hood.

Hi Alexander,
For all existing work items in ADO resolved date is visible, but when we import issues from JIRA it is not capturing resolved date even we are not mapping resolved date in config file.
can you please explain little more , how can we do it ?
Thanks,

The resolved date is simply the datetime when your issue was moved to the Resolved state. Perhaps the issue was never Resolved?

Resolved date for issue in jira is present there in jira.
But we can't migrate it using mapping can you try it from you end? Or do we need to change configuration file?
Can you suggest some changes in it?

I have managed to make the tool generate the following exception:
[E][08:53:21] Error mapping field 'Microsoft.VSTS.Common.ResolvedDate' on item 'AGILEDEMO-14'.
[System.InvalidCastException] System.InvalidCastException: Unable to cast object of type 'System.DateTime' to type 'System.String'.
   at JiraExport.JiraMapper.<>c__DisplayClass14_0`1.<IfChanged>b__0(JiraRevision r)
   at JiraExport.JiraMapper.MapFields(JiraRevision r):
   at JiraExport.JiraMapper.<>c__DisplayClass14_0`1.<IfChanged>b__0(JiraRevision r)
   at JiraExport.JiraMapper.MapFields(JiraRevision r)

Does that look familliar to you @surendra-github ?
I will try to create a fix as soon as I can :)

I was able to succesfully migrate the ResolvedDate field using the following configuration:
  {
      "source": "resolutiondate",
      "type": "datetime",
      "target": "Microsoft.VSTS.Common.ResolvedDate"
  },

This successfully creates the target field in ADO:

        {
          "ReferenceName": "Microsoft.VSTS.Common.ResolvedDate",
          "Value": "2019-12-10 13:54:24"
        },

I am demoting this issue to a support ticket. Feel free to mention me again if the issue persists!

Documenting solution in docs/faq.md

Thank you so much .


[Enhancement] Add Angular library, with core basics and ngrx

Its almost 100% going to happen that this repo will be used with Angular. As such adding
the required dependencies and common helper libraries would be very helpful.

client-logger

common-ngrx

add more ngrx+angular dependencies that are almost always used

angular-material

storybook

angular-flex-layout

angular-cdk

ngrx-store

ngrx-effects

ngrx-store-devtools

ngrx-router-store

ngrx-entity

Basics are added with:
925d5ddaf256f862f87bf1bf8a657283bf187bc4
The rxjs will need to be added later.


Document Preview page improvement

Please @DuuriimaaBargaid complete this page in lexicon/examples with the description as other example pages do

Already in the repo posted


policy: kycfile onboarding issues

When reading the kycfile, if parsePubkeyPair or parseMultisig fails, reading succeeds without any error. This was observed by having the contractintx flag disabled and thus onboarding expected a tweaked address.
Then onboarduser succeeds at creating an onboarding transaction with an empty onboarding script, which means that no address is whitelisted.
I am not sure what _decryptedStream is doing in kycfile.cpp.
Also _kycPubKey seems to never be set in onboardingscript.cpp and when added in FinalizeUnencrypted it has no effect.

_kycPubKey is only needed by onboardingscript.cpp for encrypted whitelist.
_decryptedStream is a decrypted stream of text used for writing a plaintext kycfile and is used by the readkycfile rpc.
This issue is fixed in pull request #221.

This issue is fixed in pull request #221.


Create an optimization for contiguous chars

Either create a new parser MultiChar, or change how OneOrMore and ZeroOrMore function with OneChar and NotChar as their sub-parsers. Written out explicitly, the following scenarios will change.
+"abc"_one => OneOrMore<OneChar<"abc">>
*"abc"_one => ZeroOrMore<OneChar<"abc">>
+"abc"_not => OneOrMore<NotChar<"abc">>
*"abc"_not => ZeroOrMore<NotChar<"abc">>

This can't work for +(string_view_parser) or *(string_view_parser) in general, because it can't be guaranteed to be a contiguous answer in the memory of the input string.

I'm leaning towards a MultiChar parser, because I'd rather not partially specialize the behaviour of the parser types themselves. The optimization should be left to the modifiers, but the parsers should be clear and straightforward. For example in this case, a OneOrMore<P> and ZeroOrMore<P> should always have a result_type of std::vector<typename P::result_type>, with no surprises.

Adding onto my last comment, I think a new type of parser is important for another reason. If you ever actually want to create a OneOrMore<OneChar<"abc">> that has the normal OneOrMore<> behaviour, I want this to be possible. You'll just need to use the types directly instead of the operators. If I create partial specializations for OneOrMore<> and ZeroOrMore<>, then this behaviour can't be overridden at all.
Right now I'm thinking either 2 new parser types are needed, or 1 parser type with a bool parameter.

MultiOneChar<StaticString str> and MultiNotChar<StaticString str>; or
MultiChar<StaticString str, bool Negated>

Regardless of that decision, I'll create new overloads for the following functions. operator+() and operator*() won't need any changes.

detail::operators::one_or_more(OneChar<str>)
detail::operators::one_or_more(NotChar<str>)
detail::operators::zero_or_more(OneChar<str>)
detail::operators::zero_or_more(NotChar<str>)

As a motivator, I feel like this type of thing is very common in the examples right now.
constexpr auto digits = +"0123456789"_one % join;
I think this results in too much run-time computation where it isn't needed. The parser OneChar<"0123456789"> is run repeatedly, the result is pushed to a vector every time, and then the vector is passed into Join to be collapsed back down into a std::string_view. Removing some steps here would be helpful.

On second thought, I think the types should stay the same. +"0123456789"_one % join should still give a Join<OneOrMore<OneChar<"0123456789">>>. How it actually executes is just an implementation detail. I think I should create some partial specializations for Join<> that optimize these situations, but don't change anything about which types are used.

This will use the changes in #50 to implement the Join optimizations.

I'm expanding this to:
Join<OneOrMore<Basic<str>>>
Join<ZeroOrMore<Basic<str>>>
Join<Maybe<Basic<str>>>
Join<Exactly<Basic<str>, N>>
Where Basic <- [OneChar, NotChar, Literal]. I group them together like this because the implementation for Join<OneOrMore<Basic<str>>> will be done once total, not once for each Basic. Likewise for the others in the list above.
All of these parsers are guaranteed to either parse a contiguous string_view, or to fail. There is no in between. I'd rather not allocate/construct the extra intermediary objects, if the final result is a string_view anyway.

I'm implementing this without #84, which was trying a one-size-fits-all solution in a place where it's not required or helpful.


lichdom

Description
Player can become a lich, at which point a model swap is activated. Aedenthorn already made a playable skeleton mod here, which can be studied.
Gameplay Aspects

Player must create a Phylactery at their base
The Phylactery is an expensive item and must be crafted with 10000 gold and loaded with cores
If the player loads 1 infernal core or 50 surtling cores into their Phylactery, the next time they take a blow that would kill them, they instead get teleported back to their phylactery and the core(s) is(are) consumed
The first time they die with a loaded phylactery, they become a lich after teleporting
If the player has no fuel in their phylactery, or no phylactery, then they are killed like normal and if not already a lich no transformation occurs
Liches gain standard undead benefits like immunity to poison, weakness to blunt, etc

Dragon Eggs
Use dragon eggs as a fuel for the Phylactery because they are:

heavy and a pain in the butt to move around
relatively rare
Mostly useless

As such, we bring further utility to a useless item and also make the process of fueling the phylactery a non-trivial task.

My take on lichdom design:
I feel lich transformation would unlock phylactery building, but phylactery wouldn't be the center point and isn't required to transform.
Transformation would happen by performing a ritual, which comes in a form of a consumable item, like a potion, that is crafted with late-plains/early-mistlands items (torn spirit is a good one, maybe also trophies of all, or at least most prominent, undead creatures in the game: wraith, draugr elite and rancid remains). When you use it you become a lich permanently. Should also come with a reverse counterpart item to revert the transformation, for convenience and debugging.
Lich benefits/downsides would be things that stay with you permanently after transformation:

Undead modifiers (weakness to spirit and/or fire, resistance to frost, immunity to poison and cold (night time), I think weakness to any physical damage would make the transformation more trouble than it's worth, most of the damage in this game is physical)
You gain a passive bonus to your health, stamina and eitr equal to your Necromancy skill level, but stats and/or duration of any food you consume is halfed.

I like Cheb's phylactery design, but think that surtling cores aren't a worthy price, once you know where to farm them they basically become a free resource. It should be something you really don't like losing, so fear of death doesn't just get completely removed from the game. Dragon eggs are a good example, you can't get too many of them and they're a pain to transport around, so you get saved from losing your items, buffs and skills, but dealth is still punishing.
Alternative take on phylactery, a bit of a ramble: instead of it being a buildable piece, it's an item you carry around and it works similar to the totem of undying from Minecraft.  For example: one boss trophy might give you an effect of a healing potion, second gives you temporary movement speed boost, another gives bonus damage or attack speed, etc. you can choose which trophy you use depending on your playstyle (or effect power, with later bosses obviously having more powerful effects of the roster).

The phylactery part of this is implemented in 4.1.0

Model
A custom model would be great:
-Choose parts from Draugr and Skeleton
-Each part has its own perk, like Draugr arms bonus to physikal damage
-a craftable Core that is placed at the heart (when Skelleton Torso it floats inside and when Draugr Torso it is not visible), stronger Core  = stronger buff to Magic
Ability
-Lich skill
-No Food consumption
-Health is exchanged with Magic (healthbar becomes blue)
-"Life leech" ability, you get Magic (Health) instead of food through sucking the life out of your enemies (or tames) caped through Necromancy and/or Lich skill
-somthing in exchange for Stamina (Undead are known for infinite Stamina but that would be to OP without other restriction)
-standart Undead pearks for Poison etc.
Transformation
-there is a enhancment request for an altar #229
-A ritual where you need Blood/Souls
-every creature inside radius X is Sacreficed, blood is saved and you can make several Sacrefices
-amount X is needed to use the ritual
-each ritual empty the storage even if you have more than needed


Update everything

Updated the CLI to the latest supported build
Updated packages to netstandard supported versions
Updated xunit and tests to netstandard supported packages
Skipped broken tests

/cc @KrzysztofCwalina
PS: Still fixing tests. Seeing a random failure in System.Slices

Hi @davidfowl, I'm your friendly neighborhood .NET Foundation Pull Request Bot (You can call me DNFBOT). Thanks for your contribution!
You've already signed the contribution license agreement. Thanks!
The agreement was validated by .NET Foundation and real humans are currently evaluating your PR.
TTYL, DNFBOT;