feat: add new collaboration service (WOPI)

Description
Copied from cs3-wopi-server repo.
Adjustments:

package route will be "github.com/owncloud/ocis/v2/services/collaboration" (adjusted in all the files)
go-chi updated to v5 to align to ocis (no new module required)
jwt updated to v4 to align to ocis (no new module required)
Makefile copied from another service
adjust command / service startup to align to the rest of ocis' services
"cmd/cs3-wopi-server/main.go" removed
configuration aligned with ocis

Todo:

~most of the wopi actions are dependent of the "app", which should be eventually removed~
~"health" and "version" commands, pending to be done~
~debug server~
~recheck fileinfo properties. Some of them might be dependent on the specific document server (collabora, onlyoffice, etc)~
~need some refactoring around logging because it's too dense right now.~
some of the GRPC error codes used come from experience running tests and aren't documented. They'll require comments.
recheck for potential problems on non-atomic operations (https://github.com/owncloud/ocis/pull/8374#discussion_r1521443416)

NOTE:
Since the collaboration service requires another external service (onlyoffice, collabora...), there is no plan to automatically start the service using the general ocis server command.
In order to start this service, run ocis collaboration server.
KNOWN ISSUES:

Closing the editor and showing the file list in the web might happen too fast, which could cause the file to be shown as locked because the file might not be uploaded yet (or the upload might be processing), and the unlock request might still be pending. Reloading the page after some seconds (giving enough time to fully finish the upload and unlock) is the easiest workaround.

ADDITIONAL CHANGES IN THE PR:

Additional code included in order to pass the wopi validator tests (including missing implementations)
CI will test for both the builtin wopi server (made in this PR) and the cs3org one
depguard linter is disabled for now until it's properly configured
uniuri dependency replaced with custom code (additional changes in the ocis-pkg/generators package)

Related Issue

Fixes <issue_link>

Motivation and Context

How Has This Been Tested?

test environment:
test case 1:
test case 2:
...

Screenshots (if appropriate):
Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] Technical debt
[ ] Tests only (no source changes)

Checklist:

[ ] Code changes
[ ] Unit tests added
[ ] Acceptance tests added
[ ] Documentation ticket raised:

For the list of configuration options, these are the ones currently unused:
Debug.* (not used)
GRPC.Namespace (used but not written nor configurable -> can be removed)
GRPC.TLS (not used)

HTTP.Namespace (used but not written nor configurable -> can be removed)

GRPCClientTLS (not used)

We can provide a default / hardcoded namespace such as com.owncloud.collaboration for both GRPC and HTTP. The rest of unused configuration should be either removed or commented out (debug server might be needed later and we have some code there already, so could keep it for later use).
A couple of additional things to check:

WOPI secret is generated by default using the "uniuri" module (unless the secret is explicitly provided). We should remove that dependency and generate the secret ourselves if possible.
The secret should also use the TokenManager instead of using a different config key.
HTTP.Scheme is manually provided, but it could be inferred with the HTTP.HTTPServiceTLS.Enabled configuration.

@jvillafanez pls add a readme to that service - this is absolutely necessary. You can find a "guideline" looking into another services like the proxy service.

@jvillafanez would it be in scope of this PR to hook up:

the wopi validator in the .drone.star CI setup?
the deployment example in deployments/examples/ocis_wopi?

Up to you, although I'd rather have small PRs so they're easier to review.
I think the deployment example can be delayed until we have the service "production ready". Currently, it could be in an usable state, but there are still things to be done.
For the wopi validator in the CI, I'll have a look. If it's a couple of lines I think it's fine

@jvillafanez @DeepDiver1975 awesome!
Getting closer.
Can we please hold the merge button until we have a new reva tag relesed, bumped and backported to stable-5.0.
Then we can open up the master branch for 5.1.0-prealpha new features.

Note: Some fileinfo properties have changed in order to make the test pass. We might need to adjust the fileinfo properties based in the target backend since some properties are exclusive of specific backends. I've added the task on the todo list in the initial post

@jvillafanez I have added some comments with regards to envvars. Also see this list of global envvars used.
One set of files is missing that the collaboration service will show up in the dev docs:
docs/services/collaboration
  configuration.md
  _index.md

You can derive them like from the store service.

Q: is the collaboration service scaleable like with Kubernetes?
If yes, anything to consider?
If no, we need to add it to Scaling of Services.

One set of files is missing that the collaboration service will show up in the dev docs:

Pretty sure that the "_index.md" file was auto-generated, but I didn't upload it because it seems to be ignored by git. Generated contents below:
---
title: Collaboration
date: 2024-02-14T15:24:21.442675647+01:00
weight: 20
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/services/collaboration
geekdocFilePath: README.md
geekdocCollapseSection: true
---

<!-- Do not edit this file, it is autogenerated. Edit the service README.md instead -->

## Abstract

The collaboration service connects ocis with document servers such as collabora and onlyoffice using the WOPI protocol.

Since this service requires an external service (onlyoffice, for example), it won't run by default with the general `ocis server` command. You need to run it manually with the `ocis collaboration server` command.

## Table of Contents

* [Requirements](#requirements)
* [Configuration](#configuration)
* [Example Yaml Config](#example-yaml-config)

## Requirements

The collaboration service requires the target document server (onlyoffice, collabora, etc) to be up and running.
We also need reva's gateway and app provider services to be running in order to register the GRPC service for the "open in app" action.

If any of those services are down, the collaboration service won't start.

## Configuration

There are a few variables that you need to set:

* `COLLABORATION_WOPIAPP_ADDR`: The URL of the WOPI app (onlyoffice, collabora, etc). For example: "https://office.mycloud.prv".
* `COLLABORATION_HTTP_ADDR`: The external address of the collaboration service. The target app (onlyoffice, collabora) will use this address to read and write files from ocis. For example: "wopiserver.mycloud.prv"
* `COLLABORATION_HTTP_SCHEME`: The scheme to be used when accessing the collaboration service. Either "http" or "https". This will be used to build the URL that the WOPI app needs in order to contact this service.

The rest of the configuration options available can be left with the default values.
## Example Yaml Config
{{< include file="services/_includes/collaboration-config-example.yaml"  language="yaml" >}}

{{< include file="services/_includes/collaboration_configvars.md" >}}

I can re-generate the file and upload it manually if that's good enough, but since the folder seems to be ignored and auto-generated, I don't know if there is an automatic process that will generate the file by itself and upload it to the repo.

@jvillafanez I think @mmattel is right.
We have local variables which are prefixed by the service name.
We have variables which are prefixed with OCIS  which are convenient globals for the single process mode. So in you case the gateway for example. It should also get an OCIS_GATEWAY annotation.

Taking into account that the "debug" service isn't available yet (and I'd need to check if it applies there or not), only the cs3api vars are the ones that would need to include the "OCIS_" prefix. There rest should be fine already

@micbar I think https://github.com/owncloud/ocis/pull/8374#issuecomment-1976418024 is the only piece left. I can upload the generated file manually, but everything seems to point to an automated process which I don't know if it's ready or not.

Pretty sure that the "_index.md" file was auto-generated

@kobergj can you give some input. I just dont want to miss things and we have post fix work to do... (reminds me that we have to describe that part in the new service docs in more detail)

_index.md files are autogenerated. No need to generate them manually but also does no harm.

Is the same true for configuration.yaml (in the same directory as _index.md)?

Before merging this, PR #8574 needs to be merged, a rebase done and all missing data necessary in new anvvars added according the changes in the referenced PR.

I would do the refactor in here. Adding a new service is a big task nothing we can do about.
Been in the same with the caldav stuff.

@wkloucek could you double check all the TODOs?
As far as I know, these are the files containing TODOs:
pkg/helpers/registration.go
pkg/helpers/discovery.go
pkg/server/debug/server.go
pkg/connector/fileconnector.go
pkg/connector/contentconnector.go

registration -> I can't say anything about the REVA one, but the other one seems legit and we probably should keep it for now.
discovery -> TODO to be removed (we won't log anything on success)
server -> Need to keep them. I haven't found an implementation in other services, so it's probably ok to leave it for now, but we need to implement them at some point
fileconnector -> for the refresh lock, I'm not sure about the priority. As said in the comment, we probably should make a RefreshLock call at that point, but currently it seems to be working without it, so maybe it's fine for now. As for the error code (the TODO below), I think it should return a 500 instead of 404, but I'm not sure.
contentconnector -> I think the access token is required for both, so we should remove the TODO and leave the code. For the lockID, I don't know the current state of REVA, so maybe we should remove the TODO and keep the current code?

For the recheck for potential problems on non-atomic operations (https://github.com/owncloud/ocis/pull/8374#discussion_r1521443416) I don't think we can do much more. I think locks are required except for editing an empty file, so that should be the only race condition that could happen, and it should be "fixed" by using locks. Assuming the code works as expected, it should be good enough
Other than that, I think it's almost ready to review. Last things are fixing sonarcloud report and add some comments; no big changes in the code.

I got it running once, but cannot get the token anymore but getting 500 errors. I don't see what I'm doing differently.
Here is what I'm doing:

run fake hosting discovery

while true; do echo -e "HTTP/1.1 200 OK\n\n$(cat ./tests/config/drone/hosting-discovery.xml)" | nc -l -k -q 0 -p 8080; done

run ocis
PROXY_ENABLE_BASIC_AUTH=true MICRO_REGISTRY=nats-js-kv MICRO_REGISTRY_ADDRESS="<IP_ADDRESS>:9233" NATS_NATS_HOST="<IP_ADDRESS>" NATS_NATS_PORT=9233 GATEWAY_GRPC_ADDR="<IP_ADDRESS>:9142" APP_PROVIDER_EXTERNAL_ADDR="com.owncloud.api.app-provider" APP_PROVIDER_DRIVER="wopi" APP_PROVIDER_WOPI_APP_NAME="FakeOffice" APP_PROVIDER_WOPI_APP_URL=http://localhost:8080 APP_PROVIDER_WOPI_INSECURE=true APP_PROVIDER_WOPI_WOPI_SERVER_EXTERNAL_URL=http://localhost:9300 APP_PROVIDER_WOPI_FOLDER_URL_BASE_URL=https://localhost:9200 ocis/bin/ocis server

run collaboration server
MICRO_REGISTRY="nats-js-kv" MICRO_REGISTRY_ADDRESS="localhost:9233" COLLABORATION_LOG_LEVEL="debug" COLLABORATION_APP_NAME="FakeOffice" COLLABORATION_HTTP_ADDR="localhost:9300" COLLABORATION_HTTP_SCHEME="http" COLLABORATION_WOPIAPP_ADDR="http://localhost:8080" COLLABORATION_WOPIAPP_INSECURE="true" COLLABORATION_CS3API_DATAGATEWAY_INSECURE="true" ocis/bin/ocis collaboration server

get file Id

curl -v -X PUT 'https://localhost:9200/remote.php/webdav/test.wopitest' -k --fail --retry-connrefused --retry 7 --retry-all-errors -u admin:admin -D headers.txt
export FILE_ID=$(cat headers.txt | sed -n -e 's/^.*Oc-Fileid: //p')

get token

export URL="https://localhost:9200/app/open?app_name=FakeOffice&file_id=$FILE_ID"
export URL=$(echo $URL | tr -d '[:cntrl:]')
curl -v -X POST "$URL" -k --fail --retry-connrefused --retry 7 --retry-all-errors -u admin:admin > open.json

and here I get 500 responses
logs of collaboration service:
{"level":"info","service":"collaboration","time":"2024-03-26T16:09:29+05:45","line":"/home/artur/www/ocis/ocis-pkg/registry/register.go:17","message":"registering external service collaboration-1854378d-40b5-493d-811b-c9d122d4cfaf@<IP_ADDRESS>:9301"}
{"level":"debug","service":"collaboration","AppName":"FakeOffice","Mimetypes":["application/octet-stream"],"time":"2024-03-26T16:09:29+05:45","line":"/home/artur/www/ocis/services/collaboration/pkg/helpers/registration.go:54","message":"Registering mimetypes in the app provider"}
{"level":"debug","service":"collaboration","service":{"name":"collaboration","version":"0.0.0","metadata":null,"endpoints":[],"nodes":[{"metadata":{"protocol":"grpc","registry":"cache","server":"grpc","transport":"grpc"},"id":"collaboration-1854378d-40b5-493d-811b-c9d122d4cfaf","address":"<IP_ADDRESS>:9301"}]},"time":"2024-03-26T16:09:59+05:45","line":"/home/artur/www/ocis/ocis-pkg/registry/register.go:30","message":"refreshing external service-registration"}
{"level":"debug","service":"collaboration","service":{"name":"collaboration","version":"0.0.0","metadata":null,"endpoints":[],"nodes":[{"metadata":{"protocol":"grpc","registry":"cache","server":"grpc","transport":"grpc"},"id":"collaboration-1854378d-40b5-493d-811b-c9d122d4cfaf","address":"<IP_ADDRESS>:9301"}]},"time":"2024-03-26T16:10:29+05:45","line":"/home/artur/www/ocis/ocis-pkg/registry/register.go:30","message":"refreshing external service-registration"}
{"level":"debug","service":"collaboration","service":{"name":"collaboration","version":"0.0.0","metadata":null,"endpoints":[],"nodes":[{"metadata":{"protocol":"grpc","registry":"cache","server":"grpc","transport":"grpc"},"id":"collaboration-1854378d-40b5-493d-811b-c9d122d4cfaf","address":"<IP_ADDRESS>:9301"}]},"time":"2024-03-26T16:10:59+05:45","line":"/home/artur/www/ocis/ocis-pkg/registry/register.go:30","message":"refreshing external service-registration"}
{"level":"error","service":"collaboration","error":"Failed to store data in bucket 'ONSXE5TJMNSS24TFM5UXG5DSPFPWG33MNRQWE33SMF2GS33O': nats: timeout","time":"2024-03-26T16:11:04+05:45","line":"/home/artur/www/ocis/ocis-pkg/registry/register.go:33","message":"registration error for external service collaboration"}
{"level":"debug","service":"collaboration","service":{"name":"collaboration","version":"0.0.0","metadata":null,"endpoints":[],"nodes":[{"metadata":{"protocol":"grpc","registry":"cache","server":"grpc","transport":"grpc"},"id":"collaboration-1854378d-40b5-493d-811b-c9d122d4cfaf","address":"<IP_ADDRESS>:9301"}]},"time":"2024-03-26T16:11:29+05:45","line":"/home/artur/www/ocis/ocis-pkg/registry/register.go:30","message":"refreshing external service-registration"}
{"level":"error","service":"collaboration","error":"Failed to store data in bucket 'ONSXE5TJMNSS24TFM5UXG5DSPFPWG33MNRQWE33SMF2GS33O': nats: timeout","time":"2024-03-26T16:11:34+05:45","line":"/home/artur/www/ocis/ocis-pkg/registry/register.go:33","message":"registration error for external service collaboration"}

logs of ocis:
2024/03/26 16:09:37 http: TLS handshake error from <IP_ADDRESS>:51402: remote error: tls: bad certificate
2024/03/26 16:09:45 http: TLS handshake error from <IP_ADDRESS>:34452: remote error: tls: bad certificate
{"level":"error","service":"frontend","pkg":"rhttp","traceid":"57aeab7487fa246be93232fc1630bd0b","request-id":"deepthought3/1ajLMpDm4c-000004","error":"error calling OpenInApp: grpc failed with code CODE_INTERNAL","time":"2024-03-26T16:10:13+05:45","message":"wopi: neither edit nor view app url found"}
{"level":"error","service":"frontend","pkg":"rhttp","traceid":"57aeab7487fa246be93232fc1630bd0b","host":"<IP_ADDRESS>","method":"POST","uri":"/app/open?app_name=FakeOffice&file_id=4ff2a9f9-8432-427a-a2fd-8e7e1a5def56$85c6c562-ecf0-43e3-9b19-e6276b3dfd75!7df242e2-abb8-445c-ac49-b55e4ee05dc1","url":"/open?app_name=FakeOffice&file_id=4ff2a9f9-8432-427a-a2fd-8e7e1a5def56$85c6c562-ecf0-43e3-9b19-e6276b3dfd75!7df242e2-abb8-445c-ac49-b55e4ee05dc1","proto":"HTTP/1.1","status":500,"size":86,"start":"26/Mar/2024:16:10:13 +0545","end":"26/Mar/2024:16:10:13 +0545","time_ns":8725679,"time":"2024-03-26T16:10:13+05:45","message":"http"}
{"level":"error","service":"frontend","pkg":"rhttp","traceid":"6c43cbbe35e2a4026c000affe4a9afaa","request-id":"deepthought3/1ajLMpDm4c-000006","error":"error calling OpenInApp: grpc failed with code CODE_INTERNAL","time":"2024-03-26T16:10:14+05:45","message":"wopi: neither edit nor view app url found"}
{"level":"error","service":"frontend","pkg":"rhttp","traceid":"6c43cbbe35e2a4026c000affe4a9afaa","host":"<IP_ADDRESS>","method":"POST","uri":"/app/open?app_name=FakeOffice&file_id=4ff2a9f9-8432-427a-a2fd-8e7e1a5def56$85c6c562-ecf0-43e3-9b19-e6276b3dfd75!7df242e2-abb8-445c-ac49-b55e4ee05dc1","url":"/open?app_name=FakeOffice&file_id=4ff2a9f9-8432-427a-a2fd-8e7e1a5def56$85c6c562-ecf0-43e3-9b19-e6276b3dfd75!7df242e2-abb8-445c-ac49-b55e4ee05dc1","proto":"HTTP/1.1","status":500,"size":86,"start":"26/Mar/2024:16:10:14 +0545","end":"26/Mar/2024:16:10:14 +0545","time_ns":5232673,"time":"2024-03-26T16:10:14+05:45","message":"http"}
2024/03/26 16:10:20 http: TLS handshake error from <IP_ADDRESS>:47698: remote error: tls: bad certificate

Something fails with the registration of the collaboration service in ocis, but I don't know what.
I'm not sure if it's related, but the collaboration registers the GRPC service in reva only once and doesn't refresh the registration (something to improve). I usually restart the whole stack.

Unit tests added for now should cover most of the "connector" package.
This includes content operations (upload and download) as well as file operations (locks and check file info). Some minor pieces aren't covered because they might need changes in the code and I don't think they're critical enough for now (mostly errors generating the HTTP requests, or copying the responses)
The "httpAdapter" in the "connector" package will need tests. Mocks for the dependencies need to be generated in order to test the adapter (this is still pending)
For the "helpers" package, there are tests for the GetAppUrls function, although we might need some changes in the code. I've seen some panics if the discovery is just a plain text.
I'm not sure how critical this issue is because the discovery phase happens only during the initialization of the server, so if it fails it should be ok to crash the server, at least for now.
Other than that, the only function that could be worthy to test within the "helpers" package would be the RegisterAppProvider, which doesn't have unit tests yet.
Summing up the state of the packages:

"command" -> not tested, likely too difficult to test
"config" -> not worthy, it just contains data with minimum functionality
"connector" -> partially covered with tests. Still needs tests for the "httpAdapter"
"helpers" -> partially covered with tests. The RegisterAppProvider might need tests
"logging" -> not worthy
"middleware" -> not tested, likely too difficult due to cryptographic functions
"server" -> not worthy, it contains initialization of servers and the functionality is elsewhere
"service" -> not tested. This will need tests and maybe little touches to the code

Something fails with the registration of the collaboration service in ocis, but I don't know what. I'm not sure if it's related, but the collaboration registers the GRPC service in reva only once and doesn't refresh the registration (something to improve). I usually restart the whole stack.

restarting did not help. interestingly I cannot kill the collaboration service in the console with Ctrl+C, I always have to kill it with kill -9 <processid>
collaboration is also not shown when doing bin/ocis list
I could trace down my issue to a request going to GET /wopi/iop/openinapp there is a response to that request, but its empty.
HTTP/1.1 200 OK
X-Userlog-Version: 5.1.0-prealpha+38db4d8c99
Date: Thu, 28 Mar 2024 06:31:52 GMT
Content-Length: 0

So the json.Unmarshal in https://github.com/cs3org/reva/blob/5541eeb91bddc138082e2fff07b1136b08eef330/pkg/app/provider/wopi/wopi.go#L240 fails

starting the collaboration service with OCIS_URL=https://localhost:9200 fixed the issue. I'm surprised, because that is not done in drone

starting the collaboration service with OCIS_URL=https://localhost:9200 fixed the issue. I'm surprised, because that is not done in drone

not true, worked a couple of times and now failing again

There should be logs from the collaboration service (https://github.com/owncloud/ocis/pull/8374/files#diff-9fdd676d110dbc32980fc65e66f3521dfd0046f55b7bcdde226821fa737caac4R52). All exits are covered with logs (debug log for success), so something should appear.

restarting did not help. interestingly I cannot kill the collaboration service in the console with Ctrl+C, I always have to kill it with kill -9
collaboration is also not shown when doing bin/ocis list

As far as I know, it should behave like any other service. I'll double-check in case something is different.

restarting did not help. interestingly I cannot kill the collaboration service in the console with Ctrl+C, I always have to kill it with kill -9

I think I know what's happening: basically the GRPC service keeps running despite receiving the "stop" signal, so the "function group" that the server command runs still waits for the GRPC service to finish.
The main thread is stuck in the gr.Run() line, waiting for the GRPC service.

Other services explode with an explicit os.Exit(1) when they receive the signal. This might not be a good idea in the long run because the could be operations that must finish, and an abrupt exit could leave things in an inconsistent state, specially on non-atomic operations.
For the collaboration service, I guess that both the GRPC and HTTP services keep running.

All methods have their corresponding comment now. The only exceptions are in the "mock" package, which is auto-generated.


graphic of numerical order of uploads

align of numerical order of uploads is not correct, you cant see half of it bcs of transparent banner with filename

Thanks for the feedback!
About the arrow on the right: It is indeed a bit disgracious
About the arrow on the left: I actually quite like how it looks right now, a bit stylish and not difficult to read :-)

This seems like solved.


Added a rule for null_edit processing

In the previous PR null_edit events were forgotten. Add a separate rule to process them
cc @wikimedia/services

Coverage remained the same at 95.545% when pulling 7e63be4e3557a164c7266840987bacc9785288ad on Pchelolo:null_edit into 30a50236ff79cf3e7c985bd2d27a76e48f6ddee7 on wikimedia:master.


Default values

Summary: ServerManagerConfig default params moved to rialto/CMakeLists.txt
Type: Feature
Test Plan: UT, FullStack
Jira: RIALTO-377

https://jenkins.code.rdkcentral.com/job/github-triggered/41733/ started

https://jenkins.code.rdkcentral.com/job/build-component/80616/ started.

https://jenkinsrdkm.cmf.code.rdkcentral.com/job/github-triggered/8006/ started

build-rpi-hybrid-wpe-dunfell Console Log uploaded to Gist URL: https://gist.github.com/rdkcmf-jenkins/2ae46b0e1a70f30a0c33015f0b204f89

build-rpi-mc-wpe-dunfell Console Log uploaded to Gist URL: https://gist.github.com/rdkcmf-jenkins/0a8b3c251a8950de292eafa4bd0377a3

build-rpi4-hybrid Console Log uploaded to Gist URL: https://gist.github.com/rdkcmf-jenkins/8059069503f3432109d5345ba7b74ca2

build-rpi-refapp2 Console Log uploaded to Gist URL: https://gist.github.com/rdkcmf-jenkins/b192f6b8701c68d3993571ca6ee3332b

build-rpi4-mc Console Log uploaded to Gist URL: https://gist.github.com/rdkcmf-jenkins/9fa66ab0bc226f7bde8ddc2fbadffe4a

https://jenkins.code.rdkcentral.com/job/github-triggered/41737/ started

https://jenkins.code.rdkcentral.com/job/build-component/80618/ started.

https://jenkinsrdkm.cmf.code.rdkcentral.com/job/github-triggered/8008/ started

build-rpi-mc-wpe-dunfell Console Log uploaded to Gist URL: https://gist.github.com/rdkcmf-jenkins/7a9d51fef8a93f2623f479022b066ca7

build-rpi-hybrid-wpe-dunfell Console Log uploaded to Gist URL: https://gist.github.com/rdkcmf-jenkins/613d359fb19ebdc4dd187419a9db2346

build-rpi4-mc Console Log uploaded to Gist URL: https://gist.github.com/rdkcmf-jenkins/5099db295c86fda5396fa79a0f9c12de

build-rpi-refapp2 Console Log uploaded to Gist URL: https://gist.github.com/rdkcmf-jenkins/88ef34171a81693acf78a65174fa4115

build-rpi4-hybrid Console Log uploaded to Gist URL: https://gist.github.com/rdkcmf-jenkins/5888159f7f11c387969f8dd0472c8930

https://jenkins.code.rdkcentral.com/job/github-triggered/41739/ started

https://jenkins.code.rdkcentral.com/job/build-component/80620/ started.

https://jenkinsrdkm.cmf.code.rdkcentral.com/job/github-triggered/8009/ started

build-rpi-refapp2 Console Log uploaded to Gist URL: https://gist.github.com/rdkcmf-jenkins/cbeaa73bd4612a3f4dcde7de0afac476

build-rpi-hybrid-wpe-dunfell Console Log uploaded to Gist URL: https://gist.github.com/rdkcmf-jenkins/f5c7d195392bbb69880025794d8c39cf

build-rpi-mc-wpe-dunfell Console Log uploaded to Gist URL: https://gist.github.com/rdkcmf-jenkins/839bcb37b17680efa94e2d00c95786d9

https://jenkins.code.rdkcentral.com/job/build-component/80626/ started.

https://jenkinsrdkm.cmf.code.rdkcentral.com/job/github-triggered/8012/ started

Sanity tests for ['RPI4 Hybrid'] added.
https://jenkinsrdkm.cmf.code.rdkcentral.com/job/test-components-flow/29166/

https://jenkins.code.rdkcentral.com/job/github-triggered/41749/ started

https://jenkins.code.rdkcentral.com/job/build-component/80631/ started.

https://jenkinsrdkm.cmf.code.rdkcentral.com/job/github-triggered/8014/ started

https://jenkinsrdkm.cmf.code.rdkcentral.com/job/build-component-platform/13028/ started.

Sanity tests for ['RPI4 Hybrid'] added.
https://jenkinsrdkm.cmf.code.rdkcentral.com/job/test-components-flow/29177/

Sanity tests for ['BCM Reference Refapp2'] added.
https://jenkinsrdkm.cmf.code.rdkcentral.com/job/test-components-flow/29178/

https://jenkins.code.rdkcentral.com/job/github-triggered/41754/ started

https://jenkins.code.rdkcentral.com/job/build-component/80634/ started.

https://jenkinsrdkm.cmf.code.rdkcentral.com/job/github-triggered/8018/ started

https://jenkinsrdkm.cmf.code.rdkcentral.com/job/build-component-platform/13034/ started.

Sanity tests for ['RPI4 Hybrid'] added.
https://jenkinsrdkm.cmf.code.rdkcentral.com/job/test-components-flow/29186/

Sanity tests for ['BCM Reference Refapp2'] added.
https://jenkinsrdkm.cmf.code.rdkcentral.com/job/test-components-flow/29188/


Update delete script for cleanup in Workload Cluster

Since the script is getting longer, can we move this script to the scripts directory (call the script cleanup-control-cluster.sh) and provide a link from here to that script?

Moved the script to scripts/cleanup-workload-cluster.sh, because this is used to clean workload cluster. And updated the README.

Can you push a similar change to the main branch?

Can you push a similar change to the main branch?

Pushed here https://github.com/noironetworks/cko/pull/14


git update

git update using  and <git fetch & git merge>

What's the reason for the changes in init-setup.yaml?

new roles added in init-setup.yaml and here: https://github.com/stereum-dev/ethereum2-ansible/pull/3/commits/70c8cdbcc93cb620babfd1d82c7dad655c43d390#diff-6f78551be385b5661ddd5a5793a1f17e61be9ec2aac687b5ddcddb5d994bf53aL1 'test' is added for testing purpose, but it's now removed.

We'll use init-setup.yaml as the playbook to setup a node from scratch. From my point of view there is no need to fetch for any updates because the node was just created and there can't be any new updates.

We'll use init-setup.yaml as the playbook to setup a node from scratch. From my point of view there is no need to fetch for any updates because the node was just created and there can't be any new updates.

okey, got it, then I going to remove the update roles from init-setup.yaml


+* and -* crash

The select all remove all buttons often crash if there are too many elements or units to deal with.
Requesting something like an "are you sure" popup when more than 100 units/elements would be touched or for the buttons to be greyed out when too many units/elements are visible.
Bonus, seems trying to remove (-*) when some elements are selected (tested with the orientation_elements having 5 elements selected and 5 not selected) will just add more elements.

Should be blazing fast next update(update beardlib also)


[8.x] Verify that SchemaState has the required utilities for restoring a database schema

I ran into the issue where my test environment did not have access to the mysql command. If the required command does not exist the Application will throw an Exception.
What I added is that when we try to restore an existing schema we first check if the required command exists or else we just skip that part.

Thanks for your pull request to Laravel!
Unfortunately, I'm going to delay merging this code for now. To preserve our ability to adequately maintain the framework, we need to be very careful regarding the amount of code we include.
If possible, please consider releasing your code as a package so that the community can still take advantage of your contributions!
If you feel absolutely certain that this code corrects a bug in the framework, please "@" mention me in a follow-up comment with further explanation so that GitHub will send me a notification of your response.


⚠ Bofrost.de has degraded performance

In d1c152d, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 2352 ms

Resolved: Bofrost.de performance has improved in 10d7348 after 35 minutes.


Tighten access to AWS Elasticsearch Service

At the moment, anyone coming from a Pitt or PSC IP address can read and write the hubmap-es-dev2 ES service.  User-specific access restrictions have to be added.

@shirey @jswelling: Is this issue still relevant?

@pdblood this is still relevant.  @jswelling and I should discuss.

This is now controlled with a much tighter firewall list.  Closing.


example/storage: reduce scope of app engine demo

The current example is much broader in scope than the equivalent python demo, and uses a much broader API surface than the dev_appserver will ever be able to emulate.
Consider paring back this demo so that if/when we have dev_appserver support, the app will be able to work in both production and development environments.
@rakyll @broady

I am thinking about removing example/storage and revisiting it once we finalize the dev_appserver support. The migration of the examples has started with re-authoring the references at https://github.com/GoogleCloudPlatform/golang-samples/pull/112.

The examples have been removed, so closing. @rakyll, if this still applies to the example in golang-samples, please open an issue there.

It's OK now, I think. It's a separate thing, and I think useful enough to keep around.


"no documentation available" when there is indeed documentation

The module page shows this.

But the interface is documented.
https://jsr.io/@gameplay/games/doc/poker/~/CheckAction
<EMAIL_ADDRESS>

Related issue: #454


GCC dump support throws exceptions when given a Windows compiler

https://sentry.io/compiler-explorer/compiler-explorer/issues/409375530/
Error: ENOENT: no such file or directory, scandir 'Z:/tmp/compiler-explorer-compiler1171024-55-1uxi964.7zsm'
  File "/compiler-explorer/lib/base-compiler.js", line 488, in Object.Compile.processGccDumpOutput
    var allFiles = fs.readdirSync(rootDir);

Error: ENOENT: no such file or directory, scandir 'Z:/tmp/compiler-explorer-compiler1171024-55-1uxi964.7zsm'

Currently only on the beta site. Having the gcc dump window open and then changing to a MS compiler causes the issue.
Reproducing link: https://godbolt.org/g/FuTy8n
Looks straightforward to fix by only attempting to process dumps for gcc. @dkm ?

Yes. See #638 . I wanted to produce a fix, but last few days have been really busy...

Is there something wrong with this code ? It should list files in the output directories and detect gcc dumps. How is it possible that the directory does not exist ?

Hi @dkm - sorry I didn't spot the other issue. No worries.
In reply to your second comment; no nothing wrong. The error comes from the Z:/tmp/... nature of the file: the wine backend for the MS compilers has to map back and forth between linux filenames (of the form /tmp/blah) to Windows filenames (Z:/tmp/blah) in order to get things to work. It seems that the Z:/ filename is making it to your code somehow...this is something we should fix on our side!

Is this still an issue? I don't seem to be able to reproduce it

I couldn't repro either. Closing.


just thanks

Hi,
I don't know if this is the appropriate channel, but as I couldn't find any forum or similar hub tie to this project, thanks to the gl-matrix team and collaborators for the implementation of this library.
I have borrowed the Vec2 implementation, added a few functions and embedded it in a web crowd simulator that I have been building during the past months.
Thanks,
Jacq

My pleasure! Glad you've found it useful!


get JQUERY path to work

This should make the tabs work for the intro page. Right @jiwalker-usgs ?
It at least made my local version work again
or @mbucknell ?

It should. Merge away


Batocera support

What is your question?
Hi,
I'm trying to run your universal dynamic input on my Batocera (v38) but unfortunately all I see is ? buttons.
I use the PS4's DualShock.
The analog sticks were recognized. I have tried everything described in the FAQ under ? Buttons, but couldn't find a solution.
Is there anything special to configure Delphin on Batocera?

No all desktop versions should be supported the feature is from dolphin itself not this pack.
so the solution should be here https://github.com/Venomalia/UniversalDynamicInput/wiki/Problem-solving#i-see--buttons
since you have apparently already tried this?
Can you send your GCPadNew.ini if it is a gc game or WiimoteNew.ini if it is a wii game, you can find this in the user directory in the Config folder.

GCPadNew.zip
Thanks for your help
It's dolphin 5.0-19864 on batocera v38

i found a solution:
i rename the buttons of my controller in every single UniversalDIT_Defaultstyle of the games on my Batocera System, because the have a different name in GCPadNew.ini.
GCPadNew.ini
[GCPad1]
Device = evdev/0/Wireless Controller
Buttons/A = Button 1
Rumble/Motor = Weak
Buttons/B = Button 0
D-Pad/Down = Axis 7+
Buttons/Hotkey = Button 10
Main Stick/Left = Axis 0-
Main Stick/Right = Axis 0+
Main Stick/Up = Axis 1-
Main Stick/Down = Axis 1+
C-Stick/Left = Axis 3-
C-Stick/Right = Axis 3+
C-Stick/Up = Axis 4-
C-Stick/Down = Axis 4+
Triggers/L = Axis 2+
D-Pad/Left = Axis 6-
Buttons/Z = Button 5
Triggers/R = Axis 5+
D-Pad/Right = Axis 6+
Buttons/Start = Button 9
D-Pad/Up = Axis 7-
Buttons/X = Button 2
Buttons/Y = Button 3
Main Stick/Modifier/Range = 50.0
C-Stick/Modifier/Range = 50.0
[GCPad2]
Main Stick/Modifier/Range = 50.0
C-Stick/Modifier/Range = 50.0
[GCPad3]
Main Stick/Modifier/Range = 50.0
C-Stick/Modifier/Range = 50.0
[GCPad4]
Main Stick/Modifier/Range = 50.0
C-Stick/Modifier/Range = 50.0
UniversalDIT_DefaultStyle
"evdev/0/Wireless Controller": {
"evdev/0/Wireless Controller": "../#DefaultDevices/DualShock4/Device/Icon.png",
"Button 0": "../#DefaultDevices/DualShock4/Flat/Cross.png",
"Button 1": "../#DefaultDevices/DualShock4/Flat/Circle.png",
"Button 3": "../#DefaultDevices/DualShock4/Flat/Square.png",
"Button 2": "../#DefaultDevices/DualShock4/Flat/Triangle.png",
"Button 4": "../#DefaultDevices/DualShock4/Flat/L1.png",
"Button 5": "../#DefaultDevices/DualShock4/Flat/R1.png",
"Axis 2+": "../#DefaultDevices/DualShock4/Flat/L2.png",
"Axis 5+": "../#DefaultDevices/DualShock4/Flat/R2.png",
"Button 9": "../#DefaultDevices/DualShock4/Flat/Options.png",
"Button 8": "../#DefaultDevices/DualShock4/Flat/Share.png",
"Button 10": "../#DefaultDevices/DualShock4/Flat/PS.png",
"LEFT": "../#DefaultDevices/DualShock4/Flat/Touchpad.png",
"Axis 7-": "../#DefaultDevices/DualShock4/Flat/Pad N.png",
"Axis 7+": "../#DefaultDevices/DualShock4/Flat/Pad S.png",
"Axis 6+": "../#DefaultDevices/DualShock4/Flat/Pad E.png",
"Axis 6-": "../#DefaultDevices/DualShock4/Flat/Pad W.png",
"Axis 1+": "../#DefaultDevices/DualShock4/Flat/Left Y-.png",
"Axis 1-": "../#DefaultDevices/DualShock4/Flat/Left Y+.png",
"Axis 0+": "../#DefaultDevices/DualShock4/Flat/Left X+.png",
"Axis 0-": "../#DefaultDevices/DualShock4/Flat/Left X-.png",
"Button 11": "../#DefaultDevices/DualShock4/Flat/L3.png",
"Axis 4+": "../#DefaultDevices/DualShock4/Flat/Right Y-.png",
"Axis 4-": "../#DefaultDevices/DualShock4/Flat/Right Y+.png",
"Axis 3+": "../#DefaultDevices/DualShock4/Flat/Right X+.png",
"Axis 3-": "../#DefaultDevices/DualShock4/Flat/Right X-.png",
"Button 12": "../#DefaultDevices/DualShock4/Flat/R3.png"
},

I'm having the same problem with my Xbox Series X controller. Do you know where GCPad.ini is located in Batocera so I can fine mine and fix this as well?


Add progress bar to technical skills

i have added a progressbar under the technical skills. i think this is a nice visual representation of the skills. i would like to share the code of this.
#let progress_bar(
  progress,
) = {
  // Fix for https://github.com/typst/typst/issues/3826
  if progress == 0% {
    progress = 0.1%
  }

  set block(above: 8pt, below: 8pt, spacing: 0pt)
  set par(leading: 0em)
  let light-accent = chosenAccentColor.lighten(20%)
  let grad-accent = chosenAccentColor.darken(20%)

  context {
    rect(
      height: 8pt,
      width: 100%,
      stroke: chosenAccentColor,
      fill: gradient.linear(
        (light-accent, 0%),
        (grad-accent, progress),
        (white, progress),
        (white, 100%),
      ),

    )
  }
}

#let cvSkill(
    type: "Type",
    tags: (),
    progress: none,
) = {
    skillTypeStyle(type)
    v(-1mm)
    tagListStyle(tags)
    if progress != none {
        v(1mm)
        progress_bar(progress)
    }
}

maybe you can include it to your template

Hey there!
Thanks for sharing your code, adding visual representation is a nice idea.
I have one comment: if the progress bar represents all of your programming skills at once like your screenshot suggests, I'm not sure it helps give a measure of your proficiency. I think it would be better to add one bar per skill (similar to the language section), but that may take up a lot of valuable space.
In any case, I do not plan on adding any features currently: I like the template being fairly minimal as it is right now.
If I change my mind, I'll be sure to keep you posted.
Cheers!

Hey,
Your template is really beautiful and minimalistic. It drew me more into typst to learn it. Thanks for the answer.


zsh-5.4.2 eats unterminated line in command output

Issue description
Here is what I get with zsh-5.4.2:
root@nixos> echo -n "abc\ndef"
abc

It's very confusing, if a program outputs a single unterminated error message and then exits.
With zsh-5.3.1, everything is fine:
root@nixos> echo -n "abc\ndef"
abc
def%

Steps to reproduce
Install zsh in nixos, run zsh, execute a command that outputs an unterminated line.
Technical details

System: 17.09.1502.19879836d1 (Hummingbird)
Nix version: nix-env (Nix) 1.11.15
Nixpkgs version: 17.09.1502.19879836d1
Sandboxing enabled: no

I can reproduce it

I can reproduce. Could this possibly be a zsh issue?

Most probably it is. Or it can be a problem of the prompt (I don't think Nixos uses the default zsh prompt).

5.4.2 from Nixpkgs, non-NixOS prompt, works fine/could not reproduce issue.

Wait, it might be our zshrc?
I exaplain:
zsh prompt: run zsh (uses zshrc) -> echo -n "hi" -> nothing
zsh prompt: run zsh -f -d ( no zshrc) -> echo -n "hi" -> hi%

Yes, I can reproduce it here, too.
Another strange thing:
$> echo -n"abc\ndef"
-nabc
def
$> echo "abc\ndef"
abc
def
$> echo -E "abc\ndef"
abc\ndef

Maybe things of echo builtin?

Maybe things of echo builtin?

No. I originally encountered it with my own Perl program.

bash: echo = "\n\n"
bash: echo -n = "\n"
zsh: echo = "\n"
zsh: echo -n ""
Yeah it's eating the last newline

@7c6f434c can you test if you get this output?

/nix/store/<hash>-coreutils-8.28/bin/echo -e '1\n1'
1
1
/nix/store/<hash>-coreutils-8.28/bin/echo -ne '1\n11\n111\n1111\na'
1
11
111
1111

And in fish it runs OK!

@AndersonTorres it does not depend on echo. both builtin and coretuils echo have the same (wrong) output on zsh and the same (right) output on bash

@nico202 technically, I get colour-inverted version of same, but yes, that's how I have tested the issue in the first place and that's what I get.

For reference, I am not using the default prompt.

With zsh -d -f for me it works:
nixos% echo -n "abc\ndef"
abc
def%
nixos%

@nico202 not sure if this is expected?

More investigation needed but after stumbling on this:
https://www.bountysource.com/issues/48757141-failure-to-output-content-without-newline
I added setopt prompt_sp to my ~/.zshrc (well I added it to the list) and this seems to work now.
Inclined to say "working as expected" but I'm not sure if there are things to patch up re:themes and such.

It's an upstream problem. They changed interaction of the prompt_sp option with prompt_opts (https://github.com/zsh-users/zsh/commit/43e55a9bcd2c90124a751f2597d2f33cb6e3c042), but forgot to update the built-in prompts - at least, the "walters" prompt that Nixos uses by default.

Is this resolved now?

Well the behavior is still present, but since it's an upstream problem and one everyone has (AFAIK?) I think we can go ahead and close this.

Not sure it can be closed like this. For upstream, this is a problem with one of their many prompts, but for Nixos, it's a problem that occurs in default configuration.

Thanks for speaking up and clarifying.  Reopening, sorry :).

I hit this issue, too. It took a lot of my time understanding that zsh's configuration was the problem, not my program.
autoload -U promptinit && promptinit && prompt walters

the last 'promp walters' is causing it.
It could be fixed by setopt prompt_sp
If you added this to promptInit after walter I think people (me ?) would be happy :)

Done in #66992

@LnL7 looks like this should also be updated on darwin-nix, actually if one uses oh-my-zsh with themes like agnoster, setting promptInit to empty is better, otherwise one can see CWD on left and the right side of the terminal.


Support Store decoding for generic resources

Resources such as namespaces and roles are considered generic.
Todo:

[x] draft code per component to decode resources into envoy nodes
[ ] implement codegen based on the draft implementation

Important points:

Special attention when doing modules & module fields, workflows and triggers -- they are two separate resources but should optimally be handled as one
How will we get related RBAC rules and resource translations?

[ ] codegen for compose namespace query construction (ID is wrong)


Version mismatch for google-services plugin

Expected Behaviour
Build should work.
Actual Behaviour
I get a mismatch for the play-services:
Found play-services-identity:11.8.0, but version 11.6.2 is needed for the google-services plugin.
Found play-services-auth:11.8.0, but version 11.6.2 is needed for the google-services plugin.
:app:processDebugGoogleServices FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':app:processDebugGoogleServices'.
> Please fix the version conflict either by updating the version of the google-services plugin (information about the latest version is available at https://bintray.com/android/android-tools/com.google.gms.google-services/) or updating the version of com.google.android.gms to 11.6.2.

Reproduce Scenario (including but not limited to)
Build from CLI. I'm not using Android Studio. I have play services 49 installed, and I can't find how to downgrade to v47 as shown in the docs screenshots.
Steps to Reproduce
cordova build --verbose --info -- --minSdkVersion=17
Platform and Version (eg. Android 5.0 or iOS 9.2.1)
Using cli to build. Building for Android.
Also using cordova-plugin-googleplus and cordova-plugin-facebook4.
Cordova CLI version and cordova platform version
cordova --version                                 8.1.1
cordova platform version android                  7.1.4

Plugin version
cordova plugin version | grep phonegap-plugin-push 2.2.3

Any ideas or workarounds? Perhaps downgrading play services to version 47 (and how to do that from command line)?

OK, problem found. The FCM_VERSION had to be specified. For some reason it didn't work in the xml, but it worked in the command line.
cordova plugin add phonegap-plugin-push --variable FCM_VERSION=11.8.0


Vulcanize index.html so it loads faster

Specially important for low bandwidth connections.

Fixed by #73


Stand Up an Instance of FreeCodeCamp Based on this Repo

I have added a webhook feature into Free Code Camp where a user can provide a URL in the settings page and then, every time they complete a lesson, a JSON object is emitted to the provided end point. This allows a site like www.CodeTrack.dev to track and score the work of users.
I opened an issue proposing this feature with quite a bit of information on the FreeCodeCamp repo here: https://github.com/freeCodeCamp/freeCodeCamp/issues/52542
Since it will likely take a few months to get them to put this feature into production and because I want to better test it before having them merge it, I would like to stand up an independent version of FreeCodeCamp.
This version can be deployed at any URL since it will be tested by my students.
The documentation regarding dev ops and contributing to Free Code Camp is here: https://contribute.freecodecamp.org/#/index
There may be some features that do not need to be configured (like paypal donations) since they are not necessary to testing the webhook feature.
Video demoing the feature locally: https://www.loom.com/share/88fd20c528ea487ea73fdfa061306b53

http://<IP_ADDRESS>/


Due to new NLB functions, Security Groups can now be applied to NLBs.  cfn-lint marks this as an error.

Is this feature request related to a new rule or cfn-lint capabilities?
rules
Describe the feature you'd like to request
Due to a recent change in AWS NLB functionality, you can now associate a Security Group with an NLB.  cfn-lint marks SecurityGroup data in a network type load balancer stanza as an error.
Describe the solution you'd like
Update the rules for NLB load balancer stanzas to not flag the use of a SecurityGroup descriptor node as invalid.
Additional context
No response
Is this something that you'd be interested in working on?

[ ] 👋 I may be able to implement this feature request

Would this feature include a breaking change?

[ ] ⚠ This feature might incur a breaking change

Fixed in #2853 and released with v0.79.8.


[WIP] Integration test through commonly used plugins

In order to help prevent breaking some of the most used plugins and help identify issues with using commonly used plugins as ember-cli-deploy changes, these tests will serve as details of how configurations work along with passing tests to identify breakages.
Items of note currently in flight or in the future

Need to support running the "plugin-integration" tests outside of normal npm test command as these tests will require quite a bit of env setup
Currently I opted to simply npm install the plugin vs including it in the package.json. Alternatively, I thought about creating a set of package.json files to use with these tests. That does open up quite a few rabbit holes however going down one off package.json files. I tried to limit what is included in the current package.json for ember-deploy-cli because node and dependencies. Also, that would be alot to maintain.
leveraging existing ember app in test/app for this to work and deploy.

Have quite a bit of cleanup but here's are first pass. I'll most likely need to update the travis.yml file to provide redis access.

Perhaps it would be a good idea that these smoke tests would live in their own repository. We then wouldn't need to add dependencies on then-redis etc. in ember-cli-deploy directly.

@LevelbossMike I think that would be a good idea as I would also prefer to avoid adding dependencies to ember-cli-deploy that are unnecessary. It would make it a bit simpler also and easier to create multiple package.json files and provide multiple examples of config.
Possibly a directory structure like below to support the multiple apps, etc. This other repository could build based on a hook from ember-cli-deploy master pushes maybe?
[dshultz|ember-cli-deploy-integrations]$ tree
.
├── ember-deploy-redis
│   ├── Brocfile.js
│   └── package.json
├── ember-deploy-s3
│   ├── Brocfile.js
│   └── package.json
└── test-app
    ├── Brocfile.js
    ├── README.md
    ├── app
    ├── bower.json
    ├── config
    │   └── environment.js
    ├── package.json
    ├── public
    │   ├── crossdomain.xml
    │   └── robots.txt
    ├── testem.json
    ├── tests
    │   ├── helpers
    │   │   ├── resolver.js
    │   │   └── start-app.js
    │   ├── index.html
    │   ├── test-helper.js
    │   └── unit
    └── vendor

One other strategy I was thinking about, was doing something similar to what @kategengler did in her ember-try plugin https://github.com/kategengler/ember-try
The test run would execute npm install --save-dev for the necessary dependencies for the tests and then do a git reset --hard as part of the test run to reset for reach test. In doing that, I would also include a safety check to ensure that it would not do a git reset --hard if the index was dirty.
The full integration tests would need a special flag to run or be run single file directly. The tests would primarily run on Travis but could be run locally if a person would like. The tests would leverage environment variables for s3 and redis config

Closing in favor of work in progress by @ghedamat and others


fix: links and references to Atala in the quick-start.md guide

https://input-output.atlassian.net/browse/ATL-7064
This is a non-conflicting merge of changes made before some other changes were merged from other branches.
Please let's merge this one and then we look into the other open PR's, merge and then final overhaul to make sure all fixes are in and everything is correct.
I have an automated tool to check that all links work and all replacement table is applied.

Had to open a new one as this one was far too much behind and was messy already.
https://github.com/input-output-hk/atala-prism-docs/pull/100


Remove pulse shape mention

Remove mention of pulse shape which was due to an old issue, but add a different parameter change in order to actually create that telescope. Fixes #295.

Nice and short...


Can't change FontStore filtering mode

I'm trying to import a pixel font, VT323, but FontStore defaults to linear filtering with no apparent way to change it. The result is not great when I start scaling text up.
I can open up a PR to pass filtering mode through to the underlying TextureStore, is there any reason why this isn't an option already?

The only reason is probably that no one had such a use case until now. Feel free to add this.


proposal: cmd/vet: flag invalid json struct tags

I've recently written codes for json marshaling, containing:
`json:"omitempty"`

as tag. The error only became obvious when I've added another field with the same tag. It has to be:
`json:",omitempty"`

I was wondering if vet could/should detect those (and potentially more invalid tags)?

cmd/vet does already flag certain kinds of invalid struct tags.  This would extend that to look specifically for invalid json struct tags.  And, logically, xml, etc.

One catch here will be tooling outside the Go toolchain that uses these tags. For example, easyjson adds an intern directive to the json tag to enable interning for fields on unmarshal. If go vet disallows this it might have a negative effect, though I don't know the pervasiveness of adding to the standard library's tags versus say, other tooling using their own tags like easyjson:"intern".

I'm not sure how much there is to do here? Vet already flags duplicate tags: ./bar.go:5:2: struct field State repeats json tag "omitempty" also at bar.go:4, and I don't think vet can flag a field simply for being called omitempty, as that would be a false positive if I actually wanted the field to be named that.

I don't think vet can flag a field simply for being called omitempty, as that would be a false positive if I actually wanted the field to be named that.

That's the question imho. Might be the usefulness is outweighed by this drawback.

cc @mvdan

I agree with @dominikh, vet has a high standard for no false positives, so I don't think it can be added. The buggy example you shared originally would be very quickly spotted with a test or by running the code, I presume.
I'm also not sure I get the title of the issue. json:"omitempty" is perhaps confusing, but it's not an invalid tag.

Sorry for the confusion, the title was still on the notion of omitempty affecting decoding. Closing this one on the basis of not generating false positives as vet standard. Thanks for the discussion!


Release 0.5.0

Checklist

[x] Used a fork of the feedstock to propose changes
[x] Bumped the build number (if the version is unchanged)
[x] Reset the build number to 0 (if the version changed)
[x] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[x] Ensured the license file is being packaged.

Builds are all passing, only Github did not received the message from azure correctly I guess. https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=47215&view=logs&jobId=171a126d-c574-5c8c-1269-ff3b989e923d


Show example of anyhow::Result with two type parameters

Closes #26.

Thank you for including the comments about equivalence with std::result::Result. That does completely address my original PR.


Removing the old "Address" form element

Description
With the implementation of AddressComplete, we'll need to remove the old Address form element.
Acceptance Criteria:

[ ] Old Address field is removed
[ ] We've tested this for backwards compatibility to ensure Draft and Published forms with that current selection aren't impacted

Already done by @ShadeWyrm ! Closing this issue


Suggestion: better error message with JavetProxyConverter

See this. Since you use reflection, Throwable t is an InvocationTargetException in most cases, whose getMessage() is always null.
I think it is better to use t.getCause().getMessage() or add a specific catch for InvocationTargetException.
In some cases, user cannot get the root cause Throwable, such as IJavetPromiseRejectCallback. I think it is better to replace getMessage() with toString(). The latter includes class name of Throwable, which provides more useful message.

Thank you for reporting this issue. Please let me know if the fix works or not.

With the new commit, the inner exception is more accurate. Please take a try and let me know if it works.

Sorry. I am too busy to test it in last two weeks.
It works well with my code. Thank you!

But I still have some questions. For example, with JavetProxyConverter:
runtime.getGlobalObject().set("Java", new Object() {
    public Class<?> type(String className) throws ClassNotFoundException {
        return Class.forName(className);
    }
});

try {
    runtime.getExecutor(script).setResourceName(name).executeVoid();
} catch (BaseJavetScriptingException e) {
    JavetScriptingError scriptingError = e.getScriptingError();
    runtime.getLogger().error("failed to execute: " + scriptingError.getStack());
}

Java.type('class that does not exists');

Is there a way to get the root cause (ClassNotFoundException) in the catch block?

Yes, the inner exception is the one (ClassNotFoundException). It was recently fixed. Please make sure you are using the latest snapshot build.

I just downloaded linux build for f17de21. I am sure that e.getCause() returns null.

Thank you for the info. It was a bug and just got fixed. Please try the latest build and let me know if you have any questions.

It works. Thank you!

POC:
try (V8Runtime runtime = V8Host.getV8Instance().createV8Runtime()) {
    runtime.setConverter(new JavetProxyConverter());
    runtime.setPromiseRejectCallback((event, promise, value) -> {
        System.err.println(value);
    });
    JavetStandardConsoleInterceptor console = new JavetStandardConsoleInterceptor(runtime);
    V8ValueGlobalObject global = runtime.getGlobalObject();
    console.register(global);
    runtime.getGlobalObject().set("Class", Class.class);
    try (V8Module module = runtime.getExecutor("Class.forName('abc');")
            .setResourceName("module.js")
            .setModule(true)
            .compileV8Module()) {
        module.executeVoid();
    }
    runtime.getGlobalObject().delete("Class");
    console.unregister(global);
    runtime.lowMemoryNotification();
} catch (Exception e) {
    System.err.println(e);
}

If it uses compileV8Module().executeVoid() like above, both PromiseRejectCallback and catch block handle the exception.
If it simply uses executeVoid(), only PromiseRejectCallback handles the exception, which I think is expected.

Nice catch. A routine was missed. This commit should work as you expected. Please take a try.

Yes, it works.

If a java method throws an exception in js async function, the result Promise is rejected with V8ValueError, whose error message is callback {name} failed with error message {e.getMessage()}. In this case, V8ValueError doesn't provide any way to get the cause Exception. Is it easy to implement? If no, would you consider replacing e.getMessage() with e.toString() or something like e.getClass().getSimpleName() + e.getMessage(). Class name of Exception is often useful or even essential information, such as NullPointerException (Objects.requireNonNull), FileNotFoundException (HttpURLConnection 404) and so on.

Could you provide the reproduceable code and expected output?
toString() sometimes is too much for JS to understand. The coding experience in JS needs to be considered as well. Exposing Java class info in V8 is information leakage, I think.

public class POC {
    public static void main(String[] args) throws Exception {
        try (V8Runtime runtime = V8Host.getV8Instance().createV8Runtime()) {
            runtime.setConverter(new JavetProxyConverter());
            runtime.setPromiseRejectCallback((event, promise, value) -> {
                System.err.println(value);
            });
            JavetStandardConsoleInterceptor console = new JavetStandardConsoleInterceptor(runtime);
            V8ValueGlobalObject global = runtime.getGlobalObject();
            console.register(global);
            POC poc = new POC();
            runtime.getGlobalObject().bind(poc);
            runtime.getExecutor("request('https://cdn.jsdelivr.net/npm/jqery/dist/jquery.min.js');")
                    .setResourceName("module.js")
                    .setModule(true)
                    .executeVoid();
            runtime.getGlobalObject().unbind(poc);
            console.unregister(global);
            runtime.lowMemoryNotification();
        }
    }

    @V8Function
    public String request(String url) throws IOException {
        InputStream inputStream = new URL(url).openStream();
        ByteArrayOutputStream result = new ByteArrayOutputStream();
        byte[] buffer = new byte[4096];
        for (int length; (length = inputStream.read(buffer)) != -1; )
            result.write(buffer, 0, length);
        return result.toString("UTF-8");
    }
}

Output:
Error: https://cdn.jsdelivr.net/npm/jqery/dist/jquery.min.js

Maybe better:
Error: FileNotFoundException: https://cdn.jsdelivr.net/npm/jqery/dist/jquery.min.js

Another example:
public class POC {
    public static void main(String[] args) throws Exception {
        try (V8Runtime runtime = V8Host.getV8Instance().createV8Runtime()) {
            runtime.setConverter(new JavetProxyConverter());
            runtime.setPromiseRejectCallback((event, promise, value) -> {
                System.err.println(value);
            });
            JavetStandardConsoleInterceptor console = new JavetStandardConsoleInterceptor(runtime);
            V8ValueGlobalObject global = runtime.getGlobalObject();
            console.register(global);
            runtime.getGlobalObject().set("poc", POC.class);
            runtime.getExecutor("poc.foo(null);")
                    .setResourceName("module.js")
                    .setModule(true)
                    .executeVoid();
            runtime.getGlobalObject().delete("poc");
            console.unregister(global);
            runtime.lowMemoryNotification();
        }
    }

    public static void foo(Object obj) {
        Objects.requireNonNull(obj);
    }
}

Output:
Error: Callback foo failed with error message <null>

Maybe better:
Error: Callback foo failed with error message NullPointerException

Example 1
I think there are 2 options.

Catch the IOException and throw whatever you want to throw with customized message.
Javet may enhance the Error object to carry the inner exception in JS. That could be a nice feature in the future.

Example 2
If you try Objects.requireNonNull(o, "The argument is null."); in Java, you will get Error: The argument is null. in JS.
Summary
Javet aims at being as less intrusive as possible. The current behavior allows applications to control everything to be exposed in JS to achieve maximal information security. Please let me know if you have any questions.

Catch the IOException and throw whatever you want to throw with customized message.

If you try Objects.requireNonNull(o, "the argument is null."); in Java, you will get Error: Callback foo failed with error message the argument is null. in JS.

In fact, I am developing a library based on Javet. It is a pity that I can't control everything. I will tell my library users that they can better error message in this way.

Javet may enhance the Error object to carry the inner exception in JS. That could be a nice feature in the future.

I would appreciate it if this feature is implemented.

Thank you for solving these issues!
By the way, when will 1.0.7 be released?

Is your library open-sourced? The enhancement for Error will be after v1.0.7 is released. When? Chrome 97 will be released soon (Jan 4), so will v1.0.7 be.

It is not open-sourced for now, but it is on the plan.


MQTT - Motion Start and Stop messages

Hello,
in MQTT page I am trying to configure 'Motion Start' message in this way:
{"ipcam":"sala_on"}
but when I save and reload the page, the 'Motion Start' text is only: sala_on
Using ssh and msquitto_pub I was able to send messages in this way:
mosquitto_pub -h 192.x.x.x -u mqtt_usr -P mqtt_pwd -t home-assistant/ingresso/motion_detection -m '{"ipcam":"sala_on"}'
This is my sensor configuration:
binary_sensor:
  - platform: mqtt
    name: Movimento Sala
    payload_on: "on"
    payload_off: "off"
    device_class: motion
    state_topic: "home-assistant/ingresso/motion_detection"
    value_template: >
      {% if value_json.ipcam == 'sala_on' %}
        on
      {% elif value_json.ipcam == 'sala_off' %}
        off
      {% else %}
        {{ states('binary_sensor.movimento_sala') }}
      {% endif %}

In witch way should I configure the 'Motion Start' message in more complex way (json data format)?
Thanks!

Why do you need json payload?

Well ... I don't necessarily need structured JSON to do what I need ...
It was more of a didactic purpose, to better understand how things work :)

There is a problem with the string parser.
When you save the json payload from the web page, the file .conf contains only "sala_on".

Thanks roleoroleo, at the end I use a simple string 👍
I think you can close

May I continue this topic? I have my yicam 1080 which sends presence detection payloads as well if everything is still in the room.
Any suggestion to check or adjust?

Could you explain me better your purpose?

sure, I'd like to use it as an alert if I'm not at home, BUT I show you the record of last night...

as you can see, nobody was in the kitchen at 3AM, but presence is continuously detected...

This problem was already discussed for MStar based cams.
There isn't a final solution, but this is my suggerstion:

Check the electronic devices in the room (I remember a lot of detection when my phone receveid wa messages and the led flashed)
Set the sensitivity to low

the sensitivity is already set to low.
electronic devices, uh? I'll check..
but, does it use the same alert of the stock fw?

the sensitivity is already set to low.
electronic devices, uh? I'll check..
but, does it use the same alert of the stock fw?

Yes, the motion detection is "sniffed" form the stock fw.
The motion algorithm is developed by yi.


backend/tidb: fix compatible issue in fetch table schemas

What problem does this PR solve?
Fix the compatible issue in show table t next_row_id with v4.0.0~ v4.0.2
What is changed and how it works?
In v4.0.0 tidb  show table t next_row_id don't return the ID_TYPE  column, so we should Scan by column count and set the default value without the extra row
Check List
Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects
Related changes

LGTM


feat: Add Get Conduits & Create Conduit endpoints

This doesn't implement all conduit-related endpoints on purpose since I'm slow/not sure what I'm doing so I thought doing little by little is a better idea
Questions:

The token for create_conduit & get_conduits must be an app access token. Can I check that compile-time somehow with a trait or something?
I've left a bunch of stuff on the bodies/requests (like typed-builder) that I don't really understand. Please let me know if they should be removed

The token for create_conduit & get_conduits must be an app access token. Can I check that compile-time somehow with a trait or something?

There is a way to do it, but not needed
https://docs.rs/twitch_oauth2/0.13.0/twitch_oauth2/trait.TwitchToken.html#tymethod.token_type
That should be promoted to const when available.


fopenReadStream: file not found

Getting following exception when trying to read png file,
Error exec tesseract: exit status 1 Tesseract Open Source OCR Engine v3.03 with Leptonica
Error in pixReadStream: Unknown format: no pix returned
Error in pixRead: pix not read
Error in pixGetInputFormat: pix not defined
Reading /tmp/8df990cd-9890-4e4d-67d9-ac03bc4f44c2 as a list of filenames...
Error in fopenReadStream: file not found
Error in pixRead: image file not found
Image file ┬ëPNG cannot be read!
Error during processing.
Error processing image url: .  Error: exit status 1
Error generating ocr result.  Error: exit status 1←
Can you please  advise me what went wrong for this case ?

This is impossible to debug without clear steps to reproduce.

Here is the payload I used..
-- -----BOUNDARY
Content-Type: application/json
{"engine":"tesseract"}
-----BOUNDARY
Content-Disposition: attachment; filename="attachment.txt".
Content-Type: image/png
PNG

±`>@ ¶ ²fÍý«n¸nñ?ñýýÞzËõ!)LJZ%$	É¿.Ùþ¸»wxùÞ^¯zÕÐ6;U\¼~@ ¶P<°i{ú¿{Üýãç¾PyôÎÿ^ù'7°Ô¢äÎËýÂ§v~á»Ý½_uì3-#E@ [?1Éîzò±Ú¼;.ºj»G~zázâN	IF58ö/Ç|á?ø²}ît÷	@ X¸ç©u¬qçßÞxãÞuü¥k6È#$NQ2òèùÛxß>o?/?È@ T°íbVÛï
§.;ø¨?°Èå#WqW¿~lùNëw<ö£c{~ès·[Fw@ B&V.^Ê<ì§ýíó¾£()qr×¯;¥þÐþï>@ Í
-----BOUNDARY--


implementation code

Outstanding work for promoting Chinese culture! Looking forward to the possibility of open-sourcing the code

@jjzhang-SCUT Hi! Thank you for your interest in our work, and we apologize for the delayed code update.  We have now incorporated the necessary code for reproducing the results.


db-postgres: Pagination shows incorrect totalDocs after deletion using local API with user set

Link to reproduction
https://github.com/joas8211/payloadcms/tree/repro/postgres-delete-pagination-result
Describe the Bug
When using PostgreSQL as database, deleting the only document from a collection using local API causes the immediately following find call return an incorrect totalDocs count (1 instead of 0). The problem is illustrated by a code snippet below:
await payload.delete({
  collection: 'posts',
  id: post.id,
  user, // <-- Setting user probably creates a transaction
})
const result = await payload.find({
  collection: 'posts',
})
console.log(result)
/*
sometimes:
{
  docs: [],
  hasNextPage: false,
  hasPrevPage: false,
  limit: 10,
  nextPage: null,
  page: 1,
  pagingCounter: 1,
  prevPage: null,
  totalDocs: 1,
  totalPages: 1
}

and sometimes:
{
  docs: [
    {
      id: 1,
      updatedAt: '2023-12-21T11:27:56.354Z',
      createdAt: '2023-12-21T11:27:56.354Z'
    }
  ],
  hasNextPage: false,
  hasPrevPage: false,
  limit: 10,
  nextPage: null,
  page: 1,
  pagingCounter: 1,
  prevPage: null,
  totalDocs: 1,
  totalPages: 1
}
*/

To Reproduce

Clone the reproduction branch:git clone git@github.com:joas8211/payloadcms.git --branch repro/postgres-delete-pagination-result

Setup development environment for PostgreSQL.
Run test in _community folder using PostgreSQL as database:PAYLOAD_DATABASE=postgres pnpm test:int test/_community/int.spec.ts

Payload Version
2.5.0
Adapters and Plugins
db-postgres: 0.2.2

Updated the example in the description, I originally copy-pasted the output from test using mongoose, which was correct output to pass the test, not the right one for this bug. Now there's example outputs while using Postgres adapter. It's noteworthy that the output seems to variate a bit. Sometimes the document is still in the result, sometimes not but totalDocs is still 1.


feat: allow orphaned parents

There is a use case where it is desired to use "real" nodes as parent nodes instead of creating temporary ones.
This can be accomplished with "newParentNode" by returning the dropTarget:
  newParentNode: (grabbedNode: any, dropTarget: any) => {
    return dropTarget;
  },

The new part is the addition of "allowOrphanedParents" which prevents the parent node from being deleted and allows the last child of a parent to be taken out.

(1) It looks like your use of newParentNode() is based on a side-effect rather than by design (i.e. the return value is expected to be JSON, not an existing element).  If this is to be allowed, then the documentation should be updated to make this use case explicit.
(2) Is a new option necessary if the existing newParentNode() function would be allowed to work as you propose?  Could this be made automatic?  The only reason the new parent would be removed when dragging one orphan over another is because no parent exists.  This new parent has no reason for being if the gesture is cancelled, and so it is removed.  This does not hold in your case, so I suspect you could just detect that case and then handle it appropriately.

Hello,
I think that if we do not specify a "newParentNode" function, by default the target node should become a parent node - it feels natural.
If we accept this then the "allowOrphanedParents" can be removed and automated:

If the parent node is a "real node" then always keep it and allow child nodes to exit it
If the parent node is provided from "newParentNode" then remove parent nodes (working as previous)

What do you think ?

I don't think we should be changing the defaults just now.  My main concern is that the consumer should not be specifying options that can be determined implicitly and internally by the library.
So internally, you can have something like allowOrphanedParents if that’s how you want to structure it (e.g. as a function or computed property), but it should be determined by the extension by inspecting the result of the existing options like newParentNode

On Nov 12, 2021, at 18:45, Alexander KIRILOV @.***> wrote:
Hello,
I think that if we do not specify a "newParentNode" function, by default the target node should become a parent node - it feels natural.
If we accept this then the "allowOrphanedParents" can be removed and automated:
If the parent node is a "real node" then always keep it and allow child nodes to exit it
If the parent node is provided from "newParentNode" then remove parent nodes (working as previous)
What do you think ?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub https://github.com/cytoscape/cytoscape.js-compound-drag-and-drop/pull/17#issuecomment-967731116, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHRO44TNKJZKOFYFSTAO4TULWRKJANCNFSM5HDPWFDA.

closed in favor of #22 which takes into account your suggestion for automatic detection of real parents


Most performant way of getting typed AST of single file edited after full parsing

One of the selling points of Fable is it works very well in watch mode as frontend developers are used to work. However, I still don't know what's the best way to get the typed AST of a single file edited after fully parsing a project. Right now I'm using ParseAndCheckProject for both cases (first project load and file edits after that), the only difference is I reuse the ProjectOptions for watch compilations.
Is there a better way to do this? I've seen there're some methods like ParseFileInProject but it doesn't give the FSharpImplementationFileContents. Also, should I set LoadTime in ProjectOptions to any specific value?
Thanks! (Pinging @dsyme for help 😄 )

@alfonsogarciacaro If you want FSharpImplementationFileContents then ParseAndCheckProject is correct right now
If you want a single file - I suppose we could add FSharpImplementationFileContents to the results from ParseFileInProject

Probably being able to parse a single file would be useful though some cases may be a bit complicated (for example, the edited file may have to access the AST of another file to inline a method, which may or may have not have changed as well).
My question is mainly if there's any performance trick I may be overlooking as I don't know well how the cache works. Does it depend on the LastWriteTime of the source files? Does the LoadTime property of ProjectOptions affect the cache somehow? If so, what's the proper way to set it?

@dsyme How difficult would it be to add FSharpImplementationFileContents to ParseFileInProject? Would it be able to also detect changes in dependent files?

That's great news! It's not urgent but probably parsing a single file will bring a big performance gain, as currently Fable merges sources of all dependencies in a single file so parsing the whole project on every file save can get a bit slow.
I'll try to have a look at it myself after FableConf and will ask for pointers if necessary :smile:

Wow, it should be fixed.

@vasily-kirichenko go for it!

I personally think this would be super valuable for Fable.
In our usual projects we have many "fable base libraries" that we import via nuget source code packages. These libs never change during watch mode and are always imported at the top. So even if I change the very first line of my own code these libs are not touched. So a version of the FCS which only starts reching from that line already will be a big perf win.

Changing single file results with the following output:
webpack: Compiling...
Parsing ./EsExplorer.fsproj...
fable: Compiled src\App.fs
Hash: 37c448bdf0c57a51d758
Version: webpack 3.6.0
Time: 6317ms
        Asset     Size  Chunks                    Chunk Names
    bundle.js  2.31 MB       0  [emitted]  [big]  main
bundle.js.map  2.32 MB       0  [emitted]         main
 [250] (webpack)/hot nonrecursive ^\.\/log$ 170 bytes {0} [built]
 [255] ./src/App.fs 4.05 kB {0} [built]
    + 464 hidden modules
webpack: Compiled successfully.

Does it mean a single file has been type checked? Or it still checked the entire project?

Every time you see the Parsing XXX.fsproj... it means the whole project is being checked. I didn't find another way to get the FSharpImplementationFileContents which contains the AST :/

How to debug the compiler? So I have Fable.sln open in Rider. It builds successfully. How can I replace the compiler which is used in a project with the one just built and attach with the debugger to it? Or the dev flow is completely different?

I don't use the debugger, because when I compile Fable with Debug config I get ehem SO exceptions more often than not (it works if I target .NET Framework, I removed the .fsproj files for that recently but I can add them again).
My workflow with FAKE is usually to make the modifications in the code and then run QuickFableCompilerTest which compiles Fable and then uses it to compile src/tools/QuickTest.fs so I can check the output (src/tools/temp/QuickTest.js) or use printfn statements were necessary. With VS for Mac I have this dotnet-fable.fsproj.user file so I just need to press F5:
<?xml version="1.0" encoding="utf-8"?>
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <PropertyGroup Condition=" '$(RunConfiguration)' == 'Default' ">
    <StartAction>Project</StartAction>
    <StartArguments>npm-run rollup --port free</StartArguments>
    <StartWorkingDirectory>..\..\tools</StartWorkingDirectory>
    <ExternalConsole>true</ExternalConsole>
  </PropertyGroup>
</Project>

After some thinking about it I've come to conclusion that it's impossible to speed up this. This is how I see the current implementation:

A *.fs file changed and saved to disk.
The mailbox processor got a message which path field is set to the file https://github.com/fable-compiler/Fable/blob/master/src/dotnet/dotnet-fable/StateUtil.fs#L199
The MB calls StateUtil.updateState, which calls ParseAndCheckProject and pass the result to transformFile along with the file name https://github.com/fable-compiler/Fable/blob/master/src/dotnet/dotnet-fable/StateUtil.fs#L153 https://github.com/fable-compiler/Fable/blob/master/src/dotnet/dotnet-fable/StateUtil.fs#L75
It finds the related FSharpImplementationFileContents in checkedProject.AssemblyContents.ImplementationFiles and uses it to generate JavaScript from it https://github.com/fable-compiler/Fable/blob/master/src/dotnet/Fable.Compiler/FSharp2Fable.fs#L1328

Why the optimization won't work:

Instead of calling ParseAndCheckProject we call ParseAndCheckFileInProject, which does not compile files below this one.
We store FSharpImplementationFileContents (which is not a part of FSharpFileCheckResults yet) somewhere in a map by file name.
We transform the file impl content to js.
Now we have a broken project: if *.fs files below the current one are now non-compilable due to changes made in the current file (i.e. it's API has changed), then ParseAndCheckFileInProject won't fail like ParseAndCheckProject would, and we have inconsistent JavaScript code.

Does it make sense or I don't understand something?
/cc @dsyme

(AFAIK ParseAndCheckProject is smart and rechecks files starting and below the top changed file only)

Your concerns are right but I think we could force recompilation of dependent files (not all the files below the edited one, only those accessing any of its public members). Before Fable kept a dependency map for this reason, I removed it after the Webpack integration but we could bring it back.

(AFAIK ParseAndCheckProject is smart and rechecks files starting and below the top changed file only)

I was hoping the same, but when compiling a project with many files, like the Fulma docs, parsing the project takes more or less same (around 8s) either when I edit the last file (App.fs) or one in the top (Global.fs) 😕

I was hoping the same, but when compiling a project with many files, like the Fulma docs, parsing the project takes more or less same (around 8s) either when I edit the last file (App.fs) or one in the top (Global.fs)

@dsyme I think it's a bug in FCS?

I think we could force recompilation of dependent files (not all the files below the edited one, only those accessing any of its public members

If so, the optimization may work.
However, we should fix the ^^^ bug (if it's a bug) first and see what a perf improvement it gives us.

We have a nice tool in VFT repo for testing FCS in console. This is what it shows for FCS project itself.
First run
ParseAndCheckProject(FSharp.Compiler.Private.fsproj)...
Finished successfully in 00:00:58.8495450

Second run, no changes anywhere
cParseAndCheckProject(FSharp.Compiler.Private.fsproj)...
Finished successfully in 00:00:05.8005010

Third run, no changes anywhere (@dsyme it takes 5 seconds for nothing?..)
cParseAndCheckProject(FSharp.Compiler.Private.fsproj)...
Finished successfully in 00:00:05.4934968
Run after `service.fs` was changed (it's one of the last files in the proj file)

cParseAndCheckProject(FSharp.Compiler.Private.fsproj)...
Finished successfully in 00:00:11.1900123
So the caching seems to work.

Ah, FCS has a lot of errors, so the symbol suggester takes all the CPU time:

/cc @forki

why does it have "a lot of errors"?

@forki I think it's because the perf tool has outdated project option hardcoded into it.

AFAIK ParseAndCheckProject is smart and rechecks files starting and below the top changed file only

pure speculation following: I think @auduchinok said that FCS is adding a \n or something at the end of the parsed files. Maybe just maybe all out-of-date checks evaluate to false because of this!? can we somehow trace where it starts to compute?

@forki it doesn't add it to actual files, only to buffers being parsed, changing AST ranges. It checks timestamps of files to find a file to start.

so there must be something else going on.

@forki I don't think FCS behavior is wrong at all. It rechecks only the most upper changed file and all files below it. If we switch to ParseAndCheckFileInProject, it may not give us any perf improvements because we often change view or state of a Fable page, which are referenced in top level files like App.fs, which are at the bottom in fsproj, so we decide to recheck those top level files as well and call ParseAndCheckFileInProject for them, which looks good until we realize that FCS will check all the files above them. So, effectively, all the files starting from the changed one and down to the top level files will be rechecked anyway.

As I said above: many of my files are "fable base library" files. If it would never recheck these then it's already a great thing

It checks timestamps of files to find a file to start.

Against what? Against the LoadTime of FSharpProjectOptions? I didn't notice any difference by trying different values for it 😕 Or is the time when the Checker is created what is taken into account?

It keeps last write time for each checked file and compare it with the actual one to decide if a file should be rechecked.

@alfonsogarciacaro another question: does fable watch cache parsed fsproj? When I save fs file in watch mode it always prints "Parsing ./myProj.fsproj..." - even when I didn't touch that file

Thanks for the confirmation @vasily-kirichenko! I've always wondered how costly is checking the timestamp of files, specially for a big project, but I guess this is not the bottleneck. I'll try to run a profiling session in VS with Fulma docs to see if I find something.
@forki Parsing ./myProj.fsproj... is just the way Fable has to say it's running ParseAndCheckProject and thus parsing the whole project (even if the .fsproj remains untouched). Also, there's no actual Fable watch now, each JS client must take responsibility of watching the files and send requests to Fable. I don't recall the details right now but Webpack watcher is more advanced in the sense it can detect when multiple files have been saved at the same time and send less requests so ParseAndCheckProject is only called once. However, the watcher in fable-splitter is less sophisticated and you need to parse the whole project for every tiny change. This can certainly be improved.

This can certainly be improved.

https://github.com/fsharp/FSharp.Compiler.Service/pull/810
@dsyme could you please take a look?

#810 seems to work OK. I cherry picked the changes to VFT and look at FSharpCheckFileResults.ImplementationFiles in the debugger:

Before Fable kept a dependency map for this reason, I removed it after the Webpack integration but we could bring it back.

@alfonsogarciacaro could you bring this back and try to use ParseAndCheckFileInProject with FCS from #810?

I'll do! Thanks a lot for this @vasily-kirichenko!

Does it make watch mode faster?

@forki I don't know :) It may.

Well, after deciphering the needed commands to build FCS for netstandard, I created a Fable build to test @vasily-kirichenko PR, tested with a project with many files like Fulma docs and watch compilations went from 8 seconds to less than 1 sec (the Fable part) so, yeah, it's kind of an improvement :wink: :clap: :tada: :clap: ❤ (ping @mangelmaxime).
In my quick test I didn't see any problem, but I think @vasily-kirichenko's concerns are right (and probably I also need to tweak another part of the code) so for now I'll try to release it as an opt-in.
Thanks again for this @vasily-kirichenko! Just one quick question: what's the best way to deal with the version parameter in ParseAndCheckFileInProject? For now I'm just using always 0 and seems to work well.

so what do I need to do to get it?

@alfonsogarciacaro That's a good news :)
If I can help to test or something feel free to ping me.

@alfonsogarciacaro the version is an int value that represents a "version" of file. In VFT we pass hash code of Roslyn's Document.GetTextVersion() : VersionStamp https://github.com/dotnet/roslyn/blob/master/src/Workspaces/Core/Portable/Workspace/Solution/VersionStamp.cs. AFAIK it's just a fast unique "identifier" or version of a file content (it's fast compared to, say, calculating SHA-1).
However, as far as I can see, this version thing is not used in ParseAndCheckFileInProject (which uses parseAndCheckFileInProjectCache only https://github.com/fsharp/FSharp.Compiler.Service/blob/master/src/fsharp/vs/service.fs#L2312). It's used in a couple of other caches though. So, I think we are safe for now just passing zero to it. However, if it's easy to do, we can pass something like int <file_last_wirte_time>.Ticks.

However,

Ionide uses something TextDocument.version: float (which comes from VS Code itself?)
Other editors, like Emacs, use 0, see https://github.com/fsharp/FsAutoComplete/blob/master/src/FsAutoComplete/FsAutoComplete.Stdio.fs#L18

So, we are fine :)

@vasily-kirichenko TextDocument.version in VSCode is a version of an open document. VSCode increment it each time a "TextDocument" is edited (Documents not opened in the UI have no version and closing then re-opening a document reset the version)

Great, thanks for the confirmation! Ok, I've pushed Fable 1.3.0-beta-001 which includes @vasily-kirichenko's custom FCS build and seems to work. Please give it a try. Being at opt-in you'll have to set extra.optimizeWatch in the Fable JS client options. For example, for Webpack:
            {
                test: /\.fs(x|proj)?$/,
                use: {
                    loader: "fable-loader",
                    options: {
                        define: isProduction ? [] : ["DEBUG"],
                        extra: { optimizeWatch: true }
                    }
                }
            },

Incidentally, with this version if you don't set optimizeWatch, watch compilations are slower than before. I tried to keep the non-optimized code the same, so I'm not sure what's the issue. Maybe because the custom FCS build?

Have you built it in release mode?

Hmm. It works the same for me with and without the flag - about 8 seconds from editing a file to refresh the page.

Ok, that's interesting, the trick seems to be resetting LoadTime in the FSharpProjectOptions to DateTime.Now or not.

ParseAndCheckProject best results by setting  LoadTime to DateTime.Now everytime
ParseAndCheckFileInProject best results by NOT setting  LoadTime to DateTime.Now everytime

Yes, I built FCS with release mode :+1:

like https://github.com/SAFE-Stack/SAFE-BookStore/pull/185? Do not notice improvements with it

Hmm. It works the same for me with and without the flag - about 8 seconds from editing a file to refresh the page.

Weird, what do you see? Parsing XXX.fsproj or Parsing XXX.fs? (should be the second when using the extra.optimizeWatch flag) Has anyone else tried too? @forki @MangelMaxime

@alfonsogarciacaro

so it's parsing full project

yes, I see Parsing XXX.fsproj
About LoadTime, yes you should change it only if the project file itself has been changed.

Ah, sorry, I forgot to tell you! 🤦♂  There was an issue with latest Fable and fable-loader, so you need to update fable-loader too to 1.1.3, see https://github.com/SAFE-Stack/SAFE-BookStore/pull/186

λ yarn upgrade fable-loader
yarn upgrade v0.24.6
[1/4] Resolving packages...
[2/4] Fetching packages...
warning<EMAIL_ADDRESS>The platform "win32" is incompatible with this module.
info<EMAIL_ADDRESS>is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies...
[4/4] Rebuilding all packages...
success Saved lockfile.
success Saved 1 new dependency.
└─<EMAIL_ADDRESS>Done in 6.44s.

webpack: Compiling...
Parsing ./EsExplorer.fsproj...
fable: Compiled src\Errors\View.fs
Hash: fb5708a4bcf814162a18
Version: webpack 3.6.0
Time: 3868ms
                                   Asset      Size  Chunks                    Chunk Names
                               bundle.js   2.56 MB       0  [emitted]  [big]  main
    0.1755e05c83db05bb6189.hot-update.js   25.4 kB       0  [emitted]         main
    1755e05c83db05bb6189.hot-update.json  43 bytes          [emitted]
                           bundle.js.map   2.37 MB       0  [emitted]         main
0.1755e05c83db05bb6189.hot-update.js.map   7.46 kB       0  [emitted]         main
[./node_modules/webpack/hot ^\.\/log$] (webpack)/hot nonrecursive ^\.\/log$ 170 bytes {0} [built]
[./src/Errors/View.fs] ./src/Errors/View.fs 6.85 kB {0} [built]
    + 471 hidden modules
webpack: Compiled successfully.

I get this result:
webpack: Compiling...
Parsing ./src/Fulma/Elements/Button/View.fs...
fable: Compiled src\Fulma\Elements\Button\View.fs
Hash: 5d1e08c584c2fdd92895
Version: webpack 3.6.0
Time: 7248ms
        Asset       Size  Chunks                    Chunk Names
    bundle.js    3.19 MB       0  [emitted]  [big]  main
bundle.js.map    2.26 MB       0  [emitted]         main
   index.html  913 bytes          [emitted]
 [272] ../node_modules/webpack/hot nonrecursive ^\.\/log$ 170 bytes {0} [built]
 [334] ./src/Fulma/Elements/Button/View.fs 6.93 kB {0} [built]
    + 489 hidden modules
Child html-webpack-plugin for "index.html":
     1 asset
       4 modules
webpack: Compiled successfully.

it works on https://github.com/SAFE-Stack/SAFE-BookStore/pull/185!!!!

@forki how have you got it working?

you need latest fable loader + fable.compiler beta + dotnet-fable beta - then it works

@forki What do you mean by it works ?
I mean, I am doing the watch too but the compile time is still of 7-8 sec. Even, if we see Parsing ./src/Fulma/Elements/Button/View.fs...
Is it intended or I missed something about what we are testing here ?

it works means that refresh time for that login thing in bookstore was 6s and now is 2s

@forki I don't see any prerelease in https://github.com/SAFE-Stack/SAFE-BookStore/blob/master/paket.dependencies

https://github.com/SAFE-Stack/SAFE-BookStore/pull/185 <- it's not merged

source https://nuget.org/api/v2

storage:none

nuget Fable.PowerPack prerlease
nuget FSharp.Core
nuget Fulma prerelease
nuget Fable.Core prerelease
nuget Fable.Compiler prerelease
nuget Fable.Elmish.Browser  prerelease
nuget Fable.Elmish.Debugger prerelease
nuget Fable.Elmish.React prerelease
nuget Fable.Elmish.HMR prerelease

clitool dotnet-fable prelease

STORAGE: NONE
NUGET
  remote: https://www.nuget.org/api/v2
    dotnet-fable (1.2.4) - clitool: true
      Dotnet.ProjInfo (>= 0.7.4) - restriction: >= netcoreapp2.0
      Fable.Compiler (>= 1.2.4) - restriction: >= netcoreapp2.0
      Fable.Core (>= 1.2.4) - restriction: >= netcoreapp2.0
      FSharp.Core (>= 4.2.3) - restriction: >= netcoreapp2.0
      Microsoft.NETCore.App (>= 2.0) - restriction: >= netcoreapp2.0
    Dotnet.ProjInfo (0.7.4) - restriction: >= netcoreapp2.0
      FSharp.Core (>= 4.1.17) - restriction: >= netstandard1.6
      NETStandard.Library (>= 1.6.1) - restriction: >= netstandard1.6
    Fable.Compiler (1.2.4)
      Fable.Core (>= 1.2.4) - restriction: >= netstandard1.6
      FSharp.Compiler.Service (>= 14.0.2) - restriction: >= netstandard1.6
      FSharp.Core (>= 4.2.3) - restriction: >= netstandard1.6
      NETStandard.Library (>= 1.6.1) - restriction: >= netstandard1.6
      Newtonsoft.Json (>= 10.0.3) - restriction: >= netstandard1.6
      System.Collections.Immutable (>= 1.4) - restriction: >= netstandard1.6
      System.Reflection.Metadata (>= 1.5) - restriction: >= netstandard1.6
      System.Runtime.Loader (>= 4.3) - restriction: >= netstandard1.6
      System.ValueTuple (>= 4.4) - restriction: >= netstandard1.6
    Fable.Core (1.2.4)
      FSharp.Core (>= 4.2.3) - restriction: >= netstandard1.6
      NETStandard.Library (>= 1.6.1) - restriction: >= netstandard1.6
      System.ValueTuple (>= 4.4) - restriction: >= netstandard1.6
    Fable.Elmish (1.0.0-beta-1) - restriction: >= netstandard1.6
...

:(

λ .paket\paket update --force
Paket version 5.100.3
Resolving packages for group Main:
 - Fable.PowerPack 1.2.0
 - dotnet-fable 1.2.4
 - Fable.Compiler 1.2.4
 - Fable.Core 1.2.4
...

wtf?! why is it a paket issue now. arggghhhhhh

prerlease
prelease
typos - @vasily-kirichenko you are killing me

wow. paket should fail with a good error.

We can't. It's a feature. It's allowed to use custom prerelease tags. If
it's not exactly "prerelease" it will filter for that tag.
I admit it can be usability issue
Am 29.09.2017 17:43 schrieb "Vasily Kirichenko"<EMAIL_ADDRESS>
wow. paket should fail with a good error.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/fsharp/FSharp.Compiler.Service/issues/796#issuecomment-333162411,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADgND5mz6fC_ysv3iU4lwfLMtE8NLN-ks5snRAggaJpZM4PFg79
.

I think it's a rather big issue. We should think about it more.

Ah, forget to say, it finally works and very nicely :)

@alfonsogarciacaro Ok I tested it on my personal laptop.
With the new optimization it take between 3 and 3,5 seconds versus 8-9 seconds before. I think 80% of the time is due to webpack doing some stuff. I will try to see if I can reduce webpack time too (it's probably related to the assets and plugins used).
Great job :clap:

@alfonsogarciacaro what about fable-splitter? Can we get that to work as well?

Awesome, thanks everybody for checking. I will close this issue now as it should be fixed by #810, hopefully the PR is merged soon :) If you find any issue with the beta please report it to the Fable repo. But note the dependency tree is not implemented, so when optimizeWach is set and you change something that causes an error in another file, probably Webpack will just compile happily.
@forki The change is done in Fable daemon, so it should apply to any JS client, including fable-splitter (just make sure you pass the extra.optimizeWatch in the Fable options). When we add the dependency tree I'll guess it will require some works, as Webpack manages dependencies automatically but we'll have to do it for the fable-splitter by ourselves.

@alfonsogarciacaro Don just merged my PR and published FCS 15.0.1 https://github.com/Microsoft/visualfsharp/pull/3659#issuecomment-333613347, so you can try to update to it.
About dependency tree, it does make life a bit hard because outdated js code throws cryptic exceptions at runtime :(

Ah, great! Much better to use the package reference than a custom build :) I will push a new version and try to add the dependency tree too :+1:

Thank you :)


Fix to issue #1.

https://www.compart.com/en/unicode/category/Pe
https://www.compart.com/en/unicode/category/Ps

@christianp Do you accept this, and are there any further additions to this project?  I'd like to keep STACK in sync with this list!

Sorry, I completely forgot about this! I'm off work this week for half term, but I'll look at it when I'm back.

@sangwinc now I've looked at #1 again, I see I didn't add these to symbols.json. Instead, I added a file brackets.json, listing bracket characters that aren't normalised by NFKD to the standard parenthesis, curly brace, or square bracket characters. The only ones in this PR that weren't in my list were the light left/right tortoise shell ornaments, so I've added those.
For convenience, I'll repeat what I said in my last comment to #1: you can detect bracket characters by looking for character classes Ps and Po, normalise with NFKD, then apply the mapping from brackets.json if necessary.