v0.4.4. feedback: Error building Docker image

Thank you for the update v0.4.4 @Normalnoise .
you did not really mention if Multichain.storage is now still down, but here is my feedback for the actual state
ubi task not working:
[GIN] 2024/03/04 - 14:55:31 | 200 |  385.587333ms |   <IP_ADDRESS> | GET      "/api/v1/computing/cp"
time="2024-03-04 14:55:31.604" level=info msg="receive ubi task received: {ID:28241 Name:1000-0-7-17861 Type:0 ZkType:fil-c2-512M InputParam:https://download.fogmetalabs.com/ipfs/QmNr558fVVuw2G8vMZ74fvfR2itzLdH47QYwvULciYXBBH Signature:0x90d15e5ba418c9af929587d6f58e22d493deacf0ba1af27bb9a5e4503e2ffcb942c6af8760391679a6edf47ee3eedbfa2fac341100ce32dc7b843b618adb666501 Resource:0xc000b9ad40}" func=DoUbiTask file="cp_service.go:563"
time="2024-03-04 14:55:31.604" level=info msg="ubi task sign verifing, task_id: 28241, type: fil-c2-512M, verify: true" func=DoUbiTask file="cp_service.go:603"
time="2024-03-04 14:55:31.650" level=info msg="checkResourceAvailableForUbi: needCpu: 1, needMemory: 5.00, needStorage: 1.00" func=checkResourceAvailableForUbi file="cp_service.go:1302"
time="2024-03-04 14:55:31.650" level=info msg="checkResourceAvailableForUbi: remainingCpu: 4, remainingMemory: 13.00, remainingStorage: 353.00" func=checkResourceAvailableForUbi file="cp_service.go:1303"
[GIN] 2024/03/04 - 14:55:31 | 200 |   46.700585ms |   <IP_ADDRESS> | POST     "/api/v1/computing/cp/ubi"
time="2024-03-04 14:55:31.659" level=error msg="Failed creating ubi task job: Job.batch \"fil-c2-512m-28241\" is invalid: spec.template.spec.containers[0].image: Required value" func=func1 file="cp_service.go:845"

lagrange GPU renting ok ( "pacman" ) although with 400 error
time="2024-03-04 14:59:14.893" level=info msg="submitting job..." func=submitJob file="cp_service.go:127"
time="2024-03-04 14:59:15.119" level=info msg="uuid: 41d84b94-3063-46a9-b96c-d0241ae62b22, spaceName: pac-man, hardwareName: Nvidia 3090 · 4 vCPU · 8 GiB" func=DeploySpaceTask file="cp_service.go:1035"
time="2024-03-04 14:59:15.186" level=info msg="uploading file to bucket, objectName: jobs/4f91b4f7-c31e-49a6-921a-125a6a00a56a.json, filePath: /tmp/jobs/4f91b4f7-c31e-49a6-921a-125a6a00a56a.json" func=UploadFileToBucket file="storage_service.go:52"
time="2024-03-04 14:59:15.432" level=info msg="Download 41d84b94-3063-46a9-b96c-d0241ae62b22 successfully." func=BuildSpaceTaskImage file="buildspace.go:33"
time="2024-03-04 14:59:15.437" level=info msg="Deleted ingress ing-41d84b94-3063-46a9-b96c-d0241ae62b22 finished" func=deleteJob file="cp_service.go:1084"
time="2024-03-04 14:59:15.462" level=info msg="Deleted service svc-41d84b94-3063-46a9-b96c-d0241ae62b22 finished" func=deleteJob file="cp_service.go:1090"
time="2024-03-04 14:59:16.185" level=error msg="http status: 400 Bad Request, code:400, url:https://api.swanipfs.com/api/v2/oss_file/get_file_by_object_name?bucket_uid=878494a8-6ab7-4694-96ac-fc89a2afcbe1&object_name=jobs/4f91b4f7-c31e-49a6-921a-125a6a00a56a.json" func=HttpRequest file="restful.go:127"
time="2024-03-04 14:59:16.185" level=error msg="https://api.swanipfs.com/api/v2/oss_file/get_file_by_object_name?bucket_uid=878494a8-6ab7-4694-96ac-fc89a2afcbe1&object_name=jobs/4f91b4f7-c31e-49a6-921a-125a6a00a56a.json failed, status:error, message:invalid param value:record not found" func=HttpRequest file="restful.go:154"
time="2024-03-04 14:59:16.185" level=error msg="https://api.swanipfs.com/api/v2/oss_file/get_file_by_object_name?bucket_uid=878494a8-6ab7-4694-96ac-fc89a2afcbe1&object_name=jobs/4f91b4f7-c31e-49a6-921a-125a6a00a56a.json failed, status:error, message:invalid param value:record not found" func=HttpGet file="restful.go:64"
[GIN] 2024/03/04 - 14:59:16 | 200 |       21.53µs |   <IP_ADDRESS> | GET      "/api/v1/computing/host/info"
time="2024-03-04 14:59:17.173" level=error msg="http status: 400 Bad Request, code:400, url:https://api.swanipfs.com/api/v2/oss_file/get_file_by_object_name?bucket_uid=878494a8-6ab7-4694-96ac-fc89a2afcbe1&object_name=jobs" func=HttpRequest file="restful.go:127"
time="2024-03-04 14:59:17.173" level=error msg="https://api.swanipfs.com/api/v2/oss_file/get_file_by_object_name?bucket_uid=878494a8-6ab7-4694-96ac-fc89a2afcbe1&object_name=jobs failed, status:error, message:invalid param value:record not found" func=HttpRequest file="restful.go:154"
time="2024-03-04 14:59:17.173" level=error msg="https://api.swanipfs.com/api/v2/oss_file/get_file_by_object_name?bucket_uid=878494a8-6ab7-4694-96ac-fc89a2afcbe1&object_name=jobs failed, status:error, message:invalid param value:record not found" func=HttpGet file="restful.go:64"
time="2024-03-04 14:59:19.419" level=info msg="file name:1_4f91b4f7-c31e-49a6-921a-125a6a00a56a.json, chunk size:712" func=func1 file="file.go:248"
time="2024-03-04 14:59:21.472" level=info msg="Deleted deployment deploy-41d84b94-3063-46a9-b96c-d0241ae62b22 finished" func=deleteJob file="cp_service.go:1107"
time="2024-03-04 14:59:21.606" level=info msg="jobuuid: 2b133c35-5bfa-4a80-980b-3e0c5d4f2478 successfully submitted to IPFS" func=submitJob file="cp_service.go:155"
time="2024-03-04 14:59:21.869" level=info msg="submit job detail: {UUID:2b133c35-5bfa-4a80-980b-3e0c5d4f2478 Name:Job-2b133c35-5bfa-4a80-980b-3e0c5d4f2478 Status:submitted Duration:3600 JobSourceURI:https://api.lagrangedao.org/spaces/41d84b94-3063-46a9-b96c-d0241ae62b22 JobResultURI:https://7d67303d2964.acl.swanipfs.com/ipfs/QmcqDAA2d5QrE5FYPipZVqnUNTf8qVeWFVueeU5ToDeVsQ StorageSource:lagrange TaskUUID:a4fb3004-1dab-460d-8662-f58bc221241e CreatedAt:1709560754 UpdatedAt:1709560754 BuildLog:wss://log.bitstakehaven.com:8085/api/v1/computing/lagrange/spaces/log?space_id=41d84b94-3063-46a9-b96c-d0241ae62b22&type=build ContainerLog:wss://log.bitstakehaven.com:8085/api/v1/computing/lagrange/spaces/log?space_id=41d84b94-3063-46a9-b96c-d0241ae62b22&type=container}" func=ReceiveJob file="cp_service.go:122"
[GIN] 2024/03/04 - 14:59:21 | 200 |  7.500569152s |   <IP_ADDRESS> | POST     "/api/v1/computing/lagrange/jobs"

lagrange GPU renting with stablediffusion not possible:
error 400 and Error building Docker image: Error response from daemon: invalid reference format
time="2024-03-04 15:04:42.396" level=info msg="Job received Data: {UUID:88911b11-5f3a-46a8-bb70-5efa55afdbf4 Name:Job-88911b11-5f3a-46a8-bb70-5efa55afdbf4 Status:Submitted Duration:3600 JobSourceURI:https://api.lagrangedao.org/spaces/201d7532-8284-4ebd-b348-ff5ce862beda JobResultURI: StorageSource:lagrange TaskUUID:7db9916e-397f-493d-b20f-0e5688042897 CreatedAt:1709561082 UpdatedAt:1709561082 BuildLog: ContainerLog:}" func=ReceiveJob file="cp_service.go:79"
time="2024-03-04 15:04:42.899" level=info msg="checkResourceAvailableForSpace: needCpu: 8, needMemory: 16.00, needStorage: 20.00" func=checkResourceAvailableForSpace file="cp_service.go:1227"
time="2024-03-04 15:04:42.899" level=info msg="checkResourceAvailableForSpace: remainingCpu: 4, remainingMemory: 13.00, remainingStorage: 353.00" func=checkResourceAvailableForSpace file="cp_service.go:1228"
time="2024-03-04 15:04:42.899" level=info msg="checkResourceAvailableForSpace: needCpu: 8, needMemory: 16.00, needStorage: 20.00" func=checkResourceAvailableForSpace file="cp_service.go:1227"
time="2024-03-04 15:04:42.899" level=info msg="checkResourceAvailableForSpace: remainingCpu: 23, remainingMemory: 65.00, remainingStorage: 1593.00" func=checkResourceAvailableForSpace file="cp_service.go:1228"
time="2024-03-04 15:04:42.899" level=info msg="gpuName: NVIDIA-4090, nodeGpu: map[:0 kubernetes.io/os:0], nodeGpuSummary: map[swan3:map[NVIDIA-4090:1] swan7:map[NVIDIA-3090:1]]" func=checkResourceAvailableForSpace file="cp_service.go:1235"
time="2024-03-04 15:04:42.900" level=info msg="submitting job..." func=submitJob file="cp_service.go:127"
time="2024-03-04 15:04:42.900" level=info msg="uploading file to bucket, objectName: jobs/bfbd05a0-eff5-4275-a0db-7adc9d9bab66.json, filePath: /tmp/jobs/bfbd05a0-eff5-4275-a0db-7adc9d9bab66.json" func=UploadFileToBucket file="storage_service.go:52"
time="2024-03-04 15:04:43.126" level=info msg="uuid: 201d7532-8284-4ebd-b348-ff5ce862beda, spaceName: myDiffusion, hardwareName: Nvidia 4090 · 8 vCPU · 16 GiB" func=DeploySpaceTask file="cp_service.go:1035"
time="2024-03-04 15:04:43.425" level=info msg="Download 201d7532-8284-4ebd-b348-ff5ce862beda successfully." func=BuildSpaceTaskImage file="buildspace.go:33"
time="2024-03-04 15:04:43.540" level=info msg="Download 201d7532-8284-4ebd-b348-ff5ce862beda successfully." func=BuildSpaceTaskImage file="buildspace.go:33"
time="2024-03-04 15:04:43.654" level=info msg="Download 201d7532-8284-4ebd-b348-ff5ce862beda successfully." func=BuildSpaceTaskImage file="buildspace.go:33"
2024/03/04 15:04:43 Image path: build/0x7B0CEe1939a4AdA062EC79f4862a42C1F47B1806/spaces/myDiffusion
time="2024-03-04 15:04:43.655" level=error msg="Error building Docker image: Error response from daemon: invalid reference format" func=BuildImagesByDockerfile file="buildspace.go:80"
time="2024-03-04 15:04:43.655" level=info msg="Failed to extract exposed port: unable to open Dockerfile: open : no such file or directory" func=DockerfileToK8s file="deploy.go:97"
time="2024-03-04 15:04:43.921" level=error msg="http status: 400 Bad Request, code:400, url:https://api.swanipfs.com/api/v2/oss_file/get_file_by_object_name?bucket_uid=878494a8-6ab7-4694-96ac-fc89a2afcbe1&object_name=jobs/bfbd05a0-eff5-4275-a0db-7adc9d9bab66.json" func=HttpRequest file="restful.go:127"
time="2024-03-04 15:04:43.921" level=error msg="https://api.swanipfs.com/api/v2/oss_file/get_file_by_object_name?bucket_uid=878494a8-6ab7-4694-96ac-fc89a2afcbe1&object_name=jobs/bfbd05a0-eff5-4275-a0db-7adc9d9bab66.json failed, status:error, message:invalid param value:record not found" func=HttpRequest file="restful.go:154"
time="2024-03-04 15:04:43.921" level=error msg="https://api.swanipfs.com/api/v2/oss_file/get_file_by_object_name?bucket_uid=878494a8-6ab7-4694-96ac-fc89a2afcbe1&object_name=jobs/bfbd05a0-eff5-4275-a0db-7adc9d9bab66.json failed, status:error, message:invalid param value:record not found" func=HttpGet file="restful.go:64"
time="2024-03-04 15:04:46.064" level=info msg="file name:1_bfbd05a0-eff5-4275-a0db-7adc9d9bab66.json, chunk size:712" func=func1 file="file.go:248"
time="2024-03-04 15:04:48.209" level=info msg="jobuuid: 88911b11-5f3a-46a8-bb70-5efa55afdbf4 successfully submitted to IPFS" func=submitJob file="cp_service.go:155"
time="2024-03-04 15:04:48.471" level=info msg="submit job detail: {UUID:88911b11-5f3a-46a8-bb70-5efa55afdbf4 Name:Job-88911b11-5f3a-46a8-bb70-5efa55afdbf4 Status:submitted Duration:3600 JobSourceURI:https://api.lagrangedao.org/spaces/201d7532-8284-4ebd-b348-ff5ce862beda JobResultURI:https://7d67303d2964.acl.swanipfs.com/ipfs/QmWA4aFGaH7FrBA9qMNgcX2ZA9rvW7ywH6uAVuDTPZmYeB StorageSource:lagrange TaskUUID:7db9916e-397f-493d-b20f-0e5688042897 CreatedAt:1709561082 UpdatedAt:1709561082 BuildLog:wss://log.bitstakehaven.com:8085/api/v1/computing/lagrange/spaces/log?space_id=201d7532-8284-4ebd-b348-ff5ce862beda&type=build ContainerLog:wss://log.bitstakehaven.com:8085/api/v1/computing/lagrange/spaces/log?space_id=201d7532-8284-4ebd-b348-ff5ce862beda&type=container}" func=ReceiveJob file="cp_service.go:122"
[GIN] 2024/03/04 - 15:04:48 | 200 |  6.075421328s |   <IP_ADDRESS> | POST     "/api/v1/computing/lagrange/jobs"

init still borken
rm private_key
computing-provider init --ownerAddress 0xfe017Ff8F0C7349845Ab52E58FcA96143f2c4981 --beneficiaryAddress 0x269EBeee083CE6f70486a67dC8036A889bF322A9
Contract deployed! Address: 0xa1468F15CF82f939ed0A96b5A89EFCf92021f1da
Transaction hash: 0x0d0ab9afe5caab575eb02c84bfd8698bc45c466ac940f5915c9a5bb22370b20c
Error: register cp to ubi hub failed, error: cpAccount client create GetCpAccountInfo tx error: no contract code at given address

collateral still inaccurate
HUB info for beneficiary: 8.65 sETH
HUB info for owner: 1.49 sETH
computing-provider info: Collateral(SWAN-ETH):	0.95000

Solution：

ubi task not working:

update resource-exporter,This component gets the CPU model of the machine (amd or intel) and pulls different ubi-task images.

kubectl delete ds -n kube-system resource-exporter-ds
docker rmi -f filswan/resource-exporter:v11.2.5
Reinstall resource-exporter. Refer to:  [install-the-hardware-resource-exporter]

error 400 and Error building Docker image:

Check whether the space has a dockerfile file, or check whether the dockerfile format is correct
Check whether the user running cp can run docker commands normally

init still borken:

There is a delay after the contract is deployed and synchronized to the chain. The server has a retry mechanism to obtain the cp information in the contract. If you can receive the ubi-task task, you can ignore this error.

Solution：

ubi task not working:

update resource-exporter,This component gets the CPU model of the machine (amd or intel) and pulls different ubi-task images.

kubectl delete ds -n kube-system resource-exporter-ds
docker rmi -f filswan/resource-exporter:v11.2.5
Reinstall resource-exporter. Refer to:  [install-the-hardware-resource-exporter]

error 400 and Error building Docker image:

Check whether the space has a dockerfile file, or check whether the dockerfile format is correct
Check whether the user running cp can run docker commands normally

init still borken:

There is a delay after the contract is deployed and synchronized to the chain. The server has a retry mechanism to obtain the cp information in the contract. If you can receive the ubi-task task, you can ignore this error.
@ThomasBlock

thank you for the reply. i dont't want to be rude. but it always looks like you reply to me with "standard phrases" and dont really look into the problem. its also very slow. it looks like this whole team is really uninterested, or overworked!
i am running this since several weeks. of course i know that docker runs.
i updated  resource-exporter on all nodes. ubi task is still broken
time="2024-03-05 09:55:31.728" level=info msg="receive ubi task received: {ID:29710 Name:1000-0-7-19330 Type:0 ZkType:fil-c2-512M InputParam:https://download.fogmetalabs.com/ipfs/QmT7Q8AWBziSbWY8XMZt964tqP48QZncMEvADUhkXRbXPP Signature:0x130502f6fb2adf3d88ab8c3751274b44388b6b959085e86608c4594ff50abcfd13d0b52ccd0f6f212a208f095bd148960399673f34435137b9851b782cc96dd701 Resource:0xc000b078c0}" func=DoUbiTask file="cp_service.go:563"
time="2024-03-05 09:55:31.729" level=info msg="ubi task sign verifing, task_id: 29710, type: fil-c2-512M, verify: true" func=DoUbiTask file="cp_service.go:603"
time="2024-03-05 09:55:31.762" level=info msg="checkResourceAvailableForUbi: needCpu: 1, needMemory: 5.00, needStorage: 1.00" func=checkResourceAvailableForUbi file="cp_service.go:1302"
time="2024-03-05 09:55:31.762" level=info msg="checkResourceAvailableForUbi: remainingCpu: 4, remainingMemory: 13.00, remainingStorage: 353.00" func=checkResourceAvailableForUbi file="cp_service.go:1303"
[GIN] 2024/03/05 - 09:55:31 | 200 |   33.692995ms |   <IP_ADDRESS> | POST     "/api/v1/computing/cp/ubi"
time="2024-03-05 09:55:31.770" level=error msg="Failed creating ubi task job: Job.batch \"fil-c2-512m-29710\" is invalid: spec.template.spec.containers[0].image: Required value" func=func1 file="cp_service.go:845"

invalid reference format - i posted you all the logs - please tell me if the space itself is broken. i think its okay as i ran it in the past. can you resommend a "good" space so that we can really test gpu?

time="2024-03-05 09:55:31.770" level=error msg="Failed creating ubi task job: Job.batch "fil-c2-512m-29710" is invalid: spec.template.spec.containers[0].image: Required value" func=func1 file="cp_service.go:845"

@ThomasBlock this issue should be your CP is still using the old version's resource-exporter, so you need re-install it following the steps:

kubectl delete ds -n kube-system resource-exporter-ds
docker rmi -f filswan/resource-exporter:v11.2.5
Reinstall resource-exporter. Refer to: install-the-hardware-resource-exporter

i updated resource-exporter on all nodes. ubi task is still broken

please give me your K8s cluster information, including how many nodes in your K8s cluster, master and every node information(like GPU type, cpu type)

Please execute the following command in your k8s cluster and post the log:
kubectl logs --tail=1 -fn kube-system resource-exporter-ds-xxx

i updated resource-exporter on all nodes. ubi task is still broken

please give me your K8s cluster information, including how many nodes in your K8s cluster, master and every node information(like GPU type, cpu type)

kubectl get nodes
NAME    STATUS   ROLES           AGE   VERSION
swan1   Ready    control-plane   47d   v1.28.2
swan3   Ready              42d   v1.28.2
swan5   Ready              40d   v1.28.2
swan6   Ready              16d   v1.28.2
swan7   Ready              15d   v1.28.2
swan1 = cpu ryzen7
swan3 = ryzen7 + 4090
swan5,swan6 = cpu Xeon
swan7 = ryzen7 + 3090

Please execute the following command in your k8s cluster and post the log:
kubectl logs --tail=1 -fn kube-system resource-exporter-ds-xxx

kubectl logs --tail=1 -fn kube-system resource-exporter-ds-6lssn
{"gpu":{"driver_version":"535.161.07","cuda_version":"12020","attached_gpus":1,"details":[{"product_name":"NVIDIA 4090","fb_memory_usage":{"total":"24564 MiB","used":"347 MiB","free":"24216 MiB"},"bar1_memory_usage":{"total":"32768 MiB","used":"2 MiB","free":"32765 MiB"}}]},"cpu_name":"AMD"}

kubectl logs --tail=1 -fn kube-system resource-exporter-ds-9lcxf
If the node has a GPU, this error can be ignored. ERROR:: unable to initialize NVML: 12

 kubectl logs --tail=1 -fn kube-system resource-exporter-ds-c48zq
{"gpu":{"driver_version":"535.161.07","cuda_version":"12020","attached_gpus":1,"details":[{"product_name":"NVIDIA 3090","fb_memory_usage":{"total":"24576 MiB","used":"317 MiB","free":"24258 MiB"},"bar1_memory_usage":{"total":"256 MiB","used":"2 MiB","free":"253 MiB"}}]},"cpu_name":"AMD"}

kubectl logs --tail=1 -fn kube-system resource-exporter-ds-k8xfq
If the node has a GPU, this error can be ignored. ERROR:: unable to initialize NVML: 12

kubectl logs --tail=1 -fn kube-system resource-exporter-ds-rnwxz
If the node has a GPU, this error can be ignored. ERROR:: unable to initialize NVML: 12

The resource-exporter pod of a node on your cluster cannot obtain resource information and reports an error. You can set the RUST_GPU_TOOLS_CUSTOM_GPU content to 3090 or 4090 in the fil-c2.env file to let k8s schedule it to these normal nodes.

I fixed the situation where resource-exporter could not be retrieved without nvidia-plugin installed. I would like to trouble you to upgrade the version to filswan/resource-exporter:v11.2.6 again.

I fixed the situation where resource-exporter could not be retrieved without nvidia-plugin installed. I would like to trouble you to upgrade the version to filswan/resource-exporter:v11.2.6 again.

kubectl delete ds -n kube-system resource-exporter-ds
daemonset.apps "resource-exporter-ds" deleted

5x
ctr -n k8s.io images remove docker.io/filswan/resource-exporter:v11.2.5

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: DaemonSet
metadata:
  namespace: kube-system
  name: resource-exporter-ds
  labels:
    app: resource-exporter
spec:
  selector:
    matchLabels:
      app: resource-exporter
  template:
    metadata:
      labels:
        app: resource-exporter
    spec:
      containers:
      - name: resource-exporter
        image: filswan/resource-exporter:v11.2.6
        imagePullPolicy: IfNotPresent
EOF

kubectl logs --tail=1 -fn kube-system resource-exporter-ds-5r2rv
{"gpu":{"driver_version":"535.161.07","cuda_version":"12020","attached_gpus":1,"details":[{"product_name":"NVIDIA 3090","fb_memory_usage":{"total":"24576 MiB","used":"320 MiB","free":"24255 MiB"},"bar1_memory_usage":{"total":"256 MiB","used":"3 MiB","free":"252 MiB"}}]},"cpu_name":"AMD"}

kubectl logs --tail=1 -fn kube-system resource-exporter-ds-jdpn4
{"gpu":{"driver_version":"535.161.07","cuda_version":"12020","attached_gpus":1,"details":[{"product_name":"NVIDIA 4090","fb_memory_usage":{"total":"24564 MiB","used":"347 MiB","free":"24216 MiB"},"bar1_memory_usage":{"total":"32768 MiB","used":"2 MiB","free":"32765 MiB"}}]},"cpu_name":"AMD"}

kubectl logs --tail=1 -fn kube-system resource-exporter-ds-9dg2z
{"gpu":{"driver_version":"","cuda_version":"","attached_gpus":0,"details":null},"cpu_name":""}
The node not found nvm libnvidia, if the node does not have a GPU, this error can be ignored.

kubectl logs --tail=1 -fn kube-system resource-exporter-ds-ljzbs
{"gpu":{"driver_version":"","cuda_version":"","attached_gpus":0,"details":null},"cpu_name":"INTEL"}
The node not found nvm libnvidia, if the node does not have a GPU, this error can be ignored.

kubectl logs --tail=1 -fn kube-system resource-exporter-ds-x2jqq
{"gpu":{"driver_version":"","cuda_version":"","attached_gpus":0,"details":null},"cpu_name":""}

some dont report a cpu. but this might make sense: i checked, some of them are virtual amchines with "generic" proxmox type. Some are type "host", here we can see "cpu_name":"INTEL" and "cpu_name":"AMD"

The resource-exporter pod of a node on your cluster cannot obtain resource information and reports an error. You can set the RUST_GPU_TOOLS_CUSTOM_GPU content to 3090 or 4090 in the fil-c2.env file to let k8s schedule it to these normal nodes.

ok thank you i did this. will this help me to get 32 G ubi tasks? until now i only receive 512M tasks..
RUST_GPU_TOOLS_CUSTOM_GPU="NVIDIA GeForce RTX 3090:10496,NVIDIA GeForce RTX 4090:16384"

There are only 512M tasks now, and the 32G tasks are gone now, but there will be more in the future.

I fixed the situation where resource-exporter could not be retrieved without nvidia-plugin installed. I would like to trouble you to upgrade the version to filswan/resource-exporter:v11.2.6 again.

kubectl delete ds -n kube-system resource-exporter-ds
daemonset.apps "resource-exporter-ds" deleted

5x
ctr -n k8s.io images remove docker.io/filswan/resource-exporter:v11.2.5

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: DaemonSet
metadata:
  namespace: kube-system
  name: resource-exporter-ds
  labels:
    app: resource-exporter
spec:
  selector:
    matchLabels:
      app: resource-exporter
  template:
    metadata:
      labels:
        app: resource-exporter
    spec:
      containers:
      - name: resource-exporter
        image: filswan/resource-exporter:v11.2.6
        imagePullPolicy: IfNotPresent
EOF

kubectl logs --tail=1 -fn kube-system resource-exporter-ds-5r2rv
{"gpu":{"driver_version":"535.161.07","cuda_version":"12020","attached_gpus":1,"details":[{"product_name":"NVIDIA 3090","fb_memory_usage":{"total":"24576 MiB","used":"320 MiB","free":"24255 MiB"},"bar1_memory_usage":{"total":"256 MiB","used":"3 MiB","free":"252 MiB"}}]},"cpu_name":"AMD"}

kubectl logs --tail=1 -fn kube-system resource-exporter-ds-jdpn4
{"gpu":{"driver_version":"535.161.07","cuda_version":"12020","attached_gpus":1,"details":[{"product_name":"NVIDIA 4090","fb_memory_usage":{"total":"24564 MiB","used":"347 MiB","free":"24216 MiB"},"bar1_memory_usage":{"total":"32768 MiB","used":"2 MiB","free":"32765 MiB"}}]},"cpu_name":"AMD"}

kubectl logs --tail=1 -fn kube-system resource-exporter-ds-9dg2z
{"gpu":{"driver_version":"","cuda_version":"","attached_gpus":0,"details":null},"cpu_name":""}
The node not found nvm libnvidia, if the node does not have a GPU, this error can be ignored.

kubectl logs --tail=1 -fn kube-system resource-exporter-ds-ljzbs
{"gpu":{"driver_version":"","cuda_version":"","attached_gpus":0,"details":null},"cpu_name":"INTEL"}
The node not found nvm libnvidia, if the node does not have a GPU, this error can be ignored.

kubectl logs --tail=1 -fn kube-system resource-exporter-ds-x2jqq
{"gpu":{"driver_version":"","cuda_version":"","attached_gpus":0,"details":null},"cpu_name":""}

some dont report a cpu. but this might make sense: i checked, some of them are virtual amchines with "generic" proxmox type. Some are type "host", here we can see "cpu_name":"INTEL" and "cpu_name":"AMD"

ubi-task needs to pull different docker images according to different CPU architectures. Currently, it only supports intel and amd. Other architectures are not supported for the time being.

I fixed the situation where resource-exporter could not be retrieved without nvidia-plugin installed. I would like to trouble you to upgrade the version to filswan/resource-exporter:v11.2.6 again.

kubectl delete ds -n kube-system resource-exporter-ds
daemonset.apps "resource-exporter-ds" deleted

5x
ctr -n k8s.io images remove docker.io/filswan/resource-exporter:v11.2.5

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: DaemonSet
metadata:
  namespace: kube-system
  name: resource-exporter-ds
  labels:
    app: resource-exporter
spec:
  selector:
    matchLabels:
      app: resource-exporter
  template:
    metadata:
      labels:
        app: resource-exporter
    spec:
      containers:
      - name: resource-exporter
        image: filswan/resource-exporter:v11.2.6
        imagePullPolicy: IfNotPresent
EOF

kubectl logs --tail=1 -fn kube-system resource-exporter-ds-5r2rv
{"gpu":{"driver_version":"535.161.07","cuda_version":"12020","attached_gpus":1,"details":[{"product_name":"NVIDIA 3090","fb_memory_usage":{"total":"24576 MiB","used":"320 MiB","free":"24255 MiB"},"bar1_memory_usage":{"total":"256 MiB","used":"3 MiB","free":"252 MiB"}}]},"cpu_name":"AMD"}

kubectl logs --tail=1 -fn kube-system resource-exporter-ds-jdpn4
{"gpu":{"driver_version":"535.161.07","cuda_version":"12020","attached_gpus":1,"details":[{"product_name":"NVIDIA 4090","fb_memory_usage":{"total":"24564 MiB","used":"347 MiB","free":"24216 MiB"},"bar1_memory_usage":{"total":"32768 MiB","used":"2 MiB","free":"32765 MiB"}}]},"cpu_name":"AMD"}

kubectl logs --tail=1 -fn kube-system resource-exporter-ds-9dg2z
{"gpu":{"driver_version":"","cuda_version":"","attached_gpus":0,"details":null},"cpu_name":""}
The node not found nvm libnvidia, if the node does not have a GPU, this error can be ignored.

kubectl logs --tail=1 -fn kube-system resource-exporter-ds-ljzbs
{"gpu":{"driver_version":"","cuda_version":"","attached_gpus":0,"details":null},"cpu_name":"INTEL"}
The node not found nvm libnvidia, if the node does not have a GPU, this error can be ignored.

kubectl logs --tail=1 -fn kube-system resource-exporter-ds-x2jqq
{"gpu":{"driver_version":"","cuda_version":"","attached_gpus":0,"details":null},"cpu_name":""}

some dont report a cpu. but this might make sense: i checked, some of them are virtual amchines with "generic" proxmox type. Some are type "host", here we can see "cpu_name":"INTEL" and "cpu_name":"AMD"

ubi-task needs to pull different docker images according to different CPU architectures. Currently, it only supports intel and amd. Other architectures are not supported for the time being.

okay fine. we now clarified that VMs should use a "host" processor type.
any further comments on my setup?
my ubi tasks are working now. proofs are submitted onchain. But in the last week there have been no payouts. for me and other users.

if you have not received the rewards, maybe there are some reasons:

If you submit the proof and it takes more than 10 minutes, you will not get the reward.
If your task fails the check or fails to submit three times in a row, you will be penalized for not receiving the task for the next eight hours


Changed the way batchSize is limiting the search results of this quer…

Changed the way batchSize is limiting the search results of this query making it similar to the JPA implementation.

Understandable.  Yet not easy to make this pretty.  The suggestion made by Steven @smcvb does not feel like the ultimate solution.
Personally I was thinking about an enum with limit, top, numrows as query limit types and to hide the actual syntax of the limiting injecting it like in the following example.
String sql = "SELECT " + schema.optionalTopLimit + trackedEventFields() + " FROM " + schema.domainEventTable() +
" WHERE (" + schema.globalIndexColumn() + " > ? ) schema.optionalLimit";
I think both Steven as mine solution is not the prettiest thing I have ever seen, but I also consider this an series issue and would like to see it fixed.
I also found another source with a comparison of limit clauses for different vendors: https://en.wikibooks.org/wiki/SQL_Dialects_Reference/Select_queries/Limiting_results_of_select_query.
I am willing to make a PR this week for either Stevens or my solution. Let me know how I can contribute?

An alternative implementation has been provided in f457246c7baedeaa14b10d0edf3f9bf3d3198d8c. Instead of relying on database-specific language constructs, this one simply executes an extra query when a batch comes up empty. This extra query only checks primary keys, so it is fairly cheap to execute.


Add Tracer#joinOrContinue utility method

The name and the implementation are a matter of discussion.

@rossabaker @zmccoy
Linking an existing span to the external one is a complicated task. Basically, we need to link a new span right after we join the external one. This leads us to an unfortunate case when the extra details must be passed (span name, etc).
I prefer to keep this case outside of the current scope.
Meanwhile, I did the following:

Add joinOrContinue method
Add joinOrRoot method

These two utility methods should cover quite a lot of use cases. Also, I've provided several examples to simplify the adoption.


[Potential Bug] TypeError: ImageClassifier: SwinTransformer: init() got an unexpected keyword argument 'stage_cfg'

I tried to run the following command:
python tools/train.py configs/swin_transformer/swin_large_384_imagenet.py
I got the following error:

Traceback (most recent call last):
  File "tools/train.py", line 156, in <module>
    main()
  File "tools/train.py", line 128, in main
    model = build_classifier(cfg.model)
  File "mmclassification/mmcls/models/builder.py", line 34, in build_classifier
    return CLASSIFIERS.build(cfg)
  File "lib/python3.7/site-packages/mmcv/utils/registry.py", line 210, in build
    return self.build_func(*args, **kwargs, registry=self)
  File "lib/python3.7/site-packages/mmcv/cnn/builder.py", line 26, in build_model_from_cfg
    return build_from_cfg(cfg, registry, default_args)
  File "lib/python3.7/site-packages/mmcv/utils/registry.py", line 54, in build_from_cfg
    raise type(e)(f'{obj_cls.__name__}: {e}')
TypeError: ImageClassifier: SwinTransformer: __init__() got an unexpected keyword argument 'stage_cfg'

My mmcv version is 1.3.9 and mmcls version is 0.13.0.
I have found one some what similar issue in one of the mmdetection repo here.

I'm sorry about that. We will fix it soon.
Please modify the 9th line of config file:
https://github.com/open-mmlab/mmclassification/blob/1a7cebe4b90cf72e730a7066267b75561864e811/configs/_base_/models/swin_transformer/large_384.py#L4-L9
to
stage_cfgs=dict(block_cfgs=dict(window_size=12))),

Should be solved by #355

Thanks a lot for the help !


Passing a relative path

It would be very nice because I guess I'm not the only one, first, create the script on my PC, and second, push the script to a server.
So this:
path = r'BDMV\STREAM\00001.m2ts'
ap.VideoSource(infile=path, trimlist="[None,-48]")

Instead of:
path = r'F:\MY\FULL\PATH\BDMV\STREAM\00001.m2ts'
ap.VideoSource(infile=path, trimlist="[None,-48]")

Added in https://github.com/begna112/bvsfunc/commit/ffadbf97c168b24bdb5d15d1450319558c53bd94


Improve Readability of the Website by Improving Contrast and Fonts

Unlike the previous version of CodeIgniter's website, the current website does not provide enough contrast to improve readability. Also, the choice of font or font weight needs to be improved as well.
As seen in the following screenshot, the thinness of the font weight does not provide enough contrast and makes it hard for the text to be read.

Raleway's Extra Light is literally extra light, I suggest you refrain from using it, rather use either Light or Regular. Better yet, choose a different font.

This version of the website has been live for the past few years and this is the first complaint we have received about the font&colors etc.
Upon your comment now, I have just visited the page trying it in several browsers (Firefox, Chrome, Edge and Opera) but I haven't seen such a color scheme that you see and also the font-weight is not so thin, even though we have it in our list in the css file. Because we also have normal and bold font-weights there (font-weight: 400 and 700). Also the colors of the texts are not grey. We use #252525 and #111111 for dark colors which are fair, I believe.
Thus, to investigate further, may I kindly ask which browser you are using or even the OS that you're using? Is our website the first website you have had such a problem with?
Thanks

Hi @demyr17 I'm using Google Chrome version 114.0.5735.134 to view the website. Also, I'm using Windows 11 22H2 wiht build number 22621.1848. Hope this helps.
It's been a while since I used PHP and CodeIgniter, the last version of CodeIgniter I used was 3 during which CodeIgniter 4 was in alpha stage.

macOS 12.6.6
Google Chrome 114.0.5735.133

Hi @kenjis
Unfortunately, we're not seeing the same overall appearance.

Chrome
Version 114.0.5735.134 (Official Build) (64-bit)

Firefox
Version  114.0.2 (32-bit)

Edge
Version 114.0.1823.58 (Official build) (64-bit)

@justinefication I am not sure why the text is so thin in your environment.

@kenjis so do I. Tried clearing browser cache and cookies as well as clearing my font cache. But to no avail. Tried the website on the following browsers:
Microsoft Edge
Version 114.0.1823.58 (Official build) (64-bit)
Google Chrome
Version 114.0.5735.198 (Official Build) (64-bit)
Firefox
Version 114.0.2 (64-bit)
Both Chromium-based browser rendered the page as reported. However, Firefox renders it just fine. Anyhow, thanks for the help.

Thanks for the report and ensuing investigation! It sounds like this is probably an environment or OS settings issue. Safe to close?

Hi @MGatner , It's nice to see you here again.
@justinefication , When I check our css file I see that we have Raleway ExtraLight - font-weight: 200 in our font list but we don't use it anywhere. We have font-weight:600 that are used in following lines even though we don't have font-weight: 600 in our list. We have font-weight:700. So, I will fix it. But I guess it will not solve your problem because in the end, the default weight we use for body is 400.

Hi @MGatner sorry for responding late. Thank you for taking the time to look into the issue. Yes it is safe to close this.

@demyr17 that's cool. I'll still look into it in my environment when I have the opportunity. Thank you so much.


Add licence

Based on cucumber/docs.cucumber.io#304

Not sure how to fix the CI failure...

Hmmm, looks like circleci is not able to checkout the code. It's not had a new build for a while, so possibly something changed when 2FA was required? @mattwynne you set this up originally, are there any pointers you could give to me or @ahhbristow about getting this running again?

I think the problem is that the project is using Circle 1.0 config format which is now turned off.

I've had a crack at setting up a Circle 2.0 build in #42. If that works we can rebase this on top and it should pass fine.


STARTTLS not supported and the user requires enforcement

I am using Gmail to send and receive emails, and using the SMTP credentials from my Postal setup.
Everything was working fine, until I started getting this error:

STARTTLS not supported and the user requires enforcement., code: 500

At first I thought it was nothing, then the 3rd email bounced back... 3 emails send to 3 different companies, all returned this error.
I went into my Gmail SMTP settings (Gmail > Settings > Accounts & Imports > Edit info per email), then you get a popup window where you have to fill in your SMTP info.
Well, my SMTP is not connecting anymore, and I keep getting the same error listed above.
My SMTP was working fine with TLS, port 25 in Gmail, not anymore.
Using the unsecure option with port 25, is however working and delivering my emails.

Okay, some good news (and still some bad news):
Bad news: TLS with port 25 is not working anymore in Gmail (Did Gmaiil change their policy?)
Good news: you can use unsecure in Gmail when configuring your email, instead of TLS and SSL. Postal will stil encrypt your emails and deliver it to your recipients. They won't see an ugly "This email is not encrypted" label in Gmail (yeey for Postal, 1-0)

Firstly I will say this wasn't what Postal was designed for so its possible this will never be free of problems.
Did you give an SSL certificate to the smtp_server config in your postal.yml?

It may not have been designed for this purpose but have been using it for this, for a while now and it saved me some money.
I just route emails to my Gmail and send emails from my Gmail using my company email address.
I will check smtp_server and let you know ASAP.


[Q]not a bug, ask for solution

when I used deepspeed for distributed training, I find it cost me a lot of time on forward_microstep and backward_microstep. Is there any solution to improve training efficiency？

Hi @etoilestar could you provide some additional details? For example: how large is the model? How many GPUs are you using? What is the DeepSpeed config you are using?

hello， I am using deepspeed-meagtron to train GPT which is around 22B parameter, zero-stage is 1, TP is equal to 8, and I am running on 16 nodes   with 8 gpus.

Thank you for sharing some additional details. Can you also share what kind of performance you are seeing and what you are expecting? There are many many parameters that can impact the overall performance you are seeing. For example, are maximizing GPU memory utilization? Have you tried with different TP values? etc.

Hello， thanks for your reply, I just use the same module as I used in DDP-megatron, however, it seems that the throughput is much lower when I use deepspeed-megatron， referring to https://github.com/microsoft/Megatron-DeepSpeed. of cource the GPU memory utilization is much lower. I also used different TP values and different number of parameters.


The docs badge shows "Docs: no builds" even though there clearly are.

The reStructuredText fragment is pasted directly from the Read the Docs settings page, so I'm not sure what's wrong.

This seems to have "corrected itself".


dvclive: Add save_dvc_exp to API reference.

https://github.com/iterative/dvclive/releases/tag/1.1.0

Minor thing that's been bothering me playing with this: should we rename save_dvc_exp to dvc_exp_save for consistency with the CLI?

Minor thing that's been bothering me playing with this: should we rename save_dvc_exp to dvc_exp_save for consistency with the CLI?

No strong opinion

@daavoo Just a reminder to get back to this PR. Can we get it merged tomorrow?

@daavoo Just a reminder to get back to this PR. Can we get it merged tomorrow?

Thanks for pinging 🙏 Didn't forget but neither found the time 😅 .
I will try to push some changes tomorrow

@iterative/websites Any idea why deployment isn't working? Looks to be stuck in a loop of failure and redeploying.

Gonna temporarily close for a redeploy

Deployment is working now, thanks @rogermparent

Thanks @daavoo! Nice start. Let me know if you want me to take over at any point. Probably one more major iteration and it should be in good shape.

@daavoo PTAL and merge whenever you feel it's ready 🙏 .


Close channels when normal complete or error

This is for MMENG-4117 [generic proxy] netstat find CLOSE_WAIT sockets which need to be closed
I suspect some user connections broke before sending any useful data to proxy, so I close channels when such 'bad request' happens.

LGTM. let's test it later. thanks.


Copy Stan's src/stan/services/arguments/* to CmdStan

Summary:
Stan doesn't use src/stan/services/arguments/* and it's not used in RStan or PyStan. It should be moved to CmdStan.
Current Version:
v2.9.0

This was the first step of the original refactor, which would then
get propagated to the interfaces.  Of course that the refactor died
in a flames leaving the code in an intermediate state.  I don’t think
it’s worth moving this all to CmdStan but rather implementing the
“proper” refactor that is supposedly in the works directly.
On Jun 8, 2016, at 10:57 PM, Daniel Lee<EMAIL_ADDRESS>wrote:

Summary:
Stan doesn't use src/stan/services/arguments/* and it's not used in RStan or PyStan. It should be moved to CmdStan.
Current Version:
v2.9.0
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

Once again, Mike, you're right. It's not as simple as moving things over since it's all tied together. It was blocking me conceptually, but I'll have to work around it.


[Chore] Update README

Minor update
Explain what this PR does.

Can we have a DCO commit please?

I have manually set it to pass.


Upgrade SDK to 1.5.0

Starting from January, Could Payments moving away from 32 bit integers for the TransactionId attribute to 64 bit. In order to support this, they require all clients to upgrade to 1.5.0 version of their SDK. Any plans for upgrading?

Hello, could you share where you got this information?


Rails 5 bundle install error

Fix bundle install in Rails 5
"As of sprockets-rails version 3.1.0, used in current versions of rails, this gem is deprecated." from https://github.com/evrone/quiet_assets

@frontandstart, correct! Can you please sign the CLA so I can process the patch?

 I'd like to merge your patch to give you credit, but would like to have it CLA processed... Any problem if I mention you in the commit message for the overall patch for Rails compatibility and change the line myself?

Ok, thanks. I was redirected on your website and be sure that sign you CLA in PDF. but any way thank you for mention) First commit in open source its like first sex, very exiting)

I'm closing this as it appears to be in the code base already as of this commit


bugfix/everit plugin

Closes sast-scan #229
The required plugin is downloaded previously from the external repository, getting the makeAggregateBom command ready to be ran.

Thank you!


Files for Scotch example no longer available

In the Got Scotch API? example, the Box files are no longer available:
HTTPError: 403 Client Error: Forbidden for url: https://ibm.ent.box.com/shared/static/2vntdqbozf9lzmukkeoq1lfi2pcb00j1.dataframe

@robmarkcole i'm curious if you got it recreated from original matlab file?
https://outreach.mathstat.strath.ac.uk/outreach/nessie/nessie_whisky.html

@dkhokhlov I did not


Training a new model

Hi @jts ,
Nanopolish uses 6-mers to call CpG methylations. I was wondering can we train a model with nanopolish using 5-mers rather than 6-mers?
Thanks,
Vahid.

In principle yes but it would be somewhat difficult to set up the initial 5-mer model to base the training on. This is not a supported feature right now so I can't directly help but will keep it in mind next time I work on model training.
Jared

Thanks @jts


About Reproducibility Issues

Hi there,
Thank you very much for sharing the source codes.
For reproducibility, I modified the codes as follows,
https://github.com/GFNOrg/gflownet/blob/831a6989d1abd5c05123ec84654fb08629d9bc38/mols/gflownet.py#L84
---> self.train_rng = np.random.RandomState(142857)
as well as to add
torch.manual_seed(142857)
torch.cuda.manual_seed(142857)
torch.cuda.manual_seed_all(142857)

However, I encountered an issue. I ran it more than 3 times with the same random seed, but the results are totally different (although they are close). I didn't modify other parts, except for addressing package compatibility issues.

0     [1152.62, 112.939, 23.232]
100 [460.257, 44.253, 17.728]
200 [68.114, 6.007, 8.045]
0     [1151.024, 112.603, 24.993]
100 [471.219, 45.525, 15.964]
200 [66.349, 6.174, 4.607]
0     [1263.066, 124.094, 22.128]
100 [467.747, 44.899, 18.76]
200 [61.992, 5.715, 4.841]

I am wondering whether you encountered such an issue before.
Best,
Dong

Hi Dong,
even with seeding, CUDA operations are not guaranteed to be deterministic, therefore results will change from run to run. This is especially problematic in online-RL-type setups since tiny changes in parameters can result in changes in (self-generated) data distribution which compounds very quickly and pushes models in different directions.
There are ways to get deterministic results but these usually involve using much slower operations which are undesirable in research.

OK. Thank you for your prompt reply.


When sending a pong fails, check if the WS isn't already closed

When sending an automated pong message for an incoming ping, it might happen that the WS becomes meanwhile closed. We should catch these exceptions and ignore them, if that's the case.

The issue happens here: https://github.com/softwaremill/sttp-shared/blob/master/ws/src/main/scala/sttp/ws/WebSocket.scala#L54
if pongOnPing is true (which is the default), the client tries to send a frame, which then fails for JDK ws client.
[info]   Cause: java.io.IOException: Output closed
[info]   at java.net.http/jdk.internal.net.http.websocket.MessageEncoder.encodePong(MessageEncoder.java:301)
[info]   at java.net.http/jdk.internal.net.http.websocket.TransportImpl$SendTask$1.onPong(TransportImpl.java:404)
[info]   at java.net.http/jdk.internal.net.http.websocket.TransportImpl$SendTask$1.onPong(TransportImpl.java:367)

it may be in a state that allows receiving a Close, but not sending anything. I don't know if this will happen for other clients.


Gracefully handle validate when an unknown pack is referenced

Before
thread 'main' panicked at src/packs/pack_set.rs:124:41:
/Users/perry.hertler/workspace/zenpayroll/packs/partners_api/package.yml has 'packs/tacos' in its dependencies, but that pack cannot be found. Try `packs list-packs` to debug.
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

After
1 validation error(s) detected:
/Users/perry.hertler/workspace/zenpayroll/packs/partners_api/package.yml has 'packs/tacos' in its dependencies, but that pack cannot be found. Try `packs list-packs` to debug.

anyhow:
I really like how it infers the returned error and easily propagates errors with an optional "context".  I've also noticed that some of the big names in the rust community use anyhow by default.


Ability to use external "moment" package

Hi,
I'm using this package in my project and it's great!
However when I pack my project using webpack, I notice that the bundle contains two moment libraries, one that I use in my project, and the other used in this library.
This means that the size on my bundle increases significantly (around 460kb).
In my project I'm using "moment": "^2.20.1"
In exceljs it's instead  "moment": "2.19.3"
I would suggest to change the dependence's version, but it would be even better to be able to initialized the library passing my own version of moment.
What do you think?
Thanks!

@nicoladefranceschi I think it would be easier for now to loosen exceljs's dependency requirements (to say >=2.19.3). This will satisfy your need as then npm will align the two without complicating anything.

Published in v1.0.2


s6-networking-man-pages: sync with s6-networking v<IP_ADDRESS> documentation

I wasn't entirely sure which formatting strings are required because my mandoc is terrible so it's a bit of a best guess.

Thanks for your contribution!
Re.  s6-tlsd-io.1, "an" rather than "a" is correct here (which i confirmed a while back with a friend who's an English teacher[1]). So please leave that file unchanged, and let Laurent know of this issue (which he and i have discussed before).
Re. s6-tlsd.1, please update the date in the Dd macro.
Could you also please change the commit message to simply say "Update to <IP_ADDRESS>", to match the style of previous commit messages for project-wide updates based on s6-networking version updates.
(And if you could please update this PR via an amend to the existing commit, rather than creating a new commit, that would be much appreciated. :-) )
[1] When the first sound of a word following an indefinite article is a vowel sound, as is the case with "SSL_PROTOCOL" and "SSL_CIPHER", one uses 'an' rather than 'a'.

s6-tlsd-io.1: you're right and I should have known better. That's what I get for trying to race midnight UTC to get everything in on the 22nd.
s6-tlsd.1: thought I'd caught that one. I'm (again) going to blame being in a bit of a hurry for arbitrary time boundaries.

Merged, thanks! And, heh, i definitely understand what you're saying about racing midnight - been there. :-)


How to make ml-backend for segmentation model

Hi,
I am clueless about how to prepare label-studio-ml-backend for my segmentation model. I am using a segformer for training the data.
And I just have the raw scripts for training and inference. My question is how can I prepare ml-backend from it.

Hi! Have you seen these articles?

https://labelstud.io/guide/ml.html#Start-your-custom-ML-backend-with-Label-Studio
https://labelstud.io/guide/ml_create.html#main
https://labelstud.io/tutorials/dummy_model.html
https://labelstud.io/tutorials/object-detector.html

Thanks!!!!! I will try to implement ml-backend with the help of your suggested tutorials.

Hi @makseq,
I have gone through all the articles but could not find a way to customise a ml backend for my segmentation model. I don't have mmdetection or mmsegmentation model, I want to build it step by step.
So my first question is how can I declare and initialize a class for a segmentation template

attached below.

You are using PolygonLabels, it's not popular case like bboxes. Looks like you need to integrate something like UNet model (pixel masks) and then convert image masks to polygons. As an example for bboxes you can take a look at this mmdetection ML Backend: https://github.com/heartexlabs/label-studio-ml-backend/blob/master/label_studio_ml/examples/mmdetection/mmdetection.py#L20


.Net 7.0.0 now on nuget

Hi - due to the constraint on the .net 7 packages, we can't install NSwag.AspNetCore in a project using .net 7.0.0 packages now.

Such is the life of third party dependencies, I'm sure Rico is very aware .NET 7 is GA today. So it's just a waiting game.

Aye yeah :)

Is there a Preview version that we can use? @RicoSuter

No previews or prereleases yet. The active branches last saw development 2 months ago, so chances are any work necessary to bring this package to .NET 7 isn't started yet.

In actual fact I’m using this fine in a project using rc2. It’s just the version constraint states less than 7.0.0. It’s set at 7.0 which doesn’t include the final release

Soo... How annoying is migration to Swashbuckle?

I wouldn't - I'd just wait for this - Afaik, Swashbuckle doesn't support client generation or any of that jazz.

That means I won't be able to migrate to .NET 7 for... however long the update takes. It's bad enough that Rider only supports .NET 7 in the EAP version, now packages that had months to prepare are also starting to fail.

I just added this as temporary solution. Works fine

Hello @Atulin, can you, please, explain to me, how your complaining about the free work of others is helpful?
Rico is managing libraries by himself really well, to my opinion. And I do not see a reason why anyone should complain about not publishing an upgrade the day a new feature/sdk is published.

I am not a maintainer of the library, yet I try to give to the community. Please go and fix the libraries you are using instead of writing this kind of comments. Thanks

It's not helpful, it isn't meant to be helpful. It's just me venting, that despite .NET 7 having the whole cycle of pre-releases, release-candidates, the whole shebang, some tools and libraries still manage to be behind when the day of release arrives.

I just added this as temporary solution. Works fine
    <PackageReference Include="Microsoft.Extensions.DependencyInjection.Abstractions" Version="7.0.0" />
    <PackageReference Include="Microsoft.Extensions.Options" Version="7.0.0" />

Yes, this works and everything in swagger still works too :)
I saw in the NSWag.Generations.AspCore csproj that it's only checking dotnet 5 and 6.
That is probably why nuget gets confused:
Something like this should fix it (not tested):
  <ItemGroup Condition="'$(TargetFramework)' == 'net7.0'">
    <FrameworkReference Include="Microsoft.AspNetCore.App" />
    <PackageReference Include="Microsoft.Extensions.DependencyInjection.Abstractions" Version="[7)" />
    <PackageReference Include="Microsoft.Extensions.Options" Version="[7)" />
  </ItemGroup>

It should be fixed soon: https://github.com/RicoSuter/NSwag/pull/4220

@coen22 i actually get an error even after updating
error MSB3073: The command "dotnet "/root/.nuget/packages/nswag.msbuild/13.17.0/build/../tools/Net60/dotnet-nswag.dll" run nswag.json /variables:Configuration=Debug" exited with code 150.

Probably something with Net60 folder

Wait for the nswag package update


Strip symbol names from output

rbpf does not allow symbol names greater than 64 bytes, and in https://github.com/solana-labs/move/issues/303 I have needed to generate symbol names larger than that. rustc generates huge symbol names but they are stripped before running on solana.
It seems that we have just been lucky that move-native has not generated any huge symbol names, as that library is not stripped.
Solana does this in the cargo-build-sbf tool by using its strip.sh script. That script essentially just calls llvm-objcopy --strip-all on the .so file.
I am unsure where exactly we should do this. For now we might be able to just do it in rbpf-tests.

After further experiments I don't stripping solves my problems from https://github.com/solana-labs/move/issues/303.
Stripping does not remove symbol names from the dynamic symbol table (.dynsym), and that is where the large symbols are.
So this is probably still needed for production, but does not solve my immediate problems.

In rbpf-tests.rs the linker command already adds an option to strip all symbols. We might have to keep move-native symbols short or replace them by hashes, assuming we can avoid collisions.

We already strip all symbols that are not exported.
When  move build links the final .so file it strips all the symbols. https://github.com/solana-labs/move/blob/eda1dcf69a816a2732d88a80d6f31b459d775066/language/solana/move-to-solana/src/lib.rs#L275
and this is in rbpf-tests.rs when linking the final binary https://github.com/solana-labs/move/blob/eda1dcf69a816a2732d88a80d6f31b459d775066/language/tools/move-mv-llvm-compiler/tests/rbpf-tests.rs#L278


Other Decodings

Elasticsearch version: 5.2
Moloch version: 18.1
OS name and version: Ubuntu 16.04
How was Moloch built/installed: (rpm, deb, singlehost built, ...) deb package, single host
Provide logs, stack traces and steps to reproduce:
The "Other Decodings" is no longer working  when looking at the packet data :(

Ya, known issue and on the list to add back.  For now you can use /sessions.old to get the old interface


cannot estimate gas; transaction may fail or may require manual gas limit

same issue

same issue

Did you get fixed?

estimategas it is an issue on bsc :) put a fix value

i have same issues, i already fix gas to 5 gwei, how to solve this problem? also when bot found profitable it's show error also

please refer to https://github.com/paco0x/amm-arbitrageur/pull/47

Actually contract seems to work only with Tokens that has 18 decimals


Add a custom 404 page

Description
It adds a custom 404 page, wrapped inside the Layout component with a homepage link.

#Fixes #133

hm interesting enough it doesn't seem like it works as a staticly deployed page 🤔  https://deploy-preview-134--next-wordpress-starter.netlify.app/colby
i have a few thoughts here:

maybe it doesn't work on deploy preview, if there's not an obvious solution we should probably just check before doing any additional work by merging it in
the file is /404/index.html - netlify says it looks for 404.html, maybe it doesnt work because of that? though this blog post makes me feel like it should work https://www.netlify.com/blog/2020/12/08/making-a-custom-404-page-in-next.js/ if so we could consider adding a redirects file, something like /404.html /404/index.html, though idk if that would work

https://docs.netlify.com/routing/redirects/redirect-options/#custom-404-page-handling
what do you think?

to be clear the page itself works
https://deploy-preview-134--next-wordpress-starter.netlify.app/404/
https://deploy-preview-134--next-wordpress-starter.netlify.app/404/index.html
Interestingly this one too: https://deploy-preview-134--next-wordpress-starter.netlify.app/404.html

That is weird ... dont know why is happening. Here in my machine is working  (in both modes:  dev and build + start)... As you say it may be the deploy preview or a redirection is needed ... not sure.

lets give it a try! 🚢


Add feature to attach record headers to events

I want to read hilight in picture

edit kafka.rb in intregration plug in kafka

I've migrated this feature request to the integration plugin.


[SMALLFIX] Optimize metric name construction

Since metrics will often be reported multiple times, we reduce the overhead of string construction by caching the metric names.
Also change the user name call to be much more lightweight as accessing the configuration is relatively expensive.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/20389/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/20392/
Test PASSed.


C# Wrapper - Can't Build Working Tutorials - Issue with DLL(s) in recent Releases

Required Info:

Camera Model - D400 series (applies to D415 and D435)
Firmware Version - <IP_ADDRESS>
Operating System & Version - Windows 10 (64 bit)
Platform – PC (64bit)
IDE - Visual Studio 15 2017
SDK Version - 2.19.1 WORKING
SDK Versions -  2.24.0 & 2.29.0 NOT WORKING
Language - C# WPF Desktop Application
Segment - Healthcare Research, AI Deep Learning, Motion Pattern Recognition

Issue Description

I am able to successfully build and run working samples for C# wrappers in SDK v.2.19.1.
I can also build a working application from scratch with v.2.19.1 DLLs (‘Intel.RealSense.dll’ and ‘realsense2.dll’).
I cannot do the same for versions 2.24.0 or 2.29.0, there seems to be a problem with the DLLs (‘Intel.RealSensed.dll’ and ‘realsense2d.dll’ - note the added 'd')

STEPS TO REPRODUCE
Build and Run WORKING Tutorial Project from v.2.19.1 Source Code

Download ‘librealsense-2.19.1.zip’ from https://github.com/IntelRealSense/librealsense/releases
Unzip files to ‘C:’
Create directory ‘build’ (C:\librealsense-2.19.1\build)
Using CMake v.3.14.0 GUI
a.	Click ‘Browse Source…’ and browse to directory ‘C:/librealsense-2.19.1’
b.	Click ‘Browse Build…’ and browse to directory ‘C:/librealsense-2.19.1/build’
c.	Click ‘Configure’
--- Confirm Generator, is set to ‘Visual Studio 15 2017’
--- Specify Platform, select ‘x64’
--- Click Finish and wait for CMake to Configure Build
d.	Check ‘BUILD CSHARP BINDINGS’
e.	Click ‘Generate’
Open ‘C:\librealsense-2.19.1\build\librealsense2.sln in Visual Studio 2017
a.	Right-click ‘Solution librealsense2’ at top of Solution Explorer, select ‘Properties'
--- For Startup Project, select Current Selection
b.	In Solution Explorer, select Project ‘cs-tutorial-2-capture’ (in Wrappers/csharp)
c.	Click 'Start' to Build and Run. Generates 203 Warnings, 0 Errors
d.	Launches application and streams frames to Depth and Color image controls

To Create, Build and Run WORKING Application after building v.2.19.1 Source Files in Visual Studio

Create new WPF Desktop Solution and Project in Visual Studio
Use Configuration Manager to change Platform from ‘Any CPU’ to <New…> platform ‘x64’
Add a reference to ‘C:\librealsense-2.19.1\build\Debug\Intel.RealSense.dll’
Add using statement ‘using Intel.RealSense;’
Copy ‘realsense2.dll’ from ‘C:\librealsense-2.19.1\build\Debug’ to ‘YourProject\bin\x64\Debug’

BROKEN C# Wrapper Tutorial in v.2.24.0 and v.2.29.0
After following steps outlined above, attempting to build and run ‘cs-tutorial-2-capture’ with v.2.24.0 or v.2.29.0 receives error message:
Unable to load DLL ‘realsense2’: The specified module could not be found. (Exception from HRESULT: 0x8007007E)
Observation:
In v.2.19.1, the DLLs are ‘Intel.RealSense.dll’ and ‘realsense2.dll’
In v.2.24.0 and 2.29.0 the names have changed to ‘Intel.RealSensed.dll’ and ‘realsense2d.dll’. the letter ‘d’ has been added to the end of both dlls in ‘build\Debug’
The same issues apply to creating a working application from scratch

If you run "Release" in "Solution Configuration" instead of "Debug", it creates dll without the letter 'd'.

Hi @brenteickhoff,
The "d" postfix was added for debug builds. User that doesn't intend to develop/contribute code for the SDK itself should install SDK by following installation guide instructions and build a solution in "Release" mode.
Please let us know if you need any further clarifications regarding this topic.
Thank you!


Feature: Support for UserConsentPage

The current authorization consent page is fixed by splicing HTML in the code (unable to extend the style)

This modification abstracts the UserConsentPage as an interface, which can be customized to achieve the purpose of customized page

On the basis of modifying a small amount of code, users can implant their own customized pages (for example, it can be generated by freemarker/thymeleaf, etc.)

@xzxiaoshan Please sign the Contributor License Agreement!
Click here to manually synchronize the status of this Pull Request.
See the FAQ for frequently asked questions.

recommendation:
remove UserConsentPage interface default generate method , enable util class or impl class 。
eg:
UserConsentPage -> GenerateUserConsentPage
UserConsentPage -> RedirectUserConsentPage (enable queryParam, eg: redirect nginx example.com/us?p1=&p2 ...)

@xzxiaoshan Thank you for signing the Contributor License Agreement!

@xzxiaoshan 保持 UserConsentPage 接口，实现一个 DefaultGenerateUserConsentPage 默认类，是否提供重定向这个后续使用可以自己扩展实现，一些做产品的，别人使用一些自定义页面这个很常有，尽量做到前后端分离。

Thanks for the PR @xzxiaoshan.
It was never the intention to expose UserConsentPage as a public API. It's strictly meant for internal implementation only.
The plan for allowing a custom consent page to be configured was to implement the same pattern as HttpSecurity.formLogin().loginPage().
We have a PR in progress gh-280 that has implemented this way and additionally provided a mechanism for saving/loading user consent decisions, which is needed as part of this feature/customization.
As an FYI, it's always best to reach out first before you start any work so we can align on approach and to also ensure the task is not already taken.


chore(STONEINTG-938): Improve error reporting and log annotation value for better debuggability

We're logging the value of the struct to help
us with debugging in case of race conditions.
We'll be requeueing in case we hit an error
while updating the "git-reporter-status" anntn
This will log silent errors, and help us debug.

Maintainers will complete the following section

[ ] Commit messages are descriptive enough (hints)
[ ] Code coverage from testing does not decrease and new code is covered (check the PR coverage on codecov)
[ ] Controllers diagrams are updated when PR changes controllers code  (if applicable)

Codecov Report
Attention: Patch coverage is 33.33333% with 4 lines in your changes are missing coverage. Please review.

Project coverage is 72.10%. Comparing base (dc775ef) to head (8a4d3ed).
Report is 22 commits behind head on main.

Files
Patch %
Lines

status/status.go
33.33%
4 Missing :warning:

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #734      +/-   ##
==========================================
+ Coverage   70.46%   72.10%   +1.63%
==========================================
  Files          45       42       -3
  Lines        4181     4018     -163
==========================================
- Hits         2946     2897      -49
+ Misses        929      820     -109
+ Partials      306      301       -5

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

I looked at the pod logs of build-service and saw:
2024-05-28T13:37:16.705Z	ERROR	ComponentOnboarding.PaC-setup	controllers/component_build_controller_pac.go:1017	failed to setup Pipelines as Code webhook https://pipelines-as-code-controller-openshift-pipelines.apps.rhtap-4-14-us-west-2-dn2gg.ci.stonesoupengineering.com	{"controller": "component", "controllerGroup": "appstudio.redhat.com", "controllerKind": "Component", "Component": {"name":"test-comp-pac-gitlab-ogvuvt","namespace":"gitlab-rep-yiqr-tenant"}, "namespace": "gitlab-rep-yiqr-tenant", "name": "test-comp-pac-gitlab-ogvuvt", "reconcileID": "d918d6df-63dc-409f-a273-4643ee121b48", "ComponentGitSource": {"url":"https://gitlab.com/konflux-qe/hacbs-test-project-integration","revision":"base-gitlab-ryuufs","devfileUrl":"https://gitlab.com/konflux-qe/hacbs-test-project-integration/-/blob/main/devfile.yaml?ref_type=heads"}, "repository": "https://gitlab.com/konflux-qe/hacbs-test-project-integration", "audit": "true", "error": "POST https://gitlab.com/api/v4/projects/konflux-qe/hacbs-test-project-integration/hooks: 422 {error: Maximum number of project hooks (100) exceeded}"}

The webhooks limit was exceeded because we're not deleting them. This should be fixed by STONEINTG-937

/test integration-service-e2e

Filed KFLUXBUGS-1310 for the latest job failure here.


Constructing container environments using OpenShift descriptions using podman?

/kind feature
Description
I'd like to be able to run and test batches of containers defined with the OpenShift YAML/JSON description locally. As it is now, it's not possible to test these without running a full OpenShift system, which makes it difficult to have a consistent description from development to production.
This would be kind of a superset of the docker-compose.yml issue (#746), leveraging OpenShift descriptions with content for building the containers, setting them up, and running the service.
The advantage of this is that people can easily develop correct descriptions for setting up OpenShift pods locally that transfer well to a proper OpenShift environment for production, which would only help in increasing adoption of OpenShift as a production environment platform.
Additional environment details (AWS, VirtualBox, physical, etc.):

Developer workstations running Fedora with podman and buildah installed
GitLab CI runners with Docker container (of Fedora with buildah + podman)

FWIW, @mheon discussed this at some length over IRC with the requestor @Conan-Kudo .  I think the gist of the conversation was that this would be a good change for @Conan-Kudo but @mheon thought it would also aid the transition of Podman to CRI-O/OpenShift.  Still digging/thinking to do on it.

I think it's potentially beneficial towards CRI-O integration, but also it's a cool direction to go towards if we're already thinking of something similar to compose.

Some of @baude  podman containre play code is implementding

Please use podman play kube


feat(volume): remount volume if it becomes read only

ref: https://github.com/longhorn/longhorn/issues/7843
When detecting the filesystem become read only
Try to remount the global mount point to read write again

The solution looks good to me.
I'm thinking whether an application returns to a normal state or functions as expected after the RO volume switches back to RW without a restart, but this is primarily the responsibility of the application. We can see users' feedback.

Wait for instance-manager PR merged then fix the conflicts

The solution looks good to me. I'm thinking whether an application returns to a normal state or functions as expected after the RO volume switches back to RW without a restart, but this is primarily the responsibility of the application. We can see users' feedback.
Suggest clearly describing the behavior in the official document.

Hi @derekbit I have tested that after remounting the mount point to rw
The IO works fine.
But you are right, there are lots of cases not in our control, I will update the doc

@mergify backport v1.6.1

@mergify backport v1.6.x


Model validation non-nullable reference types typo

If I am reading this correctly, it looks like this is a typo.

Internal previews

📄 File
🔗 Preview link

aspnetcore/mvc/models/validation.md
Model validation in ASP.NET Core MVC and Razor Pages

@dotnet-policy-service agree

How did we get that backwards?
Thanks for the contribution. Be sure to check out your name as a contributor about the 3rd of next month on the What's new page in the Community contributors section at the bottom.

Cool!

Check out is:issue is:open label:"code help wanted" " and is:open is:issue label:"Good first issue"

@jkrajcir @Rick-Anderson I don't understand how that was a typo.
According to the implementation, if the RequiredAttribute is inferred, AllowEmptyStrings is set to true.
Inference only happens if nullability.ReadState == NullabilityState.NotNull (which means "Non-nullable value or reference type" with an enabled nullability context).
See https://github.com/dotnet/aspnetcore/blob/main/src/Mvc/Mvc.DataAnnotations/src/DataAnnotationsMetadataProvider.cs#L359-L367

Oh, yeah looks like I was wrong, my mistake. Thanks for catching that @cn-byn .
I think I misunderstand because I didn't know empty strings are converted to null by default. It's even mentioned in the code @cn-byn referenced.
@Rick-Anderson would it be useful to add a small note about this behavior alongside the example shown in the docs?

@Rick-Anderson would it be useful to add a small note about this behavior alongside the example shown in the docs?

@jkrajcir  can you draft that note and I'll add it. Or you can add the note in another PR if you'd like the merge credit.


Add base URL as argument to the tool

Fixes #16

Coverage remained the same at 100.0% when pulling 25acc83fc0a267692cdf87f559794ba8970ff2e5 on feature/base-url-as-argument into c7e09dd81ab694191defddd8ee4929702781dbf6 on master.


Add datetime converter and util

Summary
Checklist

[ ] If code changes were made then they have been tested.

[ ] I have updated the documentation to reflect the changes.

[ ] This PR fixes an issue.
[x] This PR adds something new (e.g. new method or parameters).
[ ] This PR is a breaking change (e.g. methods or parameters removed/renamed)
[ ] This PR is not a code change (e.g. documentation, README, ...)

Is this really needed? I remember seeing Danny and the other people in d.py talk about this - they agreed that since they're no easy way for users themselves to the Discord timestamp markdown, that it really wasn't worth it, and I kinda have to agree. There's no UI to make the timestamp, meaning most users would have to hear about this markdown from some source and then actually learn how to use it, which I'm willing to bet most users won't be able to do both.
I think it's best to let bot developers decide if it's worth making a converter like this for the small amount of Discord users that know how to use said markdown.

I'm with astrea/Danny/others on this one, I don't see this as useful in the library. If discord added some sort of UI to create timestamps I could see a use case here. As it stands, I don't see this being used

I'll reiterate that as NC has suggested there is an incuded util, I think thats important for a user to be able to extract a datetime from a timemstamp. I can remove the converter however. Also having a converter doesn't hurt, but I think yeah users can implement the converters with the util.

Yeah, I can agree that the util itself makes sense - I can think of some uses a bot maker might have for it. Just think a converter doesn't make much sense at this time - it doesn't hurt, but I think it's best not to add unnecessary features if possible.


Fail to remove a storage cluster that failed to create

WARNING:asyncio:Executing <Handle Task._wakeup created at /usr/lib/python3.5/asyncio/futures.py:373> took 0.212 seconds
[Tue26 10:04] - Job.py              :57  :j.core.jobcontroller.job.8dfe3721e6ede234799f9db9d8a72871 - ERROR    - Traceback (most recent call last):
  File "/usr/lib/python3.5/concurrent/futures/thread.py", line 55, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/tmp/actions/storage_cluster/a30d49e72a959e3670cc36b4079bd8b5.py", line 405, in stop
    cluster = get_cluster(job)
  File "/tmp/actions/storage_cluster/a30d49e72a959e3670cc36b4079bd8b5.py", line 132, in get_cluster
    return StorageCluster.from_ays(job.service, job.context['token'])
  File "/usr/local/lib/python3.5/dist-packages/zeroos/orchestrator/sal/StorageCluster.py", line 38, in from_ays
    storages_server = StorageServer.from_ays(storageEngine_service, password)
  File "/usr/local/lib/python3.5/dist-packages/zeroos/orchestrator/sal/StorageCluster.py", line 160, in from_ays
    storageEngine = StorageEngine.from_ays(storageEngine_services, password)
  File "/usr/local/lib/python3.5/dist-packages/zeroos/orchestrator/sal/StorageEngine.py", line 30, in from_ays
    container = Container.from_ays(service.parent, password, logger=service.logger)
  File "/usr/local/lib/python3.5/dist-packages/zeroos/orchestrator/sal/Container.py", line 110, in from_ays
    sp = node.storagepools.get(fs_service.parent.name)
  File "/usr/local/lib/python3.5/dist-packages/zeroos/orchestrator/sal/StoragePool.py", line 59, in get
    raise ValueError("Could not find StoragePool with name {}".format(name))
ValueError: Could not find StoragePool with name cluster_2c600ccd2ae9_primary_sda
[Tue26 10:04] - RunStep.py          :84  :j.atyourservice.server         - INFO     - runstep 1: error
[Tue26 10:04] - Run.py              :108 :j.atyourservice.server         - ERROR    - error during execution of step 1 in run 47f3a22ab320066a7d817b74e5f99217
Error of job: storage_cluster!primary (stop):
*TRACEBACK*********************************************************************************
ValueError: Could not find StoragePool with name cluster_2c600ccd2ae9_primary_sda

******************************************************************************************

I fixed the above error in https://github.com/zero-os/0-orchestrator/pull/1349, but when I tried to test the behavior I hit https://github.com/Jumpscale/ays9/issues/376 issue, which ays known issue. So this issue still not resolved completely and blocked on ays's behavior


safe &'static mut references through a runtime checked macro

runtime checked implementation of japaric/cortex-m-rtfm#59 that doesn't depend on RTFM macros
TODO

[ ] bikeshed macro syntax

Looks great, I'm not too hot about the name static_ though. What about singleton?

I'm not too hot about the name static_ though.

Me neither.

What about singleton?

Sounds to better to me.
Please continue bikeshedding :-).


not getting all the tweets from all the days when using "since" and "until" when the query result is big

Issue Template
Please use this template!
Initial Check

If the issue is a request please specify that it is a request in the title (Example: [REQUEST] more features). If this is a question regarding 'twint' please specify that it's a question in the title (Example: [QUESTION] What is x?). Please only submit issues related to 'twint'. Thanks.

Make sure you've checked the following:

[X] Python version is 3.5 or higher.
[X] Using the latest version of Twint.
[X] I have searched the issues and there are no duplicates of this issue/question/request.

Command Ran
import  twint
c = twint.Config()
c.Count = True
output csv
c.Output = "file.csv"
c.Store_csv = True
c.search_name ="testmysql1_en"

c.Search = "elonmusk%20OR%20testla%20OR%20electric%20OR%20car%20OR%20apple%20OR%20volvo%20OR%20uber"

c.Since = "2017-09-01"
c.Until = "2017-10-27"
c.Lang = "en"
twint.run.Search(c)

it gives only  returns 25831 tweets from only these not consecutive days 2017-10-27, 2017-10-26 and 2017-09-27
it makes me think there is a problem with how the system manages the dates.
if I use the same parameters but changing the since and until parameters by:
c.Since = "2017-10-23"
c.Until = "2017-10-24"
then I get 81557 tweets from  2017-10-24 and 2017-10-23  Why did not appear in the previous search? this time frame is included in the previous one
and for doing it more weird... if I use the first time frame but searching for a not common stuff
import  twint
c = twint.Config()
c.Count = True
output csv
c.Output = "file.csv"
c.Store_csv = True
c.search_name ="testmysql3_en"
c.Search = "gimeno"
c.Since = "2017-09-01"
c.Until = "2017-10-27"
c.Lang = "en"
twint.run.Search(c)

I get 670 tweets from all these days:
'2017-08-28'
'2017-08-29'
'2017-08-30'
'2017-08-31'
'2017-09-01'
'2017-09-02'
'2017-09-03'
'2017-09-04'
'2017-09-05'
'2017-09-06'
'2017-09-07'
'2017-09-08'
'2017-09-09'
'2017-09-10'
'2017-09-11'
'2017-09-12'
'2017-09-13'
'2017-09-14'
'2017-09-15'
'2017-09-16'
'2017-09-17'
'2017-09-18'
'2017-09-19'
'2017-09-20'
'2017-09-21'
'2017-09-22'
'2017-09-23'
'2017-09-24'
'2017-09-25'
'2017-09-26'
'2017-09-27'
'2017-09-28'
'2017-09-29'
'2017-09-30'
'2017-10-01'
'2017-10-02'
'2017-10-03'
'2017-10-04'
'2017-10-05'
'2017-10-06'
'2017-10-07'
'2017-10-08'
'2017-10-09'
'2017-10-10'
'2017-10-11'
'2017-10-12'
'2017-10-13'
'2017-10-14'
'2017-10-15'
'2017-10-16'
'2017-10-17'
'2017-10-18'
'2017-10-19'
'2017-10-20'
'2017-10-21'
'2017-10-22'
'2017-10-23'
'2017-10-24'
'2017-10-25'
'2017-10-26'
'2017-10-27'
Description of Issue

Please use as much detail as possible.
it only gets tweets from some not consecutive days when you search for more than a month and searching for common words with tones of results

Environment Details

Using Windows, Linux? Linux What OS version? ubuntu Running this in Anaconda? NO Jupyter Notebook? No Terminal? Yes

I'll look into this while I look at https://github.com/haccer/twint/issues/106

I just noticed Twitter has added a new "Quality Filter"... maybe that has something to do with it; I just explicitly turned it off.

I read the #106 and it could be related to but even the problem likes the same, the way to do the search and the parameters were different and that's why I opened new issue. Please feel free in closing it as I am also following the #106

after applying the last commit you did, I launched again the first query and  the result was not really much better.
this is the code:
c = twint.Config()
c.Count = True
output csv
c.Output = "file.csv"
c.Store_csv = True
c.search_name ="testmysql1_en"
c.Search = "elonmusk%20OR%20testla%20OR%20electric%20OR%20car%20OR%20apple%20OR%20volvo%20OR%20uber"
c.Since = "2017-09-01"
c.Until = "2017-10-27"
c.Lang = "en"
twint.run.Search(c)
before the change this was the result:
it only returned 25831 tweets from only these not consecutive days
2017-09-27
2017-10-26
2017-10-27
after the change:
it returned 181.687 tweets which is a really bigger number than before but only for four days:
'2017-09-26'
'2017-09-27'
'2017-10-26'
'2017-10-27'
and like I checked previously, if y use
c.Since = "2017-10-23"
c.Until = "2017-10-24"
with the same query I got 81557  tweets so..... it seems there is something else :(

I suspect since this might be related to #106, I'm running the first scripts above with the master branch on GitHub and the current stable release on PyPI.

PIP script stopped at 19234
Master branch script still going

It might take me a while to attempt to debug this.... Large queries wern't necessarily a problem in the past as I collected like 650k Tweets from the #metoo search...

In the past I also got more than 500K tweets without any problem

Right now the master run has reached the point where the pip version ended which is significantly more tweets... I suspect the qualify filter option i recently added has something to do with that

I use the version with the qualifier update and I got more tweets but only from 4 days and the time range were almost 2 months :(

I have perform another test with the qualifier off.
The query was exactly the same for the four test I did and I only changed the time frames.
1st  test called testmysql1_en : c.Since = "2017-09-01" c.Until = "2017-10-27"
2nd test called testmysql2_en:  c.Since = "2017-10-20" c.Until = "2017-10-27"
3rd  test called testmysql3_en:  c.Since = "2017-10-10" c.Until = "2017-10-19"
4th test called testmysql4_en:  c.Since = "2017-10-01" c.Until = "2017-10-09"
the query was:
import  twint
#info comun
c = twint.Config()
c.Count = True
#output database
c.search_name ="testmysql1_en"
c.Search = "elonmusk OR testla OR electri OR car OR apple OR volvo OR uber"
c.Since = "2017-09-01"
c.Until = "2017-10-27"
c.Lang = "en"
twint.run.Search(c)
the number of tweets obtained in each test are:
COUNT(tweets.id), count(id)
'26552', 'testmysql1_en'
'105321', 'testmysql2_en'
'25661', 'testmysql3_en'
'25852', 'testmysql4_en'
the tweets obtained per day and per test are:
count(*), date, Search
'279', '2017-09-27', 'testmysql1_en'
'10404', '2017-10-26', 'testmysql1_en'
'15869', '2017-10-27', 'testmysql1_en'
'89451', '2017-10-26', 'testmysql2_en'
'15870', '2017-10-27', 'testmysql2_en'
'11519', '2017-10-18', 'testmysql3_en'
'14142', '2017-10-19', 'testmysql3_en'
'12944', '2017-10-08', 'testmysql4_en'
'12908', '2017-10-09', 'testmysql4_en'
I got tweets for only two or three days and the lower time frame was a week
the test one and two, the until parameter is the same and:
- the test 1 obtained 10.404 tweets on 2017-10-26 when the test 2 in the same day obtained 89.451
it is very weird :(
it does make sense these numbers

The script had finished w/
912305785987837952 2017-09-25 13:20:23 UTC <bellestarr48> @JoyAnnReid @nycsouthpaw This is from May....has anyone noticed that American citizens are dying....no food, water or electric !!! It's an island underwater!!!
[+] Finished: Successfully collected 346581 Tweets.

I think it's time implement some logging to better debug the last request of each run, because this stopped early, as since was c.Since = "2017-09-01"

There is a debug parameter, isn't it?

I closed it

I removed the debug param a while ago because it needed to be redone

There is a debug parameter,  isn't it?  How it works?   Which King of log generates?

Any guide about how can I debug it?

yeah if you want to debug on your own,
After line 23 in search.py before self.feed = [] put:
print(response, file=open("twint.log", "w", encoding="utf-8"))

Then when the script stops, we can look at twint.log which will contain the last response so we can try to figure out why it stopped early.

I will do it,  many Many thanks

Something is technically wrong.

Thanks for noticing—we're going to fix it up and have things back to normal soon.

This showed up in the twint.log when the run stopped.... this is likely due to the amount of requests being sent lol... SO what I can do is write something that'll detect this error, wait a little bit (10 second increments) then try again to see if it'll work. Sound good?

this is likely the cause of the issue for #106 too

I run the queries again last night and hereby attached the final logs
these are the results of number of tweets per test:
COUNT(tweets.id), search
'26552', 'testmysql1_en'
'22992', 'testmysql1_en_log'
'105321', 'testmysql2_en'
'20312', 'testmysql2_en_log'
'25661', 'testmysql3_en'
'19991', 'testmysql3_en_log'
'25852', 'testmysql4_en'
'20357', 'testmysql4_en_log'
and per days:
count(*), date, Search
'279', '2017-09-27', 'testmysql1_en'
'10404', '2017-10-26', 'testmysql1_en'
'15869', '2017-10-27', 'testmysql1_en'
'7159', '2017-10-26', 'testmysql1_en_log'
'15833', '2017-10-27', 'testmysql1_en_log'
'89451', '2017-10-26', 'testmysql2_en'
'15870', '2017-10-27', 'testmysql2_en'
'4479', '2017-10-26', 'testmysql2_en_log'
'15833', '2017-10-27', 'testmysql2_en_log'
'11519', '2017-10-18', 'testmysql3_en'
'14142', '2017-10-19', 'testmysql3_en'
'5856', '2017-10-18', 'testmysql3_en_log'
'14135', '2017-10-19', 'testmysql3_en_log'
'12944', '2017-10-08', 'testmysql4_en'
'12908', '2017-10-09', 'testmysql4_en'
'7472', '2017-10-08', 'testmysql4_en_log'
'12885', '2017-10-09', 'testmysql4_en_log'
it finished with error and not always fails at the same point even the query is exactly the same.
testmysql1_en_log.log
testmysql2_en_log.log
testmysql3_en_log.log
testmysql4_en_log.log

the solution you proposed sounds great :)

one comment over here even it might be does nothing to do with that.
the master branch dor not have the quialifer of, file url.py line 49:
url += "reset_error_state=false&src=typ&max_position={}&q=".format(init)
but you tried it ... in the mysql branch it is off.
Which one is finally the good one?
url += "reset_error_state=false&src=typd&qf=off&max_position={}&q=".format(init)

o
url += "reset_error_state=false&src=typd&max_position={}&q=".format(init)

I am doing some tests increasing the timeout and with this code in the search.py file:
async def Feed(self):
response = await get.RequestUrl(self.config, self.init)
print(response, file=open(self.config.search_name+".log", "w", encoding="utf-8"))#debug
if "Twitter / Error" in response:
time.sleep(30)
response = await get.RequestUrl(self.config, self.init)
print("1", file=open(self.config.search_name+"error.log", "a", encoding="utf-8"))#debug
print(response, file=open(self.config.search_name+".log", "w", encoding="utf-8"))#debug
self.feed = []
try:
self.feed, self.init = feed.Json(response)
except:
pass
To be honest I am not sure what I am doing and it it will improve anything..... for the moments it gathered more than 100K  tweets but there is something very weird, tha dates jump from 26 Oct to 27 Sep:
count(*), date, Search
'49264', '2017-09-26', 'testmysql1_en_log_1'
'16916', '2017-09-27', 'testmysql1_en_log_1'
'99842', '2017-10-26', 'testmysql1_en_log_1'
'15874', '2017-10-27', 'testmysql1_en_log_1'
I will share the outcome once it finish

well, I have the outcome almost then 700K (694986) tweets gathered but only from 6 days when the time frame was 2017-09-01  2017-10-01, 2 days from October and 5 days from September
count(*), date, Search
'34536', '2017-09-23', 'testmysql1_en_log_1'
'154594', '2017-09-24', 'testmysql1_en_log_1'
'183891', '2017-09-25', 'testmysql1_en_log_1'
'189733', '2017-09-26', 'testmysql1_en_log_1'
'16959', '2017-09-27', 'testmysql1_en_log_1'
'100021', '2017-10-26', 'testmysql1_en_log_1'
'15911', '2017-10-27', 'testmysql1_en_log_1'
it registered 7 "Twitter / Error".... and it is the same number of days ... could it be  causality but....
the last log registered is :
{"min_position":"TWEET-911651435296825344-923700770066776065","has_more_items":false,"items_html":"\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n","new_latent_count":0,"focused_refresh_interval":30000}

yeah,
{"min_position":"TWEET-911651435296825344-923700770066776065","has_more_items":false,"items_html":"\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n","new_latent_count":0,"focused_refresh_interval":30000}

is the response when there's no more tweets for the query and should be the last response

It doesn't make sense because the time frame was from 2017-09-01 until 2017-10-27 and there are only results from 7 days, 2 from October (27 and 26) and 5 from September (27,26,25,24 and 23)
What about the rest of the days???? there are only result in 7 of the almost 60 days I asked for and  I checked that there are tweets in , for example 24 and 23 of October and they don't appear in that query.....
the code was:
import  twint
#info comun
c = twint.Config()
c.Count = True
#output database
c.search_name ="testmysql1_en_log_1"
c.Search = "elonmusk OR testla OR electri OR car OR apple OR volvo OR uber"
c.Since = "2017-09-01"
c.Until = "2017-10-27"
c.Lang = "en"
twint.run.Search(c)

I think I found out where the problem is with very large queries.
If you don't specify a Timedelta, by default is 30 days. Therefore the first bath in the time frame I specified is 2017-09-27 until 2017-10-27. the outcome I got is for example something like that: there are only results from 7 days, 2 from October (27 and 26) and 5 from September (27,26,25,24 and 23).
I tried to bypass the timedelta stuff to go over the whole time frame but then I only go 3 days from October ....
The last thing I am trying is specify a timedelta = 2 days and it seems it is working fine, I think it will get the tweets from all the days in the timeframe.
the code I am referring to is in the search.py :
I write some prints for debugging
async def main(self):
if self.config.User_id is not None:
self.config.Username = await get.Username(self.config.User_id)
    if self.config.Since and self.config.Until:
        _days = timedelta(days=int(self.config.Timedelta))
        print("$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$")#debug
        print(_days)#debug
        while self.d._since < self.d._until:
            print( self.d._since)#debug
            print(self.d._until)#debug
            self.config.Since = str(self.d._until - _days)
            print(self.config.Since)
            self.config.Until = str(self.d._until)
            print(self.config.Until)
            if len(self.feed) > 0:
                await self.tweets()
            else:
                self.d._until = self.d._until - _days
                self.feed = [-1]

            if get.Limit(self.config.Limit, self.count):
                self.d._until = self.d._until - _days
                self.feed = [-1]
    else:
        while True:
            if len(self.feed) > 0:
                await self.tweets()
            else:
                break

            if get.Limit(self.config.Limit, self.count):
                break

    verbose.Count(self.config, self.count)

I will share the results when the query will finish and I will do some more test....as the queries are very large it will take some hours to get them

Do you know if it has been able to solve something realted to this problem an that of #106?

I don't know.
what I can say is using two days deltatime you can get tweets from all the time frame.
In my case I go almost 1.700.000 tweets from almost 60 days with out any problem. it's a really huge number.

What do you mean with deltatime?
Could you give me an example? please

there is a parameter in twint called timedelta
-t, --timedelta | Time interval for every request
by default is 30 days and when you ask twitter for tweets during an specific time, it request in 30 days batches.
it seems that when the query is too large and with the default 20 days value, you don't get all the data properly.
so...if you use --t 2 you request data using 2 days batches so in my case it worked very well.
you could try and let's see what happens.

I tried to do it as you said, but nothing. I keep getting 3110 tweets. It does not let me get older tweets.

I tried the comand you expose in #106 but with count and timedlta parameters:
#106

Sorry, I explained myself wrongly. You are absolutely right, with the way you have previously commented it works perfectly.
However, it does not work when the command --profile-full is added. With this added command, to get retweets too, it does not exceed 3000 tweets.

I will check it and I will share the outcome....let's see how it works for me

Closing this since the new --resume option mitigates this.


How to change backbone

I'd like to change the backbone to another architecture
but I did not find the code of this part
how to change the architecture of backbone?
thanks~
@heilaw

The network architecture is defined here.

@heilaw thank u for ur reply
i have another problem when i test my model
the model can be tested on coco val2014 with 38mAP and i think the result is acceptable because some of my training details
but when i test on coco test2017，mAP are all -1，i dont know the meaning of -1 ，i googled this but find nothing
thanks a lot!

btw，issue#120 meet the same problem like mine


Display difference of choices in Prompt Options in Emulator and web Chat

I am trying to provide a set of choice to the user using PromptOptions and ChoiceFactory in Bot Framework V4. When tested on Emulator I get the choices as suggestions:

but when testing on the Webchat I get the choice in indexed format like:
1.S
2.M
3.L
4.XL
5.XXL
6.XXXL
CODE:
step.PromptAsync("cardPrompt", new PromptOptions { Prompt = MessageFactory.Text("Please select the size of '" + userProfile.Productname + "' that you need."), Choices = ChoiceFactory.ToChoices(new List { "S", "M", "L", "XL", "XXL", "XXXL" }) }, cancellationToken);
Would be helpful  if anyone can provide suggestions on how can we achieve the choices as separate suggestion box.

Please try 4.1 bits, as soon as they become available (later today). this should be fixed.


Private methods deleted

Table of Contents
Link to GitHub issues it solves
closes #10
Description
Private methods deleted from the memo-compiler library
Contributing has more information and tips for a great pull request.

@zeobot set lc to pass


[v6] Fields component custom props documentation bug or code bug?

Checking out the new Fields component. The docs show that custom props will be passed in alongside the fields, but actually they are passed within the field properties. Apologies if this was already reported, but I didn't find it in my search of the issues here.
Relevant code is here - ConnectedFields.js line 45. It adds the rest containing the custom props.
It looks like a quick fix could be to include ...rest after ...custom in the createElement call on line 65.
Or potentially drop line 51 to remove the properties from being duplicated within each field.
Or update the documentation to show the additional props exist within each field. Personal preference is for the properties to be passed as shown in the docs.
Thanks! I have a "wizard" style form where some of the pages have fields that appear after the previous fields have been filled out. It seems the Fields component should localize the amount of this form that needs to re-render.

Apologies if this was already reported, but I didn't find it in my search of the issues here.

I reported it here #1583 and it should already be fixed here #1589 and will thus be available in the next release.

Ah. I should have tested against the head of the v6 branch. Sorry and thanks!!


Make the location view table a grid.

The use of a table for the infoj view in the location view is rather clunky. This should be turned into a grid view at some point.

testing on branch dev_locationview_as_grid

Works ok.


Use the provider secret instead of putting the plain details in the CR

Currently we pass the URL, username, password and certificate to the CR. A better way would be to just use the secret for this as it should have all these details, changing the CR to look like:
apiVersion: forklift.konveyor.io/v1beta1
kind: OvirtImageIOPopulator
metadata:
  name: ovirt-imageio1
  namespace: konveyor-forklift
spec:
  secret: ovirt-secret-s2jrf
  diskId: 1a6757a4-2af5-41b4-9076-57687be83ecf

The populator code would need to be changed and use the k8s client to get the relevant data from the secret.

@ahadas I suppose we may want an additional field for the certificate in case we use a 3rd pary one instead of the engine certificate?

@ahadas I suppose we may want an additional field for the certificate in case we use a 3rd pary one instead of the engine certificate?

yep


Some general doubts regarding the use of Vitis AI Optimizer and some specific ones about VAI Optimizer Tensorflow2

I've been working for a couple of months with VAI Optimizer and after a careful reading of both Vitis AI Optimizer User Guide (UG1333) (v3.0) and Vitis AI User Guide (UG1414) (v3.5) I have some doubts that I think are interesting to share and I hope the community or AMD-Xilinx employees can solve.

Do you expect fine-grained pruning to be supported by a DPU? If not, does it exist another Xilinx tool where we could take advantage of fine-grained pruning?
How does the tool use the model sensitivity analysis (model.sens)?
Specifically for TF2 VAI Optimizer, when I specify a certain ratio how does the tool choose between all the models that meet the pruning criteria? I imagine that evaluating all of them one by one would be prohibitively costly. Similarly, if a threshold (instead of a ratio) is specified, how does the tool choose the most adecuate model? From my very short experience, when using the ratio (instead of the threshold) as the pruning criteria, much more "Model Analysis Report"s are output, so this fact distracts me even more as I would think that the ratio criteria would imply analysing fewer models.
During the output of "Model Analysis Report" some options appear (max_depth, min_bytes, min_peak_bytes...) What does that mean?
How does the tool select which channels have to be pruned? Does it use L1 norm criteria?
Can the pruning ratio be multiple of 0.05 (and not multiple of 0.1)?
Will you support One-Step Pruning and Once For All Pruning (OFA) for Tensorflow2 in the very near future?
Even though the output of the eval function can be a metric different from the loss, the threshold can only be specified related to the loss. Am I right?
The release notes from Vitis AI 3.5 state that TensorFlow 2 Optimizer includes "Support for setting the number of channels to be a multiple of the specified number after pruning". Does this mean that we have to manually modify the "channel_divisible" field in the corresponding .spec file?
Finally, will you release the v3.5 of the UG1333 or is all the information already contained in UG1414 v3.5? I think I've read somewhere that UG1333 will be included in UG1414 but I am not 100% sure.

Thank you everyone in advance.

DPU has no support plan for fine-grained pruning and there are currently no other tools that can support it.
2 & 3. The sensitivity analysis is used to determine pruning ratio of each layer. For detailed implementation, you can refer to the code: https://github.com/Xilinx/Vitis-AI/blob/master/src/vai_optimizer/tensorflow/tf_nndct/pruning/runner.py#L421
The output is from tf profiler and users do not need to pay attention to it.
Yes, L1 norm criteria is used.
Currently pruning ratio is multiple of 0.1. (You can make some modifications to make it multiple of 0.05 as Optimizer is open sourced)
No support plan for One-Step and OFA in tensorflow2.
Threshold is related to the metric you specified in model analysis phase. It indicates how much variation you allow for this metric after pruning.
There is an argument for this in pruning API:  https://github.com/Xilinx/Vitis-AI/blob/master/src/vai_optimizer/tensorflow/tf_nndct/pruning/runner.py#L384
UG1414 contains all content of UG1333. As Optimizer was open sourced in VAI 3.5, UG1333 was deprecated and the content has been merged in UG1414.

Hi @yuwang-xlnx,
Thanks for the response and the provided references. I will have a look at them as soon as possible.