Parsing data from Device MQTT test: ERROR

When I subscribe to a broker MQTT, I can parse my data: No DevoceObject found.
Can someone help me to configure this device MQTT?
Need I to modify something to use the Device MQTT test?
Thanks
M

May I know the data model or format you are using?
This repository is a reference implementation, and the accepted format is fixed.
https://docs.edgexfoundry.org/1.2/examples/Ch-ExamplesAddingMQTTDevice/
You may try the commercial version if you need more flexible format support in MQTT.

I send this via MQTT : {"name":"MQTT test device","cmd":"val","val":"56"}
And I use the exemple device from device-mqtt-go/cmd/res/example with almost no modification in the .yml and .toml.
But when the device MQTT read the data, it cannot find the device corresponding to the name "MQTT test device"
Where should I configure this?
Thanks
M

If you go through the doc,
1.The device name is come from res/example/configuration.toml
# Pre-define Devices
[[DeviceList]]
  Name = 'MQTT-test-device'

2.val should equal to the device profile's device resource name

With the doc I got some error when the device read the sending data ({"name":"MQTT-test-device","cmd":"randfloat32","randfloat32":"56.30"}). It is the good device name and ressource name but :
level=ERROR ts=2020-05-26T07:01:28.523720796Z app=edgex-device-mqtt source=managedprofiles.go:144 msg="retrieving DeviceResource - Device MQTT-test-device not found"
level=WARN ts=2020-05-26T07:01:28.563690621Z app=edgex-device-mqtt source=incominglistener.go:79 msg="[Incoming listener] Incoming reading ignored. No DeviceObject found : topic=MathTopic msg={"name":"MQTT-test-device","cmd":"randfloat32","randfloat32":"56.30"}"
Have you any idea?
Thanks
M

You can get a list of registered devices by GETting the API of the metadata service.
curl -s http://localhost:48081/api/v1/device
Is your device MQTT-test-device included correctly in the list?

My device MQTT is running but it isn't in the device list from the API call.
http://localhost:48081/api/v1/device
but he is in the list of device service: http://localhost:48081/api/v1/deviceservice
{
    "created":<PHONE_NUMBER>207,
    "modified":<PHONE_NUMBER>207,
    "origin":<PHONE_NUMBER>205,
    "id": "840adabd-5289-4c34-8321-d0d83bb334f7",
    "name": "edgex-device-mqtt",
    "operatingState": "ENABLED",
    "addressable": {
        "created":<PHONE_NUMBER>203,

…
"url": "http://localhost:49982/api/v1/callback"
},
"adminState": "UNLOCKED"
}
]
Should I register him manually, or integrate it in the docker-compose.yml file?

If your device isn't in the list, maybe your configuration.toml is not read on its startup.
The important things are:

The place of configuration.toml which includes [[DeviceList]] for your MQTT-test-device
volumes and entrypoint for MQTT device service in the docker-compose.yml

If you are using Fuji release, please refer to the official examples.
As this example shows, place your configuration.toml under mqtt directory:
- device-service-demo
  |- docker-compose.yml
  |- mqtt
     |- configuration.toml
     |- mqtt.test.device.profile.yml

and mount mqtt directory as /custom-config in the docker-compose.yml:
  volumes:
    ...
    - ./mqtt:/custom-config

and specify /custom-config as confdir in entrypoint:
  entrypoint:
    - /device-mqtt
    - --registry=consul://edgex-core-consul:8500
    - --confdir=/custom-config

If you are using Geneva release, maybe my sample configuration will help you. It is almost same as Fuji, but there are some diferrences in the entrypoint and configuration.toml.

I am on fuji,
Thanks for this explanation, I was just doing it:
I add the config in docker-compose.yml, exactly like you said and like the docs.
I add the 2 files for config in /mqtt. (.yml & .toml)
But when I do the, my device MQTT doesn't start.
Is this really running my device MQTT from sources?
            Name                              Command               State                       Ports

-----------------------------------------------------------------------------------------------------------------------------edgex-device-mqtt                      /device-mqtt --registry=co ...   Exit 1
M

Please investigate the logs by running docker logs edgex-device-mqtt. I guess something is wrong in your configuration file...?

You were right, I got some errors:
~$ docker logs edgex-device-mqtt
Init: useRegistry: consul://edgex-core-consul:8500 profile:  confDir: /custom-config
Check registry is up... http://edgex-core-consul:8500
Pushing configuration into Registry...
Loading configuration from: /custom-config/configuration.toml
error loading config file: could not register service with Registry: Put http://localhost:8500/v1/agent/service/register: dial tcp <IP_ADDRESS>:8500: connect: connection refused

If your configuration.toml is based on this file, it's not suite to use in docker container because all of hostname defined as localhost.
Please copy whole of configuration.toml from the example and modify it to suite your environment.

ok, that is what I was doing first, just build and running from the sources.
so I comment the part on docker-compose.yml and I run my device MQTT from sources.
And if I do it, I got the issue at top of this conversation.

...Ah, sorry for the misunderstanding.
I've never tried that but I think maybe you should pass some arguments like --confdir like you specified in entrypoint in the dokcer-compose.yml when running your own device-mqtt.

In my environment, first time it shows error. But once exit with CTRL+C and simply run again, it works.
🤔
$ git clone https://github.com/edgexfoundry/device-mqtt-go.git
$ cd device-mqtt-go
$ make build
$ cd cmd/

# first time to run, new device is added successfully, but incoming reading ignored.
$ ./device-mqtt --confdir=./res/example
...
level=DEBUG ts=2020-05-28T16:24:25.340375672Z app=edgex-device-mqtt source=devices.go:22 msg="Loading pre-define Devices from configuration"
level=DEBUG ts=2020-05-28T16:24:25.340424505Z app=edgex-device-mqtt source=devices.go:28 msg="Device MQTT-test-device doesn't exist, creating a new one"
level=DEBUG ts=2020-05-28T16:24:25.340805606Z app=edgex-device-mqtt source=devices.go:60 msg="Adding Device:...
...
level=ERROR ts=2020-05-28T16:24:30.004836961Z app=edgex-device-mqtt source=managedprofiles.go:144 msg="retrieving DeviceResource - Device MQTT-test-device not found"
level=WARN ts=2020-05-28T16:24:30.004871945Z app=edgex-device-mqtt source=incominglistener.go:79 msg="[Incoming listener] Incoming reading ignored. No DeviceObject found : topic=DataTopic msg={\"name\":\"MQTT-test-device\",\"cmd\":\"randfloat32\",\"randfloat32\":\"27.2\"}"
...

# exit with CTRL+C, and simply run again, works good
$ ./device-mqtt --confdir=./res/example
...
level=INFO ts=2020-05-28T16:26:30.005008923Z app=edgex-device-mqtt source=incominglistener.go:100 msg="[Incoming listener] Incoming reading received: topic=DataTopic msg={\"name\":\"MQTT-test-device\",\"cmd\":\"randfloat32\",\"randfloat32\":\"26.1\"}"
level=DEBUG ts=2020-05-28T16:26:30.005354945Z app=edgex-device-mqtt source=utils.go:84 correlation-id=767e2cd3-e9d8-40ce-aea0-91bbfd9b3523 msg="SendEvent: EventClient.MarshalEvent encoded event"
level=INFO ts=2020-05-28T16:26:30.008087298Z app=edgex-device-mqtt source=utils.go:94 Content-Type=application/json correlation-id=767e2cd3-e9d8-40ce-aea0-91bbfd9b3523 msg="SendEvent: Pushed event to core data"
...

Oh,
Thanks, I didn't use the --confdir.
Now it's working!
Thank you kurokobo.

Thanks for your help, I got another question.
If I run the device MQTT from sources, can I see it in consul?

You need to add flag for starting
https://github.com/edgexfoundry/device-mqtt-go/blob/8b9d4518ce44b452ea6e7170c38b8c7009d29236/Dockerfile#L49
edgex-core-consul should be replaced by the Consul IP address
Also, please update SDK to v1.2.2-dev.2

I am using the fuji version of the device-mqtt-go from sources nothing is on docker-compose.yml because I do it on local.
If I use :
ENTRYPOINT ["/device-mqtt","--cp=consul://<IP_ADDRESS>:8500","--registry","--confdir=/res/example"]
It should work?

not exactly, it depends on how you deploy your services
It won't work if you are using container environment, because 127.0.01 is the device-mqtt container itself

I don't understand very well.
Can I see the Health of my device MQTT if I run from sources?
I work in a hybrid environment, all EdgeX from docker and just the device-mqtt from sources.
So if I do that, I can't see my device in consul.

running device-mqtt from source is fine.  the point is the device-mqtt can connect to Consul with the right address.  if you make the correct port mapping of Consul container, <IP_ADDRESS> should work.

HI,
Thanks for all your help. It was very helpful.
I got a last question, If I want to run a docker version of the device MQTT go, I need to configure my docker-compose to use a confdir like this?
device-mqtt:
image: edgexfoundry/docker-device-mqtt-go:1.1.0
ports:
- "49982:49982"
container_name: edgex-device-mqtt
hostname: edgex-device-mqtt
networks:
- edgex-network
volumes:
- db-data:/data/db
- log-data:/edgex/logs
- consul-config:/consul/config
- consul-data:/consul/data
- ./mqtt:/custom-config
depends_on:
- data
- command
entrypoint:
- /device-mqtt
- --registry=consul://edgex-core-consul:8500
- --confdir=/custom-config
And when I do it, I go an Exit Code 1 that Indicates a failure due to an application error.
How can I fix this?
I got this docker-compose logs:
edgex-device-mqtt    | Init: useRegistry: consul://edgex-core-consul:8500 profile:  confDir: /custom-config
edgex-device-mqtt    | Check registry is up... http://edgex-core-consul:8500
edgex-device-mqtt    | Pushing configuration into Registry...
edgex-device-mqtt    | Loading configuration from: /custom-config/configuration.toml
edgex-device-mqtt    |
edgex-device-mqtt    | error loading config file: could not register service with Registry: Put http://localhost:8500/v1/agent/service/register: dial tcp <IP_ADDRESS>:8500: connect: connection refused
edgex-device-mqtt    | Init: useRegistry: consul://edgex-core-consul:8500 profile:  confDir: /custom-config
edgex-device-mqtt    | Check registry is up... http://edgex-core-consul:8500
edgex-device-mqtt    | error loading config file: could not register service with Registry: Put http://localhost:8500/v1/agent/service/register: dial tcp <IP_ADDRESS>:8500: connect: connection refused
edgex-device-mqtt    | Init: useRegistry: consul://edgex-core-consul:8500 profile:  confDir: /custom-config
edgex-device-mqtt    | Check registry is up... http://edgex-core-consul:8500
edgex-device-mqtt    | error loading config file: could not register service with Registry: Put http://localhost:8500/v1/agent/service/register: dial tcp <IP_ADDRESS>:8500: connect: connection refused
M

If it's ok for you to use the default configuration file which included in the container image, you don't need to specify your own configuration file by confdir.
But in most cases, in my understanding, you have to specify your own configuration file or build your own container image with your own configuration file because the device service can't reach the the broker with default values and you can't override those values by environment variables. This is one of the restrictions of fuji release.

And for the second question, in fuji release, the configuration file which should be used is different for running in docker and running on your host. The differences are Host value.
This is also one of the restrictions of fuji release.
I guess you are using same configuration file that used when you built and run not in the container but on your host...? The log shows http://localhost:8500/....
You should replace the values for all of Host from localhost to the container name as this file.

...Just for your information, alternatively, you can use geneva version of docker-device-mqtt-go:1.2.0 with existing fuji release.
In 1.2.0, the restrictions that I mentioned are resolved :laughing: You can run device service without using confdir, override any values except for DeviceList by environment variables (this is the recommended way), register your device-profile and devices by APIs.

Thanks a lot.
I still use Fuji version, but with the good configuration.toml
My device is running from docker but I still got Errors:
$ docker-compose logs device-mqtt
...
edgex-device-mqtt    | level=INFO ts=2020-06-04T13:00:24.422899695Z app=edgex-device-mqtt source=driver.go:279 msg="Create MQTT client and connection: uri=tcp://admin:public@<IP_ADDRESS>:18883 clientID=CommandResponseSubscriber "
edgex-device-mqtt    | level=INFO ts=2020-06-04T13:00:24.422946804Z app=edgex-device-mqtt source=driver.go:279 msg="Create MQTT client and connection: uri=tcp://admin:public@<IP_ADDRESS>:18883 clientID=IncomingDataSubscriber "
edgex-device-mqtt    | panic: start command response Listener failed, please check MQTT broker settings are correct, Network Error : dial tcp <IP_ADDRESS>:18883: connect: connection refused
Have you an idea ?
M.

Instead of <IP_ADDRESS>, specify the IP address of the host on which the broker is running.

If the broker is running on the same machine,
edgex-device-mqtt    | level=INFO ts=2020-06-04T13:22:23.553079993Z app=edgex-device-mqtt source=driver.go:279 msg="Create MQTT client and connection: uri=tcp://admin:public@<IP_ADDRESS>:18883 clientID=CommandResponseSubscriber "
edgex-device-mqtt    | level=INFO ts=2020-06-04T13:22:23.553345748Z app=edgex-device-mqtt source=driver.go:279 msg="Create MQTT client and connection: uri=tcp://admin:public@<IP_ADDRESS>:18883 clientID=IncomingDataSubscriber "
edgex-device-mqtt    | panic: start incoming data Listener failed, please check MQTT broker settings are correct, Network Error : dial tcp <IP_ADDRESS>:18883: connect: connection refused

No, not <IP_ADDRESS>, not <IP_ADDRESS>, but some concrete IP address like <IP_ADDRESS> or <IP_ADDRESS>.
Both <IP_ADDRESS> and <IP_ADDRESS> refer only to the IP address of the container itself, not your broker.

Thanks, I think I got the correct address, but I got new errors.
edgex-device-mqtt    | level=INFO ts=2020-06-04T14:02:35.448135459Z app=edgex-device-mqtt source=driver.go:279 msg="Create MQTT client and connection: uri=tcp://admin:public@<IP_ADDRESS>:18883 clientID=CommandResponseSubscriber "
edgex-device-mqtt    | level=INFO ts=2020-06-04T14:02:35.448657595Z app=edgex-device-mqtt source=driver.go:279 msg="Create MQTT client and connection: uri=tcp://admin:public@<IP_ADDRESS>:18883 clientID=IncomingDataSubscriber "
edgex-device-mqtt    | level=INFO ts=2020-06-04T14:02:35.452769677Z app=edgex-device-mqtt source=service.go:127 msg="Service started in: 363.635614ms"
edgex-device-mqtt    | panic: start command response Listener failed, please check MQTT broker settings are correct, Network Error : %!s()

Are the MQTT broker settings correct? Can you subscribe to the same broker using the same IP address, port, credentials, and topic by any other MQTT client like mosquitto-client?

Big thank you,
I just swap the IP address of my computer to the broker one and its works.
My device MQTT is reading messages from the broker on port 1883 and Consul see its health status.
But the device didn't parse message because he didn't recognize the device name.
edgex-device-mqtt    | level=ERROR ts=2020-06-05T07:41:18.441304449Z app=edgex-device-mqtt source=managedprofiles.go:144 msg="retrieving DeviceResource - Device MQTT-test-device not found"
edgex-device-mqtt    | level=WARN ts=2020-06-05T07:41:18.441630592Z app=edgex-device-mqtt source=incominglistener.go:79 msg="[Incoming listener] Incoming reading ignored. No DeviceObject found : topic=DataTopic msg={"name":"MQTT-test-device","cmd":"va:49:mdb_simu_var1","va:49:mdb_simu_var1":"2"}"
So I add the DeviceList to the configuration.toml and 0 evolution.
Need I to modify something else?
M

Could you try completely cleaning up your environment including persistent volumes, then start it again, and watch the logs by running as follows:
docker-compose down --volumes
docker-compose up -d
docker logs -f edgex-device-mqtt

If your DeviceList is loaded correctly, the logs should show up the progress of adding the device.
And ensure that the device is registered or not by querying the metadata API.

Oh,
You were right, I got this issue :
edgex-device-mqtt    | level=ERROR ts=2020-06-08T06:47:01.315403254Z app=edgex-device-mqtt source=devices.go:43 msg="Device Profile Test.Device.MQTT.Profile doesn't exist for Device MQTT-test-device"
edgex-device-mqtt    | level=ERROR ts=2020-06-08T06:47:01.315427266Z app=edgex-device-mqtt source=devices.go:31 msg="creating Device from config failed: MQTT-test-device"
And the device doesn't appear in the device list (API)
My files tree is:
docker-compose.yml
mqtt
|_configuration.toml
|_mqtt.test.device.profile.yml
and in configuration.toml, I define my device like this:
[Device]
DataTransform = true
InitCmd = ""
InitCmdArgs = ""
MaxCmdOps = 128
MaxCmdValueLen = 256
RemoveCmd = ""
RemoveCmdArgs = ""
ProfilesDir = "./"
[Logging]
EnableRemote = true
File = "./device-mqtt.log"
[Writable]
LogLevel = "INFO"
Pre-define Devices
[[DeviceList]]
Name = "MQTT-test-device"
Profile = "Test.Device.MQTT.Profile"
Description = "MQTT device is created for test purpose"
Labels = [ "MQTT", "test"]
[DeviceList.Protocols]
[DeviceList.Protocols.mqtt]
Schema = "tcp"
Host = "<IP_ADDRESS>"
Port = "1883"
ClientId = "CommandPublisher"
User = "admin"
Password = "public"
Topic = "CommandTopic"
Thank you for your help.
M.

You need to modify ProfilesDir to the right location.  The profile might not be loaded.  It should also be "/custom-config", not "./"

With /custom config I don't have the same issue,
It seems that don't load any device profiles
edgex-device-mqtt    | Init: useRegistry: consul://edgex-core-consul:8500 profile:  confDir: /custom-config
edgex-device-mqtt    | Check registry is up... http://edgex-core-consul:8500
edgex-device-mqtt    | Register in registry...
edgex-device-mqtt    | Calling service.Start.
edgex-device-mqtt    | EnableRemote is true, using remote logging service
edgex-device-mqtt    | level=INFO ts=2020-06-08T08:46:12.795298909Z app=edgex-device-mqtt source=init.go:138 msg="Check Metadata service's status ..."
edgex-device-mqtt    | level=INFO ts=2020-06-08T08:46:12.795572884Z app=edgex-device-mqtt source=init.go:138 msg="Check Data service's status ..."
edgex-device-mqtt    | level=INFO ts=2020-06-08T08:46:12.796431793Z app=edgex-device-mqtt source=init.go:48 msg="Service clients initialize successful."
edgex-device-mqtt    | level=INFO ts=2020-06-08T08:46:12.796658101Z app=edgex-device-mqtt source=loader.go:247 msg="listen for config changes from Registry"
edgex-device-mqtt    | level=INFO ts=2020-06-08T08:46:12.797058218Z app=edgex-device-mqtt source=loader.go:271 msg="Writeable configuration has been updated. Setting log level to INFO"
edgex-device-mqtt    | level=INFO ts=2020-06-08T08:46:13.188540266Z app=edgex-device-mqtt source=service.go:146 msg="Device Service  doesn't exist, creating a new one"
edgex-device-mqtt    | level=INFO ts=2020-06-08T08:46:13.189949763Z app=edgex-device-mqtt source=service.go:202 msg="Addressable edgex-device-mqtt doesn't exist, creating a new one"
edgex-device-mqtt    | level=INFO ts=2020-06-08T08:46:13.561568115Z app=edgex-device-mqtt source=service.go:96 msg="*Service Start() called, name=edgex-device-mqtt, version=1.1.0"
edgex-device-mqtt    | level=INFO ts=2020-06-08T08:46:13.561656827Z app=edgex-device-mqtt source=service.go:100 msg="Listening on port: 49982"
edgex-device-mqtt    | level=INFO ts=2020-06-08T08:46:13.563972112Z app=edgex-device-mqtt source=driver.go:279 msg="Create MQTT client and connection: uri=tcp://admin:public@<IP_ADDRESS>:1883 clientID=CommandResponseSubscriber "
edgex-device-mqtt    | level=INFO ts=2020-06-08T08:46:13.564307739Z app=edgex-device-mqtt source=driver.go:279 msg="Create MQTT client and connection: uri=tcp://admin:public@<IP_ADDRESS>:1883 clientID=IncomingDataSubscriber "
edgex-device-mqtt    | level=INFO ts=2020-06-08T08:46:13.585402414Z app=edgex-device-mqtt source=responselistener.go:52 msg="[Response listener] Start command response listening. "
edgex-device-mqtt    | level=INFO ts=2020-06-08T08:46:13.586319068Z app=edgex-device-mqtt source=incominglistener.go:54 msg="[Incoming listener] Start incoming data listening. "
edgex-device-mqtt    | level=INFO ts=2020-06-08T08:46:13.708961773Z app=edgex-device-mqtt source=service.go:127 msg="Service started in: 1.473119971s"
edgex-device-mqtt    | level=ERROR ts=2020-06-08T08:46:45.811020907Z app=edgex-device-mqtt source=managedprofiles.go:144 msg="retrieving DeviceResource - Device MQTT-test-device not found"
edgex-device-mqtt    | level=WARN ts=2020-06-08T08:46:45.811101685Z app=edgex-device-mqtt source=incominglistener.go:79 msg="[Incoming listener] Incoming reading ignored. No DeviceObject found : topic=DataTopic msg={"name":"MQTT-test-device","cmd":"va:49:mdb_simu_var1","va:49:mdb_simu_var1":"2"}"

Your profile dir config is wrong.  There is no profile yaml in "./"
[Device]
DataTransform = true
InitCmd = ""
InitCmdArgs = ""
MaxCmdOps = 128
MaxCmdValueLen = 256
RemoveCmd = ""
RemoveCmdArgs = ""
ProfilesDir = "./"
Please change to
[Device]
DataTransform = true
InitCmd = ""
InitCmdArgs = ""
MaxCmdOps = 128
MaxCmdValueLen = 256
RemoveCmd = ""
RemoveCmdArgs = ""
ProfilesDir = "/custom-config"
to match your volume mapping:
volumes:

db-data:/data/db
log-data:/edgex/logs
consul-config:/consul/config
consul-data:/consul/data
- ./mqtt:/custom-config

#################################################################
My Device Services MQTT
#################################################################
device-mqtt:
image: edgexfoundry/docker-device-mqtt-go:1.1.0
ports:
- "49982:49982"
container_name: edgex-device-mqtt
hostname: edgex-device-mqtt
networks:
- edgex-network
volumes:
- db-data:/data/db
- log-data:/edgex/logs
- consul-config:/consul/config
- consul-data:/consul/data
- ./mqtt:/custom-config
depends_on:
- data
- command
entrypoint:
- /device-mqtt
- --registry=consul://edgex-core-consul:8500
- --confdir=/custom-config
#################################################################
And my configuration.toml :
#################################################################
[Device]
DataTransform = true
InitCmd = ""
InitCmdArgs = ""
MaxCmdOps = 128
MaxCmdValueLen = 256
RemoveCmd = ""
RemoveCmdArgs = ""
ProfilesDir = "/custom-config"
[Logging]
EnableRemote = true
File = "./device-mqtt.log"
[Writable]
LogLevel = "INFO"
Pre-define Devices
[[DeviceList]]
Name = "MQTT-test-device"
Profile = "Test.Device.MQTT.Profile"
Description = "MQTT device is created for test purpose"
Labels = [ "MQTT", "test"]
[DeviceList.Protocols]
[DeviceList.Protocols.mqtt]
Schema = "tcp"
Host = "<IP_ADDRESS>"
Port = "1883"
ClientId = "CommandPublisher"
User = "admin"
Password = "public"
Topic = "CommandTopic"

So it should work?

Ye I have tried,
With ProfilesDir=/custom, I never load any profiles.

And no devices are registered in the API (http://localhost:48081/api/v1/device).
I prefer to use profiles in file for my use case.

I notice you are using consul.  It might load the old config from Consul.  Could you please try to remove this line from docker-compose file?
- --registry=consul://edgex-core-consul:8500
Or, try docker-compose down -v and restart again

I want to keep the consul registry.
So I try to remove all the containers and recreate it.
Just for my information, if you are not using consul, what can you use?

I meant just for verifying the issue, not removing that line permanently.
Even you remove all containers, the data is still in the volume, so you need to add -v flag for down.
docker-compose down -v
Or, you can verify whether the key/value store keeps the latest config you made.
ProfilesDir = "/custom-config"

You were right,
without the consul registry, it's WORKS!

So you have an idea why it doesn't work with?

Your old configuration has uploaded to Consul, so it loads configuration from Consul instead of the toml file each time.  You can clean up the containers and volumes by
docker-compose down -v
Or, you can manually modify the configuration form Consul's key/value store to
ProfilesDir = "/custom-config"

Thanks.
It works finally.
You save my day.
M.

no problem


Try to avoid the auto restart behaviour when resuming it

Many users reported us that the application is automatically restarted when it is resumed.
The memory warning is another case : the system warns us that there is no more available memory on the device and the client is going to be killed so it seems better to restart the client to reduce its memory usage and to avoid crash.
But there are other cases in which the application restarts whereas it is not expected.
With their help, we found out that the service (events polling thread) is killed or some other required fields are not initialised (e.g active sessions list) : the client detects that something is not valid in its fields and restarts to avoid crash.
I never got such behaviour on my devices (Nexus 6 - Android M, Samsung S3 - Android 4.2.2, Zenfone 550KL - android 5.01).
I suspect that there is a custom home launcher (or something similar) which kills service or stop the client in putting a pending activity to resume it.
I assume it appends when the device memory usage becomes high without being urgent.

By now, with the help of some users (thanks to them), we identified two use cases
1- The eventStreamService was released by the OS when the application is in background.
So, when the application is resumed, the activity might be created again (after a memory pressure) but the event stream is released : the client used to restart to manage this case
By now, the event stream is started again.
2- the application is resumed from the login activity.
It happens with some launchers or when the application activities are popped up by pressing the back button until to reach the device home page.
We already found out that Intent.FLAG_ACTIVITY_BROUGHT_TO_FRONT implies to display a temporary activity above existing one (it should be managed by now).
It seems there are other flags which might triggers invalid behaviours

It seems being fixed by now.


Update how-to-setup-ssl-on-mlc.md

Added section to cover import of certificate and key values into PowerShell variables.

@alexfcurtis : Thanks for your contribution! The author, @SerinaKaye, has been notified to review your proposed change.

@alexfcurtis : Thanks for your contribution! The author, @SerinaKaye, has been notified to review your proposed change.

@SerinaKaye - can you please #sign-off

@j-martens: I'm sorry - only the author of this article, @SerinaKaye, can sign off on your changes. But we do have an exception process - if you are on the Microsoft content or product team for this product area, you can ask the PR review team to review and merge it by sending mail to the techdocprs alias.

#sign-off


Improve dynamic output multiplexing performance

The dynamic output is currently the best performance option for dynamically altering the outputs of a stream. However, since this output is strictly fan-out, when the goal is to multiplex messages across the dynamic outputs we're forced to dispatch each message to every single output and rely on processors to filter down the correct messages:
output:
  dynamic:
    outputs:
      foo:
        http_client: ...
        processors:
          - bloblang: root = if type != "a" { deleted() }
      bar:
        kafka: ...
        processors:
          - bloblang: root = if type != "a" { deleted() }
      baz:
        sqs: ...
        processors:
          - bloblang: root = if type != "b" { deleted() }
      ...

As the number of outputs increases the performance hit grows linearly, even when the number of outputs that a message dispatches to is static. Since this dynamic dispatch capability is pretty much unique to Benthos it would be cool to take this output one step further and offer a high performance multiplexing solution.
One possibility is to expand the current implementation to include a check field with each output so that the routing can be determined before dispatch. However, this still means executing a bloblang query for each output for each message, carrying an overhead per-output.
If we really want to eliminate a performance cost of having a large number of outputs then we need the check to be made once, and it returns a static list of outputs to be dispatched to. As an example we could implement behaviour whereby if a message has a metadata field dynamic_output_dispatch specified then we parse it as a comma separated list of outputs to dispatch to:
pipeline:
  processors:
    - bloblang: |
        meta dynamic_output_dispatch = match type {
          case "a" => "foo",
          case "b" => "bar",
          case "c" => "baz",
          case "d" => "baz,buz",
        }
output:
  dynamic:
    outputs:
      foo:
        http_client: ...
      bar:
        kafka: ...
      baz:
        sqs: ...
      ...

Giving us a way to multiplex across a dynamic set of outputs with a single bloblang query.

Great writeup! I'm not sure this entirely works for us though.. say a new dynamic output gets posted, intended to consume messages with type == "z". Won't this require modification of that global processor, to add a case "z" => "new_output"?

Good point, prefixes or wildcards could work but I would need to restrict dynamic output names to a more limited character set. I was originally thinking we'd have an API for editing resources (as a separate body of work) and it would be a two step process: add the output ready for routing messages to, then update the processor resource used for routing to include the new output and logic for selecting it.
I'm not sure I've captured a dynamic resources API in an issue yet but we could consider it a precursor. I'm not sure if that would suit your flow though, it'd be nice to find a generalized solution that would only include registering the new output without having to hook anything else in.

I'm not sure if that would suit your flow though

If a dynamic resources API is the clearest way forward, we're all for it! We don't mind writing a little code to make changes to dynamic resources as well as outputs.

Shower thought: Maybe the cleanest way forward here would be to fork an existing HTTP multiplexer and use that as the basis for output routing? It would be a new output type called multiplexer and it would allow you to add/remove outputs and dispatch messages using the same rules as HTTP server endpoints, which would make it much easier to document.

Shower thought: Maybe the cleanest way forward here would be to fork an existing HTTP multiplexer and use that as the basis for output routing? It would be a new output type called multiplexer and it would allow you to add/remove outputs and dispatch messages using the same rules as HTTP server endpoints, which would make it much easier to document.


Why Does Not Work?

What needs to be done for this library to work?  Previously it was corrected with changes.  What needs to be done now?

Apparently, twitter scrappers have stopped working due to some #bannedByTwitter. You may check this repo for latest legal twitter scraper.

Apparently, twitter scrappers have stopped working due to some #bannedByTwitter. You may check this repo for latest legal twitter scraper.

Thank you for answer.


raw3-6 configs

added new configs
startup new nodes on raw3-6

includes https://github.com/ipa320/cob_robots/pull/534/files
FR: OK

@ipa-bnm this pull request will overwrite this commit https://github.com/ipa320/cob_robots/pull/534/commits/e15a5cbd37f2786900b91d1d210d266a4e531b91 , which one is the right one?

@ipa-nhg resolved by merge

FR: OK


Created a title and a Favicon

@willRigney  it looks like the favicon isn't quite working, do you mind opening another pull request with the fix for it? The title looks great!


Change plugin config to be specified as a list

This makes plugin configuration more similar to output configuration,
where we can specify multiple plugins as a list. The idea behind this is
that the Telegraf agent can handle the multi-processing and error
handling better than each plugin handling that internally. This will
also allow for having different plugin configurations for different
instances of the same type of plugin.

This will NOT be a breaking change. In essence, it will make the config look like this (see bottom for plugins change):
$ ./telegraf -sample-config -filter cpu -outputfilter influxdb
# Tags can also be specified via a normal map, but only one form at a time:
[tags]
  # dc = "us-east-1"

# Configuration for telegraf agent
[agent]
  # Default data collection interval for all plugins
  interval = "10s"
  # Rounds collection interval to 'interval'
  # ie, if interval="10s" then always collect on :00, :10, :20, etc.
  round_interval = true

  # Default data flushing interval for all outputs. You should not set this below
  # interval. Maximum flush_interval will be flush_interval + flush_jitter
  flush_interval = "10s"
  # Jitter the flush interval by a random amount. This is primarily to avoid
  # large write spikes for users running a large number of telegraf instances.
  # ie, a jitter of 5s and interval 10s means flushes will happen every 10-15s
  flush_jitter = "0s"

  # Run telegraf in debug mode
  debug = false
  # Override default hostname, if empty use os.Hostname()
  hostname = ""

###############################################################################
#                                  OUTPUTS                                    #
###############################################################################

[outputs]

# Configuration for influxdb server to send metrics to
[[outputs.influxdb]]
  # The full HTTP or UDP endpoint URL for your InfluxDB instance.
  # Multiple urls can be specified but it is assumed that they are part of the same
  # cluster, this means that only ONE of the urls will be written to each interval.
  # urls = ["udp://localhost:8089"] # UDP endpoint example
  urls = ["http://localhost:8086"] # required
  # The target database for metrics (telegraf will create it if not exists)
  database = "telegraf" # required
  # Precision of writes, valid values are n, u, ms, s, m, and h
  # note: using second precision greatly helps InfluxDB compression
  precision = "s"

  # Connection timeout (for the connection with InfluxDB), formatted as a string.
  # If not provided, will default to 0 (no timeout)
  # timeout = "5s"
  # username = "telegraf"
  # password = "metricsmetricsmetricsmetrics"
  # Set the user agent for HTTP POSTs (can be useful for log differentiation)
  # user_agent = "telegraf"
  # Set UDP payload size, defaults to InfluxDB UDP Client default (512 bytes)
  # udp_payload = 512

###############################################################################
#                                  PLUGINS                                    #
###############################################################################

[plugins]

# Read metrics about cpu usage
[[plugins.cpu]]
  # Whether to report per-cpu stats or not
  percpu = true
  # Whether to report total system cpu stats or not
  totalcpu = true
  # Comment this line if you want the raw CPU time metrics
  drop = ["cpu_time"]

@gotyaoi This is a change to the way that plugins are configured, I thought you would want to be in the loop on it.
Makes plugins configured in a way similar to outputs, which means that users can specify multiple plugin types with completely different configurations.

cc @pauldix @beckettsean @gunnaraasen this PR would change the way the config file looks but is NOT a breaking change

The only things I can think of is that for some plugins, like the CPU plugin, it may not make sense to have multiple copies, and that other plugins, like procstat, have already worked around this somewhat by having the plugin be a container, which controls a list of the actual things to check.
Granted if there are multiple plugins doing this, and I think there are, it may make sense to have a generic solution, but procstat and the others might need to be changed to take advantage of it.

@gotyaoi Agreed, I was hoping to get away from plugins like procstat and httpjson, which have to do a lot of internal threading and error handling. I think it would be better for the telegraf agent to handle the threading.
Even the CPU plugin, while simple, could have a use-case for multiple instances. For example, what if you wanted cpu_time* for cpu-total, but not for cpu0, cpu1, etc. Currently that's not possible, but with this change you could do that like this:
[plugins]

# Get everything from totalcpu
[[plugins.cpu]]
percpu = false
totalcpu = true

# Drop cpu_time* from percpu measurements
[[plugins.cpu]]
percpu = true
totalcpu = false
drop = ["cpu_time"]

This particular scenario is not very likely, but I think it would be better to have this flexibility.

Sure, if it's not a breaking change +1

Works for me

@gotyaoi This is finally completed :sweat_smile:
I made some significant changes, and namely I realized that almost all of the configuration merging had to be negated in order to accomodate specifying multiple plugins as a list. Since they are a list, there is not really a way to get it to understand when it should merge and overwrite a plugin or not.
I'm sorry that I have removed some of the power of the configuration changes that you made, but I would like to move Telegraf towards a configuration that allows for plugins to be lists, and for plugins to not need to deal with large amounts of parallelism internally.

No worries. My code made sense in a context where there is only one instance of a given struct type, and that is no longer the case. There are potentially a few issues I can see though. On mobile right now, but I'll add some comments when I get home.

Last thing wouldn't relate to a specific line: My changes sort of tried to move all the details from the Config to the Agent, once all the config files were parsed. This seems to go back to keeping around the Config with the details in it, which is fine, but then there's the matter of the Agent's config, which is copied into the Agent from the Config. So now the Config has an *Ast.Table hanging around in it's agent attribute, and the agent refers to itself for some values, and to the config for other values, which feels sort of strange. Perhaps the various attributes on the agent struct should actually live on the Config, and the agent just holds a Config, which it refers to for details on what to do. This even enables a potentially nice path to config reloading in future, simply by building the new config in the background, then subbing out the Agent's Config attribute in one go.

@gotyaoi I'll make changes to have the agent config be held within config, I think that makes more sense


stop running tests on travis

tests are running on azure pipelines now
travis only does deploy
follow-up to #939 to complete #232
TODO:

[x] regenerate pypi encrypted creds for new repo, but I don't seem to have this. I see that @betatim last committed them. I'm not 100% sure this is necessary, but probably good to do now rather than wait for a release to fail.

closes https://github.com/jupyterhub/team-compass/issues/322

Hm, Azure hasn't run on the new repo, might need to poke something on pipelines

Had to update azure devops config to create a new project for the relocated repo. I guess azure doesn't link github permissions?
It's working now, but maybe we want to try out github actions on this repo?

closes https://github.com/jupyterhub/repo2docker/pull/940

Which issue/PR did you want to refer to with "closes #940"? Because 940 is this PR :D noticed after adding it to the top comment so it would auto close

After moving the pipeline do we have to re-add people so they have permissions to restart/poke around the pipeline? I can't find the button to restart the "unit" job and I think it is because I lack permissions but then I seem to always take a lot of time to find the restart button :-/

Oops! I meant https://github.com/jupyterhub/team-compass/issues/322. Fixed.
Not sure what's up with the repo2docker builds. Maybe I didn't set up the pipeline correctly? I opened #941 to figure that out.

After moving the pipeline do we have to re-add people so they have permissions to restart/poke around the pipeline?

Yes, that's why I opened #941. What email should I add for you?

<EMAIL_ADDRESS>but I think moving to GH action as in #942 is better


Connect EV indicator (because cheap tariffs are ahead)

Is your feature request related to a problem? Please describe.
We have a dynamic electricity contract, which means our charging tariffs vary significantly (usually between 15Ect/kWh and 30 Ect/kWh, sometimes beyond). Whenever tariffs are low (e.g. raw price around or below zero), I would like to charge our EV. It would be great if the SmartEVSE charger can indicate that cheap tariffs are ahead (in time) by showing a (for example) blue LED switch when we arrive home (and the EV is not yet connected and access is blocked). My home domotica system already knows the dynamic prices and can set a flag via MQTT (and possibly REST), so SmartEVSE code can stay simple.
Describe the solution you'd like
I'd like the SmartEVSE Switch/LED to show a blue color when I'm arriving home and the CheapTariffIndicator has been set by my home domotica system. Afaik, blue is not used to indicate anything yet (but open to improvements).
MQTT proposal

smartEVSE/Set/CheapTariffIndicator: unix time
smartEVSE/CheapTariffIndicator

Whenever present time is before the CheapTariffIndicator, the LED turns blue when no EV is connected and the SmartEVSE is blocked.
Describe alternatives you've considered
Look at the Electricity prices app (e.g. Tinder, Coolblue, etc) is a bit of a habit for me, but not for my spouse...
Additional context
Check out my psuedo-code additions in my own fork of this repository
Note: I'm not a github expert, usually don't program in C++. The modified code in my repository compiles ok, but I haven't tried it, because I would really appreciate @dingo35 feedback and suggestions on my modifications.

Maybe a MQTT command to set a custom RGB color to set the LED color in each possible mode: normal, smart, solar, off.

I'm not sure if we should change/override colors while charging, because those are listed in the manual. But a generic MQTT/REST command to set the color in Off mode is also a possibility.
What do you think about the time aspect I proposed above? The unix time which indicates the time until the color is set, before reverting back to the standard color (Off in my proposal).

I think external software should set the color based on "whatever", time in your case. It shouldn't be logic in the SmartEVSE.

I dont understand why you would task the end user with this; if your domotica system signals low prices coming up, just let your wifi plug in the EV as always, and let the domotica system set a delayed charge through the API?

My domotica is set up to always charge the EV when it is plugged in, using the lowest rates before the next time it will be used (usual departure time is 7am). However, it is not necessary to plug in and charge every day - the car has enough range to charge every few days. We don't want to charge when it is not needed and the prices are (relatively) high. However, we do want to charge when the prices are low.
For our EV, there is no easy API to read the SOC from, and even if that were available, the domotica doesn't know what the required range is for the next day. Hence, the approach to always charge when plugged in.
Example: This Monday the car had plenty of range left, but the overnight raw prices dropped to zero Ect/kWh - therefore it is advantageous to charge overnight.
Does that clarify?

I'm afraid this is too specific.
Although i do like the idea of customizable LED colours. Some people might want to mimic a public charging stations status led colors.

OK, good feedback here. Let's generalize the proposal in a way that people can control the color of the LED (and I can control it in the way that I'm thinking off). Do you propose to make the color configurable during all modes? Or limit it to a controllable Off mode color?

@devdems : I don't have access to the SOC and I can't configure the car to stop charging at a SOC %%, but you are right, there are more solutions to my problem. :-)
Anyway, I've implemented a more generic approach to set the color of the LED in Off mode in my fork. You can send the RGB color in off mode via MQTT (and it will publish the setting via MQTT). It compiles, and it works. I left out the integration in the web UI and /settings json for the time being.
Welcome to feedback and improvements.

Setting up custom RGB sounds useful. I can see it useful to set it up so it notifies me if I'm over my max kW in the tariff block before I run any heavy equipment in garage.
We just need to be careful that the change has no impact on the "error" RED light which needs to be prioritized.

Agreed, you can only modify the color of the 'Off' mode: (Access_bit == 0 || State == STATE_MODEM_DENIED)
These colors default to 0,0,0 (off). Since I only modify the off mode color, none of the other colors should have been affected.

@jonesPD I'm trying to keep the REST API and the MQTT API in sync; could you add your mod to the REST API and make a PR out of it?

If you ask me I would leave the option available on all modes (Off, Normal, Solar and Smart). If it's only on "Off" then you are limiting the use-cases where this could be useful like in the example that @mstegen wrote. For example, I never have the charger in Off mode but always in Smart.
But if it's "error" it should always blink red regardless of what you are sending to the charger via REST/MQTT.

@devdems : I can prepare a proposal with similar endpoints to control the color in Normal, Solar and Smart modes.

Pull request added.
I'll create a proposal for similar endpoints for the normal/solar/smart modes.
Suggestions/ideas welcome.

Additional pull request for the Normal/Smart/Solar modes.

I solved this for myself with ioBroker (like Homeassistens, NodeRed,...). He knows when the electricity tariff is cheap.
Normally the SmartEVSE is in solar mode, when the price is below a certain value the ioBroker sends an API command to switch to smart mode, if the value increases there is an API command for solar mode.
This way I can see when I get home whether the electricity is cheap or not.


Conflict with other Eigen Dependents Template Libraries

I noticed conflict happens when RandEigen is used alongside another library which also uses eigen.
The conflict only happens with the definition of plgamma
<command-line>: note: this is the location of the previous definition
In file included from /usr/include/eigen3/unsupported/Eigen/SpecialFunctions:47,
                 from /usr/include/eigen3/unsupported/Eigen/CXX11/Tensor:30,
                 from /opt/ros/noetic/include/pinocchio/fwd.hpp:31,
                 [...]
/usr/include/eigen3/unsupported/Eigen/src/SpecialFunctions/SpecialFunctionsPacketMath.h:19:8: error: redefinition of ‘template<class Packet> Packet Eigen::internal::plgamma(const Packet&)’
   19 | Packet plgamma(const Packet& a) { using numext::lgamma; return lgamma(a); }
      |        ^~~~~~~
In file included from /usr/local/include/EigenRand/RandUtils.h:15,
                 from /usr/local/include/EigenRand/Core.h:16,
                 from /usr/local/include/EigenRand/EigenRand:17,
                 [...]
/usr/local/include/EigenRand/MorePacketMath.h:181:30: note: ‘template<class Packet> Packet Eigen::internal::plgamma(const Packet&)’ previously declared here
  181 |   EIGEN_STRONG_INLINE Packet plgamma(const Packet& x)
      |                              ^~~~~~~

Once plgamma is commented, it compiles just fine. (Also removing the comments and compiling a second time does not
raise any error. The error comes for the first build or after a make clean).
I tried to avoid circular declaration in my own code, it might be an error from my side but I reported it just in case.

CXX compiler: GNU 9.3.0
cmake version 3.16.3
Eigen 3.3.7
RandEigen version 0.3.2

Hi @MehdiN,
Thank you for your reporting.
I didn't know there is another plgamma in Eigen's SpecialFunctionsPacketMath.h.
I'll rename my plgamma for preventing conflicts in the next update.

Solved in version 0.3.3.


Update it.coffee

alcune parole più

Hi, @javatlacati! Thank you for contributing. Could you sign the CLA so that I can merge your pull request?

Thanks.


Update and Fix strings

The last settings string pull request would not build so it was removed.  Does this one build ok ?

I'm sorry but i can not understand where the error is, you can tell me?

It was in settings package and the other dev removed it now so I'm not sure


I just wanna upload a new version of my site...

I'm a bit confused as to why I would want to manage a copy of my repository on the production server.
There's no way for us to really run the build as part of the deployment process, the build has nasty dependencies and must be run locally but we'd like to track what commits get deployed and log them somewhere, is it possible to put information about a different git repo in the site/deployments directory?
How would you otherwise facilitate this? I really just want to deploy a zip file with the new version of site but then log that a deployment did occur with information about the git revision from our repo in VSTS.
I currently just view Kudu as a nice little RESTful API for managing content and running commands on the server...

Take a look at the deployment API: https://github.com/projectkudu/kudu/wiki/REST-API#deployment. It has a flavor to just 'Add a deployment status to history', which does what you're asking. In fact, we created this for VSTS so they can call that in scenarios where they do the build.

@davidebbo Thank you for the link, I really appreciate you taking the time to answer.
I have so many questions I need answers to and I cannot find any technical in depth documentation to help me.
...but what happens when I push a deployment through the API, does that trigger some other behavior? Can the ID be anything I want? I feel as if the only way to answer many of these questions is to experiment and read the Kudu source code which is a really tedious task. Not that I won't do it but I was expecting some guide to go with the APIs, on why they exist and to what end for example the VSTS team is using the APIs and how. Things that help me get moving faster.
Like the file system layout, the documentation doesn't reflect everything I currently see when browsing the server. I also have a ticket with Azure support because I'm seeing a file from a different app hosted on the same app service plan, I thought those things should have been isolated from each other. (this is probably an issue with Azure and not Kudu)
I'm also really curious about the motivations behind Kudu? Why was it created, why does it exist? Will it continue to exist? What problem was it that you wanted to solve?
From my point of view, I see Kudu having an appeal if you have a "simple" app with a "simple" deployment process but beyond that, I'm a bit confused. There doesn't seem to be a straight forward guide on how to just put some files on an app service through the API. I've rewritten some of the Kudu toolings to help with this.
Though the APIs and power tools for managing and troubleshooting the service are great, as soon as you figure out what knobs to turn on in the Azure portal.

@davidebbo to give you an example
I have feeling a lot more is going on here than what's obvious from the API.
https://github.com/projectkudu/kudu/blob/2ea953e532d0a44afe291dc4b9e370df9fc51686/Kudu.Services/Deployment/DeploymentController.cs#L90
So if I HTTP PUT a deployment to this endpoint am I expected to have uploaded the actually files for deployment to some specific place for this to work? From where I'm standing all of this is undocumented and a complete black box.

@davidebbo actually wait, I think I got it now.
https://github.com/projectkudu/kudu/blob/2ea953e532d0a44afe291dc4b9e370df9fc51686/Kudu.Services/Deployment/DeploymentController.cs#L108
Looks very much like nothing really happens if we hit this branch.

That's correct. In the 'Add a deployment status to history' scenario, the only thing it does is populate a history item. It is completely detached from the actual deployment of files. It is designed to let you do your own deployment by whatever technique, and then populate the history yourself for tracking purpose.
And yes, the docs could be better. I've added clarifications to the wiki.


[Bug] Table content not  without vertical scroll or limited vertical scroll

Description
When creating queries and clicking Run, the resulting table underneath has no vertical scroll.
When add widgets from queries with large amounts of fields, not all fields show up.
Both of the issues are because the .table element-style min-width is not calculated correctly (in the former case it is 0px and in the latter it is something but too small); is there a way to change the calculation easily?
Steps to reproduce the behaviour:
Add steps to reproduce this behaviour, include console / network logs & screenshots
Create a query with a lot of fields (over 20) and click Run, you only see a few. Then + Add Widget and test there, you see more (as you can scroll now) but still not all (and the last one is usually cut-off).
Important Details

Version: Self-hosted
OS: Linux/server
browser: Chrome/Safari (it's worse under Safari but we hardly use that anyway)

It seems columncount * 150px works well for me, so as a bandaid:
setInterval(function(){ if (document.location.href.indexOf("edit/queries")>0) { var table = document.getElementsByClassName("table")[0];  if (table) table.style["min-width"] = (table.firstChild.children.length*150)+"px"; } else {var tables=document.getElementsByClassName("table"); if (tables)for(var i=0;i<tables.length;i++) tables[i].style["min-width"] = (tables[i].firstChild.firstChild.children.length*150)+"px";}}, 1500);

@tluyben thank you for reporting these! We'll ship a fix quickly

@tluyben tested this on a few devices and needed a little clarification here
Is your trouble with the horizontal or vertical scroll in the query pane? We're only seeing an issue in the horizontal scroll.
In the table widget, I can see the data is getting paginated so there is no vertical scroll, you can instead go to the next page as indicated in the image.  Was this the confusion?

Please share screenshots if this does not solve it

Oh! It was early for me; I meant horizontal ; fixing this issue. Apologies. The issue is not vertical, it's horizontal.

@tluyben thank you! We'll fix this asap and hope the issue in the widget is clear that you can access the next page from the top right section. Thank you for the contribution 😄


Bengoro6661

Signed-off-by: Village31<EMAIL_ADDRESS>

Unsubscribe
On Thu, Jan 4, 2024 at 6:30 AM Ghostman2591 @.***>
wrote:

@.**** approved this pull request.
—
Reply to this email directly, view it on GitHub
https://github.com/github/haikus-for-codespaces/pull/170#pullrequestreview-1804070467,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/BBB3FX42LQWUVBRW2TSUSPLYM2OFTAVCNFSM6AAAAAARXWPSKOVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMYTQMBUGA3TANBWG4
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>


Consistent method name casing

I really like the fact that LWJGL offers a lot of bindings to various useful libraries,
but there's one thing that drives me crazy: The inconsistent casing of method names.
Look at the examples.

:white_check_mark: glClearColor(...)
:white_check_mark: glfwDefaultWindowHints()
:white_check_mark: alDeleteBuffers(...)

All of them are lowerCamelCase and comply with the Java Conventions. :+1:
They consist of the prefix of the library followed by the name of the actual method.
Awesome, but what about for example STB and Nuklear?

:confused: stb_vorbis_open_memory(...)
:confused: nk_label_wrap(...)

They follow the same general "layout", but in snake_case.
Well, I get the point that these are just bindings to C/C++ libraries, but still:
Why not rename the methods? In all the cases I found you can simply split the name by _ and capitalize all parts except the first one. Then the examples would look like this:

:white_check_mark: stbVorbisOpenMemory(...)
:white_check_mark: nkLabelWrap(...)

What do you think about it?

I think the goal was never to make the method names consistent with the Java conventions. (it's a mere conincidence that the OpenGL function names comply with that).
The goal has been to have the names identical to their native function names, which LWJGL provides bindings for, so as to simplify porting C/C++ code to LWJGL/Java.

Java conventions are followed for class names only (including struct and callback classes).
The original symbols are used for all constants and functions. This is not an inconsistency in LWJGL, but a deliberate design decision. The idea is that new users looking for tutorials are much more likely to find resources written in native code, rather than in Java/LWJGL. Being able to copy/paste C calls inside your Java code, with minimal porting, is an explicit goal of LWJGL. That's why APIs like OpenGL are completely static, when every other Java library exposes OpenGL via instance methods. It's easier to do that (because of the way GL contexts work), but breaks the aforementioned benefit.
With that said, you're welcome to fork LWJGL and change its generator to do exactly what you propose. It should be trivial, but let me know if you need help with it.

@httpdigest Many developers use LWJGL, because they prefer Java over C (or C++). One (probably trivial sounding) good thing about Java is, that - unlike with other programming languages - developers actually care about coding style and conventions - including casing of variableNames, methodNames and CONSTANT_NAMES (static finals).
Just because the methods I'm talking of are "just" bindings they are still being use inside actual Java code and it just makes the code look a lot less clean.

@Spasi Mmhh, what about allowing both?
Due to the fact that many developers use static imports to use the methods we could have two classes:

A "normal" one with no changes to the method names
A class with a special suffix (like "J" for "Java" or "CC" for "CamelCase")

Developers would be able to decide what they want then:

Be able to copy-paste C code without renaming the methods
OR: Have a clean code by renaming the methods

I'm afraid being friendly to Java developer aesthetics is not a priority for LWJGL. LWJGL is low level, hard to use, designed for experienced developers that know what they're doing. It's also designed to be easy to wrap in higher-level frameworks and engines (see libGDX etc), which are welcome to expose the functionality in a friendly manner.
Generating twice as many classes is not an option. LWJGL is already massive and complex as it is.

@Spasi Fine, I give up.


Default database type in V4.0.1 changed from lmdb to hash?

It looks like the Postfix default database type changed from lmdb to hash in the new V4.0.1 release. Whe´re using the HELM chart to deploy to K8s.
Whe´re using a transport_maps entry transport_maps: "lmdb:/etc/postfix/recipient_dependent_relayhost" which is not working any more in the new release.
When i add postfix.default_database_type: "lmdb" to the values.yaml  it works as before.
There is also a Warning in the logs which i did not spot in older releases:
warning: inet_protocols: disabling IPv6 name/address support: Address family not supported by protocol

Can you please open a separate ticket for inet_protocols: disabling IPv6 name/address support and add your exact configuration so I can replace this, as I have not encountered this in my testing.


Style sidebar scroll bar for dark mode

Thanks for the fantastic project. I've been using it heavily and hope to contribute some code when I have time.
For now, the scroll bar could look a lot better when using dark mode. I've attached a screenshot to illustrate the issue.
Screenshot was taken in Google Chrome.

Fixed #433

Fixed #433

Thanks so much!


cmd/compile: dead code-elimination should work for sync.OnceValue

This is special case of #14840, but identified as a potential source of special-cases if the general case is hard to solve.
Compiling the following program:
package main

import "sync"

var initializeVariable = sync.OnceValue(func() string {
    return "THIS_IS_DEAD_CODE"
})

func main() {}

should not link in logic for initializeVariable since the variable is dead.
This also applies to OnceFunc and OnceValues.
At present, OnceValue is unfortunately not a drop-in replacement for for sync.Once for initializing global variables.

CC @dr2chase who has been thinking about function purity analysis, which may be related.

A related optimization we should consider is detecting that the package variable is never modified and then applying inlining at sites that call the function bound to it. We do this for closures bound to local variables, but not at package scope. (This might be a different issue, or could be thought of together with this one.)


Docker task, Build Context missing info

In the page:
https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/build/docker?view=vsts
for the property Build Context, add info explaining its value is ignored unless Use Default Context is set to false (default is true)
Also, Use Default Build Context should be Use Default Context (already reported in this issue, which is now closed but it seems it has not been fixed).

@shashankbarsin just to comment in here, as there's still some vagueness here:
buildContext docs currently state:

(Required if useDefaultContext == false) Path to the buildContext.

But don't clarify where the path is relative to. For example, should I assume that the buildContext is relative to the directory of the Dockerfilespecified, or from the location of theazure-pipelines.yml` file or from the base build directory?
Some clarity here would also be helpful.

I have also been confused by this. Having the following structure:

folder 1
| ----------  Dockerfile
folder 2
| ----------  Dockerfile
folder 3 (files manipulated by Dockerfile)

When I ran a Docker task, I was unable to retrieve contents from folder 3 before specifying the buildContext property, which is relative to the azure-pipelines.yml.
buildContext: . worked for me.

Is there any update of this? I m tryng to use a dockerfile from one repo , and send the context of other repo to the docker task
--------Yaml---------

checkout: docker-repo
path: docker-repo

checkout: self
path: self

task: Docker@1
displayName: Build image
inputs:
command: build
azureSubscriptionEndpoint: $(azureSubscriptionContainerRegistry)
azureContainerRegistry: $(azureContainerRegistry)
dockerFile: $(Agent.BuildDirectory)/docker-repo/Docker/Dockerfile
imageName: $(Build.Repository.Name):v$(Build.BuildNumber)
buildContext: $(Agent.BuildDirectory)/self

Im getting an error on the build context . It seems that the task is not seing my /self repo (where the context is)


icetime using icebox PREFIX on runtime

I think it would be useful to allow loading icebox data path during runtime. The binary would be a more flexible binary.

I agree. Added icetime -C <chipdb-file> in commit 9ab00ea.

Awesome. Thanks!


Reproduce custom width demo in codesandbox

Hello,
I'm trying to reproduce the custom width demo in codesandbox, but I am unable to get slider 2 to be between slider 1 right portion and slider 3 left portion:
https://codesandbox.io/s/y2x56plokx
What I'm getting is only slider 1 right portion with slider 2. The demo is working as expected.
I'm lost with root style, container style and slider style.
Thank you in advance for your help.
For info, the demo example link doesn't work in codesandbox because SupportTouch dependency is missing.

this problem still exists, any solution?


KartaView images not shown completely when resizing+zooming

URL
https://www.openstreetmap.org/edit#map=19/59.40386/17.93796
How to reproduce the issue?
Go to iD, enable KartaView photo overlay, open any KV image, change the size of the image window (from the top right corner, make it bigger), zoom in and then try to move the image: you'll see that you can reach the top of the image and the left side, but you can't reach the bottom and the right side of the images. It's like if the image gets cropped on two sides, the right and the bottom.
Screenshot(s) or anything else?
Watch video at: https://odysee.com/@Finix:d/kvbug022022:c
Which iD Editor versions do you see the issue on?
Released version at openstreetmap.org/edit
Which browsers are you seeing this problem on?
Firefox

I am also experiencing this issue. But it doesn't seem to be browser-specific. I tested with brave and firefox.

This issue still persists. It seems that the image is cropped every time the outer frame is moved, or the image is zoomed in, which is not the expected behaviour.


Migrate to config

afaik it isn't possible to move docker-hosted service definitions to traefik config files (one reason being that the docker network assigns a new ip to each service every time it spins up.


feat(nsis): support choosing compression algorithms

What kind of change does this PR introduce?

[ ] Bugfix
[x] Feature
[ ] Docs
[ ] New Binding issue #___
[ ] Code style update
[ ] Refactor
[ ] Build-related changes
[ ] Other, please describe:

Does this PR introduce a breaking change?

[ ] Yes, and the changes were approved in issue #____
[x] No

Checklist

[x] When resolving issues, they are referenced in the PR's title (e.g fix: remove a typo, closes #___, #___)
[x] A change file is added if any packages will require a version bump due to this PR per the instructions in the readme.
[ ] I have added a convincing reason for adding this feature, if necessary

Other information
Request from #7685
This is for dev branch.
The nsi template is slightly different from the 1.x branch (#7776) because the SetCompressor had been moved below the !define section

Since 1.x will be merged into dev, I will go ahead and close this PR and focus on the other one


Job arguments not coming through correctly

I deployed my job using sasjs. Now scheduling it from SAS Studio by going to the job and right-click: Schedule Job.
When I open the Job Properties in environment manager, it states thet _addJesBeginEndMacros is not set (or set to true), while it should be set to false (in line with the deployed job settings)

Hi @mblauw-sas,
Thanks for raising this.  After exploring the REST APIs for both Viya 3.5 and 4, and reproducing the above behaviour in Viya 3.5, I believe this is a SAS Bug.
The documentation is very sparse on the matter, but it seems to me that when you create a job definition you provide a bunch of parameters along with default values.
Here is an example of a SASjs generated job:
{
	"creationTimeStamp": "2022-07-13T11:02:47.228Z",
	"modifiedTimeStamp": "2022-07-13T11:02:47.228Z",
	"createdBy": "allbow",
	"modifiedBy": "allbow",
	"version": 2,
	"id": "5bd61d5d-deb1-49bc-8c5a-78a1a0880abc",
	"name": "makedata1",
	"type": "Compute",
	"parameters": [{
		"version": 1,
		"name": "_addjesbeginendmacros",
		"defaultValue": "false",
		"type": "CHARACTER",
		"required": false
	}],
	"code": "* Job Variables start;\n* Job Variables end;\n* SAS Macros start;\n%macro example(msg);\n  data work.example;\n    msg=symget('msg');\n    putlog msg=;\n  run;\n%mend example;\n* SAS Macros end;\n* SAS Includes start;\nfilename FREF1 temp;\ndata _null_;\nfile FREF1 lrecl=32767;\nput 'proc sql;';\nput 'create table &mylib..demotable1(';\nput '        tx_from num not null format=datetime19.3';\nput '        ,tx_to num not null format=datetime19.3';\nput '        ,vara varchar(10) not null';\nput '        ,varb varchar(32) not null';\nput '    ,constraint pk_demotable1';\nput '        primary key(tx_from, vara));';\nrun;\nfilename FREF2 temp;\ndata _null_;\nfile FREF2 lrecl=32767;\nput '/* this is just to illustrate possibilities for SAS Program inclusion */';\nput 'data work.append;';\nput '  if 0 then set &mylib..demotable1;';\nput 'infile cards dsd;';\nput 'input tx_from';\nput '  tx_to';\nput '  vara : $char.';\nput '  varb : $char.';\nput ';';\nput 'datalines4;';\nput '0,999999,A,B';\nput ';;;;';\nput 'run;';\nrun;\n* SAS Includes end;\n* Binary Files start;\n* Binary Files end;\n* JobInit start;\n%example(Job Init is executing!)\n%let mylib=WORK;\n* JobInit end;\n* Job start;\n%let pw={sas002}asdfasdf;\n%example(MakeData1 is executing)\n/* these file refs are configurable above */\n%inc FREF1;\n%inc FREF2;\nproc append base=&mylib..demotable1 data=work.append;\nrun;\ndata _null_;\n  rc=sleep(10);\nrun;\n* Job end;\n* JobTerm start;\n%example(Job Term is executing!)\n* JobTerm end;",
	"links": [{
		"method": "GET",
		"rel": "self",
		"href": "/jobDefinitions/definitions/5bd61d5d-deb1-49bc-8c5a-78a1a0880abc",
		"uri": "/jobDefinitions/definitions/5bd61d5d-deb1-49bc-8c5a-78a1a0880abc",
		"type": "application/vnd.sas.job.definition"
	}, {
		"method": "GET",
		"rel": "alternate",
		"href": "/jobDefinitions/definitions/5bd61d5d-deb1-49bc-8c5a-78a1a0880abc",
		"uri": "/jobDefinitions/definitions/5bd61d5d-deb1-49bc-8c5a-78a1a0880abc",
		"type": "application/vnd.sas.summary"
	}, {
		"method": "PUT",
		"rel": "update",
		"href": "/jobDefinitions/definitions/5bd61d5d-deb1-49bc-8c5a-78a1a0880abc",
		"uri": "/jobDefinitions/definitions/5bd61d5d-deb1-49bc-8c5a-78a1a0880abc",
		"type": "application/vnd.sas.job.definition",
		"responseType": "application/vnd.sas.job.definition"
	}, {
		"method": "DELETE",
		"rel": "delete",
		"href": "/jobDefinitions/definitions/5bd61d5d-deb1-49bc-8c5a-78a1a0880abc",
		"uri": "/jobDefinitions/definitions/5bd61d5d-deb1-49bc-8c5a-78a1a0880abc"
	}],
	"properties": []
}

By contrast, when you set up a job request, which is associated with a job definition, you can provide arguments.
It seems that Environment Manager does not honour the defaults in the parameters of the job when creating the arguments of the new job request.
I've also reproduced this by creating a job using SAS Studio, with _addJesBeginEndMacros set to false, scheduling, and opening in Environment Manager.  Same behaviour, so it's nothing to do with SASjs.
I'll raise a track.

Thank you for looking into this Allan! I have also sent this to our project team such that they are aware.

Ok!  I got a response on the ticket, they are able to reproduce.  Are we ok to close this issue?

Yes, I will close. Thanks!

Leaving the SAS note here:  https://support.sas.com/kb/69/540.html


Permission issue when updating docker...

RancherOS Version: (ros os version)
1.0.3
Where are you running RancherOS? (docker-machine, AWS, GCE, baremetal, etc.)
Openstack.
Installed the Rancher OS fine. Log in as rancher by certificate...
 Is the docker daemon running?

yup. 17.06 doesn't work with any released RancherOS, I've a patch to fix it, but its not released yet.

Cool :)

RancherOS Version: (ros os version)
1.5.1
sudo system-docker version
Client:
Version:      17.06-ros6
Tried setting sudo ros engine
docker-18.09.1
docker-18.09.2
docker-17.12.0-ce
docker-17.12.1-ce
Where are you running RancherOS? (docker-machine, AWS, GCE, baremetal, etc.)
baremetal
RancherOS loads and I can login
sudo system-docker ps
Shows the ros user-docker machine stuck in a reboot loop, but if I try to just run a regular docker user command it throws the error Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.38/containers/json?all=1: dial unix /var/run/docker.sock: connect: permission denied

@MikePadge Could you check /var/log/docker.log after switching the engine.
I cannot reproduce:
[rancher@ip-172-31-14-0 ~]$ sudo ros -v
version v1.5.1 from os image rancher/os:v1.5.1
[rancher@ip-172-31-14-0 ~]$ sudo ros engine switch docker-17.12.1-ce

[rancher@ip-172-31-14-0 ~]$ docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 17.12.1-ce

@niusmallnan
Failed to create API server: Could not load X509 key pair (cert: "/etc/docker/tls/server-cert.pem", key: "/etc/docker/tls/server-key.pem"): open /etc/docker/tls/server-cert.pem: no such file or directory
Which is in line with what I found earlier. Declaring the following in my config didn't actually create the certs.

docker:
tls: true
tls_args:

"--tlsverify"
"--tlscacert=/etc/docker/tls/ca.pem"
"--tlscert=/etc/docker/tls/server-cert.pem"
"--tlskey=/etc/docker/tls/server-key.pem"
"-H=<IP_ADDRESS>:2376"

In my config didn't actually create those certs.
Here's my full config

Cloud-Config
hostname: P3X-984
rancher:
docker:
tls: true
tls_args:
- "--tlsverify"
- "--tlscacert=/etc/docker/tls/ca.pem"
- "--tlscert=/etc/docker/tls/server-cert.pem"
- "--tlskey=/etc/docker/tls/server-key.pem"
- "-H=<IP_ADDRESS>:2376"
storage_driver: overlay2
network:
interfaces:
eth0:
address: <IP_ADDRESS>/24
gateway: <IP_ADDRESS>
mtu: 1500
dhcp: false
dns:
nameservers:
- <IP_ADDRESS>
- <IP_ADDRESS>
ssh_authorized_keys:

ssh pubkey

@MikePadge  Did you try this? https://rancher.com/docs/os/v1.x/en/installation/configuration/setting-up-docker-tls/

@niusmallnan those configuration settings have always worked, I was under the impression, I could set them in the cloud-config, and have them auto apply to at least the localhost, but I see now that's not actually declared anywhere in my cloud-config.
Thank you.


unable to verify if cc is installed

As the below, what's wrong with it, thank you!

Hey @freemking,
Sorry, this version is still under development and currently not working.
Use our last working version : heroes-service.v1.0.5
Have a great day !
Maxime


feature: save and print notification history

Did you check the docs?

[X] I have read all the snacks.nvim docs

Is your feature request related to a problem? Please describe.
nvim-notify has this useful option to view history of all previously shown messages, https://github.com/rcarriga/nvim-notify#viewing-history.
Describe the solution you'd like
Print them in the similar fashion as nvim-notify, using either a command or just a method on Snacks.notifier.
Describe alternatives you've considered
Probably some monkey patching, but I'd like to avoid that.
Additional context
No response

I agree. Definitely important to be able to see history. Right now I will continue to use mini.notify but when history option is added to Snacks.notifier I will switch over to this snack :-)
Initially I was staying with mini.notify due to LSP progress, but with that added today, the last thing left for me is the history :-)
Nice work as always Folke!


JU: scrape Power BI iframe

/rebase

@metaodi I guess this is one approach to do it. I've played around with getting the data via the POST JSON data in https://github.com/openZH/covid_19/pull/1846
It's not yet complete, I haven't really figured out all the responses / types (sometimes the data seems to be computed).
As far as I can tell your scraper should be fine too - feel free to merge this one. Or I can also try to investigate a bit further on the other approach...

I went down the same rabbit hole, and decided to use the selenium approach. Maybe we can convince JU to share the underlying data...


[Feature Request]: Check cluster resource limits/quotas to predict if notebook sizes will actually spawn

Feature description
Currently users will get no notification if any of the provided notebook size options can spawn in the current cluster unless they attempt a spawn and it fails due to scheduling.   Since we already query the cluster for worker nodes to detect GPUs, we should query available 1) worker nodes resource limit and 2) notebook namespace resourceQuotas to predict if the given notebookSize is viable from a scheduling perspective
Examples:

If the largest worker node only has 8CPUs then we should add a note to the SpawnerUI to show that the Large notebook size (limit of 14cpu) is not schedulable
If the target namespace for the user notebooks has a resourceQuota of 4GB of memory then the Small, Medium and Large notebooks are not schedulable

There should be a toggle in the ODHDashboardConfig to disable this check in the event that the cluster admin allows for autoscaling of large resource nodes and wants to disable this restriction
Describe alternatives you've considered
This is a quality of life feature for users since the notebook sizes are completely controlled by the Dashboard admin.  They would have the responsiblity to modify the global settings when ODH is installed
Anything else?
JupyterHub SingleUserProfiles PR https://github.com/opendatahub-io/jupyterhub-singleuser-profiles/pull/187 that added this feature to the JH Spawner UI

Moving to closed - it has been marked stale for the last 2 months with no further updates.  We will handle this if it is comes up from a customer ask in the future.


Fixes for pytest-8.0.0 compatibility

Two fixes for compatibility with pytest 8.0.0:

Removed pytest.warns() inside pytest.raises() where no warnings are actually raised.
Cleared recorded warnings after pytest.warns() invocation where the function expected all warnings to be captured.

Explained in detail in commit messages.

[x] Closes #8681
[ ] Tests added
[ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
[ ] New functions/methods are listed in api.rst

Great first PR. Thanks @mgorny !

Thanks!


Text query on index built with image embeddings

Hi @rom1504,
Thanks for the great work on this retrieval system. I have a question about your implementation details. In README you have this sentence: image.index containing a brute force faiss index for images. I'm wondering if the image index behind the retrieval ui is a brute force index (flat index)? From the code looks like you used autofaiss, then I would expect the index to a quantized one.
I have this question because I'm also using autofaiss to build an index on 100 million images using CLIP embeddings. What I observed is that on this index, image query gives decent performance. However, text queries return very poor performance. This issue disappears when I switched to use a flat index. I think the underlying reason is that image and text embeddings span into disjoint clusters in the CLIP latent space, as discussed here: https://twitter.com/metasemantic/status/1356406256802607112
Building a quantized Faiss index somehow increased the gap between image and text embeddings and making the retrieval fail on text queries. I'm wondering if you also had this issue and then decided to use a flat index to overcome this issue? If so, I would expect an index with 400M images to be bulky and slow?
Thanks for your kind reply!

Hello,
good catch, image.index containing a brute force faiss index for images is not correct anymore. I used a flat index only for small indices at the beginning of this project. I will fix that in the readme
This tool is indeed using autofaiss and hence using approximate knn with a degree of approximation that depends on the memory constraint given to autofaiss.
You can see the difference of performance in https://rom1504.github.io/clip-retrieval/ by choosing the index on the left.
I have observed that indices built on clip text and image embeddings (400m of them) work well if using indices of size at least 16GB, and work very well for 64GB (I can give you the exact indices kind if you interested but https://colab.research.google.com/github/criteo/autofaiss/blob/master/docs/notebooks/autofaiss_index_selection_demo.ipynb should give you a good idea about this)
as you can check on https://rom1504.github.io/clip-retrieval/ all kind of queries work very well (text to text, text to image, image to text, image to image)
I have computed knn recalls for all indices, and the best recall I got are around 70% for the largest indices
Indeed using a flat index beyond a few million of points does not work. Approximate knn are needed and usually work very well
I am wondering if you used a too small index for your 100M embeddings experiment ?
I'm glad you're using autofaiss.
I'm curious about what are your plans ?
if you want to play locally with some large clip-embeddings indices, some big indices of laion400m are available there https://the-eye.eu/public/AI/cah/laion400m-indexes/

Thanks for such a prompt reply! I was using a 16G index and running autofaiss score_index shows '1-recall@20': 0.992. I can try to increase the index size to larger and see if that helps.
It's very helpful to know that your text queries work well on the image index. At least now I know it's possible to do. My other guess is that data in LAION-400M is cleaner. One their page (https://laion.ai/laion-400-open-dataset/) it says we calculate the cosine similarity of both embedding vectors and drop all samples with a similarity below 0.3.. I'm planning to try to filter my data with the same rule and see if that helps.

what value of 20-recall@20 do you get? that one is much harder and that's the one I was mentioning
did you set max_index_memory_usage to 16GB and not only current_memory_available ?
laion400m is indeed filtered to have matching text and image pairs, and that makes it a good dataset to train models such as clip or dalle
for this specific retrieval use case however, the text and image matching does not matter as the indices are built on the text part or the image part independently: you could build such indices with only text or only images

what value of 20-recall@20 do you get? that one is much harder and that's the one I was mentioning. Usually if you get more than 50% for 20-recall@20, you'll get pretty decent results did you set max_index_memory_usage to 16GB and not only current_memory_available ? (can you maybe share the index key that was chosen by autofaiss?)

My '20-recall@20': 0.6614999999999996.
index_key:"OPQ64_448,IVF65536_HNSW32,PQ64x8"
Yes I set max_index_memory_usage to be 16G and current_memory_available is 30G.

laion400m is indeed filtered to have matching text and image pairs, and that makes it a good dataset to train models such as clip or dalle for this specific retrieval use case however, the text and image matching does not matter as the indices are built on the text part or the image part independently: you could build such indices with only text or only images

Definitely makes sense!

ok, that score seems pretty reasonable
you said that "However, text queries return very poor performance", how did you measure that exactly? what results do you observe and what did you expect? what kind of dataset is it?

I'm fixing one thing and I'll get back to you with what I got. Thank you!

So I trained a larger Faiss index. It's again on the 100M common crawl images. This time the index is 32GB and 20-recall@20 reaches 0.899. I'm still observing that text retrieval quality is poor.
For example, when I search red car in your system (https://rom1504.github.io/clip-retrieval/?back=https%3A%2F%2Fknn.laion.ai&index=laion_400m_128G&query=red+car), it returns decent red car images.
When I search the same thing in my index, it returns these images as top 3:

https://lakisarah.files.wordpress.com/2020/12/screen-shot-2020-12-26-at-6.13.20-pm.png?w=1200
https://nicichiarasa.files.wordpress.com/2021/01/horse-and-car.jpg?w=676
https://static.wixstatic.com/media/eaff7f_20766f63569a4b3fb7bf7ab4f6d1d72f~mv2.jpg/v1/fit/w_1000%2Ch_744%2Cal_c%2Cq_80/file.jpg

The results seem to be dominated by images with text. My theory is that since the index recall looks ok, the problem is on the data: LAION data is cleaner since it's filtered with similarity threshold, and these kind of images are discarded.
Interestingly, if it's a flat index, the results are much more reasonable. In the case of red car, a flat index with the same set of images would return these images as top 3:

https://janemount.files.wordpress.com/2018/07/eca8a-36853937_250754579046897_3977563955785629696_n.jpg?w=640
https://i.pinimg.com/236x/e8/b2/bf/e8b2bf49c41a2edc4fa965d2e2cdf62c.jpg
https://graffiti-unlimited.com/system/image_mixeds/258950/original_jpg/bombing_1604178770979.jpg?1604178855

Looks like the Faiss quantization somehow degrades the text retrieval quality by quite a bit. This is also noticed when I query with a smaller LAION index: https://rom1504.github.io/clip-retrieval/?back=https%3A%2F%2Fknn.laion.ai&index=laion_400m_16G&query=red+car

interesting
can you try "a photo of a red car" on your index?

Yep I was already using "a photo of a red car". I also tried "red car", it didn't make it better lol.

quite interesting you're getting results significantly worse than laion400m
I guess then yeah give it a try to keep only image and text that match with a 0.3 clip threshold (since you already have the clip embeddings it should be fast to do)

ah one important point: is your dataset fully shuffled? we extract the training set for the index among the first files so if your dataset is not shuffled (and hence the first files are not representative of the whole set), it could have a negative impact

I used https://github.com/rom1504/cah-prepro/blob/main/deduplicate/cah_stats_spark.py#L83 to shuffle

I believe it's shuffled but I need to double check.
I've also noticed that text query quality on LAION degrades quite a big between 128G and 64G. For example:
https://rom1504.github.io/clip-retrieval/?back=https%3A%2F%2Fknn.laion.ai&index=laion_400m_128G&query=red+car
vs.
https://rom1504.github.io/clip-retrieval/?back=https%3A%2F%2Fknn.laion.ai&index=laion_400m_64G&query=red+car
From this I think the performance is sensitive to the compression ratio as well. I'll also train an 128G index and see what happens.

I'm not sure I would say it degrades much from this example alone
"a photo of a red car" gives very similar results with both indices for laion

Getting back to you with some new findings!
So I tried two things:

Train a larger Faiss index

Last time I was using a 26G index. This time I trained a 238G index (for some reason autofaiss cannot found an index size in between 26G and 238G). This time index_key:"HNSW32", compression ratio:0.88, '20-recall@20': 0.77.
The top 5 results for "a photo of a red car":
https://2.bp.blogspot.com/-7PV9IYXKzD8/WcZ-LZbj3uI/AAAAAAAAqZo/AKSsTtrKH2weKdWpfOG_dpVHpUcanOTjACLcBGAs/s640/4.jpg
https://timdobbsphotography.files.wordpress.com/2018/07/holga_e100vs_009.jpg
https://www.sloannota.com/wp-content/uploads/2021/01/Screen-Shot-2021-01-28-at-11.31.04-AM-sm.png
https://pocketdiarybudapest.files.wordpress.com/2016/09/ce7e0-62.jpg?w=640&h=426
http://russellfreeman.com/wp-content/uploads/2011/11/russ_colombia_renault_repair-228(pp_w639_h479).jpg
As compared to the 26G index:
https://reddwarfquotes.com/wp-content/uploads/2016/12/But-kippers-vindaloo.jpg
https://www.mppsociety.com/wp-content/uploads/2016/12/derpyfrs-scion-frs-ssr-ms3-4-640x427.jpg
https://sketchingwordscom.files.wordpress.com/2020/10/fullsizeoutput_8ee.jpeg
https://lakisarah.files.wordpress.com/2020/12/screen-shot-2020-12-26-at-6.13.20-pm.png?w=1200
https://brattleboroadventurecom.files.wordpress.com/2019/02/img_3466-1.png
This seems to be an improvement. However, when I search for "a photo fo a red guitar", from 238G index I got:
https://cdn.shopify.com/s/files/1/0410/9110/4924/products/sunbrella-canvas-jockey-red-5403_90f4df10-86f9-4d43-bb38-6f101636e87e_1024x1024.jpg?v=1616687617
https://couleursjapon.com/1807-home_default/tissu-chirimen-rouge-uni.jpg
https://www.eqsuk.com/images/products/838_1_590x504.jpg
From 26G index I got:
http://2.bp.blogspot.com/-1Dxh3M8bOuQ/VSs9IKaxWXI/AAAAAAAAFqI/G_vbHh0ULgw/s1600/footprint%2Bguitar.jpg
https://guitartwitt.com/wp-content/uploads/2021/06/20210619_200541_0000.jpg
https://www.divideandconquermusic.com/uploads/1/3/4/0/13408633/7233565.jpg?263
This time the larger index totally fails.

Clean the common crawl data by throwing away image-text pairs with a cosine similarity below 0.3.
After cleaning, only 26% data remains.

This time when I search for red guitar, 238G index gives:
http://i612.photobucket.com/albums/tt210/gratyngen/048-1.jpg~original
https://cdn.shopify.com/s/files/1/0550/4437/4714/products/fecd4zgjvy4je5n833kh_a61a7d8d-86d9-4228-9479-1136885b9654.jpg?v=1619827290
https://sirclocdn.com/nafiriguitar/products/_200924153910_Epi_full.jpg
which looks much more reasonable.
26G index gives:
http://i612.photobucket.com/albums/tt210/gratyngen/048-1.jpg~original
https://cdn.shopify.com/s/files/1/0550/4437/4714/products/fecd4zgjvy4je5n833kh_a61a7d8d-86d9-4228-9479-1136885b9654.jpg?v=1619827290
https://bunkerguitars.com/wp-content/uploads/2019/12/nikon-photos-192.jpg
Looks like cleaning the data greatly helps.

Very interesting!
What I think this show is that clip is bad for some examples, so if you have enough such bad examples in the dataset, the index will return a lot of them in first results
regarding the index size, something I note is your hnsw recall is actually lower than the recall you had with your 32GB ivf. The size is not the only thing that makes the index quality vary
is it a good solution for you to pre-filter the dataset with clip matching?

Yeah, it's very interesting to see that Faiss quantization actually brings these bad examples to the front.
Pre-filter with CLIP is definitely helping for common crawl images specifically, but not for cases when images are not captioned. I think in these cases we need to find other methods to score the image quality.

We will also have a large list of non or poorly captioned images and will have to look into this to process them
Obțineți Outlook pentru iOShttps://aka.ms/o0ukef

De la: Zelun Wang @.>
Trimis: Thursday, December 2, 2021 1:49:02 AM
Către: rom1504/clip-retrieval @.>
Cc: Subscribed @.***>
Subiect: Re: [rom1504/clip-retrieval] Text query on index built with image embeddings (Issue #76)
Yeah, it's very interesting to see that Faiss quantization actually brings these bad examples to the front.
Pre-filter with CLIP is definitely helping for common crawl images specifically, but not for cases when images are not captioned. I think in these cases we need to find other methods to score the image quality.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHubhttps://github.com/rom1504/clip-retrieval/issues/76#issuecomment-984161076, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AACGFM63MFON3HFVQCABXXDUO2X65ANCNFSM5IUR7UIQ.


badges textarea not monotype on ipad

in e30f3ddb34bfc54a65c636a16c4183cb0f9233e7 chrome and safari. same on macbook

should be fixed by 1e2d5f60069ca4b9176408676fecd9ee6f6189dd, needs confirmation

Tried it on macbook, iphone, ipad, all good. Closing


Weird ring like blur and images

I noticed a weird artifact when shows gradient fills or blurs. This are displayed in a "ring like" manner, and I have also discovered that it is a problem of how the OS shows images in general, because Youtube videos and other images with gradients are also shown weirdly. When I take a screenshot of the problem and try to see it in another computer, it shows it just fine. So I'll post picture of the screen taken with a camera, to exemplify.
Please help me!
This is the blured background of Launchpad:
https://i.redd.it/crq31emx47s11.jpg
https://i.redd.it/9im08q2h57s11.jpg

In screen I can't see Gl553vd . It's Look like Lenovo laptop...

i use similar images, please help me 🥇

@thanhtmph06914 MacOS Launchpod, by default bluring background, it doesn't look like a real issue because that may be caused by low-resolution display, for example on my laptop blur on some wallpapers wasn't good quality but when I connected 2k display it just looks a lot better.
I think that issue can be closed.

Yeah I can't reproduce this either on my GL553VD. Either provide more details or this should be closed.

Thanks all!


Add notes from the 1.2 release party (the real 1.2 release)

Signed-off-by: Maël Valais<EMAIL_ADDRESS>

/assign @jakexks

/lgtm


Negative ion display loss in the simplified notation of carboxyl negative ions

What happened?
My Smiles Line:
[NH3+]C(CS)C([O-])=O-
Negative ion display loss in the simplified notation of carboxyl negative ions

What did you expect to happen?
Negative ion should be displayed.
How can we reproduce it (as minimally and precisely as possible)?
Update it
Anything else we need to know?
No response
Plugin version
0.3.0
Obsidian version
1.5.3

Yeah that's why we need the option Compact drawing [Sigh]. For a temporarily workaround please disable this option.
This happens upstream and it requires too much work for me to fix it. I'm about to include more render cores to bypass the problem. And, a per-structure config override is in design, you would be able to just disable the compact drawing option for only this smiles.
Thanks for your report and any following ideas are warm welcomed.

Now in Smiles Drawer there arent any negative ion at all in some cases even if compact drawing is disabled. This smiles  C[C-]CO are displayed without any charge. Yes, RDKit works good with it but it doesnt support some pretty big molecules I really want.


Clean interfaces

Q
A

Is bugfix?
❌

New feature?
❌

Breaks BC?
✔

Fixed issues
#30

What about to separate interface and make the following structure:
SimpleMutexInterface:
acquire(name, timeout)
release(name)

----

MutexFactoryInterface:
crete(name): MutexInterface

MutexInterface:
acquire(timeout)
release()

?
Also would be great to add synchronized(name, callback) method.


How to use these devcontainer file without loosing docker ?

Hi @Chuxel
I started using the .devcontainer settings from this repos after reading your comment here https://github.community/t/not-able-to-install-yarn-and-angular-cli-and-create-codespace-container/159239/3
I later found that I am not able to do use docker in container. Today I remove the .devcontainer and build again now I am able to use docker.
I am only looking for install cli and do npm install after the container create. I don't want to use .devcontainer file since it will not provide me docker but I want to apply postcreatecommand. Any help with that!
for example look at this commit https://github.com/yegasi/yegasi.github.io/commit/6b4762399f89a572e047df4cc39c66419fe77c7b I am only looking for applying the postcreatecommand.

In some other words, how we can use devcontainer for postcreatecommand without dockerfile.