mail/postfix: Duplicate Message Fix, Milter Default Action Choice and Recipient Address Verification

(Really) Fix Duplicate Message Issue #1591
Issue #1591 concerning duplicate E-Mails was not fixed, because it was anticipated that Postfix 'milter_protocol' version would specify the IP version. Instead this specifies the version of Sendmail's Milter Protocol, which in Postfix+Rspamd's case should always be version 6.
To really fix the issue I have set Postfix', Rspamd Proxy's and Rspamd Worker's Connection to Unix Sockets. Because in OPNsense Postfix and Rspamd are set up to run on the same host, there is really no need to use TCP-Connections, which makes this the most ideal solution.
Postfix ---/var/run/rspamd/milter.sock---> Rspamd Proxy Worker ---/var/run/rspamd/normal.sock---> Rspamd Normal Worker

Add milter_default_action choice
At the moment, if Rspamd becomes unavailable to Postfix, Postfix will accept the message. I have added a choice to use the 'tempfail' action. With this one, a 4xx Status Code is given until Rspamd becomes available again.
This would prevent for example that a overloaded system would accept all (spam) messages.
Delete Milter Macros
Rspamd's documentation does not require to add milter macros in Postfix' configuration anymore, because the Milter Protocol Version 6 is fully supported.
Add Recipient Address Verification
I have added a choice to enable Postfix' Recipient Address Verification. Although this may but a little bit more load onto the next SMTP hop, this may be a desired feature for many users. #1926
Although there some limitations of this feature, this is certainly something that a user should be able to choose.

Do I have to ask for a review manually/ explicitly? Since it is about postfix and rspamd probably @mimugmail and @fabianfrz need to review it? Thanks a lot!

Do I have to ask for a review manually/ explicitly? Since it is about postfix and rspamd probably @mimugmail and @fabianfrz need to review it? Thanks a lot!

@mimugmail has to review, still a few days of vacation for him :)

@mimugmail has to review, still a few days of vacation for him :)

Two things:

This PR handles 2 plugins with 2 owners, for me it's ok
We added the choice for both protocols since for some it worked with 4 and some with 6. This would stick to socket-only, correct?

Two things:

This PR handles 2 plugins with 2 owners, for me it's ok
We added the choice for both protocols since for some it worked with 4 and some with 6. This would stick to socket-only, correct?

Hey @mimugmail
thank you very much for looking at my request. I hope you had a nice vacation.
Milter protocol version

it worked with 4 and some with 6

The choice of milter_protocol should not make any difference to the user. A major change from Milter protocol version 4 to version 6 is the introduction of a EOH (End of Headers) milter stage. Otherwise they should be fully compatible to each other.
Rspamd recommends Milter version 6 in it's documentation since May 2015.
A reason to use an older milter protocol would be that a milter (Rspamd) would not support it. This choice would only change Postfix' behaviour which milter macros will be sent (and when). Because Rspamd supports all of those for a very long time, I decided to remove the setting completely, because it just introduces confusion while serving no purpose.
Unix-socket only?

This would stick to socket-only, correct?

The Rspamd Worker's configuration would allow to listen on multiple interfaces, if that would be required. For a system that is running postfix and rspamd locally, I see no purpose in doing so:

postfix will use the Rspamd Proxy Worker's socketfile when it is configured to do so.
Rspamd Proxy Workers (milter.sock) will contact Rspamd Normal Worker (normal.sock) through the socket connection as well.

There could be a quite theoretical use case, where someone uses remote clients to talk to a local Rspamd process on a OPNsense system. For that specific case, it may be suitable for Rspamd processes to listen on TCP ports as well.
For the local Postfix however I would really recommend to use a socket connection, because it removes the confusion between IPv4/ IPv6 protocols that produces issue #1591. If we would allow TCP listeners to be configured, we should of course add a IP protocol choice for real now.
That would add some complexity for a probably non-existent use case.
If you would otherwise be uncomfortable, I will add it to the pull request.

Hey @mimugmail
thank you very much for looking at my request. I hope you had a nice vacation.
Milter protocol version

it worked with 4 and some with 6

The choice of milter_protocol should not make any difference to the user. A major change from Milter protocol version 4 to version 6 is the introduction of a EOH (End of Headers) milter stage. Otherwise they should be fully compatible to each other.
Rspamd recommends Milter version 6 in it's documentation since May 2015.
A reason to use an older milter protocol would be that a milter (Rspamd) would not support it. This choice would only change Postfix' behaviour which milter macros will be sent (and when). Because Rspamd supports all of those for a very long time, I decided to remove the setting completely, because it just introduces confusion while serving no purpose.
Unix-socket only?

This would stick to socket-only, correct?

The Rspamd Worker's configuration would allow to listen on multiple interfaces, if that would be required. For a system that is running postfix and rspamd locally, I see no purpose in doing so:

postfix will use the Rspamd Proxy Worker's socketfile when it is configured to do so.
Rspamd Proxy Workers (milter.sock) will contact Rspamd Normal Worker (normal.sock) through the socket connection as well.

There could be a quite theoretical use case, where someone uses remote clients to talk to a local Rspamd process on a OPNsense system. For that specific case, it may be suitable for Rspamd processes to listen on TCP ports as well.
For the local Postfix however I would really recommend to use a socket connection, because it removes the confusion between IPv4/ IPv6 protocols that produces issue #1591. If we would allow TCP listeners to be configured, we should of course add a IP protocol choice for real now.
That would add some complexity for a probably non-existent use case.
If you would otherwise be uncomfortable, I will add it to the pull request.

I installed your branch but it throws templating errors. Didn't you get them during your tests?

root@OPNsense:~/wind # git clone https://github.com/windgmbh/plugins
Cloning into 'plugins'...
remote: Enumerating objects: 44265, done.
remote: Total 44265 (delta 0), reused 0 (delta 0), pack-reused 44265
Receiving objects: 100% (44265/44265), 8.73 MiB | 3.05 MiB/s, done.
Resolving deltas: 100% (17217/17217), done.
root@OPNsense:~/wind # cd plugins/mail/postfix/make upgrade
cd: Too many arguments.
root@OPNsense:~/wind # cd plugins/mail/postfix/
root@OPNsense:~/wind/plugins/mail/postfix # make upgrade
>>> Generating metadata for os-postfix-devel-1.18... done
>>> Staging files for os-postfix-devel-1.18... done
>>> Packaging files for os-postfix-devel-1.18:
file sizes/checksums    [82]: 100%
packing files           [82]: 100%
packing directories      [0]: 100%
Updating database digests format: 100%
Checking integrity... done (0 conflicting)
Deinstallation has been requested for the following 1 packages (of 0 packages in the universe):

Installed packages to be REMOVED:
        os-postfix-devel: 1.17

Number of packages to be removed: 1
[1/1] Deinstalling os-postfix-devel-1.17...
[1/1] Deleting files for os-postfix-devel-1.17: 100%
Reloading plugin configuration
Installing os-postfix-devel-1.18...
Extracting os-postfix-devel-1.18: 100%
Stopping configd...done
Starting configd.
Migrated OPNsense\Postfix\Antispam from 1.0.1 to 1.0.2
Keep version OPNsense\Postfix\Domain (1.0.1)
Migrated OPNsense\Postfix\General from 1.2.4 to 1.2.5
Keep version OPNsense\Postfix\Headerchecks (1.0.0)
Keep version OPNsense\Postfix\Recipient (1.0.0)
Keep version OPNsense\Postfix\Recipientbcc (1.0.0)
Keep version OPNsense\Postfix\Sender (1.0.0)
Keep version OPNsense\Postfix\Senderbcc (1.0.0)
Keep version OPNsense\Postfix\Sendercanonical (1.0.0)
Keep version OPNsense\Postfix\Address (1.0.0)
Reloading plugin configuration
Configuring system logging...done.
Reloading template OPNsense/Postfix: Execute error
Reloading template OPNsense/Syslog: OK

Jan  6 07:08:57 OPNsense.localdomain configd.py[34218]: [85f388fa-d4d2-4038-a59c-245becf4b4a4] generate template OPNsense/Postfix
Jan  6 07:08:57 OPNsense.localdomain configd.py[34218]: generate template container OPNsense/Postfix
Jan  6 07:08:57 OPNsense.localdomain configd.py[34218]: [85f388fa-d4d2-4038-a59c-245becf4b4a4] Inline action failed with OPNsense/Postfix OPNsense/Postfix/main.cf Unexpected end of template. Jinja was looking for the following tags: 'elif' or 'else' or 'endif'. The innermost block that needs to be closed is 'if'. at Traceback (most recent call last):   File "/usr/local/opnsense/service/modules/template.py", line 247, in _generate     j2_page = self._j2_env.get_template(template_filename)   File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 883, in get_template     return self._load_template(name, self.make_globals(globals))   File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 857, in _load_template     template = self.loader.load(self, name, globals)   File "/usr/local/lib/python3.7/site-packages/jinja2/loaders.py", line 127, in load     code = environment.compile(source, name, filename)   File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 638, in compile     self.handle_exception(source=source_hint)   File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 832, in handle_exception     reraise(*rewrite_traceback_stack(source=source))   File "/usr/local/lib/python3.7/site-packages/jinja2/_compat.py", line 28, in reraise     raise value.with_traceback(tb)   File "/usr/local/opnsense/service/modules/../templates/OPNsense/Postfix/main.cf", line 209, in template     {% endif %} jinja2.exceptions.TemplateSyntaxError: Unexpected end of template. Jinja was looking for the following tags: 'elif' or 'else' or 'endif'. The innermost block that needs to be closed is 'if'.  During handling of the above exception, another exception occurred:  Traceback (most recent call last):   File "/usr/local/opnsense/service/modules/processhandler.py", line 506, in execute     return ph_inline_actions.execute(self, inline_act_parameters)   File "/usr/local/opnsense/service/modules/ph_inline_actions.py", line 51, in execute     filenames = tmpl.generate(parameters)   File "/usr/local/opnsense/service/modules/template.py", line 341, in generate     raise render_exception   File "/usr/local/opnsense/service/modules/template.py", line 332, in generate     for filename in self._generate(template_name, create_directory):   File "/usr/local/opnsense/service/modules/template.py", line 249, in _generate     raise Exception("%s %s %s" % (module_name, template_filename, templExc)) Exception: OPNsense/Postfix OPNsense/Postfix/main.cf Unexpected end of template. Jinja was looking for the following tags: 'elif' or 'else' or 'endif'. The innermost block that needs to be closed is 'if'.

I installed your branch but it throws templating errors. Didn't you get them during your tests?

root@OPNsense:~/wind # git clone https://github.com/windgmbh/plugins
Cloning into 'plugins'...
remote: Enumerating objects: 44265, done.
remote: Total 44265 (delta 0), reused 0 (delta 0), pack-reused 44265
Receiving objects: 100% (44265/44265), 8.73 MiB | 3.05 MiB/s, done.
Resolving deltas: 100% (17217/17217), done.
root@OPNsense:~/wind # cd plugins/mail/postfix/make upgrade
cd: Too many arguments.
root@OPNsense:~/wind # cd plugins/mail/postfix/
root@OPNsense:~/wind/plugins/mail/postfix # make upgrade
>>> Generating metadata for os-postfix-devel-1.18... done
>>> Staging files for os-postfix-devel-1.18... done
>>> Packaging files for os-postfix-devel-1.18:
file sizes/checksums    [82]: 100%
packing files           [82]: 100%
packing directories      [0]: 100%
Updating database digests format: 100%
Checking integrity... done (0 conflicting)
Deinstallation has been requested for the following 1 packages (of 0 packages in the universe):

Installed packages to be REMOVED:
        os-postfix-devel: 1.17

Number of packages to be removed: 1
[1/1] Deinstalling os-postfix-devel-1.17...
[1/1] Deleting files for os-postfix-devel-1.17: 100%
Reloading plugin configuration
Installing os-postfix-devel-1.18...
Extracting os-postfix-devel-1.18: 100%
Stopping configd...done
Starting configd.
Migrated OPNsense\Postfix\Antispam from 1.0.1 to 1.0.2
Keep version OPNsense\Postfix\Domain (1.0.1)
Migrated OPNsense\Postfix\General from 1.2.4 to 1.2.5
Keep version OPNsense\Postfix\Headerchecks (1.0.0)
Keep version OPNsense\Postfix\Recipient (1.0.0)
Keep version OPNsense\Postfix\Recipientbcc (1.0.0)
Keep version OPNsense\Postfix\Sender (1.0.0)
Keep version OPNsense\Postfix\Senderbcc (1.0.0)
Keep version OPNsense\Postfix\Sendercanonical (1.0.0)
Keep version OPNsense\Postfix\Address (1.0.0)
Reloading plugin configuration
Configuring system logging...done.
Reloading template OPNsense/Postfix: Execute error
Reloading template OPNsense/Syslog: OK

Jan  6 07:08:57 OPNsense.localdomain configd.py[34218]: [85f388fa-d4d2-4038-a59c-245becf4b4a4] generate template OPNsense/Postfix
Jan  6 07:08:57 OPNsense.localdomain configd.py[34218]: generate template container OPNsense/Postfix
Jan  6 07:08:57 OPNsense.localdomain configd.py[34218]: [85f388fa-d4d2-4038-a59c-245becf4b4a4] Inline action failed with OPNsense/Postfix OPNsense/Postfix/main.cf Unexpected end of template. Jinja was looking for the following tags: 'elif' or 'else' or 'endif'. The innermost block that needs to be closed is 'if'. at Traceback (most recent call last):   File "/usr/local/opnsense/service/modules/template.py", line 247, in _generate     j2_page = self._j2_env.get_template(template_filename)   File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 883, in get_template     return self._load_template(name, self.make_globals(globals))   File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 857, in _load_template     template = self.loader.load(self, name, globals)   File "/usr/local/lib/python3.7/site-packages/jinja2/loaders.py", line 127, in load     code = environment.compile(source, name, filename)   File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 638, in compile     self.handle_exception(source=source_hint)   File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 832, in handle_exception     reraise(*rewrite_traceback_stack(source=source))   File "/usr/local/lib/python3.7/site-packages/jinja2/_compat.py", line 28, in reraise     raise value.with_traceback(tb)   File "/usr/local/opnsense/service/modules/../templates/OPNsense/Postfix/main.cf", line 209, in template     {% endif %} jinja2.exceptions.TemplateSyntaxError: Unexpected end of template. Jinja was looking for the following tags: 'elif' or 'else' or 'endif'. The innermost block that needs to be closed is 'if'.  During handling of the above exception, another exception occurred:  Traceback (most recent call last):   File "/usr/local/opnsense/service/modules/processhandler.py", line 506, in execute     return ph_inline_actions.execute(self, inline_act_parameters)   File "/usr/local/opnsense/service/modules/ph_inline_actions.py", line 51, in execute     filenames = tmpl.generate(parameters)   File "/usr/local/opnsense/service/modules/template.py", line 341, in generate     raise render_exception   File "/usr/local/opnsense/service/modules/template.py", line 332, in generate     for filename in self._generate(template_name, create_directory):   File "/usr/local/opnsense/service/modules/template.py", line 249, in _generate     raise Exception("%s %s %s" % (module_name, template_filename, templExc)) Exception: OPNsense/Postfix OPNsense/Postfix/main.cf Unexpected end of template. Jinja was looking for the following tags: 'elif' or 'else' or 'endif'. The innermost block that needs to be closed is 'if'.

Somehow a Jinja Syntax Error might have ended up in the final mail/postfix: Add Recipient Address Verification commit, that I have rectified now.
I ran the tests again and have gotten these results:
Build Log

>>> Generating metadata for os-postfix-devel-1.18... done
>>> Staging files for os-postfix-devel-1.18... done
>>> Packaging files for os-postfix-devel-1.18:
file sizes/checksums    [82]: 100%
packing files           [82]: 100%
packing directories      [0]: 100%
Checking integrity... done (0 conflicting)
Deinstallation has been requested for the following 1 packages (of 0 packages in the universe):

Installed packages to be REMOVED:
	os-postfix: 1.17

Number of packages to be removed: 1
[1/1] Deinstalling os-postfix-1.17...
[1/1] Deleting files for os-postfix-1.17: 100%
Reloading plugin configuration
Installing os-postfix-devel-1.18...
Extracting os-postfix-devel-1.18: 100%
Stopping configd...done
Starting configd.
Migrated OPNsense\Postfix\Antispam from 1.0.1 to 1.0.2
Keep version OPNsense\Postfix\Domain (1.0.1)
Migrated OPNsense\Postfix\General from 1.2.4 to 1.2.5
Keep version OPNsense\Postfix\Headerchecks (1.0.0)
Keep version OPNsense\Postfix\Recipient (1.0.0)
Keep version OPNsense\Postfix\Recipientbcc (1.0.0)
Keep version OPNsense\Postfix\Sender (1.0.0)
Keep version OPNsense\Postfix\Senderbcc (1.0.0)
Keep version OPNsense\Postfix\Sendercanonical (1.0.0)
Keep version OPNsense\Postfix\Address (1.0.0)
Reloading plugin configuration
Configuring system logging...done.
Reloading template OPNsense/Postfix: OK
Reloading template OPNsense/Syslog: OK

>>> Generating metadata for os-rspamd-devel-1.11... done
>>> Staging files for os-rspamd-devel-1.11... done
>>> Packaging files for os-rspamd-devel-1.11:
file sizes/checksums    [40]: 100%
packing files           [40]: 100%
packing directories      [0]: 100%
Updating database digests format: 100%
Checking integrity... done (0 conflicting)
Deinstallation has been requested for the following 1 packages (of 0 packages in the universe):

Installed packages to be REMOVED:
	os-rspamd: 1.10

Number of packages to be removed: 1
[1/1] Deinstalling os-rspamd-1.10...
[1/1] Deleting files for os-rspamd-1.10: 100%
Reloading plugin configuration
Installing os-rspamd-devel-1.11...
Extracting os-rspamd-devel-1.11: 100%
Stopping configd...done
Starting configd.
Keep version OPNsense\Rspamd\RSpamd (1.0.2)
Reloading plugin configuration
Configuring system logging...done.
Reloading template OPNsense/Rspamd: OK

Results
After applying new options in the GUI.
Postfix

main.cf (excerpt)

smtpd_milters = unix:/var/run/rspamd/milter.sock
non_smtpd_milters = $smtpd_milters
milter_protocol = 6
milter_default_action = tempfail

smtpd_recipient_restrictions = reject_unknown_sender_domain, reject_unknown_recipient_domain, reject_non_fqdn_sender, reject_non_fqdn_recipient, permit_mynetworks, reject_unauth_destination, reject_unverified_recipient

Rspamd

local.d/worker-normal.inc

upstream "local" {
  default = yes;
  hosts = "/var/run/rspamd/normal.sock";
}
bind_socket = "/var/run/rspamd/milter.sock mode=0666 owner=rspamd";

local.d/worker-normal.inc

bind_socket = "/var/run/rspamd/normal.sock mode=0666 owner=rspamd";

Sorry for my bad development attitude. I promise that I will do better next time.
How is your opinion on listening on TCP4/ TCP6 ports? If you think it is necessary, I will add it as an option.

Somehow a Jinja Syntax Error might have ended up in the final mail/postfix: Add Recipient Address Verification commit, that I have rectified now.
I ran the tests again and have gotten these results:
Build Log

>>> Generating metadata for os-postfix-devel-1.18... done
>>> Staging files for os-postfix-devel-1.18... done
>>> Packaging files for os-postfix-devel-1.18:
file sizes/checksums    [82]: 100%
packing files           [82]: 100%
packing directories      [0]: 100%
Checking integrity... done (0 conflicting)
Deinstallation has been requested for the following 1 packages (of 0 packages in the universe):

Installed packages to be REMOVED:
	os-postfix: 1.17

Number of packages to be removed: 1
[1/1] Deinstalling os-postfix-1.17...
[1/1] Deleting files for os-postfix-1.17: 100%
Reloading plugin configuration
Installing os-postfix-devel-1.18...
Extracting os-postfix-devel-1.18: 100%
Stopping configd...done
Starting configd.
Migrated OPNsense\Postfix\Antispam from 1.0.1 to 1.0.2
Keep version OPNsense\Postfix\Domain (1.0.1)
Migrated OPNsense\Postfix\General from 1.2.4 to 1.2.5
Keep version OPNsense\Postfix\Headerchecks (1.0.0)
Keep version OPNsense\Postfix\Recipient (1.0.0)
Keep version OPNsense\Postfix\Recipientbcc (1.0.0)
Keep version OPNsense\Postfix\Sender (1.0.0)
Keep version OPNsense\Postfix\Senderbcc (1.0.0)
Keep version OPNsense\Postfix\Sendercanonical (1.0.0)
Keep version OPNsense\Postfix\Address (1.0.0)
Reloading plugin configuration
Configuring system logging...done.
Reloading template OPNsense/Postfix: OK
Reloading template OPNsense/Syslog: OK

>>> Generating metadata for os-rspamd-devel-1.11... done
>>> Staging files for os-rspamd-devel-1.11... done
>>> Packaging files for os-rspamd-devel-1.11:
file sizes/checksums    [40]: 100%
packing files           [40]: 100%
packing directories      [0]: 100%
Updating database digests format: 100%
Checking integrity... done (0 conflicting)
Deinstallation has been requested for the following 1 packages (of 0 packages in the universe):

Installed packages to be REMOVED:
	os-rspamd: 1.10

Number of packages to be removed: 1
[1/1] Deinstalling os-rspamd-1.10...
[1/1] Deleting files for os-rspamd-1.10: 100%
Reloading plugin configuration
Installing os-rspamd-devel-1.11...
Extracting os-rspamd-devel-1.11: 100%
Stopping configd...done
Starting configd.
Keep version OPNsense\Rspamd\RSpamd (1.0.2)
Reloading plugin configuration
Configuring system logging...done.
Reloading template OPNsense/Rspamd: OK

Results
After applying new options in the GUI.
Postfix

main.cf (excerpt)

smtpd_milters = unix:/var/run/rspamd/milter.sock
non_smtpd_milters = $smtpd_milters
milter_protocol = 6
milter_default_action = tempfail

smtpd_recipient_restrictions = reject_unknown_sender_domain, reject_unknown_recipient_domain, reject_non_fqdn_sender, reject_non_fqdn_recipient, permit_mynetworks, reject_unauth_destination, reject_unverified_recipient

Rspamd

local.d/worker-normal.inc

upstream "local" {
  default = yes;
  hosts = "/var/run/rspamd/normal.sock";
}
bind_socket = "/var/run/rspamd/milter.sock mode=0666 owner=rspamd";

local.d/worker-normal.inc

bind_socket = "/var/run/rspamd/normal.sock mode=0666 owner=rspamd";

Sorry for my bad development attitude. I promise that I will do better next time.
How is your opinion on listening on TCP4/ TCP6 ports? If you think it is necessary, I will add it as an option.

I installed your branch now on my test system, I'll have a look at it the next days. Looks good now.
No opinion yet :)

I installed your branch now on my test system, I'll have a look at it the next days. Looks good now.
No opinion yet :)

I count @mimugmail as a maintainer of rspamd as well. He can approve that as well.

I count @mimugmail as a maintainer of rspamd as well. He can approve that as well.

I may add that I am running the Unix Socket configuration (4fe003b) on a production system since mid November, because the issue with duplicate messages had to be fixed immediately. So at least for my setup it can be considered as thoroughly tested by now.

Concerning Unix sockets another though came to my mind: Even though the Redis connection does not cause any problems with TCP connections, it could be more consistent to use Unix sockets for the Redis connections as well.
Redis is already configured to have a socket file in /var/run/redis/redis.sock, but has the wrong permissions (700) at the moment.
$ ls -l /var/run/redis/redis.sock
srwx------  1 redis  redis  0 Jan 12 16:02 /var/run/redis/redis.sock

In order to work, we could either set permissions to (777) or to 770 while adding the rspamd user to the redis group.
I don't expect that to be a security risk compared to the current configuration, because the Redis connection is already exposed on a local TCP port for all local processes to communicate with.
While this is, strictly speaking, not required to fix the issue with issue with duplicate messages, it may be better for from a design standpoint to use Unix socket for everything that has to do with rspamd.

If this is a change that you would favour as well, I could:

add this to this pull request
open a new pull request.

Please let me know, what way you would prefer.

I may add that I am running the Unix Socket configuration (4fe003b) on a production system since mid November, because the issue with duplicate messages had to be fixed immediately. So at least for my setup it can be considered as thoroughly tested by now.

Concerning Unix sockets another though came to my mind: Even though the Redis connection does not cause any problems with TCP connections, it could be more consistent to use Unix sockets for the Redis connections as well.
Redis is already configured to have a socket file in /var/run/redis/redis.sock, but has the wrong permissions (700) at the moment.
$ ls -l /var/run/redis/redis.sock
srwx------  1 redis  redis  0 Jan 12 16:02 /var/run/redis/redis.sock

In order to work, we could either set permissions to (777) or to 770 while adding the rspamd user to the redis group.
I don't expect that to be a security risk compared to the current configuration, because the Redis connection is already exposed on a local TCP port for all local processes to communicate with.
While this is, strictly speaking, not required to fix the issue with issue with duplicate messages, it may be better for from a design standpoint to use Unix socket for everything that has to do with rspamd.

If this is a change that you would favour as well, I could:

add this to this pull request
open a new pull request.

Please let me know, what way you would prefer.

Merged, thanks! ❤


runApp "appstart" event

Right now it's impossible to know when the NW.js process is starting after calling run(). The promise will be resolved when killing the application, so the promise itself can't be used for getting the NW.js process.
With this change an appstart event will be triggered right after spawning the NW.js process. There's also an error callback added, which will reject the promise in case of an error.

Any comments on this?


Deliver Missing Module

Hey, AICoE-CI!
Please deliver the package module for the following tag, as it missing on PyPI:
Tag: v0.0.2

Successfully Delivered Package module to Pulp PyPI.
Package can be found at: https://pypi.org/project/disk-health-predictor


Remove --experimental from Studio command

👀

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

@madebysid please check everywhere where --experimental is mentioned, like CLI help
https://github.com/prisma/prisma/blob/fbd241a2bba12b757f900a6f8e723ebc7285a47f/src/packages/cli/src/Studio.ts#L28
https://github.com/prisma/prisma/blob/115244b153e4bb34d0f36aa48216881891d816a0/src/packages/cli/src/CLI.ts#L158
or tests
https://github.com/prisma/prisma/blob/73f5190990c345227e2eb3b154253166a123f8ec/src/packages/cli/src/__tests__/studio.test.ts#L78

Ah, done. did a global search for --experimental and removed it from all appropriate places.


Bugfix: Change condition to give True when splits - l is 0 to give correct P_k score

Refer to https://github.com/chschock/textsplit/issues/4

Closing this one since I messed up the commits..


Enable MacOS Scripting Bridge

OS:
macOS 10.12.3
Issue Descriptions:
I use the SIP application Telephone which can turn off music players whenever there is an incoming call. This works for iTunes and Spotify but not for the Google Play Music Desktop Player.
According to the Telephone developers this works by using the macOS Scripting Bridge that allows the sandboxed applications to talk to each other. Compare the output of the following commands which show the scripting bridge exports:
sdef /Applications/iTunes.app
sdef /Applications/Google\ Play\ Music\ Desktop\ Player.app
Is there any way to add the functionality for external programs to pause the music?

@Phylu External applications can control GPMDP through the cross-platform PlaybackAPI.
I looked into the scripting bridge a while ago as AFAIC it will require a native module to be written in order to provide the native hooks required.
I will not spend time doing this but others are more than welcome to contribute.  Tentatively leaving open so other can find 👍

Out of curiosity, ( I don't own a mac ), can you post the results of those two commands @Phylu?

Here are the outputs:
GPMDP.txt
iTunes.txt

Is it possible also to control the playback using the PlaybackAPI? The documentation mentions only getting the status:

Google Play Music Desktop Player provides an interface for external applications to determine the currently played song and playback status.

The WebSocket API allows for status and control. The JSON file is one way, as in it only outputs current info to a json file.

And that native module that @MarshallOfSound mentioned, how big should it be?

That's really an arbitrary question. It's going to have to be as big as it needs to be to encapsulate the functionality required to work with apple's ScriptingBridge interface.

What exactly would you need from it? On one side it would know how to use Scripting Bridge. On the other side it would provide some Swift or Objective-C API. Is this Swift or Objective-C API what you need and could use?

I think that's what is needed, to an extent. The native module would only need to provide the bridge between the internal GPM API for methods and properties, and the ScriptingBridge. From what I can tell the ScriptingBridge would need to allow track information, state of music, and the app. It would be up to other developers to utilize that bridge. I'm not sure what would be needed in order for the VoIP application mentioned by @Phylu to automatically pause music.

I’m the author of that VoIP app and what is needed is a property like isPlaying and two methods like play() and pause() exposed via Scripting Bridge with so-called access groups. That’s how the VoIP app, being sandboxed, communicates with iTunes and Spotify apps.

I see what you're asking now. We'd need the module work with NodeJS. So the module would need to be in C++, and your bridge would probably be in Swift/Objective-C. Somehow you'd have to glue the communication together between them. The C++ would be what communicates directly with NodeJS, and in Javascript we'd populate the properties and set callbacks for the methods.

@eofster I have zero clue how to use the scripting bridge and little interest / investment in figuring it out right now 😆
This is up for grabs if anyone wants to try it, I can coach you through getting a GPMDP development environment up or an Electron environment if that's what needed 👍

Closing as a unique/stale feature. It doesn't appear there is a significant interest in this at this time.


Add an example to create/delete/list Google Cloud Storage buckets.

Thanks for the pull request but it's release week in jclouds and that means it's time to clean up the PR queue. This PR will be over 6 months old as of April 1. If you intend to continue work on it, please make a comment by April 2. Otherwise it will be closed on April 3.


App is not shown in the share sheet iOS

Hello, I tried the example app. Everything works fine on iOS, however when I try to add the extension to my app, it's not shown in the share sheet. I followed the tutorial, double-checked every config, seems like everything is fine. Maybe somebody encountered the same problem, maybe I've done some silly mistake and trying to fight with the configs? Much appreciate any help

Ok, I found the solution after countless hours staring at the stupid plist file. Just make sure the deployment target for Runner.app and the share extension is the same... in my case the app was for iOS 10 and the share extension was iOS 12.4. Could we include this info to README just in case, to save the lifetime of other people

Thanks! I just included that in the readme.


Files seen as folders in Atom 1.12.0 (beta 0 and beta 1)

I can't open any files remotely on the latest Atom betas as all of the files (and folders) are seen as folders and so they just "expend".
For example in this screenshot the .json and the .htm file aren't openable as Remote-FTP thinks they are folders.
Same version of plugin (0.9.0) works fine in Atom 1.11.0 (and 1.11.1).
Screenshot of error:

Same files locally obviously work as normal files:

Will have a look. Thanks for pointing this out

@jimmaaay Cool 👍  If you need me to test anything if you can't reproduce then let me know.

Same is happening to me, files are showing up as empty folders.. weird.

@gdevdeiv Mine has worked perfectly in the non beta (1.11.x) version of Atom.  It was only the move to 1.12.x that seemed to break the plugin.  Maybe it is a setting then?

Should be fixed in 0.9.1, if you were using SFTP. If anyone is having the same issue when using FTP please create a new issue.

@jimmaaay Seems to be working great, thanks!

This bug is not fixed in Atom 1.19.0 and Remote-Ftp 1.1.4. Some servers are working fine, others don't.
All files are downloaded as empty folders.

@zodit This was quite an old issue but it has cropped up again recently.  You're probably better off looking at #925 which is a more up to date version of the "same" bug


About the "multiView" attribute and unknown sized attributes

In exrinput.cpp the "multiView" attribute is a OpenEXR 2 StringVectorAttribute.
When handling the attribute, it does the following (https://github.com/OpenImageIO/oiio/blob/master/src/openexr.imageio/exrinput.cpp#L613-L614) :
TypeDesc sv (TypeDesc::STRING, ustrvec.size());
 spec.attribute(oname, sv, &ustrvec[0]);

This attribute has an unknown size by the caller, but the TypeDesc is "identified" by the vector size. The only solution then to retrieve this attribute from the find_attribute() function is to pass TypeDesc::UNKNOWN so we are sure it does not attempt array length equality check (because we don't know on the calling hand the size of the array).

The second line :  spec.attribute(oname, sv, &ustrvec[0]) will create an attribute with only 1 value as size, see: https://github.com/OpenImageIO/oiio/blob/master/src/libOpenImageIO/formatspec.cpp#L298
This results in the function ParamValue::nvalues() returning 1 always, even if this is in this case a string vector.
The only workaround at the moment is to call ParamValue::type() on the attribute to retrieve the original TypeDesc and check its array length.
Is this the intended way to deal with dynamic size attributes ?

I'm not sure if this is still an issue in our code any more or not (this area of the code has changed a lot since the embarrassingly long time ago that this was filed). But in any case, I think that this method of dealing with multi-view is no longer favored, and instead everybody is using multi-part exrs, which we do support.
I'm going to speculatively close this, but if you think that there's still a problem worth investigating in current versions, please re-open or re-file the issue.


Added "configFile" entry

There was a missing "configFile" entry for adding configuration options to alpha and zero.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Make the creation of dwell time map quiet and  clean; fix filename issue in the FullDetectorResponse module; add fast ts fit module

Update 1 -- SpacecraftFile and FullDetectorResponse
According to  #88, I added two parameters (quiet and save) for get_target_in_sc_frame and get_dwell_map methods:

quiet controls if intermediate messages during calculation are printed. Default is False.
save controls if the target path in the SC frame and the dwell time map are saved. Default is False.

The issue caused by FullDetectorResponse.open method when the input is a pathlib.Path object instead of str is fixed. Now the filename will be converted into a pathlib.Path object regardless of the type of input.
Update 2 -- FastTSMap
I also added two new modules: fast_norm_fit.py and fast_ts_fit.py.

fast_norm_fit.py is taken from the BusrtCube tool (simply copy and paste), so it doesn't need review (I guess so?). @israelmcmc please let me know what we should do to credit this module.

fast_ts_fit.py is the one I wrote to compute the parallel ts computation based on fast_norm_fit.py, which requires review.

The tutorial notebook is located at /docs/tutorials/Parallel_TS_map_computation.ipynb. It contains the necessary cells with simple explanatory comments to run through the whole process. I will update it in future PR with more text in detail to explain what it's actually doing. Let me know if you have any questions regarding this.

Now FastTSMap uses all the available cores to run the parallel computation. Please let me know if it significantly slows down your machine.

The data are upload to the folder /cosipy/test_data:

new_healpix_rsp_Binned_Bkg_2s_model.hdf5
new_healpix_rsp_Binned_Cosmic2s.hdf5
new_healpix_rsp_Binned_protoGRB.hdf5

I already set up the path to the files in Parallel_TS_map_computation.ipynb, but if it doesn't work on your machine, please update the path accordingly.
Thanks!

Hi Yong, I've looked through this pull request. Mostly I focused on the Parallel_TS_map_computation jupyter-notebook, and I have a few comments but these are mostly cosmetic vs functional.

It will be important to test this with the SMEX model once we have a GRB sim to do this
Is there a way to get the orientation information from the ori file instead of hardcoding it? For example, if you know the time of the burst, can you not just take the range of pointings during that time?
When running the ts_results = ts.parallel_ts_fit(... results cell, I am getting 16 copies of lots of warnings. These mostly seem related to astromodels, and 3ML
Can you include the option to select the number of cores to run with in the parallel_ts_fit, since using all of my cpu cores is limiting if I'm trying to do other work in parallel?
For the final image when you plot the known location with longitude and latitude, can you instead plot an x at the correct location so it's clear which pixel this corresponds to since the arrow isn't as obvious (is the point of the arrow the actual true location?)
Just FYI, it took my computer with 16 GB of memory and 16 cores 21 minutes to run and produce an image


How do I add my parser?

Good job.
If I want to enhance the parsers, how do I add my parse into the parser list?

Good job.
If I want to enhance the parsers, how do I add my parse into the parser list?

please take a look at https://github.com/dengyin2000/dynamic_widget#how-to-implement-a-widgetparser


VM Exception when interfacing with Kyber contract

Testing a contract that otherwise works against a ganache mainnet fork but when interfacing with the KyberProxy contract I get VM Exception while processing transaction: revert.
Kyber address: 0x818E6FECD516Ecc3849DAf6845e3EC868087B755
The code is as simple as this
function testFunc() public view returns(uint minConversionRate, uint  slippageRate){
uint value =<PHONE_NUMBER>000000;
(minConversionRate, slippageRate) = kyberNetworkProxy.getExpectedRate(ERC20(ETH_ADDRESS), ERC20(snxAddress), value);
}

and the same code works on mainnet without issue. So we can safely rule out any issues related to contract code. Additionally, JS calls to the Kyber contract directly on ganache fork are working, so this bug is specific to interfacing with this contract from another contract on ganache.

I asked about this on gitter: [x ]
Link to my question on gitter: https://gitter.im/ConsenSys/truffle?at=5ea31b31afa1f51d4e1a5b0c

Expected Behavior
Return two values from the getExpectedRate view function
Current Behavior
EVM Revert
Possible Solution
Not sure but I've tested with very high gasLimits and it wasn't any help.
Steps to Reproduce (for bugs)

run ganache-cli --fork https://mainnet.infura.io/v3/<infura-token>
run testFunc()

Context
Your Environment

Version used: Ganache CLI v6.4.3 (ganache-core: 2.5.5)
Version of Truffle/Remix/Other tools used:Truffle v5.0.7 (core: 5.0.7)
Solidity - 0.5.0 (solc-js)
NodeJS Version: Node v13.3.0

Looks like you are using a version of ganache-cli released over 1 year ago. Please update to latest (npm install -g ganache-cli) and try again.
If that doesn't fix it, I don't think there is anything on your side you can do at the moment, this is likely a bug in Ganache core itself. :-/
That said, if you don't have access to archive data at Infura, forking might not always behave as expected.
It's not immediately apparent to me what could be going wrong, and I can't jump to fix this at the moment. If you have time to dive in and try to track down whats going wrong, all the forking-related code is contained within these two files:
https://github.com/trufflesuite/ganache-core/blob/develop/lib/utils/forkedblockchain.js
https://github.com/trufflesuite/ganache-core/blob/develop/lib/utils/forkedstoragetrie.js

Troubleshooted with latest ganache version after posting this. Still facing same bug

I'm experiencing a similar thing with uniswap when forking
https://github.com/Uniswap/uniswap-v2-periphery/issues/44


Add flow typing: Part One

This is the first part of adding flow typing to react-render-server.
I decided to submit this PR now as it was getting pretty darn big. In this PR we:

Add flow-bin and flow-typed
Add linting to ensure files declare @flow or @noflow
Add flow-types to arguments.js, logging.js, secret.js and main.js
Reworked arguments.js to make mocking argument values easier
Reworked secret.js tests to use async/await
Setup chai-as-promised for all tests rather than just render_test.js

During this work, I've noticed that flow typing of Winston stuff seems pretty weak. There are also no flow-typed files for some of our other dependencies, including google cloud things. I have filled in some typing gaps for chai-as-promised and winston, but nothing serious; just copy/paste/edit from the typescript files. Another reason for getting this out now is so that subsequent typing edits, which may be more substantial, are clear and easy to review.

Thanks for the feedback, folks. I've addressed all the comments, I believe. I'm going to merge this in and continue with the rest of the changes.


This should be changed to an exact version number :  0.6.8

https://github.com/chnejohnson/vue-dapp/blob/9a948222a60622f8b339c50877ab41e10aabbcb1/yarn.lock#L3103

@kohoj Oh, I'm not familiar with yarn.lock, do you know how can I solve it?


How to turn a map in to specific hashmap?

let map = Map::from_pin_file("pin_file").unwrap();
let hasmap = HashMap::<u32, u32>::new(&map).unwrap();

The hashmap internally references map, so I can't use hashmap alone without a valid map;
If there has a way like map::into::<HashMap>(), it will be more convenient:
let map = Map::from_pin_file("pin_file").unwrap();
let hasmap: HashMap<u32, u32> = map.try_into().unwrap();

Hello @d0u9
It's really nice to see you again. Lucky day!
Thanks for your suggestion :+1:
It looks like convenient and neat!


Tokodon - Mastodon Client

Project description
Tokodon is a Mastodon client for Plasma and Plasma Mobile
Metadata

homepage URL:  https://invent.kde.org/network/tokodon
source URL: https://github.com/KDE/tokodon
license: GNU General Public License 3
platforms: Linux

Tokodon is packaged in nixpkgs in plasma-mobile (libsForQt5.plasma-mobile). It is just not exposed yet.

Tokodon is packaged in nixpkgs in plasma-mobile (libsForQt5.plasma-mobile). It is just not exposed yet.

Then why is it still not exposed?


Warning on Elixir v1.14

I'm doing a bit of preliminary testing with https://github.com/elixir-lang/elixir/releases/tag/v1.14.0-rc.1, and noticed the following warning, so I'm opening an issue to report it :
==> crontab
Compiling 7 files (.ex)
warning: Application.get_env/3 is discouraged in the module body, use Application.compile_env/3 instead
  lib/crontab/scheduler.ex:21: Crontab.Scheduler

Generated crontab app

Apparently not fixed on master:
https://github.com/jshmrtn/crontab/blob/c7641a424abc1191c3bf91ace369350097eb4890/lib/crontab/scheduler.ex#L21

Fixed by 81bd15abaa954afc8e8f9d1fd5b295fb3bf59139

Looks like the warning is still being triggered:
Compiling 7 files (.ex)
    warning: Application.get_env/3 is discouraged in the module body, use Application.compile_env/3 instead
    │
 21 │     @max_runs Application.get_env(:crontab, :max_runs, 10_000)
    │                           ~
    │
    └─ lib/crontab/scheduler.ex:21:27: Crontab.Scheduler (module)

Same here! Maybe we can reopen a new issue @sergchernata?

It seems this has been resolved in #116, but there has not been a new release?


Replacement for include_paths?

I took a look at the docs and API docs for the new Sentry python SDK and couldn't see a replacement for include_paths. Is this planned/available?

Yes, this is called in_app_include now, as argument to init. We have to document this.


style: Underlined “Keystore” and “Backup” makes me think they are separate links that do different things.  I know from talking with jojobyte that it’s just one button that does the same thing, but let’s try to not trick people.  Maybe just make them bold (remove the underline, and remove the bold from the “or”).

Fixed in PR #31


[5371] Update withdrawal date question

Context
https://trello.com/c/8D6UROhZ/5371-s-update-withdrawal-date-question
Changes proposed in this pull request

Legend updated to Date the trainee withdrew
Hint text added
3rd radio option updated to say  "Another date"
Removed unused translation key not related to these card

Guidance to review

Choose an in-training trainee
Click withdraw
Text should reflect the changes made

Important business

[ ] Does this PR introduce any PII fields that need to be overwritten or deleted in db/scripts/sanitise.sql?
[ ] Does this PR change the database schema? If so, have you updated the config/analytics.yml file and considered whether you need to send 'import_entity' events?
[ ] Do we need to send any updates to DQT as part of the work in this PR?
[ ] Does this PR need an ADR?

NB: Please notify the #twd_data_insights team and ask for a review if new fields are being added to analytics.yml

 [S] Update withdrawal date question

The changes look fine to me.


[improve] enable and fix test src checkstyle (#2610)

What's changed?
Added unit tests for PushMetricsDao, PushMonitorDao
Checklist

[x]  I have read the Contributing Guide
[x]  I have written the necessary doc or comment.
[x]  I have added the necessary unit tests and all cases have passed.

Add or update API

[ ] I have added the necessary e2e tests and all cases have passed.

What's changed?
Added unit tests for PushMetricsDao, PushMonitorDao
Checklist

[x]   I have read the Contributing Guide
[x]   I have written the necessary doc or comment.
[x]   I have added the necessary unit tests and all cases have passed.

Add or update API

[ ]  I have added the necessary e2e tests and all cases have passed.


How can we use Click One?

How can we keep the project up-to-date with Click One in linux or Mac operating systems. Could you sample this?

ClickOnce is part of the .Net Framework, so it'll only work with Windows Applications as far as I am aware.


IE11 win7, over CPU usage

There is 100% cpu utilization in case of IE11 and win7 combination. On applying profilers i narrowed down and found that
 var wrap = function(event) {
                 if (event.key === 'ngIdle.expiry') svc.interrupt();
                };

this line is taking all the cpu
this is happening when i upgraded to the latest version 1.0.0

its going on a infinite loop on that function

The same is happening with Windows 7 + IE 10.
This event is supposed to be fired only when the user is active, however is infinite as @pranaydutta89  pointed out.

Everything works fine with Firefox and Chrome so this is probably an IE's issue only.

Once again, Microsoft can't read or plainly implement a *&#$@%&ed spec. The onstorage event is not supposed to fire on the same window that caused the storage change, but they do, which is causing the infinite loop.

Thank you!

thanks a ton


TypeError: rawRequest.connection.address is not a function

Description
We are using a library that uses a Popsicle client.
There is a failure in mocking the requests using Popsicle.
Shareable Source
Pretty vanilla Polly setup:
import path from 'path'
import { Polly, PollyServer } from '@pollyjs/core'
import NodeHttpAdapter from '@pollyjs/adapter-node-http'
import FSPersister from '@pollyjs/persister-fs'

Polly.register(FSPersister)
Polly.register(NodeHttpAdapter)

export class PollyJS {
  recordingName: string
  polly: Polly

  constructor(recordingName: string) {
    this.recordingName = recordingName
    this.polly = new Polly(this.recordingName, {
      adapters: ['node-http'],
      persister: 'fs',
      persisterOptions: {
        fs: {
          recordingsDir: path.resolve(__dirname, '../__recordings__'),
        },
      },
    })
  }

  getPollyInstance(): Polly {
    return this.polly
  }

  getPollyServer(): PollyServer {
    return this.polly.server
  }
}

Error Message & Stack Trace
    TypeError: rawRequest.connection.address is not a function

      at OverriddenClientRequest.onResponse (node_modules/popsicle-transport-http/src/index.ts:316:33)
      at _respond (node_modules/nock/lib/request_overrider.js:525:17)
      at respond (node_modules/nock/lib/request_overrider.js:508:11)

The code can be found here: https://github.com/serviejs/popsicle-transport-http/blob/ef78022e431532ec710c716ca91b7d3691afe888/src/index.ts#L316
Config
See above.
Dependencies
Copy the @pollyjs dependencies from package.json:
{
    "@pollyjs/adapter-node-http": "2.6.3",
    "@pollyjs/core": "2.6.3",
    "@pollyjs/persister-fs": "2.6.3"
}

Relevant Links
Provided by @nguyentoanit:
https://github.com/nguyentoanit/popsicle-polly-error
Environment
Tell us which operating system you are using, as well as which versions of Node.js and npm/yarn. If applicable, include the browser and the corresponding version.
Run the following to get it quickly:
Node.js v10.15.3
darwin 18.7.0
npm 6.11.3

Minimal Project for reproduce error:
https://github.com/nguyentoanit/popsicle-polly-error

Thank you for the detailed issue @moltar and for the reproduction @nguyentoanit but this an issue with nock which we use under the hood. See nock/nock#1643.

@offirgolan thanks! Any suggested workarounds?

@moltar the best thing to do would be to open a PR to nock. We can also patch it on our end in the mean time but #264 needs to be addressed first.

This was fixed in Nock as of 11.7.0


Exchange TilesOverlay without flickering

Issue Type
[ X] Question
[ ] Bug
[ ] Improvement
[ ] Build system related
[ ] Performance
[ ] Documentation
Description and/or steps/code to reproduce the problem
I am trying to create a RainViewer animation on top of OpenStreetMap. The code is working but when moving through the
animation it is flickering:
At every step I am adding a new overlay for the next step in time and remove the old one. So there is always just one overlay at a time.
In order to show the change it I need to call mapController.animateTo(center).
        final MapTileProviderBasic RainViewerTileProvider = new MapTileProviderBasic(this);
        final ITileSource RainViewerTileSource = new XYTileSource(radarFrames.getJSONObject(position).getString("time"), 1, 20, 256, "/2/1_1.png", new String[]{host+radarFrames.getJSONObject(position).getString("path")+"/256/"});
        RainViewerTileProvider.setTileSource(RainViewerTileSource);
        final TilesOverlay newRadarOverlay = new TilesOverlay(RainViewerTileProvider, this);

        mapView.getOverlays().add(newRadarOverlay);
        if (oldRadarOverlay !=null) mapView.getOverlays().remove(oldRadarOverlay);
        oldRadarOverlay = newRadarOverlay;

        IGeoPoint center = mapView.getMapCenter(); //refresh
        mapController.animateTo(center);

If I uncomment
        if (oldRadarOverlay !=null) mapView.getOverlays().remove(oldRadarOverlay);

then I can use mapController.setCenter(center) instead of animateTo(center) without flickering but that does not work if I remove the previous layer. If I use setCenter(center) in this case none of the overlays is shown until I move the map.
Any ideas how to fix this?
Version of osmdroid the issue relates to:
6.1.18

I have a working solution now which uses 2 MapViews and crossfades between them in an animation:
https://github.com/woheller69/omweather/blob/osmdroid/app/src/main/java/org/woheller69/weather/activities/RainViewerActivity.java
If there is a simpler solution, let me know.


Improve fork_choice spec test runner's time

Motivation

Fork choice spec test runs really slow (20x compared to v1.5.0) due to loading ckzg for all forks

Description

Only load ckzg for deneb in fork_choice spec test runner
Fix isForkWithdrawals util

:tada: This PR is included in v1.7.0 :tada:


Possible deadlock between DB.getNameForObject and DB.commit

The gist at https://gist.github.com/batterseapower/b59125cfb831933f6a27 shows how this can occur.
A simple fix is to remove the synchronized keyword from getNameForObject (the map it accesses is synchronized anyway!) and then stop mutating that map to a dummy value in close()

Thanks, nice catch.


Disqus support

Hi! I'd like to add disqus to your theme. I know hugo has it's own internal template to add disqus. But what would be the perfect file to add the code?
{{ template "_internal/disqus.html" . }}

Thanks in advance and awesome theme! ❤

Hello and thanks for checking out the theme!
Just as a matter of personal preference, I would like to keep the theme minimal and without commenting capabilities. However, you can easily override this in your own blog, by adding a file at layouts/_default/single.html, copying over the contents add https://github.com/lingxz/er/blob/master/layouts/_default/single.html, and add the disqus shortcode at the end.

Thanks for your quick reply.


Quantizing Phi-3 128k Instruct to FP8 fails.

System Info

GPU name: L40s
CUDA: 12.1

Wed Jun  5 16:27:21 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.14              Driver Version: 550.54.14      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA L40S                    On  |   00000000:25:00.0 Off |                    0 |
| N/A   35C    P8             33W /  350W |       0MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

TritonLLM: v0.10.0
Who can help?
@Tracin
Information

[X] The official example scripts
[ ] My own modified scripts

Tasks

[X] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
[ ] My own task or dataset (give details below)

Reproduction
Ran:
python3 ./examples/quantization/quantize.py --model_dir ../../Phi-3-mini-128k-instruct/ --dtype float16 --qformat fp8 --kv_cache_dtype fp8 --output_dir ./tllm_checkpoint_1gpu_fp8 --tp_size 1
Error:
(phi-3) root@150b096d1444:/workspace/tensorrt/TensorRT-LLM# python3 ./examples/quantization/quantize.py --model_dir ../../Phi-3-mini-128k-instruct/ --dtype float16 --qformat fp8 --kv_cache_dtype fp8 --output_dir ./tllm_checkpoint_1gpu_fp8 --tp_size 1
[TensorRT-LLM] TensorRT-LLM version: 0.10.0
Initializing model from ../../Phi-3-mini-128k-instruct/
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████| 2/2 [00:01<00:00,  1.06it/s]
[TensorRT-LLM][WARNING] The manually set model data type is torch.float16, but the data type of the HuggingFace model is torch.bfloat16.
Initializing tokenizer from ../../Phi-3-mini-128k-instruct/
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Loading calibration dataset
Starting quantization...
Inserted 387 quantizers
Calibrating batch 0
[06/05/2024-16:24:58] You are not running the flash-attention implementation, expect numerical differences.
Calibrating batch 1
Calibrating batch 2
Calibrating batch 3
Calibrating batch 4
Calibrating batch 5
Calibrating batch 6
Calibrating batch 7
Calibrating batch 8
Calibrating batch 9
Calibrating batch 10
Calibrating batch 11
Calibrating batch 12
Calibrating batch 13
Calibrating batch 14
Calibrating batch 15
Calibrating batch 16
Calibrating batch 17
Calibrating batch 18
Calibrating batch 19
Calibrating batch 20
Calibrating batch 21
Calibrating batch 22
Calibrating batch 23
Calibrating batch 24
Calibrating batch 25
Calibrating batch 26
Calibrating batch 27
Calibrating batch 28
Calibrating batch 29
Calibrating batch 30
Calibrating batch 31
Calibrating batch 32
Calibrating batch 33
Calibrating batch 34
Calibrating batch 35
Calibrating batch 36
Calibrating batch 37
Calibrating batch 38
Calibrating batch 39
Calibrating batch 40
Calibrating batch 41
Calibrating batch 42
Calibrating batch 43
Calibrating batch 44
Calibrating batch 45
Calibrating batch 46
Calibrating batch 47
Calibrating batch 48
Calibrating batch 49
Calibrating batch 50
Calibrating batch 51
Calibrating batch 52
Calibrating batch 53
Calibrating batch 54
Calibrating batch 55
Calibrating batch 56
Calibrating batch 57
Calibrating batch 58
Calibrating batch 59
Calibrating batch 60
Calibrating batch 61
Calibrating batch 62
Calibrating batch 63
Calibrating batch 64
Calibrating batch 65
Calibrating batch 66
Calibrating batch 67
Calibrating batch 68
Calibrating batch 69
Calibrating batch 70
Calibrating batch 71
Calibrating batch 72
Calibrating batch 73
Calibrating batch 74
Calibrating batch 75
Calibrating batch 76
Calibrating batch 77
Calibrating batch 78
Calibrating batch 79
Calibrating batch 80
Calibrating batch 81
Calibrating batch 82
Calibrating batch 83
Calibrating batch 84
Calibrating batch 85
Calibrating batch 86
Calibrating batch 87
Calibrating batch 88
Calibrating batch 89
Calibrating batch 90
Calibrating batch 91
Calibrating batch 92
Calibrating batch 93
Calibrating batch 94
Calibrating batch 95
Calibrating batch 96
Calibrating batch 97
Calibrating batch 98
Calibrating batch 99
Calibrating batch 100
Calibrating batch 101
Calibrating batch 102
Calibrating batch 103
Calibrating batch 104
Calibrating batch 105
Calibrating batch 106
Calibrating batch 107
Calibrating batch 108
Calibrating batch 109
Calibrating batch 110
Calibrating batch 111
Calibrating batch 112
Calibrating batch 113
Calibrating batch 114
Calibrating batch 115
Calibrating batch 116
Calibrating batch 117
Calibrating batch 118
Calibrating batch 119
Calibrating batch 120
Calibrating batch 121
Calibrating batch 122
Calibrating batch 123
Calibrating batch 124
Calibrating batch 125
Calibrating batch 126
Calibrating batch 127
Calibrating batch 128
Calibrating batch 129
Calibrating batch 130
Calibrating batch 131
Calibrating batch 132
Calibrating batch 133
Calibrating batch 134
Calibrating batch 135
Calibrating batch 136
Calibrating batch 137
Calibrating batch 138
Calibrating batch 139
Calibrating batch 140
Calibrating batch 141
Calibrating batch 142
Calibrating batch 143
Calibrating batch 144
Calibrating batch 145
Calibrating batch 146
Calibrating batch 147
Calibrating batch 148
Calibrating batch 149
Calibrating batch 150
Calibrating batch 151
Calibrating batch 152
Calibrating batch 153
Calibrating batch 154
Calibrating batch 155
Calibrating batch 156
Calibrating batch 157
Calibrating batch 158
Calibrating batch 159
Calibrating batch 160
Calibrating batch 161
Calibrating batch 162
Calibrating batch 163
Calibrating batch 164
Calibrating batch 165
Calibrating batch 166
Calibrating batch 167
Calibrating batch 168
Calibrating batch 169
Calibrating batch 170
Calibrating batch 171
Calibrating batch 172
Calibrating batch 173
Calibrating batch 174
Calibrating batch 175
Calibrating batch 176
Calibrating batch 177
Calibrating batch 178
Calibrating batch 179
Calibrating batch 180
Calibrating batch 181
Calibrating batch 182
Calibrating batch 183
Calibrating batch 184
Calibrating batch 185
Calibrating batch 186
Calibrating batch 187
Calibrating batch 188
Calibrating batch 189
Calibrating batch 190
Calibrating batch 191
Calibrating batch 192
Calibrating batch 193
Calibrating batch 194
Calibrating batch 195
Calibrating batch 196
Calibrating batch 197
Calibrating batch 198
Calibrating batch 199
Calibrating batch 200
Calibrating batch 201
Calibrating batch 202
Calibrating batch 203
Calibrating batch 204
Calibrating batch 205
Calibrating batch 206
Calibrating batch 207
Calibrating batch 208
Calibrating batch 209
Calibrating batch 210
Calibrating batch 211
Calibrating batch 212
Calibrating batch 213
Calibrating batch 214
Calibrating batch 215
Calibrating batch 216
Calibrating batch 217
Calibrating batch 218
Calibrating batch 219
Calibrating batch 220
Calibrating batch 221
Calibrating batch 222
Calibrating batch 223
Calibrating batch 224
Calibrating batch 225
Calibrating batch 226
Calibrating batch 227
Calibrating batch 228
Calibrating batch 229
Calibrating batch 230
Calibrating batch 231
Calibrating batch 232
Calibrating batch 233
Calibrating batch 234
Calibrating batch 235
Calibrating batch 236
Calibrating batch 237
Calibrating batch 238
Calibrating batch 239
Calibrating batch 240
Calibrating batch 241
Calibrating batch 242
Calibrating batch 243
Calibrating batch 244
Calibrating batch 245
Calibrating batch 246
Calibrating batch 247
Calibrating batch 248
Calibrating batch 249
Calibrating batch 250
Calibrating batch 251
Calibrating batch 252
Calibrating batch 253
Calibrating batch 254
Calibrating batch 255
Calibrating batch 256
Calibrating batch 257
Calibrating batch 258
Calibrating batch 259
Calibrating batch 260
Calibrating batch 261
Calibrating batch 262
Calibrating batch 263
Calibrating batch 264
Calibrating batch 265
Calibrating batch 266
Calibrating batch 267
Calibrating batch 268
Calibrating batch 269
Calibrating batch 270
Calibrating batch 271
Calibrating batch 272
Calibrating batch 273
Calibrating batch 274
Calibrating batch 275
Calibrating batch 276
Calibrating batch 277
Calibrating batch 278
Calibrating batch 279
Calibrating batch 280
Calibrating batch 281
Calibrating batch 282
Calibrating batch 283
Calibrating batch 284
Calibrating batch 285
Calibrating batch 286
Calibrating batch 287
Calibrating batch 288
Calibrating batch 289
Calibrating batch 290
Calibrating batch 291
Calibrating batch 292
Calibrating batch 293
Calibrating batch 294
Calibrating batch 295
Calibrating batch 296
Calibrating batch 297
Calibrating batch 298
Calibrating batch 299
Calibrating batch 300
Calibrating batch 301
Calibrating batch 302
Calibrating batch 303
Calibrating batch 304
Calibrating batch 305
Calibrating batch 306
Calibrating batch 307
Calibrating batch 308
Calibrating batch 309
Calibrating batch 310
Calibrating batch 311
Calibrating batch 312
Calibrating batch 313
Calibrating batch 314
Calibrating batch 315
Calibrating batch 316
Calibrating batch 317
Calibrating batch 318
Calibrating batch 319
Calibrating batch 320
Calibrating batch 321
Calibrating batch 322
Calibrating batch 323
Calibrating batch 324
Calibrating batch 325
Calibrating batch 326
Calibrating batch 327
Calibrating batch 328
Calibrating batch 329
Calibrating batch 330
Calibrating batch 331
Calibrating batch 332
Calibrating batch 333
Calibrating batch 334
Calibrating batch 335
Calibrating batch 336
Calibrating batch 337
Calibrating batch 338
Calibrating batch 339
Calibrating batch 340
Calibrating batch 341
Calibrating batch 342
Calibrating batch 343
Calibrating batch 344
Calibrating batch 345
Calibrating batch 346
Calibrating batch 347
Calibrating batch 348
Calibrating batch 349
Calibrating batch 350
Calibrating batch 351
Calibrating batch 352
Calibrating batch 353
Calibrating batch 354
Calibrating batch 355
Calibrating batch 356
Calibrating batch 357
Calibrating batch 358
Calibrating batch 359
Calibrating batch 360
Calibrating batch 361
Calibrating batch 362
Calibrating batch 363
Calibrating batch 364
Calibrating batch 365
Calibrating batch 366
Calibrating batch 367
Calibrating batch 368
Calibrating batch 369
Calibrating batch 370
Calibrating batch 371
Calibrating batch 372
Calibrating batch 373
Calibrating batch 374
Calibrating batch 375
Calibrating batch 376
Calibrating batch 377
Calibrating batch 378
Calibrating batch 379
Calibrating batch 380
Calibrating batch 381
Calibrating batch 382
Calibrating batch 383
Calibrating batch 384
Calibrating batch 385
Calibrating batch 386
Calibrating batch 387
Calibrating batch 388
Calibrating batch 389
Calibrating batch 390
Calibrating batch 391
Calibrating batch 392
Calibrating batch 393
Calibrating batch 394
Calibrating batch 395
Calibrating batch 396
Calibrating batch 397

Calibrating batch 398
Calibrating batch 399
Calibrating batch 400
Calibrating batch 401
Calibrating batch 402
Calibrating batch 403
Calibrating batch 404
Calibrating batch 405
Calibrating batch 406
Calibrating batch 407
Calibrating batch 408
Calibrating batch 409
Calibrating batch 410
Calibrating batch 411
Calibrating batch 412
Calibrating batch 413
Calibrating batch 414
Calibrating batch 415
Calibrating batch 416
Calibrating batch 417
Calibrating batch 418
Calibrating batch 419
Calibrating batch 420
Calibrating batch 421
Calibrating batch 422
Calibrating batch 423
Calibrating batch 424
Calibrating batch 425
Calibrating batch 426
Calibrating batch 427
Calibrating batch 428
Calibrating batch 429
Calibrating batch 430
Calibrating batch 431
Calibrating batch 432
Calibrating batch 433
Calibrating batch 434
Calibrating batch 435
Calibrating batch 436
Calibrating batch 437
Calibrating batch 438
Calibrating batch 439
Calibrating batch 440
Calibrating batch 441
Calibrating batch 442
Calibrating batch 443
Calibrating batch 444
Calibrating batch 445
Calibrating batch 446
Calibrating batch 447
Calibrating batch 448
Calibrating batch 449
Calibrating batch 450
Calibrating batch 451
Calibrating batch 452
Calibrating batch 453
Calibrating batch 454
Calibrating batch 455
Calibrating batch 456
Calibrating batch 457
Calibrating batch 458
Calibrating batch 459
Calibrating batch 460
Calibrating batch 461
Calibrating batch 462
Calibrating batch 463
Calibrating batch 464
Calibrating batch 465
Calibrating batch 466
Calibrating batch 467
Calibrating batch 468
Calibrating batch 469
Calibrating batch 470
Calibrating batch 471
Calibrating batch 472
Calibrating batch 473
Calibrating batch 474
Calibrating batch 475
Calibrating batch 476
Calibrating batch 477
Calibrating batch 478
Calibrating batch 479
Calibrating batch 480
Calibrating batch 481
Calibrating batch 482
Calibrating batch 483
Calibrating batch 484
Calibrating batch 485
Calibrating batch 486
Calibrating batch 487
Calibrating batch 488
Calibrating batch 489
Calibrating batch 490
Calibrating batch 491
Calibrating batch 492
Calibrating batch 493
Calibrating batch 494
Calibrating batch 495
Calibrating batch 496
Calibrating batch 497
Calibrating batch 498
Calibrating batch 499
Calibrating batch 500
Calibrating batch 501
Calibrating batch 502
Calibrating batch 503
Calibrating batch 504
Calibrating batch 505
Calibrating batch 506
Calibrating batch 507
Calibrating batch 508
Calibrating batch 509
Calibrating batch 510
Calibrating batch 511
Quantization done. Total time used: 39.34 s.
Unknown model type Phi3ForCausalLM. Continue exporting...
torch.distributed not initialized, assuming single world_size.
torch.distributed not initialized, assuming single world_size.
torch.distributed not initialized, assuming single world_size.
torch.distributed not initialized, assuming single world_size.
torch.distributed not initialized, assuming single world_size.
torch.distributed not initialized, assuming single world_size.
torch.distributed not initialized, assuming single world_size.
current rank: 0, tp rank: 0, pp rank: 0
torch.distributed not initialized, assuming single world_size.
torch.distributed not initialized, assuming single world_size.
Cannot export model to the model_config. The modelopt-optimized model state_dict (including the quantization factors) is saved to tllm_checkpoint_1gpu_fp8/modelopt_model.0.pth using torch.save for further inspection.
Detailed export error: 'unknown:Phi3ForCausalLM'
Traceback (most recent call last):
  File "/workspace/tensorrt/TensorRT-LLM/phi-3/lib/python3.10/site-packages/modelopt/torch/export/model_config_export.py", line 364, in export_tensorrt_llm_checkpoint
    for tensorrt_llm_config, weights in torch_to_tensorrt_llm_checkpoint(
  File "/workspace/tensorrt/TensorRT-LLM/phi-3/lib/python3.10/site-packages/modelopt/torch/export/model_config_export.py", line 312, in torch_to_tensorrt_llm_checkpoint
    tensorrt_llm_config = convert_to_tensorrt_llm_config(model_config, tp_size_overwrite)
  File "/workspace/tensorrt/TensorRT-LLM/phi-3/lib/python3.10/site-packages/modelopt/torch/export/tensorrt_llm_utils.py", line 84, in convert_to_tensorrt_llm_config
    "architecture": MODEL_NAME_TO_HF_ARCH_MAP[decoder_type],
KeyError: 'unknown:Phi3ForCausalLM'
Traceback (most recent call last):
  File "/workspace/tensorrt/TensorRT-LLM/./examples/quantization/quantize.py", line 90, in <module>
    quantize_and_export(
  File "/workspace/tensorrt/TensorRT-LLM/phi-3/lib/python3.10/site-packages/tensorrt_llm/quantization/quantize_by_modelopt.py", line 340, in quantize_and_export
    with open(f"{export_path}/config.json", "r") as f:
FileNotFoundError: [Errno 2] No such file or directory: './tllm_checkpoint_1gpu_fp8/config.json'

Expected behavior
Quantized Phi-3 weights would get saved
actual behavior
Gives error: Detailed export error: 'unknown:Phi3ForCausalLM' and fails.
additional notes
None.

Could you try the TRT-LLM main branch?
i.e,
Docker base image : nvcr.io/nvidia/tritonserver:24.05-py3
tensorrt-llm version : 0.11.0.dev2024060400

Reply from Qi: Support FP8 of Phi3 is supposed to be in the next release.

Can I somehow use something else to get float8 quant? Can I maybe convert phi-3 huggingface to float 8 using: https://huggingface.co/blog/quanto-introduction and then load them using convert_checkpoint in TensorRTLLM? Will that work?

@nv-guomingz will this solution work? or is this also reliant on modelopt?

TRT-LLM only leaverage the nvidia-modelopt for quantization till now. We don't evaluate the  https://huggingface.co/blog/quanto-introduction yet.
My guts telling me that it's possible to use this tool for fp8 scaling factor generation but it needs additional efforts(glue code development) to apply them on trtllm checkponit generation.
We've filed bug to model opt team and I think this issue could be fixed soon.

Phi3 mini model changed after modelopt0.11 is out. We plan to launch a new version of modelopt soon which will fix this issue.

Hi @kalradivyanshu  , modelopt 0.13.0 had been released for several days and please have a try with it.
I've verified the python3 ./examples/quantization/quantize.py --model_dir ../../Phi-3-mini-128k-instruct/ --dtype float16 --qformat fp8 --kv_cache_dtype fp8 --output_dir ./tllm_checkpoint_1gpu_fp8 --tp_size 1 on my side and there's no crash now.
If the issue doesn't exist any more, please close this ticket.

**when we using fp8 / int4_awq with not enough vram. we saw it would automatic offload some parameters. it's really thankful!
however, it would have some bug to save weight to safetensors after quantizing. it would loss the config and the weight would save  to be modelopt.pth first, it's really confused. please help us solved.**
Quantization done. Total time used: 11516.87 s.
torch.distributed not initialized, assuming single world_size.
torch.distributed not initialized, assuming single world_size.
torch.distributed not initialized, assuming single world_size.
torch.distributed not initialized, assuming single world_size.
torch.distributed not initialized, assuming single world_size.
torch.distributed not initialized, assuming single world_size.
Cannot export model to the model_config. The modelopt-optimized model state_dict (including the quantization factors) is saved to llama3-70b-fp8/modelopt_model.0.pth using torch.save for further inspection.
Detailed export error: Cannot copy out of meta tensor; no data!
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/modelopt/torch/export/model_config_export.py", line 364, in export_tensorrt_llm_checkpoint
for tensorrt_llm_config, weights in torch_to_tensorrt_llm_checkpoint(
File "/usr/local/lib/python3.10/dist-packages/modelopt/torch/export/model_config_export.py", line 220, in torch_to_tensorrt_llm_checkpoint
build_decoder_config(layer, model_metadata_config, decoder_type, dtype)
File "/usr/local/lib/python3.10/dist-packages/modelopt/torch/export/layer_utils.py", line 1179, in build_decoder_config
config.attention = build_attention_config(layer, model_metadata_config, dtype, config)
File "/usr/local/lib/python3.10/dist-packages/modelopt/torch/export/layer_utils.py", line 649, in build_attention_config
config.dense = build_linear_config(layer, LINEAR_ROW, dtype)
File "/usr/local/lib/python3.10/dist-packages/modelopt/torch/export/layer_utils.py", line 605, in build_linear_config
config.weight = weight.cpu()
NotImplementedError: Cannot copy out of meta tensor; no data!
Traceback (most recent call last):
File "/media/mnt/sdb/willy/tensorrt_llm/TensorRT-LLM/examples/quantization/llama3-70-quantize.py", line 107, in
quantize_and_export(
File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/quantization/quantize_by_modelopt.py", line 451, in quantize_and_export
with open(f"{export_path}/config.json", "r") as f:
FileNotFoundError: [Errno 2] No such file or directory: 'llama3-70b-fp8/config.json'

My version is tensorrt-llm 0.13, and the command I used is  python3 ../quantization/quantize.py   --model_dir  phi3-model  --output_dir  ./phi_out/  \  --dtype bfloat16  \   --qformat fp8  --kv_cache_dtype fp8     no error.
But   when i use the command:trtllm-build   --checkpoint_dir ./phi_out/ --output_dir ./phi_engine/   --gemm_plugin auto   --max_batch_size 8  --max_input_len 1024   --max_seq_len 2048   .the error is  AttributeError: 'NoneType' object has no attribute 'ndim'.

Hi @kalradivyanshu , modelopt 0.13.0 had been released for several days and please have a try with it. I've verified the python3 ./examples/quantization/quantize.py --model_dir ../../Phi-3-mini-128k-instruct/ --dtype float16 --qformat fp8 --kv_cache_dtype fp8 --output_dir ./tllm_checkpoint_1gpu_fp8 --tp_size 1 on my side and there's no crash now.
If the issue doesn't exist any more, please close this ticket.

when i use the command:trtllm-build --checkpoint_dir ./phi_out/ --output_dir ./phi_engine/ --gemm_plugin auto --max_batch_size 8 --max_input_len 1024 --max_seq_len 2048 .the error is AttributeError: 'NoneType' object has no attribute 'ndim'.   how to solve?

Cannot copy out of meta tensor; no data!

have you solved it ?

Hi @kalradivyanshu  please reopen it if needed


Line break after saving file

Prerequisites

[X] Put an X between the brackets on this line if you have done all of the following:

Reproduced the problem in Safe Mode: http://flight-manual.atom.io/hacking-atom/sections/debugging/#using-safe-mode
Followed all applicable steps in the debugging guide: http://flight-manual.atom.io/hacking-atom/sections/debugging/
Checked the FAQs on the message board for common solutions: https://discuss.atom.io/c/faq
Checked that your issue isn't already filed: https://github.com/issues?utf8=✓&q=is%3Aissue+user%3Aatom
Checked that there is not already an Atom package that provides the described functionality: https://atom.io/packages

Description
Atom adds useless line break after saving file.
Steps to Reproduce
Following steps are intended to terminal:

touch test
atom test
add some text into the first line of the file
press save (cmd+s)
the see second line in use

Expected behavior:
See only lines, that I write
Actual behavior:
Atom adds line break
Reproduces how often:
10/10
Additional Information
atom version:
Atom    : 1.16.0
Electron: 1.3.13
Chrome  : 52.0.2743.82
Node    : 6.5.0

macos: 10.12.4

Isn't that normal behavior? ESLint in fact has a rule requiring you to add a line break and it's true everywhere including the ESLint:recommended configuration.

The whitespace package will do that by default. You can disable it using a checkbox in the package settings for that package. It's a core package, so it ships with all installs of Atom.

As @tangledhelix stated, this is a feature of the whitespace package and is intended behavior. Follow these instructions to disable the behavior:

Launch Atom
Open Settings View using Cmd+, on macOS or Ctrl+, on other platforms
Click the Packages tab
Search for "whitespace"
Click the Settings button on the whitespace package's card
Uncheck "Ensure Single Trailing Newline"

@lee-dohm thank you, it works


Merge of docker/fig:master and texttable from PyPI

this PR includes:

all commits from docker/fig:master until 2015/01/22
a commit where i changed the source of texttable to the PyPI, where a Python3-compatible version is now hosted

it's intended to save you some work. ;-)
however the test ServiceTest.test_recreate_containers_with_image_declared_volume fails with different outcomes under Python 2 and Python 3.

Awesome, thanks for the contribution!

The build is having issues because some earlier commits are missing the signoff. I'll merge without the build


chore(DATA-1438): 🤖 remove deprecated "issueNumber" from default fields

should we remove the others too?

No, it is sill considered as obligatory when creating the issues. But maybe we should make it optional, @p-desaintchamas do you agree ?


Manage multiple disjoint instances of RocksDB on the fly

Development Task
Provide the capability to create multiple disjoint instances from one instance, and merge multiple instances into one.
RocksDB already provides checkpoint to create a clone of an existing instance. So the first part can be trivially implemented as checkpoint followed by an out-of-range deletion.
The second part can be broken down into:

[x] Physically delete all data except for a given range (as contrary to logically deleting via key deletion marks or range deletion marks): CompactRange can do the job.
[x] Merge two instances that are physically disjoint: https://github.com/tikv/rocksdb/pull/294
[x] Preserve cache of old instances (primarily block cache)

Memo for the implementation of Merge:

Pause flush of all existing DBs.
Merge table files. This operation is trivial as table files are not overlapped.
Merge WALs. The log number of WALs from different DBs will be altered to have no conflict and be larger than the max_log_number of all table files. This number will be stable because flush is paused before.
Merge memtables by ref counting the memtable pointers from existing instances.
Log and apply the version edits of all file operations in previous steps.

Implemented in https://github.com/tikv/rocksdb/pull/294


Adding multiple views to a layout in addition to CardStackLayout

Currently we are only able to add views (inside same layout.xml) in addition to CardStackLayout with Coordination. What if we want to have a View linear or relative to CardStackLayout, or consider two CardStackLayout below one another having the same functionality.

@OmkarAmberkar Is that not possible now? Do you mean the height of cards is not adjusting according to the container or that you're not able to add multiple view along side card stack view?


Use AnyHttp with httpc adapter

https://github.com/GRoguelon/any_http/

👍

Closed in favor of https://github.com/rlopzc/logger_exporter/issues/45


Adding Krishnamurthy Number.

🚀 Feature
(A clear and concise description of what the feature is.)
A Krishnamurthy number is a number whose sum of the factorial of digits is equal to the number itself.
Have you read the Contributing Guidelines on Pull Requests?
(Write your answer here.)
Yes.
Motivation
(Please outline the motivation for the proposal.)
As a part of my Gssoc'21 contribution, I want to add Krishnamurthy number solution under C language.
Pitch
(Please explain why this feature should be implemented and how it would be used.)
This solution I found was missing in the existing problems and hence I want to give the solution.

@Madhumitamoi I assigned you, Happy Coding!😊

Added Krishnamurthy number.


feature request: choose specific calendar

Hi! I'm looking for something like this, so I'm happy that you are developing it! However, I use several of my calendars for time blocking, and I can't have "Do Not Disturb" start for those events. Can I pick just one calendar to use with this app? That might make it workable for me. Thanks.

Hi! Glad you've found this useful :)
I could see this being a useful addition. I plan to work on an update soon fixing support in MacOS ~14.4.1 and could get this added in that update. I'll update you in this issue
Thanks for the feature request!


Incorrect use of ParentDataWidget

Hello.
I'm using Dropdown Below library for a while and after upgrading my Flutter version to latest version, a problem happened. I was digging deep to find out where is the render object problem but after some time I gave up and decided to write a new screen from scratch in my application. At that point, I discovered a problem inside Dropdown Below.
This is the track of message from terminal.
`══╡ EXCEPTION CAUGHT BY WIDGETS LIBRARY ╞═══════════════════════════════════════════════════════════
The following assertion was thrown while applying parent data.:
Incorrect use of ParentDataWidget.
The ParentDataWidget Expanded(flex: 1) wants to apply ParentData of type FlexParentData to a
RenderObject, which has been set up to accept ParentData of incompatible type ParentData.
Usually, this means that the Expanded widget has the wrong ancestor RenderObjectWidget. Typically,
Expanded widgets are placed directly inside Flex widgets.
The offending Expanded is currently placed inside a ConstrainedBox widget.
The ownership chain for the RenderObject that received the incompatible parent data was:
IndexedStack ← Expanded ← ConstrainedBox ← Container ← DefaultTextStyle ← Stack ← Listener ←
_GestureSemantics ← RawGestureDetector ← GestureDetector ← ⋯
When the exception was thrown, this was the stack:
#0      RenderObjectElement._updateParentData. (package:flutter/src/widgets/framework.dart:5723:11)
#1      RenderObjectElement._updateParentData (package:flutter/src/widgets/framework.dart:5739:6)
#2      RenderObjectElement.attachRenderObject (package:flutter/src/widgets/framework.dart:5761:7)
#3      RenderObjectElement.mount (package:flutter/src/widgets/framework.dart:5440:5)
#4      MultiChildRenderObjectElement.mount (package:flutter/src/widgets/framework.dart:6228:11)
...     Normal element mounting (22 frames)
#26     Element.inflateWidget (package:flutter/src/widgets/framework.dart:3611:14)
#27     MultiChildRenderObjectElement.inflateWidget (package:flutter/src/widgets/framework.dart:6221:36)
#28     MultiChildRenderObjectElement.mount (package:flutter/src/widgets/framework.dart:6232:32)
...     Normal element mounting (32 frames)
#60     Element.inflateWidget (package:flutter/src/widgets/framework.dart:3611:14)
#61     MultiChildRenderObjectElement.inflateWidget (package:flutter/src/widgets/framework.dart:6221:36)
#62     MultiChildRenderObjectElement.mount (package:flutter/src/widgets/framework.dart:6232:32)
...     Normal element mounting (103 frames)
#165    Element.inflateWidget (package:flutter/src/widgets/framework.dart:3611:14)
#166    MultiChildRenderObjectElement.inflateWidget (package:flutter/src/widgets/framework.dart:6221:36)
#167    MultiChildRenderObjectElement.mount (package:flutter/src/widgets/framework.dart:6232:32)
...     Normal element mounting (73 frames)
#240    Element.inflateWidget (package:flutter/src/widgets/framework.dart:3611:14)
#241    MultiChildRenderObjectElement.inflateWidget (package:flutter/src/widgets/framework.dart:6221:36)
#242    Element.updateChild (package:flutter/src/widgets/framework.dart:3363:18)
#243    RenderObjectElement.updateChildren (package:flutter/src/widgets/framework.dart:5654:32)
#244    MultiChildRenderObjectElement.update (package:flutter/src/widgets/framework.dart:6243:17)
#245    Element.updateChild (package:flutter/src/widgets/framework.dart:3350:15)
#246    SingleChildRenderObjectElement.update (package:flutter/src/widgets/framework.dart:6090:14)
#247    Element.updateChild (package:flutter/src/widgets/framework.dart:3350:15)
#248    ComponentElement.performRebuild (package:flutter/src/widgets/framework.dart:4599:16)
#249    Element.rebuild (package:flutter/src/widgets/framework.dart:4267:5)
#250    StatelessElement.update (package:flutter/src/widgets/framework.dart:4655:5)
#251    Element.updateChild (package:flutter/src/widgets/framework.dart:3350:15)
#252    ComponentElement.performRebuild (package:flutter/src/widgets/framework.dart:4599:16)
#253    Element.rebuild (package:flutter/src/widgets/framework.dart:4267:5)
#254    StatelessElement.update (package:flutter/src/widgets/framework.dart:4655:5)
#255    Element.updateChild (package:flutter/src/widgets/framework.dart:3350:15)
#256    ComponentElement.performRebuild (package:flutter/src/widgets/framework.dart:4599:16)
#257    Element.rebuild (package:flutter/src/widgets/framework.dart:4267:5)
#258    ProxyElement.update (package:flutter/src/widgets/framework.dart:4922:5)
#259    Element.updateChild (package:flutter/src/widgets/framework.dart:3350:15)
#260    ComponentElement.performRebuild (package:flutter/src/widgets/framework.dart:4599:16)
#261    Element.rebuild (package:flutter/src/widgets/framework.dart:4267:5)
#262    ProxyElement.update (package:flutter/src/widgets/framework.dart:4922:5)
#263    Element.updateChild (package:flutter/src/widgets/framework.dart:3350:15)
#264    RenderObjectElement.updateChildren (package:flutter/src/widgets/framework.dart:5596:32)
#265    MultiChildRenderObjectElement.update (package:flutter/src/widgets/framework.dart:6243:17)
#266    Element.updateChild (package:flutter/src/widgets/framework.dart:3350:15)
#267    ComponentElement.performRebuild (package:flutter/src/widgets/framework.dart:4599:16)
#268    StatefulElement.performRebuild (package:flutter/src/widgets/framework.dart:4746:11)
#269    Element.rebuild (package:flutter/src/widgets/framework.dart:4267:5)
#270    StatefulElement.update (package:flutter/src/widgets/framework.dart:4778:5)
#271    Element.updateChild (package:flutter/src/widgets/framework.dart:3350:15)
#272    ComponentElement.performRebuild (package:flutter/src/widgets/framework.dart:4599:16)
#273    Element.rebuild (package:flutter/src/widgets/framework.dart:4267:5)
#274    ProxyElement.update (package:flutter/src/widgets/framework.dart:4922:5)
#275    Element.updateChild (package:flutter/src/widgets/framework.dart:3350:15)
#276    ComponentElement.performRebuild (package:flutter/src/widgets/framework.dart:4599:16)
#277    StatefulElement.performRebuild (package:flutter/src/widgets/framework.dart:4746:11)
#278    Element.rebuild (package:flutter/src/widgets/framework.dart:4267:5)
#279    StatefulElement.update (package:flutter/src/widgets/framework.dart:4778:5)
#280    Element.updateChild (package:flutter/src/widgets/framework.dart:3350:15)
#281    SingleChildRenderObjectElement.update (package:flutter/src/widgets/framework.dart:6090:14)
#282    Element.updateChild (package:flutter/src/widgets/framework.dart:3350:15)
#283    ComponentElement.performRebuild (package:flutter/src/widgets/framework.dart:4599:16)
#284    Element.rebuild (package:flutter/src/widgets/framework.dart:4267:5)
#285    StatelessElement.update (package:flutter/src/widgets/framework.dart:4655:5)
#286    Element.updateChild (package:flutter/src/widgets/framework.dart:3350:15)
#287    SingleChildRenderObjectElement.update (package:flutter/src/widgets/framework.dart:6090:14)
#288    Element.updateChild (package:flutter/src/widgets/framework.dart:3350:15)
#289    ComponentElement.performRebuild (package:flutter/src/widgets/framework.dart:4599:16)
#290    StatefulElement.performRebuild (package:flutter/src/widgets/framework.dart:4746:11)
#291    Element.rebuild (package:flutter/src/widgets/framework.dart:4267:5)
#292    StatefulElement.update (package:flutter/src/widgets/framework.dart:4778:5)
#293    Element.updateChild (package:flutter/src/widgets/framework.dart:3350:15)
#294    ComponentElement.performRebuild (package:flutter/src/widgets/framework.dart:4599:16)
#295    StatefulElement.performRebuild (package:flutter/src/widgets/framework.dart:4746:11)
#296    Element.rebuild (package:flutter/src/widgets/framework.dart:4267:5)
#297    StatefulElement.update (package:flutter/src/widgets/framework.dart:4778:5)
#298    Element.updateChild (package:flutter/src/widgets/framework.dart:3350:15)
#299    ComponentElement.performRebuild (package:flutter/src/widgets/framework.dart:4599:16)
#300    Element.rebuild (package:flutter/src/widgets/framework.dart:4267:5)
#301    ProxyElement.update (package:flutter/src/widgets/framework.dart:4922:5)
#302    Element.updateChild (package:flutter/src/widgets/framework.dart:3350:15)
#303    ComponentElement.performRebuild (package:flutter/src/widgets/framework.dart:4599:16)
#304    StatefulElement.performRebuild (package:flutter/src/widgets/framework.dart:4746:11)
#305    Element.rebuild (package:flutter/src/widgets/framework.dart:4267:5)
#306    StatefulElement.update (package:flutter/src/widgets/framework.dart:4778:5)
#307    Element.updateChild (package:flutter/src/widgets/framework.dart:3350:15)
#308    ComponentElement.performRebuild (package:flutter/src/widgets/framework.dart:4599:16)
#309    StatefulElement.performRebuild (package:flutter/src/widgets/framework.dart:4746:11)
#310    Element.rebuild (package:flutter/src/widgets/framework.dart:4267:5)
#311    BuildOwner.buildScope (package:flutter/src/widgets/framework.dart:2582:33)
#312    WidgetsBinding.drawFrame (package:flutter/src/widgets/binding.dart:875:21)
#313    RendererBinding._handlePersistentFrameCallback (package:flutter/src/rendering/binding.dart:328:5)
#314    SchedulerBinding._invokeFrameCallback (package:flutter/src/scheduler/binding.dart:1144:15)
#315    SchedulerBinding.handleDrawFrame (package:flutter/src/scheduler/binding.dart:1082:9)
#316    SchedulerBinding._handleDrawFrame (package:flutter/src/scheduler/binding.dart:998:5)
#320    _invoke (dart:ui/hooks.dart:163:10)
#321    PlatformDispatcher._drawFrame (dart:ui/platform_dispatcher.dart:259:5)
#322    _drawFrame (dart:ui/hooks.dart:126:31)
(elided 3 frames from dart:async)`
This is the version of flutter in my computer.
[✓] Flutter (Channel stable, 2.2.2, on macOS 11.4 20F71 darwin-x64, locale en-GB)     • Flutter version 2.2.2 at /Users/user/development/flutter     • Framework revision d79295af24 (3 weeks ago), 2021-06-11 08:56:01 -0700     • Engine revision 91c9fc8fe0     • Dart version 2.13.3
This is the code piece I was using perfectly before upgrading Flutter version:
`
Padding(
padding: const EdgeInsets.only(right: 16.0),
child: Row(
mainAxisAlignment: MainAxisAlignment.end,
children: [
DropdownBelow(
itemWidth: 100,
itemTextstyle: TextStyle(
fontSize: 14,
fontWeight: FontWeight.w400,
color: Colors.black),
boxTextstyle: TextStyle(
fontSize: 14,
fontWeight: FontWeight.w400,
color: Color(0XFFbbbbbb)),
boxPadding: const EdgeInsets.fromLTRB(13, 12, 0, 12),
boxWidth: 100,
boxHeight: 45,
hint: Text('Filter'),
value: _selectedTest,
items: _dropdownTestItems,
onChanged: onChangeDropdownTests,
),
],
),
),
`
And I find out that problem may come from dropdown_below.dart. Please check out line 590. The code piece fits to the cause of error message.
Line 590 as a screenshot below

I hope it helps you to make your package better. If that problem is related to my code, I'm sorry that I have disturbed you.

Hi @Trpstn, Thx to report issue!
Your explain is very helpful to me.
And now I publish new version 0.0.9 that fix this issue.
You can test and close issue if you want. XD


<IP_ADDRESS> Released on github is not usable, should it be removed from released?

This make world misunderstand the latest release of Dash.
Please kindly fix this.

We need this tag for internal use, it is not a "release". Please always refer to offical website what version is latest/stable.
Thanks,
Holger


rultor merges takes too much long

We have to merge this pull: #125. So, we triggered @rultor for this to do the job. However, merge already in progress for 40 min, and counting.. Besides that, rultor outputs a lot of logs irrelevant to our build: cargofmt lints on Rust internals on so on. What's interesting, that we didn't change the rultor image version or rultor merge script. Need to fix this.
Rultor build log: https://www.rultor.com/t/43616-2324144304

@l3r8yJ can you take this one?


Unthemable elements

Continuation of #2892.
@pdf Please take a look and see if I got everything right. Thanks a lot.
@oliviertassinari @newoga Please take a look Thanks.

@alitaheri thanks for running with this - I haven't tested it, but other than the syntax error this LGTM :+1:

@alitaheri Haven't tested but looks good so far.

Guys take another look pelase :grin:

@alitaheri Thanks for restarting this. That looks good. I think that you can squash down and merge :+1:.

Cool. I'll have @newoga and @pdf Take a final look and make sure they are satisfied with it. mean while I'll squash down. Thanks :grin:

@alitaheri I'm good with this :+1:

Thanks @newoga :grin:
@pdf Thanks a lot for getting this in the first place :+1: :+1:


Improve the CSS situation

(this comes from https://github.com/umaranis/svelte-lexical/issues/51#issuecomment-1941552464, which is a closed issue, so I opened a new one here so we can comment on this)
In my opinion, we should make something to fix the CSS mess. This are the problems I'm seeing:
1- Conflicts with other libraries (https://github.com/umaranis/svelte-lexical/issues/71). I fixed this partially by nesting all the lexical.css classes inside another class:
.lexical-div {
    body {/**/}

    header {/**/}

    header a {/**/}

    header a {/**/}
}

In this way, I put the Lexical editor inside a lexical-div div, and classes in this file only affect the editor. If I don't add this, the lexical.css file is changing a lot of classes very used in apps, like body, dropdown, modal, switch, toolbar, etc.
But I still have the reverse problem: external libraries changing svelte-lexical behaviour by defining classes like modal, dropdown, etc.
2- Output of Lexical converted to HTML cannot be displayed in the same way as it's in the editor, because in PlaygroundEditorTheme.css, all the classes are defined with a prefix. So I'm thinking on just duplicating the PlaygroundEditorTheme.css file and removing the prefix, so it can be used in standard html... But that's ugly.
So, to fix both problems, I think we should do this:
1- In lexical.css, put lexical-specific class names, like it's done at the end of the file with Modal__content or ToolbarPlugin__dialogActions. In this way, external libraries cannot break this plugin, and this plugin doesn't break external components.
2- In PlaygroundEditorTheme.css, we remove the prefix (changes in code needed, of course) and wrap everything inside a super-class, so we don't have the problem we have in 1).
@umaranis  what do you think?
I could help with a PR if you think this is a good idea.

@umaranis Did you had a second to give a look to this?
@PaddiM8 If you have a moment, I would like you to also give a look to this.
Thanks!

Sorry for the delayed response @ktecho
Here is my view on the issue. We don't want to style the components out of the box (in the library itself) because often users would like to style them according to their design system and CSS framework. That's why styles are provided separately and help in getting started. They are good enough to suffice in some of the use cases.
Now, how do we provide the users with the ability to style components as they like without having any conflicts? For the core editor (excluding dialogs, toolbar etc.), this is done with PlaygroundEditorTheme.ts file in playground/src/theme folder. This provides full control to the users to specify any styles for the various elements shown inside the editor (no hard-coded class names). The sample styling is provided in demo projects to get started. It uses prefixes with class names to make any conflicts unlikely for those who are getting started or experimenting with svelte-lexical. The styles are provided in a simple CSS file and it is left to the library users to adjust them according to their preferences and CSS framework. Because there are no hard-coded class names inside the library, we can avoid conflicts.
When it comes to other UI components like the toolbar, dialogs etc. we have a few issues:

Class names are hardcoded. We need to apply the same logic to styling these UI components as was applied to the editor. It has a precedent in facebook/lexical repo where it has been done for comments panel.
The styling provided in demo projects should use a prefix to avoid conflicts for anyone getting started with the library.

@ktecho does it make sense? Would you like to take it up and apply the same styling logic to UI components as applied in the editor? Thanks.

yes I agree with @umaranis