fontconfig: 2.12.6 -> 2.13.92

Motivation for this change
~This currently fails to build at postInstall with~
/nix/store/hmqnlh2nvjc4893bxyr0n9i3zyvsh5c3-fontconfig-2.13.92/share/xml/fontconfig/fonts.dtd:127: parser error : xmlParseElementDecl: 'EMPTY', 'ANY' or '(' expected
<!ELEMENT reset-dirs >
                     ^
/nix/store/hmqnlh2nvjc4893bxyr0n9i3zyvsh5c3-fontconfig-2.13.92/share/xml/fontconfig/fonts.dtd:127: parser error : Content error in the external subset
<!ELEMENT reset-dirs >
                     ^
unable to parse /nix/store/hmqnlh2nvjc4893bxyr0n9i3zyvsh5c3-fontconfig-2.13.92/etc/fonts/fonts.conf

Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[ ] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[ ] Fits CONTRIBUTING.md.

Notify maintainers
cc @
Closes: https://github.com/NixOS/nixpkgs/issues/73725

Is the reset-dirs element invalid ?

https://gitlab.freedesktop.org/fontconfig/fontconfig/commit/def1d00036a4e828382027292a167203c6c7a0b4

The DTD looks wrong. The element should have EMPTY as a content specifier.

Un-drafted because it isn't broke anymore. Thanks @jtojnar for the quick patch :sparkles:

Does configVersion need to be changed, and what does this mean for fontconfig_210? Do we need three nested sets of conf directories now, or only the two as we have presently?
(and how do we know/determine the answers to these questions?)
The above matters for having a good story for folks using paths built against current fontconfig, etc...

Does configVersion need to be changed, and what does this mean for fontconfig_210? Do we need three nested sets of conf directories now, or only the two as we have presently?
(and how do we know/determine the answers to these questions?)
The above matters for having a good story for folks using paths built against current fontconfig, etc...

Probably all good questions for its maintainer.
I don't think I can answer any of those questions.
Thanks for considering backwards compat though, but we do really need to upgrade our fontconfig.

What is wrong with our current version? Is not x.13 unstable? x.y.92, I would definitely expect to be.

What is wrong with our current version? Is not x.13 unstable? x.y.92, I would definitely expect to be.

Yes it is a development release, but I'd really like to get it tested while still very early in 20.03 dev.

@jtojnar Though stable is actually 2.13.1

I think fontconfig made breaking configuration changes in 2.13, so maybe configVersion should be bumped?

I think fontconfig made breaking configuration changes in 2.13, so maybe configVersion should be bumped?

I guess that would be adding a new element. However, I'm not certain how this works in the first place in nixpkgs. Why do we have fontconfig 2.10?

Ah, I see it's backported to generate font caches
  # back-supported fontconfig version and package
  # version is used for font cache generation
  supportVersion = "210";
  supportPkg     = pkgs."fontconfig_${supportVersion}";

  # latest fontconfig version and package
  # version is used for configuration folder name, /etc/fonts/VERSION/
  # note: format differs from supportVersion and can not be used with makeCacheConf
  latestVersion  = pkgs.fontconfig.configVersion;
  latestPkg      = pkgs.fontconfig;

Debian unstable seems to have this beta version https://github.com/NixOS/nixpkgs/issues/73725#issuecomment-555663378.

I will circle back to resyncing this and seeing what we can do with compatibility. In the meantime anyone is free to re-PR the changes.

Hope this request will be merge soon. I got error that I can not start nix GUI app because of the fontconfig version on my machine is newer than nixpkgs fonconfig version

please resolve conflicts :)

I hope you don't mind my fixing the merge conflict (can squash out the merge commit when it comes time to merge into master).
I'd like to triage this because it seems like it's affecting lots of people, especially those who are using nix on non-NixOS systems; it looks like the only controversy here is that fontconfig has lots and lots of reverse-dependencies, but is there anything actually left to do here? As @worldofpeace said, we really need to update our fontconfig..
cc @worldofpeace @jtojnar @sauyon @FRidh @thongpv87 @jonringer

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/prs-in-distress/3604/23

@NickHu Number of reverse dependencies is not a problem at all, the stability and backwards compatibility is.
2.13.92 is an unstable version but if it is good enough for Arch, it should probably work for us too.
The worse issue is the compatibility so that people on stable NixOS can still use programs from unstable package set and vice versa.
Someone will need to figure out what does this upgrade mean for fontconfig cache and configuration, see https://github.com/NixOS/nixpkgs/pull/73795#issuecomment-555810414 and https://github.com/NixOS/nixpkgs/pull/73795#issuecomment-557705533.
Possibly, we might need to not fallback to /etc/fontconfig in https://github.com/NixOS/nixpkgs/blob/72066757c8bd31cc592914cb944f8d7da569d741/pkgs/development/libraries/fontconfig/config-compat.patch and always use versioned path. Similar action might need to be considered for cache.

Is the simplest solution just to backport this to stable too?

I do not think backporting a major upgrade of core system component to stable is something we can do.

In line with this, if you look at https://repology.org/project/fontconfig/versions all the major distros seem to have moved on to at least 2.13.1 (latest stable, past the breaking changes AFAIK), including

ubuntu 20.04 LTS (but also since 19.10)
debian stable 😱
Arch Linux
Gentoo
openSUSE
Fedora 29 (current is 32)
CentOS 7 (current is 8)

I think there's a decent case for suggesting that at least 2.13.1 is backported to stable

@NickHu The point of stable is that it does not change under your feet. Backporting a major upgrade of a core system component would break that. The earliest release any fontconfig update will get into is 20.09 (if someone steps up to maintain it before then).

I think caches are versioned and therefore shouldn't be an issue, but configuration is.
I actually think fonts.conf should be part of the derivations, as they're not meant to be edited anyway. We'd need to patch it so that it reads from /etc/fonts/conf.d or maybe /etc/fonts/version/conf.d instead, but that should be easy enough.
For non-nixos, ideally there would be some way of specifying the system fontconfig version, which would then cause that version of fontconfig to read from the default /etc/fonts/ directory.
I don't actually think there's a problem with a generated fonts.conf as long as it's contained within the derivation.
Issues may arise when font configurations require a certain version of fontconfig, and might require dumping multiple symlinks into etc for every /version, which doesn't seem very clean. Ideally those fonts would install only the config files for the current version of fontconfig, and only install the older config versions if those versions of fontconfig are installed, but that seems rather difficult to do in practice.
Was there any work done to upstream versioned configs?

@sauyon

I think caches are versioned and therefore shouldn't be an issue, but configuration is.

Hmm, it indeed looks like cache is versioned:

https://gitlab.freedesktop.org/fontconfig/fontconfig/blob/68d021b634d1bc53cf3ec1bd28a306500c68c23a/fc-cache/fc-cache.sgml#L195-205
https://gitlab.freedesktop.org/fontconfig/fontconfig/blob/68d021b634d1bc53cf3ec1bd28a306500c68c23a/fontconfig/fontconfig.h#L60-70

I actually think fonts.conf should be part of the derivations, as they're not meant to be edited anyway. We'd need to patch it so that it reads from /etc/fonts/conf.d or maybe /etc/fonts/version/conf.d instead, but that should be easy enough.

Sounds reasonable, we already configure fontconfig 2.12 to have fonts.conf pointing to /etc/fonts/${version}/conf.d so that would ensure the loaded configuration is always correct if it exists. Not sure what happens if the configuration does not exist there, maybe we will also need to add <include>${fontconfig}/etc/fonts/conf.d</include> to the fonts.conf but I am not sure if it is possible to only load it when the first <include> does not exist as to avoid clashes.

For non-nixos, ideally there would be some way of specifying the system fontconfig version, which would then cause that version of fontconfig to read from the default /etc/fonts/ directory.

I think ignoring the system config if there is not a /etc/fonts/${version}/conf.d should be sufficient. Especially if we fall back to the upstream config from the package.

I don't actually think there's a problem with a generated fonts.conf as long as it's contained within the derivation.

We only copy it to the versioned directory so that is not issue either way. The issue is when non-derivation fonts.conf is picked up, which can only happen thanks to our patch:
https://github.com/NixOS/nixpkgs/blob/24f9bc0243ebf873cc0a34790dd0dd265d23f44f/pkgs/development/libraries/fontconfig/config-compat.patch#L25
I believe we chose that fallback to avoid loading random fonts.conf on FONTCONFIG_PATH and solution to all our problems should be as simple as removing the hunk from the patch and passing -DFONTCONFIG_FILE=${placeholder "out"}/etc/fonts/fonts.conf" to compile out:
https://gitlab.freedesktop.org/fontconfig/fontconfig/-/blob/68d021b634d1bc53cf3ec1bd28a306500c68c23a/src/fccfg.c#L2139-2141
Though, for robustness, we might want to replace it in the patch with NIX_FONTCONFIG_FILE constant so that it breaks loudly on change.

Issues may arise when font configurations require a certain version of fontconfig, and might require dumping multiple symlinks into etc for every /version, which doesn't seem very clean. Ideally those fonts would install only the config files for the current version of fontconfig, and only install the older config versions if those versions of fontconfig are installed, but that seems rather difficult to do in practice.

The solution I propose here would mean the module only installs the latest conf.d and have other versions load the upstream defaults from the package.
But as I mentioned, I am not sure how will adding <include>${fontconfig}/etc/fonts/conf.d</include> behave for the latest package. If it really clashes as I suspect, we might need to patch fontconfig to load only one of the directories (adding <includefirst> element).

Was there any work done to upstream versioned configs?

No idea.

Looking at the module, we actually copy all files in the conf.d directory so there would be no conflicts:
https://github.com/NixOS/nixpkgs/blob/53eb2155e76eececd94a4a2f0dad03d1ad80c859/nixos/modules/config/fonts/fontconfig.nix#L217-L218
We only need to fix the routing of local.conf in the package itself:
https://github.com/NixOS/nixpkgs/blob/53eb2155e76eececd94a4a2f0dad03d1ad80c859/nixos/modules/config/fonts/fontconfig.nix#L220-L225
The only problematic part is the optional removal of the user config (hmm, so this is why it is not listed in fonts.conf):
https://github.com/NixOS/nixpkgs/blob/53eb2155e76eececd94a4a2f0dad03d1ad80c859/nixos/modules/config/fonts/fontconfig.nix#L236-L240
We would need to remove it in the package (or move it to $out/etc/fonts/conf.defaults) and import it in the module when needed. Of course this would disable user config on non-NixOS but we might say it is a good thing.

I pushed the proposed changes but it is a late hour here so if you could test this or double check my logic, it would be appreciated.
cc @vcunat

The only problematic part is the optional removal of the user config (hmm, so this is why it is not listed in fonts.conf):

This explains a lot of my font configuration issues, hah.

We would need to remove it in the package (or move it to $out/etc/fonts/conf.defaults) and import it in the module when needed. Of course this would disable user config on non-NixOS but we might say it is a good thing.

Perhaps versioned user configuration should also be a thing? Though I'm sure this won't be an issue for the vast majority of users.
Just to summarize and make sure I understand: configuration files will now only be searched for in and then ${fontconfig}/etc/fonts/conf.d, which are symlinked into /etc/fonts/${version}, correct? Why do they need to be symlinked anyway?
I don't quite understand why the 2.10 fontconfig cache was ever necessary, and I feel like it shouldn't be necessary any longer; though perhaps the version should be bumped to 2.12? I feel like it would make sense at this point for the package to generate and maintain its own cache instead of doing it in the module.

Perhaps versioned user configuration should also be a thing? Though I'm sure this won't be an issue for the vast majority of users.

We could probablly do that but since we are ignoring user config outside of NixOS, it will probably be fine.

Just to summarize and make sure I understand: configuration files will now only be searched for in and then ${fontconfig}/etc/fonts/conf.d, which are symlinked into /etc/fonts/${version}, correct? Why do they need to be symlinked anyway?

The fonts.conf now includes /etc/fonts/${version}/conf.d (where the NixOS module puts it own overrides) and also ${fontconfig}/etc/fonts/conf.d. IIRC fontconfig cares about the filename, not location so both directories will be processed at the same time.
We are no longer symlinking the conf.d to /etc/fonts/${version} since we include them from fonts.conf.
The main difference that we now always use the same fonts.conf, instead of falling back to unversioned global one that would include unversioned conf.d.

I don't quite understand why the 2.10 fontconfig cache was ever necessary, and I feel like it shouldn't be necessary any longer; though perhaps the version should be bumped to 2.12?

Maybe for Steam? No idea either, I would just scrap the “support” parts from the module but we can do that later so that we do not extend the scope of this PR even more.

I feel like it would make sense at this point for the package to generate and maintain its own cache instead of doing it in the module.

IMHO the module for generating cache is useful since it allows us to rebuild the cache when system configuration changes. I prefer declarative methods like this to imperative hooks.

We could probablly do that but since we are ignoring user config outside of NixOS, it will probably be fine.

This was so that user configurations could be supported on non-NixOS, though I think your point about not extending PR scope is a good reason not to do that here. It's probably only something to do if somebody complains about it anyway.

I tried in Fedora VM and can confirm that FreeCAD no longer shows these warnings
[root@localhost-live liveuser]# /nix/store/91y2lsp822qb8cxprrkfnidm00bwn64c-freecad-0.18.4/bin/FreeCAD
FreeCAD 0.18, Libs: 0.18RUnknown
© Juergen Riegel, Werner Mayer, Yorik van Havre 2001-2019
  #####                 ####  ###   ####
  #                    #      # #   #   #
  #     ##  #### ####  #     #   #  #   #
  ####  # # #  # #  #  #     #####  #   #
  #     #   #### ####  #    #     # #   #
  #     #   #    #     #    #     # #   #  ##  ##  ##
  #     #   #### ####   ### #     # ####   ##  ##  ##

Fontconfig warning: "/etc/fonts/fonts.conf", line 5: unknown element "its:rules"
Fontconfig warning: "/etc/fonts/fonts.conf", line 6: unknown element "its:translateRule"
Fontconfig error: "/etc/fonts/fonts.conf", line 6: invalid attribute 'translate'
Fontconfig error: "/etc/fonts/fonts.conf", line 6: invalid attribute 'selector'
Fontconfig error: "/etc/fonts/fonts.conf", line 7: invalid attribute 'xmlns:its'
Fontconfig error: "/etc/fonts/fonts.conf", line 7: invalid attribute 'version'
Fontconfig warning: "/etc/fonts/fonts.conf", line 9: unknown element "description"
Fontconfig error: Cannot load config file from /etc/fonts/fonts.conf

but it crashes:
[jan@localhost ~]$ /nix/store/qmplqyfkfjma2cm70nfwdgkvn42b67x5-freecad-0.18.4/bin/FreeCAD
FreeCAD 0.18, Libs: 0.18RUnknown
© Juergen Riegel, Werner Mayer, Yorik van Havre 2001-2019
  #####                 ####  ###   ####
  #                    #      # #   #   #
  #     ##  #### ####  #     #   #  #   #
  ####  # # #  # #  #  #     #####  #   #
  #     #   #### ####  #    #     # #   #
  #     #   #    #     #    #     # #   #  ##  ##  ##
  #     #   #### ####   ### #     # ####   ##  ##  ##

(process:2863): Gtk-WARNING **: 23:29:03.105: Locale not supported by C library.
	Using the fallback 'C' locale.
Aborted (core dumped)

Do you have other programs that I could test?

I confirmed it on gedit
[jan@localhost ~]$ /nix/store/jcdvq7qgizfn403b3wa8m7i19j2gcx6j-gedit-3.36.2/bin/gedit

(gedit:3327): Gtk-WARNING **: 00:12:37.163: Locale not supported by C library.
    Using the fallback 'C' locale.

compared to
[jan@localhost ~]$ /nix/store/l2n8w3rzvyra37gpynrzy659h5aqvayl-gedit-3.36.2/bin/gedit

(gedit:3160): Gtk-WARNING **: 23:53:37.191: Locale not supported by C library.
    Using the fallback 'C' locale.
Fontconfig warning: "/etc/fonts/fonts.conf", line 5: unknown element "its:rules"
Fontconfig warning: "/etc/fonts/fonts.conf", line 6: unknown element "its:translateRule"
Fontconfig error: "/etc/fonts/fonts.conf", line 6: invalid attribute 'translate'
Fontconfig error: "/etc/fonts/fonts.conf", line 6: invalid attribute 'selector'
Fontconfig error: "/etc/fonts/fonts.conf", line 7: invalid attribute 'xmlns:its'
Fontconfig error: "/etc/fonts/fonts.conf", line 7: invalid attribute 'version'
Fontconfig warning: "/etc/fonts/fonts.conf", line 9: unknown element "description"
Fontconfig error: Cannot load config file from /etc/fonts/fonts.conf

But in both cases the program just runs, regardless of the error. Seems like the issue was misattributed.
Now the only problematic workflow will be running programs compiled against fontconfig 2.12 on NixOS with 2.13, since that will not have 2.12 config dir and will fall back to the legacy fontconfig files. Maybe we should remove them altogether, or add an empty fonts.conf to prevent the fallthrough.

Tested this with Alacritty on Arch as well.

Now the only problematic workflow will be running programs compiled against fontconfig 2.12 on NixOS with 2.13, since that will not have 2.12 config dir and will fall back to the legacy fontconfig files. Maybe we should remove them altogether, or add an empty fonts.conf to prevent the fallthrough.

I think the legacy config should be removed, though another alternative would be to backport these changes for 2.12 in stable.

I think the legacy config should be removed, though another alternative would be to backport these changes for 2.12 in stable.

Backporting complex fixes like this is out of question as mentioned above.
Fortunately, I tried the use case in question by applying the following patch on top of nixos-unstable:
--- a/nixos/modules/config/fonts/fontconfig.nix
+++ b/nixos/modules/config/fonts/fontconfig.nix
@@ -31,7 +31,7 @@ let
   # latest fontconfig version and package
   # version is used for configuration folder name, /etc/fonts/VERSION/
   # note: format differs from supportVersion and can not be used with makeCacheConf
-  latestVersion  = pkgs.fontconfig.configVersion;
+  latestVersion  = "2.13";
   latestPkg      = pkgs.fontconfig;

   # supported version fonts.conf

and it still seems to work with only a warning:
$ gedit
Fontconfig warning: "/etc/fonts/fonts.conf", line 146: blank doesn't take any effect anymore. please remove it from your fonts.conf

I would not worry about that much.

Backporting complex fixes like this is out of question as mentioned above.

I would have advocated for just the config load order change, but it seems like the point is moot in any case, as I agree it doesn't seem like any changes are required.

I've discovered that home-manager has a fontconfig module that uses the user configuration. @rycee, do you have any thoughts on its removal?

@sauyon Yeah HM does have a fontconfig module. It produces a fontconfig cache and a configuration that, if I recall correctly, is necessary for working user installed fonts on both NixOS and non-NixOS.
I install a bunch of fonts through the HM home.packages option (including some that includes additional configuration files in …/etc/fonts/conf.d) and the generated configuration on my system is
<fontconfig>
  <include ignore_missing="yes">/nix/store/303g5xci3s6h1yj9d0n2zp2x9b45fwb2-home-manager-path/etc/fonts/conf.d</include>
  <include ignore_missing="yes">/nix/store/303g5xci3s6h1yj9d0n2zp2x9b45fwb2-home-manager-path/etc/fonts/fonts.conf</include>

  <dir>/nix/store/303g5xci3s6h1yj9d0n2zp2x9b45fwb2-home-manager-path/lib/X11/fonts</dir>
  <dir>/nix/store/303g5xci3s6h1yj9d0n2zp2x9b45fwb2-home-manager-path/share/fonts</dir>
  <dir>/home/rycee/.nix-profile/lib/X11/fonts</dir>
  <dir>/home/rycee/.nix-profile/share/fonts</dir>

  <cachedir>/nix/store/303g5xci3s6h1yj9d0n2zp2x9b45fwb2-home-manager-path/lib/fontconfig/cache</cachedir>
</fontconfig>

If fontconfig 2.13 makes this configuration unnecessary then I'd be very happy. I tried to search for some kind of description of changes in fontconfig 2.13 but only found the commit logs, which is not very helpful. Does anybody happen to have a link to a higher level changelog?

@rycee this is not a change in fontconfig but in our packaging. With 9396f5b8d6f276a2a93b31139598b692434cb656, we explicitly need to link ${fontconfig.out}/etc/fonts/conf.d.bak/50-user.conf to /etc/fonts/2.13/conf.d/50-user.conf to be able to use user configuration.
The main change is that fontconfig will now only use configuration it is sure it is compatible with (or at least user needs to explicitly add it) and fall back to upstream default.
Of course if home-manager relies on 50-user.conf that will not work. Either home-manager will need to set FONTCONFIG_FILE environment variable to point to its own fonts.conf (which side steps the system one, potentially without 50-user.conf) or we need to choose a different approach (not sure what else would work except for giving up and hoping that it does not break too badly – when fontconfig cannot process the configs, it should still fall back to DejaVu).

@jtojnar Ah, right. If the concern is that user provided fontconfig config files will be incompatible between versions then would it be possible to have some 50-versioned-user.conf that is something like
<fontconfig>
  <include ignore_missing="yes" prefix="xdg">fontconfig/2.13/conf.d</include>
  <include ignore_missing="yes" prefix="xdg">fontconfig/2.13/fonts.conf</include>
</fontconfig

? I guess it would be a bit non-standard but perhaps it wouldn't break anything? Then HM could easily put the generated config there (as well).

@jtojnar I think there is a regression with the patches in this PR; now system-installed fonts are no longer properly found (they were before).
I've made a Docker image nickhu/nix-fontconfig-update to test (it's currently uploading, might take a while to be available). It's basically the archlinux base image, with nix (single-user mode) and gtk2fontsel installed three times:

via Arch's repos
via whatever you get from nix-env -iA nixpkgs.gtk2fontsel (i.e. nixpkgs aside from this PR)
via nix-env -f /path/to/this/checkout -iA gtk2fontsel (i.e. nixpkgs at commit 9396f5b8d6f276a2a93b31139598b692434cb656 - your latest commit on this PR @jtojnar)

I've also pushed this build to cachix under nickhu

You can run X11 apps inside Docker via nix-shell -p x11docker.
Running the system's version of gtk2fontsel (x11docker nickhu/nix-fontconfig-update gtk2fontsel) produces no errors and gives:

Running the standard version in nixpkgs (x11docker nickhu/nix-fontconfig-update /nix/store/wxhrj2cmzmag3rc43kr4b66g51q7zxl0-gtk2fontsel-0.1/bin/gtk2fontsel) produces a slew of errors arising from fontconfig version incompatibility, but shows all the fonts:
(process:117): Gtk-WARNING **: 17:18:50.872: Locale not supported by C library.
	Using the fallback 'C' locale.
Fontconfig warning: "/etc/fonts/fonts.conf", line 5: unknown element "its:rules"
Fontconfig warning: "/etc/fonts/fonts.conf", line 6: unknown element "its:translateRule"
Fontconfig error: "/etc/fonts/fonts.conf", line 6: invalid attribute 'translate'
Fontconfig error: "/etc/fonts/fonts.conf", line 6: invalid attribute 'selector'
Fontconfig error: "/etc/fonts/fonts.conf", line 7: invalid attribute 'xmlns:its'
Fontconfig error: "/etc/fonts/fonts.conf", line 7: invalid attribute 'version'
Fontconfig warning: "/etc/fonts/fonts.conf", line 9: unknown element "description"
Fontconfig error: Cannot load config file from /etc/fonts/fonts.conf

Running the version based off this PR (x11docker nickhu/nix-fontconfig-update /nix/store/s1h0hmd5i2bpfv7hglpkdvczpnza5bjk-gtk2fontsel-0.1/bin/gtk2fontsel) produces an error about locales and gives:
(process:117): Gtk-WARNING **: 17:17:29.325: Locale not supported by C library.
	Using the fallback 'C' locale.

@rycee

@jtojnar Ah, right. If the concern is that user provided fontconfig config files will be incompatible between versions then would it be possible to include some 50-versioned-user.conf in the package that contains something like

The issue with that is that there would be no way of disabling those includes, other than by rebuilding fontconfig. The only way forward I see is having home-manager set FONTCONFIG_FILE environment variable pointing to its own fonts.conf.

@NickHu

I think there is a regression with the patches in this PR; now system-installed fonts are no longer properly found (they were before).

Then this works as intended.
We can either have a hermetic package as introduced here, or we need to accept that there might be errors when the host fontconfig configuration does not match the fontconfig used by the package. As always, there is no good answer when trading purity for functionality, and we need to choose the variant that better suits our goals.

The only way forward I see is having home-manager set FONTCONFIG_FILE environment variable pointing to its own fonts.conf.

Is there a reason fontconfig isn't set up to load fonts installed into user profiles? That would solve this issue, right?

@sauyon The fonts.conf file already contains <dir prefix="xdg">fonts</dir>, I guess the main issue is that the system fonts are not read.

But nix doesn't install fonts to ~/.local/share/fonts, does it? I guess home-manager could link fonts into that directory by default, though I think it would make sense to have <dir>~/.nix-profile/share/fonts</dir> included by default for all systems.

@sauyon That is already there, see the full listing:
<dir prefix="xdg">fonts</dir>
<dir>/nix/store/yyazdkikgnmg9rgdvbnrfn8k3q8wgf7v-dejavu-fonts-minimal-2.37</dir>
<dir>~/.nix-profile/lib/X11/fonts</dir>
<dir>~/.nix-profile/share/fonts</dir>
<dir>/nix/var/nix/profiles/default/lib/X11/fonts</dir>
<dir>/nix/var/nix/profiles/default/share/fonts</dir>

@sauyon That is already there, see the full listing:
<dir prefix="xdg">fonts</dir>
<dir>/nix/store/yyazdkikgnmg9rgdvbnrfn8k3q8wgf7v-dejavu-fonts-minimal-2.37</dir>
<dir>~/.nix-profile/lib/X11/fonts</dir>
<dir>~/.nix-profile/share/fonts</dir>
<dir>/nix/var/nix/profiles/default/lib/X11/fonts</dir>
<dir>/nix/var/nix/profiles/default/share/fonts</dir>

Don't forget /etc/profiles/per-user/<username>/share/fonts. Unfortunately, that one is trickier since it cannot, as far as I know, be represented statically in the fontconfig XML format.

@rycee I do not think that one is necessary since on NixOS, the fonts will be managed by a NixOS module.

@sauyon That is already there, see the full listing:

:facepalm: indeed, I'd assumed it wasn't because as it was before fontconfig never used its own fonts.conf (and therefore never included fonts installed by nix). I think I'd even be ok with not including system fonts for purity since now the ones installed via nix should be available.

@jtojnar As far as I know, the NixOS module will only consider fonts that are installed in the system profile, not ones installed to per-user profiles through users.users.<name?>.packages. I might be wrong though, haven't checked that for a while.

Do you have other programs that I could test?

I have a similar problem, and can name a few cases that work badly probably because of this issue:
-Geany-1.36. It boots, but try to write underscores '_'. They show only with bold text, when the font size is 10 or more. Otherwise they are indistinguishable from whitespace.
-bless-0.6.2. The program boots, but every symbol in the main editing fields is replaced with hex value of the symbol the editor tries to show.
My fonts.conf:
<?xml version="1.0"?>
<!DOCTYPE fontconfig SYSTEM "fonts.dtd">
<!-- /etc/fonts/fonts.conf file to configure system font access -->
<fontconfig>
	<its:rules xmlns:its="http://www.w3.org/2005/11/its" version="1.0">
		<its:translateRule translate="no" selector="/fontconfig/*[not(self::description)]"/>
	</its:rules>

	<description>Default configuration file</description>

<!--
	DO NOT EDIT THIS FILE.
	IT WILL BE REPLACED WHEN FONTCONFIG IS UPDATED.
	LOCAL CHANGES BELONG IN 'local.conf'.

	The intent of this standard configuration file is to be adequate for
	most environments.  If you have a reasonably normal environment and
	have found problems with this configuration, they are probably
	things that others will also want fixed.  Please submit any
	problems to the fontconfig bugzilla system located at fontconfig.org

	Note that the normal 'make install' procedure for fontconfig is to
	replace any existing fonts.conf file with the new version.  Place
	any local customizations in local.conf which this file references.

	Keith Packard
-->

<!-- Font directory list -->

	<dir>/usr/share/fonts</dir>
	<dir>/usr/X11R6/lib/X11/fonts</dir> <dir>/opt/kde3/share/fonts</dir> <dir>/usr/local/share/fonts</dir>
	<dir prefix="xdg">fonts</dir>
	<!-- the following element will be removed in the future -->
	<dir>~/.fonts</dir>

<!--
  Accept deprecated 'mono' alias, replacing it with 'monospace'
-->
	<match target="pattern">
		<test qual="any" name="family">
			<string>mono</string>
		</test>
		<edit name="family" mode="assign" binding="same">
			<string>monospace</string>
		</edit>
	</match>

<!--
  Accept alternate 'sans serif' spelling, replacing it with 'sans-serif'
-->
	<match target="pattern">
		<test qual="any" name="family">
			<string>sans serif</string>
		</test>
		<edit name="family" mode="assign" binding="same">
			<string>sans-serif</string>
		</edit>
	</match>

<!--
  Accept deprecated 'sans' alias, replacing it with 'sans-serif'
-->
	<match target="pattern">
		<test qual="any" name="family">
			<string>sans</string>
		</test>
		<edit name="family" mode="assign" binding="same">
			<string>sans-serif</string>
		</edit>
	</match>

<!--
  Load local system customization file
-->
	<include ignore_missing="yes">conf.d</include>

<!-- Font cache directory list -->

	<cachedir>/var/cache/fontconfig</cachedir>
	<cachedir prefix="xdg">fontconfig</cachedir>
	<!-- the following element will be removed in the future -->
	<cachedir>~/.fontconfig</cachedir>

	<config>
<!--
  Rescan configuration every 30 seconds when FcFontSetList is called
 -->
		<rescan>
			<int>30</int>
		</rescan>
	</config>

</fontconfig>

Should we just merge this, and push any home-manager related compatibility tweaks to a further PR?

@rycee

@jtojnar As far as I know, the NixOS module will only consider fonts that are installed in the system profile, not ones installed to per-user profiles through users.users.<name?>.packages. I might be wrong though, haven't checked that for a while.

You are correct. At least this is not a regression so I will not address it here.

@jtojnar Ah, right. If the concern is that user provided fontconfig config files will be incompatible between versions

Yeah, that was the primary concern for this changeset. But here the issue is that in order to keep fonts.fontconfig.includeUserConf NixOS option working, we needed to remove 50-user.conf from the package.
This will not be an issue on NixOS, where the fontconfig module reintroduces it, but on other systems, packages using fontconfig from Nixpkgs will no longer load fonts.conf in the profile.

then would it be possible to include some 50-versioned-user.conf in the package that contains something like
<fontconfig>
  <include ignore_missing="yes" prefix="xdg">fontconfig/2.13/conf.d</include>
  <include ignore_missing="yes" prefix="xdg">fontconfig/2.13/fonts.conf</include>
</fontconfig>

? I guess it would be a bit non-standard but perhaps it wouldn't break anything? Then HM could easily put the generated config there (as well).

We could do that but then we would not be able to disable it using fonts.fontconfig.includeUserConf = false as introduced in 1df1305a8a313120219e70ed1f50618b1047963d. Not sure if that would be much problem since the path would be only used by home-manager.
As an alternative, home-manager could set FONTCONFIG_FILE environment variable to point to its own fonts.conf as suggested above https://github.com/NixOS/nixpkgs/pull/73795#issuecomment-635434294.
At this point I am unsure which one would be preferable.

@NickHu

Should we just merge this, and push any home-manager related compatibility tweaks to a further PR?

That is also an option that stakeholders would have to decide.

At this point home-manager should be in the NixOS org :grin:

@GrahamcOfBorg build fontcontig

Replying to https://github.com/NixOS/nixpkgs/pull/73795#issuecomment-555810414

Does configVersion need to be changed,

Yes, config for 2.13 (change introduced in 2.12.91) is not backwards compatible with 2.12. It is already bumped in the current revision.

and what does this mean for fontconfig_210?

It will continue to use unprefixed config, though we might eventually want to remove that.

Do we need three nested sets of conf directories now, or only the two as we have presently?

Only if we care for people using apps build against Nixpkgs’s fontconfig 2.12 on NixOS system with fontconfig 2.14.
We might not even need that if we backport https://gitlab.freedesktop.org/fontconfig/fontconfig/-/commit/528b17b2837c3b102acd90cc7548d07bacaccb1f, assuming that the warning from unknown description does not cause the breakage. We could then even keep the old configVersion.

(and how do we know/determine the answers to these questions?)

Need to run Arch/Manjaro VM, apply the patch to /etc and then run an app from current nixpkgs-unstable (fontconfig 2.12.6).

What remains to decide if

we want to apply a patch like

--- a/pkgs/development/libraries/fontconfig/make-fonts-conf.xsl
+++ b/pkgs/development/libraries/fontconfig/make-fonts-conf.xsl
@@ -31,6 +31,9 @@
       <!-- versioned system-wide config -->
       <include ignore_missing="yes">/etc/fonts/<xsl:value-of select="$fontconfigConfigVersion" />/conf.d</include>

+      <!-- versioned per-user config -->
+      <include ignore_missing="yes" prefix="xdg">fontconfig/nixfc-<xsl:value-of select="$fontconfigConfigVersion" />-fonts.conf</include>
+
       <!-- upstream config -->
       <include><xsl:value-of select="$fontconfig" />/etc/fonts/conf.d</include>

and have home-manager install its config to $XDG_CONFIG_HOME/fontconfig/nixfc-2.11-fonts.conf, or

have home-manager set FONTCONFIG_FILE environment variable pointing to its config (possibly also have the config include conf.d(s) in /etc), or

do nothing for now and lose home-manager fontconfig support outside of NixOS.

cc @rycee

The patch is incomplete, will also need to backport https://gitlab.freedesktop.org/fontconfig/fontconfig/-/merge_requests/100

This should probably be re-evaluated with 2.10 being insecure - I assume we might want to end up using 2.13 as a default, having $out/etc/fonts/conf.d pointing to the 2.13 config, and maybe keeping config working for 2.11+2.12 in $out/etc/fonts/2.11/conf.d, or did I miss something here?

Can we incorporate the removal of 2.10 (https://github.com/NixOS/nixpkgs/pull/92919) and its config from the module system in here?

We only use configs from 2.10 so that it should be safe.
We cannot use non-versioned config location since it might break apps from the future and from the past.
I suggest we drop non-versioned config on NixOS altogether when we decide to stop supporting apps running fontconfig <2.11.

But I would not do that in this pull request, it is already a big enough change and removing unversioned config dir would make it harder to determine what caused potential issue.

Rebased.

But I would not do that in this pull request, it is already a big enough change and removing unversioned config dir would make it harder to determine what caused potential issue.

Okay. I assume such a change only is in the module system, so we could file the followuup PR to master once this has cycled through staging.

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/thunderbird-crash-after-upgrade-to-20-09/9915/4

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/nixos-builds-a-ton-of-packages-instead-of-fetching-them-in-the-cache/13231/8


Report

Redmi/earth_global/earth:12/SP1A.210812.016/V<IP_ADDRESS>.SCVMIXM:user/release-keys
library: /data/app/~~1OYH_RjiiFMQv287-X4mTg==/me.piebridge.brevent-x-5GqW5AfZ8hP5EkbVHEBw==/base.apk!/lib/arm64-v8a/libbrevent.so
apk: /data/app/~~1OYH_RjiiFMQv287-X4mTg==/me.piebridge.brevent-x-5GqW5AfZ8hP5EkbVHEBw==/base.apk, name: lib/arm64-v8a/libbrevent.so
use exists /data/local/tmp/brevent
checking for server..started, pid: 15172
checking for stable...quited
[command] pm path me.piebridge.brevent
package:/data/app/~~1OYH_RjiiFMQv287-X4mTg==/me.piebridge.brevent-x-5GqW5AfZ8hP5EkbVHEBw==/base.apk
please report bug to https://github.com/brevent/Brevent/issues/new with logs below
Click to show logs
--- crash start ---
[command] logcat -b crash -t '05-20 11:56:00.000' -d
--- crash end ---
--- brevent start ---
[command] logcat -b main -t '05-20 11:56:00.000' -d -s BreventLoader BreventServer
--- brevent end ---

please report bug to https://github.com/brevent/Brevent/issues/new with logs above

fail without detail log, we cannot do anything.


⚠ Bofrost.be has degraded performance

In 73ef0c6, Bofrost.be (https://www.bofrost.be) experienced degraded performance:

HTTP code: 200
Response time: 1698 ms

Resolved: Bofrost.be performance has improved in 6435586 after 6 minutes.


[DPEDE-1341] Improve date picker return data when datetime mode is set

https://ctl.atlassian.net/browse/DPEDE-1341

Hi Emmerson, I am seeing two events being emitted:

one from chi-date and another one from chi-date-picker, both have the same detail but still can cause errors.In case of mode 'datetime', the two payloads ARE different and will cause errors:

I would solve it by doing stopPropagation if the event comes from chi-time. WDYT @josecarlosgonzalezv ?

@ASanchLumen Thanks a lot for your review, I have covered all of your comments in this PR: https://github.com/CenturyLink/Chi/pull/1080.


Plans to allow custom text/functions?

Are there any plans to allow for custom functions or custom text to this module? I'd love to be able to feed data into it.

You can already do that.
New-BGInfo -MonitorIndex 0 {
    # Lets add computer name, but lets use builtin values for that
    New-BGInfoValue -BuiltinValue HostName -Color Red -FontSize 20 -FontFamilyName 'Calibri'
    # Lets add user name, but lets use builtin values for that
    New-BGInfoValue -BuiltinValue FullUserName -Name "User Name X" -Color White
    New-BGInfoValue -BuiltinValue CpuName
    New-BGInfoValue -BuiltinValue CpuLogicalCores
    New-BGInfoValue -BuiltinValue RAMSize
    New-BGInfoValue -BuiltinValue RAMSpeed

    # Lets add Label, but without any values, kind of like section starting
    New-BGInfoLabel -Name "Drives" -Color LemonChiffon -FontSize 16 -FontFamilyName 'Calibri'

    # Lets get all drives and their labels
    foreach ($Disk in (Get-Disk)) {
        $Volumes = $Disk | Get-Partition | Get-Volume
        foreach ($V in $Volumes) {
            New-BGInfoValue -Name "Drive $($V.DriveLetter)" -Value $V.SizeRemaining
        }
    }
} -FilePath $PSScriptRoot\Samples\PrzemyslawKlysAndKulkozaurr.jpg -ConfigurationDirectory $PSScriptRoot\Output -PositionX 100 -PositionY 100 -WallpaperFit Center

The top ones are builtin, you can also use builtin but name it differently. Then you have New-BGInfoLabel and treat it like just section starting. ANd thne you can use New-BGInfoValue with Name and Value paramerters for custom data that you can feed however you like.


fix: fixes a monaco editor display bug for lines below 14-17

Overview

I'm hoping that if this is a draft it won't make web platform review...
Trying to fix DISC-521

PR Checklist

[ ] Related to JIRA ticket: ABC-123
[ ] I have run this code to verify it works
[ ] This PR includes unit tests for the code change

📬Published Alpha Packages<EMAIL_ADDRESS>
📬Published Alpha Packages<EMAIL_ADDRESS>
📬Published Alpha Packages<EMAIL_ADDRESS>
can you link to an integration pr environment so we can test that it fixes the bug?

can you link to an integration pr environment so we can test that it fixes the bug?

PR env being funny but I put https://github.com/codecademy-engineering/Codecademy/pull/29720 on https://smoke.codecademy.com/

looks good! @zippyzow did you have steps to reproduce the scrollbar issue? or did we decide that hiding lines below 14 was the only issue?

https://smoke.codecademy.com/

@borisonr  Can't seem to reproduce on homepage, but you can see them in forums (example).

Line 14+ issue looks good (running client-modules locally). Having cloudflare issues so cant test scrollbars quite yet.

https://smoke.codecademy.com/

@borisonr Can't seem to reproduce on homepage, but you can see them in forums (example).

I see - that's because the container in forums is too small.  I've just added overflow: hidden to the container.  Can you try (or help me try) this newly built version in forums?

📬Published Alpha Packages<EMAIL_ADDRESS>
@Codecademy/gamut do I need additional reviews before adding the "Ship-It" label?


Fix registering and removing commands

Currenly there is a issue with rate limiter when registering new commands. This could be solved using some delay or delegating async tasks with delay.
Beside this issue is problem with removing old command to update them where commands without any change are also removed.
Both thing should be resolved together.

Might also resolve: https://github.com/seker212/Discord_bot/issues/136


projects are not being saved

Dear Mrs, Ms,
I work with Scratch Junior on my notebook with Windows 10 Pro and suddenly last week the projects stopped saving and all saved projects dissappear. I have uninstaled the Scratch Jr and installed again, but the problem is same.
Could you please help me with?
Thank you very much.
Helena Sarapatkova
<EMAIL_ADDRESS>+420 775 238 909

read this https://github.com/jfo8000/ScratchJr-Desktop/issues/120


SE data from space probes

Add recipes for data from space probes from SE.

Do you think you could be more specific? I haven't played SE and if the recipe isn't already included then it likely isn't treated as an actual recipe in-game (like the rocket launch recipe). Is this a single recipe or multiple?

There are (I believe) three recipes: Asteroid belt probe -> Asteroid belt probe data, Star probe -> Star probe data, Interstellar void probe -> Interstellar void probe data. They're all rocket launching recipes, and I've got an idea. The 'rocket-launch-product' property of an item prototype describes what is made when that item is launched in a rocket. You can just make recipes like that: item + 100x Rocket Part -> item.rocket-launch-product[2]x item.rocket-launch-product[1]

I'll likely investigate this in tandem with the "burnt-result" property you brought up in order to add the nuclear reactor recipe cycle recipe. It makes sense to use both of these in place of hard-coded recipes.


Switch PCL project to profile 328

Hi Oystein,
I noticed that the PCL version of the QuantityTypes library is currently targeting profile 136. I thought it would be good to use profile 328 instead, which supports all the targets of 136, plus Windows Phone 8.1 (non-Silverlight).
To facilitate, I have created a pull request for this. If you approve of the change, please feel free to pull it in.
Best regards,
Anders @ Cureos

Sounds good!


Change PropertySource secret backends for ApplicationContext hierarchy

We have a use-case where we are creating multiple microservices via an ApplicationContext hierarchy, similar to this example. For example, we have one parent ApplicationContext "parent" and two child contexts, "child1" and "child2".
Ideally, we'd like to source properties for each ApplicationContext from different context-specific secret backends. Right now, we see that the secrets for the child contexts are always loaded from the same location, specifically for the parent context:
Located property source: CompositePropertySource [name='vault', propertySources=
[LeaseAwareVaultPropertySource {name='secret/parent'},  LeaseAwareVaultPropertySource
{name='secret/application'}]]

I'd like to change this for each child context, so context child1 loads its secrets from secret/child1 and context child2 loads from secret/child2 when starting up.
It seems like this is not possible as VaultBootstrapPropertySourceConfiguration determines the backend secret locations during the bootstrap phase. We've tried implementing a custom VaultConfigurer but even then the values returned from the configurer remain fixed and don't appear to be customizable once they're set. In addition, VaultBootstrapPropertySourceConfiguration constructs its PropertySourceLocatorConfiguration in such a way that it's not possible to customize the implementation of PropertySourceLocatorConfiguration, which could be one possible extension point to enable this.
Is there a workaround to implement this functionality with the current codebase? Or can anything be done to support this use-case?

Spring Cloud Vault is limited to the bootstrap context only. However, you can use the programmatic approach with @VaultPropertySource to declare ApplicationContext-specific property sources. Spring Cloud Vault provides the environment (such as VaultTemplate) so you can use various paths in @VaultPropertySource.
Let me know if that helps.

May I close this ticket or is there anything else I can assist you with?

Thanks Mark. I had to look into this as we're using an outdated version of spring-cloud-vault but I think context-specific VaultPropertySources is what I'm looking for.
My only question would be with with authentication, for example tokens, where I'd expect each ApplicationContext to be assigned their own unique token. Since I can't use spring.cloud.vault.token as its a singular property, I can provide each context with their own ClientAuthentication bean that obtains the token value from a context-specific configuration location, but will that take precedence to be used in VaultTemplate, or will I end up defining multiple VaultTemplate objects and any other associated objects? Ultimately I'm wondering if the beans that are being defined in the bootstrap will work against me.

No, this does not work. Spring Cloud Vault assumes a single authentication method. In your case, I suggest implementing your own configuration support and not using Spring Cloud Vault.


[bug]: img2img (and inpainting) produces only terrible noise on Mac

img2img and inpainting produces totally unrecognizable noise on Mac.
Screenshots
from whosawhatsis:

Inpainting has a lot going on with the masking and everything, but if this also happens on plain img2img then there are fewer things to track down. The difference between img2img and txt2img isn't that big in our current pipeline.
Something is the wrong data type or some get_noise function doesn't do the mps-compatible generate-noise-on-cpu thing or the input image needs an extra contiguous on it or something?

The problem may be in the upstream diffusers library. I'm having similar problems getting img2img to work in my custom script.

it is most likely a pytorch problem, in fact. ref https://github.com/pytorch/pytorch/issues/91617. if so then it's fixed in https://github.com/kulinseth/pytorch/pull/222 but no idea when this will make into pytorch nightly builds let alone release.

(the gist of https://github.com/pytorch/pytorch/issues/91617 is that under certain conditions pytorch subtractions that look perfectly reasonable will return 0s or nonsense, and the "fix" is to .clone() the tensor before doing the operation. ie, a - b = (erroneously) 0, whereas a - b.clone() = correct result.)

confirmed, running against a patched pytorch 2.0 that includes https://github.com/kulinseth/pytorch/pull/222, inpainting and img2img work fine

@damian0815 running into the exact same issue. How did you install the patched pytorch? Tried today but ran into too many hiccups, getting a lot of package inconsistency warnings in conda (granted, I have little idea what I'm doing).

(the gist of pytorch/pytorch#91617 is that under certain conditions pytorch subtractions that look perfectly reasonable will return 0s or nonsense, and the "fix" is to .clone() the tensor before doing the operation. ie, a - b = (erroneously) 0, whereas a - b.clone() = correct result.)

If we need to put that workaround somewhere, I think there are only a handful of places to experiment with, those that are used in img2img but not txt2img.

the noise functions in img2img
non_noised_latents_from_image in the pipeline

Those are the main places that look like they return a tensor that might later be used in some buggy operation. Try sprinkling a few clones() around and see if anything shows up.

@pbakaus afaik the only way at this point is to compile it manually. i documented my process here https://medium.com/@damian0815/how-to-debug-native-mps-operations-in-pytorch-f094b7f4e8f0 but i wouldn't recommend it

i've managed to isolate the problem to, i think,   def non_noised_latents_from_image(self, init_image, *, device, dtype) which works on CPU but returns rubbish on MPS. going to see if i can dig any further into why.

img2img and inpainting both work in #2458


Change 2405 to be a Baseline update

As per

2405 is a Baseline release, not an Update release.

@cblackuk Thank you so much for your contribution. Sincerely appreciate your effort. We did merge this change yesterday through this PR: https://github.com/MicrosoftDocs/azure-stack-docs-pr/pull/15223
As such I will be closing this PR. Please reach out to me if you need any clarifications. #please-close

@cblackuk Thank you so much for your contribution. Sincerely appreciate your effort. We did merge this change yesterday through this PR: https://github.com/MicrosoftDocs/azure-stack-docs-pr/pull/15223
As such I will be closing this PR. Please reach out to me if you need any clarifications. #please-close

Oh, great 😊 my bad, I did look if anything was open for it and could not find it, also the link gives 404 but for as long as it is handled already, all good 😊
Sorry for causing you extra work.


Support for PHP7

This projects heavily relies upon mysql_* functionality from PHP and no other MySQL interface can be chosen, since it is hardcoded. Therefore it does not work with PHP7. The underlying database shouldn't be too much of a deal and should be configurable. At the very least mysqli should be used to stay compatible with PHP7.

Please see several other tickets for work in progress moving to PDO. (Sorry, on my phone so no numbers handy)
Sent from my PDP-11/780

On 15 Oct 2016, at 17:05, Karol Babioch<EMAIL_ADDRESS>wrote:
This projects heavily relies upon mysql_* functionality from PHP and no other MySQL interface can be chosen, since it is hardcoded. Therefore it does not work with PHP7. The underlying database shouldn't be too much of a deal and should be configurable. At the very least mysqli should be used to stay compatible with PHP7.
―
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

#33 #34 #38 #43 #44 #45
And: https://network-weathermap.com/2016/09/04/september-2016-current-issues/
This work is happening in the database-refactor branch, and is almost complete.

@kbabioch, are you using weathermap standalone, with Cacti or something else? I assume Cacti, since otherwise the database stuff wouldn't affect you at all.
I think the current database-refactor commit works in PHP 7.0, aside from a couple of obscure rendering issues. I'd be interested to know if you agree. I'd also be interested in knowing more about your environment, if it doesn't - Cacti version, OS etc...

@howardjones
0.98a works for me on php7.0.11-1  and cacti 0.8.8h. Thanks for the update

@howardjones I'm using it in conjuction with Cacti, but I'm having issues with the actual installation. Installing network-weathermap from an official release works just fine, but I'm having trouble with the database-refactor branch. I'm trying it in the following way:

Remove the weathermap directory in Cacti's plugin directory
Checkout the database-refactor branch from the git repostiry
Invoke make
Put the content from the resulting archive in Cacti's plugin directory
Trying to open Plugin management from within Cacti

At this point the page stays white. The error log states:
AH01071: Got error 'PHP message: PHP Warning:  require_once(/usr/share/webapps/cacti/plugins/weathermap/lib/database.php): failed to open stream: No such file or directory in /usr/share/webapps/cacti/plugins/weathermap/setup.php on line 12\nPHP message: PHP Fatal error:  require_once(): Failed opening required '/usr/share/webapps/cacti/plugins/weathermap/lib/database.php' (include_path='.:') in /usr/share/webapps/cacti/plugins/weathermap/setup.php on line 12\n', referer: xxx

The file /usr/share/webapps/cacti/plugins/weathermap/lib/database.php is not there in either case (official release vs. self-made release), so maybe some include logic is broken?
@jasonkack: How did you test it? Even when installing the plugin from the official release and then trying to replace all files I run into the same problem and cannot open a single Cacti page.

Aaah. OK. The makefile is what I use to make releases - mostly, it compiles the manual from XML sources, and zips up the file that's uploaded as a release. However, it uses the packing.list file to decide what ends up in the zip file. I haven't updated that recently, and there are a lot of new files since 0.98. So instead, for the moment, I'd clone directly into plugins/weathermap. That how I test until very close to release time.

Ok, after having a look at the Makefile it all makes sense now. For the time being I've basically copied all files from the repository into the plugin directory. This works fine. So far I couldn't spot any problems. Integration with Cacti works fine, although I certainly haven't tested everything yet. Going to do so in the next couple of days. I will report back any problems I encounter. Is there any ETA when this will be released officially?

There is one failing test (or group of tests) related to the precedence of background colours for nodes. There are some changes I still want to add in for the new SNMPv2c and SNMPv3 plugins.
There's some documentation (and obviously packing.list!) to update.
I expect a week or two in real time.
Thanks for your feedback!

We have an issue when we set the: "SET rrd_use_poller_output 1"  option in the config because we use boost:
Plugin 'weathermap': Uncaught Error: Call to undefined function mysql_real_escape_string() in /opt/cacti/plugins/weathermap/lib/datasources/WeatherMapDataSource_rrd.php:72
We are on the database-refactor branch but the wmrrd_read_from_poller_output function still seems to use php5 code.
If you need anymore info I would be glad to help.

Thanks for the report! This is a known issue - see #45. Several of the data plugins still have outstanding work to switch to PDO database calls.

All mysql_* functions are replaced now.


[SPARK-45652][SQL][3.5] SPJ: Handle empty input partitions after dynamic filtering

This is a cherry-pick of https://github.com/apache/spark/pull/43531 to branch-3.5, with a few modifications.

What changes were proposed in this pull request?

Handle the case when input partitions become empty after V2 dynamic filtering, when SPJ is enabled.
Why are the changes needed?

Current in the situation when all input partitions are filtered out via dynamic filtering, SPJ doesn't work but instead will panic:
java.util.NoSuchElementException: None.get
	at scala.None$.get(Option.scala:529)
	at scala.None$.get(Option.scala:527)
	at org.apache.spark.sql.execution.datasources.v2.BatchScanExec.filteredPartitions$lzycompute(BatchScanExec.scala:108)
	at org.apache.spark.sql.execution.datasources.v2.BatchScanExec.filteredPartitions(BatchScanExec.scala:65)
	at org.apache.spark.sql.execution.datasources.v2.BatchScanExec.inputRDD$lzycompute(BatchScanExec.scala:136)
	at org.apache.spark.sql.execution.datasources.v2.BatchScanExec.inputRDD(BatchScanExec.scala:135)
	at org.apache.spark.sql.boson.BosonBatchScanExec.inputRDD$lzycompute(BosonBatchScanExec.scala:28)

This is because the groupPartitions method will return None in this scenario. We should handle the case.
Does this PR introduce any user-facing change?

No
How was this patch tested?

Added a test case for this.
Was this patch authored or co-authored using generative AI tooling?

No

Merged to branch-3.5. Thank you, @sunchao and @viirya


Better logging system

As of now, some wrapper classes to make the API nicer for the built in logging module are used, but it's not exactly ideal. The thing that make's this tough is that we need to interop with logging to make hijacks possible.

Basically, something like this:
# internally
from ._logging import new_logger, Level
import logging

service = new_logger("Service")
service.info("Hi!")
server.level(Level.INFO)  # enum
service.hijack(logging.getLogger("uvicorn.error"))  # redirects into this logger

Event listeners also need to be possible in order to allow fancy mode:
@service.event(Level.INFO)
def logged_info(message: str):
    ... # write to file

Or, a complete redirect:
@service.redirect(Level.INFO)
def info(message: str):
    ...  # send to fancy


Empty section when right-clicking multiple instances in scene tree

Godot version
4.0 9d732aa / 3.x
System information
Windows 10 x64
Issue description

This appears, because "Change Type" is not available for instances. One of the separators should be gone in this case.
Steps to reproduce

Instance a couple of scenes
Select them (don't select non-instanced node)
Right-click the selection

Minimal reproduction project
No response

Hey there! I'm new to developing the project and would like to try and tackle this issue. Since it looks like the issue exists in multiple versions, would it be best to fix it in the 3.x version or the 4.0 version?

It should be fixed in 4.0 first. The commit can then be cherry-picked.


Add function to compute lifted edges from rag and set of offsets

This implementation works for me, but I think it could be improved in some points (see TODOs in get_lifted_edges_from_rag_and_offsets.cxx ).
Most importantly, I could not find an easy way to release the gil to allow multiple threads.

@abailoni  what about   py::gil_scoped_release release;

Hey @DerThorsten, my question was related to the fact that usually, you release the gil for a specific scope:
xt::pytensor<uint64_t, 2> outTensor({shapeX, shapeY});
{
    py::gil_scoped_release allowThreads;
    computeLongRangeEdgesFromRagAndOffsets(rag, offsetVector, outTensor, numberOfThreads);
}
return outTensor;

But in this case I know the dimension of the output tensor only after calling the function. This alternative of course would not work:
{
	py::gil_scoped_release allowThreads;
 	out = computeLiftedEdgesFromRagAndOffsets(rag, offsetVector, numberOfThreads);
}
return out;

I can imagine that the solution is very simple, but so far I could not see an obvious way to declare the output tensor out of the scope before to know its final shape.

@abailoni I have fixed the tests on master,could you rebase to it?
I will think about the gil too.


ManagedClusterInfos monitoring

Signed off by: @jnpacker

Add support to monitor ManagedClusterInfos - Instead of ManagedCluster, this saves a RoleBinding resource
Added extend cluster-installer role (not used as Hive controller overwrites this role periodically)
Added flag.set so that verbose level 2 is now available
End to end deployment validated
Add TTL for the curator-job pod

/cc @hanqiuzh @Randy424

/lgtm

/retest


Use plugin structure for Glyphs3

Please use the templates (mostly the binary in the "MacOS" folder) from: https://github.com/schriftgestalt/GlyphsSDK/tree/Glyphs3/Python Templates

Done.


nixci: 0.5.0 -> 1.0.0

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/srid/nixci/releases.
meta.description for nixci is: Define and build CI for Nix projects anywhere
meta.homepage for nixci is: https://github.com/srid/nixci
Updates performed

Rust version update

To inspect upstream changes
Impact
Checks done

built on NixOS
The tests defined in passthru.tests, if any, passed
found 1.0.0 with grep in /nix/store/mnsqmq23w2s65i79d0r2bc3szkh7cnyx-nixci-1.0.0
found 1.0.0 in filename of file in /nix/store/mnsqmq23w2s65i79d0r2bc3szkh7cnyx-nixci-1.0.0

Rebuild report (if merged into master) (click to expand)

2 total rebuild path(s)

2 package rebuild(s)

First fifty rebuilds by attrpath

nixci

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/mnsqmq23w2s65i79d0r2bc3szkh7cnyx-nixci-1.0.0 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A nixci https://github.com/r-ryantm/nixpkgs/archive/6cd32b6758e5eb5e80769cb4dc02eadf6f2bdbbf.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/6cd32b6758e5eb5e80769cb4dc02eadf6f2bdbbf#nixci

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/mnsqmq23w2s65i79d0r2bc3szkh7cnyx-nixci-1.0.0
ls -la /nix/store/mnsqmq23w2s65i79d0r2bc3szkh7cnyx-nixci-1.0.0/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

1 package built:

nixci

Maintainer pings
cc @srid @shivaraj-bh for testing.

closing in favour of https://github.com/NixOS/nixpkgs/pull/329645.


How to train the model on multi-class problem, for example including headline.

Who can post example code ? Help me


Dictionary destructuring

Describe the solution you'd like
Vectors can be destructured but dictionaries cannot. It would be great to be able to do.
Describe alternatives you've considered
Pulling out dictionary values with juxt and destructuring the vector from that.
Suggestion
(let {:a a, :b b} {:a 123 :b 234})
; a = 123
; b = 234

Retrieving the rest and entire dictionary too would be useful:
(let {:a a, & b, _ c} {:a 123 :b 234})
; a = 123
; b = {:b 234}
; c = {:a 123 :b 234}

What would also be lovely is just being able to provide keys that are automatically turned into variables:
(let {:keys [a b c]} {:a 123 :b 234 :c 345})
; a = 123
; b = 234
; c = 345

Actually I'd really love to be able to do what you can do in JavaScript:
(let {x y} {:x 1 :y 2})
; x = 1
; y = 2

I feel the terse and simple nature would be more valuable than being able to rename keys, as a trade-off.


Parsing tables with missing 	 and
 is different than in browsers

Hi, and thank you for this package that will probably allow me to get a proof-of-concept out (and maybe more) much more easily!
I’m having an issue with really bad HTML coming from the backend, where the result when parsed with the library is different than what the browsers do.
Basically it’s a table with missing 	 and
 at the end of each row:

    A1
    B1

    A2
    B2

    A3
    B3

Instead of returning the correct 2×3 table (as if the missing ending tags were there), the result is that the second row is nested into the second cell of the first row, the third row is nested into the second cell of that nested row, etc.
Expected:

Actual:

In my case, I will probably fix the backend since I can. But because your goal seems to be to match the behaviour of browsers, the issue seemed relevant to mention here.

Good find!
At a glance, I suppose the element parser can be patched to special-case more autoclosing rules.
Thinking out loud, since I'm not on a dev machine at the moment:
Since tr and td are in the parser's list of autoclosing tags, they already automatically close when encountering an open tag with their same name, like
a
b.
I think, the minimal fix for this issue is to make open
 autoclose  like how I special-case open  to autoclose :
https://github.com/danneu/html-parser/blob/cfa2acb7e44c6f675b9c30019ae4ec71a4e6dfd4/src/Html/Parser.elm#L525-L540
Maybe this is sufficient:
  if tag == "head" then
      notNode cfg [ tag, "body" ]
+ else if tag == "td" then
+    notNode cfg [ tag, "tr" ]
  else
      ...

Thanks for your time. I'll have a look.

I published this fix as v2.0.3: https://package.elm-lang.org/packages/danneu/html-parser/

Works perfectly, thank you so much!


React Native Debugger is broken

i got error message when start debugger, its work yesterday and didn't nothing change
deltaUrlToBlobUrl.js:24 Uncaught (in promise) TypeError: Cannot read property 'get' of undefined
    at deltaUrlToBlobUrl (deltaUrlToBlobUrl.js:24)
    at getBlobUrl (?platform=android&dev=true&minify=false:230)
    at WebSocket.ws.onmessage (?platform=android&dev=true&minify=false:185)

Update
its work when using <IP_ADDRESS>:8081 instead of localhost:8081
React Native version:
  React Native Environment Info:
    System:
      OS: macOS High Sierra 10.13.6
      CPU: (4) x64 Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz
      Memory: 68.80 MB / 8.00 GB
      Shell: 3.2.57 - /bin/bash
    Binaries:
      Node: 8.11.4 - /usr/local/bin/node
      Yarn: 1.15.2 - /usr/local/bin/yarn
      npm: 5.6.0 - /usr/local/bin/npm
      Watchman: 4.9.0 - /usr/local/bin/watchman
    SDKs:
      iOS SDK:
        Platforms: iOS 12.1, macOS 10.14, tvOS 12.1, watchOS 5.1
      Android SDK:
        API Levels: 22, 23, 24, 25, 26, 27, 28
        Build Tools: 23.0.1, 23.0.2, 24.0.1, 25.0.0, 25.0.1, 25.0.2, 26.0.1, 26.0.2, 26.0.3, 27.0.2, 27.0.3, 28.0.1, 28.0.3
        System Images: android-24 | Intel x86 Atom_64
    IDEs:
      Android Studio: 3.1 AI-173.4819257
      Xcode: 10.1/10B61 - /usr/bin/xcodebuild
    npmPackages:
      react: 16.6.3 => 16.6.3
      react-native: 0.59.9 => 0.59.9

Steps To Reproduce
1.Start Remote JS Debugging
Describe what you expected to happen:
just want to debug

I'm leaning towards closing this for two reasons: 1) insufficient repro steps (apparently it stopped working without any changes to the product code), and 2) you already worked around this by using the localhost ip address directly.


ci: Use cargo-nextest from within a pixi task

Also remove test_environment_json and test_old_lock_install since neither work when run from within pixi

I dont think you should remove the rust cache, that is not cached in a pixi environment right?

I dont think you should remove the rust cache, that is not cached in a pixi environment right?

At least the target directory is part of .pixi and therefore cached. Does the rust cache action cache more than that?


Release/v0.15.0 bookkeeping

Codecov Report

Merging #72 into master will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master      #72   +/-   ##
=========================================
  Coverage     99.34%   99.34%
  Complexity      657      657
=========================================
  Files            49       49
  Lines          1538     1538
  Branches        164      164
=========================================
  Hits           1528     1528
  Misses            4        4
  Partials          6        6

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 503a318...e67a4e1. Read the comment docs.


ModuleNotFoundError: No module named 'axes.handlers'

Hi, I just updated axes from version 4.5.4 to 5.0.12, make the change that was in the documentation for the use of is_locked, but this error is giving me ModuleNotFoundError: No module named 'axes.handlers',
This is my settings and views.
settings.py
AXES_COOLOFF_TIME = 24
AXES_FAILURE_LIMIT = 10
AXES_HANDLER = 'axes.handlers.database.AxesDatabaseHandler'
AXES_LOCK_OUT_BY_COMBINATION_USER_AND_IP = True
AXES_LOCKOUT_TEMPLATE = '429.html'
AXES_META_PRECEDENCE_ORDER = ['HTTP_X_FORWARDED_FOR', 'REMOTE_ADDR']
AXES_RESET_ON_SUCCESS = True
AXES_USE_USER_AGENT = True

views.py

def post(self, request):
        form = self.get_form()
        if not form.is_valid():
            credentials = {'username': form.cleaned_data.get('username')}
            if AxesProxyHandler.is_locked(request=request, credentials=credentials):
                tasks_access_block_send_email(email=form.cleaned_data.get('username'))

            user_login_failed.send(sender=User, request=request, credentials=credentials)

            return self.form_invalid(form)

        user = authenticate(
            request=request,
            username=form.cleaned_data.get('username'),
            password=form.cleaned_data.get('password'),
        )
        if user is not None:
            login(request, user)
            user_logged_in.send(sender=User, request=request, user=user)

            return redirect(self.get_success_url())

        credentials = {'username': form.cleaned_data.get('username')}
        user_login_failed.send(sender=User, request=request, credentials=credentials)
        return self.form_invalid(form)

I would like to know how I could solve it, could you please help me, thanks

Ensure to import from axes.handlers.proxy import AxesProxyHandler

Your code looks good to me. Can you post your imports?
Do you think the docs could be improved for the custom authentication implementations? If you have any improvement ideas, please feel free to open a PR with changes to the rST files, they are auto built by the RTD website on commit to master so you can just update the source on this repo and the changes go live once upstreamed :)


Document new configuration to enable ssl for ingesting via mqtt.

Codecov Report

:exclamation: No coverage uploaded for pull request base (ssl_mqtt@9e0146e). Click here to learn what that means.
The diff coverage is n/a.

@@             Coverage Diff             @@
##             ssl_mqtt    #6272   +/-   ##
===========================================
  Coverage            ?   83.38%
  Complexity          ?    12838
===========================================
  Files               ?     1478
  Lines               ?    48432
  Branches            ?     5919
===========================================
  Hits                ?    40384
  Misses              ?     5305
  Partials            ?     2743

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9e0146e...aad725c. Read the comment docs.


Refactor to use the functions in component-helpers to calculate pod resource

What would you like to be added:
Refactor the code to use library functions in component-helpers:
https://github.com/kubernetes/kubectl/blob/master/pkg/util/resource/resource.go#L40
https://github.com/kubernetes/kubectl/blob/master/pkg/util/resource/resource.go#L96
Why is this needed:
PodRequests and PodLimits method exists in staging/src/k8s.io/component-helpers/resource/helpers.go. Refactor the code to avoid duplicating podRequests and podLimits method.

When I checked in the referenced PR, there are plenty of changes and it is really hard to confirm that this won't create any regressions. In my opinion, we can leave it as is. However, maintainers can still drop comments on the PR but there is no need to have a separate issue for this;
/close


Add spark csv and json tests

What is this PR for?
As it was discussed here: https://github.com/apache/zeppelin/pull/3290#issuecomment-457206170
These two tests will make sure Apache Spark in Zeppelin can read both csv and json files.
What type of PR is it?
[Improvement]
Todos

[ ] - Task

What is the Jira issue?

Open an issue on Jira https://issues.apache.org/jira/browse/ZEPPELIN/
Put link here, and add [ZEPPELIN-Jira number] in PR title, eg. [ZEPPELIN-533]
https://github.com/apache/zeppelin/pull/3290

How should this be tested?

First time? Setup Travis CI as described on https://zeppelin.apache.org/contribution/contributions.html#continuous-integration
Strongly recommended: add automated unit tests for any new or changed behavior
Outline any manual steps to test the PR here.

Screenshots (if appropriate)
Questions:

Does the licenses files need update? Yes
Is there breaking changes for older versions? No
Does this needs documentation? No

I forgot th JSON test! I will commit to this pull request, in fact there should be 2 tests one for csv and one for json.

@zjffdu I copied the wrong test by mistake for JSON, now there are two tests one for CSV and one for JSON exactly as they were in branch-0.8.
Many thanks :)

LGTM, will merge if no more comment


Add T-SQL delete with join support

Add initial support
Add feature test / regression test - tests passing locally & on CI
Partially Fixes #16

Initial support, please let me know if you have any input. I will review again later to see if I want to clean up more before requesting to merge

@ceastwood interesting!, I will merge it to a separate branch, and I will try to support the other compilers.
Do you know how to merge this to a new  branch :) ?

@ahmad-moussawi  The branch is in this repo-- tsql-delete-with-joins. I had made it a PR so we could have discussion & tracking on the feature.
I want to finish up the MySql and test + throwing an exception where unsupported by the compiler.

@ahmad-moussawi
I'm not sure of the performance, but with PostgreSQL-- there can be multiple using's for a delete, also possibly an inner self-join with the actual joins attached to that reference.
CREATE TABLE test (
  testId INT
);
CREATE TABLE test2 (
  test2Id INT,
  testId INT
);
CREATE TABLE test3 (
  test3Id INT,
  test2Id INT,
  testId INT
);

INSERT INTO test (testId) VALUES (1);
INSERT INTO test (testId) VALUES (2);
INSERT INTO test (testId) VALUES (3);
INSERT INTO test2 (test2id, testId) VALUES (100,1);
INSERT INTO test2 (test2id, testId) VALUES (200,2);
INSERT INTO test3 (test3id, test2Id, testId) VALUES (1000, 100, 1);
INSERT INTO test3 (test3id, test2Id, testId) VALUES (2000, 200, 2);

-- multiple usings
--  : should return only 3 on query side
--DELETE FROM test tst
--USING test2 tst2,
--	test3 tst3
--WHERE tst.testId = tst2.testId AND tst3.test2id = tst2.test2id

-- inner self-join
--  : should return only 3 on query side
--DELETE FROM test tstMain
--USING test as tst
--INNER JOIN test2 tst2 on tst.testId = tst2.testId
--INNER JOIN test3 tst3 on tst2.test2Id = tst3.test2Id
--WHERE tstMain.testId = tst.testId

-- left self-join
--  : should return only 3 on query side
DELETE FROM test tstMain
USING test as tst
LEFT JOIN test2 tst2 on tst.testId = tst2.testId
LEFT JOIN test3 tst3 on tst2.test2Id = tst3.test2Id
WHERE tstMain.testId = tst.testId AND tst3.test3Id IS NOT NULL

-- query side
SELECT * FROM test;`

I can rename the branch I've made since it isn't T-Sql anymore and continue working on the feature there.


[Question] How to monitor Thanos for "no space left on device" or "out of memory" can we add such to /metrics

How to monitor Thanos for "no space left on device" or "out of memory" can we add such to /metrics

So memory and CPU usage is part of Golang metrics, but I would focus to monitor "host" that Thanos is running on. E.g docker container (e.g cAdvisor metrics for cpu, mem, disk, network limits and usage), or Node metrics if it's bare node.

"no space left on device" is a node problem so I suggest you use https://github.com/prometheus/node_exporter to get metrics for it


Fixes issue 3363: Normalize casing in UI text in English

Fixes #{3363}
Changes proposed in this pull request:
Make some modification based on pvcodes's commit(ecc63c8)
Normalize some phrases pvcode missed with lowercase. eg: line 75 "Select Files to import" to "Select files to import"
Capitalize the SQL statement. eg: line 293 "Include Drop statement" to "Include DROP statement"

Closing since the corresponding issue was fixed by another PR.


Potential Optimization: duk_remove(-2) -> duk_replace(-2)

While optimizing my internal Duktape emulation layer in miniSphere, it occurred to me that the following two operations are equivalent:
duk_remove(-2);
duk_replace(-2);

Both have the same stack signature:
remove(-2)  - [ ... value1 value2 ] -> [ ... value2 ]
replace(-2) - [ ... value1 value2 ] -> [ ... value2 ]

The difference: One requires a memmove and the other does not.  As duk_remove(-2) is a common enough idiom to have an internal helper (duk_remove_m2()), it might make sense to use duk_replace for this case instead and see if there's a performance impact.

While it's true that the memmove in question for duk_remove(-2) is only the size of a single value stack element (assumedly a pointer) and likely quite fast--especially if the compiler uses an intrinsic--duk_replace as a whole can probably be optimized by the compiler on some platforms into a swap instruction or two, while the same almost surely isn't the case for duk_remove's memmove since the amount of data moved will vary depending on where the value to be removed is.
Benchmark results are of course needed before making any decision; this is currently just a curiosity of mine. :)

Outside of performance, I've recently found in my own code that jsal_replace() communicates the actual intent of the code better than jsal_remove() in most of the cases where I was using the latter.  That is, I put one thing on the stack, derive the value you need from that, and then replace the original intermediate value with the value you actually wanted.


Wait for selector

Hopefully this resolves https://github.com/chanzuckerberg/axe-storybook-testing/issues/34. It adds the waitForSelector parameter for stories. That way stories can wait for specific selectors to be visible on the page, if necessary.

@ahuth do you know if this works in the inverse? For example, yesterday you helped me debug my color contrast issue, and it was a temporary notification on one of our stories. That notification disappears after four seconds, so:
waitForSelector: "not:(.notification-box)"

would probably work for us. Might be worth adding a test?

@kevinmpowell thanks for the idea. I tested out waiting for the not pseudo selector, and it didn't work. We'll need to come up with another way to wait for something to not be present.
Simplest workaround I can think of in the meantime is to separate the component from the temporary notification, and only render the component itself (without notification) in storybook.

For an actual solution we could maybe add some sort of waitForSelectorToNotBePresent option.


RobotFeedback Class Development

Est 5-10d
Full SensorFeedback class development
Please work with Rhys when you get to this point
Refer to EncoderImplementation.ino (https://drive.google.com/file/d/1P1TEhiQYkMvv-Y2d6aORED3SsEXtWyE-/view?usp=share_link) note: this was written for a different encoder from what we will use for this implementation.
We want to avoid the use of interrupts but if it is absolutely necessary, this code can be implemented on the second esp core, the issue with interrupts is: the rest of the program may not be called at all.
Function for reading the encoder state once every loop, store these values locally as counts, etc…
Function to calculate the counts/mm
Function to output the counts/mm for each encoder
The object needs to be static, get sensor input needs to be called inside main.cpp, if this is not possible calling it inside of Drive::Update() will suffice
Notes for development: remember that there are going to be more wheel encoders added later, so a true oop class could be helpful here, (one class object for each encoder). Refer to MotorControl for more information on implementation

 Bot Feedback Software development


Fix typo in Readme.md

:tada: This PR is included in version 1.8.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Uptime color

⚠ Please verify that this feature request has NOT been suggested before.

[X] I checked and didn't find similar feature request

🏷 Feature Request Type
UI Feature
🔖 Feature description

Actually, uptime is always green regardless the value. I think it's a problem, Uptime of 1% isn't normal, it's must be displayed with red color. Yellow under 100%, orange under 99% for exemple. Because when a open my dashboard, I can't see if there is an uptime problem.
✔ Solution
Add uptime color according to uptime value. For exemple :

❓ Alternatives
No response
📝 Additional Context
No response

The problem is that this is the current state. If the dot was red, it could indicate at first glance that the service was not working.
In fact, the current solution seems logical to me and this would confuse a lot of people.

In fact, it would be good to adjust the design of monitors on the status page so that the icon showing the current status and uptime in percentages separately.
Like BetterUptime:

And as you can see, they don't have colored uptime :)


Opentofu support

Description
Add support to enable OpenTofu usage in tests.
Release required?

[ ] No release
[x] Patch release (x.x.X)
[ ] Minor release (x.X.x)
[ ] Major release (X.x.x)

Release notes content

add support to enable OpenTofu in testOptions

Run the pipeline
If the CI pipeline doesn't run when you create the PR, the PR requires a user with GitHub collaborators access to run the pipeline.
Run the CI pipeline when the PR is ready for review and you expect tests to pass. Add a comment to the PR with the following text:
/run pipeline

Checklist for reviewers

[ ] If relevant, a test for the change is included or updated with this PR.
[ ] If relevant, documentation for the change is included or updated with this PR.

For mergers

Use a conventional commit message to set the release level. Follow the guidelines.
Include information that users need to know about the PR in the commit message. The commit message becomes part of the GitHub release notes.
Use the Squash and merge option.

/run pipeline

/run pipeline

/run pipeline

/run pipeline

/run pipeline

:tada: This PR is included in version 1.30.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Export/Import settings with same Folder as BOM-Location

When Exporting/Importing JSON Files with the Settings, i would be convinient, if this would start in the same directory as the current location of the BOM File.
We usually make separate Setting-JSONs for every single Batch or Board to be produced in order to keep Track of their status and therefore store them next ot the according BOM-html.
Its a bit clumsy to navigate through the Folder-Structure everytime and would save time, if it already suggests to store it in mentioned location.
Even better: Allow to name the resulting JSON with a "Save as" dialouge!

This is not technically feasible, there is no javascript API to do this. It will be possible when https://wicg.github.io/file-system-access/ is accepted and widely supported but it can be years until that happens.
Until then the only thing ibom can do is generate a data blob and let browser "download" it. Whether you get a "Save as" dialog depends on your browsers download settings. For example in Chrome you can choose to have it show a dialog for every file or always save to a specific directory without asking.


Pre process final

This branch includes a preprocess folder of code for calculation of preliminary resources given input data sets (which are not included --- I plan to load a 'data' folder up on CloudStor containing these GHS and OSM resources which can be downloaded to merge with the repo process/data folder.
The caveat is, this is used for for calculation of basic resources as previously delivered --ie. OSM destinations (and POS entry points, for any and large).  However, it doesn't yet include additional code to incorporate custom destinations (Maiduguri) or regular PT data; and I haven't yet tested out that all that will be required for evaluating access to POS and regular PT data has been done.  For example, the OSM type destinations were pre-associated with their nearest node locations -- I can't recall if that information is made use of in how we ended up using the code; if the analysis assumes these fields are present for destinations, then we'll have to either add them for POS and PT, or combine those destinations altogether with the OSM ones.
At a basic level though, the resources are prepared enough to make a start on analyses I believe, and I think the analysis process code should allow us to post hoc go back and analyse the other destinations as we add them.
tl/dr: I expect updates will be made to the pre-processing code, as required!

@carlhiggs I see there's also a pre_process branch... shall we delete that one? Is it outdated?

Hi @gboeing , yup we shall delete it --- when i first created it (and related to the other issue i posted earlier) i overlooked that I wasn't on the master branch.  I attempted to merge the master into it post hoc, but it didn't seem to resolve the conflicts so it made more sense to just create a new branch and do things properly.  I'm in the process of cleaning up folders on my computer (as the pre-processing work I did the past few days was actually on that branch), and once I've sorted things out so I know i won't lose data if i switch branches I'll delete the other one.

At a basic level though, the resources are prepared enough to make a start on analyses I believe, and I think the analysis process code should allow us to post hoc go back and analyse the other destinations as we add them.
tl/dr: I expect updates will be made to the pre-processing code, as required!

Ok, I'll hold off on merging this with master until I get the official word from you.

As per discussion on e-mail, I'm working on updating the docker image definition and making sure the validation report documentation works as intended when run back in January (so merge is still pending while I run through the code)  @gboeing

Hi @gboeing  @shiqin-liu @nicholas-david @duelran ,
I think this repository is good to be pulled in now!
I've updated the pre-process code so it now is able to generate the preliminary data validation report --- I'll attach Maiduguri's as a proof of concept here.  It requires all cities defined in the configuration file to have been processed to successfully run, and if attempt is made to run it without doing this the user will be prompted (because the key all cities summary table won't exist).  The summary tables used to be manually constructed from CSV, but now are built using some customised latex generated from Pandas --- I'll attach here in case of use for our other paper(s) (e.g. it contains index rows for continent, country and city --- so the data columns could be repurposed for other uses).  The python code for that is in the _all_cities_summary_tex_tables_for_report.py script.
global_liveability_maiduguri.pdf
zipped tex file (as it wouldn't allow upload of the tex file itself -- I didn't include this tex file in the repository as its subject to change pending configuration and data)
cities_data.zip
cheers,
Carl

Thanks @gboeing I just amended the docker image file as suggested (main RUN all in one --- the run calls to python and ipython are still separate, but I don't expect that matters as was that way in original, I think).  I gave it a test run and re-built the pdf report successfully.  I haven't re-run all the other code since previous runs, but can see no reason why this shouldn't also work.  Perhaps before final release we can do a full run through all code using the master branch.  ----- So I believe this pull request should be good to merge if all agree!

Perhaps before final release we can do a full run through all code using the master branch.

Yep, I agree!


[BUG] Failure in creating Azure node/driver and AKS using keys from China region

Rancher Server Setup

Rancher version: v2.7.9
Installation option (Docker install/Helm Chart): Docker

Describe the bug
Use china Azure Cloud Credential

Create China Azure Cloud Credential Error

The 'env' parameter was not passed to the backend

Create rke2 azure

Js error

The 'aksCheckCredentials' cannot recognize Azure China's Cloud Credential.

"failed to get locations: subscription.SubscriptionsClient#ListLocations: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code=\"SubscriptionNotFound\" Message=\"The subscription 'ebca86b2-b2ae-4eb2-a69c-9ee96034073e' could not be found.\""

Expected Result
Rancher should support other regions within Azure.

Tracking Info
https://github.com/rancher/rancher/issues/43608 is the core issue
https://github.com/rancher/rancher/pull/43605 contains the fix backend side
https://github.com/rancher/dashboard/pull/10071 contains fix frontend side

Tracking Info #43608 is the core issue #43605 contains the fix backend side rancher/dashboard#10071 contains fix frontend side

Thanks for your response. Let me rephrase the 2 issues encountered by the front end:

When creating the node-driver rke2 using a China-based key, the default region for China is not DEFAULT_REGION = 'westus'. Our front end should load the region based on the passed key to fill in the default area. If the above process uses a China-based key and loads data using westus, it results in the aksVMSizesV2 interface returning null. Consequently, this triggers an error in this.vmSizes.filter.

The environment itself is hardcoded during key creation and should be disabled when creating rke2.


Segmentation fault with pytorch 1.0

I'm facing a segfault when using torch==1.0 and onnx-coreml from the current master (commit https://github.com/onnx/onnx-coreml/tree/da33f4f0efde2245925b3a2444973786a3a91e0d) on Ubuntu 16.04 servers. Conversion is successful in the same environment with torch==0.4.1.
Script to reproduce:
import onnx
import onnx_coreml

import torch
from torch import nn

class Dummy(nn.Module):

    def __init__(self):
        super().__init__()

    def forward(self, inp):
        a, b = inp

        result = b + nn.functional.interpolate(a, scale_factor=2, mode="nearest")
        return result

def main(device='cuda:0'):
    model = Dummy()
    model = model.to(device)
    dummy_input = [torch.randn(1, 128, 32, 32).to(device), torch.randn(1, 128, 64, 64).to(device)]

    onnx_path = '/tmp/dummy.onnx'
    coreml_path = '/tmp/dummy.mlmodel'
    torch.onnx.export(model,
                      dummy_input,
                      f=onnx_path,
                      input_names=['input'],
                      output_names=['output'],
                      verbose=False,
                      export_params=True,
                      )
    onnx_model = onnx.load(onnx_path)
    coreml_model = onnx_coreml.convert(onnx_model)
    coreml_model.save(coreml_path)

if __name__ == '__main__':
    main()

I've investigated this a bit, and there still might be an issue with onnx-coreml.
PyTorch 1.0 uses a new version of Upsample op from ONNX.
Changes were introduced by this commit: https://github.com/pytorch/pytorch/commit/02d3787a192b1d51e572fc82f75284b523e4921d
I was able to get past the segfault on a Mac OS Mojave (without CUDA) and actually compile the model following these steps:

Install onnx from source:

brew install protobuf --without-python@2
brew install cmake
git clone  https://github.com/onnx/onnx
cd onnx
python setup.py install

Replace device='cuda:0' with cpu in before-mentioned script.
Manually patch onnx_coreml/_operators.py:

diff --git a/onnx_coreml/_operators.py b/onnx_coreml/_operators.py
index caf2938..e112183 100644
--- a/onnx_coreml/_operators.py
+++ b/onnx_coreml/_operators.py
@@ -1510,8 +1510,9 @@ def _convert_upsample(builder, node, graph, err):  # type: (NeuralNetworkBuilder
         height_scale = int(scales[2])
         width_scale = int(scales[3])
     else:
-        height_scale = int(node.attrs.get('height_scale', 1))
-        width_scale = int(node.attrs.get('width_scale', 1))
+        height_scale = int(node.input_tensors[node.inputs[-1]][2])
+        width_scale = int(node.input_tensors[node.inputs[-1]][3])
     mode_convert = {
         "nearest": "NN",
         "bilinear": "BILINEAR",

@aseemw , perhaps something like this can be done in onnx-coreml to fix the issue?

I see. Thanks for investigating and the solution!
For now onnx-coreml has only been tested against onnx 1.3.0, which is the latest released version on PyPI.
This can be done once onnx support is upgraded in onnx-coreml. I'll keep the issue open as an enhancement.

I have the same problem with a PyTorch 1.0 exported model. I raised an issue here. Not sure why this current issue is closed, as following the pip install instructions leads directly to this problem.
https://github.com/onnx/onnx-coreml/issues/369

@marvelousNinja I tried your fix, but the current onnx_coreml/_operators.py function below looks different than what you have in your comment.
Also, I set a debug breakpoint at the start of _operators.py before trying the conversion, but _operators.py never appears to be called during the conversion process as the breakpoint was never triggered.
https://github.com/onnx/onnx-coreml/blob/da33f4f0efde2245925b3a2444973786a3a91e0d/onnx_coreml/_operators.py#L1505-L1518

@Dimbl4 yes your solution works well, you should submit a PR. My solution was to install onnx-coreml using pip install -U onnx-coreml as usual. Then:

Place the https://github.com/onnx/onnx-coreml/tree/master/onnx_coreml directory in the local project directory, so import onnx_coreml statements would direct to this local folder.
Use your code to modify the _convert_upsample() function in https://github.com/onnx/onnx-coreml/blob/master/onnx_coreml/_operators.py#L1517. The changes look like this:

def _convert_upsample(builder, node, graph, err):  # type: (NeuralNetworkBuilder, Node, Graph, ErrorHandling) -> None
    # NEW SECTION -----
    key = next(iter(node.input_tensors.keys()))
    scales = node.input_tensors[key]
    if len(scales) != 4 or scales[0] != 1.0 or scales[1] != 1.0:
        err.unsupported_op_configuration(builder, node, graph, "Unsupported scales {} for upsample".format(scales))
    height_scale = int(scales[2])
    width_scale = int(scales[3])
    # NEW SECTION -----

    # if 'scales' in node.attrs:
    #     scales = node.attrs["scales"]
    #     if len(scales) != 4 or scales[0] != 1.0 or scales[1] != 1.0:
    #         err.unsupported_op_configuration(builder, node, graph, "Unsupported scales {} for upsample".format(scales))
    #     height_scale = int(scales[2])
    #     width_scale = int(scales[3])
    # else:
    #     height_scale = int(node.attrs.get('height_scale', 1))
    #     width_scale = int(node.attrs.get('width_scale', 1))

@arsenyinfo could you please give a try to https://github.com/onnx/onnx-coreml#new-beta-onnx-coreml-converter-with-core-ml-3 latest beta?
Feel free to re-open if you still face this issue


Creating persistence volume using StorageClass (pd-standard or pd-sdd) on GCE and Kubelet Client Certificates

I found couple of issues with the current configuration where control nodes and worker nodes are not able to recognise persistence volume, giving following error while creating POD with PVC:
"Kubernetes: Failed to get GCE GCECloudProvider with error "
to fix that need to add "--cloud-provider=gce " in following service configuration files
controller nodes: kube-apiserver.service, kube-controller-manager.service
worker nodes: kubelet.service
Another one was related to issue regarding Kubelet Client Certificates, when try to connect to pod using ssh or  getting logs using kubectl log  command, getting following error:
"Unable to connect to the server: x509: certificate is valid for worker-0, not worker-0.europe-west2-a.c..internal"
adding following in the hostname should fix the certificate issue:
-hostname=${instance},${EXTERNAL_IP},${INTERNAL_IP}, ${instance}.c..internal \
Hope that helps,
Thanks

@amitmahajan78 in #502 I used --hostname-override worker-0, which meant I didn't need to regenerate certs.


Workbooks with non-published Views get migrated with all Views published

HI,
When migrating a Project I found that there were many extra Views which got migrated over. On analyzing, I found that we had many Workbooks where only the Dashboards were Published and not the Views, on the Source environment. However, when we migrated using the TabMigrate tool, all the Views were migrated as well. We had to manually open each Workbook with the Tableau Desktop and Publish again with just the Dashboards.
Please advise how we can migrate without this issue. Thanks

We just ran into the same problem.
As the workbooks are downloaded and uploaded again, the information is currently nowhere persisted. I guess it might be possible to include in the content metadata (assuming one can access that information) but that would need some more development.
For now we will go with your mentioned approach: Download and only publish the needed sheets