Proposal: Refine Light and Dark ThemeResources - Use Pure Black for OLED only

Proposal: Refine Light and Dark ThemeResources
There has been a lot of feedback in the FeedbackHub about making changes to the Dark Theme colours to move to using a Dark Grey base colour instead of a pure black #000000.
This would work well on LCD displays like those commonly used on Desktop PCs, but would negate some of the power saving effects that Black gives for (AM)OLED displays commonly found on mobile devices.
Xbox already uses a specialised range of colours which avoids pure black and white which is beyond the non HDR range of television colour gamuts.  Apps from Microsoft like OneNote which has Dark Theme support avoids using pure black and so has to override the default DarkTheme resources.

Summary
My proposal is that with WinUI 3.0 these Light and Dark ThemeResource brushes be re-examined and some mechanism be found to detect the display type and provide adjusted colours on OLED devices.
Rationale

Moving to a Dark Grey base will avoid "glowing" artifacts around light coloured elements;
Fluent ThemeShadows will show up for elements with elevation translations;
White on Dark Grey avoids some eye strain;
There is strong support for this change in the Feedback Hub;

Scope

Capability
Priority

Change the XAML ThemeResources to use Dark Grey as a base
Should

Allow OLED devices to use pure Black as a base
Could

Important Notes
Dark Modes on other platforms avoid using pure Black as a base colour, and have special affordances to refine the colours used by the mode.
macOS Mojave uses a dark grey but then takes the predominant colour from the desktop background and then blends it into the colours to give a refined palette which blends in nicer than pure neutral greys.

Android Q is getting a dark mode, which specifically lightens the background colour of elements as they increase in elevation in order to give a sense of layer depth and heirachy

Office apps like OneNote also avoid using a pure black base colour, and that team's use of depth and subtle shadows were influenced by the Fluent Design guides, and so Windows should probably take note of this.
FabricWeb from the Office team, as well as the new Edge built on Chronium, also have moved away from using entirely neutral shades of Black > Grey > White - in favour of a warmer tone.

EDIT: I have made an image to demonstrate how the ThemeResources could work with these blended Base colours with the Dark Theme

EDIT: Also whilst there is no outcry for changes to the Light Theme, Here is how the same blended approach could work for it.

Open Questions
This whole topic is an open question.  So I propose this to be as much a discussion as to the merits of the change, as well as how to implement it.
Should the Light and Dark themes use the warmer tones chosen and recommended by the Office and Fabric teams?
Should the Accent colour or the automatically detected colour from the wallpaper, be used to blend into the Dark Grey to White shades of colour, similar to how Apple's macOS Mojave Dark Mode works?  This could help the dark app windows blend and sit more comfortably on the screen...
I am mentioning here @RudyHuyn who posted a poll and discussion on twitter.
https://twitter.com/RudyHuyn/status/1135968903585181696
Also mentioning @JeGentle who called out the Collection in the Feedback Hub asking for a Light, Dark, and Black theme for Windows.
https://aka.ms/AA1dxxf

I'd like to see any newly proposed standard dark color demonstrated in Settings and Security app of Windows 10. Below are picks of how they currently look:

For the settings app, there is a nice color contrast between the navigation pane on the left and the actual content on the right, something I'd definitely want to preserved.

In case of the Security app not only do we still have the above color contrast (btw the navigation view should really be extended into the titlebar to match the Settings app) but we also have a powerful contrast between the dark background and the status color (green, yellow, red) for each security section, making it stand out which is desired here.
With those two examples above, any change in dark color should be subtle (looking at your color table above at most Gray200 I'd think - but even then I'd like to see that color in action for the apps above first before making a verdict).
I also don't think the dark/light theme color should be influenced by the current wallpaper. For one, just have one light/dark theme and don't introduce more variations again if what you want is consistency in the system. More importantly, however, the system has no control over the frequency of wallpaper switches. If a user decides to have a new background every minute, you'll end up with extremely irritating theme changes every minute as well.

The way macOS handles it's dark mode colour blending is not exposed as options for the user, but when dark mode is on, it then automatically tints the colours

Regarding dark theme color shades, the Material Design docs have the right idea (as mentioned by @mdtauk):

Higher elevation, lighter surface
The higher a surface’s elevation (raising it closer to an implied light source), the lighter that surface becomes. That lightness is expressed through the application of a semi-transparent white overlay.

Although I'm not sure that a  'semi-transparent white overlay' is the best approach, beyond just using notion of it to decide on which shades of grey to display.
The UWP Fluent Design docs do go over the importance of elevation & visual hierarchy, but only in reference to how shadows are impacted by z-depth. The brief dark theme section also doesn't mention color usage in regards to elevation.
So something to consider with this conversation is if Dark theme colors should be tied to z-depth, similar to how shadows are handled via ThemeShadow. This does introduce a couple of important questions:

How to best assign colors to z-depths, including upper and lower bounds + default height?
Should color transitions be smooth if/when z-depth changes are animated?
Should composition lighting be used to change the shading (complicated by application to content vs background)?
Should this only impact dark theme colors or light as well (and how)?

@Felix-Dev I used the Live Visual Tree while debugging to make some tweaks to the Settings app, I am using a very Dark Grey instead of Black and I think it looks fine

I took my wallpaper and used the Average filter in Photoshop to get an average pixel colour #8b8cab
I then added a black layer on top and changed the opacity to 90% and then took that colour to use to replace the pure black in the example above.  #0e0e11
Then I tweaked some of the controls to add the corner rounding that is coming with WinUI 3.0 and tweaked some font styles.

Here is black and the new colour side by side.

And here is the Acrylic fallback colour which is 10% lighter than the base dark colour, I think the contrast still works well. #272729

As for elevation, I think these will have to be provided as XAML brushes generated by the OS, rather than actually manipulating the colour at the rendering or composition level.  The Z-depth Translation value could be linked to certain brushes, maybe even make it a XAML property, instead of having to do it all in code.
<Grid Elevation="03">
</Grid>

@mdtauk
Thank you for providing me with a chance to see how a darker gray would look like! I agree, it still looks good to me and as such that would be a color value I definitely would agree on (in fact, I barely notice a difference in the settings app 😃)
Looks like a perfect color compromise to me!

@Shaheedmalik @crse
Care to elaborate on your down vote/confusing reaction? Contrast in Windows apps has to be kept in mind when changing the current dark theme color to a lighter dark color.

Here is an image showing how the various shades used by the controls, and the Elevations would work with the blended Dark Base Colour.

Is this something that would be OS-level? Essentially an additional mode, a "neutral dark" in addition to the existing light and dark modes? If the true black stays for OLEDs, I assume this is how it would need to be done.
My personal preference would be to keep the true black for all displays. I find dark grey to be incredibly hard on my eyes - more contrast is better. It's cleaner, sharper, easier to decipher colours (that first OneNote pic, the purple text is barely readable), and overall much easier to read. The Office apps and the new OneNote are some of the only apps that I don't use dark mode with - I can't stand the grey. Too hard to decipher UI elements and text.
Having 3-4 system-wide modes would be a valid option. This is already pretty common - Office desktop apps and Adobe apps both have 4 modes: black/dark > medium dark > light > all white. It would satisfy all users, and wouldn't be any harder to support in an app than the existing two modes are.
I'm not seeing a huge rationale for dark grey - it might serve as a "transition" for people used to light themes, but doesn't seem to have actual notable benefits. If eye strain is the concern, would not dimming the entire screen (and thus reducing that contrast) be a more logical option? I don't really see how "absence of light" is the issue here...
Most Fluent graphics (the new ones they just rolled out across Windows Dev Center) are using gradient colours on true black. If bold colour effects are going to be common in UI, that also needs to be considered.
I do like the idea of having lighter shades for transient UI and other elevated elements, it does provide useful visual hierarchy. Would light theme get the inverse (darkening with elevation increase), or would it continue to rely purely on shadows? Currently, all elevated surfaces are the same shade of grey with the light theme. Having different shades might help there too.
As for shadows... they do help with visual hierarchy. But again, I don't think slightly lightening the background would solve that. The effect would still be far more subtle than it is for light theme. The dark theme may need an alternate method, similar to how Reveal highlights are dark on the light theme. Perhaps an "underglow" effect, similar to Xbox's Reveal Focus (but desaturated)? Something like the element floating over the light, if that makes sense? Not sure.
The tinting/blending idea from macOS is interesting, but I think Acrylic already covers that effect. Any place that would benefit from that effect, would likely benefit from Acrylic.

Is this something that would be OS-level? Essentially an additional mode, a "neutral dark" in addition to the existing light and dark modes? If the true black stays for OLEDs, I assume this is how it would need to be done.

My proposal would be just using this new approach with the existing Dark Theme - but an OEM making a device with an OLED screen, would be able to tell the OS, and it would use pure Black in that circumstance.  The display tech, would dictate that option.

My personal preference would be to keep the true black for all displays. I find dark grey to be incredibly hard on my eyes - more contrast is better. It's cleaner, sharper, easier to decipher colours (that first OneNote pic, the purple text is barely readable), and overall much easier to read. The Office apps and the new OneNote are some of the only apps that I don't use dark mode with - I can't stand the grey. Too hard to decipher UI elements and text.
Having 3-4 system-wide modes would be a valid option. This is already pretty common - Office desktop apps and Adobe apps both have 4 modes: black/dark > medium dark > light > all white. It would satisfy all users, and wouldn't be any harder to support in an app than the existing two modes are.

This would be another way to approach it, by making an additional mode or even a Toggle checkbox which shows up when the App mode is set to Dark.
[X]  Adjust colours to avoid eye strain
or perhaps something like
[X]  Use neutral theme colours
This may be a more global option if it is felt that the light theme would benefit from this colour blending also.

I'm not seeing a huge rationale for dark grey - it might serve as a "transition" for people used to light themes, but doesn't seem to have actual notable benefits. If eye strain is the concern, would not dimming the entire screen (and thus reducing that contrast) be a more logical option? I don't really see how "absence of light" is the issue here...

Pure black on a desktop screen causes white text and elements to exhibit a "halo" glowing affect, as well as making shadows impossible to see for elevated elements.  It also causes Windows to appear totally transparent on Hololens, and can be painful to read text on - in the same way that pure white backgrounds with black text causes eye strain over time, and in darkened rooms.
There are design studies that probably have more qualititive arguments, and I am sure there are usability studies.  The arguments for using pure black as a base colour, are mostly the battery benefits on OLED devices, and aesthetically blending into the bezels on black electronics.

Most Fluent graphics (the new ones they just rolled out across Windows Dev Center) are using gradient colours on true black. If bold colour effects are going to be common in UI, that also needs to be considered.

I took a look at the Windows Dev Centre dashboard and site, and in Dark Mode, the base colour is actually #1e1e1e and not pure black.

I do like the idea of having lighter shades for transient UI and other elevated elements, it does provide useful visual hierarchy. Would light theme get the inverse (darkening with elevation increase), or would it continue to rely purely on shadows? Currently, all elevated surfaces are the same shade of grey with the light theme. Having different shades might help there too.

Lighter shades for elevated surfaces would require guidance to pick appropriate brush resources or some kind of mechanism to take the XAML brush colour applied to the element, and an adjusted colour be applied to the element.

As for shadows... they do help with visual hierarchy. But again, I don't think slightly lightening the background would solve that. The effect would still be far more subtle than it is for light theme. The dark theme may need an alternate method, similar to how Reveal highlights are dark on the light theme. Perhaps an "underglow" effect, similar to Xbox's Reveal Focus (but desaturated)? Something like the element floating over the light, if that makes sense? Not sure.

I believe elevated elements should use a dark outer, and lighter inner border, which would do a lot to help reinforce this fact, and help mitigate the harder to see shadow.  The Light border is subtle in a light theme where as the shadow is strong, and in Dark the light border stands out more.
That again is a guidance issue.  Perhaps it could be made a material in the same way as Acrylic and Reveal.  Perhaps Elevated and Acrylic Elevated

The tinting/blending idea from macOS is interesting, but I think Acrylic already covers that effect. Any place that would benefit from that effect, would likely benefit from Acrylic.
Acrylic is used for transient surfaces and shows elements behind - but has performance implications.  The colour blending I am suggesting is Opaque and is no different to the current brushes, only the colour values have a tint applied, either taken from the Background wallpaper, or using the Accent colour.

I'm all for user personalization choices, and a recent poll which asked for people if they would prefer pure black or a darker gray was pretty close (52% for a "very dark gray background" with 48% for pure black with a total vote count of 2422).
@mdtauk did post an example color above which isn't pure black but still provides good contrast IMO. If the new dark color is set to a lighter gray, however, I'd definitely call for an additional theme option to be added to the system: "black" (#000000), "medium dark" (tbd) and "light" (possibly as it is now).
To sum it up, having two dark-themes to choose from seems like the way to go: "black" (#000000) (would probably be up for a color very very slightly lighter too) and "medium dark" (tbd), with "medium dark" addressing those who prefer a darker gray over pure black (see the poll).
(Reference to the poll here: https://twitter.com/RudyHuyn/status/1135968903585181696 - already posted by Martin above)

Having a options for Light, Dark, and Black would be the easiest.
Light (Currently)
Dark (Currently)
Black (#000000)

@shaheedmalik
I'm not sure what you mean with "Dark (Currently)"? The current (base) color used by the "Dark theme" is pure black (#000000). (As seen by the Settings/Security app and also when creating a new UWP app with the system-provided black theme.)

@shaheedmalik
I'm not sure what you mean with "Dark (Currently)"? The current (base) color used by the "Dark theme" is pure black (#000000). (As seen by the Settings/Security app and also when creating a new UWP app with the system-provided black theme.)

That's not the current base for Dark Mode. In fact, those apps haven't been updated since to reflect current dark mode.
Dark mode (around #373737) is currently reflected in apps such as the Feedback Hub.

Something to add to the idea of having two dark modes in the system:
As with many things it will be up to the developers to provide support for themes in their apps. Currently, with a single "dark" theme each developer can still ship provide its own version of it (see OneNote app/ToDo app/Settings app). There is no specific color associated with the "Dark" setting in the Settings app.
However, once we add more themes and even start to stricly bind them to a color, such as "Theme Black" (#000000) or "Theme LightGray" (some Light Gray color value) app developers might choose to not provide support for this particular theme because it wouldn't fit well with the overall look of their app.
That would introduce a whole new set of problems, i.e. a user sets the theme to "pure black" just to find out it won't get respected. Compare that to the current liberal description of just "Dark".
So, once we add more them options and/or tie themes to specific colors there could be a risk of disgruntling users because not every app or system component they will use on the system might have support for that particular theme.

@shaheedmalik
I just created a new UWP test app targeting the newest SDK (SDK 18362). Setting the requested theme to "Dark" gives me a base color of pure black (#000000).
So according to that, the feedback hub is using its own dark theme.

@shaheedmalik
I just created a new UWP test app targeting the newest SDK (SDK 18362). Setting the requested theme to "Dark" gives me a base color of pure black (#000000).
So according to that, the feedback hub is using its own dark theme.

The Store also uses a different colour - I suspect this is because it was a mistake having Black as a base in the first place - but that leads to the problem of the Store on Windows 10 Mobile now being in grey instead of Black.
If the Dark Theme is updated, those apps need to move to the new default.

My proposal would be just using this new approach with the existing Dark Theme - but an OEM making a device with an OLED screen, would be able to tell the OS, and it would use pure Black in that circumstance.  The display tech, would dictate that option.

There is already an option for OEMs to specify OLED - it's used for taskbar transparency. This is hidden from users, by burying it in the registry. Since opinions seem pretty split on this, might be better off surfacing the option in the Settings app. The OLED option could still be used to set the default.
Also the Fluent site I was referring to was this one with the new Fluent logo: https://docs.microsoft.com/en-us/windows/uwp/design/ (the logo is on a background that shows as #000000 in Photoshop for me). What I was specifically referring to, we haven't seen it yet in actual UI, but the use of colours.
Anyways... for built-in elevated controls like MenuFlyout, I suppose they could take the theme resources for shading - not sure if such resources are already present. These already take on both light and dark styles automatically, so no additional work there (assuming that darker=higher works for light theme). For custom controls, they can just use these resources as appropriate. I suppose it could just be tied to the app's set z indexes (as I believe the new drop shadows are), but that might be undesirable if it affects all controls.
The border colours is definitely something that needs to be standardized. In the Windows taskbar alone, there are at least four different border styles. A light+dark border would be great for ensuring clear separation, especially for transient UI.

However, once we add more themes and even start to strictly bind them to a color, such as "Theme Black" (#000000) or "Theme LightGray" (some Light Gray color value) app developers might choose to not provide support for this particular theme because it wouldn't fit well with the overall look of their app.

Developers don't set colours right now. It takes 0 extra work to support an additional theme. The OS handles everything.
Right now you pick out preset theme resources, which are like different shades. The OS will automatically pick out colours based on the user's chosen theme, and then use whatever shade was specified. i.e you specify "background colour" and the OS will automatically put white (for light) or black (for dark).
The current mess of shades is because of apps (seemingly) disregarding this behavior. Clearer guidelines might help. There isn't really any clear indication of what shade to use where, just that apps should use the theme brushes: https://docs.microsoft.com/en-us/windows/uwp/design/style/color#using-theme-brushes
A proposed "medium dark" theme would apply to existing apps with 0 changes. Apps are allowed to request a specific theme - presumably any that currently request Dark can continue using the current dark theme, to avoid any visual issues that'd require changes.

@KyleNanakdewa
My point was that for each app developers might think a modified theme is better suited than the default-privided ones. With helper apps like the Fluent Xaml Theme Editor, you can easily modify brushes present in all those themes and thus create our own "Light" theme or "Dark theme".  That's how you get a different black color in apps like Feedback Hub compared to the Settings Hub, and the ToDo app for example uses yet another set of values for their dark themes.
That's what I like about the current choice. You have all those different apps, each with their own dark theme version which the devs think is best suited for the kind of content their app displays, and yet all those apps have a "dark theme". The moment specific colors can be selected in the Settings app for themes, you might not get fully support for it across First-Party and Third-Party apps.

@shaheedmalik
I just created a new UWP test app targeting the newest SDK (SDK 18362). Setting the requested theme to "Dark" gives me a base color of pure black (#000000).
So according to that, the feedback hub is using its own dark theme.

I am going off around what the actual apps are using. At this point all of the Inbox apps have to be rethemed. Dark being black and all the apps having varing degrees of gray proves that they are all off template. And have been for a while.
I have checked like three apps and I saw, 1E1E1E, 373737, and 363636.

It's possible that the developers of these apps preferred (for whatever reason) the grey (over black), and thus decided to create their own theme for it. Therefore, if such a theme was available in the OS, they might update their apps to remove the custom theme and revert to the standard/recommended theme resources.
Of course that's just speculation. We don't know why apps like the Store went with grey instead of black... but they (probably) wouldn't need to write their own theme if there was one built-in.

It's possible that the developers of these apps preferred (for whatever reason) the grey (over black), and thus decided to create their own theme for it. Therefore, if such a theme was available in the OS, they might update their apps to remove the custom theme and revert to the standard/recommended theme resources.
Of course that's just speculation. We don't know why apps like the Store went with grey instead of black... but they (probably) wouldn't need to write their own theme if there was one built-in.

The developers of those apps is Microsoft. Since there wasn't a universal theme for dark gray, it looks like each theme did a gray updated without talking with the other teams and making a new template.

Of course that's just speculation. We don't know why apps like the Store went with grey instead of black... but they (probably) wouldn't need to write their own theme if there was one built-in.

The developers of those apps is Microsoft. Since there wasn't a universal theme for dark gray, it looks like each theme did a gray updated without talking with the other teams and making a new template.

We can assume there was a dislike of the pure Black, and so it was changed.  The Store also updated it's Light Theme to use a light grey to match their website colours.
Eitherway apps using default theming, would get the updated colours.  This change should probably apply to WinUI 3.0 and beyond.

Speaking of, it would be neat if apps and websites tried to avoid pure white and pure black and reserved that for content.

A lot of apps are currently providing own personalization options.
I think it indicates lack of the system options. 4 themes like "black", "dark", "light", "white" could cover most preferences.
I'd really welcome such change as a user and would be also happy as a developer because I won't need to implement it myself in my apps 😃

A lot of apps are currently providing own personalization options.
I think it indicates lack of the system options. 4 themes like "black", "dark", "light", "white" could cover most preferences.
I'd really welcome such change as a user and would be also happy as a developer because I won't need to implement it myself in my apps 😃

@cherepts
And that's the issue. Providing a small finite set of pre-defined themes perhaps won't be the solution as you might imagine it to be. To quote someone from twitter:

Before that, let's start simple: 171717, 1e1e1e, 262626, 000000, 363636. These are the hex codes for the "dark" background color in 5 Microsoft apps: News, windows settings, Visual Studio, Word, Edge/Chromium.

As you can see, 5 MS apps with 5 different dark theme base values. Now count in all the third-party apps as well where each app might want to use its own base theme value.
As I said above, the issue I see is that Windows does not know which color combination works best for apps. Hence adding specific themes, such as "black", "white",... might introduce UI options which won't get respected anyway. So, you will have an app where the developer says "black", "white" just won't work because of how the app is is designed. A user, however, sets that theme in Settings, then uses such apps and will start complaining why his preference won't get respected.
To put it differently: It's up to the developers, not Windows, to determine which themes an app will look good with and which it won't.
That's what I like about the current setting. All those 5 MS apps use their own dark theme, but it is still a "dark theme" nonetheless. Thus, the user preference gets respected by all those apps.

@Felix-Dev Ok, 5 apps now have different themes. That means that at least 4 developers did create theme manually and that every user is unhappy about at least 4 apps. I wouldn't say it sounds good.
Personally I prefer #000000 and think that other apps look dirty so I rarely use apps like News of Xbox Companion.

@cherepets
Black does look nice in Settings/Security app. I'm also fine with the much lighter gray used in the MS ToDo app.
The problem is that to a great extent every developer picks his own dark theme which he thinks looks best for his app. That won't go away even if more theme options in the UI are introduced. Either you provide customization for developers or you have to make it "hardcoded". And with MS big on enabling easy branding of apps, you will never get consistency anyway. More options could instead make the current situation worse as seen with the example above - devs opting not to support some options, thus ignoring user preferences, which the user will never like to see.
Such as it is today with devs having to opt-into respecting theme preferences by the user (it can always be ignored) at best I feel app developers could be advised to provide more theme options, such as a black one, a lighter gray and a light one (or more). In the end it will be up to them how far they will go.

The reasoning for different apps using different shades of grey seems pretty obvious - there is no established standard whatsoever. They're using different shades because there's no guidance on what grey to use, not in spite of it.
The chosen shade of grey is rarely, if ever, a branding choice. I cant think of a single case where the specific shade of grey is relevant to an app's identity.
If a specific app cannot achieve necessary contrast using the existing theme resources, I would say that's a failing on the part of that app's developer - the app is already supposed to be designed to adapt to the theme resources, given that they change between light and dark themes. If it is truly necessary for a specific app or control, hard coded colours continue to work, or the theme resources can be overridden if such a specific change is needed on a larger scale. This is nothing new - it has been part of Windows 10 from the beginning.
Also worth noting that the suggested change (dark grey backgrounds) is not massively different than the existing black. If an app is negatively affected by having to adapt to both colours, it likely had poor contrast from the start. This isn't a drastic change in colours.

The reason the gray those apps are inconstent is because each team did those changes off template. And customers complained about it. (I know I did.)
It had nothing to do wanting a different choice.

@KyleNanakdewa
You say "there is no standard". Well, the thing with standards is that you reduce dev choice without having any idea about the branding/content of apps. The work done by the Windows Team (Fluent Xaml Theme Editor,...) is all about supporting dev customization so they can create their own themes. Windows shouldn't be in any position to dictate how apps will have to look like.
And we've just seen how that turns out because I'd argue there indeed is a standard: the default color black. As written above, out of the box, the base color for the "dark" theme is black! Yet we still have devs which went with it and others who went with a whole set of other base values.
So, perhaps to clarify your point: How would you think a standard should look like? What makes you think the standard will be followed by others? A standard only makes sense if it's broadly used and there should be made no such assumption in this case here.

The chosen shade of grey is rarely, if ever, a branding choice. I cant think of a single case where the specific shade of grey is relevant to an app's identity.
It's not about the base color but about the complete look of an app. Depending on the colors chosen to reflect a branding (such as foreground color), one kind of gray/black might be better suited than others. The base theme color might not be starting point of a custom theme design process but it certainly will play a role at some point. It's not up to Windows to decide which kind of gray works best with other app colors.

Also worth noting that the suggested change (dark grey backgrounds) is not massively different than the existing black.
That's true and I said previously, I'd be up to such a change. That hasn't been my issue here anyway. Changing the base color of the dark theme to a subtly lighter color won't address the issue above, you will still have devs who will think black/an even lighter gray is better suited for their apps and no one should tell them they can't chose their own app theme. The problem of different black themes will still exist.

@shaheedmalik

It had nothing to do wanting a different choice.
Can you clarify which perspective you are describing here? The user perspective or the dev perspective?

So, to put it together:
At the end of the day, you will still have all kinds of different grays used in apps. It's not up to us to speculate the reasoning behind modified themes for apps. Fact of the matter is, we have all kinds of colors in use today and I fail to see why a small set of pre-defined theme values will make the situation any better (and as I argued above I think it will make it worse) The base color to use is dependent on many factors and some of those factors cannot be decided by Windows nor should they be.

Windows shouldn't be in any position to dictate how apps will have to look like color-wise.

This is literally the exact primary purpose of theme resources, which is what we're discussing here.
https://docs.microsoft.com/en-us/windows/uwp/design/style/color#using-theme-brushes
The whole idea of this proposal is to change the choice of colour, as it is made available to apps in the form of theme resources. I'm not sure what you're getting at when you say this should not be determined by Windows, because that doesn't fit the context of this proposal.
Furthermore, many apps already support multiple dark themes - Office and Adobe apps are big ones, plus many existing UWPs - I use Nextgen Reader, Ouga, Skype, Quarrel, all of which support both a black theme (normal dark theme), and a grey theme. Currently, apps must supply their own definition of the grey theme - it would make sense to add a standard, similar to how we already have Dark, Light, and High Contrast themes. This would reduce the work needed by developers.

As @KyleNanakdewa has said there is a difference between what colours an app chooses, and what Windows provides.
At present, Windows has a Dark and Light theme, which then tells the app which Theme Resources to apply when the app is running.
When designing an app, they are provided as two ResourceDictionary XAML files included in the SDK.  But those same colours, values, and metrics are included in the framework in memory for apps running.
So for the avoidance of doubt, I am talking about the Light and Dark Theme Resources provided by default, and not about preventing developers from overriding those styles.
Many of the apps made by Microsoft are perfectly happy to use the default Light Theme, but feel the need to customise the Dark themes.  I believe this is to avoid the use of pure black as a base colour.
With WinUI 3.0 becoming decoupled from the OS, it will fall on the framework to provide those new default styles for controls and the themes, and so I am suggesting improvements to the Dark Theme in particular, but open to seeing the same changes come to the Light Theme too.
Improvements which other platforms have made to their Dark Theme implementations, based on best practices, and hoping that all those apps who have chosen to "do their own thing" with their Dark Themes - will choose to return to the defaults if they address those concerns.
Netflix, the BBC, and Facebook will be free to use their own brand colours, and their own control designs and colours - to match their cross platform apps.

@shaheedmalik

It had nothing to do wanting a different choice.

Can you clarify which perspective you are describing here? The user perspective or the dev perspective?
So, to put it together:
At the end of the day, you will still have all kinds of different grays used in apps. It's not up to us to speculate the reasoning behind modified themes for apps. Fact of the matter is, we have all kinds of dark colors in use today and I fail to see why a small set of pre-defined theme values will make the situation any better (and as I argued above I think it will make it worse) The base color to use is dependent on many factors and some of those factors cannot be decided by Windows nor should they be.

All the Dark Mode apps started off at #000000.
Feedback requested a lighter Dark Mode, each team made a different looking dark mode. They weren't consistent.
Feedback complained about app consistency specifically of MS apps.
Had all the apps had guidelines for the exact same lighter dark mode color, the apps would have looked more consistent thus less feedback on app consistency.
What should have been done was a new universal color should have been discussed  among the teams and new template should have been made. Apps retemplated = Consistency.
Let's fix this problem.

What should have been done was a new universal color should have been discussed among the teams and new template should have been made. Apps retemplated = Consistency.
Let's fix this problem.

100% this - and while fixing the problem with pure black, let's consider these colour blends, to add a little polish to these themes!

There's a thread on Reddit talking about this very issue.
https://www.reddit.com/r/Windows10/comments/c0gewl/i_tinkered_with_the_colors_to_try_to_match/

I'm skeptical here, after all, that was my whole point. Deciding on a new standard value which at least all the different MS teams with their apps have to follow - not sure there will be a solution. Apps are different, and thus some might want to have a certain gray color which won't be the perfect choice for other apps (see ToDo app vs Settings app with a navigation view and content pane, for example). If you start with a base color several apps use today and then you change it slightly because of the request of other app teams, you might get new complaints from the teams responsible for the former apps.
From my point of view perfect color match trumps consistency. If there are (MS) apps for which a darker/lighter gray is better suited for the displayed content than given by the "new universal color", those apps should always ship with that base color. If consistency means negative impact on in-app contrasts, it's not the way to go.
I'm interested in seeing how the process of finding the "new universal color" will go and how it will look for a variety of apps.

I really appreciate if you could read this before posting a new comment
Hello everyone,
Thank you so much for very active and engaging conversation that is taking place in this proposal. I am very happy to see the community engaged to provide us with valuable feedback and proposals for making our UX/UI better.
Let me introduce myself first for those of you who don’t know me already from other WinUI proposals. I also want to make sure I am super crisp on what I can and I cannot do.
I’m a PM in WinUI team. Currently, I play a role sort of an “interpreter” who distills user, developer, and design requests around UX/UI into specific features and propose changes that WinUI would make. I also generalize very specific design changes into changes that we can systematically and coherently make with the set of controls we offer. You could also view me as one of developer advocates specifically in design decisions that are made.
While I work very closely with many different design teams, I am not in the position to change design decisions that have already been made by respective organization. However depending on issues/proposals being raised here, it is possible for me to help influence design teams to take the idea into design considerations and/or modify the design if the impact to developer is significant.
As you can see, better I understand your ideas and issues, better I can be your voices in design discussions that happen here for WinUI. In other words, help me help you. 😊
That being said, let me help this group make a better case for the ideas being discussed here. Here are things you should consider doing when making proposals and discussions for design. Again, this helps me take your ideas to the right people better so I appreciate your help!

State the problem(s) you are trying to solve clearly. Depending on the problem, the solution and approach can be vastly different.
HINT: Providing evidence of the problems you are observing in real-life is a plus.
Rather than creating one single big issue, break them down into small, independently actionable issues.
HINT: If multiple issues make up one big feature, you could create one “master” feature and refer to smaller items.
If proposing design change for a control design that’s already well established, explain what the problem current design impacting our users and how the new design help solve these specific problems.

I’ve noticed that there are multiple problems being discussed here in this thread but I wonder if we all have the same problem in mind. Let me try to list out the problems I think I hear following from the series of discussion here?

There are not enough theme options for USERS.
There are not enough theme options for DEVELOPERS.
USERS have different opinions as in what color of gray is better.
DEVELOPERS do not agree with what color of gray is better for their application.
Gray colors being used by Microsoft products are not consistent.
Gray colors being used by non-Microsoft products are not consistent.
It appears MacOS has better solution compared with Microsoft?
Maybe more?

It would be very helpful if the group could come to consensus as to what the problem we are trying to solve here. Perhaps we are trying to solve multiple problems and it is OK. I just want to make sure I am understanding the problems and why we need to solve these problems before we start discussing any specific solutions for them.

@KyleNanakdewa, @Felix-Dev, @shaheedmalik, @cherepets – I believe you all suggested we introduce more themes. Are we trying to solve a problem for users or developers or both?
@mdtauk – It appears you suggested a separate feature of blending background that could be a stand-alone proposal?
Many of you expressed frustration with Microsoft products not being consistent. I hear you and design teams are working together to make our applications coherent (as you heard from our //Build talk). There is nothing I can share here right now. As far as I could tell, none of the solutions being discussed here in this proposal did not feel like a silver bullet to solve this once and for all?
@KyleNanakdewa, @Felix-Dev, @shaheedmalik, @cherepets, @kmgallahan, @Poopooracoocoo are discussing @mdtauk’s original topic of OLED detection/support. Would you help distill key problems you are trying to solve with this solution?

I’m sorry for a very lengthy post. FWIW, this is how we design and discuss design changes at very high level and having the community follow this similar model will help us take your idea into next step faster and with greater impact.
Thanks! Looking forward to your inputs!

@chigy  Thank you for such a detailed response.  Would you prefer I close this issue, and make a new one with a more detailed proposal, breaking things down?
I would suggest those asking for more than the Light and Dark themes, should make their own separate proposal, as they are not part of what I am proposing.

@mdtauk ,

@chigy Thank you for such a detailed response. Would you prefer I close this issue, and make a new one with a more detailed proposal, breaking things down?

Since you are the one who made the proposal to the original request, I'll leave it up to you to keep it to discuss the OLED topic or close and open new one. Though, you will need to remind folks to keep the conversation on point as this contains many different topics. If this conversation results in something I feel I can pitch on our side, I may end up creating a new one just so it is cleaner but we can think about that when it is at that point.

Yeah, there's a couple things being discussed here. Might be better to split this up.
Dark Themes (adding an additional theme)
It's looking like three themes may be the way to go forward, for the following reasons:

Provides the neutral grey theme for users (or developers) that prefer this look.
Makes it easy for apps/controls to support a refined/additional theme with no additional work, if they are already using theme resources - same as already supporting both light and dark themes in an app/control.
Maintains the black theme for users that prefer that look.
Maintains the black theme for OLED screens.
Does not affect existing apps by changing their colours, as the black theme is still present.
Still allows developers to override the existing themes and provide their own replacements (such as by using the Fluent XAML Theme Editor app), though they may need to provide three replacement themes now, instead of two.

This proposal would involve adding a third theme, in addition to the existing Dark and Light. Not sure how it could be named, possibly "Neutral Dark", or maybe rename the existing dark theme to "Black".
The theme would be very similar to the existing dark theme, but avoiding pure black.
While themes can be changed within an app, as they can be now, it would be best to have an OS-level setting for users to change the setting across all apps. This is obviously outside the scope of WinUI, but given the response in Feedback Hub, I think it's safe to say there's a demand for an OS-level neutral dark theme.
Dark theme (replacing the existing theme)
An alternate proposal is to outright replace the existing dark theme with more neutral greys.

Provides the neutral grey theme, as the only option.
Does not add an additional theme, so apps/controls just have to support the new dark theme, and can ignore the old theme entirely.
Does not keep the black theme for users that prefer that look.
No way to detect OLED screens and adjust accordingly - unless the theme resource is determined at runtime, but that's essentially just adding a third theme that needs to be designed for.
Makes it easier for developers to override the theme and provide their own, as it's one less theme to override. As far as I know, currently a developer would need to replace each theme separately, so this could be a benefit.

OLED detection
This is also an OS-level feature. If a third theme is added, the OS could set the user's default theme based on the device. With this approach, it is 100% outside the scope of WinUI - it would simply set the default theme (black vs. neutral dark vs. light).
Colour tinting/blending (like from MacOS)
I think this is a separate proposal entirely. It would affect all themes (whether that's just light and dark, or a proposed additional dark theme), and thus isn't really related to the addition or refinement of dark themes.
Consistency across apps
This is more of a guidelines change than anything else. However, providing a neutral dark theme may help consistency, as currently devs have to design their own, if they want to add it. There is no set guidelines right now.

@chigy, you're right: providing extra themes and OLED screen detection are separate topics.
Extra themes could solve problems for both users and developers:

Developers will not have to implement their own lighter version of dark theme anymore
User will get the colors they prefer in more apps

Thanks again for the response 🙂

Is there any progress/direction on this? I've been messing around with WinUI 3.0, and by default dark mode is jet black, which is hugely different from everything else in the OS now that Windows 11 has released. I was under the impression that the goal of WinUI/UWP is to get developers creating applications that fit the OS visual design, but it seems at the moment that it does the opposite.

@JamesLear92 , yes as a matter of fact, we updated colors of WinUI controls. We made the change to WinUI 2.6 and this change is going to be available to WinUI 3 soon. Due to schedule coordination, we haven't had chance to introduce this in WinUI 3 yet but is coming.
Please see the change from design perspective:
https://docs.microsoft.com/en-us/windows/apps/design/signature-experiences/color

@chigy
This is beautiful. Thank you.
Do you know if there is an issue raised/pull request for this to come to WinUI 3?

Do you know if there is an issue raised/pull request for this to come to WinUI 3?

@JamesLear92 , actually, it is already available in the latest preview build.
https://docs.microsoft.com/en-us/windows/apps/windows-app-sdk/preview-channel#version-10-preview-3-100-preview3
There are some bug fixes that are coming with 1.0 GA which is just around the corner.


💡 RFC: the language tools shouldn't offer JS/TS completions in HTML code

Background & Motivation
When writing HTML I'm getting all sorts of weird suggestions. Such as:

Not sure if this is a bug or a feature... but these JS/TS suggestions should not be happening when writing HTML.
Proposed Solution
Possible solutions
JS/TS suggestions should only occur in JS contexts such as the front matter or dynamic expressions inside {}.
Detailed Design
No response
Help make it happen!

[ ] I am willing to submit a PR to implement this change.
[ ] I am willing to submit a PR to implement this change, but would need some guidance.
[ ] I am not willing to submit a PR to implement this change.

Could you share the file in which this happens?

I guess it's ok since it's a public blog post.
https://gist.github.com/PierBover/ad23cc9f1134b06a56d13ff1d0aad933

Awesome, thank you! I'll take a look as soon as possible. It might be tough to fix because of how this works internally, but I'll see what I can do.

but you're writing in an .astro file? not an .html file. Isn't that just the regular IntelliSense feature to insert astro components (which get automatically linked in the frontmatter, in case you haven't used that component before)?

But that intellisense should only trigger after typing <, no?
It's now triggering in actual text.

But that intellisense should only trigger after typing <, no?
It's now triggering in actual text.

doesn't this depend on your VS Code settings? I think by default if you type "p" or "div" without "<" it also autosuggests creating a p or div element. Hierachy wise that's not really different to entering a  predefined , so i would also expect "component" to show up as an autosuggestion if i start typing "com".

doesn't this depend on your VS Code settings? I think by default if you type "p" or "div" without "<" it also autosuggests creating a <p> or <div> element. Hierachy wise that's not really different to entering a predefined <component />, so i would also expect "component" to show up as an autosuggestion if i start typing "com".
VSC Docs: Customizing IntelliSense

It's possible but how come it only started happening a few weeks ago?
And also why does this only happen in .astro files not in .svelte or even .jsx files?
It could very well be something on my end... but I've been using Astro for a couple of years now and AFAIK the only thing that has changed is the Astro tools extension.

@Princesseuh I'm not sure how I could send you a repro. The issue is not in the .astro files themselves.
I just crested a new empty Astro project with 4.5.5, VSCode 1.87.2, and v2.8.3 of the tools. I'm not seeing the issue on this project.
It must be something on that other project. If I ever figured it out I will reopen this issue.

Okay, no worries! I'm sorry for the inconvenience, would've loved to investigate and fix this!

I haven't figured out the origin of the issue yet but I'm still having problems even in new empty Astro projects.
For example if I have a word and write a period . right after it I get these file completions:

Which produces a JS import.
This would make sense inside a tag to import a component or maybe right after a < character. But this happens in regular HTML text.
I'm not reopening the issue yet because it's not clear what's causing this. It does seem like it's the Astro dev tools though.


Clarifying focusing essence of windowByIndex()

Yep, this is better.

I'm closing this PR due to lack of response over the past year.
@3n-mb, please feel free to open a new PR that addresses the changes that @codebytere suggested. I'd be happy for a fixed PR to be merged.


feat: filter votes by choice

See https://github.com/snapshot-labs/snapshot-hub/issues/290#issuecomment-1113165105
but this breaks pagination :(

Closing this PR for now, it's unlikely we will allow query for specific choice as the choice field is a JSON and query into doesn't scale well.


Transclusions and links

It would be useful if the "create note" could have an option to prepend or append a link or a transclusions to the created note.
This way we can achive the following:

Place a button in a note.
Every time you press the button a new note is created, and a link or a transclusion is prepend before or append after the button.
The new note can be opened in a split pane, so it can be edited.

You can accomplish this with a templater template.

Hi! I'd be interested learn how?


Overhaul volto customization training

Modified Installation docs (Use Cookieplone only)
Rearranged some chapters and added the latest content.
Removed Extending teasers per type chapter as its out of scope for this training.

📚 Documentation preview 📚: https://plone6--860.org.readthedocs.build/

@nileshgulia1 do you have permission to request a review from other developers?
I have a huge backlog of documentation PRs to review before I can get to this one, but I would like to take a look, if I find time. Thank you!

@nileshgulia1 do you have permission to request a review from other developers?
I have a huge backlog of documentation PRs to review before I can get to this one, but I would like to take a look, if I find time. Thank you!

@stevepiercy Although its ready, I would do it during the weekdays. Just don't want to bother you during weekend :). But here you go!

@stevepiercy Could you please approve this if everything looks okay? Apologies for the delay—I've been busy finishing up tasks before the conference. Thank you!

@nileshgulia1 sorry, I didn't realize you were done. I'll start a review tonight.

@stevepiercy Unfortunately I'm not authorized to merge this PR :(.

@nileshgulia1 oh, my, that's not right. Invite sent! Please accept, and let me know if you can merge it now.

@stevepiercy Thanks! 🚀


Comment in first line in worsemeower.js

Why was that line commented out??
https://github.com/3r1s-s/meo/commit/8bb5233ccc731477e83c7717e8430a8013413781

Send screenshot

Too spammy might disable fr in the future

oh, sad

i would close

i would close

Oh yeah, you're right


UI: Sandbox page overflows from the bottom

Affected module
Does it impact the UI, backend or Ingestion Framework?
UI
Describe the bug
A clear and concise description of what the bug is.
The UI overflows from the bottom on Mac M1
To Reproduce

Go to https://sandbox.open-metadata.org/signin
Scroll down at the end

Screenshots or steps to reproduce

Expected behavior
A clear and concise description of what you expected to happen.

Assign to me


Add support for accounts on lbry.tv

From notion:
LBRYTV Flows v0.2
Anonymous

If no session is present UI calls internal-api/user/new and auth_token is retrieved
UI forwards auth_token to lbryweb, a user record is created and an account is registered with the SDK

Registration

internal-api/user_email/new method is called with user’s email as an argument
An email with a confirmation link is sent to the user
UI starts querying internal-api/user_email/status
User clicks the link in the confirmation email, calling internal-api/user_email/confirm
UI sends auth_token to lbryweb, a user is marked as registered

Blocked by https://github.com/lbryio/lbrytv/pull/28

@tzarebczan will open new ticket with remaining changes required


Fix the QPACK offline interop tool qif.

The [bin] entry has been commented so it was left behind and couldn't run.

Yeah, I think this would be significantly simplified by just embracing anyhow's dynamically dispatched error type. There's not much value to an error enum that will never, ever be matched on.

I left that because I was pretty sure Error was also used for logic. I doesn't seem to be the case.
Here, using an enum as still one goal: reduce call site error message duplication.
I'll rewrite all this.

Got rid of err_derive and enum Error. It's actually much better now :)
I also added a small doc to get started with the tool.


chore: Remove Helm v2 support

Helm v2.17.0 was the final release of Helm v2 in October 2020. Helm v2 is unsupported since November 2020, as detailed in Helm 2 and the Charts Project Are Now Unsupported.
 This means that the plugin is restricted to using Helm v3.1.3 (last Helm v3 release which uses Kubernetes Go Client pre 1.18) which was released in April 2020.
It therefore makes sense to remove Helm v2 support.
Closes #50
Signed-off-by: Martin Hickey<EMAIL_ADDRESS>

Thanks for the review @marckhouzam. I have updated with your comments.

Will you be updating the plugin version in a separate PR?

Lets do it in this PR. Bumped to "0.3.0"/


use react-helmet to properly inject <head> elements

Hey @sonnylazuardi!
I made this change based on the code review session I did with Imre yesterday. I might add more PRs based on the things I pointed out in the video too, so do look forward to that!
First things first, I wrapped elements intended for the <head> element inside react-helmet so that they get properly injected into <head> instead of inside the <body>.
Before:

After:

Thanks for the PR 🙇


how to open the gripper ?

I have a multi finger gripper but it is in completely closed position.
Are there any packages or tools that i could use to open the gripper and convert to graspit required format ?
Any suggestions ? Thanks!

If your robot is in URDF, you can use urdf2graspit, see the wiki. If it is not in URDF, then you need to convert it to URDF first.
If you have it in URDF, it is highly likely that the joints of the gripper are movable already.


Fix README

Since it was copied directly from Google docs there are some formatting errors.

Added a fix for 1) a typo (in.aci instead of int.aci) and 2) a wrong command (add-image instead of add)

@derekchiang thanks for the patch! would you mind squashing your commits?

@jonboulle done


RPN class losses is computed only for positives

While the Fast RCNN paper states that a batch of size 256 contains positive/negative ratio of 1:1 at most. if there are less than 128 positive, the batch is padded to 256 by negatives.
See:
https://github.com/CharlesShang/FastMaskRCNN/blob/master/libs/nets/pyramid_network.py
Lines:  359-410
The result is that the network does not compute a valid probability for each bounding box - it just outputs the constant 1.

@CharlesShang, can you please comment to make sure i'm not missing anything?


Question about updating values for radii in hyperball representation for crystal supercell

Hi guys! I am trying to use ngl to embed representation of crystal supercell:
view_structure.html.zip
lifepo4_supercell.pdb.zip
and trying to figure out how to set different radii for LI, FE, P and O instead of covalent or Van der Waals radii. I have tried to define new constants for radii to override radius constants somenthing like this:
const CovalentRadii = {
'LI': 1, 'O':3, 'Fe':2, 'P':2}
but this did not work, also I have use something like this:
const radiusData = {}
comp.structure.eachAtom(ap => {
radiusData[ap.index] = 1.
});
comp.addRepresentation("hyperball", {scale:.3, radiusType:"covalent", radiusData:radiusData});
this should set all radii to 1. but nothing happend.
Please can somebody give me some clue how to do that or maybe some examples and documentation to learn how to update and change parameters like for example radii of atoms or colors.

Hi @zorankiki, sorry I missed this. Does  radiusType: 'data' work?


add-schema function - no export.json file

The schema definition generated with the add-schema function does not produce an export.json file when creating the schema definition.
Could an export.json file be added to the generated definition?

I believe this is done with the latest version of Nunaliit.


Reference level - alien species

The threshold value for the alien species indicator is set to 4 on the original sevel-step R7 scale. This corresponds to about 5-30% alien species cover.
This thread collects arguments for keeping the threshold value (and reference values) as is, or changing them.

I guess 3 represents 1-5%. I would say that this would be a better threshold for good ecol. cond. than accepting up to 30%. I appreciate that just above 1% is quite harsh too, but accepting covers towards 30% as good is simply insane. With 1-5% invasive cover a system already is already on a road to perdition unless action is taken. Setting the threshold to 4 will mask the problem while the invasive spread will continue...and that all the while early intervention is supposed to be key when dealing with invasives.

Marianne Evju:
Basert på definisjonen av fremmedartsindikatoren, så vil jeg si at trinn 3 er en riktigere definisjon av god økologisk tilstand enn trinn 4. Middels sterk effekt (trinn 4), med stor ulikhet i artssammensetning fra referanseverdien, mener jeg ikke er innafor god økologisk tilstand. Grenseverdien bør settes på trinn 3 Nokså svak effekt – det kan være flere fremmedearter, men de har relativt lav effekt på artssammensetningen.

see also #87

I agree, and I've changed the threshold value to be 3 rather than 4


fix: correct the SnpPlatformStatus struct

This adds the missing fields!
Signed-off-by: Harald Hoyer<EMAIL_ADDRESS>

LGTM


Error handling within on_record callback

Describe the bug
Version: 4.8.5
Errors throwing within the on_record callback are not effected by the skip_lines_with_error option, this is making it difficult for me to utilise the on_record feature as an uncaught exception will fail the entire file.
To Reproduce
Playground: https://repl.it/@RobertPitt/CSV-Paese
const parse = require('csv-parse')
const assert = require('assert')

const data = `a,b,c,d
a,b,c,-` // bad data row

const on_record = ([a, b, c, d]) => {
  if(d !== "d") throw new TypeError('Something went wrong')
  return [a, b, c, d]
}

const parser = parse(data, {
  on_record,
  delimiter: ',',
  skip_lines_with_error: true,
})

console.log(parser.read())
console.log(parser.read())

Result
Exception is thrown within the on_record call, error breaks stream and throws a global exception which results in a crash.
Expected Result
Exception to be caught and emitted as a skip event with a CSVError.
Additional context
My plan is to use the on_record handler to transform the parsed CSV data before it gets emitted out as a data event, however my concern is that if an unexpected error occurs within the on_record call this will result in a crash and  loss of the entire file/stream.

Current behavior is to interprent the error of skip_lines_with_error as a parsing error, an invalid CSV data, not a user thrown error. If you wish to skip this line, return null instead of throwing an error.

Errors are now caught and correctly handled. Note, the original error is sent and not a CSVError as you requested. A new patch version is available.

Hey @wdavidw,
Thanks for the super quick change for this, however I have just tweaked the playground I setup to run v4.8.6 but still seeing exceptions occurring and not going the skip handler
const { PassThrough } = require('stream')
const parse = require('csv-parse')
const assert = require('assert')

const on_record = ([a, b, c, d]) => {
  if(d !== "d") throw new TypeError('Something went wrong')
  return [a, b, c, d]
}

const parser = parse("a,b,c,d\na,b,c,-", {
  on_record,
  delimiter: ',',
  skip_lines_with_error: true,
})

parser.on('skip', () => console.log('Skipped'))
parser.on('error', () => console.log('Error'))
parser.on('data', (d) => console.log('Data', d))
parser.on('end', () => console.log('End'))

Playground: https://repl.it/@RobertPitt/CSV-Paese


Use a central directory for remaining tests that create a temporary file

Finishing up the work from #2393, use std.file.deleteme instead.  This is useful on Android, which doesn't allow writing to arbitrary directories and doesn't have a global /tmp directory, so you can easily hack in the app's absolute path in one place.
I had to remove the pure attribute for std.stdio.testFilename to have it use std.file.deleteme, but that didn't matter anywhere in the tests.  I don't think we guarantee such test helper functions will stay the same for users, so I doubt dropping the attribute will matter.

LGTM

LGTM

Auto-merge toggled on


5/17 Code for Boston Agenda

Previous issue: #475
Whole group:
Doing the whole-group CFB project update tonight will be @erose.
Team flow:
Start off the night with this stuff.

Review milestones - help improve it!
Quick sync up, who's working on what, who doesn't know what they're working on right now?
Process thoughts and possible experiments.

Assign yourselves to issues on Github that you're working on! (Permissions problems? Can be solved by adding people as collaborators. Tie into https://github.com/studentinsights/studentinsights/pull/462?)

Coordinating new folks tonight: @erose.

See CONTRIBUTING.md.

This is my personal list of things I'd like to get done tonight.

Get @awalker4's pull request, https://github.com/studentinsights/studentinsights/pull/464 merged. In order to do this we need to work on a spec first.
Work on spec for AcademicSummary.js with @BWieber! (https://github.com/studentinsights/studentinsights/issues/458)
Get react-modal or a similar modal dialog box framework working.

Note to @kevinrobinson from @erose and myself: we've discovered that the react-modal component seems to require window.react when our version of React sets window.React. We think we can work around this by adding a window.react = window.React somewhere after React is included but before the other React libs. Is there a better way to solve this? Also, next time you're here we might want to pick your brain about the Rails asset pipeline :)


Range operator produce nested array

Hello
return [1,2] AQL result is: (which is expected)
[
  [
    1,
    2
  ]
]

but return [1..2] will return:
[
  [
    [
      1,
      2
    ]
  ]
]

Shouldn't the both queries had the same result?

sorry i shouldn't use bracket with Range operators


Latitude property is misspelt on Request

Lattitude currently has three t's but should have two - i.e. Latitude

@tonysurma Since I'll be adding a migration for #813 I'll include this issue in that PR later.


Updates to expensive recipes for 0.15

Not really an issue.. but with 0.15 out, are there any plans on updating the recipe ratios when expensive recipe difficulty are selected with map generation?? For example, with expensive recipes green circuits require 10 copper wire and 2 iron (making a 5/1 ratio copper wire/green circuit)..
Would love to keep using this with 0.15 expensive recipes..

Just came here looking for this myself. Wanted to see what me and my friends were getting into with our new marathon game.

There's a fork that teraflop posted on the reddit that contains the 0.15 new normal recipes. reddit thread: https://www.reddit.com/r/factorio/comments/67ma95/i_updated_doomeers_webbased_planning_tool_for_015/
as I commented on that thread, the expensive recipes could be done by modifying the recipes.ml file, but I haven't had the time to do so..

Quick and dirty, a branch for expensive recipes can be done. Adding a switch for normal/expensive would be longer.

I merged s3bash's pull request. If there are missing or wrong recipes, please tell me! Or better yet, make a pull request :D Thanks again!


[BUG] Variable is sometime resetting between scenes in build

Describe the bug
In build, Unity unloads unused assets when switching scenes, so sometimes AtomsVariable are reseting, sometimes not.
Example:
SceneA is referencing a FloatVariable
SceneB is referencing the same FloatVariable
SceneC is not referencing the Variable.
So if you go A -> B, the asset is in used, Unity don't unload it, the value is kept
If you go A -> C, the asset is considered unused, Unity unload it
To Reproduce
Reproduction repository: https://github.com/bguyl/unity-so-bug
Build download: https://github.com/bguyl/unity-so-bug/releases/tag/0.0.1
Expected behavior
Variable and event behaviors should be consistent between editor play and build.
Either values should reset at each scene change or the should not reset at all.
Screenshots

Environment

Unity Version: 2020.2.1
Version of Unity Atoms: 4.4.2

Additional context
If you mark SOs as Addressable, the issue is fixed as the asset lifetime is manually managed by you.
Two others way to enforce consistency behavior is:

Mark SOs as DontUnloadUnusedAsset to make them act like a "Global" variable
Reset values on ActiveSceneChanged

Hello, i would like to know is your scene marked as Addressables?
and generally is Addressables installed in your project?

Hello, i would like to know is your scene marked as Addressables?
and generally is Addressables installed in your project?

👋
No, in the reproduction project, addressable is not installed.
When I talk about addressable that's because I use the package in another project.
Do you have something in mind?

👋
No, in the reproduction project, addressable is not installed.
When I talk about addressable that's because I use the package in another project.
Do you have something in mind?

So from what I know, scene will be converted to another form of addressable even if you don't mark it as one.
When addressable is built, if a scriptable object is referenced in the scene, it will grab that asset and pack it together.
If there's 2 scenes referencing a scriptable object, each scenes will have an independent reference of the asset, because it sad never know that those assets referenced are the same assets.
That is why when you mark the scriptable object as an addressable, the referenced asset is maintained between scenes.
So this is the behaviour with scriptable object in general and asset bundles in general - these are the basic behaviour in Unity, not specifically happening to Addressable or Unity Atom.
I don't have a direct answer for your issue, but I hope I gave you some insights. 🙂

So from what I know, scene will be converted to another form of addressable even if you don't mark it as one.
When addressable is built, if a scriptable object is referenced in the scene, it will grab that asset and pack it together.
If there's 2 scenes referencing a scriptable object, each scenes will have an independent reference of the asset, because it sad never know that those assets referenced are the same assets.
That is why when you mark the scriptable object as an addressable, the referenced asset is maintained between scenes.
So this is the behaviour with scriptable object in general and asset bundles in general - these are the basic behaviour in Unity, not specifically happening to Addressable or Unity Atom.
I don't have a direct answer for your issue, but I hope I gave you some insights. 🙂

Thank you, that's very interesting !
I test to add the Addressable package without marking anything as addressable, and it doesn't change a thing.
The second test I made was to mark the 3 scenes as addressable and have a ManagerScene that load the scenes A, B or C.
And indeed, each scene has it own variable instance.

The build: https://github.com/bguyl/unity-so-bug/releases/tag/0.0.2
I know that all of this is classic Unity behavior, but I thought it could be great to have documentation on this topic somewhere or to implement a workaround in Atoms as it could be confusing

Thank you, that's very interesting !
I test to add the Addressable package without marking anything as addressable, and it doesn't change a thing.
The second test I made was to mark the 3 scenes as addressable and have a ManagerScene that load the scenes A, B or C.
And indeed, each scene has it own variable instance.

The build: https://github.com/bguyl/unity-so-bug/releases/tag/0.0.2
I know that all of this is classic Unity behavior, but I thought it could be great to have documentation on this topic somewhere or to implement a workaround in Atoms as it could be confusing

Thanks for the details!
I was blocked by this problem for days... when I finally pinpointed it to multiple instances of the same atoms and came to check the known issues, this was the most recent one just waiting for me :-)
I was afraid I would have to add each atom as an Addressable Asset and try not to forget when creating a new one, but instead you can place them all in one folder and make that folder AA, problem solved very cleanly!

Thanks for the details!
I was blocked by this problem for days... when I finally pinpointed it to multiple instances of the same atoms and came to check the known issues, this was the most recent one just waiting for me :-)
I was afraid I would have to add each atom as an Addressable Asset and try not to forget when creating a new one, but instead you can place them all in one folder and make that folder AA, problem solved very cleanly!

Thanks for the details!
I was blocked by this problem for days... when I finally pinpointed it to multiple instances of the same atoms and came to check the known issues, this was the most recent one just waiting for me :-)
I was afraid I would have to add each atom as an Addressable Asset and try not to forget when creating a new one, but instead you can place them all in one folder and make that folder AA, problem solved very cleanly!

This is exactly what i did to my project too! Placing them all in one folder and add that folder into Addressables. :)

Thanks for the details!
I was blocked by this problem for days... when I finally pinpointed it to multiple instances of the same atoms and came to check the known issues, this was the most recent one just waiting for me :-)
I was afraid I would have to add each atom as an Addressable Asset and try not to forget when creating a new one, but instead you can place them all in one folder and make that folder AA, problem solved very cleanly!

This is exactly what i did to my project too! Placing them all in one folder and add that folder into Addressables. :)


Installation failed on Ubuntu 22.04 with Julia 1.10.0-beta3

OS: Ubuntu 22.04
g++: 11.4.0
Julia version: 1.10.0-beta3
Error message after running make install:
[  7%] Building CXX object CMakeFiles/jluna.dir/.src/exceptions.cpp.o
[ 15%] Building CXX object CMakeFiles/jluna.dir/.src/unsafe_utilities.cpp.o
/home/lyx/opt/jluna/.src/unsafe_utilities.cpp: In function ‘void jluna::unsafe::resize_array(jluna::unsafe::Array*, uint64_t)’:
/home/lyx/opt/jluna/.src/unsafe_utilities.cpp:197:41: error: cannot convert ‘_jl_value_t*’ to ‘jl_datatype_t*’ {aka ‘_jl_datatype_t*’}
  197 |             auto* tuple = jl_new_struct(tuple_type, jl_box_int64(static_cast<Int64>(one_d)));
      |                                         ^~~~~~~~~~
      |                                         |
      |                                         _jl_value_t*
In file included from /home/lyx/opt/jluna/include/julia_wrapper.hpp:13,
                 from /home/lyx/opt/jluna/include/typedefs.hpp:8,
                 from /home/lyx/opt/jluna/include/unsafe_utilities.hpp:8,
                 from /home/lyx/opt/jluna/.src/unsafe_utilities.cpp:6:
/home/lyx/.julia/juliaup/julia-1.10.0-beta3+0.x64.linux.gnu/include/julia/julia.h:1573:55: note:   initializing argument 1 of ‘jl_value_t* jl_new_struct(jl_datatype_t*, ...)’
 1573 | JL_DLLEXPORT jl_value_t *jl_new_struct(jl_datatype_t *type, ...);
      |                                        ~~~~~~~~~~~~~~~^~~~
In file included from /home/lyx/.julia/juliaup/julia-1.10.0-beta3+0.x64.linux.gnu/include/julia/julia.h:80,
                 from /home/lyx/opt/jluna/include/julia_wrapper.hpp:13,
                 from /home/lyx/opt/jluna/include/typedefs.hpp:8,
                 from /home/lyx/opt/jluna/include/unsafe_utilities.hpp:8,
                 from /home/lyx/opt/jluna/.src/unsafe_utilities.cpp:6:
/home/lyx/.julia/juliaup/julia-1.10.0-beta3+0.x64.linux.gnu/include/julia/julia_threads.h:245:12: note: class type ‘_jl_value_t’ is incomplete
  245 |     struct _jl_value_t *sig_exception;
      |            ^~~~~~~~~~~
/home/lyx/opt/jluna/.src/unsafe_utilities.cpp: In function ‘void jluna::unsafe::resize_array(jluna::unsafe::Array*, uint64_t, uint64_t)’:
/home/lyx/opt/jluna/.src/unsafe_utilities.cpp:222:41: error: cannot convert ‘_jl_value_t*’ to ‘jl_datatype_t*’ {aka ‘_jl_datatype_t*’}
  222 |             auto* tuple = jl_new_struct(tuple_type, jl_box_int64(static_cast<Int64>(one_d)), jl_box_int64(static_cast<Int64>(two_d)));
      |                                         ^~~~~~~~~~
      |                                         |
      |                                         _jl_value_t*
In file included from /home/lyx/opt/jluna/include/julia_wrapper.hpp:13,
                 from /home/lyx/opt/jluna/include/typedefs.hpp:8,
                 from /home/lyx/opt/jluna/include/unsafe_utilities.hpp:8,
                 from /home/lyx/opt/jluna/.src/unsafe_utilities.cpp:6:
/home/lyx/.julia/juliaup/julia-1.10.0-beta3+0.x64.linux.gnu/include/julia/julia.h:1573:55: note:   initializing argument 1 of ‘jl_value_t* jl_new_struct(jl_datatype_t*, ...)’
 1573 | JL_DLLEXPORT jl_value_t *jl_new_struct(jl_datatype_t *type, ...);
      |                                        ~~~~~~~~~~~~~~~^~~~
In file included from /home/lyx/.julia/juliaup/julia-1.10.0-beta3+0.x64.linux.gnu/include/julia/julia.h:80,
                 from /home/lyx/opt/jluna/include/julia_wrapper.hpp:13,
                 from /home/lyx/opt/jluna/include/typedefs.hpp:8,
                 from /home/lyx/opt/jluna/include/unsafe_utilities.hpp:8,
                 from /home/lyx/opt/jluna/.src/unsafe_utilities.cpp:6:
/home/lyx/.julia/juliaup/julia-1.10.0-beta3+0.x64.linux.gnu/include/julia/julia_threads.h:245:12: note: class type ‘_jl_value_t’ is incomplete
  245 |     struct _jl_value_t *sig_exception;
      |            ^~~~~~~~~~~
make[2]: *** [CMakeFiles/jluna.dir/build.make:90: CMakeFiles/jluna.dir/.src/unsafe_utilities.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:113: CMakeFiles/jluna.dir/all] Error 2
make: *** [Makefile:166: all] Error 2

It seems the issue is caused by the Julia version 1.10.0-beta3. With Julia 1.9.3, I can install Jluna successfully on the same machine.

Sorry for the late replay, I can reproduce this, I assume they changed the API under us so it should be a simple fix.

Fixed by https://github.com/Clemapfel/jluna/commit/3eac77882ed696a0c51cc5d088f62a584c85a3a5, this will go live with version 1.1.

Thanks! I am looking forward to using Jluna with Julia 1.10.


TorchIO has been supporting Python 3.12 since 0.19.6

Comment:
TorchIO has been supporting Python 3.12 since 0.19.6:
https://github.com/fepegar/torchio/blob/13407ab7f6d562a0953fa6642ce198f297129589/setup.cfg#L53
Details can be found at fepegar/torchio#1114
The Python version requirement should be updated to <3.13 since Python 3.12 has become the default version on many platforms:
https://github.com/conda-forge/torchio-feedstock/blob/fa631419ee8e19af25ef23390e0c3845cc8c4193/recipe/meta.yaml#L21-L36

Good point - thank you!

Good point - thank you!

Thanks! I have created a PR, please review it later.


Test results filter on result value

Related Issue or Background Info

https://app.zenhub.com/workspaces/simplereport-better-data-team-605b43d51ea9f700119a48ee/issues/cdcgov/prime-simplereport/1027

Changes Proposed

added optional result parameter to getTestResults(Count) gql queries
changed TestEventRepository to accept Specifications, using those for result searching now
added generation of static metamodel classes (in build.gradle) for type-safe query building for Specifications
added tests

Checklist for Author and Reviewer

[ ] Run this branch locally
[ ] Confirm you can run the testResults() and testResultsCount() queries in the graphQL playground locally using different combinations of facilityId, patientId, and result value

UI

[ ] Any changes to the UI/UX are approved by design
[ ] Any new or updated content (e.g. error messages) are approved by design

Testing

[ ] Includes a summary of what a code reviewer should verify

Changes are Backwards Compatible

[ ] Database changes are submitted as a separate PR

[ ] Any new tables that do not contain PII are accompanied by a GRANT SELECT to the no-PHI user
[ ] Any changes to tables that have custom no-PHI views are accompanied by changes to those views

[ ] GraphQL schema changes are backward compatible with older version of the front-end

Security

[ ] Changes with security implications have been approved by a security engineer (changes to  authentication, encryption, handling of PII, etc.)
[ ] Any dependencies introduced have been vetted and discussed

Cloud

[ ] DevOps team has been notified if PR requires ops support
[ ] If there are changes that cannot be tested locally, this has been deployed to our Azure test environment for verification

I guess @nullflux @nickclyde please ignore the Sonar failure? It is lying to you about 0% test coverage!! How mean.

Just ran locally, everything looks to be working! Awesome work on this. Only note I have, is that there maybe shouldn't be any margin between the select box and the "Results" filter, so that it's more in line with the patient name search box:

this should be fixed now @nickclyde !

https://user-images.githubusercontent.com/76444566/120844046-dbb95d00-c53c-11eb-8df1-03115de9766b.mov
@hmyers-cms @katiealoisi how's this look?

also the screen recording above has a small bug where the test results count doesn't update when selecting a result filter, but i've now updated it so that no matter what filters are applied, the results count reflects whatever is displayed on screen, accounting for filters.

The person search is a bit wonky full stop but that's definitely a separate design effort to improve. Right now, the full-width I think makes it a bit hard to tell what you're supposed to do next, because the "filter" button is so far away from the search box itself. Can we limit the max-width of that dropdown?

@hmyers-cms how about now?
https://user-images.githubusercontent.com/76444566/120859988-fdbdda00-c552-11eb-8c0e-64e06ea66e71.mov


Finish Redis protocol support

Complete redis protocol support RESP2 (redis protocol repo todos etc)

Closing as complete


Can you migrate a sample into this repository?

Now, demo is divided into two different repositories.
So I think there are some disadvantages when developing features.

Package versions are different in demo and release.
Inefficient because verification and development are separate repositories.

I would like to consider including this sample in this repository.
https://github.com/richardtallent/vue-scrolling-table-sample

Hi @kimulaco,
I could have them in the same repo, but this wouldn't solve the problems you describe. In a combined repo, each would still be required to have their own folder structure, package.json, webpack and Vue configuration, etc.
I would love for Vue to have a Project Template for a Vue component (or set of components) with an accompanying example app, but there's not a canonical way to do this.
I have run across the same issue in my other components. I do try to keep the sample apps and components aligned in version numbers, but I do forget to bump the sample apps sometimes.
When I'm working on one of the components, I clone both repos into "sibling" folders and import them separately into Vue UI. I switch the import statement for the sample app to use the component in the relative sibling folder rather than the online npm version. This allows my component changes to hot-reload in the sample app. When I'm done, I switch back to referring to the component as loaded through the dependencies. This also allows me to confirm that the new version has been published on npm correctly.
You can see this in the sample app here:
https://github.com/richardtallent/vue-scrolling-table-sample/blob/master/src/App.vue#L83
It's not a perfect solution. For example, if my eslint or prettier preferences aren't identical, I end up with the code throwing linter warnings or getting reformatted. When Vue UI provides a better solution for component and library authors to include an example app, I'll probably be the first to switch!

Thank you for answer!
I understand your idea!
This is an example, but using Vue CLI v3.x seems to be somewhat easier.
https://cli.vuejs.org/guide/build-targets.html


Fix code scanning alert - Clear-text logging of sensitive information

Tracking issue for:

[x] https://github.com/cortip/isomera/security/code-scanning/1

Fixed in another commit


Fix build for forked repos

This removes a few random whitespaces, and fixes the build script for docker so it works on forked repos. It'll still obviously be the same for this repo too.

Dunno how I missed this, sorry! Thanks for the contribution


Gas should be metered separately for internal contract calls.

Inside InternalTransactionExecutor when we make a Create or TransferFunds contract call, we should be creating a new GasMeter instance and injecting that into the virtual machine.
Currently it always uses the meter from the initial contract call.
This way smart contract developers can declare how much gas they want to allow to be used in a internal call / create.

Building on this: The default amount of gas to be supplied should start at (BaseFee * 2) - 1 as this allows some computation inside contract calls, without allowing the contract to start calling more contracts (or to re-enter).
Discuss this part before implementing.


Changed default log level for StopContainer event from INFO to WARN

@tushargosavi Kindly review and merge

Add a JIRA id in the commit message. you can use the old Jira as this is a minor change.

@tushargosavi Used old JIRA ID in commit message


Unable to open config page during API connection failure

Hi,
Last weekend I had to do some maintenance in the fuse box at home so I cut the power for some hours, so everything was offline. Homey, WiFi, SolarEdge.
When I brought everything back online I noticed a red exclamation mark on the SolarEdge tile in the Homey App. On opening the app it showed an error on the API. When I opened the settings screen (gear icon) I got a black screen and was unable to get out of it. Only way was to forcefully close the Homey app. I've made a screencast:
https://user-images.githubusercontent.com/2821657/126325966-adf4cd1f-22b6-45ec-bff3-ffaa9c4b1641.mp4
After restarting the SolarEdge app within the Homey settings it was able to reach the API again and all was back to normal. So this problem is solved but maybe something to try and reproduce for future updates.

This is normal as the timestamp is not correct anymore, rebooting the app is the solution ;-)
unable to get out of the black screen is a homey thing.

This is normal as the timestamp is not correct anymore, rebooting the app is the solution ;-)
unable to get out of the black screen is a homey thing.

I posted this after contacting homey.
This is what they said:

Het issue dat je ervaart ligt helaas niet aan de Homey software, maar aan de community app die je gebruikt. Het beste wat je kunt doen is even contact leggen met de ontwikkelaar van deze community app zodat hij op de hoogte is dat zijn app niet goed werkt.

I guess we’re in a ‘from the closet to the wall’ situation now 😬

Haha, I don’t think it is a major problem.
I crashed the app a lot during development, and I could always close the window.
To avoid issues in the future, you could make a flow that when Homey boots, it restarts the Solaredge app after +/- 15 minutes. (When your internet, network and inverter is running again after a power failure)


new placeholders

trello ticket
added required option
added "this page" / "multiple pages" designations
added logic for "single line" placeholders (less than 50px tall)
updated docs
updated styling

OLD PLACEHOLDERS:

NEW PLACEHOLDERS:

👍


ASGI zipfly on Django 4.2 first packages the file(s) to memory, then sends to client

I've tried multiple times to make this work but I've ran out of options. So this is more like a cry for help.
Kinda sucks, since the same application this happens on, would benefit from having websockets enabled.

Without seeing the code, I really couldn't tell you what the problem is.
Having said that, the easy solution is to separate the downloads from the app itself.

The same code works on both WSGI and ASGI, and it produces the zipfile as expected. The only difference is that in ASGI mode it first creates the zipfile in memory and then sends it to client.
Tried fooling around with this today. It seems to boil down to Django's StreamingHttpResponse not accepting synchronous iterators when in ASGI mode. It throws this warning:
/usr/local/lib/python3.11/site-packages/django/http/response.py:534: Warning: StreamingHttpResponse must consume synchronous iterators in order to serve them asynchronously. Use an asynchronous iterator instead.

Spent a few hours trying to splash async/await and @sync_to_async decorators all around the code, but alas; no success yet.


GeoSPARQL Functions

Implement GeoSPARQL Functions
Table 1 – Simple Features Topological Relations in the GeoSPARQL Specification.

[x] geof:sfEquals
[x] geof:sfDisjoint
[x] geof:sfIntersects
[x] geof:sfTouchs
[x] geof:sfWithin
[x] geof:sfContains
[x] geof:sfOverlaps
[x] geof:sfCrosses

As an example, geof:sfWithin has already been implemented in https://github.com/ontop/ontop/commit/7f7c08b79d637041bc01e335f36a9cec68e4ab5d.
The test case is implemented in the https://github.com/ontop/ontop/commit/9db841c7af7285e3f1d5e798bf90a7881fa72033 in the class GeoSPARQLTest using H2GIS.

SRID
According to GeoSPARQL standard:
Req 11 The URI http://www.opengis.net/def/crs/OGC/1.3/CRS84 shall be assumed as the spatial reference system for geo:wktLiterals that do not specify.
E.g.
"Point(46.49067 11.33982)"^^geo:wktLiteral
and
"<http://www.opengis.net/def/crs/OGC/1.3/CRS84> Point(46.49067, 11.33982)"^^geo:wktLiteral
are equivalent
Examples in other SRIDs

"<http://www.opengis.net/def/crs/EPSG/0/4326> POINT (11.33982 46.49067)"^^geo:wktLiteral
"<http://www.opengis.net/def/crs/EPSG/0/3044> POINT (668682.853 5122639.964)^^geo:wktLiteral

Proposal:

support WKT literals as constants with SRIDs inside (as above)
support WKT literals from Literals  template with constant SRIDs inside, e.g,
“{geom}”^^geo:wktLiteral -> default CRS84
"<http://www.opengis.net/def/crs/EPSG/0/3044> {geom}”^^geo:wktLiteral
"<http://www.opengis.net/def/crs/EPSG/0/3044> POINT ({x} {y})^^geo:wktLiteral
NOT support storing the literals with SRID URLs in one DB column, e.g.
“{geom}”^^geo:wktLiteral with geom is column with a value “<http://www.opengis.net/def/crs/EPSG/0/3044> POINT (668682.853 5122639.964)”.
Such values do not make sense in GIS systems.
NB: PostGIS uses a non-standard EWKT syntax storing SRID AsEWKT “SRID=3044;POINT (668682.853 5122639.964)”

Table 2 – Egenhofer Topological Relations in the GeoSPARQL Specification.
For some of the the following functions the same equivalent representations from the simple feature topological relations are used. They have been noted where it is the case:

[x] geof:ehCovers
[x] geof:ehCoveredBy
[x] geof:ehEquals <-> geof:sfEquals
[x] geof:ehDisjoint <-> geof:sfDisjoint
[x] geof:ehMeet <-> geof:sfTouches
[x] geof:ehContains <-> geof:sfContains

For the following relations, the respective simple relation functions are currently being used, however the equivalence might not be 100% and is being reviewed.

[ ] geof:ehOverlap <-> geof:sfOverlaps
[ ] geof:ehInside <-> geof:sfWithin

The DE-9IM model specified for each function in http://schemas.opengis.net/geosparql/1.0/geosparql_vocab_all.rdf was used to determine equivalence where applicable.

Table 3 – RCC8 Topological Relations in the GeoSPARQL Specification.
Egenhofer relations only apply to 2-dimensional geometries (e.g. polygons).

[x] geof:rcc8eq <--> geof:sfEquals
[x] geof:rcc8dc <--> geof:sfDisjoint
[x] geof:rcc8ec <--> geof:sfTouches
[x] geof:rcc8po <--> geof:sfOverlaps
[x] geof:rcc8tppi <--> geof:sfContains
[x] geof:rcc8tpp <--> geof:sfWithin
[ ] geof:rcc8ntpp
[ ] geof:rcc8ntppi

geof:rcc8ntppi --> The respective function ST_CONTAINSPROPERLY in PostGIS implementes this function. There is no implementation for H2GIS.
No SQL implementation of geof:rcc8ntpp, unlike geof:sfContains, the geometries' boundaries should not touch.

Is geof:rcc8ntpp not the same as geof:sfWithin?
9-Intersections are equivalent to RCC-8 for a suitably restricted class of spatial objects (e.g., polygons). However, one has to be a bit more careful even with "2-dimensional geometries" - there are bizarre objects there.

Table 4 – Non-Topological Functions in the GeoSPARQL Specification.
[x] geof:distance
[ ] geof:boundary
[x] geof:buffer
[ ] geof:convexHull
[ ] geof:difference
[ ] geof:envelope
[x] geof:intersection
[ ] geof:getSRID
[ ] geof:symDifference
[ ] geof:union
[ ] geof:relate
Except for buffer, distance and relate, all other functions take geometries as inputs and have fixed arity.


Nuxeo collection harvesting fails at "content harvesting" stage (UC Merced/Cooperative Extension complex image objects)

Running full harvests of 2 Nuxeo collections (UC Merced/Cooperative Extension) -- both are collections of complex image objects (scanned images of documents), getting the same error for both failing at content harvesting stage. There are no specific logs to point to from Airflow (also not sure how to link to the specific DAG runs):
1] Error: fails at "map page" stage
Registry #23389
Rikolti mapper: nuxeo.nuxeo
Airflow Run ID: manual__2024-04-10T18:08:52+00:00
https://7a8067cb-3b99-477e-a883-7e311175a9b4.c3.us-west-2.airflow.amazonaws.com/dags/harvest_collection/graph?execution_date=2024-04-10+18%3A08%3A52%2B00%3A00
2] Error: fails at "content harvesting" stage
Registry #27851
Rikolti mapper: nuxeo.nuxeo
Airflow Run ID: manual__2024-04-10T18:09:15+00:00
https://7a8067cb-3b99-477e-a883-7e311175a9b4.c3.us-west-2.airflow.amazonaws.com/dags/harvest_collection/graph?execution_date=2024-04-10+18%3A09%3A15%2B00%3A00

@aturner @christinklez I'd like to try running one of these large collections from scratch (and not try rerunning any failed mapped tasks) so that I can examine the logs and hone in on what exactly is causing the workers to be killed. OK if I kick off 23389 via the Airflow UI?

@aturner @christinklez I'm figured it's OK for me to go ahead and kick this off. Leaving this note here just FYI :)

Thank you, @barbarahui! That sounds like a great plan!

OK, I think my updates have fixed things. I ran 27143 as a test and it's successfully on stage (although the validate_collection task failed -- but I think I remember that's a known issue?)
Unfortunately you will need rerun any large collections that previously failed from scratch, i.e. fetch the metadata again. This is because the changes I made batch up the data differently starting with the fetching.
You should now see a max of 50 []mapped_tasks for the map_page[] and content_harvest[] steps. For very large collections, it can take a long time to run things, particularly the content harvest task. Airflow will try polling ECS for up to 4 hours before giving up and saying that the task has failed. So hopefully that's enough time for even very large collections to run. If not, let me know.

Related PRs:
https://github.com/ucldc/rikolti/pull/875
https://github.com/ucldc/rikolti/pull/877

I increased the time that Airflow will wait for content harvester tasks finish to 24 hours. I reran monster collection 23389 and it took 10 hours for a couple of the tasks to run (way longer than I expected; I guess they must have a lot of huge media files). I'm not sure how big our biggest collection is, but hopefully 24 hours is enough time.
PR: https://github.com/ucldc/rikolti/pull/878


global_variables_initializer errors

Hey,
import tensorflow as tf
import gpflow

kernel = gpflow.kernels.Matern32(1)

# ... some other code that create variables and depends on kernel ...

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())

the code above throws the following error
InvalidArgumentError (see above for traceback): You must feed a value for placeholder tensor '0/Matern32/lengthscales/initial_unconstrained_value' with dtype double
	 [[Node: 0/Matern32/lengthscales/initial_unconstrained_value = Placeholder[dtype=DT_DOUBLE, shape=[], _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

I know that I can fix this error simply by running passing the kernel.initializable_feeds feed_dict, but is there a way to avoid creating placeholders for the initial values in the first place? It's a bit annoying to have to go back and collect all the feeds from different GPs in a larger system.
Other than that I like how much easier it is to deal with gpflow 1.0 as a subpart of a graph👍
Thanks,
Felix

Hi @befelix! Thank you for feedback! Unfortunately, placeholders are part of parameters and provide other important features in GPflow framework. You can derive from Parametrized class and create a model-like collection of kernels, parameters, other models under one roof with nice access to full list of initializable_feeds. Something like:
class MyGPCollection(gpflow.Parameterized):
    def __init__(self):
        self.kernel = gpflow.kernels.Matern32(1)
        ...

gp_collection = MyGPCollection()
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer(), feed_dict=gp_collection.initializable_feeds)

But, even better way is putting all tensors under parameterized object and override initializables property, therefore you could just call collection.initialize() within new session or at default gpflow's tensorflow session.


Make optimizers importable

I know optimizers.py is not part of autograd's core function, but it is often needed when using autograd especially for training neural networks. Therefore, why not make autograd a bit easier to use by making optimizers.py importable just like util? Currently, it seems one has to download it to local folder to import it.

I found climin is a natural complementary tool for autograd. Therefore, I no longer think my above suggestion is necessary if one uses climin.


Add method accessible_from to test security group rules

Added accessible_from? to security groups and RDS instances.
The semantics for security groups are:

There is an ingress rule where the CIDR matches, or is a superset of the supplied CIDR range.

The semantics for RDS instances is slightly different:

Is there a security group which satisfies the CIDR
If the CIDR is for a public subnet, is the database publicly accessible

There are additional semantics that I consider out of scope, since they should probably be tested separately:

If the CIDR is an internal address, is it within the CIDR of the associated VPC (if the RDS instance is in a VPC)
VPC Peering CIDRs

@SaltwaterC This PR is ready to go.
I'll give you the opportunity to review before I merge.

@SaltwaterC Would you be able to push a release?

Sure. I'll need to tag the release and bump the version. I'll do it once I get to my office as I don't have the gem environment configured on this box.

Release pushed. Thank you for taking your contributions.