Add a playground group in the group list

Right now the group list is the landing page. Probably a lot of people come to karrot.world and just want to try it out. It would be great if they didn't have to create an account, or if they wouldn't randomly join open groups. My proposal is the following:

[ ] Take one of the unused groups, rename it to 'Playground' and write a login in the public description.
[ ] Display the playground group as the very first in the group list.
[ ] Add a yellow frame around it to make it even more salient.

Any resistance to me tackling this?

I really like the idea! A couple of questions though.

How are we keeping the playground user from entering other groups?
Is renaming the group sufficient or should it be translatable?
Are we keeping people from changing the password for the demo group?

Therefore, I am unsure if it wouldn't be easier to just deploy a demo version of karrot to demo.karrot.world and add a demo user (or multiple ones) there, since the demo user cannot mess with the production database then. The database could also be copied and replaced to the original then from time to time. If you feel confident to prevent the demo user from changing non-demo data though, I am happy either way.

Oh, very good points you have there, @D0nPiano!

I didn't think of that, you're right. But actually, users who just wan't to play around will probably not make the effort of joining a whole lot of groups, I guess. If they wanted to, they could always make an account for themselves and just hop into any unprotected group they want to, so this is not really a big deal for me... (Also, as soon as we have the autoremoval of inactive users (#868) this won't be grave anyways...)
Having it translatable would be fancy, for sure! But how would this work? Normally group names are not translatable, so how can we just make this one group name different?
Sounds reasonable, yes. Okay, looks like this group wull have unique features in more than one way...

I'd love to have the playground directly in production, so that the demo experience is closest to what actual users have. That's also why I'm more in favor of this solution than linking to dev.karrot.world and giving out foo for testing purposes. I think it could confuse some users...

Yeah, makes sense. I just usually think I can't destroy anything when I am on a demo page, wherefore I just click around randomly. Especially because you are anonymous as a demo user and don't even have to provide an email address for signing in.
I think having it translatable is just as complicated as adding a yellow frame. The playground group should probably be names "Playground", and in vue.js the store could then filter out the group and do something like
if (group.name === 'Playground') {
   group.name === $t('PLAYGROUND.TITLE')
}

Furhermore, the demo user should also be special (also a special name? or another special attribute?). This could open possibilities like disabling buttons if it is the current user
<q-btn disable="user.name === 'Playground User'">Join Group</q-btn>

...it's not a perfectly safe solution (since not done in backend) but would still be good enough I think. And I can see your point why it would be nicer in production.

Good idea!
Just an additional thought: if we want to have custom behavior for playground group at some point, it could make sense to have it marked in the database. I'm thinking of two initial pieces here: a new field that marks a group as playground and new database migration that creates the testgroup.

A script could reset the playground group + users to a certain state periodically. So, kick the playground users out of any other groups they wandered into, kick any other random users out of the playground group, and set the data back sensible defaults.

With "playground user", do you mean predefined users with predefined password? I would find it nice if I can join the playground group with my personal user account and not getting kicked out of my other groups ;)

So I renamed group 16 to PLAYGROUND and wrote encouraging things in the description. How about a login? I guess a special demo user would need to be created in the backend..?

I'll add a group status field, replacing the newly created active field. It can be set to values active, inactive or playground, because they seem mutually exclusive. I discussed it with @nicksellen and it seems to make most sense to do it like this right now.

I'll implement the playground group into the frontend. It will

offer the user to join the playground group after signup
show the playground group at the bottom of the group list if the user is logged in
don't show the playground group if the user is logged out (because it's not a real foodsaving group)
show a persistent alert if the user is on a playground group page
disable a few options, e.g. to change the group password

Backend-related changes:

it would be good to disable email notifications by default for the playground group
disable the group password option permanently

#960 is ready for review :)

Playground group is in place, let's see how it turns out. I would close this issue now. We can open another one for the playground user if we want.


Vulkan Instance allocation

We should be able to allocate a VkInstance and pass it into the vkCreateInstance() method the same as is currently done with VkInstanceCreateInfo instead of it requiering a PointerBuffer as it currently does.
. . .
VkInstance vkInstance = VkInstance.malloc();
VK10.vkCreateInstance(vkInstanceCreateInfo, null, vkInstance);
. . .

No, sorry, that would effectively make VkInstance objects mutable.
It would be possible to make vkCreateInstance return a VkInstance, but it already returns a value (the error code). An option would be to swap the VkInstance parameter and the return value like:
IntBuffer pi = ...
VkInstance inst = vkCreateInstance(inst_info, null, pi);
int err = pi.get(0);

but I don't think that's any better.

How would it make it more mutable that it already is? Currently you create a PointerBuffer outside of the object then just pass it in later so why not make it so that the object itself (or a static method) creates the pointer?

So I should have added this but It would be more like a builder method than the current allocation method. so
VkInstance inst = VkInstance.malloc(inst_info);

where the method would automatically create an address for the object and then that address would be used in the vkCreateInstance() method instead of a PointerBuffer so almost nothing would change overall but it would make it look cleaner and less work for an end user. It would also be a bit more like the C code.

How would it make it more mutable that it already is?

Because between VkInstance foo = new VkInstance() and vkCreateInstance(..., foo), you have an object that is invalid and cannot be used for anything. If vkCreateInstance fails, you still have that object. If vkCreateInstance succeeds, it would be possible to call it again and modify the existing object. It's bad design.

So I should have added this but It would be more like a builder method than the current allocation method.

Best I can do is a factory method that calls vkCreateInstance for you. Something like (in the VkInstance class):
public static VkInstance create(VkInstanceCreateInfo pCreateInfo, VkAllocationCallbacks pAllocator) {
	try ( MemoryStack stack = stackPush() ) {
		PointerBuffer pp = stack.mallocPointer(1);
		int err = vkCreateInstance(pCreateInfo, pAllocator, pp);
		if ( err != VK_SUCCESS )
			throw new RuntimeException("...");

		return new VkInstance(pp.get(0), pCreateInfo);
	}
}

and similarly for VkDevice. Error handling is still a problem though.

Because between VkInstance foo = new VkInstance() and vkCreateInstance(..., foo), you have an object that is invalid and cannot be used for anything. If vkCreateInstance fails, you still have that object. If vkCreateInstance succeeds, it would be possible to call it again and modify the existing object. It's bad design.

Thats how the current vulkan method works anyway and you can just keep using the same pointer over and over and over as well.

Best I can do is a factory method that calls vkCreateInstance for you. Something like (in the VkInstance class)

It would probably be better to leave it up to the user but that is similar to what I currently do.

This is something which a layered library on top of LWJGL should be responsible for. Not LWJGL itself. IMO, LWJGL should be as lean and mean and close to native code as possible. Any fancy OO and convenience stuff like easy to use factory methods and ways to translate vk error codes to Java exceptions should be in another library.

public static int vkCreateInstance(VkInstanceCreateInfo pCreateInfo, VkAllocationCallbacks pAllocator, VkInstance pInstance) {
		MemoryStack stack = stackGet(); int stackPointer = stack.getPointer();
		try {
		PointerBuffer pointer = stack.callocPointer(1);
		int code = nvkCreateInstance(pCreateInfo.address(), pAllocator == null ? NULL : pAllocator.address(), memAddress(pointer));
		if(code != VK_SUCCESS) {
			return code;
		} else {
			pInstance.setData(pointer.get(0), pCreateInfo);
		}
		} finally {
			stack.setPointer(stackPointer);
		}
	}

though setData could just always run but the way I did it is if there is an error code it just returns that code and VkInstance would be basically invalid anyway.
And the VkInstance code would be something like this.
. . .
/** Wraps a Vulkan instance handle. */
public class VkInstance extends DispatchableHandle {

	public VkInstance() {

	}

	/*
	 * Allows the vkCreateInstance() method to set the proper data to the container
	 * without allow subclasses to access the method.
	 */
	void setData(long address, VkInstanceCreateInfo ci) {
		this.address = address;
		capabilities = getInstanceCapabilities(address, ci);
	}

	. . .
}

Though other container type objects used by the Vulkan bindings would also need to be modified as well. Also in the above samples I made it look a lot like what the code would look like if done currently though it might look better if the VkInstance was allocated and the handle placed there. I think it is just a pointer value so normally we would represent it as a long but the VkCapabilities are also stored in the Instance as well as in several other objects used by Vulkan.
now to go off my own topic but maybe something similar could be done with OpenAL to get rid of us needing to set the ALCapabilities when we change the context.
Also there are a few unneeded null checks in the generated code that uses stack.UTF8() as that method has a built in null check that it does first thing. (OpenAL code)

Also there are a few unneeded null checks in the generated code that uses stack.UTF8() as that method has a built in null check that it does first thing.

Thanks, fixed with d20f8f1a1bd0819be961ff2a2667bc823e8b8f6e.


Dev: Warn about adding GitHub CLI to .devcontainer

Preinstall the GitHub CLI ('gh') for Codespaces and Dev Containers

@tylerbutler - Thank you for catching.  I'll add the warning for now and follow up.

@tylerbutler - Thanks for finishing the job for me.  :-)


"native-audio" sample doesn't show all buttons

This is what the IDE shows about the layout of the sample app "native-audio" (here) :

And yet this is what I see when I run it (tested on Pixel 2 with Android P DP3 ) :

Yeah there is a bug that does not play the http: linked file, so some of those buttons disabled: the internal bug was not proceeding, I have not get my time to trace that one yet. It has been for a long while...  if you could, please move to oboe, support is higher there

@ggfan What's "oboe" ?

oboe is a new open source lib, wrapping Android AAudio  and OpenSLES; easier to handle.

I see. When was oboe talked about ? Are there lectures of Google about it?
BTW, the buttons aren't disabled. They are gone.

Oboe has been there for a while ( got to be more than a year ).  Docs are on its github page, Don's talk is here. It is pretty active there on the github, if you want some features, feel free to notify there.

Thank you!

The video about it mostly talk about reducing latency.
Also, they said they will publish Midi sample, but I don't think they did:
https://youtu.be/Pjgfje52Yv0?t=2461
Anyway, since what I'd like is just about audio recording, I've written about it here:
https://github.com/google/oboe/issues/135

yeah, that is the good place to file it. Don is pretty quick on that repo. When I get the time, I will also look at that ( it is interesting to look at the containers )

So this repository is in deprecated mode?
If AAudio is new, why not put a sample to it, here too?
It requires NDK too, no?
Sorry for all the questions. I'm very new to this, and I haven't even written in C/C++ for ages (Mostly Java and Kotlin). Even when I did have the time to mess around with C/C++ on Android, I made just a small repository for it (here), which for some reason became a bit popular.

For Audio, yeah, like to fade out from this repo.
AAudio was released with Android O, it is C interface, so it is exposed via NDK.  AAudio does have its own place. Starting with Oboe for your new projects would be good for the long run as we are pretty much heavy ( if not all ) on Oboe. New Audio API samples will not coming here for sure, and probably video and graphics new samples not coming here either

You have some cool new features/libraries to show on video/graphics too?

Vulkan 1.1 enabled video texture, could be a good helper for apps using camera input as texture. That is coming out with Android P with android hardware buffer extension.
Congratulations to your repo!

Sadly Vulkan has serious issues:

Not supported on most Android versions
A very new API, meaning less places to read about and even less samples, and yet requires to learn from scratch.

I don't get why Google doesn't just publish a gaming library that will support all Android versions, and use Vulkan/OpenGL under the hood. Make it available for all, using Java/Kotlin.
The current situation only makes third party engines more attractive to developers than what Google offers, because why waste time and effort on something that others will work hard to generalize...
Only die hard developers would develop straight in Vulkan.
That's how I feel from OpenGL 2. I learned OpenGL 1 in the past, but it has gone so different on version 2 that I had a hard time to study it on my own. Not because I need it, but because I wanted, as I was curious.
But the more this field changes with more and more APIs and languages, the less I think it's attractive to develop straight in it, instead of using engines that are dedicated to use the best of each...
Thank you about the repository. I have others too (here), and I also make some tiny spare time apps (here)

Thank you!


PDFPageProxy.render.promise does not return/resolve in XR immersive mode

Attach (recommended) or Link to PDF file here:
Configuration:

Oculus Quest 2 browser
Oculus Quest OS
PDF.js version: pdfjs-dist 2.7.570
Is a browser extension: NA

Steps to reproduce the problem:
const url = "https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf";
const pdf = await PDFJS.getDocument(url).promise;
const page = await this.currentPdf.getPage(1);
const context = canvas.getContext("2d") as CanvasRenderingContext2D;
console.log("Before render");
await page.render({ canvasContext: context, viewport: viewport }).promise
console.log("After render");

Run above in immersive mode on a VR device
Observe that "After render" is never printed to the console. As such, the render() functionality cannot be used whilst in immersive mode

What is the expected behavior? (add screenshot)
PDGPageProxy.render.promise resolves
What went wrong? (add screenshot)
The promise never resolves and the code awaits infinitely.
Link to a viewer (if hosted on a site other than mozilla.github.io/pdf.js or as Firefox/Chrome extension):

Run above in immersive mode on a VR device

First of all, please note that there's no mention in either the README or the FAQ about the PDF.js library supporting e.g. VR devices.
Secondly, your code-snippet doesn't look like valid JavaScript (e.g. the use of as looks strange) and it also cannot be run-as; please see https://github.com/mozilla/pdf.js/blob/master/.github/CONTRIBUTING.md (emphasis mine):

If you are developing a custom solution, first check the examples at https://github.com/mozilla/pdf.js#learning and search existing issues. If this does not help, please prepare a short well-documented example that demonstrates the problem and make it accessible online on your website, JS Bin, GitHub, etc. before opening a new issue or contacting us in the Matrix room -- keep in mind that just code snippets won't help us troubleshoot the problem.

@Snuffleupagus, the code snippet is written in Typescript.
I understand there isn't support for VR devices at present. Is this on the roadmap?


Fix CI

We need to decide on what our CI process will be, what tools we will employ and at what exact steps these need to execute.
For reference, currently, the auto-generated workflows are:

[x] On each and every pull request (or push to main), the whole build workflow is executed, including verifying the plugin (takes ~4m) and qodana (first and last time took ~15m, perhaps next time it will take less, need to check) - fixed in #14

[x] #16

[x] #17

@JChrist I took the liberty of breaking this down into further sub-tasks.

closing as all sub-tasks completed.


Grab min max from OWM daily like precip?

What happened?
See title
How to reproduce
See title
Relevant log output
No response
Which version are you running?
Beta
Diagnostics file

[X] I have attached a diagnostics file

Additional information
No response

When selecting OWM as the source for a sensor there is no option for using average, first, last, maximum, median, minimum, sum.  If the auto-update is collecting these values every hour thru the day, which of those options are you using behind the scenes to calculate the bucket size?
It seems odd that there is no sensor selection for Temperature, but there are sensors for Maximum Temperature and Minimum Temperature.   I'm not clear why there are also no sensor selectors for Dew Point Max and Min, Humidity Max and Min, Solar Rad Max and Min, etc...

I see OWM as the set it and forget it option. So no aggregates needed, we handle it. Averages for everything except precip which is last and max and min temp. Makes sense, right? Max temp and min temp exist as separate sensors because it's needs for the calc. Other max min values are not used in calcs. Hope this helps.
Regarding what this issue is for: I will not change this unless I get feedback from you in the other issue that what it does now is not right.

That does make sense.  I'll let mine sit for another day to look at the comparison between the 2.  I think my issue has been that I've been uninstalling, reinstalling, updating to new betas, changing sensor options, emptying buckets, updating and calculating so many times that it never has had a chance to actually collect a full days worth of values.  So at 23:00 it just grabs a current set of values, and at that time of day the average doesn't mean much!

I really appreciate all your efforts. I am looking forward to what it looks like on your end. We can do the experiment outlined in this issue in case we're not happy with what we're seeing. I don't think it changes much if your collect data more than once a day. However, if you set it to collecting data once a day, then it does make a difference if we collect max/min temperature from "current" or from "daily". in "current" there is no max/min really - just temperature, so if you collect only once these will be the same. In "daily" there is a max / min temperature, but historical they have not been very accurate compared to collecting multiple "current" values and calculating max/min temperature from there. Anyway, let's see.

It's not a full days worth of data, but this is what I see this morning.   I'm using the "Daily" Sensor for my min and max temps, and since that first value is from 22:55, the max temp is from the previous day and is much higher than the next 8 hours reported.  Same thing for the min when using the Daily sensor as a source.
I'm wondering if instead of using the Daily min/max as a sensor I should just use my local temp sensor (using the last value).  That would more closely match the hourly intervals logged by OWM and make a better comparison for the average?
Reset Buckets: 22:50
Auto Update: 22:55
Auto Calculate: 23:00
OWM Max:      My Sensors Max:
21.9      31.88
20.63      22.38
20.5      22.38
19.78		22.38
19.47		22.38
19.29		22.38
18.89		22.38
18.69		22.38
18.78		22.38
20.02		22.38

OWM Min:	My Sensors Min:
21.9		18.27
20.63		21.77
20.5		21.72
19.78		21.11
19.47		20.72
19.29		19.77
18.89		19.77
18.69		19.77
18.78		19.77
20.02		19.72

It's not a full days worth of data, but this is what I see this morning. I'm using the "Daily" Sensor for my min and max temps, and since that first value is from 22:55, the max temp is from the previous day and is much higher than the next 8 hours reported. Same thing for the min when using the Daily sensor as a source.
I'm wondering if instead of using the Daily min/max as a sensor I should just use my local temp sensor (using the last value). That would more closely match the hourly intervals logged by OWM and make a better comparison for the average?
Reset Buckets: 22:50 Auto Update: 22:55 Auto Calculate: 23:00
OWM Max: My Sensors Max: 21.9 31.88 20.63 22.38 20.5 22.38 19.78 22.38 19.47 22.38 19.29 22.38 18.89 22.38 18.69 22.38 18.78 22.38 20.02 22.38
OWM Min: My Sensors Min: 21.9 18.27 20.63 21.77 20.5 21.72 19.78 21.11 19.47 20.72 19.29 19.77 18.89 19.77 18.69 19.77 18.78 19.77 20.02 19.72

I think this is fair comparison. I think we should try with OWM getting min/max from daily data and do compile the same comparisons. If that's closer I think we should adopt that approach.

so I switched the behavior around in beta11: min and max temp are now retrieved from daily OWM data instead of current. This is better for the situation that you're only updating once a day and should do us no harm for multiple updates a day, but that remains to be seen. Please wait for a day and do the same comparison. Curious to the result!

This is not correct approach. For example, if I want to update after midnight, or not on daily basis, calculations will use only current day OW?
In case you using own aggregates you should care they are in sync with zone updates, i.e. reset them after each calculation via automations (for DailySensor uncheck automatic reset at 0:00, call daily_sensor_max_temp_reset at 23:05, or after smart_irrigation calculation event? ).
On other hand, it would be easier to debug with built in aggregates if zone sensors have attributes for weather sensors.

This should work:
`automation:

alias: "Reset Daily Sensors for Zone MySensors"
trigger:

platform: state
entity_id: sensor.zone_ow
not_from:

"unknown"
"unavailable"
not_to:
null # avoid only attributes changes
"0" # not reset when bucket resets
action:

service: daily.daily_max_my_temperature_reset
service: daily.daily_min_my_temperature_reset`

I am not sure what this has to do with the experiment of getting max temp data somewhere. Let me clarify what I am doing here.
OWM provides current and daily weather data. Current is the current weather when the API is called and does not provide a MAX or MIN temperature, because, well, it's current. The daily weather contains the weatherdata for that day up until the point the API is called. So if you call it at 23:00 it will contain the MAX/MIN temperature for that day.
The V1 of this integration used the daily info exclusively and that's why it updated only daily, at 23:00. Made sense at the time.
However, in V2 there is more flexibility. People can update once a day, at a time they choose or more often. And here is the problem. Here are some scenarios to explain the issue and why I want to see if getting max/min temp from daily is better:

Update frequency
(First) update
Temp from current
Temp from Daily
Winner

1/day
2300
Max temp = Min temp = Temp @2300
Max temp = max temp for day until 2300, min temp = min temp for day until 2300
Daily

1/day
1000
Max temp = Min temp = Temp @1000
Max temp = max temp for day until 1000, min temp = min temp for day until 1000
Daily

1/hour
0100
Max temp = max(temp @0100, 0200...). Min temp = min(temp @0100,0200,...)
Max temp = max(max temp for day until @0100,0200...), Min temp = min(temp for day until @0100,0200...)
Daily

See, in all cases getting it from daily is better.

Lets choose your second example: daily update at 10:00 with following weather conditions by OW:

previous day: sunny afternoon, cloudy evening with minimum temp 15C at 05:00, max temp 28C at 14:00, and 24C at 10:00
today till 10:00: cloudy, minimum temp 20C at 05:00, max temp 24C at 10:00

I assume when using "OW current", calculations will be based on min temp of 20C and max temp of 28C.
And I guess that using "OW daily max, min": 20C (or 15C?) and 24C correspondently.

I think there's a problem with your calculations when converting Fahrenheit to Celsius using the Daily Sensor for the Temperature Max value.  I've attached my smart_irrigation.storage.  Our daily high has been around 85-87 degrees, and my Daily Sensor matches that closely.  The OWM reported max temp values are mostly in the range of 29.6 - 30.8, which also matches the Daily Sensor Max after conversion.  But all of the max temp values reported in the .storage file for My Sensors using the Daily Sensor Max are in the range of 18.3 - 21.9.  That results in an approx 15 degree Fahrenheit difference.  The min values all look normal, so just an issue with the max.
smart_irrigation.storage.txt

So behind latest beta the implementation OW sensors is static, no history was recorded? I've not get it.

I don't think I understand what you mean and I don't know to explain this in another way than what I have already said. Id suggest: do as @dlindnegm is doing: test it and report back on comparisons with your own sensors. I am doing the same and I see better results with the updated behavior than before but the differences for me were not so profound as for @dlindnegm.

I think there's a problem with your calculations when converting Fahrenheit to Celsius using the Daily Sensor for the Temperature Max value. I've attached my smart_irrigation.storage. Our daily high has been around 85-87 degrees, and my Daily Sensor matches that closely. The OWM reported max temp values are mostly in the range of 29.6 - 30.8, which also matches the Daily Sensor Max after conversion. But all of the max temp values reported in the .storage file for My Sensors using the Daily Sensor Max are in the range of 18.3 - 21.9. That results in an approx 15 degree Fahrenheit difference. The min values all look normal, so just an issue with the max.
smart_irrigation.storage.txt

So your saying that your max temperature is not stored correctly? Just that one? All others are fine? That's weird but maybe somewhere the max temp gets a double conversion or so, not sure.

So behind latest beta the implementation OW sensors is static, no history was recorded? I've not get it.

I don't think I understand what you mean and I don't know to explain this in another way than what I have already said. Id suggest: do as @dlindnegm is doing: test it and report back on comparisons with your own sensors. I am doing the same and I see better results with the updated behavior than before but the differences for me were not so profound as for @dlindnegm.

If you collect data only once then the latest beta provides better temperature estimations that the previous ones. Why? Because max and min temp now come from daily and will provide different results where previously we got them from current, which if you collect data only once will result in min and max being the same, because current only provides the current temperature.

What about custom sensors? Their data is recorded to calculate aggregates? Why not make OW API requests at same intervals, record received current temperature then calculate max, min, mean same way?

I think there's a problem with your calculations when converting Fahrenheit to Celsius using the Daily Sensor for the Temperature Max value. I've attached my smart_irrigation.storage. Our daily high has been around 85-87 degrees, and my Daily Sensor matches that closely. The OWM reported max temp values are mostly in the range of 29.6 - 30.8, which also matches the Daily Sensor Max after conversion. But all of the max temp values reported in the .storage file for My Sensors using the Daily Sensor Max are in the range of 18.3 - 21.9. That results in an approx 15 degree Fahrenheit difference. The min values all look normal, so just an issue with the max.
smart_irrigation.storage.txt

So your saying that your max temperature is not stored correctly? Just that one? All others are fine? That's weird but maybe somewhere the max temp gets a double conversion or so, not sure.

Not sure, but it seems there's a major difference between the max values from midnight up until about 8:00 am.  Maybe something to do with when each of those sensors rolls over or resets?

What about custom sensors? Their data is recorded to calculate aggregates? Why not make OW API requests at same intervals, record received current temperature then calculate max, min, mean same way?

that is exactly what is happening here... my point is that if there was only 1 call to the OWM API per day then the min and max would be be same unless we pull it from Daily. Now, if you do multiple calls both Daily and Current will return multiple values, both of which are aggregated anyway, so in that sense there is no difference.

I think there's a problem with your calculations when converting Fahrenheit to Celsius using the Daily Sensor for the Temperature Max value. I've attached my smart_irrigation.storage. Our daily high has been around 85-87 degrees, and my Daily Sensor matches that closely. The OWM reported max temp values are mostly in the range of 29.6 - 30.8, which also matches the Daily Sensor Max after conversion. But all of the max temp values reported in the .storage file for My Sensors using the Daily Sensor Max are in the range of 18.3 - 21.9. That results in an approx 15 degree Fahrenheit difference. The min values all look normal, so just an issue with the max.
smart_irrigation.storage.txt

So your saying that your max temperature is not stored correctly? Just that one? All others are fine? That's weird but maybe somewhere the max temp gets a double conversion or so, not sure.

Not sure, but it seems there's a major difference between the max values from midnight up until about 8:00 am. Maybe something to do with when each of those sensors rolls over or resets?

so there seems to be a bug here. the max should never go down and the min should never go up. Not sure what is happening at 10:00  for MAX and 7:00 for MIN. What timezone are you in again? Maybe the OWM API resets what "daily" means at one of those points? That could potentially explain one of them.
What did those charts look like before, in the previous beta?

As I understood data collection for aggregates starts from last calculate. Then what for is setting in General "First Update at ..."?

Eastern Time Zone.  I never knew where to look for this data before, so this is the first I've dug into it.  Here's my Daily Sensor Max from HA.  Since it resets at midnight it makes sense, but the OWM max doesn't seem to reset?  I can change my Daily Sensors to not auto reset if you think that helps isolate this.

As I understood data collection for aggregates starts from last calculate. Then what for is setting in General "First Update at ..."?

I don't know where you got that but, no. The schedule dictates when to collect data. The first update is just when the schedule starts. So if you say: collect data every hour, you need some ability to influence if you wanted to collect at the hour and start at midnight for example. That's what that is for.

Eastern Time Zone. I never knew where to look for this data before, so this is the first I've dug into it. Here's my Daily Sensor Max from HA. Since it resets at midnight it makes sense, but the OWM max doesn't seem to reset? I can change my Daily Sensors to not auto reset if you think that helps isolate this.

no I think the Daily thing does what it is supposed to do - after all, I wrote it ;). I am just thinking that OWM daily data follows some weird 'reset' schedule that we can't control.

https://openweather.my.site.com/s/article/min-and-max-temperature-in-current-weather-api-and-forecast-api-2019-10-24-21-47-24
Are you grabbing from the current/historical api or the daily weather forecast api?  That article indicates that the current/historical api is a snapshot in time, whereas the forecast api is a daily number.  I also thought that the current weather api was free, but the forecast data was only available from a paid subscription?  It seems like the OWM min and max are using a sliding 24 hour window with no resets.  I'll take another look at this tonight after I have a full 24 hours of data.

so we're calling the OneCall API: https://openweathermap.org/api/one-call-api, so the "current" and "daily" are both from there. I take the the first "daily" to calculate the max/min and precip because that's the expected / forecasted values for today. So it's not a true representation of what happened today, but it's a good approximation. Calling the daily forecast multiple times a day has shown the for precip/max and min temperature to change so it's not static. That's what you're seeing here. I still cannot really explain why the min / max change for you and why that happens at two different moments in time.

As I understood data collection for aggregates starts from last calculate. Then what for is setting in General "First Update at ..."?

I don't know where you got that but, no. The schedule dictates when to collect data. The first update is just when the schedule starts. So if you say: collect data every hour, you need some ability to influence if you wanted to collect at the hour and start at midnight for example. That's what that is for.

I got it from your words: https://github.com/jeroenterheerdt/HAsmartirrigation/issues/235#issuecomment-1706984974

Nowhere I said that data is automatically collected after calculate. That is your interpretation of my words, which, well eh, is wrong. The calculation and updates are only related in the sense that weatherdata collected by updates is used by calculate to do it's work and once it is successful (ie calculated a duration) it will wipe that data. Nowhere in these sentences is it implied that calculate does somehow influence the collection of data or start it again, because, well it doesn't.

Ok, collecting weather data is a continuous process. New update calculates aggregates in a time slice since previous update till the new update. Is this correct?

Daily OW means daily, from 0:00 till 23:59 o'clock in your time zone, they are not like
start: "{{ now() - timedelta( hours = 24 ) }}"
end: "{{ now() }}"
OneCall subscription is free up to 1000 calls per hour.

Daily OW means daily, from 0:00 till 23:59 o'clock in your time zone, they are not like start: "{{ now() - timedelta( hours = 24 ) }}" end: "{{ now() }}"
OneCall subscription is free up to 1000 calls per hour.

Right, so that is perfect then, right? You want the values for the day. The more I think about it it sounds like you are trying to go for a scenario that I don't see / understand and have not designed the integration for.

Ok, collecting weather data is a continuous process. New update calculates aggregates in a time slice since previous update till the new update. Is this correct? Sorry, I mean new Calculate Zone Call manual or automatic (not update). It uses only data with timestamp after previous Calculate?

You are over complicating things. I have explained multiple times how this works. The timestamp is something you made up. We don't do anything with timestamping. It is as simple as I have explained it multiple times now. Nothing more, nothing less.

Speaking of timestamps...
How much trouble would it be to add a timestamp into the .storage file for each update?  That would make it easier to put those values into a spreadsheet to see what's happening.  Something like this:
      },
      {
        **"Time": 9/1/2023 23:55,**
        "Dewpoint": 21.222222222222225,
        "Humidity": 99.0,
        "Maximum Temperature": 21.722222222222218,
        "Minimum Temperature": 20.500000000000004,
        "Precipitation": 0.0,
        "Pressure":<PHONE_NUMBER>,
        "Solar Radiation": 3.8741760000000003,
        "Temperature": 21.5,
        "Windspeed": 0.0
      },

Sorry, I missed were I can read how zone bucket calculates in timeline? Just as simple as "When you call Calculate service, the collected sensor data with interval X from time A to time B is analysed to get predefined aggregates ..." and "for OW sensors it is different (or same)?"
If it is different why we compare results? "Just choose one and trust it!"
I give up, really can't help you. Better spend time to write own python script for pyETO and automations.

I set up a new Daily Sensor Test that doesn't reset at midnight and the screenshots show how the daily max can be quite different depending on what time of day you grab it.  You have a much better understanding of how this all works than I, but it seems to me that the OWM max temp is a sliding 24 hour average, but the default Daily Sensor max is not, since it resets at midnight.  By removing the midnight reset it looks more like the OWM max.
And in my mind, that could account for why when I have 2 zones that are set up exactly the same, with the exception of one using OWM max and one using the Daily Sensor max, the OWM bucket can be much larger depending on when the calculation was performed.  Sometimes the bucket using OWM max is -4.x and the bucket using the Daily Sensor max is -.1.  The OWM max seems to always be the actual maximum temp for the day, but the Daily Sensor Max (with a midnight reset) can show the maximum temp as much as 10 degrees F less that the actual high temp, and thus it's calculating a much smaller bucket/runtime.

Speaking of timestamps...
How much trouble would it be to add a timestamp into the .storage file for each update? That would make it easier to put those values into a spreadsheet to see what's happening. Something like this:
      },
      {
        **"Time": 9/1/2023 23:55,**
        "Dewpoint": 21.222222222222225,
        "Humidity": 99.0,
        "Maximum Temperature": 21.722222222222218,
        "Minimum Temperature": 20.500000000000004,
        "Precipitation": 0.0,
        "Pressure":<PHONE_NUMBER>,
        "Solar Radiation": 3.8741760000000003,
        "Temperature": 21.5,
        "Windspeed": 0.0
      },

I might be able to do that.

Sorry, I missed were I can read how zone bucket calculates in timeline? Just as simple as "When you call Calculate service, the collected sensor data with interval X from time A to time B is analysed to get predefined aggregates ..." and "for OW sensors it is different (or same)?" If it is different why we compare results? "Just choose one and trust it!"
I give up, really can't help you. Better spend time to write own python script for pyETO and automations.

And here I was thinking I was helping you. I really don't understand what you want to hear. I tried. It's very simple. There is no relation between calculate and any interval. You call it and all data that is on the sensor group is used and after a successful calculate that is wiped. How that data got there is irrelevant.

I set up a new Daily Sensor Test that doesn't reset at midnight and the screenshots show how the daily max can be quite different depending on what time of day you grab it. You have a much better understanding of how this all works than I, but it seems to me that the OWM max temp is a sliding 24 hour average, but the default Daily Sensor max is not, since it resets at midnight. By removing the midnight reset it looks more like the OWM max.
And in my mind, that could account for why when I have 2 zones that are set up exactly the same, with the exception of one using OWM max and one using the Daily Sensor max, the OWM bucket can be much larger depending on when the calculation was performed. Sometimes the bucket using OWM max is -4.x and the bucket using the Daily Sensor max is -.1. The OWM max seems to always be the actual maximum temp for the day, but the Daily Sensor Max (with a midnight reset) can show the maximum temp as much as 10 degrees F less that the actual high temp, and thus it's calculating a much smaller bucket/runtime.

I think you might be on to something here, but I don't know a good solution yet. We can't chadnge what OWM daily provides. So if we wanted to go back to OWM current how do we account for the situation that someone does only one update a day and gets wonky results? Remove that option for the integration and force an update at least twice a day? Or just leave it like this and document that you should not do this? Ideas?

Sorry, I missed were I can read how zone bucket calculates in timeline? Just as simple as "When you call Calculate service, the collected sensor data with interval X from time A to time B is analysed to get predefined aggregates ..." and "for OW sensors it is different (or same)?" If it is different why we compare results? "Just choose one and trust it!"
I give up, really can't help you. Better spend time to write own python script for pyETO and automations.

And here I was thinking I was helping you. I really don't understand what you want to hear. I tried. It's very simple. There is no relation between calculate and any interval. You call it and all data that is on the sensor group is used and after a successful calculate that is wiped. How that data got there is irrelevant.

Ye, I do not understand what is behind the GUI.
 Is it normal?

Sorry, I missed were I can read how zone bucket calculates in timeline? Just as simple as "When you call Calculate service, the collected sensor data with interval X from time A to time B is analysed to get predefined aggregates ..." and "for OW sensors it is different (or same)?" If it is different why we compare results? "Just choose one and trust it!"
I give up, really can't help you. Better spend time to write own python script for pyETO and automations.

And here I was thinking I was helping you. I really don't understand what you want to hear. I tried. It's very simple. There is no relation between calculate and any interval. You call it and all data that is on the sensor group is used and after a successful calculate that is wiped. How that data got there is irrelevant.

Ye, I do not understand what is behind the GUI.  Is it normal?

Yes, of course. You asked us to calculate multiple times, so that's what we do.  That way the integration is agnostic to what you use to run your irrigation.

I think you might be on to something here, but I don't know a good solution yet. We can't chadnge what OWM daily provides. So if we wanted to go back to OWM current how do we account for the situation that someone does only one update a day and gets wonky results? Remove that option for the integration and force an update at least twice a day? Or just leave it like this and document that you should not do this? Ideas?

Can you provide the syntax for the API call that includes the OWM min and max?  I'd like to compare those results to the Daily Sensor numbers.
I'm thinking that the best approach might be to go back to grabbing the OWM current temps every hour, and drop any reference to min and max temp sensors altogether.  Even the Daily Sensor Max without a midnight reset can have dramatically different results depending on when it's calculated.
So folks would only need to provide a current temp sensor of their own, or use the OWM current temp.  You continue to log those values every minute, hour, day (user selectable), and at every calculate you grab whatever is logged into the .storage file and select the max and min values for each.  I don't see any issue with the number of API calls, since the free tier provides 60/minute and 1 million calls per month.
Leave the default like it is - update every hour.  If someone wants to use only one update per day that's their choice, and you can document the issues with frequency of updates and their effect on the results.  I think you could drop the option to update for less than one hour, as I don't see that improving accuracy more than once per hour.  So drop the option to update every x minutes.  And if someone only wanted to update every 5 days! they would have 120 current temp entries logged, and you just grab the max and min value from those.  Would that make everybody happy?

that is certainly an option. I can make a new version with this proposal. It will take me a bit of rewiring, but it sounds like that's a good middle ground. thanks!

Sorry, I missed were I can read how zone bucket calculates in timeline? Just as simple as "When you call Calculate service, the collected sensor data with interval X from time A to time B is analysed to get predefined aggregates ..." and "for OW sensors it is different (or same)?" If it is different why we compare results? "Just choose one and trust it!"
I give up, really can't help you. Better spend time to write own python script for pyETO and automations.

And here I was thinking I was helping you. I really don't understand what you want to hear. I tried. It's very simple. There is no relation between calculate and any interval. You call it and all data that is on the sensor group is used and after a successful calculate that is wiped. How that data got there is irrelevant.

Ye, I do not understand what is behind the GUI.  Is it normal?

Yes, of course. You asked us to calculate multiple times, so that's what we do.  That way the integration is agnostic to what you use to run your irrigation.

Ok, I got more:

At 18:00 I called Update All Zones, Calculate All Zones and got buckets -5,0 mm for OW zone and -5,1 mm for MySensors. Then "Irrigate" 5 mm and Reset All Buckets.
Now I wish "Irrigate" again but only ETO past two hours, did same Update, Calculate and got - 2,7 mm for OW and 5,0 mm for MySensors.  First, evidently this is not past 2 hours ETO, and second why such difference between OW and MySensors this time?

that is a good question. However, please open a separate issue and provide your diagnostic file as requested. without it is impossible to explain.

I think you might be on to something here, but I don't know a good solution yet. We can't chadnge what OWM daily provides. So if we wanted to go back to OWM current how do we account for the situation that someone does only one update a day and gets wonky results? Remove that option for the integration and force an update at least twice a day? Or just leave it like this and document that you should not do this? Ideas?

Can you provide the syntax for the API call that includes the OWM min and max? I'd like to compare those results to the Daily Sensor numbers.
I'm thinking that the best approach might be to go back to grabbing the OWM current temps every hour, and drop any reference to min and max temp sensors altogether. Even the Daily Sensor Max without a midnight reset can have dramatically different results depending on when it's calculated.
So folks would only need to provide a current temp sensor of their own, or use the OWM current temp. You continue to log those values every minute, hour, day (user selectable), and at every calculate you grab whatever is logged into the .storage file and select the max and min values for each. I don't see any issue with the number of API calls, since the free tier provides 60/minute and 1 million calls per month.
Leave the default like it is - update every hour. If someone wants to use only one update per day that's their choice, and you can document the issues with frequency of updates and their effect on the results. I think you could drop the option to update for less than one hour, as I don't see that improving accuracy more than once per hour. So drop the option to update every x minutes. And if someone only wanted to update every 5 days! they would have 120 current temp entries logged, and you just grab the max and min value from those. Would that make everybody happy?

If you have own solar radiance sensor (that is more accurate than estimate from temp), you will not get accurate mean with 1 hour update because clouds can cover the sun for minutes. May be better to move Update' and Auto Calculate constants into Sensor Group objects.

config_entry-smart_irrigation-6e18a9ad38f8ac00ef4285672ad17839.json-6.txt

that is certainly an option. I can make a new version with this proposal. It will take me a bit of rewiring, but it sounds like that's a good middle ground. thanks!

implemented this in beta12! Also added the timestamps as you suggested: retrieval time on weatherdata in the storage file, last updated data time on the sensor group and last calculation time is stored on zone.
Let me know if you see improvements now!

Installed beta12 and will watch it tomorrow.  One thing I just noticed with beta11 - Before I upgraded to beta12 I manually copied off the smart_irrigation.storage file and noticed that there were only 11 entries for weather data.  My setting for Automatic weather data update is set to 23:50, but only 11 entries means that something cleared all of the weather data sometime before or at 11:50???  There's no calculation data in the file, so it doesn't seem like that's what reset it.   Adding the time stamps will show if it continues tomorrow.

Thanks, the timestamps should help. Note that the timestamps are not just on weatherdata but also on mappings and zones.

I don't see any "retrieved" timestamps for each update for my sensors, just the OWM updates.  It does show the data_last_updated at the end, just not for each hourly update.
OWM:
{             "Windspeed": 0.628278903141073,             "Pressure": 1016,             "Humidity": 72,             "Temperature": 18.53,             "Dewpoint": 13.39,             "Precipitation": 4.1,             "retrieved": "2023-09-09T08:50:01.177685"           }         ],         "data_last_updated": "2023-09-09T08:50:01.180663" }
My Sensors:
{             "Dewpoint": 19.222222222222218,             "Humidity": 93.0,             "Precipitation": 0.0,             "Pressure":<PHONE_NUMBER>,             "Solar Radiation": 5.040576000000001,             "Temperature": 20.500000000000004,             "Windspeed": 0.0           }         ],         "data_last_updated": "2023-09-09T08:50:01.183119"       }

I'm apologise for my flood without knowing how exactly FAO-56 Penman-Monteith method works.
After close look into pyETO I found that in case we "Estimate Solar radiation" only daily calculations are correct.

I'm apologise for my flood without knowing how exactly FAO-56 Penman-Monteith method works. After close look into pyETO I found that in case we "Estimate Solar radiation" only daily calculations are correct.
Also about OWM OneCall data: what "daily" data are you discussing? Per [https://openweathermap.org/api/one-call-3#current] there is currently observed data and forecast: minutely, hourly and daily.
No max, min, mean in current:
"current": {
        "dt":<PHONE_NUMBER>,
        "sunrise":<PHONE_NUMBER>,
        "sunset":<PHONE_NUMBER>,
        "temp": 15.3,
        "feels_like": 13.98,
        "pressure": 1017,
        "humidity": 42,
        "dew_point": 2.48,
        "uvi": 0.29,
        "clouds": 100,
        "visibility": 10000,
        "wind_speed": 5.9,
        "wind_deg": 16,
        "wind_gust": 9.58,
        "weather": [
            {
                "id": 804,
                "main": "Clouds",
                "description": "overcast clouds",
                "icon": "04d"
            }
        ]
    },

OWM data updated every 10 minutes and suggests collecting data as frequently as possible (for correct aggregates in our case). Free 1000 calls in a day is 1 minute limit if you don't use other OWA integrations. I think 5-10 minutes is optimum.

When I say daily I mean daily.

I don't see any "retrieved" timestamps for each update for my sensors, just the OWM updates. It does show the data_last_updated at the end, just not for each hourly update.
OWM: { "Windspeed": 0.628278903141073, "Pressure": 1016, "Humidity": 72, "Temperature": 18.53, "Dewpoint": 13.39, "Precipitation": 4.1, "retrieved": "2023-09-09T08:50:01.177685" } ], "data_last_updated": "2023-09-09T08:50:01.180663" }  My Sensors: { "Dewpoint": 19.222222222222218, "Humidity": 93.0, "Precipitation": 0.0, "Pressure":<PHONE_NUMBER>, "Solar Radiation": 5.040576000000001, "Temperature": 20.500000000000004, "Windspeed": 0.0 } ], "data_last_updated": "2023-09-09T08:50:01.183119" }

That's a mistake on my side, apparently added the retrieved only on OWM data.

Like that?:
"daily": [
    {
        "dt":<PHONE_NUMBER>,
        "sunrise":<PHONE_NUMBER>,
        "sunset":<PHONE_NUMBER>,
        "moonrise": 0,
        "moonset":<PHONE_NUMBER>,
        "moon_phase": 0.83,
        "summary": "Expect a day of partly cloudy with rain",
        "temp": {
            "day": 14.25,
            "min": 10.1,
            "max": 15.7,
            "night": 10.1,
            "eve": 15.18,
            "morn": 11.07
        },
        "feels_like": {
            "day": 12.93,
            "night": 8.6,
            "eve": 13.88,
            "morn": 10.3
        },
        "pressure": 1016,
        "humidity": 46,
        "dew_point": 2.63,
        "wind_speed": 7.74,
        "wind_deg": 12,
        "wind_gust": 9.78,
        "weather": [
            {
                "id": 500,
                "main": "Rain",
                "description": "light rain",
                "icon": "10d"
            }
        ],
        "clouds": 56,
        "pop": 0.44,
        "rain": 0.66,
        "uvi": 2.14
    },

In addition to no "retrieved" timestamps being written to my local sensor updates, I just noticed that the "Automatic weather data update" doesn't survive a restart of HA.  All of my hourly updates stopped after I restarted HA this morning for the 2023.9.1 Core update.  So I won't have any hourly data until tonight's calculation is performed.  Hopefully there's a way to restart that after a HA restart?

Like that?:
"daily": [

yes.

In addition to no "retrieved" timestamps being written to my local sensor updates, I just noticed that the "Automatic weather data update" doesn't survive a restart of HA. All of my hourly updates stopped after I restarted HA this morning for the 2023.9.1 Core update. So I won't have any hourly data until tonight's calculation is performed. Hopefully there's a way to restart that after a HA restart?

you will have hourly data, it has been collected all day for you by the time the calc is performed. I will look into what's happening here. It should be persisted between restarts. I'll open a separate bug because this thread becomes too confusing with all things in here.

Like that?:
"daily": [

yes.

But this is "daily forecast" with first timestamp of this day at 09:00:00 GMT+0000.
"current": {
"dt":<PHONE_NUMBER>,            Sat Sep 09 2023 17:28:08 GMT+0000
"daily": [
{
"dt":<PHONE_NUMBER>,       Sat Sep 09 2023 09:00:00 GMT+0000
Although it is updated through the day, it is still forecast. For example, for me two hours ago tmin was 10.1C, now it is 10.7C. But daily minimum temperature can't rise by definition.

You're going to come to the same conclusion I have a while back and why I started v2. Look back at this thread and you will see that we have stopped getting data from daily except for precipitation as there is no reliable way to tell that from current. Unless you have a good alternative for that that works based on OWM I think the discussion you're trying to have here is irrelevant as we have already moved passed it. Did you read the other messages in this thread?

Yes, I read, no found any reference that you were tring to use forecast data. No mind.

Going to use this https://github.com/petergridge/openweathermaphistory.
I don't see problems with getting current precipitation with frequent updates.


[test-only] ApiTest. checking the owner of the project space

related https://github.com/owncloud/ocis/issues/4419

@ScharfViktor can we add this to the expected failures? I want to test it in https://github.com/owncloud/ocis/pull/5341 which should make it pass.

@ScharfViktor can we add this to the expected failures and merge it? I want to test it in #5341 which should make it pass.

done

thanks, i cherry pick it in https://github.com/owncloud/ocis/pull/5341 to see if it passes, give me one more ci run ;)
THX

@ScharfViktor i had a look, the space owner is still the spaceId (project space only).
I've change the owner in AsResourceInfo to manager, this is used in stat requests.
Therefor the test still fails (the graph drives endpoint still uses the spaceId as owner).
@butonic FYI

Let me know if this is ok for you :)

For me it's okay because it doesn't break any business logic and everything still looks good on the UI


feat(documentation): add content for the welcome page

Preview environments are getting posted here, as soon as they are ready!


Workshop update

Update the entry for the workshop
cc @pwalczysko @will-moore @jrswedlow

I do not see the Oxford workshop in this list, although the other entries from this PR seem to have been built successfully.
I mean

(Oxford missing)
but

(Oxford present as expected, shows that the PR has been included).

@pwalczysko done

Works as expected.
I just realized, we have the ELMI workshops still to do - but I guess this can be done in another PR.

Again, missing in Events list

Did you rebuild?
refresh it should be in

@jburel \yes, sorry, refresh was needed.
Also the ELMI capitalization is now built and reads well.
Ready to merge


[Enhance] separate disk assign and disk status update process

What this PR does / why we need it:
separate disk assign and disk status update process
Special notes for your reviewer:
NONE
Does this PR introduce a user-facing change?

Codecov Report

Merging #815 (df6230f) into main (847af4d) will increase coverage by 0.27%.
The diff coverage is 66.66%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@            Coverage Diff             @@
##             main     #815      +/-   ##
==========================================
+ Coverage   29.45%   29.72%   +0.27%
==========================================
  Files          25       25
  Lines        1616     1618       +2
==========================================
+ Hits          476      481       +5
+ Misses       1074     1072       -2
+ Partials       66       65       -1

Impacted Files
Coverage Δ

.../local-disk-manager/handler/localdisk/localdisk.go
19.81% <0.00%> (ø)

...k-manager/handler/localdiskclaim/localdiskclaim.go
26.27% <0.00%> (ø)

...roller/localdiskclaim/localdiskclaim_controller.go
35.05% <76.92%> (+3.86%)
:arrow_up:

pkg/local-disk-manager/filter/filter_disk.go
76.38% <100.00%> (+2.06%)
:arrow_up:

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


add feature: parse format: 'double' to Float

Thank you for your PR, but we just accepted change, which makes number according to Float


Respect the search.useIgnoreFiles setting when ignoring files

Code has a setting search.useIgnoreFiles which instructs its search to ignore files matching patterns listed in .gitignore and .ignore.
It would be great if Intelephense respected this setting, and also ignored those files when the setting is in use.
Great work on the extension. Keep it up!

v1.0.0 has it's own config setting for this now intelephense.files.exclude that accepts an array of globs.


Do not select always transitions when no regular transition gets selected

This essentially reverts https://github.com/statelyai/xstate/pull/2343 . I open this PR to spark the conversation about this behavior. It wasn't a change requested by any user - we just did it to comply with SCXML. However, this behavior "promotes" impurity and I really wonder if we want to keep this behavior.

What about the failing tests?

Those are directly related to this change. So if we decide to go with this change then I'll just adjust them before landing this.

@davidkpiano reopened this one like we discussed offline


Fix classless static routes

Classless static routes are broken.
Why?:

Subsequent definitions of of {rfc3442,ms}-classless-static-routes override their predecessors.  Perhaps this was not true in older versions of dhcpd, although I doubt it.
There is no ability to define a /0 masked static route.  With dhcpcd and perhaps other DHCP clients, option 3 (router) is ignored when rfc3442-classless-static-routes is specified, meaning there is no non-hackish way to specify a default route.

Have I tested this?
Yes, but only with Puppet 4.x and Debian family systems.
Styling:
I've chosen a styling that I think appropriately balances readability of the template with readability of the resulting config.  There may be a better way to do this, I'm not that familiar with Ruby.  I am happy to make changes to better balance the two as per your preferences.

I'm confused about the failing build, none of the failures seem to have anything to do with my changes, and the "expected/got" are all exactly the same, but maybe I'm missing something.

Thank you for your contribution. We have tests that verify the rendered output with various parameters in https://github.com/theforeman/puppet-dhcp/blob/master/spec/defines/pool_spec.rb and they need to be updated as well for the tests to pass.

Ok, I'll figure it out.  Do you want me to squash/fixup the next commit after I get the build passing?

It doesn't really matter during review, but we indeed do prefer squashed commits before we merge.

There we go, it was just one extra space after ms-classless-static-routes before the routes.

Would you mind adding a the /0 network to the current testcase?

Sure thing.

All good?  If so I'll squash.

Alright, made that one change on split and squashed them all up.

merged, thanks @Izzette !


buildEnv fails when one of the paths to link is a symlink to a directory

Describe the bug
As per title. Check the example at https://gitlab.com/ggpeti/buildEnv-sscce. There are 2 top level directories, a and b. b has an empty file called c in it. a has a symlink to b in it. When one tries to create a trivial buildEnv like:
buildEnv {
  name = "test";
  paths = [ ./a ];
  pathsToLink = [ "/b" ];
}

the following, weird error message surfaces:
> collision between `/nix/store/sv597kz16cwk4g15w4665w6alj9na81q-a/b' and `'

Steps To Reproduce
nix build gitlab:ggpeti/buildenv-sscce
Expected behavior
buildEnv should treat symlinks to directories just like directories. It should not fail but create the environment.
Additional context
This issue probably underlies https://github.com/LnL7/nix-darwin/issues/320 and https://github.com/nix-community/home-manager/issues/1994.
Notify maintainers
@edolstra
@urkud
@aszlig
@Ma27
@MarcWeber
@vcunat
@rasendubi
@romildo
@lheckemann
@abbradar
Maintainer information:
# a list of nixpkgs attributes affected by the problem
attribute: [ buildEnv jdk zulu ]
# a list of nixos modules affected by the problem
module: []

Sorry, false alarm. This issue is the same as #82685.


Accessibility attributes: use ngAria or manually

aria attributes should be set for screenreader support. Angular provides some magic:
https://docs.angularjs.org/guide/accessibility
Check if this is really good or needs too much workarounds (e.g. tabindex). See also #107

I'm closing this to focus on features and code improvements first. Accessibility is not to be neglected, but can be improved at a later point.


9.23

Please use the form below, leaving the prefilled data to help us. Thank you.
Page link: 9.23
Document link: 9.23.md
My Issue/Suggestion
I was not able to install this version, but I found a workarround.
I got the message "Internal error: ShellExecuteEx returned hProcess=0." Click OK the same message is displayed. Click OK again and the installation process terminates.

Here the workarround - if I don't click OK in the first message, the installation process continues.

Hi @JZelenak thanks for the message. This channel is for feedback on documentation itself, and this looks like a product problem that requires a Support ticket. Please contact Mendix Support - https://support.mendix.com/hc/en-us - to communicate this problem.
Cheers,
Adam

Hi @JZelenak , reopening this issue, as I created a Mendix Support ticket for it. I will update here upon resolution.

Hi @JZelenak I heard back from Mendix Support:
Thanks for reporting this issue to us and sharing the resolution.
This issue doesn't seems to be specific to Mendix Platform but a system issue.
Please make sure all the System requirements mentioned in the below document are installed correctly.
https://docs.mendix.com/refguide/system-requirements/
You can check below document to resolve the issue.
https://www.thewindowsclub.com/fix-shellexecuteex-failed-code-error-in-windows-10
https://answers.microsoft.com/en-us/windows/forum/all/shellexecuteex-error/4cec9d0c-8497-48ab-b273-67871e9f03c7
If you want to report any platform-related Mendix issue we recommend you to reach out to Mendix Support and you can also post your queries on our Mendix Forum.
https://support.mendix.com/hc/en-us
https://forum.mendix.com/


Draft: Add UPDR algorithm

Added an initial UPDR implementation based on the one in mypyvy. Created as a project in the course "Automatic verification of systems" taught by Prof. Sharon Shoham Buchbinder.
The implementation itself takes 11.31 secs to run on lockserver.fly as opposed to the 5.47 secs it takes mypyvy to run updr on lockserv.pyv. Sadly, when attempting more complex problems, such as consensus.fly , we get a timeout. This implementation will require some more work until it can be merged IMO.
The PR also contains some helper functions that the mypyvy implentation used and were added to flyvy:

Conversion to CNF.
Splitting a CNF term into clauses.
Negation that pushes the - inward in a term.
Removal of prime from a term.

If any of these functions feel useful enough, I can open a separate PR for them.

I cleaned up the the git history and rebased this on top of main. We should be good to merge it for now, and we can continue to iterate on the code.


chore(deps): upgrade braze

This PR resolves PHIRE-899
Description

Upgrades braze from deprecated react-native-appboy-sdk to newer dep.
This is still a ways behind latest but getting to latest is a bit of a can of worms so trying to take baby steps. 😅
Draft while I test.
PR Checklist

[ ] I have tested my changes on iOS and Android.
[x] I hid my changes behind a feature flag, or they don't need one.
[x] I have included screenshots or videos, or I have not changed the UI.
[x] I have added tests, or my changes don't require any.
[x] I added an app state migration, or my changes do not require one.
[x] I have documented any follow-up work that this PR will require, or it does not require any.
[x] I have added a changelog entry below, or my changes do not require one.

To the reviewers 👀

[ ] I would like at least one of the reviewers to run this PR on the simulator or device.

Changelog updates
Changelog updates

Cross-platform user-facing changes

iOS user-facing changes

Android user-facing changes

Dev changes

upgrade braze - brian

Need help with something? Have a look at our docs, or get in touch with us.

This PR contains the following changes:

Dev changes (upgrade braze - brian)

  Generated by :no_entry_sign: dangerJS against 18a08358f406e8f1d23d264cc5b00998e58ddb6e


Restructured Quick Time Events to Facilitate Functioning In Combat

Task: Restructured Quick Time Events to Facilitate Functioning In Combat
Feature: [Team 10] - Cleaning Up, Minor Improvements, and Bug Fixes (#539)
Overview:
This task focuses on restructuring Quick Time Events (QTEs) to function seamlessly within the combat screen. We have modified QTEs to store the previous screen state by adding necessary variables such as entities, services, and previous screen state. Additionally, we've made the combat screen pausable and implemented state-saving functionality. This allows us to restore the combat screen state when exiting QTEs.
Objectives:

Implement storage of previous screen state in QTEs
Add variables for entities, services, and previous screen state
Make the combat screen pausable
Implement state-saving for the combat screen
Create an onExit function in QTEs to restore the combat screen

Milestones & Deadlines:

[x]  Implement state storage mechanism for QTEs (Oct. 9 )
[x]  Develop pausable combat screen functionality (Oct. 10)

Documentation

[Main description of feature](/wiki/Combat Screen)
JavaDoc

Member

Sneha Zazen (sneha-zazen on GitHub) (zaza7_13 on Discord)

merged to 10/Integrate-QTE-to-combat-moves branch


Installation yml files

I believe the digests are broken and don't work. They might need to be removed in order to facilitate the installation of the environments.

@Michael-C-Strobel is working on updating these so I reopened.

@Michael-C-Strobel #14 contains screenshots of installation errors from another user.

In the end I used the Colab Notebook. Those requirements and the version dependencies might be of use!


The entry about the installation of a module that was not installed by interruption in the preload is not deleted

zpm:USER>load https://github.com/SergeyMi37/zapm
Cloning into 'zapm'...

Load of directory started on 12/17/2022 12:04:53
Load finished successfully.

[USER|zapm]	Reload START (/usr/irissys/mgr/Temp/664604348117402302/zapm/)
[zapm]	Reload FAILURE
ERROR! This application must be installed in the "%SYS" namespace
  > ERROR #5490: Error running generator for method 'RunOnLoad:%ZAPM.ext.Lifecycle'
ERROR: %ZAPM.ext.Lifecycle.cls(RunOnLoad) of generated code
    > ERROR #5030: An error occurred while compiling class '%ZAPM.ext.Lifecycle'
zpm:USER>
...
zapm:USER>list
zapm 1.0.42 (DeveloperMode)

The error is also reproduced with any erroneous loading of the module

zapm:iris-test>IRIS>SWAGGER>list
swagger-ui                      1.0.0
openapi-suite                   1.0.0 (DeveloperMode)
openapi-server-gen              1.0.0 (DeveloperMode)
openapi-common-lib              1.0.1 (DeveloperMode)
swagger-converter-cli           0.0.2 (DeveloperMode)
openapi-client-gen              2.1.1 (DeveloperMode)
swagger-validator-cli           0.0.1 (DeveloperMode)
yaml-utils                      0.1.2
objectscript-openapi-definition 1.1.7
sslclient                       1.0.1
zapm:iris-test>IRIS>SWAGGER>uninstall swagger-validator-cli
[SWAGGER|swagger-validator-cli] Clean START
[SWAGGER|swagger-validator-cli] Unconfigure START
[SWAGGER|swagger-validator-cli] Unconfigure SUCCESS
[SWAGGER|swagger-validator-cli] Clean SUCCESS
zapm:iris-test>IRIS>SWAGGER>
zapm:iris-test>IRIS>SWAGGER>
zapm:iris-test>IRIS>SWAGGER>list
swagger-ui                      1.0.0
openapi-suite                   1.0.0 (DeveloperMode)
openapi-server-gen              1.0.0 (DeveloperMode)
openapi-common-lib              1.0.1 (DeveloperMode)
swagger-converter-cli           0.0.2 (DeveloperMode)
openapi-client-gen              2.1.1 (DeveloperMode)
yaml-utils                      0.1.2
objectscript-openapi-definition 1.1.7
sslclient                       1.0.1
zapm:iris-test>IRIS>SWAGGER>zpm "load https://github.com/lscalese/iris-swagger-validator-cli.git"
[SWAGGER|swagger-validator-cli] Reload START (/opt2/isc/iris/mgr/Temp/6666049283930640369/iris-swagger-validator-cli/)
[SWAGGER|swagger-validator-cli] requirements.txt START
[SWAGGER|swagger-validator-cli] requirements.txt FAILURE
[swagger-validator-cli] Reload FAILURE
ERROR! Running command returned error.
zapm:iris-test>IRIS>SWAGGER>
zapm:iris-test>IRIS>SWAGGER>list
swagger-ui                      1.0.0
openapi-suite                   1.0.0 (DeveloperMode)
openapi-server-gen              1.0.0 (DeveloperMode)
openapi-common-lib              1.0.1 (DeveloperMode)
swagger-converter-cli           0.0.2 (DeveloperMode)
openapi-client-gen              2.1.1 (DeveloperMode)
yaml-utils                      0.1.2
objectscript-openapi-definition 1.1.7
sslclient                       1.0.1
swagger-validator-cli           0.0.1 (DeveloperMode)


[celery] setting default

This PR remedies an issue we've been seeing with celery_statsd where celery.current_app refers to the default rather than the app instance. Per this thread it seems like enforcing the app to be the default for all threads remedies the issue, which I verified via testing locally.
to: @michellethomas @mistercrunch @timifasubaa

Codecov Report

Merging #5812 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #5812      +/-   ##
==========================================
+ Coverage   63.82%   63.83%   +<.01%
==========================================
  Files         364      364
  Lines       23099    23100       +1
  Branches     2587     2587
==========================================
+ Hits        14744    14745       +1
  Misses       8340     8340
  Partials       15       15

Impacted Files
Coverage Δ

superset/utils.py
89.16% <100%> (+0.01%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update eb41756...d4d696f. Read the comment docs.


[FEATURE REQUEST] Add ability to turn off underline for internal links

I was seeing this setting in the Minimal's Style settings. In my opinion, not only I want to make internal links seamless in the text. Thank you in advance!

Relevant options have been added in the new version.

Thank you, it works!


Styled subtitles are converted incompletely

When converting subtitles from .ttml to .srt, inline spans causes subtitles to be cut off prematurely.
Original (.ttml):

Ein lever berre éin gong. Ingen gjer
noko for meg. Eg må gjere det sjølv.

Conversion (.srt):
Ein lever berre éin gong. Ingen gjer
noko for

The following part is missing:
meg. Eg må gjere det sjølv.

Verbose output:
~$ youtube-dl -f mp4 --all-subs --convert-subtitles srt --embed-subs 'https://tv.nrk.no/serie/skeive-jenter/KOID37003614/sesong-1/episode-10'
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'-c', u'-i', u'-f', u'mp4', u'--all-subs', u'--convert-subtitles', u'srt', u'--hls-prefer-native', u'--audio-format', u'm4a', u'-o', u'%(title)s.%(ext)s', u'--verbose', u'https://tv.nrk.no/serie/skeive-jenter/KOID37003614/sesong-1/episode-10']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2015.12.23
[debug] Python version 2.7.11 - Darwin-15.2.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 2.8.3, ffprobe 2.8.3
[debug] Proxy map: {}
[NRKTV] KOID37003614: Downloading webpage
[NRKTV] KOID37003614: Downloading f4m manifest
[NRKTV] KOID37003614: Downloading m3u8 information
[info] Writing video subtitles to: Skeive jenter.no.ttml
[debug] Invoking downloader on u'http://nordond16c-f.akamaihd.net/i/no/open/fb/fbaf6ba0eecbb3d23c7546e9e97371ef2fe4b798/9e7d670e-a5c0-4f61-a87d-1145f7f05315_,141,316,563,1266,2250,.mp4.csmil/index_4_av.m3u8?null='
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 73
[download] Destination: Skeive jenter.mp4
[download]   0.0% of ~216.35MiB at  2.42MiB/s ETA 01:50
frame= 1250 fps= 11 q=-1.0 Lsize=   14201kB time=00:00:50.00 bitrate=2326.4kbits/s
video:13187kB audio:977kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.258942%
Exiting normally, received signal 2.
[download] 100% of 214.49MiB in 04:11
[ffmpeg] Converting subtitles
WARNING: You have requested to convert dfxp (TTML) subtitles into another format, which results in style information loss
~$

Post the full output of youtube-dl when run with --verbose option.

Done

the same error happen to me.
the original line:

Security guard, sir. Mr Barlow, I
don't know what you've heard,

the srt line:
Security guard, sir.
don't know what you've heard,

in the python documentation about text and tail attributes of xml.etree.ElementTree.Element https://docs.python.org/3/library/xml.etree.elementtree.html#xml.etree.ElementTree.Element.text:

the text attribute holds either the text between the element’s start tag and its first child or end tag, or None, and the tail attribute holds either the text between the element’s end tag and the next tag, or None. For the XML data

so this code can't extract the text between tags.
        out = str_or_empty(node.text)

        for child in node:
            if child.tag in (_x('ttml:br'), _x('ttaf1:br'), 'br'):
                out += '\n' + str_or_empty(child.tail)


support.mrfioc2: 2.6.0 -> 2.7.1

I approve

I remove the patch from kernel modules mrf and squash my commits


[DIR-1353] Add filters to gateway routes list

Description
Users should be able to filter a large amount of gateway routes by typing in a search input field.
It will search for the keyword in the file name and in the path.
Another change in this PR:
The Refresh Button changed appearance everywhere, because the icon was mirror-inverted.
The icon before showed an arrowhead pointing in the counter-clockwise direction, but when clicking it, it was rotating clockwise. Rotating clockwise is a more popular choice for the use case "Refresh", so the icon now displays an arrow pointing in the same direction.
Checklist

[x] Documentation updated if required
[x] Test coverage is appropriate

Checklist Internal

[x] Linear issue linked (e.g. [DIR-XXXX] pull request title)
[x] Has the PR been labeled

Cool, thanks! 🙂
Yeah, sorry for keeping you busy with the detective work. I added details about the icon change in the PR description now!
That's a great solution to search for both categories but in one search field. I will add that functionality now.


Implement rule "flying general"

Please implement the "flying general" rule.

Please re-open, the issue is not fixed.


Rely on bash default value interpolation

The whitelistedEnv() function writes a bash export for the environment using a default value syntax, e.g.:
export MY_VAR=${MY_VAR:-"value sent in NM env"}

Which makes checking if MY_VAR exists in the environment passed into the YARN launch extraneous. The downside of this extra check is that there are some envvars related to Docker YARN LCE containers that appear to be set to empty strings.
For example YARN_CONTAINER_RUNTIME_DOCKER_IMAGE appears to always be set to an empty string unless explicitly set. In my test environment I have something like the following in my yarn-env.sh:
export YARN_CONTAINER_RUNTIME_DOCKER_IMAGE='hadoop-yarn-lce:some-tag'

Checking /proc/{PID}/environ of the NodeManager process shows that YARN_CONTAINER_RUNTIME_DOCKER_IMAGE is set as expected.
I also have YARN_CONTAINER_RUNTIME_DOCKER_IMAGE set as a whitelisted envvar with yarn.nodemanager.env-whitelist in yarn-site.xml.
When I try and submit a job to be run within Docker with something like:
yarn \
  org.apache.hadoop.yarn.applications.distributedshell.Client \
  -jar /usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell.jar \
  -shell_command sleep -shell_args 10 \
  -shell_env YARN_CONTAINER_RUNTIME_TYPE=docker

My expectation is that the default Docker image value set in the NM would be used however I get the following error because an empty string is causing containesKey() to true and skipping loading of the NM default value:
Exception from container-launch.
Container id: container_e5300_1732314455042_0036_01_000002
Exit code: -1
Exception message: YARN_CONTAINER_RUNTIME_DOCKER_IMAGE not set!
Shell error output: <unknown>
Shell output: <unknown>

[2024-11-22 23:54:04.582]Container exited with a non-zero exit code -1.

This issue happens with YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS and perhaps other envvars as well. Interestingly it does not happen with YARN_CONTAINER_RUNTIME_DOCKER_CONTAINER_HOSTNAME

Description of PR
How was this patch tested?
For code changes:

[ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
[ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 19s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
32m 13s

trunk passed

+1 :green_heart:
compile
0m 50s

trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
compile
0m 51s

trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

+1 :green_heart:
checkstyle
0m 26s

trunk passed

+1 :green_heart:
mvnsite
0m 31s

trunk passed

+1 :green_heart:
javadoc
0m 34s

trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javadoc
0m 26s

trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

+1 :green_heart:
spotbugs
0m 59s

trunk passed

+1 :green_heart:
shadedclient
20m 27s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 21s

the patch passed

+1 :green_heart:
compile
0m 46s

the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javac
0m 46s

the patch passed

+1 :green_heart:
compile
0m 45s

the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

+1 :green_heart:
javac
0m 45s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
0m 15s

the patch passed

+1 :green_heart:
mvnsite
0m 23s

the patch passed

+1 :green_heart:
javadoc
0m 21s

the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javadoc
0m 21s

the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

+1 :green_heart:
spotbugs
0m 51s

the patch passed

+1 :green_heart:
shadedclient
20m  3s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
23m 19s

hadoop-yarn-server-nodemanager in the patch passed.

+1 :green_heart:
asflicense
0m 26s

The patch does not generate ASF License warnings.

105m 58s

Subsystem
Report/Notes

Docker
ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7182/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/7182

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux 043f2c83173a 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 19550b333eab06bb94ae323805c3cfd5de2481ac

Default Java
Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7182/1/testReport/

Max. process+thread count
552 (vs. ulimit of 5500)

modules
C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7182/1/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.


Allow injection of extra materials via files

Some build systems could create files that contain materials used during
the build (like docker images, operating system packages etc.). These
files would contain a JSON array with Items. When set on the
command-line flag -extra_materials, slsa-provenance will read and
parse these files and add the found materials to the statement.

This looks great. Could you elaborate a bit on the usecase and where such provenance file would originate from.

To build PowerDNS packages we use pdns-builder, which is used a git sub-module.
pdns-builder takes some templates and generates a Dockerfile that has all the instructions to build the RPM or Debian package, which is docker cp'd out of the container by the build script when done. The pdns-builder repo also has a few helper scripts to generate the list of installed OS packages in the in-toto materials format.
SLSA specifies that the URL of materials is a "ResourceURI" and that is RECOMMENDED to be a "package url" and has some examples on how these urls can look.
Being able to add this package information to the SLSA material provenance would allow anyone to create a replica of the build environment and reproduce the package. It also proves/tells us what versions of dependencies we linked against.

Maybe add a reference to slsa or maybe we can clarify a bit in the README.md.

SLSA has the "Dependencies complete" requirement. It is described as

Provenance records all build dependencies that were available while running the build steps. This includes the initial state of the machine, VM, or container of the build worker.
MUST include all user-specified build steps, sources, dependencies.
SHOULD include all service-provided artifacts."

For us, the initial state is the docker base-image we pulled (e.g. debian:buster) and the service-provided artifacts are the packages that installed as part of the build-process (via e.g. apt-get).

We are also still learning on slsa and the provenance…

So are we :).

E.g. how would this work with docker images? We also have an open issue to work on docker image provenance support #73

This might not work for actually generated docker images. This is about the docker images used as a base for the builds (which, combined with the exact installed packages should lead to something that would allow for a reproducible build).
P.S. I did notice I didn't update the action.yaml to allow this option to be set, I'll add a separate commit for that.

Thanks for clarifying. Now it starts to make more sense to me. Would it be possible to add a small usage example here in the README.md.
https://github.com/philips-labs/slsa-provenance-action/blob/main/README.md?plain=1#L188
That would be very beneficial for other consumers of the action.

Would it be possible to add a small usage example here in the README.md.

Done!

P.S. I did notice I didn't update the action.yaml to allow this option to be set, I'll add a separate commit for that.

Done!


[FBCode->GH] Specify tls cert file to decoder through config

Summary:
Upgrading service to platform009 enforces tls1.3 authentication.
Specify tls cert file to decoder through config.
Context: https://fb.workplace.com/groups/2020511478062223
Differential Revision: D26026259
fbshipit-source-id: e89f88ff7e41ec6b6a62e85d9015d78f6377aa65

The failing tests are due to conda issues and they are unrelated to this PR.

The failing tests are due to conda issues and they are unrelated to this PR.