Device Tracker State

The device_tracker states use the friendly name for the state rather than the proper state name. This can cause issues when using the visual editor for automations.
For example, iCloud3 sets the state of the device_tracker to Away.
When creating an automation the visual editor has the friendly name of Away as an option, but actually uses the state of not_home.
Automation doesn't get triggered as iCloud3 doesn't set the device_tracker state to not_home.

See iCloud3 configuration > Event Log Parameters > Device tracker entity HA entity override parameter. Set to off.

Thanks Gary.
Changed that setting and the state is being set as the non-friendly name now, however the zone trigger is still not being triggered in the automation. So it must be something else.
I have a person entity with the iCloud3 device_tracker as it's only tracked device.
The automation is set to trigger when the person entity exits a zone.
When you exit the zone the person state changes but the zone exit trigger is not set off.
The only difference I can see between v2 and v3 is that on v2 the GPS coordinates show as two decimal places in more info / attributes, rather than lots of decimal places with v3.
Under Developer Tools / States, the GPS coordinates show lots of decimal places with both v2 and v3.

If the ic3 d_t state value changes correctly which changes the person correctly but HA does not catch the change and issue the trigger based on the person, it’s probably a HA timing issue, not an ic3 issue.  I would not use the person as the trigger entity. I use the sensor.[devices]_zone_name as my triggers, not the d_t or person. I actually don’t use person at all.  I like the friendly name on the d_t entity. the sensor zone_name is under ic3 control, not HA control so I know what’s going on.

OK. I realise I can work around it in different ways, but it is a regression from v2 behaviour.

Here is what I'm seeing for the decimal places using the app device tracker and then the iCloud3 v3 device tracker against the person entity.
v2 does the same as the HA App example.

Thanks Gary.
No, not wondering why the iOS and iCloud3 gps is different. I'm wondering why iCloud3 v3 seems to always ignore the iOS data. This was not the case in v2.
The screenshots above are to illustrate that v3 is setting the device tracker location somehow differently to how v2 did and how the HA app does. The two more info entities shown are the same entities as to their left. For some reason the HA App (and v2) more info rounds to two decimal places, while the v3 more info does not, while dev tools / states always show the unrounded coordinates. I'm not sure if the rounding itself is causing v3 not to correctly trigger zone entry/exit triggers for HA, or another underlying difference in how v3 is updating the device tracker location compared to v2.

I am not having any problems with iOS app triggering events, that I know of.
So I need detailed logs from v2 and v3 to see what you are seeing. Just seeing a screenshot May identify the problem but I’m going to need some raw data to see what is going on.  I assume you are running them in parallel on different computers or in virtual environments where v2 and v3 are tracking the same devices at the same time so the updates on both should be the same.

Set log level Debug in both v2 and v3..
In v3, delete the icloud3-debug.log file before the reboot.
On both, delete the ‘home-assistant.log‘ files so I don’t have to go through a lot of old data.
Restart HA so both v2 and v3 are starting fresh. Let everything run until you are sure the problem has occurred for a while.
Email me the iCloud3-debug.log and home_assistant.log files.
You can also do a Event Log > Actions > Export Event Log on both v2 and v3.
Identify some of the triggers on v2 that should show up on v3 but aren’t. They should be on the Event Log. That way I can look at the raw data in v2 and compare it to v3 from the same time period when the problem is occurring. I’ll have a lot of data to look at and will try to see what you are seeing.

Thanks.
Gary

Thanks Gary.
Sorry, I don't currently have a second environment to test in parallel. I've been manually changing between v2 and v3. I'll see what I can set up.
Version 2 and version 3 screenshots below for coordinate rounding comparison.
I'll turn on the Show Tracking Monitors to see what it reveals.
I'm tracking two devices.
Just so we're on the same page...

iOS app is correctly triggering iCloud3 events. The problem is iCloud3 seems to always be ignoring the iOS app location data and only getting location data from iCloud. (Other issue - #68 )
Separate to that issue, HA automation zone entry/exit triggers are not firing due to how v3 is updating the device tracker entity compared to how v2 does. (This issue - #65 )

Dev tools > states screenshot, iCloud3 v2 and HA App, which both work with HA automation zone entry/exit triggers

More info screen shots of the same entities at the same time with rounded coordinates
device_tracker.brads_iphone

device_tracker.brads_iphone_2

Dev tools > states screenshot, iCloud3 v3 and HA App, iCloud3 v3 doesn't work with HA automation zone entry/exit triggers

More info screen shots of the same entity at the same time with iCloud3 non-rounded coordinates

There has to be a setup issue with the iOS app if you are never getting triggers and location updates from the iOS app. Send me the iCloud-debug.log file<EMAIL_ADDRESS>to see what you r set up is. Also send me an Event Log > Actions > Export Event Log of v3. Set log level to raw data to show what is being picked up by the iosapp sensors iCloud3 is monitoring.
And go to the Update iCloud3 Devices configuration screen and look at the iOS app monitor device assignment. You can set to a specific iOS app device or scan for one starting with the FamShr device name. Select the actual device to monitor, not the scan selection.
About the gps 2-decimal places in v2, I haven’t changed v2 since Dec2021 so I guess it’s always been that way.. I can only make sure v3 is filling in its d_t and sensor entities correctly. Which it looks like it is. How person handles it is out of my control if the gps in the d_t attributes are correct.

I've done a clean install of v3 without upgrade. Will monitor what it does over the next few days.
The monitor device assignment is set to the specific devices, so that's ok.
With the decimal places, is v2 using a different method to update the device_tracker compared to what v3 is doing?
How person and device_tracker handles it is out of your control, but it is handling the way v2 updates it correctly and failing with v3.

Everything is different between v2 and v3. V2 creates all the d_t and sensor entities as temporary ones where v3 is a true Integration that creates permanent ones using HA standard methods where they are added to the HA entity registries.
Maybe by changing back and forth, the person is tracking the v2 entity, not the v3 one. The person i is set up in the core/. Storage/person file. It has the d_t entities assigned to that person. The device_registry and entity_registry also have the dt name. When an entity is added, HA will add a suffix to it and use that name internally instead on the first one you entered. Maybe the person  has the v2 name, not the v3 one. Look in those registry files for the v3 name and see if that is the case. Set up a new person when v3 is running and see what it follows.

My person entity is set up correctly. It is really related to the iCloud3 device tracker entity because an automation with zone trigger looking directly at that has the same result.
I'm not sure I can do anything more from my end.
If you're so inclined, it should be straightforward to reproduce the problem on your end. You just need a simple automation with a zone trigger looking at the iCloud3 device tracker.

I would be glad to try to reproduce the problem on my end but I need you send me the yaml code for your automations that show what is not working. It would be futile and time consuming for me to try to guess what you are seeing/using try to recreate the problem when I really do not know what it is and still not get it.  I do have a lot on my plate and guessing is not a good use of my time.
The device tracker entity state value has 2 options- the actual HA zone entity name and a Friendly Name that is either the HA zone friendly name or a reformatted name. I need to know exactly what you want it to be if that is not enough. Otherwise, I'm sure that you will be changing your automation triggers to use what iC3 is providing.

I've tried both options for the device tracker entity state with the same result.
Here's an example automation that fails to trigger with v3. Works fine with v2:
alias: iCloud3 Zone Test
description: ""
trigger:
  - platform: zone
    entity_id: device_tracker.brads_iphone
    zone: zone.home
    event: enter
condition: []
action:
  - service: notify.mobile_app_brads_iphone_2
    data:
      message: Welcome home
mode: single

I’m seeing similar issues with my setup. If I set it to use the HA name, then if I go to a zone called store the state will be “store” but the zones display name would be “Store” and wouldn’t match. If I set it to the Formatted name, then Store will be “Store”, but then my home zone gets messed up. The home state would be “Home” but the state that my house is expecting would be “home”. This worked fine under v2 but does not work in v3. It’s almost like we need to set the state to lowercase but display the state in cards in uppercase, or use the friendly name for displays.

Go to developer tools and look at device tracker state. What are for the iCloud3 and iOS app values? Upper case, lower case, both different. I can force to lower state if HA zone entity  I have seen home (lower) as the state value but display as Home (upper). But other zones have state value as lower (store) and display as lower (store). I’m not sure what HA is doing. See if it makes sense for you.

I did a lot of errands today and monitored the device_tracker state in the developer tools screen, what was displayed on the icloud3 screen and in the Attributes section of the device_tracker entity. From what I can tell, iCloud3 is setting the d_t state value to the zone.zone entity name for the various zones, not the friendly_name or anything else. Attached are screen captures for you to compare to what you are seeing. I have the iC3 > Configurator > Event Log Parameters > Device_tracker set to HA zone entity name.

Yes that matches what I see as well, but now create an automation for when you enter or leave a zone. It is looking for Warehouse or Publix MM, etc, not the lowercase values so it doesn’t trigger.

The HA iOS app is using the friendly name as the state as shown in the pics above, not the zone entity name. Maybe HA is looking for a match on the friendly name, not the entity name. Show me you automation that is not triggering, the d_t state and friendly name and the zone entity and friendly name so I can understand the issue and exactly what option you would like the d_t state values to be.
If you want, send me Automation and the /config/.storage/entity_registry file so I can see exactly what the values in the entity registry file matches to your setup. In the entity registry file, the zone actually has 3 names, the entity name, the name, and the original name.

His automation is exactly how mine is set up, just a little more detailed because it’s not a test one. Ill see if I can grab those files soon and see how they look.

I just added a zone - ‘Back in Form’:

entity_reg -> entity_id=‘back_n_form’, original_name=‘Back in Form’
zone -> entity_id=‘zone.back_in_form, name=‘Back in Form’

Change the zone name to ‘BackInForm’:

entity_reg -> no changes
zone -> name=‘BackInForm’

HA changed the zone file but not the entity reg.
v2 got the zone’s friendly name from zone, v3 gets it from entity_reg.
On the Area & Zone screen, it uses names from zone
When you list the zones on a name to select from, it shows the entity_reg name.
Looking at the actual code in the zone component trigger.py module, the friendly name comes from zone, not the ent_reg
This is an inconsistency in HA where it sometimes lists the name from zone and in other places, uses the one from entity_reg. I think it is trying to keep them in sync but that does not always happen.
I’ll change it back to the zone file for trigger purposes and maybe show the one from the entity_reg value for list selection purposes on the configuration screens.

I have added some options to the d_t state value (zone entity, friendly name, reformatted iC3 names based on the entity id). The friendly name comes from the zone setup file instead of the entity registry. Plus a lot of other stuff.
Attached is the beta13 preliminary release. See the ChangeLog for a complete list of items. There are some important things in it.
I have looked at the code that generates the zone enter/exit triggers and tried to figure out what ha is doing by following the code in the icloud3 component. I do not think it is looking at the state values but, instead, is looking at the lat/long attributes and then using them to determine if the device is in a zone and then checking to see if the device's zone changed. It then issues a trigger that fires the automation.
I have set up several automations, selecting a device_tracker entity, a zone and a trigger. My automations do not trigger and never run. I think there are some settings inside ha that need to be set up that I have not figured out. I will continue to look at it but will need to find a better example than the convoluted icloud component, Let me know it something works for you.
Inzip into the iCloud3 directory like you have been doing.
icloud3_v3-beta-13p4.zip

Just tried beta 13 and set the state to the HA Zone Friendly Name used by automation triggers option, but it shows my device_tracker.brians_iphone as Home, whereas my _2 device tracker is home. So it seems like there is still something off with the formatting.

I assume your _2 device is the iOS app device. The iOS app is inconsistent with setting the device_tracker state value for the home zone. It will always display it as home. With iCloud3 set to the friendly name, iCloud3 will set it to Home. If you move to another zone (entity=cosco, Friendly Name =Cosco), the iOS app displays the friendly name Cosco.
Obviously, I can change the home zone only to always display home but was never sure if that was the thing to do since you would then have an inconsistency that may affect others. The home could be the friendly name setting by default with another option to set it to Home.
I’ll post a question and see what the response is.

Yea the _2 is the iOS app. And I know that’s unaffected by ic3, I was just showing that the iOS app has the “correct” state. When ic3 is set to Home, my person entities are not home and my house shuts down. But when it’s set to home, I am home.
The issue still is if I have my home set in lowercase, the zone enter/exit automations still fail to trigger. These same automations worked perfectly under ic3_v2

Get beta 13a just uploaded for the 'home / 'not_home` state value issue. These values are now always used when the Friendly Name is selected so iCloud3 displays the same values as the iOS App.
About the zone enter/leave trigger. I thought I had it figured out and made some code changes but it is going to take some more figuring out what HA is doing. It worked in v2 because v2 was not an integration and all state value updates were handled by HA. As an integration, the updates are handled by iC3.

Gotcha, if you have anything we can help with let us know and we’ll help test or whatever you need.

At my in laws and the ic3 shows the zone name, and the _2 shows the formatted name

Check configuration settings. Make sure it is friendly name. NoT entity name. What will display shows in Stage 2 off startup log when zones are set up

If I set it to friendly name, when I’m home it’s set to Home which makes my person entity not registered as actually being home if that makes sense.

Here is the beta 14. It has not been released yet so be sure to let me know if you run into anything weird.. Unzip into the icloud3 directory. See #3.
Here are all the changes in this beta:

iCloud Account Authentication - When the iCloud Account is verified using a password, a session access
token is created and stored as a cookie. The password login also generates an email saying someone has
logged into your iCloud account. This beta changes the method of authenticating the account to using
the token first and then using the password if it has expired. This should eliminate or greatly reduce
the login emails generated by Apple.
Configuration Wizard > iCloud Account Credentials screen - Updated it to improve readability and make
it easier to set up the account information. Several rare problems were also fixed where (1) logging
into an iCloud Account with an incorrect password would still allow access when there was a valid token
and (2) logging into an iCloud Account with a changed password would not work if the token has expired.
Now, logging in with a new password causes a new 2fa request to be made.
Zone enter/leave event for an iCloud3 device_tracker entity - This can now be used as an automation
trigger.
Latitude/longitude values - They are now stored as numeric fields in the device_tracker attributes instead
of a string field. They are now displayed with 2-decimal places on a Lovelace device_tracker/attributes
window (an HA limitation). A new gps attribute has been added to display the latitude/longitude with it's
full precision.
Passive Zones - Fixed a problem where passive zones could be selected. Also found out that the iOS App
will generate enter/exit triggers for passive zones. These will now be ignored by iCloud3.
Configuration Wizard > Sensors screen - Changed it  for readability and to list the important sensors
first.
Event Log - Made some minor formatting changes. Also added a new icon to the top-right of the Event Log
so you can become an iCloud3 Stargazer. Click it to go to the iCloud3_v2 GitHub repository. Then click
the star in the top-right corner.

icloud3_v3-beta-14.zip

Updated to 14 last night, and this morning my zone entry/exit automation works again! Thanks!

Glad it’s working for you.  I’ve been running with it for a few days. Since there are no HA docs, it took me a while to find a component with the code that was doing it. The base icloud component was no help. Turned out to be very simple and was being done by the mobile_app component.
b14 messed up the distance precision sensor values for distances less than 1 mile. Will have a fix a little later this morning.

This b14 version addresses sensor precision:
8.  Distance Sensors - (1) Changed the distance precision so the distance value is within 6-inches/
10-centimeters of the measurement between the device and the zone center. (2) Changed the
'_moved_distance' sensor to display in feet/meters when less than 1mi/1km. (3) Added moved from time
and moved to time to the '_moved_distance' sensor.
icloud3_v3-beta-14.zip


Rails 6 Support

Rails 6 has been officially released.
The current gem version does not support rails 6, due to the lock on to railties < 6 in the gemspec.
Is there any incompatibilities with railties 6?

I get in contact with fastly support and they said that this repo is not in active maintenance and going to be deleted in the near future

@waruboy could you perhaps post the exchange if you still have it? I'm a little bit worried about this gem being abandoned :D

@rhymes sorry, I can't post it. But the last time they said they are not maintaining this gem and intend to delete this repo, so I think this gem is indeed abandoned. I'd suggest you confirm with fastly support if you have doubt

@waruboy sent an email to fastly support, let you know what they tell me as well, thanks a lot!

@rhymes any news?

Sorry @ahnick, I did receive a response and forgot to update this issue. I've been told that fastly-rails is unmaintained and deprecated. We should use only fastly-ruby.
This is what they said:

Thanks for contacting support! We are not actively supporting the repo anymore. The code is really only useful as a teaching example, and I wouldn't recommend it's used in a production application without extensive integration work.
You can check https://docs.fastly.com/api/clients#ruby and https://github.com/fastly/fastly-ruby instead.

I suggested them to update the repo description so people won't start using it.


Capability to install to versions in parallel?

Good morning,
is there a capability to install the adapter in 2 versions (0.3.0 and 0.4.2)  with 2 instances?
Would make testing of the new version much easier.
Second question:
Is the ToDo: "Migration from < 0.4.0 to new datastructure" still open?
Thanks for your support!

no sorry the ioBroker framework does not allow to install 2 different versions of 1 adapter.


⚠ Bofrost.at has degraded performance

In 06f82a8, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 1092 ms

Resolved: Bofrost.at performance has improved in 5cf2c56 after 20 minutes.


Support for multiple outgoing IPs

Hi,
I'm looking to add support for multiple outgoing IPs (similar to Squid's tcp_outgoing_address) based on some sort of an arbitrary ACL.
Can one of the devs point me towards where the current selection is performed on the code body (i.e: where the connection is initialized)? I figured asking would be faster than fumbling through the code.
Thanks!

To forward all requests via a particular IP address of machine use proxyServer.UpStreamEndPoint
https://github.com/justcoding121/Titanium-Web-Proxy/blob/develop/Titanium.Web.Proxy/ProxyServer.cs#L233
To forward an per request basis from client use args.WebSession.UpStreamEndPoint.
https://github.com/justcoding121/Titanium-Web-Proxy/blob/develop/Titanium.Web.Proxy/Http/HttpWebClient.cs#L31
Note that changing connections too often on a per request basis will affect performance. See below for what happens internally.
https://github.com/justcoding121/Titanium-Web-Proxy/blob/develop/Titanium.Web.Proxy/RequestHandler.cs#L363
On a per host (server hostname) basis would be better args.WebSession.Request.RequestUri.Host.

Hi @justcoding121
Thanks for that, I'll try it out.
Is there a way to tell which local endpoint (ExplicitEndpoint) a request came in via? I tried looking into the OnRequest(object sender, SessionEventArgs eventArgs) arguments, but don't think I noticed it.

So, this didn't seem to exist, but adding it was quite trivial. I've created a pull request, can you please review? @justcoding121

May I close this issue?

@honfika Absolutely.


'Question: [RCTView support cursor property]'

Now i need to modify cursor when mouseEntered and restore it after mouseExited.
if can export a prop likes cursor, eg: <View cursor="pointingHand"/>
Thanks.

I resolved it by onMouseEnter and onMouseLeave event hooks,
And set cursor by a nativemodule react-native-app-interface!

nativemodule react-native-app-interface

could you please shed some light how did you do that ? I have installed "react-native-app-interface" via npm. What should be done next? Could you paste some code sample please?


I created a parser that infers the offset and length
I punted on reverse engineering the tags in the idx file.
I run scangrib on each group and map one to one in order from the idx file to the scangrib group the idx tags.
I am traveling this week - happy to follow up more next week.

@martindurant the notebook in that directory provides the overview of how to use the these methods. The pangeo showcase talk has my narrated version.

FWIW, I've started tinkering with .idx parsing in Rust (in my nascent hypergrib project).
@mpiannucci please shout if you'd prefer the .idx parsing to live in gribberish!
Rust's csv crate makes it pretty easy to get started:
#[derive(PartialEq, Debug, serde::Deserialize)]
struct IdxRecord {
    msg_id: u32,
    byte_offset: u32,
    init_time: String,       // TODO: Use DateTime<Utc>
    nwp_variable: String,    // TODO: Use NWPVariable enum?
    vertical_level: String,  // TODO: Use VerticalLevel enum?
    forecast_step: String,   // TODO: Use TimeDelta?
    ensemble_member: String, // TODO: Use EnsembleMember enum?
}

/// `b` is the contents of `.idx` file.
fn parse_idx(b: &[u8]) -> anyhow::Result<Vec<IdxRecord>> {
    let mut rdr = csv::ReaderBuilder::new()
        .delimiter(b':')
        .has_headers(false)
        .from_reader(b);
    let mut records = vec![];
    for result in rdr.deserialize() {
        records.push(result?);
    }
    Ok(records)
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_parse_idx() -> anyhow::Result<()> {
        let idx_text = "\
1:0:d=2017010100:HGT:10 mb:anl:ENS=low-res ctl
2:50487:d=2017010100:TMP:10 mb:anl:ENS=low-res ctl
3:70653:d=2017010100:RH:10 mb:anl:ENS=low-res ctl
4:81565:d=2017010100:UGRD:10 mb:anl:ENS=low-res ctl
";
        let records = parse_idx(idx_text.as_bytes())?;
        assert_eq!(records.len(), 4);
        assert_eq!(
            records[0],
            IdxRecord {
                msg_id: 1,
                byte_offset: 0,
                init_time: String::from("d=2017010100"),
                nwp_variable: String::from("HGT"),
                vertical_level: String::from("10 mb"),
                forecast_step: String::from("anl"),
                ensemble_member: String::from("ENS=low-res ctl"),
            }
        );
        Ok(())
    }
}

Thanks @JackKelly . Take whatever you need from the kerchunk files and outstanding PRs on the topic, of course. Most of that is not around the idx files themselves (which are simple) but how to map them onto sets of grib files, and how to query these mappings to make logical data sets.

@mpiannucci I see that commit 01745ada07d892a6059930958ffe7ae12998abc8 implemented MessageMetadata::as_idx(&self, index: usize) -> String.
Is the intention of this github issue to implement a MessageMetadata::from_idx(idx_data: &[u8]) -> MessageMetadata method?

It would be great to have a complete solution in rust but if you want to jump start the real work in rust is reading the grib files. You should be able to change the codec in these demo files pretty easily if you want to experiment.

This is sooo cool to see!

Is the intention of this github issue to implement a MessageMetadata::from_idx(idx_data: &[u8]) -> MessageMetadata method?

Yeah that was my intention, but i clearly have not gotten to build it. I would LOVE to have this as part of the gribberish repo if folks are open to that.

@emfdavid wrote:

the real work in rust is reading the grib files

I hear you: The main performance benefits of using Rust will come from reading the grib files themselves, rather than the .idx files. But, for the MVP of hypergrib, I'm starting by "cheating" and creating my manifest by reading existing .idx files (although, soon after the MVP, I will have to read the grib files to create my manifest because the .idx files don't include essential information like the grid definition... maybe gribberish could define and generate "extended .idx" sidecar files which include additional metadata about each message?!).
For the MVP of hypergrib I'm focused on the NODD GEFS dataset (which already has .idx files). So, I agree that tinkering with .idx files in Rust isn't necessarily the fastest way to a big performance boost. But reading .idx files in Rust is a requirement of the hypergrib MVP :slightly_smiling_face:.
Plus, I'd love to help with gribberish. If only because it has such an awesome name :slightly_smiling_face:

@mpiannucci wrote:

I would LOVE to have [a MessageMetadata::from_idx function] as part of the gribberish

Great! Let's make it happen :slightly_smiling_face:
First, please may I ask a few quick questions about how best to approach this?
1) Where do I find the most authoritative definition of .idx files?
Is the best "definition" the source code for wgrib?
2) .idx files don't include all the required fields in MessageMetadata
For example, the body of an .idx file doesn't include data_compression, proj, or crs.
Some potential paths forward might include:

Change the definition of struct MessageMetadata so almost all the fields are Option

Advantages: Conceptually simple.
Disadvantages: Does downstream code in gribberish require that MessageMetadata is fully populated? If so, all that code will have to be changed to handle metadata fields that are set to None.

Force the user to supply missing metadata. e.g. implement a Builder pattern whereby some (but not all) required fields can be populated from an .idx file. But the user will have to supply the remaining fields before they're allowed to construct a MessageMetadata instance.

Advantages: Ensures gribberish has all the metadata it needs. Minimal changes to downstream gribberish code.
Disadvantages: Makes life much harder for users!

Hardcode the missing metadata for each dataset into gribberish such that gribberish knows, for example, which data_compression is used by GEFS. The user would just have to supply an .idx file and tell gribberish which dataset that .idx file comes from (GEFS, HRRR, etc.)

Advantages: Makes life easy for users! And guarantees that downstream gribberish code gets all the fields it expects.
Disadvantages: More work for us! We'd have to hardcode data for lots of datasets! (But there aren't that many NWPs). And what would gribberish do if it comes across an .idx files from a dataset it doesn't know about?

We define an "extended idx" format :slightly_smiling_face:

I quite like this idea. But let's park it for now :slightly_smiling_face:

.idx files don't include all the required fields in MessageMetadata

This is exactly what we deal with in kerchunk - we scan the first of a set of files to get all the data we need, and then use the .idx files for all the rest to infer the complete data for all of them. From your point of view, this is somewhere between solutions 2) and 3).

That's great to know, thanks @martindurant. Sorry to ask a slightly off-topic question but: What does kerchunk do if the metadata changes over the course of a dataset? e.g. an NWP dataset which is, say, 0.5 degree horizontal resolution from 2015 to 2020. But the NWP gets upgraded in 2021 to 0.25 degree resolution? (I've been worrying about this exact issue for hypergrib recently. I don't have a great solution!)

It doesn't "do" anything, you would I think have to process the two batches separately.
If the chunking is actually different in the overall dataset, then this can't be represented in the zarr model at all without at least variable chunking, and maybe not at all. This is why virtualizarr is a more general solution, and the two projects should probably be tied more closely together :)

On the topic of how to handle .idx metadata in gribberish...
I'm sure everyone has considered this already but it feels like there are three distinct types of message metadata:

The metadata that locates the 2D grib message in the huge, virtual, 5-dimensional array that xarray thinks about. Specifically, those 5 dims are the reference time, forecast step, product (AKA variable), vertical level, and the ensemble member.
The physical location of the grib message in storage: the path, offset, and length.
Metadata that helps us interpret the grib message: the crs, proj, data compression, etc.

IIUC, the .idx file always gives us everything we need for "metadata type 1" and gives us 2 or the 3 fields we need for "metadata type 2".
Maybe a 5th option (following on from the 4 "potential paths forward" listed above) would be to split gribberish's MessageMetadata into 2 or 3 separate structs?

These are exactly the issues we struggled with last winter. @martindurant and @Anu-Ra-g did a great job adding documentation about how the the kerchunk solution works.
What we found is that the idx file does not provide compete Type I metadata - at least not without reverse engineering the wgrib fortran code that wrote the "attrs" column which includes the product level and step in a form better suited to humans than machines.

The solution implemented for kerchunk is to build a mapping from this attrs string to the Type I meta data you need to locate each chunk in the hyper kube.
Resulting a table that stores all the type I & II data

From this, and that static metadata about compression, dimensions and product attributes you can than construct any logical dataset you like.
Operationally we know when NOAA is going to change the model - it is an operational product, but just to be sure, I compare reading the grib file directly with parsing the idx file for 1 in 1000 files to make sure nothing has changed.
I would love to get rid of the mapping and improve the api for building the logical dataset, but I hope understanding one working solution can give you a boost to find a better way.

And the name gribberish is totally awesome!

This is all super-useful, @emfdavid!
I think I'm up for trying to fully understand the wgrib source code, and to build something in gribberish that can understand any .idx file created by wgrib. That said, I haven't fully digested what this entails, so please don't hold me to this if it turns out to be far more work than I'm imagining! Maybe my first step will be to convert the wgrib source code to a human-readable spec for .idx files, if such a spec doesn't already exist?
On the topic of scanning grib files... One of my ultimate goals is to (help) run a service that maintains a public manifest of NODD grib datasets. So users can just run xr.open_dataset(MANIFEST_URL, engine="hypergrib") to lazily open petabyte-scale grib datasets. I'm talking myself into the idea that the "public manifest service" will have to scan every grib file that NODD publishes, so it knows exactly when the dataset changes (e.g. when the NWP upgrades its horizontal resolution etc.). Which is one of the reasons I'm interested in making our own "extended .idx" format! Or using your trick, @emfdavid, of storing the manifest in a database!
(Actually, I'm hoping I won't have to scan the entirety of every grib file. I'm hoping I can use the existing .idx files to tell me where each message begins, and only "scan" the metadata sections, not the binary payload. Although reading lots of small byteranges may be inefficient on cloud object storage...)

build something in gribberish that can understand any .idx file created by wgrib

This is fantastic - the data driven mapping between the IDX files and the cf grib metadata is definitely a bit of a hack. If you can build and maintain tables/code based on wgrib that would be strictly better.

So users can just run xr.open_dataset(MANIFEST_URL, engine="hypergrib") to lazily open petabyte-scale grib datasets

The full dataset for all products, all steps & runtimes is billions of chunks for a single perturbation of GEFS.
Would you have different MANIFEST_URLs for different slices or maybe a time range operator as part of the url? It will be a very large manifest response. When you are ready, I would like to discuss the 'reinflate' method we built in kerchunk to turn a manifest and an axis specification into a logical dataset.

ultimate goals is to (help) run a service that maintains a public manifest

I would love to see NODD actually build and maintain the manifest database for us. I think we could do it if the community comes together around a table schema. I think there is funding and a path to implementation.

@rabernat , you might have some thoughts on the DB approach being sketched out here and related kerchunk threads. This is not exactly a kerchunk manifest, but a set of chunk details that can be made into various datasets, but no one dataset can use all the references (they wouldn't fit in the data model, with mutually conflicting coordinates).

I would love to see NODD actually build and maintain the manifest database for us.

This is the big goal! No one of us wants to read however many PB of data they have, or support the storage/query interface it would need to use. Making manifests for some specific view is doable, but the data is always being updated, so there needs to be a process too.

Have NOAA hinted that they might have appetite for maintaining a public manifest for NODD?

The full dataset for all products, all steps & runtimes is billions of chunks for a single perturbation of GEFS

Yeah. I'm interested in defining a very concise manifest format but, even if the manifest only uses a single byte per chunk - which sounds impossibly concise(!) - then we're still looking at a manifest that's gigabytes in size. These dang NWP datasets are so BIG! Hmm, I see what you mean... this definitely needs more thought... It feels solvable though...

(BTW, I'm going to propose that we move discussion of giga-scale manifests etcetera to the hypergrib discussion forum, so we can focus this GitHub issue on the question of how gribberish can handle .idx files... I'll post a link to relevant hypergrib discussions within a few hours. Sorry, my fault for getting carried away!)

Yes - if we do determine we need a manifest and we came up with a information model that supports the community needs (hypergrib, virtualizarr, kerchunk) I think we could definitely find funding and enthusiastic support from NODD to operate it.
If we can deliver a tool that doesn't need the manifest at all and can load whole data sets looking up chunks algorithmically - as you are now proposing - well that would be even better!

@mpiannucci I've made a start on parsing .idx files into gribberish's parameters.
I don't (yet) have great Rust macro skills. Do you think it should be possible to write a Rust macro to automatically convert a string like "TMP" to gribberish::templates::product::parameters::meteorological::TemperatureProduct::Temperature, using the #[abbrev = "TMP"] annotation that you've already painstakingly defined?!
At the moment, I'm manually mapping from abbreviation strings to the appropriate enum variant (here's my code so far - we can move this into gribberish if you'd like).

I'm afraid I'm moving away from the idea of parsing .idx files into gribberish types (sorry!). For the hypergrib MVP, I'm going to keep the parameter abbreviations and levels as strings.
After the MVP, hypergrib will decode the parameter abbreviations and levels, but will probably use the GRIB2 tables encoded as .csv files in gdal.

That looks like a good resource for a machine readable form... but to get all the NOAA special variables you will have to take a look at the ncep libs library. I opened an issue asking for the data to be exposed in a machine readable form, but I don't think it has happened yet.

Sorry, I linked to the wrong gdal path in my comment above. This path contains more GRIB tables as CSVs: gdal/frmts/grib/data
confusingly, the README for that directory is here: gdal/frmts/grib/degrib/README.TXT

to get all the NOAA special variables you will have to take a look at the ncep libs library

Does this CSV contain what you need: gdal/frmts/grib/data/grib2_table_4_2_local_NCEP.csv

a useful contribution would be "just" to collate all these GRIB code tables into a single place

Yes! I wonder how often new entries are added... Of course, many "definitions" come with an implementation too, for example all the coordinate projections.

After posting the comment where I said:

a useful contribution would be "just" to collate all these GRIB code tables into a single place

I discovered that the GDAL codebase appears to contain a bunch of vendor-specific tables:

Is this sufficient? Does gdal already contain all the tables we need?

Obviously I don't know, but I wouldn't be too surprised if GDAL was mostly on top of this. Of course, how up to date that is, is another matter - but probably they have a pretty active user base pushing for updates as they arise.

The trouble is that there are multiple formats for the code tables. GDAL is probably the most comprehensive resource

I hate the code tables impl in gribberish and was going to codegen them but never had the time.

Oh, I'm in awe of your pure-Rust representation of the GRIB tables! Reading your code genuinely expanded my understanding of what can be represented in pure Rust!
Making no promises... but I'm wondering about writing a new Rust crate which:

Contains a copy of GDAL CSVs (with a README which gives full attribution to GDAL, of course!)

A subsequent PR could convert the CSVs to a binary format for better performance at runtime.

Contains some simple Rust code to read the GRIB tables into memory to provide a way to look up information?

This is definitely more of a people problem than a technical problem.
The existing gdal work certainly looks the most comprehensive - a great find!
I wonder if osgeo.org would consider moving the data to a separate data only repository and treating it as a proper versioned dependency that could be easily shared?
It would be great to find the correct docs to link from NCEP and WMO/ECMWF and the reference implementations they maintain.

I wonder if osgeo.org would consider moving the data to a separate data only repository

Good question. I've just requested to be subscribed to the gdal-dev mailing list, so I can ask this question! (The gdal github issues page says that github issues are only for feature requests and bug reports)

On the topic of generating Rust code which represents the GRIB code table... I'm excited about this approach! I've started a new issue: #63

can we now parse the "level" and "step"

Not yet! I haven't written any code to parse the body of the .idx files yet. (Although, when we do write code to parse the body, we could use nom).
I'm afraid I won't get round to parsing "level" and "step" from the body of .idx files for a while (a few months). For the hypergrib MVP I'm initially focused on GEFS. And, in GEFS, a bunch of information is available in the .idx filesnames in a form that's easier to parse than the body of the .idx. Specifically, the GEFS filenames include the initialisation datetime, the ensemble member, and the step.
I will extract the level string and product abbreviation from the body of the .idx files, but for the hypergrib MVP I'm just gonna leave these as strings.
My plan for the development of hypergrib is here. The ultimate plan is definitely to fully parse this information.
Also, before we can parse the body of .idx files I'm guessing we should first complete #63

My little grib_tables Rust crate is now available: https://crates.io/crates/grib_tables
grib_tables just loads the GDAL CSV files into memory and allows the user to map from the parameter abbreviation string (e.g. "VTMP") to the full details of the param. You can also map from the numeric parameter ID to the full param details.

Thanks @JackKelly ! The source code is at https://github.com/JackKelly/hypergrib/tree/main/crates/grib_tables , if that wasn't obvious.

Oooh, good point, I've just updated the repository in the Cargo.toml to point to that crates/grib_tables subdirectory!


Caret is able to select within emoji characters on the second line of text with emojis

Type multiple lines of emojis.
The caret is able to go in between the emojis, but only on the second line and further.
The caret is unable to appear in the first line.

This casues a crash on iOS.  Steps to reproduce:
enter multiple lines of emoji's
move carret to middle of emoji
type any letter
The cause of the crash is that the JSON going into and/or coming back from the engine becomes invalid UTF8, and we end up asserting that we didn't get a null return from the JSON routines.
I spent a while trying to see if I could make them tolerate this case, but it seems really difficult to get iOS to do that and would be best to just fix this bug instead.
See also #31468

The linked engine patch makes this not be a crash.  I'll remove the label once that lands.

Should be fixed by https://github.com/flutter/engine/pull/10063


Audit, and if possible, test, if( failed ) in database

This logic in particular:  https://github.com/cryptonomex/graphene/blob/d827c5df281eafe43d6d0d00533ea0a0f2643c1c/libraries/chain/db_block.cpp#L298-L309

Could/should it be triggered by tx expiring between making it into the pool and making it into a block?

Totally rewritten in current version, this ticket is obsolete.


Implement a way to tell if the remote connection was closed

It would be very helpful if I was able to tell that the remote server closed the connection.
From my conversation with @Lukasa, he said it "...Should be possible for the code to work out if the conn was closed ‘cleanly’ (I.e. with no outstanding work)"
For background:
When I use Hyper to connect to Apple's Push Notifications Servers, they require a persistent connection to be kept with their servers, but they can close the connection at any time. My implementation would be simplified if I had a way to determine that they had closed the connection before attempting to send a notification. (I'd still have to catch exceptions, but I could cleanly attempt a reconnection before trying to send a notification, as most disconnections are due to idle timeout)

So, here's my thoughts on this.
The easiest way for us to be able to do this is to enhance the BufferedSocket implementation with a fill_nowait method, that avoids blocking by only making a recv_into call if the socket is readable. That can then spot connection termination and also look for control frames like GOAWAY that good implementations will issue before connection termination.
We could then call that method before any method that sends data. We could then spot connection termination and check whether or not the connection is in a relatively clean state (no streams outstanding), and then take advantage of that to cleanly reconnect and do other nice things.
This is trivial in the single-threaded case, but in the multi-threaded case everything gets worse. This is because reading from that socket needs to become serialized, and we don't necessarily want to block any data sending method behind a socket read.
@tbetbetbe do you have suggestions for how to deal with that?

I like the idea of the fill_nowait method
for simplicity, it's best if all behaviour that needs synchronization happens in one place so I think the detection of the go-away frame should happen in connection.py
For multithreading, I think we can still call the fill_nowait method before sending, but wherever we do, we should make sure the caller owns the connection-wide lock.

This is because reading from that socket needs to become serialized, and we don't necessarily want to block any data sending method behind a socket read.

is this really a problem ?  We already serialize access by requiring any reader/writer to acquire the connection object before acting.  As long as we kept the goaway detection logic within the existing synchronized sections, the synchronization cost should be no worse in the cases where there is no goaway frame.

is this really a problem ? We already serialize access by requiring any reader/writer to acquire the connection object before acting. As long as we kept the goaway detection logic within the existing synchronized sections, the synchronization cost should be no worse in the cases where there is no goaway frame.

Heh, I think that's what I'm asking you. You've been using the project multithreaded in anger, so you've got a much better feel for what it seems like.
If you're optimistic that it won't hurt, then I'm happy to prototype this and you can play around with it in the multithreaded case and see if it hurts too much.


error: OpenCV(4.0.0) /io/opencv/modules/dnn/src/caffe/caffe_io.cpp:1132: error: (-2:Unspecified error) FAILED: fs.is_open(). Can't open "frozen_east_text_detection.pb" in function 'ReadProtoFromBinaryFile'

Expected behaviour
I have installed opencv-python  OpenCV(4.0.0)  but seems it cannot find 'frozen_east_text_detection.pb' when running cv2.dnn.readNet
error: OpenCV(4.0.0) /io/opencv/modules/dnn/src/caffe/caffe_io.cpp:1132
Actual behaviour
---------------------------------------------------------------------------
error                                     Traceback (most recent call last)
<ipython-input-15-9b59c98d5831> in <module>()
      4
      5 CTextDetector = ClassTextDetector()
----> 6 boxes = CTextDetector.getTextRegions_EAST(inputImg_1, W, H)

~/disk_user/OCR_project/Classes/ClassTextDetector.py in getTextRegions_EAST(self, inMat, outW, outH)
     20     def getTextRegions_EAST(self, inMat, outW, outH):
     21         layerNames = ["feature_fusion/Conv_7/Sigmoid", "feature_fusion/concat_3"]
---> 22         net = cv2.dnn.readNet('frozen_east_text_detection.pb')
     23
     24         blob = cv2.dnn.blobFromImage(inMat, 1.0, (outW, outH),(123.68, 116.78, 103.94), swapRB=True, crop=False)

error: OpenCV(4.0.0) /io/opencv/modules/dnn/src/caffe/caffe_io.cpp:1132: error: (-2:Unspecified error) FAILED: fs.is_open(). Can't open "frozen_east_text_detection.pb" in function 'ReadProtoFromBinaryFile'

Steps to reproduce

example code

inputImg_1 = thresh.copy()
W = int(inputImg_1.shape[1]/32) * int(32)
H = int(inputImg_1.shape[0]/32) * int(32)

CTextDetector = ClassTextDetector()
boxes = CTextDetector.getTextRegions_EAST(inputImg_1, W, H)

def getTextRegions_EAST(self, inMat, outW, outH):
        layerNames = ["feature_fusion/Conv_7/Sigmoid", "feature_fusion/concat_3"]
        net = cv2.dnn.readNet('frozen_east_text_detection.pb')

        blob = cv2.dnn.blobFromImage(inMat, 1.0, (outW, outH),(123.68, 116.78, 103.94), swapRB=True, crop=False)
        start = time.time()
        net.setInput(blob)
        (scores, geometry) = net.forward(layerNames)
        end = time.time()
        # show timing information on text prediction
        print("[INFO] text detection took {:.6f} seconds".format(end - start))

        #print(scores.shape)
        #print(scores)
        (numRows, numCols) = scores.shape[2:4]
        rects = []
        confidences = []

        # loop over the number of rows
        for y in range(0, numRows):
            # extract the scores (probabilities), followed by the geometrical
            # data used to derive potential bounding box coordinates that
            # surround text
            scoresData = scores[0, 0, y]
            xData0 = geometry[0, 0, y]
            xData1 = geometry[0, 1, y]
            xData2 = geometry[0, 2, y]
            xData3 = geometry[0, 3, y]
            anglesData = geometry[0, 4, y]

            # loop over the number of columns
            for x in range(0, numCols):
                # if our score does not have sufficient probability, ignore it
                if scoresData[x] < 0.5:
                    continue

                # compute the offset factor as our resulting feature maps will
                # be 4x smaller than the input image
                (offsetX, offsetY) = (x * 4.0, y * 4.0)

                # extract the rotation angle for the prediction and then
                # compute the sin and cosine
                angle = anglesData[x]
                cos = np.cos(angle)
                sin = np.sin(angle)

                # use the geometry volume to derive the width and height of
                # the bounding box
                h = xData0[x] + xData2[x]
                w = xData1[x] + xData3[x]

                # compute both the starting and ending (x, y)-coordinates for
                # the text prediction bounding box
                endX = int(offsetX + (cos * xData1[x]) + (sin * xData2[x]))
                endY = int(offsetY - (sin * xData1[x]) + (cos * xData2[x]))
                startX = int(endX - w)
                startY = int(endY - h)

                # add the bounding box coordinates and probability score to
                # our respective lists
                rects.append((startX, startY, endX, endY))
                confidences.append(scoresData[x])

        # apply non-maxima suppression to suppress weak, overlapping bounding
        # boxes
        boxes = non_max_suppression(np.array(rects), probs=confidences)

        return boxes

operating system
Ubuntu 18.04.1 LTS
architecture (e.g. x86)
opencv-python version
4.00

That file is not shipped with opencv-python. Additionally, I think you will have to have that file in your working directory for cv2.dnn.readNet('frozen_east_text_detection.pb') to work properly.


Update b.md

Added Branch to the list.
@gray-adeyi @OSCA-Ado-Ekiti @Elijah699

@Victorayo Kindly make all your changes just a single PR, you just splited the PR which can rasie issues of spamming.

@Victorayo Kindly make all your changes just a single PR, you just splited the PR which can rasie issues of spamming.

It can't be made as a single PR. The directories are different. One is MD file, while the other is a J's file.. @Elijah699
Instead of closing this. Kindly open it back and merge.
The PRs can only be recognized as spam if it was sent twice. This isn't.


Habit Tracker App in Flutter

Is your feature request related to a problem? Please describe.
Beginners might be confused as to where to start with designing apps in Flutter
Describe the solution you'd like.
This app tutorial will be easy to follow, and will also teach readers the importance and implementation of good UI, how to integrate other tools for backend such as Hive, and create a functioning app easily.
Describe alternatives you've considered.
The best way to learn a tool is by practicing it. So beginners can use this project to learn how to use Flutter to build amazing apps and the third party tools that can be integrated to provide additional features that are used in most apps today.
Add any other context or screenshots about the feature request here.
I am a GSSOC '24 contributor and I am working on many Flutter apps currently, so I can easily work on this. Please assign this to me

@Kushal997-das Since this is a level 1 issue, is it okay if I make the app that depicts its UI without using firebase? I can parallelly raise another issue where I create an app with both frontend and backend. Please let me know if this works fine with you.

@Kushal997-das could you please let me know? I am done with all the simple functionalities since this is a level 1 issue, I can add progress trackers and calendars as well if you ask me to. Thanks!

@vamzx Don't focus on level1 or level2 initially. Instead, concentrate on making high-quality contributions. If your PR is of good quality, we will assign an appropriate level. However, please note that only one PR is allowed per user in my repo for the first time. Avoid creating multiple PRs; just consolidate all your changes into a single PR.
Note: Ensure that the same project is not added to multiple repos, as it will be marked as spam.

@Kushal997-das Sure, got it! So I have two issues assigned at the moment, one is for a game and one is for this tracker. Should I raise my PR with both the issues together?

@Kushal997-das I have been trying to raise a PR but it says "Nothing to compared, both have different commit histories"
Could you please help me out? I have finished worked on the project, I am just not able to raise a proper pull request for some reason

@vamzx Refer this

I would like to work on this issue. Can you assign it to me

@Kushal997-das Could you please check the PR for this issue?


Remove glossary left-overs

Looks like removing the front-matter in f17ebae568a15033c914d2024f5e939dee89a4a8
caused the output to break, resulting in a JavaScript error.
Looking at where this file was used, it turned out that it was loaded, but
never used anywhere.
This commit removes the remaining parts of the glossary search functionality,
which was not used.

@StefanScherer ptal 🤗


Issue line/column information with parsing errors

That would simplify debugging quite a bit :)

This issue was moved to scala-ide/scalariform#204


GH-35056: [Python][CI] Don't install gdb on Windows

Because it's not available on Windows.
Rationale for this change
The gdb conda package isn't available on Windows:
https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow/builds/46748401#L266
C:\projects\arrow>mamba create -n arrow -y -c conda-forge   --file=ci\conda_env_python.txt    --file=ci\conda_env_gandiva_win.txt --file=ci\conda_env_cpp.txt    "ccache"   "cmake"   "ninja"   "nomkl"   "pandas"   "fsspec"   "python=3.10"     || exit /B
conda-forge/win-64                                          Using cache
conda-forge/noarch                                          Using cache
Looking for: ['boto3', 'cffi', 'cython', 'cloudpickle', 'fsspec', 'gdb', 'hypothesis', "numpy[version='>=1.16.6']", 'pytest', 'pytest-faulthandler', 'pytest-lazy-fixture', "s3fs[version='>=2021.8.0']", 'setuptools', 'setuptools_scm', "clangdev[version='<15']", "llvmdev[version='<15']", 'aws-sdk-cpp==1.10.13', "benchmark[version='>=1.6.0']", "boost-cpp[version='>=1.68.0']", 'brotli', 'bzip2', 'c-ares', 'cmake', 'flatbuffers', 'gflags', 'glog', "gmock[version='>=1.10.0']", "google-cloud-cpp[version='>=1.34.0']", 'grpc-cpp', "gtest[version='>=1.10.0']", 'libprotobuf', 'libutf8proc', 'lz4-c', 'make', 'ninja', 'orc', 'pkg-config', 'python', 'rapidjson', 're2', 'snappy', "thrift-cpp[version='>=0.11.0']", 'xsimd', 'zlib', 'zstd', 'ccache', 'cmake', 'ninja', 'nomkl', 'pandas', 'fsspec', 'python=3.10']
Could not solve for environment specs
The following package could not be installed
└─ gdb   does not exist (perhaps a typo or a missing channel).
C:\projects\arrow>set lastexitcode=1

What changes are included in this PR?
Don't install gdb on Windows.
Are these changes tested?
Yes.
Are there any user-facing changes?
No.

Closes: #35056

Do we want to add gdb to conda_env_unix to keep the number of conda files down? It may be a little imprecise.

gdb is only needed for conda-python images. conda-cpp images don't need it.
But I don't have strong opinion for this. Should I use ci/conda_env_unix.txt?

OK. I've moved gdb to conda_env_unix.txt.

@github-actions crossbow submit test-conda-python-3.10

Appveyor is still failing, but now it's an issue in the tests (not in setting up the env). Seems to be hanging / taking too much time.


[BUG] Data Prepper process threads stop when processors throw exceptions

Describe the bug
When a processor throws an exception, the current Data Prepper processor thread stops processing. Data Prepper is not shutting down the pipeline, nor is it attempting to restart the thread.
To Reproduce
Create a pipeline with a processor that will throw an exception. The date processor throws an exception when the date_when value has in valid expression.
Run Data Prepper and send data through.
The threads stop.
Expected behavior
There are two things I'd expect:

The whole pipeline shuts down
Data Prepper attempts to restart the thread

Analysis
The exact cause is this line:
https://github.com/opensearch-project/data-prepper/blob/2be8166ac8ac1f80784201f2a6e33302145cb5a8/data-prepper-core/src/main/java/org/opensearch/dataprepper/pipeline/ProcessWorker.java#L91-L93
The ProcessWorker catches the exception. Then the ProcessWorker::run method exits. After this, the thread remains, but it is waiting on a task to run.
You can see that the thread remains with a thread dump:
"my-pipeline-pipeline-processor-worker-1-thread-1" #41 prio=5 os_prio=0 cpu=2889321.30ms elapsed=70643.67s tid=0x0000aaaadebd3310 nid=0xa8 waiting on condition  [0x0000ffff08bdf000]
   java.lang.Thread.State: WAITING (parking)
	at<EMAIL_ADDRESS>Method)
	- parking to wait for  <0x00000005e1d1c890> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.22/LockSupport.java:194)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.22/AbstractQueuedSynchronizer.java:2081)
	at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.22/LinkedBlockingQueue.java:433)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.22/ThreadPoolExecutor.java:1054)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.22/ThreadPoolExecutor.java:1114)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.22/ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(java.base@11.0.22/Thread.java:829)

Environment (please complete the following information):
Data Prepper 2.6.1

I propose a couple of solutions:

Catch exceptions from the processor and then drop that batch.

We can have a try-catch around this code:
https://github.com/opensearch-project/data-prepper/blob/2be8166ac8ac1f80784201f2a6e33302145cb5a8/data-prepper-core/src/main/java/org/opensearch/dataprepper/pipeline/ProcessWorker.java#L129-L136

When the processor thread fails, let it throw an exception. It will be handled by the PipelineThreadPoolExecutor.

https://github.com/opensearch-project/data-prepper/blob/cf86674ab2e5098361d802d567e6eb5955a835b0/data-prepper-core/src/main/java/org/opensearch/dataprepper/pipeline/common/PipelineThreadPoolExecutor.java#L63-L85


⚠ ClamAV has degraded performance

In 29da018, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 68 ms

Resolved: ClamAV performance has improved in da65bc0.


Adds new db to redis-commander service

What do these changes do?
A new db for tracking cached content was added to Redis.
Related issue/s
Related PR/s
Checklist

[ ] I tested and it works

is this stale?
@GitHK

@YuryHrytsuk nope, did not got back to this PR chain. I'm a bit delayed on it. Will be taking care of it in the next week.


Email subscriptions for document collections

When a user signs up to alerts on a document collection, research
has shown an expectation of being notified when changes are made to
documents within that collection.
The relationship between a document collection and the documents it
contains is stored in the links attribute sent to this application by
email alert service. See [1]
The current logic that matches subscriber lists to a content change
matches on content ID AND tags or links. This code was added in [2] when we began
supporting single page email notifications. I've followed that pattern here. eg
if the ContentChange has tags as well as a content id, the SubscriberListQuery will
only match it to a subscriber list if it matches on the tags AND the content ID.
In the case of a user being subscribed to a single page AND a collection
that includes that page, they will recieve two emails.
[1] https://github.com/alphagov/email-alert-service/pull/577
[2] https://github.com/alphagov/email-alert-api/commit/069ef4eafe87b751ce7cf352f8137b7bd99ac0fd
Trello https://trello.com/c/tJvdhfdd/1455-support-topic-like-subscriptions-for-document-collections-email-alert-api-s
⚠ This repo is Continuously Deployed: make sure you follow the guidance ⚠

We will no longer be making changes to email alert api to support this functionality. I will update this comment with a link to the PR's in email alert frontend and email alert service once they're open.


Move getGenericListPropertyNames out of RLMObjectBase

I think this is preferable to having the PRIVATE prefix.

You ok with this @jpsim ?


[bugfix] deckgl legend is not interactive

Somehow when clicking categories in the legend, nothing happened. The
problem seemed to be around the getDerivedStateFromProps override
which gets triggered after each render and would alter the state and
reset the categories state to their origin.
Changed the method name to not be an override and just be called once
in the constructor
https://reactjs.org/docs/react-component.html#static-getderivedstatefromprops

Codecov Report

Merging #6365 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #6365   +/-   ##
=======================================
  Coverage   77.31%   77.31%
=======================================
  Files          67       67
  Lines        9581     9581
=======================================
  Hits         7408     7408
  Misses       2173     2173

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8b2cae0...96ce982. Read the comment docs.

@hughhhh 👁 👁


Tests returning Tasks should not simply return completed tasks

We should report an issue in case a test that returns a Task returns a Task.CompletedTask. In such case the test method can be of type void.

Be aware of async.


ActionController::RoutingError (No route matches [GET] "/users")

Hello,
I have configured devise 4.3.1 with rails 5.1.1. But i have facing one problem.
The scenario is that.
Step 1 click on sign up button at that time it will redirect to "http://localhost:3000/users/sign_up"
Step 2 click on submit button without fill any value.
Step 3 It will redirect to ""http://localhost:3000/users".
Step 4 in url ""http://localhost:3000/users" at that time do Hard Refresh.
Step 5 It gives error link "No route matches [GET] "/users".
Please update me ASAP.
routes.rb
Rails.application.routes.draw do
devise_for :users , :controllers => { sessions: 'users/sessions', registrations: 'users/registrations' }
get 'home/index'
root to: "home#index"
end

That is normal because there is no route for GET / users.
You can customize the devise controller to redirect back to / users / sign_up or create a route for GET / users.


[WIP] alternate AnnotationData backend

This is  work in progress, but implements an alternate backend for annotation data.
TODO:

[x] trim
[x] slice
[x] deep serialization (eg for vector format)
[ ] pretty printing
[x] purge all pandas references

This change is

Okay, this PR is now functional in that all tests pass again.  So if you never touched a JamsFrame object directly in application code (and there are few reasons to do so), then everything should just work.
That said, the design is up for discussion though, particularly seeking input from @justinsalamon and @ejhumphrey.
The new implementation replaces the JamsFrame object by an AnnotationData object, which at the schema level covers both DenseObservation and SparseObservationList, though not using JObjects because I don't think we should have sandbox support in the observation list.
Under the hood, AnnotationData encapsulates the observations with an ordered (by time) list of sparse observations.  Observations are implemented as named tuples (time, duration, value, confidence), again not using JObject so that we can preserve slots and field ordering.
This namedtuple implementation also makes individual observations immutable, which I see as a Good Thing. If you want to modify observations, it's best to make a new annotation object and re-insert them after modification.  Note that this is the reverse of how the DataFrame implementation worked: the individual observations there were mutable (it's easy to modify entries in-place), but the container itself was much more rigid (adding and deleting observations was difficult).  I think the mutable-container/immutable-observation model makes a lot more sense for the common cases (e.g., adding observations to an existing annotation).
You can iterate over observations in the usual way:
for obs in ann.data:
    print(obs.time, obs.duration, obs.value, obs.confidence)

@justinsalamon and I kicked around the idea of having an export-to-dataframe method, which should be pretty trivial to cook up.  Similarly, we might want to consider adding other mir_eval-style converters, such as to_event_values for instantaneous observations.
Pretty-printing is the last major hurdle, but I think we can get over that fairly easily.

well, first off, 💯  for a PR / RFC that deletes more code than it adds.
I do actually like that this gets us away from pandas (maybe as an optional dependency?), but we only trade dependencies. Is the development velocity on sortedcontainers lower than pandas, and that's why it's a win?
question: didn't pandas provide some neat wins for muda? shifting annotations / observations in time and whatnot?

I do actually like that this gets us away from pandas (maybe as an optional dependency?), but we only trade dependencies. Is the development velocity on sortedcontainers lower than pandas, and that's why it's a win?

I think we still need pandas as a dependency for loading lab/csv files (utils, import scripts, etc).  But it's good to get the hacky dataframe dependencies out from under the jams object hierarchy.
Sortedcontainers is a much simpler project, and seems reasonably mature at this point.

question: didn't pandas provide some neat wins for muda? shifting annotations / observations in time and whatnot?

It did, but that's not a huge deal.  It's better IMO to make observations immutable and put a little more friction into annotation manipulation.  Muda will need a little retrofitting, but it's probably about an afternoon's worth of work.

Note: coverage is down here because I added html rendering for AnnotationData.  I don't think unit tests are critical on that function.

Note: coverage is down here because I added html rendering for AnnotationData.

Would it be a good/bad idea, since we still have the pandas dependency anyhow, to do viz by exporting to a dataframe and then using the pandas built in viz? This could be nice in that you could use the variety of viz options offered by the pandas API for dataframes (e.g. peak(), head(), etc.)

Would it be a good/bad idea, since we still have the pandas dependency anyhow, to do viz by exporting to a dataframe and then using the pandas built in viz?

I don't think we should rely on it, but it's always an option.  The dataframe converter is done anyway, so you can do
df = ann.data.to_dataframe()
df.head(10)

etc.

I thought some more about this PR last night, and I think I'm convinced that we should drop the separate AnnotationData class, and promote most of its functionality up to the AnnotationData level.  The data field will then be a SortedListWithKey object, and the Annotation object will handle all (de)serialization and management of observations.
As noted in the review threads, this would break some API for folks that expect the ann.data object to behave in certain ways, but we're already doing that anyway.  The result should be a slightly simpler API, eg:
ann.to_interval_values()

instead of
ann.data.to_interval_values()

etc.
Is there any reason not to do this within this PR?

This will likely break more things in libs that build on jams, but as you say if we break things might as well go all in if it's worth it.
If the data structure is completely hidden and only accesible via the API, we should make sure the API supports operations for e.g. checking if the annotation is empty.
The one possible caveat to this change might be that since an annotation also has other fields (metada, time, duration, sandbox), the lack of an explicit data object could get a tad confusing in certain cases, e.g. len(ann)=? print(ann)=? etc

If the data structure is completely hidden and only accesible via the API, we should make sure the API supports operations for e.g. checking if the annotation is empty.

Well, the data field is specified in the schema as a list/array (actually, either a denseobservationlist or a sparseobservationlist), so we shouldn't make it entirely private.

The one possible caveat to this change might be that since an annotation also has other fields (metada, time, duration, sandbox), the lack of an explicit data object could get a tad confusing in certain cases, e.g. len(ann)=? print(ann)=? etc

I think it's reasonable to treat Annotation as a collection of observations with some metadata on the side. Then we can define len(ann) as the number of observations, iter(ann) iterates over observations, and __contains__ tests if an observation belongs to the list (not that we really need that, but it doesn't hurt).
repr(ann) currently returns a JObject-style repr string like <Annotation: data, annotation_metadata, ...>, so we wouldn't necessarily need to change anything there.  I wouldn't mind changing JObject's repr format overall, since it's not terribly informative for distinguishing between different objects of the same tyep, but that's a separate issue.

Amending my previous statement: I don't think we can consistently make Annotation implement the Collection ABC because it would change the semantics inherited from JObject for __len__ and __contains__, which both apply to the object's __dict__.  OTOH there's no problem in juts implementing __iter__.

@ejhumphrey I think this one's stable for eyeballs again.
Coverage drop is primarily due to html rendering not being covered, but I'd prefer to push that to #93 and implement it across the board.

Couple thoughts, minor apologies if / when relevance is a stretch:

Not concerned with the coverage drop; does coveralls have a setting that says "don't worry about drops if the total coverage is over a certain level" (say, 98?)
It has since occurred to me that all of my other projects using jams have things like for idx, obs in ann.data.iterrows() ... is this a common pattern (or am I weird); and if it is common, are we throwing deprecation warnings for this usage? or is it gone outright?

Not concerned with the coverage drop; does coveralls have a setting that says "don't worry about drops if the total coverage is over a certain level" (say, 98?)

I'd prefer to keep the strict coverage check in place, just so that we're aware of it at all times.

It has since occurred to me that all of my other projects using jams have things like for idx, obs in ann.data.iterrows() ... is this a common pattern (or am I weird); and if it is common, are we throwing deprecation warnings for this usage? or is it gone outright?

Gone outright: any previous access to the data field is not guaranteed to work in general.  We throw deprecation warnings for some things in 0.2.3, but it was too much of a mess to do it in full generality without also flagging internal access that was going to change anyway.
The idx, obs pattern can be recuvered by
for idx, obs in enumerate(ann):
    ...

though there's almost never a good reason to want the index itself.

Hashed out in person with @ejhumphrey  -- this one's good to go.


Added arabic locale files

I have added the arabic locale files. There need to be some modifications to the code in order to detect the language direction and add it to the current document body properties.
For example we can use this:
import { appWithTranslation, useTranslation } from "next-i18next";
import React, { useEffect } from "react";
const { t, i18n } = useTranslation();
const direction = i18n.dir();
useEffect(() => {
document.body.dir = direction;
}, [direction]);
This should be added to the app file. It uses next-i18next to automatically get the current language and sets the body text direction (ltr or rtl) depending on the selected language.

@psychedelicious Tagging you here.
The issue with this PR is that Arabic UI's are generally flipped because the language reads from right to left. Flipping the UI is extremely easy because i18n provides a direction value that we can use to flip the window entirely.
But this leads to a bunch of issues with our UI.

Any element that is absolutely placed does not get flipped over. We have quite a few of these.
Elements which have directional styles look weird. For example, some buttons tat have rounded corners on side and not the other end up looking weird.
In most cases the flex and grid boxes adjust well but some of the flips just look odd. I don't know if this is because I am not used to seeing them that way or they are actually seem oddly placed.

I cannot think of an obvious solution to this.

We do not flip the UI. Let the translation occur and the language reads right to left anyway but we do not flip the UI at all.
We flip the UI and ignore these UI anomalies. This would lead to some bad UI experience for the users on the language.
We manually design alternate styles for these items or symmetrize all their designs so they look the same even when flipped. This is a lot of work and something we'll need to be mindful of in the future design work too. Not to mention, I don't know how this'll work with the node tree.

I think we go with #1. Add the language but do not flip the UI. What are your thoughts on that experience @ParisNeo?

Hi, sorry for being late, yesterday I was hard on myself at work so when I got home I slept for 10 hours strait :)
It is OK for me to keep the layout direction for now. But I will have to change all the sentences that contain mixes of arabic and english words to do the english part phonetically. That will at least keep the sentences consistant.
The third option is what should be done in the future. We just build a custom style for right to left. Use the same code that flips the page to select the right to left style and that should be ok. This will allow supporting more unconventional languages like urdu and others.
But I don't think the localization of the ui is the most important part of this work. I think there are quite a few Ideas way more important to implement than localization. Actually most arab people use english language. So let's keep it that way, provide the arabic translation in phonetic manner for english names like Stable diffusion and that should be ok.
I want to know though, is there any prompt translation tool? In my personal ui for stable diffusion, I have added a translation step so that if you say want to write your prompt in french, you can select the language and when the query is sent, the backend verifies the selected language. If it is not english, it first translates it to english, then continue. I think we can bring this to your ui as I didn't see a language selection in your ui. That would be a more useful tool.
I use open ai's whisper to do the translation. It works really well and supports multiple languages. But we can go with a different tranlator like the ones provided by meta.
What do you think about this?

I would love to provide an equally approachable UX for LTR and RTL languages, but yeah, styling the UI in such a way that is can be flipped will be a very complex task. This is the first I've thought about it, to be honest!
What if we do 1, but offer 3 as an option in settings, tagged as experimental?
Maybe that would require two different Arabic translations, but it sounds like you @ParisNeo have already RTL properly, are willing to do a LTR version.

Ok, I can do another translation this weekend to resolve the ltr problem and use the solution 1.
There is a possible way in between though. Which is switching only the text and not the entire UI.
I google it really fast so I'm not sure this will work, but i found this blog post that can be helpful:
<EMAIL_ADDRESS>I'll see what I can do this weekend as I have no time during this week to work on this project.

Great, thank you. Please do not delete the original translation - we can use it if we provide the option to switch the UI to RTL. I just tried it with the translation and it doesn't look too bad:

In this image, I have increased the font size a bit. The Arabic script looks kinda small to me, compared to the Latin script used by English. Maybe I just lack familiarity with it, so it looks small.
Also, please let us know how to write "Arabic" in Arabic :)

Arabic in arabic is :
العربية
I see that some parts are simple assets anyway, so if we translate, they are not translated. In the screenshot you sent we can see: connected in glowing green. This is another challenge if we want 100% localization :).
OK, I'll keep a copy of the current translation

Oh, good catch. There are a few things that are not in the translations at all - that is one of them.

Lool. I have to translate some model names like GFPGAN or code former. Some sounds do not exist in arabic so I have to go with the nearest. For Invoke  AI, I have simply translated its meaning, so the people understand that yo uare invoking artificial intelligence. I don't know if this is the best cuz there is no V sound in arabic and the pholetic translation seemed odd.

I have finished removing all Latin text. I have translated translatable stuff and use phonetic approximation for the rest.

Hi, Can someone review my PR?

Thanks @ParisNeo . I've fixed a couple technical things.
@blessedcoolant Need your review
I've made a change to the language picker. Previously, the picker showed all languages in the currently selected language. This seems unhelpful, for finding your language, which you expect to be listed in your own language.
I've removed all language strings except for languages in their own language, and added these to en. So now the picker always displays all languages in their own language.

Seems good.

Will merge this now. We can figure out the direction flip for RTL languages in a later patch. Instead of doing a hackjob, we do a proper UI fix. Currently not at the top on our list of todo things but if @ParisNeo wants to give a shot at it, will be more than happy to take a look.

@lstein can you merge this manually please? Same issue with the frontend build again. There's another PR that address this. But until then ..


Memory leaks in gbt.load()

I think I have detected a memory leak in gbt.load. Could be I'm failing to understand how R handles memory (and the imperfections of that process), but maybe adding a manual garbage collection in the C subroutine could be the way to solve this?
Minimal replication:
x <- runif(50000, 0, 10)
y <- rnorm(50000, x, 1)

mod <- gbt.train(y, as.matrix(x))
gbt.save(mod, 'gbt_model.gbt')

Memory usage (by the R process, not the objects in R workspace) increments on my system by roughly 1mb every time the below line is run:
mod <- gbt.load('gbt_model.gbt')

Howdy.  @sondreus had reached out to me as we have had a few earlier DMs about his most excellent model and its use of agtboost.
The package is in good standing on CRAN, and Prof Ripley keeps us all honest by running checks with the (very powerful) valgrind tool on his instances.  I have been at the receiving end of such "valgrind detected a leak" notes myself.
And agtboost comes out clean as a whistle.
I can replicate that with a quick R -d valgrind -f issue55.R where the R file re-groups the demo above. (Even with the load() part in a loop for extra effort.)  No leak ... in the sense of memory being allocated and allocated and allocated and never freed.
The fact that @sondreus sees increase memory usage may have mean that R assigns new objects it keeps track of. It is tricky to disambiguate that.  I'll think some but at glance the package seems to not be doing anything.  Happy to help some more and chat but no smoking gun as far as I can tell.
PS I always find it instructive to have a baseline. So putting, say, these two lines in a file and running them as above with valgrind does generate the output below (among much more output)
Rcpp::cppFunction("double leakMe() { double *p = new double[1000]; return p[0]; }")
leakMe()

yields
==4133893== LEAK SUMMARY:
==4133893==    definitely lost: 8,000 bytes in 1 blocks
==4133893==    indirectly lost: 0 bytes in 0 blocks
==4133893==      possibly lost: 0 bytes in 0 blocks
==4133893==    still reachable: 56,622,975 bytes in 11,188 blocks
==4133893==                       of which reachable via heuristic:
==4133893==                         newarray           : 4,264 bytes in 1 blocks
==4133893==         suppressed: 0 bytes in 0 blocks
==4133893== Reachable blocks (those to which a pointer was found) are not shown.
==4133893== To see them, rerun with: --leak-check=full --show-leak-kinds=all

We have none of that in agtboost.  Which is good!

Thanks @eddelbuettel!
Look forward to hear from @Blunde1 who might look into this too. My guess is that my language is imprecise, so that there is no memory leak in the proper sense, just that when run many times, gbt.load takes up ever more memory (I got a colleague to replicate this to be sure), even as the loaded objects are no longer usable by any process from within R.
All the best,
Sondre

Yes. R uses copy-on-write internally so a changed object always becomes a new object (where plain assignment copied A <- B for whatever B do not yet create a new one but keep a reference).
I had massive headaches with things like too in the past (though maybe mostly in the distant past, on Windows, with smaller memory systems) so I don't have a great set of recommendations.  A simple rm(A); gc() always looks promising but in practice may not work.
In theory, "maybe" we could bring the loading of the model to the C++ side (where we have finer control) and use accessors for the data.  In practice that maybe harder because ... these model structures are larger.  But I guess they are regular?
We could possibly try alternate serialization from the C++ side too.  I'd be game for helping a little but I too would love to hear
what @Blunde1 thinks as he of course knows his model inside out...

As the cliche goes, I just had some more thoughts about this while on the morning run (and in the shower, cliche of cliches).
To clarify: @sondreus you have this issue when bootstrapping / looping over N model samples and fits, correct?  I may have an idea or two (which do not require @Blunde1 to change anything) so maybe you and can (and should) take this to a scrap repo to noodle around?

@eddelbuettel Yes indeed. I've "solved" the issue by dividing this portion of the update process into two batched, and moving the offending portion to a separate script and running it in a new R process which closes upon completion (so that the rest of the update process won't run out of memory).
https://github.com/TheEconomist/covid-19-the-economist-global-excess-deaths-model/blob/main/scripts/0_excess_deaths_global_estimates_autoupdater_B.R

That is pretty much what I was about to suggest. I happen to use external scripts lot and really like batching for R -- and wrote littler to give us r (at least on Linux, on macOS one has rename to lr or alike as it thinks r == R thanks to the genuises in Cupertino).  And if one does not like that / is on a different OS then Gabor et al also wrote callr which should do the same.  The key is to process each i-th R job to really get the memory back.

@sondreus Thanks, I am looking into this. I observe the same as you, and also from running mod <- gbt.train(y, as.matrix(x)) multiple times the memory seems to increase. I will "noodle around" some myself to try to find the cause (the only thing I can think of right now is perhaps improper use of new keyword on C++ side, or something similar...?) and a proper fix. For the time being, it seems you and @eddelbuettel have a workaround in place (nice to be aware of littler!).
@eddelbuettel thanks for the checks and running valgrind (and also Rcpp and its extensions)!

In theory, "maybe" we could bring the loading of the model to the C++ side (where we have finer control) and use accessors for the data. In practice that maybe harder because ... these model structures are larger. But I guess they are regular?
We could possibly try alternate serialization from the C++ side too. I'd be game for helping a little but I too would love to hear what @Blunde1 thinks as he of course knows his model inside out...

I believe the (de-)serialization can be improved upon, and would love to hear your thoughts on it. The models are regular in the sense that they are deterministic and in essence consists of nodes that are easily traversable (directed graph), thus the same logic may be applied all over. Do you know of any code/packages from which inspiration could/should be drawn on? In particular, are there any "using accessors for the data"-examples you could share?

Yes, it's a weird issue and it would be good to get a handle on it.  We may still be doing something wrong but it apparently it is not something obvious.
Serialization is a fantastic (and complicated !!) topic.  Per se you are doing nothing wrong, but of course writing "in text" is typically as seen as the most expensive way (formating out the way out, parsing on the way in).  It is also human-readable and portable so there is that on the plus side.   If you are only serializing 'for yourself here' (i.e. agtboost writes and reads) then saveRDS / loadRDS will be faster at the expense of human readability (and portability). I personally like fst an awful lot: it is very fast and backed by a library part one could use elsewhere (directly from C++, to interface Python, ...)  And then there are msgpack and feather and writing to database (SQLite is good for this too) and more.  You can go and on.
But you don't have to. Simple is good too.  For now, I think, we need to understand why/how the memory use grows. Maybe try another baseline where we just repeatedly assign a matrix in which we change one or two values (to trigger R's copy-on-write) ?  I would think that by assigning a new one in loop 'i' it would free / recover the memory from iteration 'i-1'.  Why this does not happen here is odd.  (And I think I am more puzzled about R here than about agtboost but we should double check).

@eddelbuettel Thanks!
For the memory issues I wanted to see if this was something that happened purely with agtboost, or if I would observe the same behaviour of increasing memory with a more simple C++ class exposed with RcppModules. I thus stole the yada example from the vignette
// yada.cpp
#include <Rcpp.h>
class World {
public:
    World() : msg("hello") {}
    void set(std::string msg) { this->msg = msg; }
    std::string greet() { return msg; }
private:
    std::string msg;
};
RCPP_MODULE(yada){
    using namespace Rcpp;
    class_<World>("World")
        .constructor()
        .method("greet", &World::greet)
        .method("set", &World::set)
    ;
}

and then use the set() function from R where a very large string (approx 1GB) is passed to the C++ object.
# yada.R
Rcpp::sourceCpp("yada.cpp")
n <- 2e8
long_string <- strrep("hello",n) # approx 1 GB
# Running the next two lines multiple times, memory seems to be incremented
world <- new(World)
world$set(long_string)

From running the last two lines incrementally I seem to experience the same memory issue. This is seen from visual inspection in RStudio, and from inspecting the rsession object in the Activity Monitor (MacOS). I will see if this is replicated on other systems. Please also tell me if I am doing something wrong here, and it would be interesting if you also observe this behaviour.
If you also indeed observe the above behaviour, I would assume it is not a problem with agtboost but rather some weird (because I do not understand it 😄) behaviour on the language-memory-handling side. Which seems to be aligned with your comments above.

That's exactly what's needed and maybe also a possible fix I had not tought about.
The long story is that some time many years ago (but after I had started using R too) the Core team changed string representation and made it much better. (It was fairly awful before and one of the reason for the old bias of using matrices over data.frames or anything indexed by rownames as those used to be expensive.)  String got a better hashed representation, but that representation is still different from numbers (int, double, ...) which are all consecutive for a vector,  CharacterVectors are essentially pointers to to individual 'words' (the individual string entries).
Moreover, these are not shared between C++'s std::string and R''s whereas we shared int or double vectors!  So I think when we invoke new(World) twice R cannot know and does not know it already has the vector and creates a new one.
This may be different if we used Rcpp::CharacterVector.  That is more of a promise and hunch from me than a firm guarantee.
But it gets us back to @sondreus issue because your serialization is ... vectors of strings, right?  We might avoid it all if we went  to storing what are for just numbers as numbers.  That may be more work, and more fragile, so we would make sure it is worth it.
But great existence proof above.  This may lead us down a road to improve the issue.

@eddelbuettel thanks, the long story and its insights is appreciated! I think there is no way I would have understood this on my own.
I changed the attribute of World (in the example above) to a Rcpp::NumericVector x and correspondingly in the World::set() function, and from the R side passed a very large numerical vector.
For agtboost, a model object holds the loss function as an std::string object, but this could (and probably should) just be an enum. All other attributes are numerical. I will take a look and perhaps do some refactoring so that the model objects are purely numerical.

Really nice work @Blunde1!  I think the 'how strings are represented' is a bit of an insider story I have happened to have picked up on.
I'd be up for cotinuing to discuss and prototype (and help, once this vacation is over,,,,) with a 'more efficient' serializartion approach.  This is essentially undocumented stuff so if we can put up a better reference implementation it may help others.

[ ] Make agtboost model object entirely numerical
[ ] Refactor serialization to be more efficient

Sorry for the silence, but had some time off.  I just set up a very minimal check to see if our hypothesis concerning string representation holds.  I made these two minimal changes to save and load as rds if the filename ends in rds (very simplistic ...)
modified   R-package/R/gbt.load.R
@@ -23,7 +23,12 @@ gbt.load <- function(file)
 {
     # check valid file path

-    mod <- new(ENSEMBLE)
-    mod$load_model(file)
-    return(mod)
-}
\ No newline at end of file
+    if (grepl("rds$", file)) {
+        mod <- readRDS(file)
+        mod
+    } else {
+        mod <- new(ENSEMBLE)
+        mod$load_model(file)
+        mod
+    }
+}
modified   R-package/R/gbt.save.R
@@ -47,7 +47,10 @@ gbt.save <- function(gbt_model, file)
     if(length(error_messages)>0)
         stop(error_messages)

-    # Store model in file
-    gbt_model$save_model(file)
+    ## Store model in file
+    if (grepl("rds$", file))
+        saveRDS(gbt_model, file)
+    else
+        gbt_model$save_model(file)

 }

And based on a casual check of loading the (same) file fifty times, memory use seems stable as it should be.  Let me know if this helps, I can probably expand on it.

@eddelbuettel thanks -- time off is time off which needs to be spent wisely I am realizing more and more. Here we just had our few good days of weather for the summer, which warranted time outside.
I will try this as soon as possible. It would be amazing if this indeed works! I will still try to change the model class as intended by introducing an enum loss-function attribute, and some general refactoring, but perhaps that can be done after introducing your solution as it might very well resolve the issue. 👏
As a side-note I remember "studying up" on serialization and in particular for C++ objects from R, and thought that I had unsuccessfully tried the save/loadRDS format at that time. Thus I am very glad if this indeed works, and appreciate the help immensely. 😄

It's easy to get lost in the weeds there.  I just accessed RDS input/output from R.  But as you mention it, I once needed it for Redis so I extracted the (otherwise 'hidden' and unexported) R code here in this repo (and on CRAN) but we should not need this.  You know your model data so much better than I do but I tried a few tricks over the years with serialization -- if RDS works for you I'd stick with that for now.

@eddelbuettel I created an early PR based on your suggestion here
I get an Error when running the suggested code
> mod <- gbt.load('gbt_model.rds')
> mod
Error in sprintf("C++ object <%s> of class '%s' <%s>", externalptr_address(pointer),  :
  external pointer is not valid

which seems to be similar to my earlier attempts in the past, I think.
I googled a little bit, and stumbled upon (again) the ever-helpful Rcpp Gallery with this post on serialization of C++ objects. Perhaps I should do this first and then call save/loadRDS? I would be surprised if this did not work, but it would however require further dependencies...
While thinking about the serialization I will continue to fix the string issues in #58 as I believe this might also resolve the memory issues.

Howdy.  Happy to get more involved and send you what I had as a PR (or push it to a fork, or ...) but can you in a first instance debug locally?  What I offered (to switch based on the final characters in the file) may be 'too loose'.  It may be simpler to use your load and save from ascii as before, and to explicitly call readRDS() and saveRDS().
The Rcpp Gallery piece uses a different serialization (from Boost), we may not want that.  I do have a helper package RApiSerialize (on CRAN for many years, I happen to have just sent a minor update to CRAN) but that is only needed if you want to serialize from C++ which we may not need here (but I guess we could).
The error message above seems to suggest that some wires are crossed for the Rcpp Module part.