Gazetteer model

I'm submitting a first draft of

a somewhat reworked GeographicItem model
a new Gazetteer model
a 'New Gazetteer' task

All of those are unfinished (and possibly somewhat broken), but I wanted to put something out to make sure I'm on the right track. All/any feedback welcome, but I'm not looking for a full review at this point - I consider this all to be in the early stages, so 'big' changes are fine with me.
Some notes/issues/questions I've encountered so far:

I doubt all the specs pass, though GeographicItem currently should. GeographicItem specs have been duplicated, but I haven't changed the second copy at all yet to cover the new geography column.
Some changes in GeographicItem:

shape_column_sql(shape) returns, e.g. the polygon column if that's present, or the geography column if that column holds a polygon. This will replace many of the existing CASE statements once everything switches over to a single geography column.
More st_* methods for working with st_* statements (feedback welcome), see also select_self.

In GeographicItem I changed all ST_Contains to ST_Covers. I couldn't tell why one was meant over the other in particular cases, and ST_Covers seems easier to reason about (to me). (The difference is ST_Contains doesn't contain things in its boundary, so e.g. a polygon wouldn't contain its vertices, etc.) That could be a breaking change, and would be easy to revert.

One question would be which should be used in new code (maybe it depends on situation).
If there's no builtin reason for one over the other then a couple other spots in the code could be changed to ST_Cover instead of ST_Contains, if we wanted to settle on one over the other.

There are nearly 500 lines of code in GeographicItem that are either

not referenced at all
are only used in specs to test themselves
are only used in specs to test other things
I put all of those in a separate GeographicItem::Deprecated module for now, but it's still included in GeographicItem (for now) and I don't have any real opinion on whether that's the way forward or not.
Personally my thought was to put the used-only-for-specs methods somewhere else where they would only be visible in specs, but I wasn't seeing any clean way to do that.
I am now assuming that PostGIS >= 3.0 so that we should be able to handle all GeometryCollections in ST_* functions.

The 'New Gazetteer' task allows the user to add multiple shapes to a new gazetteer before saving (i.e. they can draw multiple shapes in leaflet, for example, for one gazetteer) - they all get bundled into a single collection behind the scenes. That differs from the georeference model I think, but fits more closely with GeographicArea shapes (I think) - this could also be a path to combining existing GeographicItems into a Gazetteer, depending on how we want to store that. Leaflet shapes and WKT shapes (and eventually other input methods) can be combined in one gazetteer - I don't keep track of which shapes came from which input types on the backend.

One question is how to deal with intersections of multiple shapes. I'm not sure if that may cause issues or not. One leaflet polygon drawn inside another combines to a donut (inside leaflet iirc), which is probably a useful thing, allowing users to create donuts, but maybe in other cases it's unexpected/more tricky/doesn't-combine-well-with-wkt, I'm not sure.
I've turned off circle creation in leaflet for now at least - we'd need to approximate by a polygon on the backend I think, which is a whole thing (is it already covered in a library somewhere?).

As @mjy suggested, I've gone with a write once read/replace pattern for the geographic item shape of a gazetteer.
I've yet to take any position on meaning of Gazetteer parent in the code.
Most of the crud create/list/show stuff should be mostly working, but it's not all finished yet.

Thanks @kleintom

I changed all ST_Contains to ST_Covers

I feel this is a safe assumption.

which should be used in new code?

This is essentially everything but me vs everying AND me.  I feel most will want everything AND Me (covers) so I would go ahead and make covers the primary workhorse in new code.  There should be decent spec coverage on this, if there is a lot on contains version then it suggests that it was important too...

There are nearly 500 lines of code in GeographicItem that are either ...  not referenced at all

Rip it out.  Make a single commit that eliminates it and the corresponding specs so that we can diff against it down the road.  A bigger challenge is to eliminate all the items we don't test against as well (support/geo stuff). Let's not spend time on understanding code we don't use, or might use, let's clean slate things.

One question is how to deal with intersections of multiple shapes.

I think you're asking whether we should compute and store a single shape, or store all the shapes.  My initial thought was compute an intersect before persisting.  Perhaps a virtual attribute ('combine shapes').   I guess we have the flexibility to make this optional.  I think performance wise combining selected shapes into one, and possibly simplifying them as well, might be desirable options.  A major use case is to define, by selecting multiple shapes a squared off printable base-map, with borders roughed in.

all get bundled into a single collection behind the scenes. That differs from the georeference model I think,

Not really.  Individual Geoferences can each be cited and annotated, we don't want the shapes from individually performed Georeferences to be combined in the back end.

I've turned off circle creation in leaflet for now at least - we'd need to approximate by a polygon on the backend I think, which is a whole thing (is it already covered in a library somewhere?)

Hmm.  This is a major consideration.  Perhaps we will have to store a radius value for retroactive edit/adjustment functionality, and use a before hook to calculate the shape?

It may be worth considering introducing the ordered bounding box (á la OvertureMaps) in GeographicItems (and maybe also GeographicAreas) to provide a quick (maybe orders of magnitude faster) selection of different Venn cases in searches.  Early Taxonworks overlaid results with a bounding box that was flawed especially when dealing with antimeridian spanning shapes -- not then recognizing (or correctly calculating) the necessary west to east ordering.

The main new thing here is specs for the new geography column of GeographicItem. There are a couple significant departures from existing GeographicItem specs (though for the most part the things being tested are the same), suggestions/feedback welcome.

Fewer, simpler shapes, more descriptive names

fewer shapes in the sense that, e.g. a single polygon can provide multiple shapes: a polygon, lines, points, multilines, multipoints that can participate independently in specs as needed. This makes shapes more tightly coupled, but reduces the number of shapes to keep track of. I'm open to thoughts on which is preferred, or if there's a better option.
Simpler axis-aligned shapes. If specs pass for those but fail for more complicated shapes, my take is that that's an rgeo/postgis/geos issue.
Axis-aligned along the "equator" also allows for more precise distance specs (though again this couples the existing shapes to be at certain distances).
More descriptive names: the hope is that one can, more or less, read the new specs without looking up what the shapes are. I may have gone overboard in this respect (especially for someone without autocomplete), some names are quite long.

Other updates/changes:

geographic_item now has one gz rather than many. Am I missing a reason that shouldn't be the case? Oops, see the next point.
I removed the (unimplemented) clone option that I copy-pasted there, and then I remembered this comment, regarding sharing across projects:

Finally (and this is where I would lean to start) we could simply clone Gaz entries, while pointing to the identical shape. This would permit discovery and forking of use.

Do you mean users in one project could somehow see/search gzs in another project and clone the ones they want into their project?
May also need to clone (GeographicItems from) GAs if we're going to keep track of the separate GAs we're using to form a union (see also comments below).

I can't get leaflet to produce donuts anymore - I thought at one time it was doing that when I drew one polygon inside another. You can still create a donut via wkt, maybe that will be the only way (as it is for georefs I think).
Saving a shape that crosses the anti-meridian fails in the expected way, that will need to be fixed.
All of the crud stuff should be working now
When you save multiple leaflet shapes/wkt shapes/etc they're now unioned on the back end (instead of collected). Depending on how other issues mentioned here work out, this may need to be adjusted.

pros: simplicity, performance
cons: you lose track of your original shapes, harder to read the combined shape in some cases

A major use case is to define, by selecting multiple shapes a squared off printable base-map, with borders roughed in.

I assume you mean selecting multiple separate gzs(/gas/grs?), not multiple shapes from those within a single gz?

As a first step I added a kind of support for circles - a circle drawn in leaflet gets converted to a polygon on the back end and forgets it was ever a circle (which means it no longer gets reported as a circle in the ui). Number of polygon segments is configurable. I think georef radii are limited, should there be a similar limit here?

Perhaps we will have to store a radius value for retroactive edit/adjustment functionality, and use a before hook to calculate the shape?

I'm probably overemphasizing the ability to add multiple shapes to a gz, but a couple thoughts: if we only want to allow one circle then we can just have a gz radius column, if we wanted to allow multiple then we need a separate circles table (I think). If we just add a radius column then we need some way to associate the radius with a particular point within the collection of shapes for that gz (some of which may be other points), so probably better to have a separate circles table (with radii and centers), or only allow a single lone circle shape if you want a circle. This would all require keeping shapes separated (not unioned), or at least keeping circles separate. That could all be done though I think.
Next steps for me:

Add filtering on a gz
Add import via shapefile/some other means

Other remaining issues:

Parentage needs much cleaner semantics in Gaz, if indeed we use it at all (is it geopolitical, spatial, set defining). Let's not try to sync it with GA for now, because there it means "as used in TW to build the Gaz, roughly geopolitical sets", which could be better defined.

I see what you mean now, I'll go with whatever you think here.

Forming unions of GAs and/or other gzs. Implementation to be decided, see comments above.

A little review/thoughts on the present state:
IIRC the strategy is to land this PR then do a followup where we merge everything down to the single geography column, correct? The following expands on that thinking-

Let's not try to add too much more CASE/conditional handling for the multiple column types.  Since we're assuming we're going to merge everything into a single column with a data migration the PR "after" this, we can minimize the exception handling work at this point.
I think you can create a unifying migration to copy the existing data columns to the new column.  We'll leave the data in place in the existing columns and include a cleanup commit after this is merged.
Do we need a sublcass of GeographicItem if it is going to move to a single column?  Maybe not?

Fewer, simpler shapes, more descriptive names

This looks great, thanks.

Do you mean users in one project could somehow see/search gzs in another project and clone the ones they want into their project?

Something like this, yes.  For example there are people with multiple projects, they might want to "copy" their Gazs b/w them.

When you save multiple leaflet shapes/wkt shapes/etc they're now unioned on the back end (instead of collected).

Thanks for pointing out the pros/cons. I agree, let's see what kind of responses we get as people begin to use the Gaz.

I assume you mean selecting multiple separate gzs(/gas/grs?), not multiple shapes from those within a single gz?

Yes!

As a first step I added a kind of support for circles...no longer gets reported as a circle in the ui)

Seems OK for now, again we can wait to see how this plays out.  I don't anticipate a lot of circle use, but I could be wrong. Thanks for the observations on multiple circles.  I think we're talking significantly more complexity than we want to initially introduce in that scenario. Our basic idea is not editability, it's stored shapes.  We can argue that those shapes are best defined in interfaces that edit well, our UI can be "rudimentary".

The main new things in the latest batch of commits:

Added a filter-on-gz facet for ce's, co's, otu's, and ad's (everywhere there's currently a ga facet I think).

in the gz case the Descendants and Exact options don't apply (at least not yet), so there's only the spatial option.

One important point I overlooked in the new gi::geography specs is that, for example, I defined box_centroid in terms of box, which means anytime you instantiate box_centroid you also instantiate box. That defeats the idea of being able to use these shapes independently, so now all shapes are defined in terms of independent geometric data sufficient to describe the base shape (like box). Multi-shapes still instantiate their constituent shapes.

Rewrote GeographicItem ST_* methods using Arel. See if that's what you had in mind - I think there was a little bit of Arel creep, in the sense that I had to Arelize in a couple places where I was calling into an ST_ function (I think, not sure), but overall it does feel a lot more solid. (Any comments on the state of Arel regarding the possiblity of breaking changes in the future? The benefits it provides now outweigh that possibility?)

Added the input-a-point option for the New Gz task - I'm pretty much trying to follow CE GR input here, which also has the geolocate option. Is the geolocate option something you'd like to have for GZs as well? (I still need to add the iso_a34 input stuff in any case.)

Added a new Import Gazetteers task. Right now it supports importing from a shapefile; that should basically work, but it it's not built out yet. (Would you like other import options at this point?) I think there's still quite a bit to settle on there:

It's seeming like this import needs some kind of progress indicator; you may be able to just update the GZs list until you see your last shapefile Gz if we decide not to do everything in a transaction (see next point), but that's really not that great in any case. The solutions I've seen/considered all seem to use either extra columns or a separate table that background jobs can write to to poll from - the polling could involve actioncable, but so far at least it seems like the extra columns/table would be necessary. Any thoughts on solutions (has dwcimport already done this?), or could I think a little more about a proposal to present?
Should the entire import fail if one record import fails? Right now I'm ignoring record fails due to bad shapes or bad names (i.e. not present), but other failures cause a transaction abort; that probably won't be where things stay though. Currently things run in the foreground (will change to background - the Ecoregions file took 1 hour 20 minutes on my desktop for 847 records) and shape and name errors are reported after import.
Issues:

It might not be so easy to tell if your shapefile has "clean" shapes until you try to import, especially if you just downloaded it from somewhere. e.g. the Ecoregions shapefile (I think I used this one: https://www.geographyrealm.com/terrestrial-ecoregions-gis-data/ ) had quite a few self-intersections; after I ran it through ogr2ogr -f "ESRI Shapefile" -makevalid fixed_shapefile.shp Ecoregions2017.shp there were still two errors on import, even though ogr now said the file was valid - both failing shapes were anti-meridian crossing, it seems rounding issues using our factory are turning longitudes like<PHONE_NUMBER>8 (meant to skirt the anti-meridian) into -179.9999999999, and then those shapes get self-intersections and can't be imported.
Should the entire import fail in that case? I have two possible alternatives in mind, can you think of others?

support a "test run" option, like a sandbox that runs in a transaction, collecting failure information as it goes, and then aborts the transaction at the end (or maybe doesn't abort if there were no errors?). Reporting the errors would be an issue though, a progress indicator/similar would have to display those when the job ends, or as they arise (right?).
another option might be a checkbox saying Ignore shape failures, but that could be troublesome - e.g. here the original shapefile had many bad polygons, if I import the "good" ones, then see how many bad ones there were and run ogr to fix them, now I have no good way to reimport without duplicating the original good ones. On the other hand if I don't allow that then I'm left trying to fix the two errors above by hand, or removing them from the shapefile (ST_Valid returned true on them iirc, it was only on the import rounding that they became invalid, and then it was too late because the record import failed before their geometry could even be examined or fixed). Maybe a stern warning would be fine.

Another issue around failures is names - users have to provide the name of the name column (we're not guessing); if they enter a typo then every record will fail. This can at least be checked before the background job is kicked off; if the first 5 records fail on name assume a typo. But if we think some shapefiles might have multiple empty names and we want to worry about that then we'll have to decide how to deal with it.

Apparently rgeo-shapefile uses whatever projection is defined by the factory provided when the reader is opened. For us that's wgs84, so every record is assumed to have wgs84 coordinates (I'm not sure how the mercator projected projection of our factory fits in here). Right now a prj file is required, but I'm thinking that will be dropped with a note that wgs coordinates are required, and if prj is provided we can confirm or raise error.  (Again if people assume wgs84 and then see their shapes are all wonky when they import, how can they delete all of them except one at a time?)
Would it be helpful to add an import source column as in GA? Maybe autocomplete could include it as a way for users to try to figure out if the shapes from a particular shapefile/source have already been imported? And/or add a filter gzs task to provide other similar means, possibly along with bulk delete (on semi-failed/completely-failed imports).

Left to do:

Migrate to geography column, associated cleanup
Support creating gzs as unions of gas/gzs. Three options I can think of, maybe you'll have more:

Just combine everything on the backend, same as it is now for other added shapes; I think this would be simplest, though you lose track of what the union members were
Use a combination of leaflet/wkt/point-unioned shapes and closure_tree union members (via parent-child relationships); leaflet/etc shapes could be combined and stored as they are now, and then on access that would be unioned with child gzs on the fly (does that make sense?).
Allow the union, but dont allow other leaflet/etc shapes in that case. Parent can only be sti-polymorphic in closure_tree, so i think you'd either need to clone GAs into GZs to support unions, or add a polymorphic reference on GZ which could point to a GA or GZ for use in unions - I guess that option would make it easier to support unioning with other GZs (otherwise the child Gz would need to be cloned in some way from a Gz at some other location in the tree). Again the union would need to be created on the fly (right? or cached?).

Plenty of cleanup left to do.

Thanks!

It might not be so easy to tell if your shapefile has "clean" shapes until you try to import

I noticed RGeo::Shapefile does have a allow_unsafe: true option to open (documented in the readme but not on the open rdoc) which allows you to read invalid shapes. You can then rgeo make_valid them, though it's not a fast process on large shapes (maybe there would be some limit on the number of shapes that could be fixed before giving up?).

Right now a prj file is required, but I'm thinking that will be dropped with a note that wgs coordinates are required

Now I'm thinking a proj file should be required...
Maybe given those two changes there would be less chance for mass numbers of records failing and less need for test runs/mass deletes/progress reporting errors etc. that I suggested as options above.

Would it be helpful to add an import source column

Then GZs could be unique on name + import source and different shapefiles  could import shapes of the same name.

@kleintom Peeking back at this.  Let me know if I can help out in some regard.  Again, not critical, just seeing if we can facilitate. Thanks!

It's been awhile, so I'll try to summarize the main changes and issues here:

GeographicItem has been refactored, the main changes:

there's now a single geography column that supports a shape of any type (point, polygon, etc) rather than a separate column for each type

a migration copies the existing GeographicItem shape to the new geography column without deleting the existing shape columns (deletion will occur in a future pr)
all code referencing the old type columns is gone, GI is no longer STI
all unused GI code has been removed, ~500 lines iirc

a new basis of st_*_sql methods has been created to support a cleaner interface for code using the postgis ST_* methods

raw pg code has largely been replaced with Arel NamedFunctions which are more cleanly composable and provide more checks
geographic_item_spec.rb has been rewritten with a simpler, hopefully easily digestable suite of shapes (other geo test files have not yet been changed)

the new GI code has switched to just the simpler-to-reason-with ST_Covers instead of mixed use of that and ST_Contains - it's possible there will be some edge cases where behavior has changed (I did try to add specs to document at least some of those edges).

A new Gazetteer model has been added, essentially these are user-creatable named shapes. They also hold optional iso_3166 a2 and a3 values, but there's (currently at least) no hierarchical relationship between Gazetteers based on any of these data - each Gazetteer exists as an independent name/shape.

Currently the main use is for filtering geo data: asserted distributions, collecting events, collection objects, and otus.

A New/Edit Gazetteer task

Create by:

drawing shapes on leaflet, including circles
entering WKT, point coordinates
adding existing geographic area shapes and/or existing gazetteer shapes
TODO? Add Geolocate entry option as in georef?

On submission all of those shapes get combined together into a single shape that is then no longer editable - no record of the original input shapes is kept. That final shape is made valid if it's not already (more details further down).

There's a preview checkbox so you can check on what your final combined shape will look like on the leaflet map before you submit

An Import Gazetteers From Shapefile task

Add the individual files from a shapefile to TW as Documents (zipped shapefile not supported, but you can multi-select all of your files at once in the New Document selector in the task): shp, shx, dbf, and prj are required, cpg is optional but should be included if it exists; any other shapefile files are ignored and don't need to be uploaded.

Only shapefiles in the WGS 84 (EPSG: 4326) Coordinate Reference System (CRS) are supported.

TODO? If a prj file contains an epsg id for its CRS then it may be possible, at least in some cases, to convert to WGS 84 on the fly (but my impression so far is that most prj files don't actually include that??)

All relevant documents need to exist in TW, but the interface only requires selecting the .shp file - the other files will be found by name (unless there are duplicates (which I think is the only reason the document selector needs to be able to work with more than a .shp file)).
Gazetteers require non-empty names; you can either enter your shapefile's name field by hand or click to select it from a list of all of the shapefile record fields.

TODO restrict that list to only those columns that hold Strings.
Currently every record is required to have a non-empty name value - the job won't run unless that's the case. That could be adjusted.

One thing we can't do anything about is the possibility of the user accidentally selecting the wrong name field and then potentially importing many useless gazetteers.

Shapefile processing runs in a background job. Large jobs can take hours, so there's a GazetteerImport model that tracks the status of each job and displays that status in the task.

Up until a few days ago I meant for the job to run in a transaction: either all records would be converted correctly to Gazetteers or none would. I came to discover that doing that with long jobs with lots of records was a memory sink (see more comments in the commit that changes that behavior), so now the known errors that can occur during processing are caught and reported in the status of the job and the job continues (unknown errors end the job early). That means that shapefile records with no issues get imported, those that have some issue do not get imported but you should get some idea of what went wrong in the ui (this is not a great solution currently if there are lots of errors).

I feel much better now about known shape errors and dealing with those as best we can, but shape errors that do occur may require some more specialized knowledge to fix and rerun without simply reimporting the whole file and creating duplicates of those already successfully imported in the initial run.

All shapes to be imported are checked for validity; if an invalid shape is encountered it's make_valid-ed (can that fail?) and the valid version is imported.

TODO make_valid can introduce lower-dimensional artifacts which should be removed (I think).
The shapes known to fail with current processing are things like invalid shapes that also cross the anti-meridian - see below. The other known failure is when RGeo::Shapefile tries to load a troubled shape that Geos raises on before we ever even get a reference to it - n.b. those shapes can be valid according to both RGeo and postgis!

I haven't looked into DWC's imports yet since I was thinking in transaction terms, but now that there's not a transaction that may be a possibility.

I added the ability to choose to import your Gazetteers on creation into any multiple projects you're a member of - let me know if that's not what you'd like and I can pull it out (or just hide it for now).

The projects imported into are listed in the jobs status UI, but only in the project you imported from, not in all projects you imported to.
Either a given GZ will be saved to all projects or none (that part does run in a transaction).
There's currently no way to clone an existing GZ into another project (but that's certainly doable).

A couple of minor nagging issues:

When you drop multiple files into the New tab of the document selector for import, sometimes not all get imported, sometimes the page reloads, sometimes the import seems to import some files, hang for 10-15 seconds, and then complete, sometimes it all works fine. (I feel like this has somewhat improved recently though?)
Sometimes shapes drawn in leaflet disappear when you pan. Maybe more common with multiple shapes on the map? I haven't figured out STR.

There are validations on ISO_3166 a2/a3 codes to ensure they're two or three letters, but aside from that there are no controls: your shape need not actually correspond to your codes, A2 and A3 could be codes for completely different entities, etc.

codes can be searched in GZ autocomplete, other than that I believe they're currently unused

Both New GZ and Import GZs attempt to follow the same behavior for ensuring that any shapes created for GZs are:

valid and
do not cross the anti-meridian

My hot take as a person new to dealing with anti-meridian-crossing issues: "Wow, what a hot mess". Completely undocumented, with different behaviors both between libraries and within libraries. :p This was meant a little tongue-in-cheeck - TW at least has nice specs, maybe other librairies do too, maybe I just haven't looked hard enough. If there are things I'm just missing here please let me know.
the main issue as I see it is that functions processing anti-meridian-crossing shapes can treat a line between points on either side of the anti-meridian in different ways: some treat it as crossing the anti-meridian, some treat it as not crossing the anti-meridian but rather the meridian (the "long way around"). So even if your shape is reported valid?, other geo operations might operate on that shape in its opposite interpretation form and give "nonsense" results. Or your initial shape may be interpreted in the wrong way as invalid and then made valid in the wrong interpretation. So I've attempted to take the view that anti-meridian-crossing shapes just shouldn't exist in the db: some, maybe not all, are eventually going to explode or give the wrong result (though maybe not on operations we currently perform on them). I believe no GeographicArea shapes cross the anti-meridian(?).
Here was an instructive shape for me:

as drawn in leaflet:

as interpreted by leaflet after being normalized by our Gis::FACTORY, where lines between points now cross the meridian:

On that normalized shape, which is what we work with internally, GeographicItem.crosses_anti_meridian?, which checks for intersection with the anti-meridian line, returns true, as though the shape is the first picture, while rgeo valid? and postgis ST_IsValid return false as though the shape is the second picture (there's now a self-intersection at the right end which may be hard to make out). If you make_valid that shape it makes the second version valid, which is not what we want.

The coordinate normalization performed by our Gis::FACTORY is significant here, with different normalization of the coordinates you get different results. This is where our use of ST_ShiftLongitude comes into play (or rather perhaps why it's needed?).
I've attempted to document some of these issues with new specs/xspecs in the anti_meridian_spec file. Again, if I'm missing things or overly complicating things please do let me know.
the current behavior is to first check a new shape for crosses_anti_meridian? and if it does to use the new split_along_anti_meridian to cut it into pieces that don't cross the anti-meridian, and then to run make_valid on that shape. Since we're not checking valid before splitting along the anti-meridian, that can explode, and that's where I'm currently accepting failure: on (some) invalid shapes that cross the anti-meridian. Hopefully everything else is handled.

You can import the Ecoregions shapefile as found on the internet and it should complete with one error shape, but I'll have more to say on that file in the coming week. Maybe there's some way to provide a version of that shapefile that fixes that error shape and some other issues as well.
If anybody has shapefiles that they'd like me to test or that you test and that produce errors, I'd be more than happy to take a look at what's going on if you can provide me with the files or a link.

@jlpereira When you have time can you do another review of the JS side of things please?

@kleintom I talked with some folks here, where going to try and get a couple different shape-files to experiment with, some simpler, some complicated.  More feedback in general coming soon.

I feel like my understanding of anti-meridian behavior improved enough in the past couple days to make it worth adding some more clarification in the code. I'll try not to make anymore changes now.
It looks like the push is failing on a full db:migrate, which does https://github.com/SpeciesFileGroup/taxonworks/blob/development/db/migrate/20220927154908_add_cached_total_area_to_geographic_item.rb where GeoraphicItem::Point is referenced. Is the right fix for that to resurrect that class?

where going to try and get a couple different shape-files to experiment with, some simpler, some complicated. More feedback in general coming soon.

Sounds great, thanks!

migrate/20220927154908_add_cached_total_area_to_geographic_item.rb

This is a bit of a problem.  We very strongly don't want to go back and alter migrations, however here I think we could try to add the logic "if the model is present, then run, if it is not, then skip".  In this case it might be pretty safe.  I'm not sure of another way to do this outside of injecting something into the migration wrapper, which feels very janky.
Hitting this issue also indicates that we likely need to rollup our individual migrations into one, and start fresh on new ones, noting this "fork" in the schema.

@kleintom Don't worry about pausing development, keep tacking on changes (date-line) if you wish.

Hello @kleintom
What fields are require to create a gazetteer? I'm trying to create one but I get error 500
ActiveRecord::StatementInvalid at /gazetteers.json
==================================================

PG::InternalError: ERROR:  Option string entry 'structure' lacks separator '='

> To access an interactive console with this error, point your browser to: /__better_errors

app/models/geographic_item.rb, line 1136
----------------------------------------

ruby
  1131     end
  1132
  1133     def self.select_one(named_function)
  1134       # TODO use select_value instead of execute?
  1135       # This is faster than select(...)
> 1136       ActiveRecord::Base.connection.execute(
  1137         'SELECT ' +
  1138         named_function.to_sql
  1139       )
  1140       .to_a.first
  1141     end

App backtrace
-------------

 - app/models/geographic_item.rb:1136:in `select_one'
 - app/models/geographic_item.rb:361:in `make_valid_non_anti_meridian_crossing_shape'
 - app/models/gazetteer.rb:155:in `block in convert_geojson_to_rgeo'
 - app/models/gazetteer.rb:141:in `convert_geojson_to_rgeo'
 - app/models/gazetteer.rb:104:in `combine_shapes_to_rgeo'
 - app/models/gazetteer.rb:82:in `build_gi_from_shapes'
 - app/controllers/gazetteers_controller.rb:55:in `create'

parameters:
    "gazetteer": {
      "name": "Testing",
      "shapes": {"geojson":["{\"type\":\"Feature\",\"properties\":{},\"geometry\":{\"type\":\"Polygon\",\"coordinates\":[[[-69.023438,-28.214061],[-69.023438,-25.151913],[-62.958985,-25.151913],[-62.958985,-28.214061],[-69.023438,-28.214061]]]}}"],"wkt":[],"points":[],"ga_union":[],"gz_union":[]}},
      "projects":[13]
  }

What fields are require to create a gazetteer? I'm trying to create one but I get error 500

Based on the error message I think/hope I fixed the issue, though it wasn't causing an error for me running in docker before or after the upgrade to pg16 (maybe you're running a pg with different config?).
Gazetteer should only require a shape and a (non-empty) name.

Thanks, It works!

@kleintom The blocker to us getting to Rails 7.2 has landed (https://github.com/rgeo/activerecord-postgis-adapter/pull/405#issuecomment-2455646003 ).  We're still seeing issues on our test PR: https://github.com/SpeciesFileGroup/taxonworks/actions/runs/11672461015/job/32501118816, but I haven't poked at what might be the problem.
Given that we're likely a little away from landing this branch, I think we should target it landing on the 7.2 PR. Can you please update your branch to reflect this (should largely be pointing to a couple new gems)?
If there are spatial issues, which where the blockers in the adapter getting to 7.2, then your geo work/context might be the best place to discover these.

Can you please update your branch to reflect this (should largely be pointing to a couple new gems)?

I copied @LocoDelAssembly's code to update to 7.2 and added what seems like a reasonable fix to get specs passing again (though I don't know exactly what changed in 7.2 to require it). It looks like ci won't run automatically anymore though since Gemfile.lock is going to be in conflict until the 7.2 pr lands.

@LocoDelAssembly if you have time can you look into the Gemfile/CI issue please.
@kleintom This might be the thread re the issue: https://github.com/rgeo/activerecord-postgis-adapter/issues/402#issuecomment-2371701919; and the patch: https://github.com/rgeo/activerecord-postgis-adapter/pull/417 ... maybe

@kleintom We saw that there is a migration to remove type from geographic_items (I think).  Is it possible to leave that there until after this PR lands, or does it raise/get in the way?  The issue is switching dev branches back and forth requires re-adding it.

Is it possible to leave that there until after this PR lands

No problem, I just needed to make type nullable since we don't set it anymore.

I was going to refactor code I added to normalize longitudes of new Points - to get this GI validation to pass: https://github.com/SpeciesFileGroup/taxonworks/blob/a804e9cef2e541bdf49af05f9b129c1aec7660f6/app/models/geographic_item/point.rb#L27-L31 - when I realized that the same issue arises with Multipoints and Points/Multipoints contained in GeometryCollections: rgeo normalizes longitudes for any shapes that have edges, but not for those mentioned above. Is point-longitude normalization something we want to pursue for Multipoints and Points/Multipoints in GeometryCollections? And/or do we want to keep it for Points? (Is the normalization mainly for UI display purposes?)

Reading this I had a momement of panic thinking maybe we had swtiched GeographicItem to Geometry type :( !, which would have been bad.

Multipoints and Points/Multipoints contained in GeometryCollections: rgeo normalizes longitudes for any shapes that have edges, but not for those mentioned above. Is point-longitude normalization something we want to pursue for Multipoints and Points/Multipoints in GeometryCollections?

I don't forsee needing either Multipoints or Collections thereof.  Let's let users request it before we move on that front (I can't wrap my mind on what it would mean to assign such, but maybe, like in a Lot of fish lumped into a single barrel).

And/or do we want to keep it for Points? (Is the normalization mainly for UI display purposes?)

I don't recall why.  It might be related to downstream checks at GBIF requiring such.  Mapping might also have been an issue.  If we can normalize on render (e.g. to map, or DwC, or...) that's a viable solution too, as long as we persist meaning unambiguosly.

I'll have more to say, probably tomorrow, on the last new commit, but I wanted to get the first new commit out today since it fixes my previous attempt to restore GI type, which I think actually causes an exception on any read of an existing GA with a non-null type (specs all passed since no spec references such a GA (apparently)).

I finally got some notes written up (and figured out, I think) for the Ecoregions2017 shapefile - I put them here for now: https://gist.github.com/kleintom/c4c851de52380b0d08b6950e198aa8ba
People just wanting a version of that shapefile that imports cleanly into TaxonWorks can just check the first couple paragraphs and skip everything else, which is more technical regarding the issues I encountered when importing the original shapefile (not all having to do with problems with the shapefile itself). There's a summary at the end so the longer account can be skipped. If there are any issues I missed please let me know.
I don't see a way around the first two RGeo-related issues (given our factory), that said I think those issues are both going to be quite rare, involving shapes crossing +/- 85.051127 latitude (two shapes involving Antarctica in the Ecoregions case) and lines intended to take the  long way around the globe.
The third issue regarding what happens when imported shapes are made_valid has more wiggle room I think, maybe more reporting/warnings would be helpful there.
Lastly I'll point out the following postgis oddity (x)spec'ed in my last commit above:
select ST_Intersects(ST_GeographyFromText('LINESTRING (91 10, 0 0)'), ST_GeographyFromText('LINESTRING (180 89.0, 180 -89.0)'));

select ST_Intersects(ST_GeographyFromText('LINESTRING (91 0, 0 10)'), ST_GeographyFromText('LINESTRING (180 89.0, 180 -89.0)'));

The first returns false, as expected (for me at least): the line LINESTRING (91 10, 0 0) does not cross the anti-meridian, the second returns true: the line LINESTRING (91 0, 0 10) does cross the anti-meridian. The issue only occurs when one of the vertices is at exactly longitude 0, and the second vertex seems to have to be longitude-span > 90 away from 0, other than that I'm not seeing the pattern (feel free to shout if you know what's going on :-). Again, I think this is unlikely to be run into, but if it does become an issue I think the workaround outlined in the last commit message might work and not be too bad (or maybe it can get fixed in postgis if it is a bug).
I'm ready for more shapefile issues is anybody runs into some :-)

@kleintom Thanks very much.  I just got my hands on some shapefiles and hope to hit the interfaces a little today.  We also will collectively explore the UI during tomorrows help sessions, so a first wave of feedback should be coming shortly.

@kleintom can you please add import_gazetteers queue in exe/delayed_job and any other new queue this PR is adding?

Can we scope choosing Gaz shapes in union to some set of projects (maybe, bonus)

Yes, this is doable; we've also mentioned adding the ability to clone from one project to another. Thoughts on which/either/both of those to add?

IIF there are two shapes can we have the option to make the operation INTERSECT

You can now choose Union or Intersection as the method to combine; both operations work on any number of inputs.

Bug- gazetteer_id is likely not permitted in filter params

Yes, filter otus had a bug (the others should have been working), thanks!

Upload shapes
UI - clarify which files might be missing in drag/drop options

Added a visual indicator to the New tab of the document selector to help with this, see what you think.

Keep warning messages live rather than popup?

I reworded a couple messages; if there are still issues here let me know specifics and I can make a change.

Clarify projection translation options if any

You should now be able to import any shapefile that has a .prj file (... and the prj wkt is valid and proj4 knows how to convert that projection to WGS84), for both projected and geographic CRSs. More testing here would be appreciated.

Resetting new needs tweaks

I cleared radio selections from the document selector on submission, if there's anything else I missed feel free to let me know.

Clear failed option

Field names that are the wrong type are now cleared on failed submission, if there's anything else I missed feel free to let me know.
Thanks for the feedback!


miss spell ect on chapter 4

On The web stack of Chapter 4, "ect" may be miss spell of "etc"
"It includes most of what you’d expect such a tool to have, such as a template language, database access, configuration, url routing, ect."

https://github.com/pvlpenev/fullstacklisp/blob/2de60d76da699d364be848c2bba7b20e2851f725/manuscript/caveman.txt#L5 is the specific line.
There's also another ect here: https://github.com/pvlpenev/fullstacklisp/blob/2de60d76da699d364be848c2bba7b20e2851f725/manuscript/sample-intro.txt#L47
(Providing specific links saves time for everyone.)


The offline first approach is breaking usability in search

While saving resources is laudable in general, it comes at the expense of usability.
99% of the time, when I click the search icon on the keyboard after typing a query, I expect fresh and complete results.
In this case, I'm confronted with offline results, that lead me to believe that Open Food Facts has partial to no results?
https://user-images.githubusercontent.com/1689815/116215227-c3893d80-a747-11eb-892d-fa31ddb78623.mp4

@teolemon The thing is that we don't have a partial search API on the server side, do we? That would help!
That being said, we could also be more verbose and state that those are only local results and that you need to click on the search button in order to use the server.

I agree with @teolemon, if we have network, then we should issue a live query. We can show local results first if we have some, but we should issue the query and update the results when get them back.

I agree with @teolemon, if we have network, then we should issue a live query. We can show local results first if we have some, but we should issue the query and update the results when get them back.

@stephanegigandet Like the current search? I've just tested a server search on 'ab': the result is slow (8 seconds), is limited to 500 items, and no systematic match with 'ab'.
I've tested 'abr' and found only one result. In French. Anything with "abricot" should match.
In short, the current server search is either not well set up (smoothie side or server side), and is perhaps not adapted anyway for those fast suggestions. How do you guys do that on the website?

What I can do is link the suggestions to the server instead of a local search: we'll see if it's satisfactory.
In a next step, the idea would be to combine local search and server search results. I don't know if the suggestion tools offer that dual provider mechanism.

I've tried to put the server search instead of the local search for suggestions:

nothing happens in the first seconds, because there it takes time to get the results and there's no progress indicator in the meantime
there are tons of results (e.g. "lentilles" in French) that cannot be sorted anyway, you have to scroll again and again - it's not user-friendly at all

Perhaps a solution would be to:

roll back to local search
clearly indicate that those are local results
and let the user decide to click on the "server search" button

Anyway, I think we missed something: in general, the suggestion tools provide suggestions for keywords, not results.
If I type the first letters of a word in Google, I see suggestions of words, not the corresponding results.

Hence the question: is there an API that provides a list of suggestion Strings from an input String?
For instance it would suggest nutella from nutel, or orange juice and orange cake from oran?
I've just tried on the website, but there are only limited and curious suggestions and I don't know where they come from: only 3 suggestions babybel, biorogg and bolognaise veggie for b, and only babybel for ba.
Without that API we should

remove the suggestion text field altogether
or at least specify that the results are only local and that you have to click on the server button to get server results

My suggestion:

making it obvious that it's only a local search (with the "local search" subtitle)
making more obvious the button for server search (with the "cloud download" icon)

This is a good start, but we should definitely switch to a server side search in the long term.

Fixed in #424.
Given the current server search duration and the open question about "should we display product suggestions, or keyword suggestions like google?", the best we can do now is to avoid possible confusion between server search and local results, and to give users access to both server and local search.


armnn v22.08 Get stuck in rk3399 mali-T860

https://github.com/ARM-software/armnn/issues/638

I have already update acl && armnn to v22.08，this problem still exists.
Is there any method to clean or kill the mali device and related threads, so that memory can be released when stuck happend, and restart armnn functions.

Hi,
My understanding from reading the original issue is that at some point between 5000 and 10000 inferences Arm NN hangs. Can you answer the following for me please:

Are all inferences occurring in a single process?
Are all inferences on the same Network instance(IOptimizedNetworkPtr)? i.e. you load one network and are repeating inferences with different input data.
Are all inferences using the same Runtime instance? You are not loading the network each time.
Are any other instances of Arm NN running in a different process that may access the device at the same time?
Could you give me a rough idea of how large the model being executed is and what data type it is using please.

Thanks,
Colm.

Hey thanks!

yes ,all inferences occurring in a single process's single thread.
I load two network to one runtime instance. I built the first net with Multiplication Layer as a image rescale net. and the second net is a 1mb onnx(from yolov5) net.
yes, all inferences using the same runtime instance, i load two network in sequence, and all inference will running as such sequence
nope
1mb onnx file,  using 320*320 rgbnchw images as input.
plus: it is wired that the stuck occured only when running on rk3399(armv8). but it never stucked when running on rk3288(armv7)
here's my funcs cpp file, thanks again!
lost-det.zip

Ok, that's a very interesting use case.
Looking at the differences in rk3399 I can see that you'll have access to NEON which isn't available on the rk3288. Can you change your test case to exclude CpuAcc and see if the problem persists. If you think all of the layers are available on GpuAcc perhaps you could remove CpuRef too. Similarly you could run only on CpuAcc and eliminate the Gpu as a source of the problem.
I will attempt to recreate something similar on a hikey 960 and see if I get the same problem.
Colm.

@Colm-in-Arm
Oh！this works，I seems found out the problem
When I disable the cpuacc and cpuref use, occurs an error like this.
Warning: ERROR: Layer of type Concat is not supported on any preferred backend [GpuAcc ]
So, the problem may related with the concat layer was not supported in gpuacc. it may stuck when gpuacc results pollin to cpuacc when concat when rk3399 running 5k+ times?
further, it's there any doc for me to develop the gpuacc concat layer support, or it's there any other solution to avoid this stuck both rk3288 rk3399 rk3588 this three development boards I mainly use.

plue, cause of the shortage of cpu resources, I hope that the vast majority of calculations are completed in the gpu~

here's the detail of the error concat.
here comes an strange concat layer with just one input in my onnx model after transed form pt model.and this layer may cause 'armnn::InvalidArgumentException'
what():  Number of inputs provided does not match network.
I can manage the onnx model to erase this layer or add an single input concat layer to set the output same as the input.
This may help to running the program with gpuacc without cpuacc and cpuref, but i am not sure whether this can solve the stuck problem~

I'm pleased you're making some progress. I agree that the concat layer looks very odd but you said this previously worked on v7 + GpuAcc. Could you try the combination of armnn::Compute::GpuAcc, armnn::Compute::CpuRef in that order please. In this case the concat layer might work on the CpuRef implementation. The entire network except that layer would run on the GPU. That way we could isolate the fault to some interaction with CpuAcc.
Colm.

@Colm-in-Arm
I've tried combine gpuacc and cpuref. This stuck still exists.

I've solved the concat problem, and try running all inferences in gpuacc now. but this stuck still exists.

Hey，
I am trying on several rk3399 boards. The boards installed on the robot shows high frequency of this problem (I can guarantee that only my thread using gpu resources). I also try to reproduce this problem on several clean rk3399 boards( installed in computer room). The clean boards have run for 200k+ times and this problem has not yet occurred.
I suspect that it may be a problem with the power supply of the gpu.
Our RK gpu device use simple_ondemand goverment. The voltage maybe cannot keep up with the frequency. So I changed goverment from odemand to performance whitch has the highest frequency, and it will no longer de regulate the frequency.
But the problem is the stuck thread been straced to event (pollin), It seems like a data copy problem?
I'm not sure LOL.
I'll keep following this and update this issue~
thanks~

some dead locks~

Is it possible that the voltage instability causes gpu fail to return an infer result,  then the lock cannot be opened?
I found out when this stuck happend, gpu can back to normal soon，but this stuck will be forever. I could not unload
classifier network by UnloadNetwork function when stuck. After I kill my process and re run, infer will back to normal again.

I'm afraid you've gone beyond my knowledge of the GPU architecture.
I did try running many thousands of inferences on a HiKey 960 device and I didn't hit any problems.
I might refer this to my colleagues in Arm Computer Library. They have the GPU experience.
Colm.

I don't think OpenCL has an abort mechanism. But since you are able to kill the process and restart it, the process is not in an uninterruptable sleep state.
If you really want to avoid killing the process and restarting, I'd maybe try installing a signal handler for SIGUSR1 in main, then starting a watchdog thread that (somehow) detects the hang state and sends SIGUSR1 to the process. That will cause any system calls to wake up, which might give them a chance to detect the error state, return an error, and let you unload/reload the network, etc.
I really don't know if it would be worth the trouble, though! It's quite possible that something in the OpenCL driver or Arm NN would be left in an inconsistent state and require restarting the process anyway :-(
Hope that helps,
Matthe

Thanks for reply！ @MatthewARM
My current method is to turn the infer task into an independent process,  and let it communicate with my main process. (However, there will be a lot of performance loss caused by data copy and trans), so I prefer a watch dog.
And I also tried add a watchdog thread, but I found that when the blocking occurs, armnn unload mode cannot release the network (UnloadNetwork function returns false), When this happens, how can I release it correctly？

Here's two process,

Robot Process(main process, control my robot behaviour, which should never be killed or restarted)
Infer Process(infer images and get recognition results)

I wrote a watchdog thread in infer process with a function which would cause a segment fault.
When watchdog detected the infer thread not update infer result for 30s, call this segemt fault function and whole process will be killed.
Or I may call system("sudo systemctl restart infer_process.service") to restart infer process by service.
But is this the only way to solve this problem?
I dont want an independent infer process(data communicate between already became a very complex logic, not to say the image data copy and send loss). I wish an infer thread, when watchdog detect infer been stuck,  I could release armnn network and resources by watchdog thread, and armnn throw an exception when the infer could not get results(unlock the lock when network unload detected  while condition var waitting ).
Is that doable? LOL

Hi @junyuancat1
Since we could not reproduce the problem on our devices and there has been no activity I'm closing this issue. If you still need help with this then please create a new ticket.


Issues with unit tooltip generation

A general bucket for any outstanding issues with automated unit tooltip generation.

Searing bite has an icon and a real ability, it appears twice in the unit description


Missing key_previous field

https://github.com/OTRF/OSSEM-CDM/blob/dacbc880b04e9b8ca0ae07b42614c1e0f9bcc8bf/schemas/entities/registry.yml#L23
Both Sysmon event 14 and Microsoft Defender 365 "RegistryKeyRenamed" events reference both the original and the new keys, and hence two key fields are needed "key" and "key_modified"

We have a some questions about this. We don't want to close this issue yet. Let's discuss it in our next community meeting. We can review the previous commit.


TimePicker Disabled

How i can disabled timepicker?

We're working on the big refactor. According to this issue the feature of re-initing, manual initing or disabling will be available soon.


Fix crashing docker daemon in binderhub-staging

Ref https://github.com/2i2c-org/infrastructure/issues/3588

The process of debugging this was documented in https://youtube.com/live/lWp5BItlde4?feature=share

Thanks @sgibson91


LeagueState Change redesign

Start daily process only if "auto state change" is selected for a region.
This is a redesign and related to issue  #344
Enable auto state change only for registration->select and select->freeze
and when dates are set

preferred:
registration->selection starts at 08:00
selection->freeze starts at 20:00

auto_state_change is disabled for now.
Following dates can be set on region level:

open selection
close selection
open scheduling
close scheduling
close referees

Two daily jobs are started:

runs at 08:00 and handles open state changes
runs at 20:00 and handles close state changes

If no date set, nothing happens.
If clubs are notified about state changes

To re-introduce auto-state-change:
If true
then a daily job is started
and all dates are ignored and league state changes are handled by completeness rules only
I not true the job is killed


Large Model Support

System information

TensorFlow version (you are using): 2.2 nightly
Are you willing to contribute it (Yes/No): Code exists

code is here:
https://github.com/IBM/tensorflow-large-model-support
Describe the feature and the current behavior/state.
TensorFlow Large Model Support (TFLMS) is a feature in the TensorFlow provided by IBM Watson Machine Learning Community Edition (WML CE) that allows the successful training of deep learning models that would otherwise exhaust GPU memory and abort with "out-of-memory" errors. LMS manages this oversubscription of GPU memory by temporarily swapping tensors to host memory when they are not needed.
Will this change the current api? How?
will add:
tf.config.experimental.set_lms_enabled(True)
Who will benefit with this feature?
everyone who trains models

Hi @bela127, are you proposing to implement this feature (i.e. will you send a PR at some point)?

CC @reedwm

Hi,
I definitely can help, but I'm not that deep into tensor flow backend, and more a python developer, than c++, so I guess I will need help.
The code is ready for tf 2.1 , so one could try to do the merging, but 2.2 changed a lot of API, at least at the python side, so I'm not shure how much has to be changed.

I think having an RFC for this feature would be the best process to getting it into TensorFlow. The link RFC documentation describes you should first email<EMAIL_ADDRESS>to discuss the proposal.

@reedwm
ok, thanks.
if i understand the RFC documentation correctly, i should email<EMAIL_ADDRESS>directly, without creating a RFC document or feature-request before?
if this is correct, i will send a mail with a short description and a link to this issue.

@alextp can you confirm emailing<EMAIL_ADDRESS>is the best process? And since @bela127 is fairly busy, is it better to wait to start the RFC process until they have more time?

Email with a description of the plan to<EMAIL_ADDRESS>is the right way to start.
I will say though that per the TF API guidelines in https://github.com/tensorflow/community/blob/master/governance/design-reviews.md#programmability--flexibility (specially the programmability section) I am not a fan of a one-push-button switch for this behavior. Can we build it instead out of other pieces in the TF API that would in principle be recombinable to achieve other results?

Ok, Il send the mail.
@alextp
I just reviewed the TF API guidelines, and instead of the "one button" solution with tf.config.experimental.set_lms_enabled(True) (as it is implemented by IBM) one could implement a context manager.
something like:
with auto_swap_tensors_to_dev("cpu"):
every tensor defined in the context gets swaped, but you can swap different tensors/model parts to other devs.
The only other solution i can think of, would be several evaluation hooks:
do_before_evaluation_step(func) func gets called before every calculation
get_next_tensors_in_pipeline() gets names/symbolic tensors that are needed for calculation
get_unused_tensors() gets names/symbolic tensors that are currently not needed for calculation
swap_tensors_from_dev_to_dev(tensor, dev_source, dev_target) dev copy already possible, but this should not create "new" symbolic tensor, instead it performs a dev swap.
with this one could write:
def before_evaluation_callback():
   unused = get_unused_tensors()
   swap_tensors_from_dev_to_dev(unused, "gpu", "cpu")
   next = get_next_tensors_in_pipeline()
   swap_tensors_from_dev_to_dev(next, "cpu", "gpu")

do_before_evaluation_step(before_evaluation_callback)

instead of:
swap_tensors_from_dev_to_dev(tensor, dev_source, dev_target)
it could be slitted into, smaler steps:
copy_tensor_to_dev(tensor, dev_target) Is already possible
clear_tensor_from_dev(tensor, dev_source)
load_tensor_from_dev(tensor, dev_source)
this would give the possibility to implement various kind of memory management options:

the proposed LMS (https://github.com/IBM/tensorflow-large-model-support) -> see above
gradient checkpointing (https://github.com/cybertronai/gradient-checkpointing) -> just delete every n tenors, and they are recomputed if needed. -> how to select good tensors to delete?

i hope i could provide a basis for discussion.

My main concern is that multiple one-button solutions tend to not combine
well with each other and with unforeseen features of TF.
On Wed, Apr 1, 2020 at 12:13 PM Bela Böhnke<EMAIL_ADDRESS>wrote:

Ok, Il send the mail.
@alextp https://github.com/alextp
I just reviewed the TF API guidelines, and instead of the "one button"
solution with tf.config.experimental.set_lms_enabled(True) (as it is
implemented by IBM) one could implement a context manager.
something like:
with auto_swap_tensors_to_dev("cpu"):
every tensor defined in the context gets swaped, but you can swap
different tensors/model parts to other devs.
The only other solution i can think of, would be several evaluation hooks:
do_before_evaluation_step(func) func gets called before every calculation
get_next_tensors_in_pipeline() gets names/symbolic tensors that are
needed for calculation
get_unused_tensors() gets names/symbolic tensors that are currently not
needed for calculation
swap_tensors_from_dev_to_dev(tensor, dev_source, dev_target) dev copy
already possible, but this should not create "new" symbolic tensor, instead
it performs a dev swap.
with this one could write:
def before_evaluation_callback():
unused = get_unused_tensors()
swap_tensors_from_dev_to_dev(unused, "gpu", "cpu")
next = get_next_tensors_in_pipeline()
swap_tensors_from_dev_to_dev(next, "cpu", "gpu")
do_before_evaluation_step(before_evaluation_callback)
instead of:
swap_tensors_from_dev_to_dev(tensor, dev_source, dev_target)
it could be slitted into, smaler steps:
copy_tensor_to_dev(tensor, dev_target) Is already possible
clear_tensor_from_dev(tensor, dev_source)
load_tensor_from_dev(tensor, dev_source)
this would give the possibility to implement various kind of memory
management options:

the proposed LMS (
https://github.com/IBM/tensorflow-large-model-support) -> see above
gradient checkpointing (
https://github.com/cybertronai/gradient-checkpointing) -> just delete
every n tenors, and they are recomputed if needed. -> how to select good
tensors to delete?

i hope i could provide a basis for discussion.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/37982#issuecomment-607439505,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAABHROHHS4SXRV424ZUPPLRKOG63ANCNFSM4LVFXJMQ
.

--

Alex

I fully understand, one button solutions are nice for high level work, but for low level work, multiple single functionality is much better composable.
that is why i tried to decompose the one button solution

i just recognized that this feature request is basically, the request for a hook into tensorflows tensor memory management strategy.
at the moment tensorflow only knows the, "calculate and keep" strategy, but there are a lot of possible other strategies and every thing between them, like moving tensors to other memory devs.
High Level "one button" switches to choose between different strategies, would be nice, like the introduced strategies for distribution but for memory management.
but a low level api for building ones one strategy, as suggested by @alextp would although be nice, likewise a low level api for building distribution strategies exists with explicit dev placement, also Im not aware of an api for building custom automated distribution strategies.

the features could be implemented with a Api to, the meta optimizer
I found in the slides the memory optimizer, but i cant find it in the api docs, is this undocumented or not available?
In the slides It seems it already supports the features I asked for, there is also this side, but i cant find anything in the API Docs how to configure it.
If a known and good API, it would be the perfect place for implementing all memory management strategies, and support for custom ones

Hi,
Thank you for opening this issue. Since this issue has been open for a long time, the code/debug information for this issue may not be relevant with the current state of the code base.
The Tensorflow team is constantly improving the framework by fixing bugs and adding new features. We suggest you try the latest TensorFlow version with the latest compatible hardware configuration which could potentially resolve the issue. If you are still facing the issue, please create a new GitHub issue with your latest findings, with all the debugging information which could help us investigate.
Please follow the release notes to stay up to date with the latest developments which are happening in the Tensorflow space.


Update submissions-admin.md

The images are missing please contact me and I can provide them alias madalenamedo.
Thank you!

Learn Build status updates of commit 3416393:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

microsoft-365/security/office-365-security/submissions-admin.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the learn.microsoft.com contributor guidesPost your question in the Learn support channel

@microsoft-github-policy-service agree [company="Microsoft"]

@microsoft-github-policy-service agree company="Microsoft"

@JohanFreelancer9 done, thank you!

Learn Build status updates of commit ac276ec:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

microsoft-365/security/office-365-security/submissions-admin.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the learn.microsoft.com contributor guidesPost your question in the Learn support channel

Learn Build status updates of commit 6e1e869:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

microsoft-365/security/office-365-security/submissions-admin.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the learn.microsoft.com contributor guidesPost your question in the Learn support channel

Learn Build status updates of commit 728eb5b:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

microsoft-365/security/office-365-security/submissions-admin.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the learn.microsoft.com contributor guidesPost your question in the Learn support channel

Learn Build status updates of commit 41cdf35:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

microsoft-365/security/office-365-security/submissions-admin.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the learn.microsoft.com contributor guidesPost your question in the Learn support channel

Images requested. Will review and merge once they're in.

Learn Build status updates of commit a1660d6:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

microsoft-365/security/office-365-security/submissions-admin.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the learn.microsoft.com contributor guidesPost your question in the Learn support channel


Resize picture if very large

1600x1200 (~300KB) should be enough to display comfortably on computer screen.
Let the user choose original file instead :
| Uploading resized picture (300KB) ... |
[ OK ]
[ Upload original file (13MB) instead ]

Resizing works fine using Bitmap.createScaledBitmap()

Resize by default looks legit, and works fine.
But, we'd still like the possibility to upload the full original picture file.


SetImplicitShowAnimation not working in ItemsControl with ItemsStackPanel

Describe the bug
When an ImplicitShowAnimation is set on an element that's hosted inside of an ItemsControl that uses <ItemsStackPanel>  as the ItemsPanelTemplate, the animation appear to not run.
Steps to reproduce the bug
Download and run the complete sample from CompositionAnimationsItemsControl.zip
To summarize:

Set up the ItemsControl

<ItemsControl>
    <ItemsControl.Items>
        <Rectangle x:Name="Rect" Height="200" Width="200" Fill="Red" Visibility="Collapsed" Opacity="0" />
    </ItemsControl.Items>

    <ItemsControl.ItemsPanel>
        <ItemsPanelTemplate>
            <!--  Orientation does not matter  -->
            <ItemsStackPanel Orientation="Horizontal" />
        </ItemsPanelTemplate>
    </ItemsControl.ItemsPanel>
</ItemsControl>

Set up the ImplicitShowAnimation

Compositor compositor = ElementCompositionPreview.GetElementVisual(Rect).Compositor;
var animation = compositor.CreateScalarKeyFrameAnimation();
animation.Duration = System.TimeSpan.FromSeconds(1);
animation.InsertKeyFrame(0, 0);
animation.InsertKeyFrame(1, 1);
animation.Target = "Opacity";

ElementCompositionPreview.SetImplicitShowAnimation(Rect, animation);

Trigger the animation

Rect.Visibility = Windows.UI.Xaml.Visibility.Visible;

Expected behavior
The rectangle should fade in.
Screenshots
No response
NuGet package version
Microsoft.UI.Xaml 2.7.0
Windows app type

[x] UWP
[ ] Win32

Device form factor
Desktop
Windows version
No response
Additional context
Tested on Windows 11 22000.348

@gegao18 or @stmoy is this an understood issue? I don't see anything obviously wrong.

I've been hitting this bug with my app. Any updates here?

Bumping, based on https://github.com/microsoft/microsoft-ui-xaml/discussions/8638

Sorry for the bump, just hit this myself. Any updates?


[RFC] Trusted storage proposal for CoCo

Motivation
The CoCo stack utilizes hardware-based TEE technology for runtime protection which provides the confidential memory. Current CoCo uses confidential memory for data storage like container images, container RW layers. Memory is an expensive resource and we need a low cost solution to provide trusted storage as part of CoCo stack. Thanks @jiangliu and @liangzhou121, this proposal is based on https://github.com/confidential-containers/documentation/pull/23 and https://github.com/confidential-containers/documentation/issues/20 .
Short Term Goals

Provide Confidential Ephemeral Storage

Auto crypt setup(encrypt) for the storage, use dm-crypt to protect the block volume
Use dm-integrity to protect the integrity of encrypted block volume
Host downloaded container images (Pod level)
Host RW layer of container rootfs
Provide ephemeral volume for containers

Compatible with current Kubernetes PodSpec

Use pause sidecar container to host the block volume

Integrate with at least one Kubernetes CSI driver

open-local: https://github.com/alibaba/open-local
topolvm: https://github.com/topolvm/topolvm

Long Term Goals

Provide Confidential Persistent Storage

Auto crypt setup(encrypt/decrypt) for the storage
Use dm-integrity to protect the integrity of encrypted block volume

Transparent for the pod spec of enduser

Utilize Kubernetes MutatingAdmissionWebhook to auto inject the config

Kubernetes support Pod level storage resource definition and allocation

High Level Architecture

This is the high level architecture for trusted storage for CoCo. The CSI driver will dynamically provision and allocate a block volume for the pod. We will associate this block volume with a sidecar container with pause image. Since the block volume will be used to store container images, to avoid the bootstrap issue, the pause image will be in the rootfs by default to avoid image downloading. After the block volume is attached to guest through virtio-blk, the kata-agent will do the initialization and crypt setup work, if the block device is encrypted, kata-agent will get the decrypt key from trusted party like attestation-agent.
Implementation Details

PodSpec Definition
Since the volumes consumed by a pod need associate with one container, we choose use a sidecar container with pause image to host the block volume:

apiVersion: v1
kind: Pod
metadata:
  name: trusted-lvm-block
spec:
  runtimeClassName: kata-cc
  containers:
   - name: sidecar-trusted-store
     image: pause
     volumeDevices:
     - devicePath: "/dev/trusted_store"
       name: trusted-store

  volumes:
   - name: trusted-store
     persistentVolumeClaim:
       claimName: trusted-store-block-pvc

Kata Guest Kernel
Add dm-crypt/integrity required kernel config
Kata rootfs
Add pause image, cryptsetup and filesystem utilities related packages into the rootfs
Kata-agent
Detect the block volume with device path: /dev/trusted_store, start the crypt setup and file system initialization for this volume and mount the encrypted disk to the container image store path: /run/image. The following containers images and container RW layers will saved in this directory.

This is the demo link:
https://asciinema.org/a/W90GYRZmFoCb1nEIKS3BjAToH
Potential GAPs

Container launch sequence
The initContainer will launch sequencely and will wait previous container finished successfully, but normal container for runc runtime will launch in paralle, but for kata runtime I found they also run sequencely, but we need define the sidecar container before normal application container.

Storage resouce restriction for each container
After kata-agent setup the block volume and export as directory for other containers to use, currently there don't have resource restriction for each container

Container defined mount path support
In pod yaml file, the volumes can be used as shared volume to allow the communication between containers, after kata-agent manage the block volume, the definition for each containers to define the mount path also need support.

Block volume sharing between pods
Since the volume is encrypted and can not mount by other pods, sharing between pods is not supported

Devel Plan for CoCo first release
For CoCo September first release, we will support the ephemeral trusted storage to host container images and RW layers.

[x] Guest kernel support dm-integrity, dm-crypt
[x] Integrate cryptsetup and filesystem utilities into Kata rootfs
[x] Package pause image inside Kata rootfs
[x] Kata agent crypt setup raw block volume with shell scripts
[ ] Kata agent crypt setup raw block volume with rust native implementation(one GAP is libcryptsetup-rs don't support musl build)
[x] Mount the encrypted disk for image/container RW layer store

The next step will be support the persistent trusted storage with Kata direct volume features.

[ ] Kata agent retrive disk encrypted key from attestation-agent
[ ] Kata agent crypt setup persistent encrypted raw block vloume

Hello. Is there any possibility to share pvc between containers within a single pod in the current version? If I understand correctly, now each container has its own layer (/run/image/overlay//upperdir) and the data is isolated?

We can close this issue now:
Trusted Ephemeral Storage for container images: https://github.com/confidential-containers/confidential-containers/blob/main/guides/ephemeral_storage.md
Trusted persistent storage: https://github.com/confidential-containers/guest-components/blob/main/confidential-data-hub/docs/SECURE_STORAGE.md


Add example for C2487

Added an example and a "See also" section.

@TylerMSFT

Can you review this PR?
IMPORTANT: When this content is ready to merge, you must add #sign-off in a comment or the approval may get overlooked.

#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team


⚠ DesiTorrents has degraded performance

In 7716d95, DesiTorrents ($DST) experienced degraded performance:

HTTP code: 200
Response time: 1114 ms

Resolved: DesiTorrents performance has improved in 8bd5f53 after 13 minutes.


Synthesis failed for java-iam

Hello! Autosynth couldn't regenerate java-iam. :broken_heart:
Please investigate and fix this issue within 5 business days.  While it remains broken,
this library cannot be updated with changes to the java-iam API, and the library grows
stale.
See https://github.com/googleapis/synthtool/blob/master/autosynth/TroubleShooting.md
for trouble shooting tips.
Here's the output from running synth.py:
aa818de324632b5
2021-06-02 01:54:44,517 autosynth [DEBUG] > Running: git log -1 --pretty=%at bb5ef5a99df6d439c63fddf0f7b6fe58e0679b10
2021-06-02 01:54:44,520 autosynth [DEBUG] > Running: git log -1 --pretty=%at 8ac413e632a517438e03779e6d788384d6e2898d
2021-06-02 01:54:44,523 autosynth [DEBUG] > Running: git log -1 --pretty=%at a2de127f90799cd5bace653b1dcc5dd2ca8b19c9
2021-06-02 01:54:44,526 autosynth [DEBUG] > Running: git log -1 --pretty=%at 1778de119522ea9c9b18763c2a4577e22a60433c
2021-06-02 01:54:44,529 autosynth [DEBUG] > Running: git log -1 --pretty=%at d543abd7c1f500bdbe530a35ca174ee374fde8f6
2021-06-02 01:54:44,532 autosynth [DEBUG] > Running: git log -1 --pretty=%at 7c50355dbd51333f43f53211d586a173b02d26df
2021-06-02 01:54:44,535 autosynth [DEBUG] > Running: git log -1 --pretty=%at c3e41da0fa256ad7f6b4bc76b9d069dedecdfef4
2021-06-02 01:54:44,539 autosynth [DEBUG] > Running: git checkout 708625ec17a9e01a7d2e61e2b64098638fb1b5ab
Note: switching to '708625ec17a9e01a7d2e61e2b64098638fb1b5ab'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 708625e build(deps): update dependency org.apache.maven.plugins:maven-project-info-reports-plugin to v3.1.2 (#139)
2021-06-02 01:54:44,546 autosynth [DEBUG] > Running: git checkout 46f4f7054fc945d7af9474fe5d70e623c46d0dd5
Note: switching to '46f4f7054fc945d7af9474fe5d70e623c46d0dd5'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 46f4f705 chore: update ruby rules to use the correct assembly_pkg rule
2021-06-02 01:54:44,561 autosynth [DEBUG] > Running: git checkout c3e41da0fa256ad7f6b4bc76b9d069dedecdfef4
Note: switching to 'c3e41da0fa256ad7f6b4bc76b9d069dedecdfef4'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at c3e41da fix: make `generate_index_ts()` deterministic (#1104)
2021-06-02 01:54:44,639 autosynth [DEBUG] > Running: git branch -f autosynth-430
2021-06-02 01:54:44,643 autosynth [DEBUG] > Running: git checkout autosynth-430
Switched to branch 'autosynth-430'
2021-06-02 01:54:44,647 autosynth [INFO] > Running synthtool
2021-06-02 01:54:44,648 autosynth [INFO] > ['/tmpfs/src/github/synthtool/env/bin/python3', '-m', 'synthtool', '--metadata', 'synth.metadata', 'synth.py', '--']
2021-06-02 01:54:44,648 autosynth [DEBUG] > log_file_path: /tmpfs/src/logs/java-iam/430/sponge_log.log
2021-06-02 01:54:44,649 autosynth [DEBUG] > Running: /tmpfs/src/github/synthtool/env/bin/python3 -m synthtool --metadata synth.metadata synth.py --
2021-06-02 01:54:44,881 synthtool [DEBUG] > Executing /home/kbuilder/.cache/synthtool/java-iam/synth.py.
On branch autosynth-430
nothing to commit, working tree clean
2021-06-02 01:54:45,041 synthtool [DEBUG] > Cloning googleapis.
2021-06-02 01:54:45,042 synthtool [DEBUG] > Using precloned repo /home/kbuilder/.cache/synthtool/googleapis
2021-06-02 01:54:45,046 synthtool [DEBUG] > Generating code for: //google/iam/v1:iam_java_proto.
2021-06-02 01:55:47,233 synthtool [ERROR] > Failed executing bazel --max_idle_secs=240 build //google/iam/v1:iam_java_proto:

Loading:
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
INFO: Call stack for the definition of repository 'com_google_protobuf' which is a http_archive (rule definition at /home/kbuilder/.cache/bazel/_bazel_kbuilder/a732f932c2cbeb7e37e1543f189a2a73/external/bazel_tools/tools/build_defs/repo/http.bzl:296:16):
 - <builtin>
 - /home/kbuilder/.cache/synthtool/googleapis/WORKSPACE:48:1
WARNING: Download from https://github.com/protocolbuffers/protobuf/archive/v3.15.3.tar.gz failed: class java.io.IOException GET returned 500 Internal Server Error
ERROR: An error occurred during the fetch of repository 'com_google_protobuf':
   java.io.IOException: Error downloading [https://github.com/protocolbuffers/protobuf/archive/v3.15.3.tar.gz] to /home/kbuilder/.cache/bazel/_bazel_kbuilder/a732f932c2cbeb7e37e1543f189a2a73/external/com_google_protobuf/v3.15.3.tar.gz: GET returned 500 Internal Server Error
ERROR: no such package '@com_google_protobuf//': java.io.IOException: Error downloading [https://github.com/protocolbuffers/protobuf/archive/v3.15.3.tar.gz] to /home/kbuilder/.cache/bazel/_bazel_kbuilder/a732f932c2cbeb7e37e1543f189a2a73/external/com_google_protobuf/v3.15.3.tar.gz: GET returned 500 Internal Server Error
INFO: Elapsed time: 62.156s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)
FAILED: Build did NOT complete successfully (0 packages loaded)

Traceback (most recent call last):
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmpfs/src/github/synthtool/synthtool/__main__.py", line 102, in <module>
    main()
  File "/tmpfs/src/github/synthtool/env/lib/python3.6/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/tmpfs/src/github/synthtool/env/lib/python3.6/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/tmpfs/src/github/synthtool/env/lib/python3.6/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/tmpfs/src/github/synthtool/env/lib/python3.6/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/tmpfs/src/github/synthtool/synthtool/__main__.py", line 94, in main
    spec.loader.exec_module(synth_module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/kbuilder/.cache/synthtool/java-iam/synth.py", line 85, in <module>
    build_proto(f"//google/iam/{version}:iam_java_proto")
  File "/home/kbuilder/.cache/synthtool/java-iam/synth.py", line 58, in build_proto
    cwd=googleapis,
  File "/home/kbuilder/.cache/synthtool/java-iam/synth.py", line 47, in bazel_build
    shell.run(bazel_run_args)
  File "/tmpfs/src/github/synthtool/synthtool/shell.py", line 39, in run
    raise exc
  File "/tmpfs/src/github/synthtool/synthtool/shell.py", line 33, in run
    encoding="utf-8",
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['bazel', '--max_idle_secs=240', 'build', '//google/iam/v1:iam_java_proto']' returned non-zero exit status 1.
2021-06-02 01:55:47,297 autosynth [ERROR] > Synthesis failed
2021-06-02 01:55:47,297 autosynth [DEBUG] > Running: git reset --hard HEAD
HEAD is now at 708625e build(deps): update dependency org.apache.maven.plugins:maven-project-info-reports-plugin to v3.1.2 (#139)
2021-06-02 01:55:47,302 autosynth [DEBUG] > Running: git checkout autosynth
Switched to branch 'autosynth'
2021-06-02 01:55:47,307 autosynth [DEBUG] > Running: git clean -fdx
Removing __pycache__/
Traceback (most recent call last):
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmpfs/src/github/synthtool/autosynth/synth.py", line 356, in <module>
    main()
  File "/tmpfs/src/github/synthtool/autosynth/synth.py", line 191, in main
    return _inner_main(temp_dir)
  File "/tmpfs/src/github/synthtool/autosynth/synth.py", line 336, in _inner_main
    commit_count = synthesize_loop(x, multiple_prs, change_pusher, synthesizer)
  File "/tmpfs/src/github/synthtool/autosynth/synth.py", line 68, in synthesize_loop
    has_changes = toolbox.synthesize_version_in_new_branch(synthesizer, youngest)
  File "/tmpfs/src/github/synthtool/autosynth/synth_toolbox.py", line 259, in synthesize_version_in_new_branch
    synthesizer.synthesize(synth_log_path, self.environ)
  File "/tmpfs/src/github/synthtool/autosynth/synthesizer.py", line 120, in synthesize
    synth_proc.check_returncode()  # Raise an exception.
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/subprocess.py", line 389, in check_returncode
    self.stderr)
subprocess.CalledProcessError: Command '['/tmpfs/src/github/synthtool/env/bin/python3', '-m', 'synthtool', '--metadata', 'synth.metadata', 'synth.py', '--']' returned non-zero exit status 1.

Google internal developers can see the full log here.

Autosynth passed, closing! :green_heart:


Set feedback textarea character limit

We're seeing long feedback get truncated in Sentry. We should consider adding a limit (and character counter) to help set expectations.

For example, the message attribute is limited to 1024 characters  https://docs.sentry.io/accounts/quotas/#attributes-limits

Sentry is capping the message at 250 characters right now, which is different than the 1024 value that their documentation states. I sent an email to their support team to figure out which number is correct.
250 feels like it could be a little too limiting.

From Sentry support:

By default, we cap it to 250 characters. You can configure your SDK and add the option maxValueLength within your SDK config.
More information here https://docs.sentry.io/platforms/javascript/#truncating-strings-on-the-event.

We should look up the average sent feedback length to help determine what the max should be.

The average feedback length is 95.5 characters, but I think we should set the character limit at 1,000.

I created a Mode report to examine the feedback lengths of all feedback: https://app.mode.com/mapbox/reports/ed77d021f548
Only 0.75% of all feedback was >1,000 characters. These extra long feedback were mostly junk (user's copying and pasting long examples) or non-actionable.
Looking at the feedback between 500-1000 characters, I do see good reports from user that did require that length.

@mapbox/docs any concerns with setting the feedback limit to 1000 chars?


ActivityType Hierarchy

Hey,
I'm currently working on my masterthesis where I process the Google Location History. I stumbled across this Project (sadly a little bit late) and I love it. Your work helped me quite a bit to understand the data.
In the ActivityType Segment it is written that some types are sub-activity of IN_VEHICLE. From my finding I can confirm this. However I would add to this that  IN_BUS and IN_CAR is  is also a  sub-activity  of IN_FOUR_WHEELER_VEHICLE. If analyzed my data an for every activity IN_FOUR_WHEELER_VEHICLE was included IN_BUS and IN_CAR added up to the exact confidence of IN_FOUR_WHEELER_VEHICLE.
I also found that IN_FOUR_WHEELER_VEHICLE and IN_TWO_WHEELER_VEHICLE add up to IN_ROAD_VEHICLE (plus minus 1, which is probably a rounding mistake)
Also in a lot of cases (there are also a lot of cases where it not the case) IN_FOUR_WHEELER_VEHICLE, IN_TWO_WHEELER_VEHICLE and IN_RAIL_VEHICLE add up to IN_VEHICLE. In the cases where these values don't add up they are always bigger than IN_VEHICLE.
The hierarchy probably looks like this.
                    IN_VEHICLE
                       │   │
                       │   │
                       │   │
                       │   │
IN_RAIL_VEHICLE ◄──────┘   └───────► IN_ROAD_VEHICLE
                                          │   │
                                          │   │
                                          │   │
                                          │   │
                IN_FOUR_WHEELER_VEHICLE ◄─┘   └─► IN_TWO_WHEELER_VEHICLE
                         │   │
                         │   │
                         │   │
                         │   │
                IN_CAR ◄─┘   └──► IN_BUS

[
  {
    "type": "IN_VEHICLE",
    "confidence": 97
  },
  {
    "type": "IN_ROAD_VEHICLE",
    "confidence": 97
  },
  {
    "type": "IN_FOUR_WHEELER_VEHICLE",
    "confidence": 59
  },
  {
    "type": "IN_CAR",
    "confidence": 42
  },
  {
    "type": "IN_TWO_WHEELER_VEHICLE",
    "confidence": 37
  },
  {
    "type": "IN_BUS",
    "confidence": 17
  },
  {
    "type": "STILL",
    "confidence": 1
  },
  {
    "type": "UNKNOWN",
    "confidence": 1
  },
  {
    "type": "IN_RAIL_VEHICLE",
    "confidence": 1
  }
]

So there is definitely more to the story than there are subtypes to IN_VEHICLE. I hope this helped and please let me know if I should do some further testing.

Hi Jonathan!
I'm happy you are finding this project useful :)
And hey, this is very interesting, thanks for the analysis and thanks for sharing! I hadn't considered that a more complex hierarchy could exist.
I checked with my data and I could replicate your observations.
I was curious if there was more so I also built a quick script to automatically find these kind of acitivity relationships in my data (using a very simple logic, basically find all pairs of activitity types A and B, where A.confidence is always >= B.confidence).
And the output was exactly the same hierarchy as the one you suggested :D (and also the already known ON_FOOT -> (RUNNING | WALKING)).
I think we can assume that this hierarchy is correct (and as far as we know complete), so I'll update the documentation with these new findings in the coming days.
Thanks again, and do share if you find anything else!

Hi again!
I've now added this hierarchy information in the ActivityType section.
This is the best way I've found to put it, unfortunately everything has to go in a single line string in the JSON schema (here) so it is a bit difficult to work with, I'm open to suggestions on how to handle and represent this information better.
For now I'll close the issue :)


JAR not working

If I run the EscherConverter.main(String...) from the IDE or command-line(like in Travis), everything works fine. But if I try to run the exported runnable JAR, it throws:
Error: Could not find or load main class edu.ucsd.sbrg.escher.EscherConverter

I don't what's causing this, but this is a big problem as it will be released as a JAR, so it needs to be resolved. @draeger Can you look into what might be causing this? The manifest is also correct.

This is most likely caused by either a wrong entry in the Manifest file or the class is missing in the JAR. Interstingly, this did work before (there is a downloadable version of EscherConverter). How did you build the JAR?

I'm using the maven assembly plugin, and there have been no changes in its configuration since the start.
I went through the old commits, building them one by one, see it stopped working, and I found out that when you add the reconfigured the libs here. Maybe there is some problem with the pom files, I'm not really sure. Can you check?
It is not evident to me what exactly happened in that commit.

This is almost solved now with the latest commit. There is now just one problem that needs to be solved, namely it doesn't find one resource file, but the POM builds an executable Jar.

If you stumble across this issue years later like I do, remember to build the JAR using
mvn clean compile assembly:single


Streaming and caching

Is possible do such things with the library?
start playing .mp3 from url immediately (like streaming or streaming)
cache it and track caching progress during it's playback
save cached file to local storage

Currently there is no support in this library to cache incoming audio. We used this library to stream from SoundCloud, which did not allow us to cache audio for legal reasons. However, the feature would be welcome!


QAT Engine tested with latest openssl and tls1.3?

Hi, Is QAT Engine tested with openssl 1.1.1 and tls1.3? Thanks.

Hi @felix94536,
We expect OpenSSL to make the official OpenSSL 1.1.1 release tomorrow (11th Sep 18) unless something unexpected crops up as they froze their repo to all commits on Sunday.
At that point we will begin testing against OpenSSL 1.1.1. We'll begin with testing TLS 1.2 for regression and move onto TLS 1.3 to test RSA-PSS and elliptical curves for handshaking. I expect there to be a few weeks lead time before we make a new QAT Engine release confirming things are working correctly. As part of the engine release we will be moving our default QAT Engine build to be against the OpenSSL 1.1.1 baseline (as it is the LTS baseline) and an additional configure option will be required to continue building against OpenSSL 1.1.0.
Kind Regards,
Steve.

Sounds great. I will check back after a few weeks. Thanks, Steve!


Cleanup DLQ on create-consumer-group error

Found one more case we missed .. that's causing us to leak DLQs.

Changes Unknown when pulling 9d1e3285d52e48e142d7db6a5e556f815420fdaf on cleanup-dlq-dest into ** on master**.

Coverage increased (+0.6%) to 67.304% when pulling 606d0bc7fc9d26ae1370d3c336d2d845edc40345 on cleanup-dlq-dest into 371b784adafa05f9ce37ca6311a0860b223f7484 on master.

Coverage increased (+0.6%) to 67.304% when pulling 606d0bc7fc9d26ae1370d3c336d2d845edc40345 on cleanup-dlq-dest into 371b784adafa05f9ce37ca6311a0860b223f7484 on master.