📶 🤳 Offline scanning (Tracker)

What
### Tasks
- [ ] openfoodfacts/openfoodfacts-server#6988
- [ ] https://github.com/openfoodfacts/smooth-app/issues/4067
- [ ] https://github.com/openfoodfacts/smooth-app/issues/4066
- [x] Add a local product cache
- [ ] #2452
- [ ] #2450
- [ ] openfoodfacts/smooth-app#2446
- [ ] openfoodfacts/smooth-app#2444
- [x] openfoodfacts/smooth-app#2445
- [x] openfoodfacts/smooth-app#2447
- [x] #2451
- [ ] https://github.com/openfoodfacts/smooth-app/issues/4025

Why

No network or slow in supermarkets
Add offline scanning with a reduced local copy of the database so that users can scan offline.
This feature has been documented - Design document

Part of

openfoodfacts/smooth-app#526

@teolemon I'm interested!

👋 @monsieurtanuki
The full minimized DB is located at https://world.openfoodfacts.org/data/offline/en.openfoodfacts.org.products.small.csv.zip
The minimal implementation we've done so far on the classic iPhone app is to unzip it, load it into the database, and show the values stored, and update them as we receive a value from the live API.
We re-download the file monthly and update the local db.
And that's it.
Smoothie is a little more complex with customized rankings and all, but I think we should start with something simple we can iterate on.
If you'd like to have an idea of additional complexities and sophistications, there's a (somewhat confusing) Google Doc:
https://docs.google.com/document/d/1URdZL2pxIP-lCkM9ZzUVMnoSr2o-d3741TyWimzrJI8/edit#
There's also the Swift code for the iOS implem if you'd like to have a look: https://github.com/openfoodfacts/openfoodfacts-ios/commit/20723dab571fd0c155197093190d293886f876bd

also, wanna join the channel on slack ?
https://slack.openfoodfacts.org

(for the moment I have no access to the google docs file)
The csv file is about 80Mb big, has 1,555,491 lines plus a header, and the header is

code
product_name
quantity
brands
nutrition_grade_fr
nova_group
ecoscore_grade

Some stats (with a Unix command like cat en.openfoodfacts.org.products.small.csv | awk -F\t '{print $7}' | sort | uniq -c)...

ecoscore_grade is never populated
nutrition_grade_fr is never populated
nova_group:

977,769 lines with no value
61,660 lines with value 1.0
11,029 lines with value 2.0
119,693 lines with value 3.0
385,340 lines with value 4.0

That means that there are only 577,722 lines with an attached value.
In addition to that, I saw tons of products in non-latin alphabets.
For the moment I don't see the use case...

aha…
ecoscore: normal: not live yet.
nutrition_grade_fr >> that's a bug, congratulations for finding it 🎉  :-)

Non latin alphabets: that's because that's the whole db for all countries

I've made a fix server side @monsieurtanuki
https://github.com/openfoodfacts/openfoodfacts-server/pull/4649

Non latin alphabets: that's because that's the whole db for all countries

That's my point about not understanding the use-case: what's the point of pre-downloading the data of all countries? (disclaimer: I have a smartphone with limited memory and disk space)
What I had in mind is: I'm in a supermarket and I want to check the different scores of - say - breakfast cereals, but I have a bad internet connection. Why would I think about downloading in advance data about foods sold in Russia or Japan? There's no "just in case" argumentation.
And my personal eco behavior does not find it relevant either. Beyond the fact that most of the time (as mentioned in an early comment) there's no actual data extracted.
What may make sense is to select food categories and countries, and to focus on that. Maybe automatically: I scan a Kellog's in France, therefore there's a good chance I'm interested both in breakfast cereals and products sold in France, and that's the range of foods we could cache and refresh periodically.

It was not a just in case choice, but a speed one:
The absence of division by country (or by product popularity) was to drastically simplify initial implementation (by removing the need for anything complex server side and client side).
I've shared the design document, where there's a discussion on how to implement properly all of that.
Unzipped, the file is more or less 100mb, which sounded like a good compromise until we could add sophistication to save space (generally we care about phone space, the Android app is just 5mb for instance)

@monsieurtanuki @stephanegigandet says we should start using this country-specific route as you suggested:
https://fr.openfoodfacts.org/api/v2/search?fields=code,product_name&page_size=1000
And he will ensure the transition to 10K products will be transparent, once there's something working

About https://fr.openfoodfacts.org/api/v2/search?fields=code,product_name&page_size=1000:

1000 products (page size is 1000, page number is 1)
the file size is 67850 bytes
it's basically a list of {barcode, name}, e.g. {"code":"5010477348357","product_name":"Country Crisp 4 noix"}
in total there are 871104 products

the total file / download/ database size would then be around 60Mb
we need to be very lucky when downloading the data page by page given the download size and the number of iterations - probable side-effects for products once in page X+1 and later in page X
we would probably need to use SQFlite again, because even in "lazy" mode hive takes time at init in proportion to the number of records, and would (to be double-checked) pre-load at least the keys (with EAN13 that means around 11Mb)

I still don't understand the use-case: downloading 60Mb of world data for a limited added value (barcode => name). And we need to refresh it altogether time after time.
@teolemon As I don't understand the purpose I think you should find someone else for this issue (e.g. someone who understands it) - that's why I removed my assignment to this issue about a year ago.
But I can still answer questions, write comments and make suggestions (for instance about alternate solutions).

Speed up or enable (in worst case scenario) scan in supermarkets.
@jasmeet0817 told me that he started using the app in real life, after the update for panel expansion.
The highlights of his experience were scanning overheating after some time, and difficulty scanning due to network issues.

About https://fr.openfoodfacts.org/api/v2/search?fields=code,product_name&page_size=1000:

1000 products (page size is 1000, page number is 1)
the file size is 67850 bytes
it's basically a list of {barcode, name}, e.g. {"code":"5010477348357","product_name":"Country Crisp 4 noix"}
in total there are 871104 products

the total file / download/ database size would then be around 60Mb
we need to be very lucky when downloading the data page by page given the download size and the number of iterations - probable side-effects for products once in page X+1 and later in page X
we would probably need to use SQFlite again, because even in "lazy" mode hive takes time at init in proportion to the number of records, and would (to be double-checked) pre-load at least the keys (with EAN13 that means around 11Mb)

I still don't understand the use-case: downloading 60Mb of world data for a limited added value (barcode => name). And we need to refresh it altogether time after time.

@teolemon what's the end goal? To pre-download just (barcodes => name) or the whole product. If it's just names, I agree with @monsieurtanuki. If it's the whole product it's going to take a lot of memory, and the database is always increasing. We would also need a syncing mechanism.
Yes I did have issues where I scanned products and then I was just waiting for it to load, but my phone was also very heated up so I'm not really sure if the root cause was the heating up or network issues. Even if it was network issues, I would try to do a thorough analysis before working on this.
Since this is a non trivial task I would first validate if we really need this, I would add some metrics on how often product fetch calls timeout and if it's higher than a permissible threshold I would go for this feature. If needed, I would also suggest compressing the response payload as much as possible.

@teolemon As I don't understand the purpose I think you should find someone else for this issue (e.g. someone who understands it) - that's why I removed my assignment to this issue about a year ago.
But I can still answer questions, write comments and make suggestions (for instance about alternate solutions).

@teolemon There are separate problems here:

why did @jasmeet0817's smartphone heat that much? Was it the network, was it the camera, was it that there were many pending internet connections, was it something else? Please run tests (in profiler mode?) to see what is causing the heating.
the quantity of data for each product is too large (10Kb). For a low connectivity use case, I guess we could download the product in 2 steps - a first very quick step where we download a minimum set of fields (e.g. name,  brand), or a compact mode, enough to be displayed, and then a (cancelable) full-size product download

cf. https://world.openfoodfacts.org/api/v0/product/093270067481501.json and its 19558 bytes ("all" fields here)

a good product for you, in json
{
	"code": "093270067481501",
	"product": {
		"_id": "093270067481501",
		"_keywords": ["fact", "open", "good", "beverage", "fruit", "you", "plant-based", "based", "eu", "organic", "meal", "trade", "soup", "product", "for", "and", "food", "vegetable", "fair", "france"],
		"added_countries_tags": [],
		"additives_n": 0,
		"additives_old_n": 0,
		"additives_old_tags": [],
		"additives_original_tags": [],
		"additives_tags": [],
		"allergens": "",
		"allergens_from_ingredients": "",
		"allergens_from_user": "(en) ",
		"allergens_hierarchy": [],
		"allergens_lc": "en",
		"allergens_tags": [],
		"amino_acids_tags": [],
		"brands": "Open Food Facts",
		"brands_imported": "Open Food Facts",
		"brands_tags": ["open-food-facts"],
		"carbon_footprint_from_known_ingredients_debug": "en:butternut-squash 50% x 0.3 = 15 g - en:carrot 10% x 0.3 = 3 g - ",
		"carbon_footprint_percent_of_known_ingredients": 60,
		"categories": "Plant-based foods and beverages,Plant-based foods,Fruits and vegetables based foods,Meals,Soups,Vegetable soups",
		"categories_hierarchy": ["en:plant-based-foods-and-beverages", "en:plant-based-foods", "en:fruits-and-vegetables-based-foods", "en:meals", "en:soups", "en:vegetable-soups"],
		"categories_imported": "Plant-based foods and beverages, Plant-based foods, Fruits and vegetables based foods, Meals, Soups, Vegetable soups",
		"categories_lc": "en",
		"categories_properties": {
			"agribalyse_proxy_food_code:en": "25903",
			"ciqual_food_code:en": "25969"
		},
		"categories_properties_tags": ["all-products", "categories-known", "agribalyse-food-code-unknown", "agribalyse-proxy-food-code-25903", "agribalyse-proxy-food-code-known", "ciqual-food-code-25969", "ciqual-food-code-known", "agribalyse-known", "agribalyse-25903"],
		"categories_tags": ["en:plant-based-foods-and-beverages", "en:plant-based-foods", "en:fruits-and-vegetables-based-foods", "en:meals", "en:soups", "en:vegetable-soups"],
		"checkers_tags": [],
		"cities_tags": ["saint-maur-des-fosses-val-de-marne-france"],
		"code": "093270067481501",
		"codes_tags": ["code-15", "093270067481xxx", "09327006748xxxx", "0932700674xxxxx", "093270067xxxxxx", "09327006xxxxxxx", "0932700xxxxxxxx", "093270xxxxxxxxx", "09327xxxxxxxxxx", "0932xxxxxxxxxxx", "093xxxxxxxxxxxx", "09xxxxxxxxxxxxx", "0xxxxxxxxxxxxxx"],
		"compared_to_category": "en:vegetable-soups",
		"complete": 0,
		"completeness": 0.8,
		"correctors_tags": ["org-openfoodfacts", "stephane"],
		"countries": "France",
		"countries_hierarchy": ["en:france"],
		"countries_imported": "France",
		"countries_lc": "en",
		"countries_tags": ["en:france"],
		"created_t":<PHONE_NUMBER>,
		"creator": "org-openfoodfacts",
		"data_quality_bugs_tags": [],
		"data_quality_errors_tags": [],
		"data_quality_info_tags": ["en:packaging-data-incomplete", "en:ingredients-percent-analysis-ok", "en:carbon-footprint-from-known-ingredients-but-not-from-meat-or-fish"],
		"data_quality_tags": ["en:packaging-data-incomplete", "en:ingredients-percent-analysis-ok", "en:carbon-footprint-from-known-ingredients-but-not-from-meat-or-fish"],
		"data_quality_warnings_tags": [],
		"data_sources": "Producers, Producer - openfoodfacts",
		"data_sources_imported": "Producers, Producer - openfoodfacts, Producers, Producer - openfoodfacts",
		"data_sources_tags": ["producers", "producer-openfoodfacts"],
		"debug_param_sorted_langs": ["en", "fr"],
		"ecoscore_data": {
			"adjustments": {
				"origins_of_ingredients": {
					"aggregated_origins": [{
						"origin": "en:france",
						"percent": 100
					}],
					"epi_score": 93,
					"epi_value": 4,
					"origins_from_origins_field": ["en:france"],
					"transportation_scores": {
						"ad": 57,
						"al": 0,
						"at": 38,
						"ax": 67,
						"ba": 14,
						"be": 85,
						"bg": 21,
						"ch": 69,
						"cy": 40,
						"cz": 48,
						"de": 61,
						"dk": 39,
						"dz": 45,
						"ee": 71,
						"eg": 35,
						"es": 37,
						"fi": 69,
						"fo": 62,
						"fr": 100,
						"gg": 78,
						"gi": 4,
						"gr": 49,
						"hr": 30,
						"hu": 26,
						"ie": 47,
						"il": 34,
						"im": 50,
						"is": 53,
						"it": 47,
						"je": 76,
						"lb": 39,
						"li": 64,
						"lt": 63,
						"lu": 82,
						"lv": 71,
						"ly": 56,
						"ma": 60,
						"mc": 52,
						"md": 29,
						"me": 37,
						"mk": 29,
						"mt": 57,
						"nl": 77,
						"no": 20,
						"pl": 25,
						"ps": 42,
						"pt": 13,
						"ro": 31,
						"rs": 7,
						"se": 15,
						"si": 38,
						"sj": 53,
						"sk": 24,
						"sm": 40,
						"sy": 26,
						"tn": 9,
						"tr": 7,
						"ua": 40,
						"uk": 68,
						"us": 0,
						"va": 29,
						"world": 0,
						"xk": 28
					},
					"transportation_values": {
						"ad": 9,
						"al": 0,
						"at": 6,
						"ax": 10,
						"ba": 2,
						"be": 13,
						"bg": 3,
						"ch": 10,
						"cy": 6,
						"cz": 7,
						"de": 9,
						"dk": 6,
						"dz": 7,
						"ee": 11,
						"eg": 5,
						"es": 6,
						"fi": 10,
						"fo": 9,
						"fr": 15,
						"gg": 12,
						"gi": 1,
						"gr": 7,
						"hr": 5,
						"hu": 4,
						"ie": 7,
						"il": 5,
						"im": 8,
						"is": 8,
						"it": 7,
						"je": 11,
						"lb": 6,
						"li": 10,
						"lt": 9,
						"lu": 12,
						"lv": 11,
						"ly": 8,
						"ma": 9,
						"mc": 8,
						"md": 4,
						"me": 6,
						"mk": 4,
						"mt": 9,
						"nl": 12,
						"no": 3,
						"pl": 4,
						"ps": 6,
						"pt": 2,
						"ro": 5,
						"rs": 1,
						"se": 2,
						"si": 6,
						"sj": 8,
						"sk": 4,
						"sm": 6,
						"sy": 4,
						"tn": 1,
						"tr": 1,
						"ua": 6,
						"uk": 10,
						"us": 0,
						"va": 4,
						"world": 0,
						"xk": 4
					},
					"values": {
						"ad": 13,
						"al": 4,
						"at": 10,
						"ax": 14,
						"ba": 6,
						"be": 17,
						"bg": 7,
						"ch": 14,
						"cy": 10,
						"cz": 11,
						"de": 13,
						"dk": 10,
						"dz": 11,
						"ee": 15,
						"eg": 9,
						"es": 10,
						"fi": 14,
						"fo": 13,
						"fr": 19,
						"gg": 16,
						"gi": 5,
						"gr": 11,
						"hr": 9,
						"hu": 8,
						"ie": 11,
						"il": 9,
						"im": 12,
						"is": 12,
						"it": 11,
						"je": 15,
						"lb": 10,
						"li": 14,
						"lt": 13,
						"lu": 16,
						"lv": 15,
						"ly": 12,
						"ma": 13,
						"mc": 12,
						"md": 8,
						"me": 10,
						"mk": 8,
						"mt": 13,
						"nl": 16,
						"no": 7,
						"pl": 8,
						"ps": 10,
						"pt": 6,
						"ro": 9,
						"rs": 5,
						"se": 6,
						"si": 10,
						"sj": 12,
						"sk": 8,
						"sm": 10,
						"sy": 8,
						"tn": 5,
						"tr": 5,
						"ua": 10,
						"uk": 14,
						"us": 4,
						"va": 8,
						"world": 4,
						"xk": 8
					}
				},
				"packaging": {
					"non_recyclable_and_non_biodegradable_materials": 0,
					"packagings": [{
						"ecoscore_material_score": "81",
						"ecoscore_shape_ratio": "1",
						"material": "en:glass",
						"recycling": "en:recycle",
						"shape": "en:bottle"
					}, {
						"ecoscore_material_score": "76",
						"ecoscore_shape_ratio": "0.1",
						"material": "en:steel",
						"recycling": "en:recycle",
						"shape": "en:bottle-cap"
					}],
					"score": 78.6,
					"value": -2
				},
				"production_system": {
					"labels": ["en:eu-organic"],
					"value": 15
				},
				"threatened_species": {}
			},
			"agribalyse": {
				"agribalyse_proxy_food_code": "25903",
				"co2_agriculture": "0.099163319",
				"co2_consumption": "0.0079267752",
				"co2_distribution": "0.025340163",
				"co2_packaging": "0.098786634",
				"co2_processing": "0.13521664",
				"co2_total": "0.49805172",
				"co2_transportation": "0.1316182",
				"code": "25903",
				"dqr": "2.42",
				"ef_agriculture": "0.023834065",
				"ef_consumption": "0.004012423",
				"ef_distribution": "0.0095369029",
				"ef_packaging": "0.014974093",
				"ef_processing": "0.030870892",
				"ef_total": "0.093465939",
				"ef_transportation": "0.010237564",
				"is_beverage": 0,
				"name_en": "Soup, mixed vegetables, prepacked, to be reheated",
				"name_fr": "Soupe aux légumes variés, préemballée à réchauffer",
				"score": 97
			},
			"grade": "a",
			"grades": {
				"ad": "a",
				"al": "a",
				"at": "a",
				"ax": "a",
				"ba": "a",
				"be": "a",
				"bg": "a",
				"ch": "a",
				"cy": "a",
				"cz": "a",
				"de": "a",
				"dk": "a",
				"dz": "a",
				"ee": "a",
				"eg": "a",
				"es": "a",
				"fi": "a",
				"fo": "a",
				"fr": "a",
				"gg": "a",
				"gi": "a",
				"gr": "a",
				"hr": "a",
				"hu": "a",
				"ie": "a",
				"il": "a",
				"im": "a",
				"is": "a",
				"it": "a",
				"je": "a",
				"lb": "a",
				"li": "a",
				"lt": "a",
				"lu": "a",
				"lv": "a",
				"ly": "a",
				"ma": "a",
				"mc": "a",
				"md": "a",
				"me": "a",
				"mk": "a",
				"mt": "a",
				"nl": "a",
				"no": "a",
				"pl": "a",
				"ps": "a",
				"pt": "a",
				"ro": "a",
				"rs": "a",
				"se": "a",
				"si": "a",
				"sj": "a",
				"sk": "a",
				"sm": "a",
				"sy": "a",
				"tn": "a",
				"tr": "a",
				"ua": "a",
				"uk": "a",
				"us": "a",
				"va": "a",
				"world": "a",
				"xk": "a"
			},
			"score": 122,
			"scores": {
				"ad": 122,
				"al": 114,
				"at": 120,
				"ax": 122,
				"ba": 116,
				"be": 122,
				"bg": 117,
				"ch": 122,
				"cy": 120,
				"cz": 121,
				"de": 122,
				"dk": 120,
				"dz": 121,
				"ee": 122,
				"eg": 119,
				"es": 120,
				"fi": 122,
				"fo": 122,
				"fr": 122,
				"gg": 122,
				"gi": 115,
				"gr": 121,
				"hr": 119,
				"hu": 118,
				"ie": 121,
				"il": 119,
				"im": 122,
				"is": 122,
				"it": 121,
				"je": 122,
				"lb": 120,
				"li": 122,
				"lt": 122,
				"lu": 122,
				"lv": 122,
				"ly": 122,
				"ma": 122,
				"mc": 122,
				"md": 118,
				"me": 120,
				"mk": 118,
				"mt": 122,
				"nl": 122,
				"no": 117,
				"pl": 118,
				"ps": 120,
				"pt": 116,
				"ro": 119,
				"rs": 115,
				"se": 116,
				"si": 120,
				"sj": 122,
				"sk": 118,
				"sm": 120,
				"sy": 118,
				"tn": 115,
				"tr": 115,
				"ua": 120,
				"uk": 122,
				"us": 114,
				"va": 118,
				"world": 114,
				"xk": 118
			},
			"status": "known"
		},
		"ecoscore_extended_data": {
			"impact": {
				"ef_single_score_log_stddev": 0.0800212307271874,
				"likeliest_impacts": {
					"Climate_change": 0.012534284630903,
					"EF_single_score": 0.00409189070065541
				},
				"likeliest_recipe": {
					"en:butternut_squash": 50.0548448998379,
					"en:carrot": 10.0109689799676,
					"en:leek": 2.20570054418209,
					"en:potato": 1.14589787403023,
					"en:salt": 0.392984356259901,
					"en:sweet_potato": 3.56236029441026,
					"en:water": 32.7369328509879
				},
				"mass_ratio_uncharacterized": 0,
				"uncharacterized_ingredients": {
					"impact": [],
					"nutrition": []
				},
				"uncharacterized_ingredients_mass_proportion": {
					"impact": 0,
					"nutrition": 0
				},
				"uncharacterized_ingredients_ratio": {
					"impact": 0,
					"nutrition": 0
				},
				"warnings": []
			}
		},
		"ecoscore_extended_data_version": "4",
		"ecoscore_grade": "a",
		"ecoscore_score": 122,
		"ecoscore_tags": ["a"],
		"editors_tags": ["stephane", "ecoscore-impact-estimator", "org-openfoodfacts"],
		"emb_codes": "FR 94.068.042 EC",
		"emb_codes_imported": "FR 94.068.042 CE",
		"emb_codes_orig": "EMB 53062, FR 94.068.042 EC",
		"emb_codes_tags": ["fr-94-068-042-ec"],
		"entry_dates_tags": ["2022-01-07", "2022-01", "2022"],
		"environment_impact_level": "",
		"environment_impact_level_tags": [],
		"expiration_date": "",
		"food_groups": "en:soups",
		"food_groups_tags": ["en:fruits-and-vegetables", "en:soups"],
		"generic_name": "",
		"generic_name_en": "",
		"generic_name_fr": "",
		"grades": {},
		"id": "093270067481501",
		"informers_tags": ["org-openfoodfacts"],
		"ingredients": [{
			"id": "en:butternut-squash",
			"percent": "50",
			"percent_estimate": "50",
			"percent_max": "50",
			"percent_min": "50",
			"rank": 1,
			"text": "Butternut squash",
			"vegan": "yes",
			"vegetarian": "yes"
		}, {
			"id": "en:water",
			"percent_estimate": 27.5,
			"percent_max": 45,
			"percent_min": "10",
			"rank": 2,
			"text": "water",
			"vegan": "yes",
			"vegetarian": "yes"
		}, {
			"id": "en:carrot",
			"percent": "10",
			"percent_estimate": "10",
			"percent_max": "10",
			"percent_min": "10",
			"rank": 3,
			"text": "carrots",
			"vegan": "yes",
			"vegetarian": "yes"
		}, {
			"id": "en:sweet-potato",
			"percent_estimate": 5,
			"percent_max": "10",
			"percent_min": 0,
			"rank": 4,
			"text": "sweet potato",
			"vegan": "yes",
			"vegetarian": "yes"
		}, {
			"id": "en:leek",
			"percent_estimate": 3.75,
			"percent_max": "10",
			"percent_min": 0,
			"rank": 5,
			"text": "leeks",
			"vegan": "yes",
			"vegetarian": "yes"
		}, {
			"id": "en:potato",
			"percent_estimate": 1.875,
			"percent_max": "10",
			"percent_min": 0,
			"rank": 6,
			"text": "potatoes",
			"vegan": "yes",
			"vegetarian": "yes"
		}, {
			"id": "en:salt",
			"percent_estimate": 1.875,
			"percent_max": 8,
			"percent_min": 0,
			"rank": 7,
			"text": "salt",
			"vegan": "yes",
			"vegetarian": "yes"
		}],
		"ingredients_analysis_tags": ["en:palm-oil-free", "en:vegan", "en:vegetarian"],
		"ingredients_from_or_that_may_be_from_palm_oil_n": 0,
		"ingredients_from_palm_oil_n": 0,
		"ingredients_from_palm_oil_tags": [],
		"ingredients_hierarchy": ["en:butternut-squash", "en:vegetable", "en:water", "en:carrot", "en:root-vegetable", "en:sweet-potato", "en:leek", "en:potato", "en:salt"],
		"ingredients_n": 7,
		"ingredients_n_tags": ["7", "1-10"],
		"ingredients_original_tags": ["en:butternut-squash", "en:water", "en:carrot", "en:sweet-potato", "en:leek", "en:potato", "en:salt"],
		"ingredients_percent_analysis": 1,
		"ingredients_tags": ["en:butternut-squash", "en:vegetable", "en:water", "en:carrot", "en:root-vegetable", "en:sweet-potato", "en:leek", "en:potato", "en:salt"],
		"ingredients_text": "Butternut squash 50%, water, carrots 10%, sweet potato, leeks, potatoes, salt.",
		"ingredients_text_en": "Butternut squash 50%, water, carrots 10%, sweet potato, leeks, potatoes, salt.",
		"ingredients_text_en_imported": "Butternut squash 50%, water, carrots 10%, sweet potato, leeks, potatoes, salt.",
		"ingredients_text_fr": "",
		"ingredients_text_with_allergens": "Butternut squash 50%, water, carrots 10%, sweet potato, leeks, potatoes, salt.",
		"ingredients_text_with_allergens_en": "Butternut squash 50%, water, carrots 10%, sweet potato, leeks, potatoes, salt.",
		"ingredients_text_with_allergens_fr": "",
		"ingredients_that_may_be_from_palm_oil_n": 0,
		"ingredients_that_may_be_from_palm_oil_tags": [],
		"ingredients_with_specified_percent_n": 2,
		"ingredients_with_specified_percent_sum": 60,
		"ingredients_with_unspecified_percent_n": 5,
		"ingredients_with_unspecified_percent_sum": 40,
		"interface_version_created": "import_csv_file - version 2019/09/17",
		"interface_version_modified": "20150316.jqm2",
		"known_ingredients_n": 9,
		"labels": "Organic,EU Organic,Fair trade",
		"labels_hierarchy": ["en:organic", "en:eu-organic", "en:fair-trade"],
		"labels_imported": "Organic, EU Organic, Fair trade",
		"labels_lc": "en",
		"labels_tags": ["en:organic", "en:eu-organic", "en:fair-trade"],
		"lang": "en",
		"lang_imported": "en",
		"languages": {
			"en:english": 4,
			"en:french": 1
		},
		"languages_codes": {
			"en": 4,
			"fr": 1
		},
		"languages_hierarchy": ["en:english", "en:french"],
		"languages_tags": ["en:english", "en:french", "en:2", "en:multilingual"],
		"last_edit_dates_tags": ["2022-01-07", "2022-01", "2022"],
		"last_editor": "ecoscore-impact-estimator",
		"last_modified_by": "ecoscore-impact-estimator",
		"last_modified_t":<PHONE_NUMBER>,
		"lc": "en",
		"lc_imported": "en",
		"link": "",
		"main_countries_tags": [],
		"manufacturing_places": "",
		"manufacturing_places_tags": [],
		"minerals_tags": [],
		"misc_tags": ["en:nutrition-fruits-vegetables-nuts-estimate-from-ingredients", "en:nutrition-all-nutriscore-values-known", "en:nutriscore-computed", "en:ecoscore-extended-data-computed", "en:ecoscore-extended-data-version-4", "en:ecoscore-no-missing-data", "en:ecoscore-computed", "en:main-countries-new-product"],
		"no_nutrition_data": "",
		"nova_group": 3,
		"nova_group_debug": " -- categories/en:meals : 3",
		"nova_groups": "3",
		"nova_groups_tags": ["en:3-processed-foods"],
		"nucleotides_tags": [],
		"nutrient_levels": {
			"fat": "low",
			"salt": "moderate",
			"saturated-fat": "low",
			"sugars": "low"
		},
		"nutrient_levels_tags": ["en:fat-in-low-quantity", "en:saturated-fat-in-low-quantity", "en:sugars-in-low-quantity", "en:salt-in-moderate-quantity"],
		"nutriments": {
			"carbohydrates": 4,
			"carbohydrates_100g": 4,
			"carbohydrates_unit": "g",
			"carbohydrates_value": 4,
			"carbon-footprint-from-known-ingredients_100g": 18,
			"carbon-footprint-from-known-ingredients_product": 36,
			"energy": 120,
			"energy-kj": 120,
			"energy-kj_100g": 120,
			"energy-kj_unit": "kJ",
			"energy-kj_value": 120,
			"energy_100g": 120,
			"energy_unit": "kJ",
			"energy_value": 120,
			"fat": 0.5,
			"fat_100g": 0.5,
			"fat_unit": "g",
			"fat_value": 0.5,
			"fiber": 3,
			"fiber_100g": 3,
			"fiber_unit": "g",
			"fiber_value": 3,
			"fruits-vegetables-nuts-estimate-from-ingredients_100g": 60,
			"fruits-vegetables-nuts-estimate-from-ingredients_serving": 60,
			"nova-group": 3,
			"nova-group_100g": 3,
			"nova-group_serving": 3,
			"nutrition-score-fr": -3,
			"nutrition-score-fr_100g": -3,
			"proteins": 1,
			"proteins_100g": 1,
			"proteins_unit": "g",
			"proteins_value": 1,
			"salt": 0.4,
			"salt_100g": 0.4,
			"salt_unit": "g",
			"salt_value": 0.4,
			"saturated-fat": 0.2,
			"saturated-fat_100g": 0.2,
			"saturated-fat_unit": "g",
			"saturated-fat_value": 0.2,
			"sodium": 0.16,
			"sodium_100g": 0.16,
			"sodium_unit": "g",
			"sodium_value": 0.16,
			"sugars": 0.5,
			"sugars_100g": 0.5,
			"sugars_unit": "g",
			"sugars_value": 0.5
		},
		"nutriscore_data": {
			"energy": 120,
			"energy_points": 0,
			"energy_value": 120,
			"fiber": 3,
			"fiber_points": 3,
			"fiber_value": 3,
			"fruits_vegetables_nuts_colza_walnut_olive_oils": 60,
			"fruits_vegetables_nuts_colza_walnut_olive_oils_points": 1,
			"fruits_vegetables_nuts_colza_walnut_olive_oils_value": 60,
			"grade": "a",
			"is_beverage": 0,
			"is_cheese": 0,
			"is_fat": 0,
			"is_water": 0,
			"negative_points": 1,
			"positive_points": 4,
			"proteins": 1,
			"proteins_points": 0,
			"proteins_value": 1,
			"saturated_fat": 0.2,
			"saturated_fat_points": 0,
			"saturated_fat_ratio": 40,
			"saturated_fat_ratio_points": 6,
			"saturated_fat_ratio_value": 40,
			"saturated_fat_value": 0.2,
			"score": -3,
			"sodium": 160,
			"sodium_points": 1,
			"sodium_value": 160,
			"sugars": 0.5,
			"sugars_points": 0,
			"sugars_value": 0.5
		},
		"nutriscore_grade": "a",
		"nutriscore_score": -3,
		"nutriscore_score_opposite": 3,
		"nutrition_data": "on",
		"nutrition_data_per": "100g",
		"nutrition_data_per_imported": "100g",
		"nutrition_data_prepared": "",
		"nutrition_data_prepared_per": "100g",
		"nutrition_data_prepared_per_imported": "100g",
		"nutrition_grade_fr": "a",
		"nutrition_grades": "a",
		"nutrition_grades_tags": ["a"],
		"nutrition_score_beverage": 0,
		"nutrition_score_warning_fruits_vegetables_nuts_estimate_from_ingredients": 1,
		"nutrition_score_warning_fruits_vegetables_nuts_estimate_from_ingredients_value": 60,
		"obsolete": "",
		"obsolete_imported": "0",
		"obsolete_since_date": "",
		"origin": "France",
		"origin_en": "France",
		"origin_en_imported": "France",
		"origin_fr": "",
		"origins": "France",
		"origins_hierarchy": ["en:france"],
		"origins_imported": "France",
		"origins_lc": "en",
		"origins_tags": ["en:france"],
		"other_nutritional_substances_tags": [],
		"owner": "org-openfoodfacts",
		"owner_fields": {
			"brands":<PHONE_NUMBER>,
			"carbohydrates":<PHONE_NUMBER>,
			"categories":<PHONE_NUMBER>,
			"countries":<PHONE_NUMBER>,
			"data_sources":<PHONE_NUMBER>,
			"emb_codes":<PHONE_NUMBER>,
			"energy-kj":<PHONE_NUMBER>,
			"fat":<PHONE_NUMBER>,
			"fiber":<PHONE_NUMBER>,
			"ingredients_text_en":<PHONE_NUMBER>,
			"labels":<PHONE_NUMBER>,
			"lang":<PHONE_NUMBER>,
			"lc":<PHONE_NUMBER>,
			"nutrition_data_per":<PHONE_NUMBER>,
			"nutrition_data_prepared_per":<PHONE_NUMBER>,
			"obsolete":<PHONE_NUMBER>,
			"origin_en":<PHONE_NUMBER>,
			"origins":<PHONE_NUMBER>,
			"packaging_text_en":<PHONE_NUMBER>,
			"product_name_en":<PHONE_NUMBER>,
			"product_name_fr":<PHONE_NUMBER>,
			"proteins":<PHONE_NUMBER>,
			"quantity":<PHONE_NUMBER>,
			"salt":<PHONE_NUMBER>,
			"saturated-fat":<PHONE_NUMBER>,
			"sugars":<PHONE_NUMBER>
		},
		"owners_tags": "org-openfoodfacts",
		"packaging": "",
		"packaging_tags": [],
		"packaging_text": "Glass bottle to recycle\r\nSteel cap to recycle",
		"packaging_text_en": "Glass bottle to recycle\r\nSteel cap to recycle",
		"packaging_text_en_imported": "Glass bottle to recycle\nSteel cap to recycle",
		"packaging_text_fr": "",
		"packagings": [{
			"material": "en:glass",
			"recycling": "en:recycle",
			"shape": "en:bottle"
		}, {
			"material": "en:steel",
			"recycling": "en:recycle",
			"shape": "en:bottle-cap"
		}],
		"photographers_tags": [],
		"pnns_groups_1": "Fruits and vegetables",
		"pnns_groups_1_tags": ["fruits-and-vegetables", "known"],
		"pnns_groups_2": "Soups",
		"pnns_groups_2_tags": ["soups", "known"],
		"popularity_key": 0,
		"product_name": "A good product for you",
		"product_name_en": "A good product for you",
		"product_name_en_imported": "A good product for you",
		"product_name_fr": "Un bon produit pour vous",
		"product_name_fr_imported": "Un bon produit pour vous",
		"product_quantity": "200",
		"purchase_places": "",
		"purchase_places_tags": [],
		"quantity": "200 g",
		"quantity_imported": "200 g",
		"removed_countries_tags": [],
		"rev": 5,
		"scores": {},
		"sources": [{
			"fields": ["product_name_en", "product_name_fr", "packaging_text_en", "quantity", "brands", "categories", "labels", "emb_codes", "countries", "origin_en", "data_sources", "nutrition_data_per", "nutrition_data_prepared_per", "ingredients_text_en", "nutrients.carbohydrates_unit", "nutrients.carbohydrates_value", "nutrients.fat_unit", "nutrients.fat_value", "nutrients.fiber_unit", "nutrients.fiber_value", "nutrients.proteins_unit", "nutrients.proteins_value", "nutrients.salt_unit", "nutrients.salt_value", "nutrients.saturated-fat_unit", "nutrients.saturated-fat_value", "nutrients.sugars_unit", "nutrients.sugars_value"],
			"id": "org-openfoodfacts",
			"images": [],
			"import_t":<PHONE_NUMBER>,
			"manufacturer": 1,
			"name": "openfoodfacts",
			"url": null
		}, {
			"fields": ["origins"],
			"id": "org-openfoodfacts",
			"images": [],
			"import_t":<PHONE_NUMBER>,
			"manufacturer": 1,
			"name": "openfoodfacts",
			"url": null
		}, {
			"fields": ["emb_codes"],
			"id": "org-openfoodfacts",
			"images": [],
			"import_t":<PHONE_NUMBER>,
			"manufacturer": 1,
			"name": "openfoodfacts",
			"url": null
		}],
		"states": "en:to-be-completed, en:nutrition-facts-completed, en:ingredients-completed, en:expiration-date-to-be-completed, en:packaging-code-completed, en:characteristics-to-be-completed, en:origins-completed, en:categories-completed, en:brands-completed, en:packaging-to-be-completed, en:quantity-completed, en:product-name-completed, en:photos-to-be-uploaded",
		"states_hierarchy": ["en:to-be-completed", "en:nutrition-facts-completed", "en:ingredients-completed", "en:expiration-date-to-be-completed", "en:packaging-code-completed", "en:characteristics-to-be-completed", "en:origins-completed", "en:categories-completed", "en:brands-completed", "en:packaging-to-be-completed", "en:quantity-completed", "en:product-name-completed", "en:photos-to-be-uploaded"],
		"states_tags": ["en:to-be-completed", "en:nutrition-facts-completed", "en:ingredients-completed", "en:expiration-date-to-be-completed", "en:packaging-code-completed", "en:characteristics-to-be-completed", "en:origins-completed", "en:categories-completed", "en:brands-completed", "en:packaging-to-be-completed", "en:quantity-completed", "en:product-name-completed", "en:photos-to-be-uploaded"],
		"stores": "",
		"stores_tags": [],
		"teams": "pain-au-chocolat,shark-attack",
		"teams_tags": ["pain-au-chocolat", "shark-attack"],
		"traces": "",
		"traces_from_ingredients": "",
		"traces_from_user": "(en) ",
		"traces_hierarchy": [],
		"traces_lc": "en",
		"traces_tags": [],
		"unknown_ingredients_n": 0,
		"unknown_nutrients_tags": [],
		"update_key": "ing2",
		"vitamins_tags": []
	},
	"status": 1,
	"status_verbose": "product found"
}

@jasmeet0817 @monsieurtanuki  I didn't adapt the query.

The knowledge panels are most probably off-limits due to their sheer size
It would probably be the summary card only (basic info + attributes to allow a personalized summary card)
If that's still too big, we could reduce, but just the name & brand is obviously useless

@monsieurtanuki yes, the heating is totally unrelated and being taken care of already (long discussion about scanning during the last community meeting on Thursday evening). I was just giving it for context, as I remembered my discussion about top annoyances. Sorry for the ambiguity introduced.

Yes I did have issues where I scanned products and then I was just waiting for it to load

@teolemon @jasmeet0817 For the low connectivity use-case, I suggest that we add in dev mode a switch between the current set of extracted fields, and a minimum set of fields. And then we send @jasmeet0817 go shopping :) (bad luck, it's crowded on Saturdays)
What do you think of that, at least for test purposes? Faster scan, faster carousel. And when we go to the product page, we download all the fields (to be coded in a second step, if relevant).
Anyway, it's a bit paradoxical to use scores for fast assessment of products ("it's A, it's D") and flood users with tons of the detailed data from which the scores were computed. In a "more about it..." button, fair enough. But when you're in a busy supermarket with low (or expensive) connectivity, the faster the better.
And more or less that's similar to the OP: a downgraded mode, and a full mode. The difference is that "my" downgraded mode does not handle offline queries and doesn't imply pre-loading tons of data.
I can work on that downgraded mode (= limited fields) / full mode.
Following is the current list of product fields we extract for products:

NAME
BRANDS
BARCODE
NUTRISCORE
FRONT_IMAGE
IMAGE_FRONT_SMALL_URL
IMAGE_FRONT_URL
IMAGE_INGREDIENTS_URL
IMAGE_NUTRITION_URL
IMAGE_PACKAGING_URL
SELECTED_IMAGE
QUANTITY
SERVING_SIZE
PACKAGING_QUANTITY
NUTRIMENTS
NUTRIENT_LEVELS
NUTRIMENT_ENERGY_UNIT
ADDITIVES
INGREDIENTS_ANALYSIS_TAGS
LABELS_TAGS
LABELS_TAGS_IN_LANGUAGES
ENVIRONMENT_IMPACT_LEVELS
CATEGORIES_TAGS_IN_LANGUAGES
LANGUAGE
ATTRIBUTE_GROUPS
STATES_TAGS
ECOSCORE_DATA
ECOSCORE_GRADE
ECOSCORE_SCORE
ENVIRONMENT_IMPACT_LEVELS

PS: my understanding of offline scanning hasn't changed, as described in the following video :)
https://www.youtube.com/watch?v=oyll1XxKh-M

For the other attributes, an example: my sister is gluten intolerant.
There are several reasons why offline will probably be helpful:

Speed, reliability and peace of mind (we have an insanely fast and minimum experience regardless of the network speed / absence, potential slowdowns or downtime with the servers)
Metered networks (in some areas of the world, or while roaming, people choose to disable mobile data)
Use cases like tablets

@teolemon The list of fields I provided in the previous comment was the current one - I guess gluten is already there, probably in ATTRIBUTE_GROUPS.
I suggest that you define a list of product fields that you think we definitely need: with that we can estimate the volume of the full offline database for France.
The query for the 1000 first products with our current list of fields is: https://fr.openfoodfacts.org/api/v2/search?page_size=1000&fields=code,product_name,brands,nutrition_grade_fr,image_small_url,image_front_small_url,image_front_url,image_ingredients_url,image_nutrition_url,image_packaging_url,selected_images,quantity,serving_size,product_quantity,nutriments,additives_tags,nutrient_levels,nutriment_energy_unit,ingredients_analysis_tags,labels,labels_tags_,environment_impact_level_tags,categories_tags_,lang,attribute_groups,states_tags,ecoscore_grade,ecoscore_score,ecoscore_data
The resulting size is 16,840,056 bytes, for 1000 products. For 871104 products it means 14.5 Gb, with all the current fields.

https://fr.openfoodfacts.org/api/v2/search?page_size=1000&fields=code,product_name,brands,nutrition_grade_fr,image_front_small_url,quantity,lang,attribute_groups,ecoscore_grade
Probably this to display only the summary card ?

We could remove images alltogether. What would be the cost of having 1K to 10K of those ?
https://images.openfoodfacts.org/images/products/20301415/front_fr.37.100.jpg

There are some fields that we possibly don't need anymore thanks to knowledge panels and attributes

8.5Mb for https://fr.openfoodfacts.org/api/v2/search?page_size=1000&fields=code,product_name,brands,nutrition_grade_fr,image_front_small_url,quantity,lang,attribute_groups,ecoscore_grade
That means 7Gb for the 900K records.
Now that you mention the images, let me point that image data (png, jpg) are not even included, just the urls.

yup, I know, we could save space by removing image urls, or opposedly we could decide to let the user even get images.
But probably not for 900k records.

If we really want offline storage, then the best would be to store the bare
minimum data for only the top X popular products in a country.
On Sat, Jan 29, 2022, 15:37 monsieurtanuki @.***> wrote:

8.5Mb for
https://fr.openfoodfacts.org/api/v2/search?page_size=1000&fields=code,product_name,brands,nutrition_grade_fr,image_front_small_url,quantity,lang,attribute_groups,ecoscore_grade
That means 7Gb for the 900K records.
Now that you mention the images, let me point that image data (png, jpg)
are not even included, just the urls.
—
Reply to this email directly, view it on GitHub
https://github.com/openfoodfacts/smooth-app/issues/18#issuecomment-1024923448,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABZRHYQTK2LLHAXX4EKEE7LUYP3RZANCNFSM4SCUNPYQ
.
You are receiving this because you were mentioned.Message ID:
@.***>

If we really want offline storage, then the best would be to store the bare minimum data for only the top X popular products in a country.

@jasmeet0817 @teolemon Looks like a very good idea! We could even download everything for those top X popular products.
Additional data following...
Without image_front_small_url: 8,442,485 bytes per 1K products - we don't win anything
https://fr.openfoodfacts.org/api/v2/search?page_size=1000&fields=code,product_name,brands,nutrition_grade_fr,quantity,lang,attribute_groups,ecoscore_grade
Without attribute_groups: 281,493 bytes per 1K products - we do win a lot: 30 times smaller!
https://fr.openfoodfacts.org/api/v2/search?page_size=1000&fields=code,product_name,brands,nutrition_grade_fr,quantity,lang,ecoscore_grade,image_front_small_url
The thing is that attribute_groups are too fat and redundant.
For instance, the first attribute of the first product:
{
    "description": "", // we don't need
    "icon_url": "https://static.openfoodfacts.org/images/attributes/nutriscore-a.svg", // we can use a reference
    "name": "Nutri-Score", // we can use a reference
    "title": "Nutri-Score A", // we can use a reference
    "grade": "a", // mandatory
    "id": "nutriscore", // mandatory
    "match": 100, // mandatory
    "status": "known", // we can assume that the status is known if there's a match > 0
    "description_short": "Très bonne qualité nutritionnelle" // we probably can use a reference in most cases
  }

I tried to "simplify" the attributes of the first product, and I compressed from 8635 to 2308 bytes (modulo the \n)

before
[{
	"attributes": [{
		"description": "",
		"description_short": "Très bonne qualité nutritionnelle",
		"grade": "a",
		"icon_url": "https://static.openfoodfacts.org/images/attributes/nutriscore-a.svg",
		"id": "nutriscore",
		"match": 100,
		"name": "Nutri-Score",
		"status": "known",
		"title": "Nutri-Score A"
	}, {
		"description_short": "0.021 g / 100 g",
		"grade": "a",
		"icon_url": "https://static.openfoodfacts.org/images/attributes/nutrient-level-salt-low.svg",
		"id": "low_salt",
		"match": 97.2,
		"name": "Sel",
		"status": "known",
		"title": "Sel en faible quantité"
	}, {
		"description_short": "0 g / 100 g",
		"grade": "a",
		"icon_url": "https://static.openfoodfacts.org/images/attributes/nutrient-level-fat-low.svg",
		"id": "low_fat",
		"match": 100,
		"name": "Matières grasses",
		"status": "known",
		"title": "Matières grasses en faible quantité"
	}, {
		"description_short": "0 g / 100 g",
		"grade": "a",
		"icon_url": "https://static.openfoodfacts.org/images/attributes/nutrient-level-sugars-low.svg",
		"id": "low_sugars",
		"match": 100,
		"name": "Sucres",
		"status": "known",
		"title": "Sucres en faible quantité"
	}, {
		"description_short": "0 g / 100 g",
		"grade": "a",
		"icon_url": "https://static.openfoodfacts.org/images/attributes/nutrient-level-saturated-fat-low.svg",
		"id": "low_saturated_fat",
		"match": 100,
		"name": "Acides gras saturés",
		"status": "known",
		"title": "Acides gras saturés en faible quantité"
	}],
	"id": "nutritional_quality",
	"name": "Qualité nutritionnelle"
}, {
	"attributes": [{
		"debug": "1 ingredients (0 unknown)",
		"grade": "a",
		"icon_url": "https://static.openfoodfacts.org/images/attributes/no-gluten.svg",
		"id": "allergens_no_gluten",
		"match": 100,
		"name": "Gluten",
		"status": "known",
		"title": "Ne contient pas : Gluten"
	}, {
		"debug": "1 ingredients (0 unknown)",
		"grade": "a",
		"icon_url": "https://static.openfoodfacts.org/images/attributes/no-milk.svg",
		"id": "allergens_no_milk",
		"match": 100,
		"name": "Lait",
		"status": "known",
		"title": "Ne contient pas : Lait"
	}, {
		"debug": "1 ingredients (0 unknown)",
		"grade": "a",
		"icon_url": "https://static.openfoodfacts.org/images/attributes/no-eggs.svg",
		"id": "allergens_no_eggs",
		"match": 100,
		"name": "Œufs",
		"status": "known",
		"title": "Ne contient pas : Œufs"
	}, {
		"debug": "1 ingredients (0 unknown)",
		"grade": "a",
		"icon_url": "https://static.openfoodfacts.org/images/attributes/no-nuts.svg",
		"id": "allergens_no_nuts",
		"match": 100,
		"name": "Fruits à coque",
		"status": "known",
		"title": "Ne contient pas : Fruits à coque"
	}, {
		"debug": "1 ingredients (0 unknown)",
		"grade": "a",
		"icon_url": "https://static.openfoodfacts.org/images/attributes/no-peanuts.svg",
		"id": "allergens_no_peanuts",
		"match": 100,
		"name": "Arachides",
		"status": "known",
		"title": "Ne contient pas : Arachides"
	}, {
		"debug": "1 ingredients (0 unknown)",
		"grade": "a",
		"icon_url": "https://static.openfoodfacts.org/images/attributes/no-sesame-seeds.svg",
		"id": "allergens_no_sesame_seeds",
		"match": 100,
		"name": "Graines de sésame",
		"status": "known",
		"title": "Ne contient pas : Graines de sésame"
	}, {
		"debug": "1 ingredients (0 unknown)",
		"grade": "a",
		"icon_url": "https://static.openfoodfacts.org/images/attributes/no-soybeans.svg",
		"id": "allergens_no_soybeans",
		"match": 100,
		"name": "Soja",
		"status": "known",
		"title": "Ne contient pas : Soja"
	}, {
		"debug": "1 ingredients (0 unknown)",
		"grade": "a",
		"icon_url": "https://static.openfoodfacts.org/images/attributes/no-celery.svg",
		"id": "allergens_no_celery",
		"match": 100,
		"name": "Céleri",
		"status": "known",
		"title": "Ne contient pas : Céleri"
	}, {
		"debug": "1 ingredients (0 unknown)",
		"grade": "a",
		"icon_url": "https://static.openfoodfacts.org/images/attributes/no-mustard.svg",
		"id": "allergens_no_mustard",
		"match": 100,
		"name": "Moutarde",
		"status": "known",
		"title": "Ne contient pas : Moutarde"
	}, {
		"debug": "1 ingredients (0 unknown)",
		"grade": "a",
		"icon_url": "https://static.openfoodfacts.org/images/attributes/no-lupin.svg",
		"id": "allergens_no_lupin",
		"match": 100,
		"name": "Lupin",
		"status": "known",
		"title": "Ne contient pas : Lupin"
	}, {
		"debug": "1 ingredients (0 unknown)",
		"grade": "a",
		"icon_url": "https://static.openfoodfacts.org/images/attributes/no-fish.svg",
		"id": "allergens_no_fish",
		"match": 100,
		"name": "Poisson",
		"status": "known",
		"title": "Ne contient pas : Poisson"
	}, {
		"debug": "1 ingredients (0 unknown)",
		"grade": "a",
		"icon_url": "https://static.openfoodfacts.org/images/attributes/no-crustaceans.svg",
		"id": "allergens_no_crustaceans",
		"match": 100,
		"name": "Crustacés",
		"status": "known",
		"title": "Ne contient pas : Crustacés"
	}, {
		"debug": "1 ingredients (0 unknown)",
		"grade": "a",
		"icon_url": "https://static.openfoodfacts.org/images/attributes/no-molluscs.svg",
		"id": "allergens_no_molluscs",
		"match": 100,
		"name": "Mollusques",
		"status": "known",
		"title": "Ne contient pas : Mollusques"
	}, {
		"debug": "1 ingredients (0 unknown)",
		"grade": "a",
		"icon_url": "https://static.openfoodfacts.org/images/attributes/no-sulphur-dioxide-and-sulphites.svg",
		"id": "allergens_no_sulphur_dioxide_and_sulphites",
		"match": 100,
		"name": "Anhydride sulfureux et sulfites",
		"status": "known",
		"title": "Ne contient pas : Anhydride sulfureux et sulfites"
	}],
	"id": "allergens",
	"name": "Allergènes",
	"warning": "Il est toujours possible que les données sur les allergènes soient manquantes, incomplètes, incorrectes ou que la composition du produit ait changé. Si vous êtes allergique, vérifiez toujours les informations sur l'emballage réel du produit."
}, {
	"attributes": [{
		"grade": "a",
		"icon_url": "https://static.openfoodfacts.org/images/attributes/vegan.svg",
		"id": "vegan",
		"match": 100,
		"name": "Végétalien",
		"status": "known",
		"title": "Végétalien"
	}, {
		"grade": "a",
		"icon_url": "https://static.openfoodfacts.org/images/attributes/vegetarian.svg",
		"id": "vegetarian",
		"match": 100,
		"name": "Végétarien",
		"status": "known",
		"title": "Végétarien"
	}, {
		"grade": "a",
		"icon_url": "https://static.openfoodfacts.org/images/attributes/palm-oil-free.svg",
		"id": "palm_oil_free",
		"match": 100,
		"name": "Sans huile de palme",
		"status": "known",
		"title": "Sans huile de palme"
	}],
	"id": "ingredients_analysis",
	"name": "Ingrédients"
}, {
	"attributes": [{
		"description": "",
		"description_short": "Aliments non transformés ou minimalement transformés",
		"grade": "a",
		"icon_url": "https://static.openfoodfacts.org/images/attributes/nova-group-1.svg",
		"id": "nova",
		"match": 100,
		"name": "Groupe NOVA",
		"status": "known",
		"title": "NOVA 1"
	}, {
		"grade": "a",
		"icon_url": "https://static.openfoodfacts.org/images/attributes/0-additives.svg",
		"id": "additives",
		"match": 100,
		"name": "Additifs",
		"status": "known",
		"title": "Sans additifs"
	}],
	"id": "processing",
	"name": "Transformation des aliments"
}, {
	"attributes": [{
		"description": "",
		"description_short": "Impact environnemental inconnu",
		"grade": "unknown",
		"icon_url": "https://static.openfoodfacts.org/images/attributes/ecoscore-unknown.svg",
		"id": "ecoscore",
		"match": 0,
		"name": "Éco-Score",
		"status": "unknown",
		"title": "Eco-Score non calculé"
	}, {
		"description": "",
		"description_short": "Pour l'instant seulement pour les produits avec du poulet ou des oeufs",
		"grade": "e",
		"icon_url": "https://static.openfoodfacts.org/images/attributes/forest-footprint-not-computed.svg",
		"id": "forest_footprint",
		"match": 0,
		"name": "Empreinte forêt",
		"status": "known",
		"title": "Empreinte forêt non calculée"
	}],
	"id": "environment",
	"name": "Environnement"
}, {
	"attributes": [{
		"description": "L'agriculture biologique vise à protéger l'environnement et à conserver la biodiversité en prohibant ou limitant l'utilisation d'engrais synthétiques, de pesticides et d'additifs alimentaires.",
		"description_short": "Les produits bios encouragent la durabilité écologique et la biodiversité.",
		"grade": "e",
		"icon_url": "https://static.openfoodfacts.org/images/attributes/not-organic.svg",
		"id": "labels_organic",
		"match": 0,
		"name": "Agriculture biologique",
		"status": "known",
		"title": "Pas un produit bio"
	}, {
		"description": "Quand vous achetez des produits du commerce équitable, les producteurs dans les pays en développement sont payés un prix plus haut et plus équitable, ce qui les aide à atteindre des plus hauts standards sociaux et environnementaux et à les conserver.",
		"description_short": "Les produits du commerce équitable aident les producteurs des pays en voie de développement.",
		"grade": "e",
		"icon_url": "https://static.openfoodfacts.org/images/attributes/not-fair-trade.svg",
		"id": "labels_fair_trade",
		"match": 0,
		"name": "Commerce équitable",
		"status": "known",
		"title": "Ne provient pas du commerce équitable"
	}],
	"id": "labels",
	"name": "Labels"
}]

simplified version
[{
	"attributes": [{
		"grade": "a",
		"id": "nutriscore",
		"match": 100
	}, {
		"description_short": "0.021 g / 100 g",
		"grade": "a",
		"id": "low_salt",
		"match": 97.2
	}, {
		"description_short": "0 g / 100 g",
		"grade": "a",
		"id": "low_fat",
		"match": 100
	}, {
		"description_short": "0 g / 100 g",
		"grade": "a",
		"id": "low_sugars",
		"match": 100
	}, {
		"description_short": "0 g / 100 g",
		"grade": "a",
		"id": "low_saturated_fat",
		"match": 100
	}],
	"id": "nutritional_quality",
	"name": "Qualité nutritionnelle"
}, {
	"attributes": [{
		"grade": "a",
		"id": "allergens_no_gluten",
		"match": 100
	}, {
		"grade": "a",
		"id": "allergens_no_milk",
		"match": 100
	}, {
		"grade": "a",
		"id": "allergens_no_eggs",
		"match": 100
	}, {
		"grade": "a",
		"id": "allergens_no_nuts",
		"match": 100
	}, {
		"grade": "a",
		"id": "allergens_no_peanuts",
		"match": 100
	}, {
		"grade": "a",
		"id": "allergens_no_sesame_seeds",
		"match": 100
	}, {
		"grade": "a",
		"id": "allergens_no_soybeans",
		"match": 100
	}, {
		"grade": "a",
		"id": "allergens_no_celery",
		"match": 100
	}, {
		"grade": "a",
		"id": "allergens_no_mustard",
		"match": 100
	}, {
		"grade": "a",
		"id": "allergens_no_lupin",
		"match": 100
	}, {
		"grade": "a",
		"id": "allergens_no_fish",
		"match": 100
	}, {
		"grade": "a",
		"id": "allergens_no_crustaceans",
		"match": 100
	}, {
		"grade": "a",
		"id": "allergens_no_molluscs",
		"match": 100
	}, {
		"grade": "a",
		"id": "allergens_no_sulphur_dioxide_and_sulphites",
		"match": 100
	}],
	"id": "allergens"
}, {
	"attributes": [{
		"grade": "a",
		"id": "vegan",
		"match": 100
	}, {
		"grade": "a",
		"id": "vegetarian",
		"match": 100
	}, {
		"grade": "a",
		"id": "palm_oil_free",
		"match": 100
	}],
	"id": "ingredients_analysis",
	"name": "Ingrédients"
}, {
	"attributes": [{
		"grade": "a",
		"id": "nova",
		"match": 100
	}, {
		"grade": "a",
		"id": "additives",
		"match": 100
	}],
	"id": "processing",
	"name": "Transformation des aliments"
}, {
	"attributes": [{
		"grade": "unknown",
		"id": "ecoscore",
		"match": 0,
		"status": "unknown"
	}, {
		"grade": "e",
		"id": "forest_footprint",
		"match": 0,
		"name": "Empreinte forêt",
		"status": "known",
		"title": "Empreinte forêt non calculée"
	}],
	"id": "environment",
	"name": "Environnement"
}, {
	"attributes": [{
		"grade": "e",
		"id": "labels_organic",
		"match": 0,
		"status": "known"
	}, {
		"grade": "e",
		"id": "labels_fair_trade",
		"match": 0,
		"status": "known"
	}],
	"id": "labels",
	"name": "Labels"
}]

I've just simplified again the attributes, this time "à la SQL", and it looks less poetic but much more compact (should take half the space - for attributes):

SQL version
product(id, barcode)
0,<PHONE_NUMBER>123

attribute (id, id_text)
0, nutriscore
1, low_salt
2, low_fat
3, low_sugars
4, low_saturated_fat
5, allergens_no_gluten
6, allergens_no_milk
7, allergens_no_eggs
8, allergens_no_nuts
9, allergens_no_peanuts
10, allergens_no_sesame_seeds
11, allergens_no_soybeans
12, allergens_no_celery
13, allergens_no_mustard
14, allergens_no_lupin
15, allergens_no_fish
16, allergens_no_crustaceans
17, allergens_no_molluscs
18, allergens_no_sulphur_dioxide_and_sulphites
19, vegan
20, vegetarian
21, palm_oil_free
22, nova
23, additives
24, ecoscore
25, forest_footprint
26, labels_organic
27, labels_fair_trade

attribute_product (product_id, attribute_id, match, grade, description_short, status):
0, 0, 100, 'a', ''
0, 1, 97.2, 'a', "0.021 g / 100 g"
0, 2, 100, 'a', "0 g / 100 g"
0, 3, 100, 'a', "0 g / 100 g"
0, 4, 100, 'a', "0 g / 100 g"
0, 5, 100, 'a', ''
0, 6, 100, 'a', ''
0, 7, 100, 'a', ''
0, 8, 100, 'a', ''
0, 9, 100, 'a', ''
0, 10, 100, 'a', ''
0, 11, 100, 'a', ''
0, 12, 100, 'a', ''
0, 13, 100, 'a', ''
0, 14, 100, 'a', ''
0, 15, 100, 'a', ''
0, 16, 100, 'a', ''
0, 17, 100, 'a', ''
0, 18, 100, 'a', ''
0, 19, 100, 'a', ''
0, 20, 100, 'a', ''
0, 21, 100, 'a', ''
0, 22, 100, 'a', ''
0, 23, 100, 'a', ''
0, 24, 0, 'unknown', '', 'unknown'
0, 25, 0, 'e', '', 'known'
0, 26, 0, 'e', '', known'
0, 27, 0, 'e', '', known'

As I said before you don't change a SQL database structure like you do with JSON files, therefore we should be very careful with what we really want:

a much smaller space than the current 45Kb json for each product
a limited set of data - barcode, name and attributes? categories?
a vague idea of what we'll use the data for, especially regarding the whereclauses - obviously on the barcode, but on the name? on the categories? on some attributes?

@teolemon @g123k continuing from #5392
I would compute the number of products (1M for France?) multiplied by the size of each product (in a mini version: barcode, name, main image), and then I'll realize that the server doesn't accept more than 10 queries a minute.
That would also mean a specific MiniProduct table.
Actually I don't know the use-case, or more precisely: how minified should the product version be?

I think we should drop images altogether, in favor of less size or more info (eg image status, attributes…)
Core use cases are:

scanning in a supermarket with no network and getting the score
core contributors taking photos of products during a scan party (does the product exist, and potentially do we have photos for it)

@teolemon you can emulate the no-network-scan-session after downloading a significative set of products (cf. dev mode / offline) and switching to flight mode. Immediate results.
Then you can think about which data a user would really need in this or that use case.


Convert to Kotlin

Convert to Kotlin and develop it as a Mobile Application. Or create this in Flutter, I see some problems in Java here.

will do thanks


unsuitable description in help file for command cleos

hi,
"contract TEXT               The contract that operates the currency"
This description confuse me, and i would use contract name rather than account name to get the currency balance. actually , the account name is right, am i right?
MacBook:eos easylend$ cleos get currency balance eosio.token whytest2
MacBook:eos easylend$ cleos get currency balance contracttest whytest2
75.0000 KAO
MacBook:eos easylend$ cleos get currency balance contracttest
ERROR: RequiredError: account
Retrieve the balance of an account for a given currency
Usage: cleos get currency balance contract account [symbol]
Positionals:
contract TEXT               The contract that operates the currency
account TEXT                The account to query balances for
symbol TEXT                 The symbol for the currency if the contract operates multiple currencies

The contract referred above is referred to the account that hosts the eosio.token contract


[BACKLOG-28778] Connection Dialog - No suitable Driver error when Host Name has a special character

[BACKLOG-28778] Adding changes to show custom dialog on error or success
[BACKLOG-28778] Capturing specific SQL Exception and setting message for user
[BACKLOG-28778] Moving exception text to properties file

:white_check_mark: Build finished in 31m 1s
Build command:
mvn clean verify -B -e -Daudit -amd -pl core,ui

:ok_hand: All tests passed!
Tests run: 4764, Failures: 0, Skipped: 7    Test Results

:information_source:  This is an automatic message

:white_check_mark: Build finished in 31m 1s
Build command:
mvn clean verify -B -e -Daudit -amd -pl core,ui

:ok_hand: All tests passed!
Tests run: 4764, Failures: 0, Skipped: 7    Test Results

:information_source:  This is an automatic message

:white_check_mark: Build finished in 31m 42s
Build command:
mvn clean verify -B -e -Daudit -amd -pl core,ui

:ok_hand: All tests passed!
Tests run: 4764, Failures: 0, Skipped: 7    Test Results

:information_source:  This is an automatic message

:white_check_mark: Build finished in 31m 3s
Build command:
mvn clean verify -B -e -Daudit -amd -pl core,ui

:ok_hand: All tests passed!
Tests run: 4764, Failures: 0, Skipped: 7    Test Results

:information_source:  This is an automatic message


samples: net: httpi_client: Disable OPTIONS method

As many HTTP servers do not support OPTIONS method, disable it
by default.
Fixes #4325
Signed-off-by: Jukka Rissanen<EMAIL_ADDRESS>

Typo in the commit message: "httpi_client" (instead of "http_client").

Shouldn't this be fine, with the server returning a Not Implemented error?

When we run the same application in IPv4 domain, then  don't see any issues with OPTIONS method. So is the issue related to IPV6 domain.
I have applied the patch of #4348 then it fails to send the POST request in Ipv6.


TICtxtWrd-common: Test Improvements, Context Wording, ecosystem common.

Test Improvements, Context Wording, ecosystem common.

As I have deployed and merged https://github.com/dependabot/dependabot-core/pull/9856


Running devfile integration tests on kubernetes cluster

What type of PR is this?
/kind documentation
[skip ci]
What does does this PR do / why we need it:
Added documentation for running devfile integration tests on kubernetes cluster.
Which issue(s) this PR fixes:
Fixes #3205
How to test changes / Special notes to the reviewer:
Manually verify the doc change.

Flake : https://deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gcs/origin-ci-test/pr-logs/pull/openshift_odo/3233/pull-ci-openshift-odo-master-v4.2-integration-e2e/594#1:build-log.txt%3A711
Already tracked in https://github.com/openshift/odo/issues/3193
/retest

/lgtm

/approve

/lgtm


Note disappearing when changing encounter modes

Switching the Workflow dropdown to Post-encounter or Pre-encounter used to cause the editor contents to disappear. This PR resolves that so the editor does not lose contents. Communication between components is accomplished by adding a couple more props at the FullApp level - using global state seems less than ideal to me, there may be a better way to accomplish this.
Running tests now.
Pull requests into Flux require the following. Submitter and reviewer should :white_check_mark: when done. For items that are not-applicable, note it's not-applicable (N/A) and :white_check_mark:.
Submitter:
Running the Application

[X] Manually tested in Chrome
[X] Manually tested in IE

Documentation

[X] This pull request describes why these changes were made
[X] Recognizes any potential shortcomings/bugs in the description
[N/A] Cheat sheet is updated
[N/A] Demo script is updated
[N/A] Documentation on Wiki has been updated
[N/A] Additional technical components and libraries are documented and added to wiki as needed

NoteParser

[N/A] Note parser has been updated if structured phrases change

Code Quality

[X] 4-space indents - convert any tabs to spaces
[X] Use public class field syntax for binding functions - ex. handleChange = () => { ... };
[X] Code is commented

Tests

[X] Existing tests passed
[N/A] Added UI tests for slim mode
[N/A] Added UI tests for full mode
[N/A] Added backend tests for fluxNotes code
[N/A] Added note parser examples to test new functionality

Reviewer 1: Name

[ ] Code is maintainable and reusable, reuses existing code and infrastructure where appropriate, and accomplishes the task’s purpose
[ ] The tests appropriately test the new code, including edge cases
[ ] You have tried to break the code

Reviewer 2: Name

[ ] Code is maintainable and reusable, reuses existing code and infrastructure where appropriate, and accomplishes the task’s purpose
[ ] The tests appropriately test the new code, including edge cases
[ ] You have tried to break the code

[GQ1] In pre-encounter mode, I hit new note then type in some content. Switch to post-encounter and the note stays. If I close that note and switch back to pre-encounter mode, the in-progress note is re-opened. I don't think it should be
[GQ] With an in-progress note open, switching from post-encounter to pre-encounter maybe shouldn't keep that open but instead close it (it's still in an in-progress note so not lost)?
I think the correct behavior needs to be thought through a little more here.

Closed because we need to think through all the scenarios here (see GQ). When should the open note close? When should a new note be created? Should an existing in-progress note ever be auto-opened?
Should we just eliminate the workflow drop-down and not expose that to the user?


Ignore pip warning

WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

https://github.com/christian-ci/action-yaml-github-output/issues/9

@christian-ci Thanks, it works!


Added missing import in docs/ref/contrib/contenttypes.txt.

Maybe I missed it but I believe the documentation hasn't explicitly shown you how to import GenericPrefetch. You can click on the anchor of the class and look at the address bar to get it, but I don't think that's very obvious.
I added this to the Python console example, following the documentation for other classes.

@laymonage Thanks :+1:


bug: can't switch to alternate-file with CTRL-6 like in vim-vinegar or dirbuf.nvim

Did you check the docs and existing issues?

[X] I have read the docs
[X] I have searched the existing issues

Neovim version (nvim -v)
NVIM v0.9.0-dev-623+g0996430392
Operating system/version
archlinux
Describe the bug
can't switch to alternate-file after open file with oil.nvim
Steps To Reproduce

open a normal file A
use Oil open another file B
press CTRL-6

Expected Behavior
switch to file A
Directory structure
No response
Repro
-- DO NOT change the paths and don't remove the colorscheme
local root = vim.fn.fnamemodify("./.repro", ":p")

-- set stdpaths to use .repro
for _, name in ipairs({ "config", "data", "state", "cache" }) do
  vim.env[("XDG_%s_HOME"):format(name:upper())] = root .. "/" .. name
end

-- bootstrap lazy
local lazypath = root .. "/plugins/lazy.nvim"
if not vim.loop.fs_stat(lazypath) then
  vim.fn.system({
    "git",
    "clone",
    "--filter=blob:none",
    "--single-branch",
    "https://github.com/folke/lazy.nvim.git",
    lazypath,
  })
end
vim.opt.runtimepath:prepend(lazypath)

-- install plugins
local plugins = {
  "folke/tokyonight.nvim",
  {
        "stevearc/oil.nvim",
        config = function()
            require"oil".setup() -- add any needed settings here
        end,
  },
  -- add any other plugins here
}
require("lazy").setup(plugins, {
  root = root .. "/plugins",
})

vim.cmd.colorscheme("tokyonight")
-- add anything else here

Should be fixed!

Seems not work in float mode :(.

open file A.
:Oil --float and open file B
press CTRL-6
jump to incorrect file

Fixed for floating windows too


java.lang.OutOfMemoryError:  at es.voghdev.pdfviewpager.library.adapter.SimpleBitmapPool.createBitmapAtIndex (SimpleBitmapPool.java:83)

This error is coming on nearly all versions of Android, And I don't get the why this error will come. So Please Help me out.And give solution to remove this error.
Thank you..

What's the size of the PDF you are trying to load? If it crashes in all Android devices, probably your PDF file is the cause

size of pdf is 4 to 5 mb. Yes it will crashes on all android devices

Have you tried another PDF such as the one in the example project?

sometime it will working and sometime it will give error..?

Your App is definitely getting out of memory. Probably you have many resources loaded into memory, and when you try to load the Pdf renderer, it overflows. Sometimes it does not crash probably because you have navigated through the app for just a few time, and memory is still not overloaded. I've seen a similar situation recently with a heavy PDF document. I suggest to create your own subclass of BasePdfPagerAdapter and try to render the PDF document using less memory


Problem: Driver tests failing

Solution: Fix necessary tests for release.

@vrde Looking at the code back again, I see that any fixutre which uses transactions_api_full_url appends ?mode=commit to it. So maybe we can update the fixutre to,
def transactions_api_full_url(api_root):
    return api_root + '/transactions?mode=commit'

@kansi I think that's a good solution!

Oh noees tests are still failing for py3.5

@vrde @codegeschrei It passes now.


Experiments on forward view scenes

Thanks for your excellent work!
I noticed the visualized results are all experiments of the drone view scenes.
Is there any visualized results on scenes like block-nerf data?

Hi, we currently don't provide the videos of Block-NeRF because we haven't gotten proper camera paths for Block-NeRF.


API FR: Encode/Decode Base64

At least plugin routing it holds.
I`m writing GUI for FX chain, and seems LUA cant decode base64 to hex (and back, I guess) properly.
http://forum.cockos.com/showthread.php?t=168478

+1 on this one. I have written myself a encoder/decoder in Lua but it's extremly slow on huge strings(due a lot of concatenation needed) so native functions to do it would be great.
And as in some areas Reaper uses hexstrings, I would like to ask for Ascii2Hex and Hex2Ascii-functions as well, who convert a string into their hex-counterparts and reverse.
(I have them coded in Lua as well but...they have the same problem being too slow on huge string due some of Lua's design decisions.)
Thnx :)

@cfillion What do you think about exposing the WDL Base64 functions to ReaScript?
https://github.com/justinfrankel/WDL/blob/4acb00d6298047cbb4b82f67a0ea632e23b7a77f/WDL/wdl_base64.h


Add support for MongoDb

I needed support for MongoDB to use this project, so I thought that I might as well add it, and create this pull request.

Thank you so much.

MongoDB provider nuget has been published.


Throw validation error if request is invalid json/yaml

Currently the user can run into NPE's when the request contains invalid json because the validator doesn't run in this case.

Fixed in #14.


Recency in clv.utils.clv_summary()

I was wondering if the definition of recency is correct. According to the paper of Hardie (http://brucehardie.com/notes/008/pareto_nbd_MATLAB.pdf) recency is defined as:

his “recency” (when his last transaction occurred)

However in this function (also in the function of the lifetimes package) "recency" is the timespan between the first and the last purchase.
So shouldn't be recency be recency = observation_period_end - max(transaction_date) rather than recency = max(transaction_date) - min(transaction_date) ?

Hey @matthias-Q,
The definition of recency you proposed is used in RFM Analysis, which I'm considering implementing into pymc-marketing because it'd be useful for cohort studies.
For BTYD modeling however, time between transactions is one of the key aspects we're trying to capture, hence the timespan definition of recency. This fits Hardie's definition because with any given customer, T=0 for min(transaction_date)  so recency indicates the time period when the last transaction occurred.
Speaking of Hardie, @juanitorduz has actually been in contact with him about this library, and he's very interested in the progress we're making.

Thanks for the quick answer. I think I understand. Technically, the different definitions contain the same information.
What I find confusing, that all of the BTYD papers also use the recency definition that I mentioned above (time since last interaction)
https://cran.r-project.org/web/packages/BTYD/vignettes/BTYD-walkthrough.pdf
http://brucehardie.com/papers/018/fader_et_al_mksc_05.pdf

Different definitions for the same term in this domain of modeling can definitely cause some confusion. However, I don't see any discrepancies in how the papers define recency. Quoted from BTYD-walkthrough.pdf:

We have to keep track of a
“time zero” as a point of reference for recency and total time observed

"Time zero" is referring to the customer's first purchase. If they did not make any repeat purchases, then recency will also be zero.
Time since last interaction is valuable for bucketing customers in RFM analysis because recent purchases suggest the customer is still active. However, total time observed is just the end date of the study period, chosen by the researcher. We need to know the customer's recency relative to their first purchase for predictive modeling.

If there are no further questions, I'm gonna go ahead and close this.


Fixed some broken API commands

It looks like the volume mute commands wasn't working because /volume/:vol was being greedy and swallowing them all, so I changed it to POST and corrected the readme to say this and the other volume commands were POST.
Others appeared to not be working because they had the IP repeated in their URL segments.

Hi @PeteWilliams , do this repo works with Marantz SR5012?
Actually I get null value for all methods invokation.


[BUG] After following 'How to install django CMS by hand' getting error 'TypeError: issubclass() arg 1 must be a class'

I followed https://docs.django-cms.org/en/latest/how_to/install.html exactly (I believe).
When I ran manage.py cms check I got the error:
TypeError: issubclass() arg 1 must be a class
I got the same error when running manage.py runserver
Steps to reproduce

Follow the install instructions exactly: https://docs.django-cms.org/en/latest/how_to/install.html
Run python manage.py cms check
See error

Expected behaviour
I expect to see no error, but I have never seen it run successfully so I'm not sure what to expect for the success case.
Additional information (CMS/Python/Django versions)
I'm using Django 3.2 as the instructions state.
I'm running on Windows 11 with Python 3.10
Do you want to help fix this issue?

[X] Yes, I want to help fix this issue and I will join #workgroup-pr-review on Slack to confirm with the community that a PR is welcome. But I'm going to be away from my keyboard next week so I'll have to connect after that.
[ ] No, I only want to report the issue.

What version of django-cms are you using please?

I am using version 3.10.0.

Thanks for confirming @charris-msft.
I wanted to confirm that I don't have an issue with this same setup, so we'll need to reach out to the wider community on slack.
% python --version
Python 3.10.3
% pip freeze | grep "Django=="
Django==3.2.13
% pip freeze | grep "django-cms=="
django-cms==3.10.0
% python manage.py cms check

********************************
Checking django CMS installation
********************************

...

I posted this issue on the django-cms slack channel and got the following advice:

How to install django CMS by hand
The easiest way to install django CMS is by using the automated django CMS installer. This is the recommended way to start with new projects, and it’s what we use in the tutorial section of this doc...

I found some good instructions for installing and using the djangocms-installer here: Installing django CMS


Feature/2501 blui prev version

Fixes PXBLUE-2501 .

Changes proposed in this Pull Request:

[X] @pxblue => @brightlayer-ui

[X] PX Blue => Brightlayer UI

[X] PXBlueDrawer => BLUIDrawer

Run yarn start:storybook from root.

Hi @emclaug2 , I tried to fix build for https://github.com/pxblue/react-component-library/pull/331 but no luck, so created new fresh branch from commit https://github.com/pxblue/react-component-library/commit/0630bb2a6eff03d496dca4301b2936088465afc0. Build is passing for this branch. Closing PR 331.

Storybook looks great to me, could you update the components/package.json version to be 4.2.0 ?

Some things that will need updated:

[ ] root LICENSE file says Power Xpert Blue... let's change that to Brightlayer UI
[ ] the welcome story /demos/storybook/stories/welcome.stories.tsx has a Power Xpert Blue reference
[ ] the DrawerHeader title is Power Xpert Blue in /demos/storybook/stories/drawer/with-nested-list-items.tsx

4.2.0

Done.

Now that is this merged, I'll do a manual publish for the component library at v4.x.xx.


Merge versioning feature branch into develop.

Ticket: []
Feature flag: n/a

Purpose

Summary of Changes

Screenshot(s)

Side Effects

QA Notes

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

362 of 444 (81.53%)  changed or added relevant lines in 46 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.5%) to 73.083%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

lib/osf-components/addon/components/registries/review-actions-list/review-action/component.ts
7
8
87.5%

lib/osf-components/addon/components/registries/update-dropdown/list-item/component.ts
4
5
80.0%

lib/registries/addon/branded/moderation/index/route.ts
0
1
0.0%

lib/osf-components/addon/components/registries/overview-form-renderer/component.ts
10
12
83.33%

lib/osf-components/addon/components/registries/registration-list/card/component.ts
12
14
85.71%

lib/osf-components/addon/components/registries/schema-block-renderer/editable/files/component.ts
5
7
71.43%

lib/registries/addon/edit-revision/route.ts
14
17
82.35%

lib/osf-components/addon/components/node-card/component.ts
14
18
77.78%

lib/registries/addon/edit-revision/page/controller.ts
8
12
66.67%

lib/osf-components/addon/components/registries/new-update-modal/component.ts
2
8
25.0%

Totals

Change from base Build<PHONE_NUMBER>:
0.5%

Covered Lines:
5160

Relevant Lines:
6795

💛  - Coveralls


Add secret_token field to S3

Livebook v0.12 may now set this field. We need to:

Add this field to S3 here
Pass down this field in Explorer
Receive this field from Livebook in Kino

@josevalim we have a token field that represents the session_token. Is this the same thing? I mean, is it secret_token or session_token? And if it's the same, should we rename the token field to session_token?

Perfect, they are the same and we chose to rename it from session_token to token in Livebook. :) So this is done! Thank you!


Added MainThread to 'ON' statements

Main Thread - <on.....

Not really needed atm, not having problems with walk. Also this is out of date in that we don't use this module anymore.


all: fix remaining prealloc issues

As discussed previously, this check shouldn't be run by default as part
of the precheck Makfile target due to the possibility of false
positives (see #10806 for the previous approach). Thus, we'll run
prealloc manually once at the end of a release cycle and manually
review its results and thix them where appropriate.

test-me-please

Coverage increased (+0.01%) to 46.708% when pulling 3c06ca75cfe06261a43695a1a8466737c9b89ea1 on pr/tklauser/prealloc-check into f5425fec71bff6762a5bcc5dc3f3078728020141 on master.

test-me-please


Android KeyStore: Throw exceptions for missing keys

Android was returning null secrets when they were missing instead of raising an exception (like the other implementations do). This would cause callers that expected non-failure to be a real secret to fail when they tried to decode null secrets.
:eyeglasses: @keybase/react-hackers

(rebased because the change from https://github.com/keybase/client/pull/4571 was accidentally also in this commit)


Migrate to RxJS Next

Let's use this issue to document our progress as we migrate.
TODO

[x] RxJS Next peer dependency in Cycle Core, tests passing
[x] RxJS Next peer dependency in Cycle DOM, tests passing
[ ] RxJS Next peer dependency in Cycle HTTP Driver, tests passing
[ ] Use publishBehavior instead of replay
[ ] No use of _unsubscribe() in any Cycle code

@TylorS Could you elaborate on the rough edges besides the use of _unsubscribe()?

@staltz You had mentioned to me that lift() is not a one to one equivalent to the previous use of asObserver(). I see that as a rough edge. Is there something better?
Also the fact that I seemingly have to use _unsubscribe() instead of unsubscribe()
Also it's strange that some of the tests are very time sensitive.

@TylorS Which tests were that? Were they consistently time consuming?

@Frikki Check my comments on this commit
https://github.com/TylorS/cycle-core/commit/a4ec0ba787e7a675117bddd458322cebd2f2d6c2

Also, tests take almost 400ms to complete.

Each test?

No, overall.

@staltz It’s test no. 8 that takes long time, ~300 ms, because delay() only accepts minimum value of 100 if the test shouldn’t fail.
let number$ = Rx.Observable.range(1, 3)
        .concatMap(x => Rx.Observable.of(x).delay(100));

@staltz I think we can cross off two bullet points.

Cycle DOM and what else?

I thought Cycle HTTP was ready.

@Frikki not totally ready https://github.com/cyclejs/cycle-http-driver/pull/21#discussion_r42650320

@staltz But @TylorS wrote, 'Done' two days ago(?!)

Done as in I did what he asked me to. What should work doesn't, and what shouldn't work does. If I understand correctly.

Do we have an issue open on what does and doesn’t work?

@Frikki just read those comments

That's what this issue is for

I just feel that issues should be broken down into smaller issues and tracked separately. This issue seem to combine all issues into one giant blob of information. While I think it is good to have a combining issue, I think that a milestone would be more appropriate. And, indeed, cycle-http-driver should have its own milestone for RxNext and its own issues on that repo. Likewise, cycle-dom.

Yes, let's do that

@staltz I think it's safe to close this as it's being addressed with Cycle Diversity.
We can reopen if need be.


Proposal: CI Framework

New CI framework proposal.

I like the idea. Will there be a way to input values when necessary as well? For instance, for mediawiki you need to specify a password and there is no default. In this case it seems the there would need to be a way to address this in input.

@jmontleon I forgot to address config. Thanks for pointing that out.
In order to keep the CI yaml as simple as possible, I was thinking we could have a default location to drop and APB config file.  Since each APB will have it's own git repo, we can curl the location of any config we need.  I think this is a good way to start since it's simple. As things progress, we could always add config override or custom config file locations.
There's also @shawn-hurley's work with the test apb playbook and that playbook will have a set of defaults. Maybe we can share those defaults?

I have a general question here, what are we attempting to verify with this CI?
Are trying to make sure that the APB's are good to be published or are we trying to verify that the ASB can deploy and manage a configurable setup of APBs? Alternatively, verifying a third option that I am not aware of.

This is to replace the existing CI we have, which does a provision and bind test between mediawiki and postgresql.
The goal is to cover: ASB can deploy and manage a configurable setup of APBs, but it can also tell us: APB's are good to be published.

ASB can deploy and manage a configurable setup of APBs

Makes complete sense for this.

APB's are good to be published

This should be covered by the test action that we have defined. I think that makes more sense. These are going to be doing the same basic things, running a configurable number of actions on an APB. The benefit of this PR is that we can have two APBs interact while running the configurable number of actions, which IMO should be a test of the broker and should tell us nothing of the APBs status.

Each APB repo will have a config file for running a CI job.

This to me still feels wrong. I think the APB should be able to be tested independently of any service catalog/service broker for it's CI. The requirement that we make an APB test with specific other APBs, IMO, takes away from their generic nature (until we get to have dependencies). Also, in your example you're requiring and APB that has set bindable to false, to test the binding of a database APB. Which would we mark as failed if binding failed?  I think this can be easily remedied by moving the configuration out of the APB repo.

There's also @shawn-hurley's work with the test apb playbook and that playbook will have a set of defaults. Maybe we can share those defaults?

I agree, we will need to add to the test documentation, that there is a specific default location, and will need to add some sort of mechanism for passing those in on certain actions, but yes this makes total sense to me.

APB's are good to be published

This should be covered by the test action that we have defined. I think that makes more sense. These
are going to be doing the same basic things, running a configurable number of actions on an APB. .

I agree with what your saying.  The goal of this framework is to enable complex testing.  But, basic testing is unavoidable when doing complex testing.  In the gate, the mediawiki and postgresql CI completely covers all of the smoke tests for mediawiki and postgresql. It's unavoidable.  The difference between this in the test playbook is that the playbook is portable and this CI sits in the repo. They are different use cases.

The benefit of this PR is that we can have two APBs interact while running the configurable number of
actions, which IMO should be a test of the broker and should tell us nothing of the APBs status

I slightly disagree. It does tell us the APBs status, but only in the master branch. It does not tell us the status of any published APB image you pull from dockerhub. That's where the test playbook will give you a status because it's shipped with the APB code while the framework is not.

Each APB repo will have a config file for running a CI job.

This to me still feels wrong. I think the APB should be able to be tested independently of any
service catalog/service broker for it's CI.

I think it's great to have APBs be independently tested from the catalog and broker, but it will improve our coverage if we also supplement that by testing with the broker and catalog.  I would like test bind, multi APB apps, and whether one APB works with another.

The requirement that we make an APB test with specific
other APBs, IMO, takes away from their generic nature (until we get to have dependencies).

The framework is designed to make it really easy to test with other APBs, but it's not a requirement.  The point of making it easy is to encourage APB developers to do it so we improve our CI testing.

Also, in your example you're requiring and APB that has set bindable to false, to test the binding of
a database APB. Which would we mark as failed if binding failed? I think this can be easily
remedied by moving the configuration out of the APB repo.

The are no requirement that the APB does not itself impose. So if the APB can't bind to anything and tries to it will fail as expected.

Spoke with @shawn-hurley.  To summarize our conversation:

We want to try and reuse test.yml with the CI framework if possible.
There's overlap with the test.yml work, but the CI framework will test APB using the broker and catalog while the test.yml is a portable smoketest.
We want apb devs to use both the smoke test and CI framework when building new APBs.


add kubernetes system namespaces to generated support bundles

What this PR does / why we need it:

Which issue(s) this PR fixes:

Does this PR require a test?

Does this PR require a release note?

Does this PR require documentation?

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Add README info about playing native recorded audios

Why current description is not working?
(Also, can you replace chrome to Chrome)

It's because there isn't clear that audios recorder without the polyfill will not play on safari

Can you make it shorter and more to the first paragraph?

Right now text looks like arguing with load polyfill only for Safari and Edge. Do you not like this condition polyfill loading?

Yes, i think that's the point, if you will just load the polyfill with these browser, and record something on Chrome for example, you won't be able to reproduce the recorded data in Safari (and probably not on edge as well) unless you pre-process it somehow. It's because native MediaRecorder always sets the mime type to audio/ogg which is not supported by Safari!

Polyfill hurts performance for recording client and even using MP3 has 2x size compare to OGG.
I think file convertion on back-end is still the best option.
But we can mention that files from Chrome will not be playable in Safari and you need to convert them on back-end or force polyfill with MP3 encoder for everyone (with the price of client's performance).

Totally agree, i changed the description, let me know if it's not clear enough!

👍


Updates to SBT 0.13.9-RC3

SBT 0.13.9 contains features that seem to help with GC and large
dependency trees. 0.13.9-RC3 is a stabalizing release, which will become
final next week, if there are no problems. This changes the build so
that zipkin can help test it.
See https://groups.google.com/forum/#!topic/sbt-dev/7k5sTaT9ev4

PTAL this is not failing in new ways :)

+1

rebasing.. then will merge


Figure out better place to init for Django apps

from #19

actually @dcramer I am confused about the motivation for this. Do you want to capture exceptions happening during import of the application?

One thing to note is that we monkeypatch Django anyway, we don't really rely on things like middleware order (because we want to capture as much as possible)

I'm closing this because the SDK is now released as-is and this issue is non-actionable as it stands.


AttributeError: module 'pyaudio' has no attribute 'PyAudio'

i get this error when i run the visualization script
sudo python3 visualization.py scroll
Traceback (most recent call last):
File "/home/pi3b/dancyPi-audio-reactive-led/python/visualization.py", line 368, in
microphone.start_stream(microphone_update)
File "/home/pi3b/dancyPi-audio-reactive-led/python/microphone.py", line 8, in start_stream
p = pyaudio.PyAudio()
AttributeError: module 'pyaudio' has no attribute 'PyAudio'

uninstalled and reinstalled pyaudio and it worked


[loadgenerator] loadgenerator doesn't sent requsets to other service

Bug Report
Which version of the demo you are using?
0.3.4-alpha
Symptom
I use helm to install the demo, the loadgenerator service start sucessfully, but it doesn't sent requsets to other service, the only logs I get are:
 [2022-09-14 03:01:03,338] my-otel-demo-loadgenerator-69b7bcf679-9qzwp/INFO/locust.main: Starting web interface at http://<IP_ADDRESS>:8089 (accepting connections from all network interfaces)

 [2022-09-14 03:01:03,373] my-otel-demo-loadgenerator-69b7bcf679-9qzwp/INFO/locust.main: Starting Locust 2.6.1

env:
kube version: v1.21.5
os: Linux 3.10.0-1160.6.1.el7.x86_64

hi @Frapschen
this bug should fixed :)
plz try again.
ref: https://github.com/open-telemetry/opentelemetry-helm-charts/pull/361


Creating a script/a folder using Rider or adding a new [Export] var using it doesn't cause Godot to refresh its cache.

Tested versions

Reproductible in 4.4 on multiple machines
Not reproductible in 4.3.

System information
Godot v4.4.dev2.mono - Windows 10.0.22631 - Vulkan (Forward+) - dedicated NVIDIA GeForce RTX 4090 (NVIDIA; <IP_ADDRESS>94) - AMD Ryzen 7 7800X3D 8-Core Processor (16 Threads)
Issue description
Using the latest version of Rider with the 4.4 dev2 Godot version, I've been creating new folders, files and exported variables. Every time, Godot doesn't recognize the changes and requires a full project reboot to have it detect and import everything. This is only a recent issue, and was unable to reproduce in 4.3.
Steps to reproduce

Create a C# Godot project and open the solution in Rider.
Create a script or folder using Rider.
Switch to Godot and the file will be missing from the File Explorer, as well as from the Create Node menu (if the script is a new Node).

Minimal reproduction project (MRP)
Can be achieved using a new project.

Related to #74830 perhaps?

Probably. I tried building the engine with branch 4.3, which seems to have that cherry-picked fix, but it crashes on opening, so I can't really test it.


[Design][Bug] Esteemer user feedback when no candidates are selected

Currently we have no consensus on how the pipeline should handle the error it generates when 'Keymax' is empty after running esteemer.
With the right setup of esteemer this really should be an edge case, however this is how it looks for the user:
  File "/Users/mack/Code/Display-Lab/precision-feedback-pipeline/main.py", line 188, in createprecisionfeedback
    node,spek_es=es.score()
                 ^^^^^^^^^^
  File "/Users/mack/Code/Display-Lab/precision-feedback-pipeline/esteemer/esteemer.py", line 422, in score
    Keymax = max(zip(score_dict.values(), score_dict.keys()))[1]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: max() arg is an empty sequence

To generate this error, use the following performance data:
    [7,"GLU01","2023-01-01",45,55,100,45,45,50,1],
    [7,"GLU01","2023-02-01",45,55,100,45,45,50,1],
    [7,"GLU01","2023-03-01",45,55,100,45,45,50,1],
    [7,"GLU01","2023-04-01",45,55,100,45,45,50,1],
    [7,"GLU01","2023-05-01",45,55,100,45,45,50,1],
    [7,"GLU01","2023-06-01",45,55,100,45,45,50,1],
    [7,"GLU01","2023-07-01",45,55,100,45,45,50,1],
    [7,"GLU01","2023-08-01",45,55,100,45,45,50,1],
    [7,"GLU01","2023-09-01",45,55,100,45,45,50,1],
    [7,"GLU01","2023-10-01",45,55,100,45,45,50,1],
    [7,"GLU01","2023-11-01",45,55,100,45,45,50,1],
    [7,"GLU01","2023-12-01",45,55,100,45,45,50,1]

When restricted to social better/worse, could be having an issue with equal gaps to average and 90th percentile? Perhaps the set rules are incomplete, perhaps not deciding a winner out of the tie therefore no candidates? Unsure, need to better understand Esteemer implementation to give more detailed analysis.

This error is already resolved. I tested the same test case again. The esteemer works fine. Since Patrick was not available on Wednesday(11/1/2023) and Thursday (11/2/2023), he may not have known. There is a communication gap and this bug does not exist.I would request Patrick to retest again.

Bug still present. See #129.

Only surfacing with Gaile input_message from persona smoketest, which is a great improvement. Once resolved pipeline will be passing smoketest without 500 errors, which is amazing progress!


Sanity check config file for insufficiently restrictive permissions.

Probably should do this with HIL, too.

I meant obmd config file.

@ianballou, rebased.


Difficult to get overview of machine pool/machines on cluster detail view when lots of machines

See below:
When you have lots of machines, you only see one pool header and the first set of machines - you have to scroll to see more.
Suggest we:

Make it easier to see how many pool and total machines there are
Make it easier to see how many machines are in each pool

Ticket #6142 - Test Results
With Docker on a single-node instance:
Reproduced on rancher v2.6.5:

Fresh install of rancher v2.6.5
Provision a downstream RKE1 AWS Node driver cluster
Reproduced - UI before enhancement

Screenshot:

Verified on rancher v2.6-328b12b43716c7d30a605905d73a0b799ec2a4bd-hea:

Fresh install of rancher v2.6-head
Provision a downstream RKE1 AWS Node driver cluster
Verified - UI enhanced

Screenshot:


Use default event handler for richtext

Until the new release
Jira ID:
Test URLs:

Before: https://main--exlm--adobe-experience-league.hlx.page/en/docs/integrations-learn/experience-cloud/solution-categories/content-management
After: https://--exlm--adobe-experience-league.hlx.page/en/docs/integrations-learn/experience-cloud/solution-categories/content-management

not needed anymore


Enable "Allow Hidpi" by default

This fixes window management issues across platforms on hiDPI monitors.
The --low-dpi command line argument has been removed as similar functionality (forcing low-DPI mode on DPI-aware programs) is already provided by Windows and macOS itself.
This closes https://github.com/godotengine/godot-proposals/issues/3681, closes https://github.com/godotengine/godot/issues/13017 and closes https://github.com/godotengine/godot/issues/24922.

Thanks!

Any chance of this in Godot 3.5?

Any chance of this in Godot 3.5?

This is a compatibility-breaking change. While I think the breakage would be worth it here, it would also mean with having to deal with a lot of support requests related to this default setting change after 3.5 is released :slightly_smiling_face:


Source: files (s3 and local)

Not 100% done yet but here is a fairly refined draft. Just a little more cleanup to do and a tiny bit more work left on Azure, and then documentation assuming we like the approach.
Currently extends FileSource (which itself had some minor adjustments to better accommodate extension).

TODO:

update local file docs
add azure docs
double check azure functionality (been focused on S3 first)
maybe add a simple file download queue?
look over Tomas's updated replay timestamp logic and see if changes are needed

This is ready to merge, depending on final thoughts on the client stuff for S3.


Turn on the version enforcer for all environments

Background
We have a nifty VersionEnforcer that will silently refresh your page if a new commit is pushed, but it's only enabled in production.
Action requested
Enable the VersionEnforcer in all environments. (This may require some Azure work to ensure the /app/static/commit.txt endpoint is available on all environments.)
Acceptance Criteria
VersionEnforcer is running in our lower environments, not just production.
Additional context
It'd be nice if we didn't have the 15 minute wait enabled and could truly not have to worry about backwards-compatibility, but I assume that was done for a reason...this PR may have more details.

Closed: will revisit at a later time. (06/18/2024)


Avoid renameat2 on Android

Motivation:
Until very recently Android didn't support the renameat2 syscall, this means that on most systems NIO will fail to compile. renameat2 is used to delay the materialization of files on Linux. On Darwin a different code path is used which we can make use of on Android too.
Modifications:

Update a few if-defs to force Android onto the alternative path

Result:
NIO builds on Android

@finagolfin could you verify this fixes your build?

I'll apply it locally and let you know in a little bit, thanks.

This isn't building either, some issues with optionals, will try modifying this patch and let you know within an hour.

This isn't building either, some issues with optionals, will try modifying this patch and let you know within an hour.

Suspect you'll need this commit too: https://github.com/apple/swift-nio/pull/2627/commits/600a21069a773d7401f830b9b6ef92f50caccec9

Another problem is that Bionic doesn't have confstr() at all:
/home/swift-nio/Sources/NIOFileSystem/Internal/System Calls/Syscalls.swift:389:12: error: cannot find 'confstr' in scope
    return confstr(name, buffer, size)                                                                                ^~~~~~~

I think this pull works to fix the renameat2 issue, but then other compilation errors from your large pull pop up on Android, looking into them now.

Another problem is that Bionic doesn't have confstr() at all:
/home/swift-nio/Sources/NIOFileSystem/Internal/System Calls/Syscalls.swift:389:12: error: cannot find 'confstr' in scope
    return confstr(name, buffer, size)                                                                                ^~~~~~~

We can workaround confstr too as it looks like we're only using it on Darwin.

I think this pull works to fix the renameat2 issue, but then other compilation errors from your large pull pop up on Android, looking into them now.

Great, thanks for looking!

I don't see other issues related to renameat2, so I think this pull works. If you can ifdef confstr() out on Android too, please go ahead and merge these changes.
I will work through the remaining compilation issues related to optionals, because the latest Android NDK added nullability annotations all over Bionic that your pull will have to be slightly modified for, and submit another pull for that later.

I don't see other issues related to renameat2, so I think this pull works. If you can ifdef confstr() out on Android too, please go ahead and merge these changes.

Great, will do!

I will work through the remaining compilation issues related to optionals, because the latest Android NDK added nullability annotations all over Bionic that your pull will have to be slightly modified for, and submit another pull for that later.

Do you have any timeline on this? We'd like to tag a release and would rather not break you!

Do you have any timeline on this?

Most likely later today.

We'd like to tag a release and would rather not break you!

I mostly build trunk right now, so shouldn't be an issue.

Applied this locally and all the remaining errors appear to be type and optional mismatches related to slightly different Android declarations, so you can go ahead and merge this when you like.

No, I think that should work, as I don't think that syscall is filtered out on Android.

See #2628

Tried that Musl syscall pull out, seems to work well after adding my patch for the type/optional mismatches and the confstr() commit from this pull, except I see several tests failing because they call linkat(), not sure where that's coming from.

I will look into the linkat() testing issues on Android in the next couple days. Meanwhile, if @glbrntt would reduce this pull to the confstr() commit and merge, that would be helpful.