injector: Rewrite app health probes; pass through Envoy to payload

This PR is a proposal for a way to handle Kubernetes health probes (liveness, readiness, startup) to a payload, which has been joined OSM.
Background
If a pod has liveness, readiness, startup probes defined, when that pod is augmented with an Envoy sidecar - health probes start to fail.  Probes from kubelet to pod fail because kubelet does not have enough context to mTLS through Envoy to the original payload. This results in the pod failing.  More context is available in the issues linked here: https://github.com/openservicemesh/osm/issues/2207
Proposed solution

Mutating webhook will rewrite original liveness, readiness, startup probes and change both port and path of probes
Mutating webhook will create dedicated static HTTP listeners on the Envoy proxy for liveness, readiness, startup - one per type via the bootstrap Envoy config
Envoy proxy would pass-through traffic (when port and path match) to the payload

Example
A pod with the following probe:
          livenessProbe:
            httpGet:
              path: /liveness
              port: 80

will have the probe rewritten (by the mutating webhook) to:
          livenessProbe:
            httpGet:
              path: /osm-liveness-probe
              port: 15901

The Envoy proxy will be bootstrapped (by the mutating webhook) with statically configured HTTP listener on port 15901, which will route requests for /osm-liveness-probe to the underlying payload on port 80 and at the original /liveness URL.
The init container will configure iptables w/ the Envoy ports for the dedicated static health probe listeners.
This PR demonstrates this already working -- bookstore pod in CI is instrumented w/ probes and these pass.
This PR does not add tests yet - these will be added with the following PR: https://github.com/openservicemesh/osm/pull/2233 (to make it easy to understand the current PR)

Affected area:

New Functionality      [X]
Documentation          [ ]
Install                [ ]
Control Plane          [ ]
CLI Tool               [ ]
Certificate Management [ ]
Networking             [ ]
Metrics                [ ]
SMI Policy             [ ]
Security               [ ]
Tests                  [ ]
CI System              [ ]
Performance            [ ]
Other                  [ ]

Please answer the following questions with yes/no.

Does this change contain code from or inspired by another project? If so, did you notify the maintainers and provide attribution? no

Codecov Report

Merging #2233 (562a1cb) into main (a1f0060) will decrease coverage by 1.35%.
The diff coverage is 12.75%.

@@            Coverage Diff             @@
##             main    #2233      +/-   ##
==========================================
- Coverage   58.13%   56.77%   -1.36%
==========================================
  Files         145      147       +2
  Lines        6375     6568     +193
==========================================
+ Hits         3706     3729      +23
- Misses       2664     2834     +170
  Partials        5        5

Impacted Files
Coverage Δ

pkg/injector/envoy_config_health_probes.go
0.00% <0.00%> (ø)

pkg/injector/health_probes.go
5.12% <5.12%> (ø)

pkg/injector/envoy_config.go
89.14% <46.15%> (-4.82%)
:arrow_down:

pkg/injector/envoy_container.go
64.70% <52.00%> (-35.30%)
:arrow_down:

pkg/injector/patch.go
70.00% <100.00%> (+0.76%)
:arrow_up:

pkg/envoy/route/config.go
96.03% <0.00%> (+0.79%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a1f0060...562a1cb. Read the comment docs.

Codecov Report

Merging #2233 (562a1cb) into main (a1f0060) will decrease coverage by 1.35%.
The diff coverage is 12.75%.

@@            Coverage Diff             @@
##             main    #2233      +/-   ##
==========================================
- Coverage   58.13%   56.77%   -1.36%
==========================================
  Files         145      147       +2
  Lines        6375     6568     +193
==========================================
+ Hits         3706     3729      +23
- Misses       2664     2834     +170
  Partials        5        5

Impacted Files
Coverage Δ

pkg/injector/envoy_config_health_probes.go
0.00% <0.00%> (ø)

pkg/injector/health_probes.go
5.12% <5.12%> (ø)

pkg/injector/envoy_config.go
89.14% <46.15%> (-4.82%)
:arrow_down:

pkg/injector/envoy_container.go
64.70% <52.00%> (-35.30%)
:arrow_down:

pkg/injector/patch.go
70.00% <100.00%> (+0.76%)
:arrow_up:

pkg/envoy/route/config.go
96.03% <0.00%> (+0.79%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a1f0060...562a1cb. Read the comment docs.

Tests will be added with a separate PR:  https://github.com/openservicemesh/osm/pull/2269

Tests will be added with a separate PR:  https://github.com/openservicemesh/osm/pull/2269

In a follow-up PR we will convert the map[string]interface{} structures into xDS structs and marshal these into YAML with https://github.com/openservicemesh/osm/issues/2263

In a follow-up PR we will convert the map[string]interface{} structures into xDS structs and marshal these into YAML with https://github.com/openservicemesh/osm/issues/2263


datastore.save transforming JS Array into arrayValue

When saving data to the datastore, it seems that when a Javascript Array is contained inside an object that is a class instance (as opposed of a pure JS object),  the JS Array is being transformed into an 'arrayValue'.
Interestingly, the same does not happen with a pure JS Object. The code below demonstrates this happening and also shows how arrays inside JS Objects remain unchanged.
Sample JS:
'use strict';
const gcloud = require('gcloud');
const config = require('../config/config');

const ds = gcloud.datastore({
  projectId: config.get('GCLOUD_PROJECT')
});

class MyClass {

}

const myInstance = new MyClass();
myInstance.myArray = ['A', 'B', 'C'];

const myObject = {
  myArray: ['A', 'B', 'C', 'D']
}

const data =
  [
    {
      name: 'key1',
      value: 'value1'
    },
    {
      name: 'key2',
      value: ['A', 'B', 'C', 'D']
    },
    {
      name: 'myObject',
      value: myObject
    },
    {
      name: 'myInstance',
      value: myInstance
    }
  ];

const key = ds.key('test');

const entity = {
  key: key,
  data: data
}

console.log('myInstance before save: ', myInstance);
console.log('myObject before save: ', myObject);
ds.save(entity, err => {
  if (err) {
    console.log(err);
  }
  console.log('myInstance after save: ', myInstance, '// arrayValue instead of a regular JS Array');
  console.log('myObject after save: ', myObject, '// Arrays in Objects keep unchanged!');
});

Output:
myInstance before save:  MyClass { myArray: [ 'A', 'B', 'C' ] }
myObject before save:  { myArray: [ 'A', 'B', 'C', 'D' ] }
myInstance after save:  MyClass { myArray: { arrayValue: { values: [Object] } } } // arrayValue instead of a regular JS Array
myObject after save:  { myArray: [ 'A', 'B', 'C', 'D' ] } // Arrays in Objects keep unchanged!

If you are still having issues, please be sure to include as much information as possible:
Environment details

OS: OS X El Capitain (10.11.6 )
Node.js version: v4.2.6
npm version: 2.14.12
google-cloud-node version: ~0.36.0

Steps to reproduce

run the script above

Thanks for reporting! Confirmed and fixed in PR #1552.


Error loading data source package 'Biber::Output::bibtex': Biber/Output/bibtex.pm did not return a true value at (eval 413) line 1.

I used to let biber generate a subset of my BIB file, containing only cited items. This worked flawlessly with biber 2.13 but fails after updating to biber 2.15 (beta). Any help is appreciated!
Problem:

biber 2.15 (beta) (on Windows 10 using MiKTeX)
command: biber --output_file=out.bib --output_format=bibtex document
output:

INFO - This is Biber 2.15 (beta)
INFO - Logfile is 'document.blg'
ERROR - Error loading data source package 'Biber::Output::bibtex': Biber/Output/bibtex.pm did not return a true value at (eval 413) line 1.

INFO - ERRORS: 1

expected output: with biber 2.13, the same command writes only the entries cited in "document.tex" to "out.bib".

Reproducible example:
%document.tex
\documentclass{scrartcl}

\begin{filecontents}{mybib.bib}
@Article{Test2020,
	author       = {Test, The},
	date         = {2020-09-08},
	journaltitle = {Journal of Tests},
	title        = {A Test Article},
}

@Article{Test2020b,
	author       = {Test, The},
	date         = {2020-09-08},
	journaltitle = {Journal of Tests},
	title        = {A Second Test},
}

\end{filecontents}

\usepackage{biblatex}
\addbibresource{mybib.bib}

\begin{document}
	\textcite{Test2020b}
\end{document}

% expected output: out.bib

@ARTICLE{Test2020b,
  AUTHOR = {Test, The},
  DATE = {2020-09-08},
  JOURNALTITLE = {Journal of Tests},
  TITLE = {A Second Test},
}

Is there a reason you are using the beta version of 2.15? 2.15 is released now - can you try to update your installation and also run biber --cache and delete whichever folder this reports?

@plk The Biber installed by MikTeX (at least the 64bit version) identifies as biber version: 2.15 (beta). This doesn't appear to be a MikTeX issue since the Biber from CTAN (https://www.ctan.org/tex-archive/biblio/biber/binaries/Windows) also says '(beta)'. The binary on SF (https://sourceforge.net/projects/biblatex-biber/files/biblatex-biber/current/binaries/Windows/) also has '(beta)'.

@plk As @moewew wrote, biber 2.15 (beta) is what MiKTeX installs. I installed a full version of MiKTeX yesterday and checked for updates today. Yet, the biber version is 2.15 (beta).
I tried deleting the directory reported by biber --cache but this had no effect.

That is very strange - the other binaries appear to be non-beta flagged as they should be. Can you try the 2.15 from sourceforge and just overwrite your current 2.15 binary? That error looks like a corrupted install if the cache clearing didn't fix it.

I just downloaded biber from https://sourceforge.net/projects/biblatex-biber/files/biblatex-biber/current/binaries/Windows/biber-MSWIN64.zip/download . This version is also 2.15 (beta).
Did you mean another version?

No, that's correct for now but I just wanted to see if this version actually works for you without the error.

No, unfortunately it doesn't.
When I download 2.14 the error vanishes – but the (expected) error about the incompatible biblatex version occurs instead.

Can you try getting 2.15 (current folder) from SF again now? I have rebuilt and re-uploaded there. Just make sure that the date on the version you get says something sensible to suggest it was updated in the last few minutes as SF takes a while to replicate round the world.

I downloaded the new version (the last modified date is about 19 minutes in the past), but it still self-identifies as 2.15 (beta) and still throws the same error.

To make sure that all this is not just due to something weird going on with my machine, I just took my spare laptop (with biber 2.13) and updated MiKTeX. Same issue.

Ah, my build farm win10 VM has a corrupt git clone ... looking into it.

Thanks for your effort, plk!

Please get 2.15 again from SF - should just have been updated - it seems my build server had a corrupted git clone and so was building an old, broken version ...

The new version solves the problem – thank you!
However, I noticed another issue that seems to be new in 2.15: As soon as the directory contains a (valid) biber.conf file (the actual content does not seem to matter), the following message is printed:
Log4perl: Seems like no initialization happened. Forgot to call init()?

After this message biber apparently works fine.

Hmm, I can't reproduce this with either 2.15 or 2.16 DEV - where do you see this, in the standard console output?

Here's a reproducible example: I'm using mybib.bib from the initial posting and the following biber.conf:
<?xml version="1.0" encoding="UTF-8" ?>
<config>
	<sourcemap>
		<maps datatype="bibtex" level="user">
			<map map_overwrite="1">
				<map_step map_field_set="author" map_field_value="Cool Guy" />
			</map>
		</maps>
	</sourcemap>
</config>

Then I run biber --tool --output_file=test.bib mybib.bib on the command line. Version 2.15 reports:

>biber --tool --output_file=test.bib mybib.bib
Log4perl: Seems like no initialization happened. Forgot to call init()?
INFO - This is Biber 2.15 running in TOOL mode
INFO - Config file is 'C:\Users\loehne01\TMP_lokal\Paper_CL\biber.conf'
INFO - Logfile is 'mybib.bib.blg'
INFO - Looking for bibtex format file 'mybib.bib'
INFO - LaTeX decoding ...
INFO - Found BibTeX data source 'mybib.bib'
INFO - Overriding locale 'en_US' defaults 'variable = shifted' with 'variable = non-ignorable'
INFO - Overriding locale 'en_US' defaults 'normalization = NFD' with 'normalization = prenormalized'
INFO - Sorting list 'tool/global//global/global' of type 'entry' with template 'tool' and locale 'en_US'
INFO - No sort tailoring available for locale 'en_US'
INFO - Writing 'test.bib' with encoding 'UTF-8'
INFO - Output to test.bib

test.bib is as expected. With 2.14 I get the same output, except for the first line with Log4perl (and two INFOs about "globbing data source").
I'm using an example with tool mode because 2.14 is incompatible with my biblatex version. However, if I just run biber 2.15 on a regular TEX document, I also get the Log4perl message (either on the command line after biber document or in TeXStudio's messages pane).
I would have tested with 2.16 DEV, but could not find it. The "development" version on SF is 2.15 (beta).

Hmm, I still can't reproduce this with 2.15 or 2.16 DEV in tool or normal mode. I tried your biber.conf - also fine. The only thing I haven't tried is your .bib - perhaps you can post a minimal .bib that also shows this? I have updated the development biber folder with 2.16 beta to try too.

Thanks for investigating!
I'm using mybib.bib generated by the filecontents environment in the TEX document I postet initially.
2.16 beta shows the Log4perl  message just like 2.15.
Tomorrow I can test on a different machine – I'm curious to see if I can reproduce the issue.

I tested with the dev binaries for Windows and the Log4perl message appears for me as well. But only in the terminal output and not in the .blg file.

Windows or linux?

I tested on Windows 10 (and I think so did @ClaudiusL if I read the first post correctly).

Sorry, could not make it yesterday.
On another machine (yes: also Windows) I get the same behavior:

2.14 works fine.
2.15 has the Log4perl message in the command line output (but not in the BLG file).
2.16 (beta) behaves as 2.15.

By the way, MiKTeX still serves the broken 2.15 (beta) – would it be possible to update this?

I can reproduce this on Windows but not on OSX - @moewew - can you reproduce this on Linux?

I see the issue, it's a harmless logging issue with the new code for dealing with UTF-8 filenames on windows. Should be fixed in 2.16 DEV now.

I can confirm that 2.16 DEV fixes the message on Windows.
Thanks, plk!


JSBin-like tab management system

Has this not been in the workings for a few months with regards to the multi tab editor? It surely shouldn't be that hard to implement. If @Berkeleytrue is busy I can take this one on!

@atjonathan it will be a huge part of our upcoming UI overhaul. Perhaps we can get you on the phone and show you what we're thinking.


Formfyxer is Back

Reverts the reversion in #577.  Should be merged only after https://github.com/SuffolkLITLab/FormFyxer/pull/44 is merged and published on Pypi.

Should be ready for review now. If anyone wants to try to install this branch.


"Uncaught TypeError: Cannot read property 'exifdata' of null" exif.js:339

I am running into an error in the exif.js:339.  It appears to be kicking back null instead of a boolean response for the 'return !!(img.exifdata);'.  I have tried three different .jpg images and same issue with all of them.

any solution to this I'm experiencing the very same :(


Update etree.py

To conform with the output from UFOnormalizer (and the rest of the XML file).
Along similar lines, I wonder what needs to happen to write a tabbed XML file (instead of two-spaced, as lxml does)?

thanks @frankrolf, however I am inclined to keep it as it is.
We decided to intentionally modify the default XML declaration to use single quotes, even though it would constitute a breaking change from the output of the standalone ufoLib. The reason was to make the output be consistent with that of lxml and the built-in etree library, which both use single quotes in their XML declaration (for historical reason, there is no better or worse, they are both valid).
see https://github.com/fonttools/fonttools/pull/1335#issuecomment-431118568
Same thing about using tabs vs two-space indentation. Lxml doesn't allow to customize the output from its pretty_print. For simplicity I'd like to keep it that way.

Why is lxml consistency more urgent than ufoLib consistency?

speed

the purpose of that fontTools.misc.etree is to provide a compatibility shim for when lxml is not installed, as we didn't feel ready to add a hard requirement on lxml (being a native extension module). So the idea is to keep consistent the output of fontTools.ufoLib when installed with or without lxml. And since the former cannot be customized (and it's good enough, and it's super fast), the latter is made to conform to it.
I hope that makes sense

It's rather unfortunate that everyone who has UFOs under version control will now have to commit this otherwise nonsensical change in hundreds or thousands of files.
It's probably too late to go back now, but really, this can't happen again in the future. Same with the plist indentation change. For users it's a gratuitous change that causes many headaches.

I know..
Anyway, lxml does have support for XSLT transformations, maybe that could be a way for the user to adopt a particular indentation style and the like. I remember @moyogo experimented with something like that

More info at https://lxml.de/xpathxslt.html#xslt


First pass at memoizing Leaf envelope calls

[x] I agree to follow the project's code of conduct.
[ ] I added an entry to rstar/CHANGELOG.md if knowledge of this change could be valuable to users.

While this does seem straight forward to implement, doesn't it have the downside that we keep the cached envelopes around instead of throwing them away after building? There is probably no additional memory consumption in many cases if T is smaller than Vec<RTreeNode<T>> so that RTreeNode does not grow, but if T is larger than Vec<RTreeNode<T>> this could lead to higher long-term memory consumption.
(As a general observation unrelated to the question of caching envelopes, I suspect that storing a single Vec<RTreeNode<T>> and have children: Range<usize> in ParentNode, i.e. tree-wide arena allocation of nodes, could have a positive performance impact.)

This is going to be super-annoying to test because rstar doesn't know what a linestring is. The more I have to actually work with this crate, the more I find that every aspect of its interaction with the geo ecosystem is filled with papercuts at every turn.
We really need to do something here: A fast spatial index is a cornerstone of a massive amount of functionality for geo's algorithms, and every attempt to improve anything is foundering because you first have to re-invent the wheel. I just don't have time to spent two days figuring out how to integrate wkt just so I can dump a few arrays into a data structure.

@urschrei Agreed; it's not the best yet.  I've added some simple polygon based tests here.  Please merge it if good.   The bulk loading was actually not using the RTreeNode optimization we've added, and I've fixed that too.  It gives a decent 10x boost even on polygons with 64 sides.
Bulk load complex geom  time:   [398.98 µs 400.06 µs 401.10 µs]
                        change: [-99.043% -99.040% -99.037%] (p = 0.00 < 0.05)
                        Performance has improved.
Bulk load complex geom  time:   [5.1578 ms 5.1630 ms 5.1721 ms]
                        change: [+1189.8% +1193.3% +1196.6%] (p = 0.00 < 0.05)
                        Performance has regressed.

I do agree with @adamreichold concern about space too.  This will blow up the rtree for the "usual" point case by about 3x.

@rmanoka Amazing, thank you!
Regarding memory use, could we specify three RTreeNode members? That would allow us to choose a memoizing (fast bulk inserts, larger memory use) version or the existing (slower inserts, lower memory use). It would mean some extra complexity as we match on the leaf variant and then choose a bulk load function. Very much thinking out loud here, though…
(I'm inclined to merge as-is soon, given the improvement, but happy for more discussion).

Regarding memory use, could we specify three RTreeNode members? That would allow us to choose a memoizing (fast bulk inserts, larger memory use) version or the existing (slower inserts, lower memory use). It would mean some extra complexity as we match on the leaf variant and then choose a bulk load function. Very much thinking out loud here, though…

Enums have the size of their largest variants plus space for the tag, so without additional boxing this would not help. Also due to the additional complexity, I think we would fare better just doing the work of wiring in an explicit cache like HashMap<*const T, T::Envelope> which can be summarily dropped after building is done.

Enums have the size of their largest variants plus space for the tag, so without additional boxing this would not help

Ugh, completely forgot.

I think we would fare better just doing the work of wiring in an explicit cache like HashMap<*const T, T::Envelope> which can be summarily dropped after building is done.

Where would it "live" though?

Where would it "live" though?

My first try would be on the stack at the top of bulk_load_sequential and passed as &mut HashMap<*const T, T::Envelope> into bulk_load_recursive and from there threaded through PartitioningTask etc.

If my reading of the code is correct, the main change to use an explicit on-stack cache besides threading parameters through would be that Envelope::partition_envelopes needs to access the cache.
Theoretically, we could even pass it remaining: &mut [(T, T::Envelope)] and avoid the hash table entirely at the cost of computing envelopes up front instead of lazily on demand. (I think this would mean that we uselessly compute the envelopes of the leaf nodes. Has anyone any idea how expensive this would be?)

So something like the following diff for eagerly computing the envelopes
diff --git a/rstar/src/aabb.rs b/rstar/src/aabb.rs
index 63f619e..c12bb54 100644
--- a/rstar/src/aabb.rs
+++ b/rstar/src/aabb.rs
@@ -36,7 +36,7 @@ where
     pub fn from_point(p: P) -> Self {
         AABB {
             lower: p.clone(),
-            upper: p.clone(),
+            upper: p,
         }
     }

@@ -206,14 +206,13 @@ where

     fn partition_envelopes<T: RTreeObject<Envelope = Self>>(
         axis: usize,
-        envelopes: &mut [T],
+        envelopes: &mut [(T, Self)],
         selection_size: usize,
     ) {
         envelopes.select_nth_unstable_by(selection_size, |l, r| {
-            l.envelope()
-                .lower
+            l.1.lower
                 .nth(axis)
-                .partial_cmp(&r.envelope().lower.nth(axis))
+                .partial_cmp(&r.1.lower.nth(axis))
                 .unwrap()
         });
     }
diff --git a/rstar/src/algorithm/bulk_load/bulk_load_sequential.rs b/rstar/src/algorithm/bulk_load/bulk_load_sequential.rs
index 886cf8c..b91f1c9 100644
--- a/rstar/src/algorithm/bulk_load/bulk_load_sequential.rs
+++ b/rstar/src/algorithm/bulk_load/bulk_load_sequential.rs
@@ -11,7 +11,7 @@ use num_traits::Float;

 use super::cluster_group_iterator::{calculate_number_of_clusters_on_axis, ClusterGroupIterator};

-fn bulk_load_recursive<T, Params>(elements: Vec<T>, depth: usize) -> ParentNode<T>
+fn bulk_load_recursive<T, Params>(elements: Vec<(T, T::Envelope)>, depth: usize) -> ParentNode<T>
 where
     T: RTreeObject,
     <T::Envelope as Envelope>::Point: Point,
@@ -20,7 +20,10 @@ where
     let m = Params::MAX_SIZE;
     if elements.len() <= m {
         // Reached leaf level
-        let elements: Vec<_> = elements.into_iter().map(RTreeNode::Leaf).collect();
+        let elements: Vec<_> = elements
+            .into_iter()
+            .map(|(element, _envelope)| RTreeNode::Leaf(element))
+            .collect();
         return ParentNode::new_parent(elements);
     }
     let number_of_clusters_on_axis =
@@ -43,7 +46,7 @@ where
 /// A partitioning iterator will take this item from its work queue and start partitioning "elements"
 /// along "current_axis" .
 struct PartitioningState<T: RTreeObject> {
-    elements: Vec<T>,
+    elements: Vec<(T, T::Envelope)>,
     current_axis: usize,
 }

@@ -89,7 +92,7 @@ impl<T: RTreeObject, Params: RTreeParams> Iterator for PartitioningTask<T, Param
 /// A multi dimensional implementation of the OMT bulk loading algorithm.
 ///
 /// See http://ceur-ws.org/Vol-74/files/FORUM_18.pdf
-pub fn bulk_load_sequential<T, Params>(elements: Vec<T>) -> ParentNode<T>
+pub fn bulk_load_sequential<T, Params>(elements: Vec<(T, T::Envelope)>) -> ParentNode<T>
 where
     T: RTreeObject,
     <T::Envelope as Envelope>::Point: Point,
diff --git a/rstar/src/algorithm/bulk_load/cluster_group_iterator.rs b/rstar/src/algorithm/bulk_load/cluster_group_iterator.rs
index 2b60838..089610f 100644
--- a/rstar/src/algorithm/bulk_load/cluster_group_iterator.rs
+++ b/rstar/src/algorithm/bulk_load/cluster_group_iterator.rs
@@ -7,14 +7,14 @@ use num_traits::Float;

 /// Partitions elements into groups of clusters along a specific axis.
 pub struct ClusterGroupIterator<T: RTreeObject> {
-    remaining: Vec<T>,
+    remaining: Vec<(T, T::Envelope)>,
     slab_size: usize,
     pub cluster_dimension: usize,
 }

 impl<T: RTreeObject> ClusterGroupIterator<T> {
     pub fn new(
-        elements: Vec<T>,
+        elements: Vec<(T, T::Envelope)>,
         number_of_clusters_on_axis: usize,
         cluster_dimension: usize,
     ) -> Self {
@@ -28,7 +28,7 @@ impl<T: RTreeObject> ClusterGroupIterator<T> {
 }

 impl<T: RTreeObject> Iterator for ClusterGroupIterator<T> {
-    type Item = Vec<T>;
+    type Item = Vec<(T, T::Envelope)>;

     fn next(&mut self) -> Option<Self::Item> {
         match self.remaining.len() {
@@ -56,7 +56,7 @@ where
     // The depth of the resulting tree, assuming all leaf nodes will be filled up to MAX_SIZE
     let depth = (number_of_elements as f32).log(max_size).ceil() as usize;
     // The number of elements each subtree will hold
-    let n_subtree = (max_size as f32).powi(depth as i32 - 1);
+    let n_subtree = max_size.powi(depth as i32 - 1);
     // How many clusters will this node contain
     let number_of_clusters = (number_of_elements as f32 / n_subtree).ceil();

diff --git a/rstar/src/envelope.rs b/rstar/src/envelope.rs
index 91978f4..cefa493 100644
--- a/rstar/src/envelope.rs
+++ b/rstar/src/envelope.rs
@@ -60,7 +60,7 @@ pub trait Envelope: Clone + PartialEq + ::core::fmt::Debug {
     /// than envelopes[selection_size + 1..].
     fn partition_envelopes<T: RTreeObject<Envelope = Self>>(
         axis: usize,
-        envelopes: &mut [T],
+        envelopes: &mut [(T, Self)],
         selection_size: usize,
     );
 }
diff --git a/rstar/src/rtree.rs b/rstar/src/rtree.rs
index 167f5de..0eaf4b7 100644
--- a/rstar/src/rtree.rs
+++ b/rstar/src/rtree.rs
@@ -235,7 +235,7 @@ where
     ///
     /// For more information refer to [RTree::bulk_load]
     /// and [RTreeParams].
-    pub fn bulk_load_with_params(elements: Vec<T>) -> Self {
+    pub fn bulk_load_with_params<I: IntoIterator<Item = T>>(elements: I) -> Self {
         Self::new_from_bulk_loading(elements, bulk_load::bulk_load_sequential::<_, Params>)
     }

@@ -447,11 +447,18 @@ where
         &mut self.root
     }

-    fn new_from_bulk_loading(
-        elements: Vec<T>,
-        root_loader: impl Fn(Vec<T>) -> ParentNode<T>,
+    fn new_from_bulk_loading<I: IntoIterator<Item = T>>(
+        elements: I,
+        root_loader: impl Fn(Vec<(T, T::Envelope)>) -> ParentNode<T>,
     ) -> Self {
         verify_parameters::<T, Params>();
+        let elements = elements
+            .into_iter()
+            .map(|element| {
+                let envelope = element.envelope();
+                (element, envelope)
+            })
+            .collect::<Vec<_>>();
         let size = elements.len();
         let root = if size == 0 {
             ParentNode::new_root::<Params>()

Doing it lazily using once_cell::unsync::OnceCell also seems reasonably even though I would want to see benchmarks showing that it is actually worth it compared to computing all the envelopes up front which most likely has much better cache locality and could also be trivially parallelized:

Full diff
diff --git a/rstar/Cargo.toml b/rstar/Cargo.toml
index 65b235d..9fccd4c 100644
--- a/rstar/Cargo.toml
+++ b/rstar/Cargo.toml
@@ -18,6 +18,7 @@ maintenance = { status = "actively-developed" }
 [dependencies]
 heapless = "0.7.10"
 num-traits = { version = "0.2", default-features = false, features = ["libm"] }
+once_cell = { version = "1.17.1", default-features = false }
 serde = { version = "1.0", optional = true, default-features = false, features = ["alloc", "derive"] }
 smallvec = "1.6"

diff --git a/rstar/src/aabb.rs b/rstar/src/aabb.rs
index 63f619e..8fd73c9 100644
--- a/rstar/src/aabb.rs
+++ b/rstar/src/aabb.rs
@@ -1,6 +1,7 @@
 use crate::point::{max_inline, Point, PointExt};
 use crate::{Envelope, RTreeObject};
 use num_traits::{Bounded, One, Zero};
+use once_cell::unsync::OnceCell;

 #[cfg(feature = "serde")]
 use serde::{Deserialize, Serialize};
@@ -36,7 +37,7 @@ where
     pub fn from_point(p: P) -> Self {
         AABB {
             lower: p.clone(),
-            upper: p.clone(),
+            upper: p,
         }
     }

@@ -206,15 +207,14 @@ where

     fn partition_envelopes<T: RTreeObject<Envelope = Self>>(
         axis: usize,
-        envelopes: &mut [T],
+        envelopes: &mut [(T, OnceCell<Self>)],
         selection_size: usize,
     ) {
         envelopes.select_nth_unstable_by(selection_size, |l, r| {
-            l.envelope()
-                .lower
-                .nth(axis)
-                .partial_cmp(&r.envelope().lower.nth(axis))
-                .unwrap()
+            let l = l.1.get_or_init(|| l.0.envelope());
+            let r = r.1.get_or_init(|| r.0.envelope());
+
+            l.lower.nth(axis).partial_cmp(&r.lower.nth(axis)).unwrap()
         });
     }
 }
diff --git a/rstar/src/algorithm/bulk_load/bulk_load_sequential.rs b/rstar/src/algorithm/bulk_load/bulk_load_sequential.rs
index 886cf8c..371ac7c 100644
--- a/rstar/src/algorithm/bulk_load/bulk_load_sequential.rs
+++ b/rstar/src/algorithm/bulk_load/bulk_load_sequential.rs
@@ -5,13 +5,17 @@ use crate::params::RTreeParams;
 use crate::point::Point;

 use alloc::{vec, vec::Vec};
+use once_cell::unsync::OnceCell;

 #[allow(unused_imports)] // Import is required when building without std
 use num_traits::Float;

 use super::cluster_group_iterator::{calculate_number_of_clusters_on_axis, ClusterGroupIterator};

-fn bulk_load_recursive<T, Params>(elements: Vec<T>, depth: usize) -> ParentNode<T>
+fn bulk_load_recursive<T, Params>(
+    elements: Vec<(T, OnceCell<T::Envelope>)>,
+    depth: usize,
+) -> ParentNode<T>
 where
     T: RTreeObject,
     <T::Envelope as Envelope>::Point: Point,
@@ -20,7 +24,10 @@ where
     let m = Params::MAX_SIZE;
     if elements.len() <= m {
         // Reached leaf level
-        let elements: Vec<_> = elements.into_iter().map(RTreeNode::Leaf).collect();
+        let elements: Vec<_> = elements
+            .into_iter()
+            .map(|(element, _envelope)| RTreeNode::Leaf(element))
+            .collect();
         return ParentNode::new_parent(elements);
     }
     let number_of_clusters_on_axis =
@@ -43,7 +50,7 @@ where
 /// A partitioning iterator will take this item from its work queue and start partitioning "elements"
 /// along "current_axis" .
 struct PartitioningState<T: RTreeObject> {
-    elements: Vec<T>,
+    elements: Vec<(T, OnceCell<T::Envelope>)>,
     current_axis: usize,
 }

@@ -89,7 +96,7 @@ impl<T: RTreeObject, Params: RTreeParams> Iterator for PartitioningTask<T, Param
 /// A multi dimensional implementation of the OMT bulk loading algorithm.
 ///
 /// See http://ceur-ws.org/Vol-74/files/FORUM_18.pdf
-pub fn bulk_load_sequential<T, Params>(elements: Vec<T>) -> ParentNode<T>
+pub fn bulk_load_sequential<T, Params>(elements: Vec<(T, OnceCell<T::Envelope>)>) -> ParentNode<T>
 where
     T: RTreeObject,
     <T::Envelope as Envelope>::Point: Point,
diff --git a/rstar/src/algorithm/bulk_load/cluster_group_iterator.rs b/rstar/src/algorithm/bulk_load/cluster_group_iterator.rs
index 2b60838..a930b47 100644
--- a/rstar/src/algorithm/bulk_load/cluster_group_iterator.rs
+++ b/rstar/src/algorithm/bulk_load/cluster_group_iterator.rs
@@ -1,20 +1,21 @@
 use crate::{Envelope, Point, RTreeObject, RTreeParams};

 use alloc::vec::Vec;
+use once_cell::unsync::OnceCell;

 #[allow(unused_imports)] // Import is required when building without std
 use num_traits::Float;

 /// Partitions elements into groups of clusters along a specific axis.
 pub struct ClusterGroupIterator<T: RTreeObject> {
-    remaining: Vec<T>,
+    remaining: Vec<(T, OnceCell<T::Envelope>)>,
     slab_size: usize,
     pub cluster_dimension: usize,
 }

 impl<T: RTreeObject> ClusterGroupIterator<T> {
     pub fn new(
-        elements: Vec<T>,
+        elements: Vec<(T, OnceCell<T::Envelope>)>,
         number_of_clusters_on_axis: usize,
         cluster_dimension: usize,
     ) -> Self {
@@ -28,7 +29,7 @@ impl<T: RTreeObject> ClusterGroupIterator<T> {
 }

 impl<T: RTreeObject> Iterator for ClusterGroupIterator<T> {
-    type Item = Vec<T>;
+    type Item = Vec<(T, OnceCell<T::Envelope>)>;

     fn next(&mut self) -> Option<Self::Item> {
         match self.remaining.len() {
@@ -56,7 +57,7 @@ where
     // The depth of the resulting tree, assuming all leaf nodes will be filled up to MAX_SIZE
     let depth = (number_of_elements as f32).log(max_size).ceil() as usize;
     // The number of elements each subtree will hold
-    let n_subtree = (max_size as f32).powi(depth as i32 - 1);
+    let n_subtree = max_size.powi(depth as i32 - 1);
     // How many clusters will this node contain
     let number_of_clusters = (number_of_elements as f32 / n_subtree).ceil();

diff --git a/rstar/src/envelope.rs b/rstar/src/envelope.rs
index 91978f4..df5e493 100644
--- a/rstar/src/envelope.rs
+++ b/rstar/src/envelope.rs
@@ -1,4 +1,5 @@
 use crate::{Point, RTreeObject};
+use once_cell::unsync::OnceCell;

 /// An envelope type that encompasses some child nodes.
 ///
@@ -60,7 +61,7 @@ pub trait Envelope: Clone + PartialEq + ::core::fmt::Debug {
     /// than envelopes[selection_size + 1..].
     fn partition_envelopes<T: RTreeObject<Envelope = Self>>(
         axis: usize,
-        envelopes: &mut [T],
+        envelopes: &mut [(T, OnceCell<Self>)],
         selection_size: usize,
     );
 }
diff --git a/rstar/src/rtree.rs b/rstar/src/rtree.rs
index 167f5de..651b8b4 100644
--- a/rstar/src/rtree.rs
+++ b/rstar/src/rtree.rs
@@ -14,6 +14,7 @@ use crate::params::{verify_parameters, DefaultParams, InsertionStrategy, RTreePa
 use crate::Point;

 use alloc::vec::Vec;
+use once_cell::unsync::OnceCell;

 #[cfg(feature = "serde")]
 use serde::{Deserialize, Serialize};
@@ -235,7 +236,7 @@ where
     ///
     /// For more information refer to [RTree::bulk_load]
     /// and [RTreeParams].
-    pub fn bulk_load_with_params(elements: Vec<T>) -> Self {
+    pub fn bulk_load_with_params<I: IntoIterator<Item = T>>(elements: I) -> Self {
         Self::new_from_bulk_loading(elements, bulk_load::bulk_load_sequential::<_, Params>)
     }

@@ -447,11 +448,15 @@ where
         &mut self.root
     }

-    fn new_from_bulk_loading(
-        elements: Vec<T>,
-        root_loader: impl Fn(Vec<T>) -> ParentNode<T>,
+    fn new_from_bulk_loading<I: IntoIterator<Item = T>>(
+        elements: I,
+        root_loader: impl Fn(Vec<(T, OnceCell<T::Envelope>)>) -> ParentNode<T>,
     ) -> Self {
         verify_parameters::<T, Params>();
+        let elements = elements
+            .into_iter()
+            .map(|element| (element, OnceCell::new()))
+            .collect::<Vec<_>>();
         let size = elements.len();
         let root = if size == 0 {
             ParentNode::new_root::<Params>()

I guess the call to envelope_for_children in ParentNode::new_parent should also make use of the cached envelopes if out-of-line storage is used. I think this suggests that the lazy variant would not be useful after all, since this would access exactly those envelopes which might not have been accessed during selection.

I guess the call to envelope_for_children in ParentNode::new_parent should also make use of the cached envelopes if out-of-line storage is used. I think this suggests that the lazy variant would not be useful after all, since this would access exactly those envelopes which might not have been accessed during selection.

I've just finished applying your first diff, and the timing vs @rmanoka's changes (baseline is 873.54 µs) is not encouraging so far:
Bulk load complex geom  time:   [148.10 ms 148.51 ms 148.91 ms]
                        change: [+15882% +16210% +16515%] (p = 0.00 < 0.05)
                        Performance has regressed.

I've opened a draft PR with your initial suggested changes and the new benchmark: https://github.com/georust/rstar/pull/117
If you check out that branch or PR you can push directly to it – I'm not sure where you'd like the new_parent change to go.

I've just finished applying your first diff, and the timing vs @rmanoka's changes (baseline is 873.54 µs) is not encouraging so far:

Note that the branch you pushed uses the original size of 4096 whereas this branch here uses 64. If I uses 4069 in both cases, I get
Bulk load complex geom  time:   [27.620 ms 27.653 ms 27.689 ms]

for this branch and the eager out-of-line computation
Bulk load complex geom  time:   [36.660 ms 36.819 ms 37.012 ms]

which is still worse, but more reasonably close.

I was just wondering whether there was an obvious blunder somewhere, given the magnitude of the difference.

If you check out that branch or PR you can push directly to it – I'm not sure where you'd like the new_parent change to go.

I don't think I can push there because I cannot push to this repository at all, but it isn't really necessary I think.
The diff for using the pre-computed envelopes when combing the leaf nodes into a parent node is just
diff --git a/rstar/src/algorithm/bulk_load/bulk_load_sequential.rs b/rstar/src/algorithm/bulk_load/bulk_load_sequential.rs
index 6b9aa63..bc7fbcb 100644
--- a/rstar/src/algorithm/bulk_load/bulk_load_sequential.rs
+++ b/rstar/src/algorithm/bulk_load/bulk_load_sequential.rs
@@ -20,11 +20,20 @@ where
     let m = Params::MAX_SIZE;
     if elements.len() <= m {
         // Reached leaf level
-        let elements: Vec<_> = elements
+        let envelope = elements.iter().fold(
+            T::Envelope::new_empty(),
+            |mut envelope, (_element, envelope1)| {
+                envelope.merge(envelope1);
+                envelope
+            },
+        );
+
+        let children: Vec<_> = elements
             .into_iter()
             .map(|(element, _envelope)| RTreeNode::Leaf(element))
             .collect();
-        return ParentNode::new_parent(elements);
+
+        return ParentNode { children, envelope };
     }
     let number_of_clusters_on_axis =
         calculate_number_of_clusters_on_axis::<T, Params>(elements.len());

which for me brings the benchmark down to
Bulk load complex geom  time:   [23.106 ms 23.137 ms 23.177 ms]

which looks like the best result so far.
(The other call to ParentNode::new_parent does not use the pre-computed envelopes, but I don't think this is an issue as it will produce parents-of-parents and those already internally "cache" their envelope.)

I am sorry but the numbers above are also incorrect as I only switch the CPU frequency governor but forgot to disable frequency boost. Doing that yields
Bulk load complex geom  time:   [28.556 ms 28.605 ms 28.659 ms]

using the eager computation for me which is slightly worse than the approach presented here, but comes without the indefinite increase in memory usage.

With the latest changes in #117 I'm now getting
Bulk load complex geom  time:   [121.15 ms 121.52 ms 121.92 ms]
                        change: [-0.2388% +0.2131% +0.6614%] (p = 0.35 > 0.05)
                        No change in performance detected.

Which is very encouraging.

Note that
diff --git a/rstar-benches/benches/benchmarks.rs b/rstar-benches/benches/benchmarks.rs
index 7de1601..5a3ec16 100644
--- a/rstar-benches/benches/benchmarks.rs
+++ b/rstar-benches/benches/benchmarks.rs
@@ -58,7 +58,7 @@ fn bulk_load_complex_geom(c: &mut Criterion) {
         let polys: Vec<_> = create_random_polygons(DEFAULT_BENCHMARK_TREE_SIZE, 4096, SEED_1);

         b.iter(|| {
-            RTree::<Polygon<f64>, Params>::bulk_load_with_params(polys.clone());
+            RTree::<Polygon<f64>, Params>::bulk_load_with_params(polys.iter().cloned());
         });
     });
 }

reduces the runtime further for me to
Bulk load complex geom  time:   [24.518 ms 24.567 ms 24.625 ms]

by avoiding to create the intermediate Vec, but of course, downstream code would need to make the same changes to benefit if we go for the IntoIterator-based API.

OK, we're currently still neck-and-neck on the new bulk load benchmark, but with lower memory usage in #117.

Here's the full suite:  #117 vs master
     Running benches/benchmarks.rs (target/release/deps/benchmarks-f2d29554c7e4f7dd)
Bulk load baseline      time:   [172.08 µs 173.89 µs 176.19 µs]
                        change: [+6.0470% +6.9826% +7.8936%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 3 outliers among 100 measurements (3.00%)
  2 (2.00%) high mild
  1 (1.00%) high severe

rstar and spade benchmarks/rstar sequential
                        time:   [1.8192 ms 2.0567 ms 2.3794 ms]
                        change: [+17.971% +27.835% +41.962%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 11 outliers among 100 measurements (11.00%)
  5 (5.00%) high mild
  6 (6.00%) high severe

Bulk load complex geom  time:   [119.35 ms 120.23 ms 121.23 ms]
                        change: [-82.387% -82.243% -82.102%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
  2 (2.00%) high mild
  1 (1.00%) high severe

bulk load quality       time:   [169.08 µs 170.87 µs 173.31 µs]
                        change: [-15.622% -8.0801% -1.2220%] (p = 0.03 < 0.05)
                        Performance has improved.
Found 4 outliers among 100 measurements (4.00%)
  1 (1.00%) high mild
  3 (3.00%) high severe

sequential load quality time:   [219.94 µs 221.89 µs 223.86 µs]
                        change: [-6.5737% -0.3851% +4.7198%] (p = 0.91 > 0.05)
                        No change in performance detected.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild

locate_at_point (successful)
                        time:   [341.55 ns 346.39 ns 353.21 ns]
                        change: [-2.8073% -1.4548% -0.1566%] (p = 0.03 < 0.05)
                        Change within noise threshold.
Found 4 outliers among 100 measurements (4.00%)
  2 (2.00%) high mild
  2 (2.00%) high severe

locate_at_point (unsuccessful)
                        time:   [450.39 ns 471.72 ns 506.93 ns]
                        change: [+3.1001% +6.1859% +9.5738%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 12 outliers among 100 measurements (12.00%)
  5 (5.00%) high mild
  7 (7.00%) high severe

Here's the full suite: https://github.com/georust/rstar/pull/117 vs master

Not sure if it explains all of the regression, but I think "Bulk load baseline" would also benefit from replacing points.clone() by points.iter().cloned(). For anything but "Bulk load complex geom", I do not understand how the changes from #117 would improve or regress performance...

I prefer this PR to #117 because this will also ensure memoizing during query and not just bulk insertion.  Isn't that a better approach in general for the r-tree?  I suppose complex geoms in a r-tree is a fairly typical use-case too.  For instance, we do this in the geo crate.
I wonder if we can add associated types to RTreeObject trait to handle the memory problem.  So we add default cache types that default to envelope type, but for point types, we could just make the  envelope type as ()

Coming to think of it, I also wouldn't mind simply informing the user very clearly that the r-tree is not even looking at anything other than the envelope of the geometry, and it is probably useless to store complex geoms in the r-tree except as just data.  They can use GeomWithData<AABB, GeomType> where it is clear that the actual geometry is just a payload, and the r-tree just handles the envelope.

I prefer this PR to https://github.com/georust/rstar/pull/117 because this will also ensure memoizing during query and not just bulk insertion. Isn't that a better approach in general for the r-tree?

Also though about this, but I think the important envelopes (those of the parents) are already cached during querying in the current design. The problem really is bulk insertion where they are not available as we do not know which elements will becomes parents or leafs yet.
If the user really wants the envelope to be cached all the time, they can already do that inside of their type implementing RTreeObject. The bulk insertion performance without caching was just a much larger performance foot gun than querying due to the above.

Coming to think of it, I also wouldn't mind simply informing the user very clearly that the r-tree is not even looking at anything other than the envelope of the geometry, and it is probably useless to store complex geoms in the r-tree except as just data. They can use GeomWithData<AABB, GeomType> where it is clear that the actual geometry is just a payload, and the r-tree just handles the envelope.

I think this is slightly too simplified: For parents we really look only at envelopes (as they obviously have no other geometry of their own) and their it is also cached. But for the leafs, some operations do look at the individual geometry, like the nearest neighbour search via the PointDistance trait, e.g.
https://github.com/georust/rstar/blob/101773a596f5e6827466772f8cce883d8067fe2a/rstar/src/algorithm/nearest_neighbor.rs#L66-L69
If we really want envelopes to be cached all the time, but without any memory overhead if it is not necessary, I think we could also change the RTreeObject trait from
fn envelope(&self) -> Self::Envelope;

to
fn envelope(&self) -> &Self::Envelope;

which is basically impossible to implement without caching, but trivial to implement if the geometry type basically is AABB.
But as written above, I find the current trade-off (caching only parent envelopes and leaving leaf envelopes to the implementor) preferable if we fix the bulk insertion performance foot gun, i.e. I would argue for #117.

Note that both the PRs add a 5-10% regression to the points case.  I'm thinking we should aid with ergo. wrappers like GeomWithData that cache explicitly rather than incur unnecessary extra regression for the most typical cases.
For larger redesign, I quite like the returning ref. idea:
fn envelope(&self) -> &Self::Envelope;

Definitely worth evaluating this more (separately).

So something like
struct WithCachedEnvelope<O: RTreeObject> {
  object: O,
  envelope: O::Envelope,
}

impl<O: RTreeObject> WithCachedEnvelope {
  fn new(object: O) -> Self {
    let envelope = object.envelope();

    Self { object, envelope }
}

impl<O: RTreeObject> RTreeObject for WithCachedEnvelope where O::Envelope: Clone {
  type Envelope = O::Envelope;

  fn envelope(&self) -> Self::Envelope {
    self.envelope.clone()
  }
}

? I think this would be nicely composeable and would support such a change as well.
I am somewhat surprised by the statement

for the most typical cases.

though. Wouldn't one usually use the simpler k-d trees for point-like data and turn to R* trees only for more complicated geometry?

I am somewhat surprised by the statement
for the most typical cases.
though. Wouldn't one usually use the simpler k-d trees for point-like data and turn to R* trees only for more complicated geometry?

My experience (which may not be universal) is that in GIS / most OGC-adjacent geo applications there isn't much distinction made between types of spatial index; there's just the spatial index library that the ecosystem provides, plus PostGIS (which I believe allows more granular control over indices, but in general isn't customised much).

My experience (which may not be universal) is that in GIS / most OGC-adjacent geo applications there isn't much distinction made between types of spatial index; there's just the spatial index library that the ecosystem provides, plus PostGIS (which I believe allows more granular control over indices, but in general isn't customised much).

Thank you for explanation, this does explain my surprise: My initial contact with spatial indexes was for simulations where I think it is not untypical to use different kinds of indexes for different kinds of geometry and/or different computational tasks. I guess being part of the georust organisation, providing a robust default choice is indeed a high priority requirement for this crate.

Yes, but I'm also conscious of the fact that our use case isn't a monopoly or even necessarily the most important. We maintain the crate, but I don't want to privilege our use cases over others, so I think we're trying to figure out how to accommodate everyone while balancing the relatively small amount of dev resources we have.

Closing since we went for #118 instead.
@urschrei Could you delete branch if you do not need it anymore?


Add support for the System.Version class

A (small) vote for supporting the Version class. I'm trying to build a Bridge/ASP.NET WebAPI replacement for PoSh deployment scripts, and there's lots of version numbers flying around. Being able to handle Versions in the client properly would be handy (eg. knowing which versions are newer/older).
Not a big deal if you think this is out of scope; but thought it was worth asking :)

Can you please clarify how you would use Version on the client side. Is it just an instance to store Major, Minor etc properties?

That, and also sorting/comparing. Eg. being able to check:
// Comparing versions
if (oldVer < newVer)

// Sorting
myReleases.OrderBy(r => r.Version)

I remember there being something slightly strange about comparison (eg. if you create new Version(1, 1, 1) vs new Version(1, 1, 1, 0)).. Looking at the source I think it's just that build/revision are treated as -1 if not specified.

I guess Version.Parse would be handy too, though I'm not sure anything of mine specifically needs it (since the JSON serialised versions come back as an object with the 4 properties, so I could just use the constructor during deserialisation).

Hi @DanTup!
We implemented the Version class. The implementation was to take .Net source code, translate it with Bridge, ammend the js output a little, write some tests and that's it :)
Please check and let us know it works as expected for your use case.


Very problematic table

First of all, thanks for developing Tabula! it is a fantastic help in my project.
I am extracting large scale data from old scientific papers for ecological studies as part of my research project.
From all the tables I tried to work out, this is by far the most tricky (see attachment). Do you know any suggestion to make it work??
Thanks a lot in advance!!
Alejandro
trial.pdf

You might try rotating the pages of the PDF (so that the table contents are oriented normally, rather than being flipped vertically) in another tool, then uploading that modified PDF to Tabula. Tabula doesn't do well with rotated text.


#8 Map::keys() and Map::into_keys()

@rultor merge

@rultor merge

@yegor256 OK, I'll try to merge now. You can check the progress of the merge here

@Zefick thanks for your contribution!


sum of odd and even number code in python

here i implemented the python code for sum of odd and even numbers .

output:


Update azblob with the latest azcore

This tactially resolves the small number of breaking changes in azcore.

[ ] The purpose of this PR is explained in this or a referenced issue.
[ ] The PR does not update generated files.

These files are managed by the codegen framework at Azure/autorest.go.

[ ] Tests are included and/or updated for code changes.
[ ] Updates to CHANGELOG.md are included.
[ ] MIT license headers are included in each file.

dev-azblob has changes which we need to include in the upcoming release i.e. v0.3.0. Hence changes the destination branch.


[Excel] (custom functions) debugging with manifest-only option

Fixes 3229

Docs Build status updates of commit 6fcc70d:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/excel/custom-functions-debugging.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


[REEF-569] Add RAT excludes for JVM error logs and Mesos runtime

hs_error*.log that JVM can produce on fatal errors (e.g, REEF-89).
directory that mesos runtime creates, during tests.

JIRA:
REEF-569
Pull Request:
This closes #

Is it possible to move these created files into a target folder? That way our builds are easier to understand.

Last I checked its difficult to force JVM to output fatal error logs to a particular location.
On Thu, Aug 6, 2015 at 8:18 PM -0700, "Markus Weimer"<EMAIL_ADDRESS>wrote:
Is it possible to move these created files into a target folder? That way our builds are easier to understand.
—
Reply to this email directly or view it on GitHub.

Makes sense.


Manage org-wide app settings for Microsoft 365 Government

Manage org-wide app settings for Microsoft 365 Government
please review this steps
There is no "Org-wide app settings" in "Permissions policy"

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: b51eda6c-168c-7d99-7cc1-992b3d8feef2
Version Independent ID: 118a1b8f-f4b7-a778-70b1-d3161fc548ab
Content: Manage app permission policies in Microsoft Teams - Microsoft Teams
Content Source: Teams/teams-app-permission-policies.md
Service: msteams
GitHub Login: @cichur
Microsoft Alias: v-cichur

Thank you for submitting feedback and contributing to the docs. We are currently investigating this.

Hello @RascoPK how are you?
Can you kindly confirm if you instead find Org-wide app settings going to Teams Apps - Manage Apps in your tenant?
Thanks

Sorry, I cant find it now. When I was looking at this in the first time I was able to see this bug.
Sorry for confusion.


Providing .js extension for angular config file not just angular.json

🚀 Feature request
Description
Take a look Semantic Release Config
Accepting js extension can provide more flexible and deduplicated configuration.
Describe the solution you'd like
Accepts angular.js, angular.json
Describe alternatives you've considered
No

Related to https://github.com/angular/angular-cli/issues/12943, please see @clydin comment https://github.com/angular/angular-cli/issues/12943#issuecomment-634746071


EKF2: Check published local position in altitude-only fix mode

The local position set points in alt mode are really large. Need to clarify why. Harmless in flight, but needs inspection.
http://logs.uaventure.com/view/YMZnYaFAvK5icjSNzwHT5A

Defaulting numbers to zero now until position is valid.


Support new device in Dyson component

The problem

I am trying to use Hass with my Dyson Pure Hot+Cool (note that it is not a Pure Hot+Cool Link). The device is detected correctly by the Dyson component but the climate settings do not appear. This is because components/dyson/climate.py is written to support only the libpurecool.dyson_pure_hotcool_link and not libpurecool.dyson_pure_hotcool - this is an error.
Environment

Home Assistant release with the issue: 0.106.0
Last working Home Assistant release (if known): N/A
Operating environment (Hass.io/Docker/Windows/etc.): Docker
Integration causing this issue: Dyson
Link to integration documentation on our website: https://www.home-assistant.io/integrations/dyson/

Problem-relevant configuration.yaml

# Configure a default setup of Home Assistant (frontend, api, etc)
default_config:

# Uncomment this if you are using SSL/TLS, running in Docker container, etc.
http:
  base_url: hass.example.com

# Dyson
dyson:
  username: !secret dyson_username
  password: !secret dyson_password
  language: GB
  devices:
    - device_id: ZZ9-UK-MGA9998X
      device_ip: "<IP_ADDRESS>"
    - device_id: ZZ9-UK-MGA9999X
      device_ip: "<IP_ADDRESS>"

group: !include groups.yaml
automation: !include automations.yaml
script: !include scripts.yaml
scene: !include scenes.yaml

Traceback/Error logs

2020-02-27 22:43:36 INFO (SyncWorker_10) [homeassistant.components.dyson] Connected to Dyson account
2020-02-27 22:43:36 INFO (SyncWorker_19) [pyicloud.base] Authentication completed successfully
2020-02-27 22:43:38 INFO (SyncWorker_10) [homeassistant.components.dyson] Connected to device DysonPureHotCool(serial=ZZ9-UK-MGA9998X,active=None,name=Master bedroom,version=ECG2PF.<IP_ADDRESS>3,auto_update=True,new_version_available=True,product_type=527,network_device=NetworkDevice(name=Master bedroom,address=<IP_ADDRESS>,port=1883))
2020-02-27 22:43:39 INFO (MainThread) [homeassistant.components.device_tracker] Setting up device_tracker.icloud
2020-02-27 22:43:39 INFO (SyncWorker_10) [homeassistant.components.dyson] Connected to device DysonPureHotCool(serial=ZZ9-UK-MGA9999X,active=None,name=Living room,version=ECG2PF.<IP_ADDRESS>3,auto_update=True,new_version_available=True,product_type=527,network_device=NetworkDevice(name=Living room,address=<IP_ADDRESS>,port=1883))
2020-02-27 22:43:39 INFO (MainThread) [homeassistant.setup] Setup of domain dyson took 3.5 seconds.
2020-02-27 22:43:39 INFO (SyncWorker_14) [homeassistant.loader] Loaded fan from homeassistant.components.fan
2020-02-27 22:43:39 INFO (SyncWorker_7) [homeassistant.loader] Loaded air_quality from homeassistant.components.air_quality
2020-02-27 22:43:39 INFO (SyncWorker_36) [homeassistant.loader] Loaded vacuum from homeassistant.components.vacuum
2020-02-27 22:43:39 INFO (SyncWorker_29) [homeassistant.loader] Loaded climate from homeassistant.components.climate
2020-02-27 22:43:39 INFO (MainThread) [homeassistant.setup] Setting up air_quality
2020-02-27 22:43:39 INFO (MainThread) [homeassistant.setup] Setup of domain air_quality took 0.0 seconds.
2020-02-27 22:43:39 INFO (MainThread) [homeassistant.setup] Setting up climate
2020-02-27 22:43:39 INFO (MainThread) [homeassistant.setup] Setup of domain climate took 0.0 seconds.
2020-02-27 22:43:39 INFO (MainThread) [homeassistant.setup] Setting up fan
2020-02-27 22:43:39 INFO (MainThread) [homeassistant.setup] Setup of domain fan took 0.0 seconds.
2020-02-27 22:43:39 INFO (MainThread) [homeassistant.components.air_quality] Setting up air_quality.dyson
2020-02-27 22:43:39 INFO (MainThread) [homeassistant.setup] Setting up vacuum
2020-02-27 22:43:39 INFO (MainThread) [homeassistant.setup] Setup of domain vacuum took 0.0 seconds.
2020-02-27 22:43:39 INFO (MainThread) [homeassistant.components.climate] Setting up climate.dyson
2020-02-27 22:43:39 INFO (MainThread) [homeassistant.components.fan] Setting up fan.dyson
2020-02-27 22:43:39 INFO (MainThread) [homeassistant.components.vacuum] Setting up vacuum.dyson

Additional information
As you can see from the trace, the device is correctly detected (by libpurecool) as a DysonPureHotCool - climate.py just doesn't know how to handle it.

Per above PR - @etheralm has a fixed version of this in their fork and will work to merge next week. Closing this to prevent stale issues.

You can keep this issue open if you want, I'll link the PR to it when I submit it.
There's been some discussion in the issue opened in the upstream library here: https://github.com/etheralm/libpurecool/issues/4,
but having more visibility on the progress would be good.


SignUp Button

I fixed the button and the PR was merged, Here is the link- https://github.com/Susmita-Dey/Sukoon/pull/240
The current deployed website doesn't have these changes.
Should I raise a PR again?

@Vanshiii203 We have 2 types of login signup html files. Someone needs to delte the old ones.

@Vanshiii203 We have 2 types of login signup html files. Someone needs to delte the old ones.
I was talking about-

For that login signup file, I have linked the new file in my previous PR which is currently not merged

@Vanshiii203 Make the changes in that PR first and go with this. Also, after deleting the old files, rename the new ones as login.html and signup.html


Offer of Faction Symbols, and Other Vector Work

Friends,
I enjoy the fruits of your labor, and want to contribute.  I'm not a coder/font designer, but can do vector-file gruntwork.
I noticed your font doesn't have symbols for the Galactic Republic or the Separatists, and that the Empire Symbol and Scum symbols are a little wonky.
I have vector files for all these, created by me, that I'd be willing to donate.  Oh yeah, and First Order too.
So please let me know what format you'd like the files in, what scale, etc.
Thanks!
Ned Ash

SVG works best pulling into my font app. I can work with EPS as well.    zip and upload them here or email them to<EMAIL_ADDRESS>and I'll see how they look compared to the current glyphs


Created USSurveyFoot unit

Equal to exactly 1200/3937 meters, a US Survey foot differentiates from the International foot because of existing survey data based on an old definition from 1893.
1 ft = 0.999998000004 ft (survey)
The difference is important in large scale mapping applications.
https://en.wikipedia.org/wiki/Foot_(unit)#Survey_foot

@anjdreas  This looks good to me, but would you please double check?

@JKSnd Found a couple of things, see my review. It's particularly important to verify the test values are correct. It was not in this case and would give wrong values in conversions, even if all your tests are green.
Most often the problem is that the conversion functions are defined "backwards", as in this case, and should be swapped. Maybe the naming could be improved to avoid that, not sure.
Pro tip: Often you can just copy and paste the property name USSurveyFeetInOneMeter into google and it will give you the conversion you need, but if not I usually go find an online converter.

@anjdreas please remove my recent changes.

@anjdreas thanks! I'll definitely do a more thorough job of checking conversion factors in the future. I foolishly assumed that since the tests were passing that all was ok. Bad, bad me.

Oops!  I meant to write "review" my changes and not "remove".  I commit changes that incorporated most of his points and on others I need responses.  With respect to using fractions vs. decimal, I read in the instructions that "10 significant digits" was plenty.  It might help to add a comment that using fractions for numbers is preferable whenever possible (especially in the case of irrational numbers).

Oh God, I think my brain farted on the last review. @mludlum you are absolutely correct that it was me who got it backwards. Sorry!

Gentlemen, let me know what I need to do to keep this moving forward.  Thanks.

Requested changes finished.

Looks good to me, @JKSnd ?

I'm good with it. I'll do a release.

I'm good with it, I'll handle the release on this one.


Feature request: more info in list command

fholo list displays the x/y/z coordinates of each hologram
> fholo list
[11:22:06 INFO]: List of holograms:
[11:22:06 INFO]: Page 1/3
[11:22:06 INFO]:  - lobby-yellowdog-block (130.62/5.4/134.12)
[11:22:06 INFO]:  - lobby-city (134/4.5/142.25)
[11:22:06 INFO]:  - lobby-skyblock (142.25/4.5/139)
[11:22:06 INFO]:  - lobby-playground-block (133.88/5.4/130.62)
[11:22:06 INFO]:  - lobby-yellowdog (130.75/4.5/134)
[11:22:06 INFO]:  - playground-spawn (0/66.75/-2)
[11:22:06 INFO]:  - lobby-skyblock-block (142.38/5.4/138.88)
[11:22:06 INFO]:  - lobby-westeros (139/4.5/142.25)
[11:22:06 INFO]:  - lobby-luckyblock-block (142.38/5.4/133.88)
[11:22:06 INFO]:  - lobby-westeros-block (139.12/5.4/142.38)

It would be nice to add the world name, something like
> fholo list
[11:22:06 INFO]: List of holograms:
[11:22:06 INFO]: Page 1/3
[11:22:06 INFO]:  - lobby-yellowdog-block (130.62/5.4/134.12) in world_lobby
[11:22:06 INFO]:  - lobby-city (134/4.5/142.25) in world_lobby
[11:22:06 INFO]:  - lobby-skyblock (142.25/4.5/139) in world_lobby
[11:22:06 INFO]:  - lobby-playground-block (133.88/5.4/130.62) in world_lobby
[11:22:06 INFO]:  - lobby-yellowdog (130.75/4.5/134) in world_lobby
[11:22:06 INFO]:  - playground-spawn (0/66.75/-2) in world
[11:22:06 INFO]:  - lobby-skyblock-block (142.38/5.4/138.88) in world_lobby
[11:22:06 INFO]:  - lobby-westeros (139/4.5/142.25) in world_lobby
[11:22:06 INFO]:  - lobby-luckyblock-block (142.38/5.4/133.88) in world_lobby
[11:22:06 INFO]:  - lobby-westeros-block (139.12/5.4/142.38) in world_lobby

Once FancyHologram gets the the commands revamp, it will also look nicer. I just added the world name for now.

Thanks. Looks good.
Please add yaw and pitch in list and info.


add missing conf so akka can find tables

We need to let akka know which tables to use.

Thanks @pvlugter that makes more sense.

Think this is what we are looking for now.

doh! .. ok hows this?

I've made some changes in #396 so that the tests pass. I've rebased this PR.

@coreyauger and @pvlugter, nice job!


Create models

We need models for each table and connecting tables for our database

Philip - user
Lavanya - post
Maddie - categories


Remove errant console log for resetScroll

Describe the bug
Removing this console log statement that prints "resetScroll".
The log is being emitted at this line.
Your Example Website or App
https://github.com/TanStack/router/blob/beta/packages/router-core/src/router.ts#L1713
Steps to Reproduce the Bug or Issue

Add the  component into your app.
Navigate from one route to another.
Check the console for the log.

Expected behavior
As a user, I shouldn't be seeing any info or warn level console logs from this library when in production.
Screenshots or Videos

Platform

OS: macOS
Browser: Chrome
Version: 116

Additional context
No response

Fixed


File System Read-only after fstab optimizations

Hello,
When performing the optimizations to the fstab file via sed method or direct editing of fstab with nano, my root subvolume always becomes read-only following the recommended reboot.
I can correct the ro to rw using the following command: mount -n -o remount /dev/mapper/luks-##### /.
Though this corrects the issues, it is not a persistent fix or a complete fix as some portions of the file system remain stubbornly in read-only mode, such as the NetworkManager files for establishing WiFi connections.
All file output appears exactly as your code, but when I do mount -v | grep /dev/mapper, all subvolume appear just like yours with the exception of root, which appears as such: /dev/mapper/luks-##### on / btrfs (ro,relatime,seclabel,ssd,space_cache=v2,commit=120,subvolid=257,subvol=/@)
I am running a fresh install of Fedora 36 with the 5.17 kernel (I have attempted with both 5.17 and 5.18). Any assistance would be appreciated :)

I was able to solve the issue by reinstalling and configuring subvolumes through the Advanced (Blivet-GUI) rather than after the fact.


Clean up icon file in case UnregisterAll is not called

Describe the bug
We want to make sure we don't leak icons, even in the case the developer doesn't call AppNotificationManager::UnregisterAll().
Steps to reproduce the bug
Task filed from a feature in progress.
Expected behavior
No response
Screenshots
No response
NuGet package version
No response
Packaging type
No response
Windows version
No response
IDE
No response
Additional context
No response

This is more of implementing an opportunistic way of cleaning up icons belonging to uninstalled apps which didn't call UnregisterAll prior to uninstalling.

What icons? What creates them? When are they created? Where are they created that 'leak' is a concern?


fix: Fixes #244. Concatenate balances to 2 decimal places on account overview page without rounding

With these changes, given an account with 0.999432 ETH and 1233.8945 TIB.
Previously the account summary page would show:
Ether: 1.00
THIBCoin: 1233.89
Now it shows:
Ether: 0.99
THIBCoin: 1233.89
Previously the Send Ether page would show: 0.999432
Now it shows: 0.999432
Previously the Send THIBCoin page would show: 1233.894500
Now it shows: 1233.8945

Note that another approach could be to use BigNumber.js rounding http://mikemcl.github.io/bignumber.js/#constructor-properties


Request to prioritize telegram connection

Hello again. It would be great if you could implement discord or telegram message scraper. I dont know which is faster to deliver messages, but in my case, i get the Telegram messages the fastest. maunally entering the coin is too slow.

Hey, I'm looking into this. I use Discord more than Telegram but I'm happy to add Telegram support too. I can't give an ETA on this though

If you're able to find resources (videos or node modules) on retrieving messages from Telegram and/or Discord, that would be helpful

Hi alright found a few..
https://stackoverflow.com/questions/46117493/telegram-api-how-do-i-get-messages-from-a-public-channel-which-i-am-not-partic
https://core.telegram.org/method/channels.getMessages

https://core.telegram.org/api/obtaining_api_id
and perhaps this
https://www.npmjs.com/package/telegram-bot-api

https://www.npmjs.com/search?q=keywords:telegram client

https://github.com/yagop/node-telegram-bot-api
might be same as the npmjs link above

although this says is FAST.....which we want :-)
https://www.npmjs.com/package/@mtproto/core

and discord.
https://www.npmjs.com/search?q=discord

https://javascript.plainenglish.io/nodejs-telegram-bot-5e372f67f57

This might be the one you need actually :-)
https://github.com/gram-js/gramjs

and this one for discord
https://www.npmjs.com/package/discord-client-api

MTProto core seems like the module needed for Telegram. Thanks!

Perhaps something here to skip reinventing code?
https://stackoverflow.com/questions/64594824/mtproto-core-telegram-replay-to-another-channel


Explore using popovers for property sheet

The current method of expanding the dashboard item to show its property sheet results in UI element shifting position. With auto-closing of property sheets on a timer, this means that another property sheet being editing can shift out from under the mouse, which is pretty horrible UX.
So, explore using bootstrap's popovers instead, so that the property sheet can be displayed and hidden without any shifting of the elements being edited.
http://getbootstrap.com/javascript/#popovers

https://github.com/mjolnic/bootstrap-popover-picker/


Namespace environment variable not set

Expected Behavior
Namespace environment variable set on daprd process.
Actual Behavior
Namespace variable not set (verified with ProcessExplorer).
DAPR_* variables are set.
Steps to Reproduce the Problem
CLI version: 1.4.0
Runtime version: 1.3.0
Man.Dapr.Sidekick 1.1.0
Windows 10

Create component config which points to non-existing IP (see below)
Set Namespace in json config to

only components from configured namespace should be loaded

daprd will try to load component configuration and show connection error despite namespace in sidecar config

process component mqtt-pubsub error: init timeout for component mqtt-pubsub exceeded after 5s"

pubsub.yml
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
name: mqtt-pubsub
namespace: remote
spec:
type: pubsub.mqtt
version: v1
metadata:

name: url
value: "mqtt://<IP_ADDRESS>:1883"

appsettings.json
"DaprSidekick": {
"Sidecar": {
"Namespace": "local"
}
}
Release Note
RELEASE NOTE: FIX Missing Namespace environment variable.

Now available in Release v1.2


fix!: lipome compute task optim

Companion PR

https://github.com/Substra/substra/pull/350

Breaking changes

The endpoint /compute_plans/<pkey>/tasks now returns the ComputeTask without their associated inputs and outputs.
The endpoint /tasks now returns the ComputeTask without their associated inputs and outputs.
The endpoint /tasks/<pkey> still returns the full view of the ComputeTask (including inputs and outputs).

Description
Fixes slow page display (list view of compute tasks linked to a compute plan).
Two issues have been fixed:

add some missing prefetch_related statements
duplicate the ComputeTaskSerializer to have one for the list view and one for the detail view.

The serializer for the list view does not contain any information about inputs and outputs, since these information are not used currently and make a lot of queries to get nested objects.
The serializer for the detail view is the same as it was before (optimised queries though).
How has this been tested?
This has been tested manually for the frontend part, comparing the version actually in production with a local cluster built on my branch. Pages that were checked:

/compute_plans
/compute_plans//task
/compute_plans//task/
/tasks
/task/

For the backend, I have added non-regression tests that check that the number of queries is in O(1). The so-called n+1 queries issue is caused by some missing prefetch_related statement that causes the number of queries to be linear in the number of items. In our case, because of nested items, we actually have a complexity that is higher than linear without the prefetch_related.
Impact

URL
nb items
main
this branch

/compute_plans//tasks
10 / 60 tasks
118/672
14

/tasks
10/60 tasks
38 / 142
14

/task/
4 / 10 inputs
122 / 182
18

TODO

[ ] Evaluate impact to delegate the querying of inputs / outputs to the SDK
[ ] If not delegated to the SDK, add unit test to cover the new query parameter
[ ] Update changelog

Checklist

[ ] changelog was updated with notable changes
[ ] documentation was updated

/e2e --tests sdk,frontend,substrafl --benchmarks mnist,camelyon

End to end tests: :hourglass_flowing_sand: STARTING

/e2e --tests sdk,frontend,substrafl --benchmarks mnist,camelyon

End to end tests: :hourglass_flowing_sand: STARTING

/e2e --tests sdk,frontend,substrafl --benchmarks mnist,camelyon

End to end tests: :hourglass_flowing_sand: STARTING

/e2e --help

Usage: /e2e [options] [help]

/e2e may appear anywhere as long as it is on its own line

Options:
  --refs <value>                                         Extra refs (branch or tag) with format REPO=GIT_REF,REPO=GIT_REF.
  Supported repositories: hlf-k8s, orchestrator, substra-backend, substra-frontend, substra-tools, substrafl, substra, substra-tests, substra-ci.
  Example: /e2e --refs substra-backend=some_branch,orchestrator=some_tag (default: {})
  --tests-to-run, --tests <tests-to-run>                 Comma-separated list of tests to run. Valid options: sdk,substrafl,frontend or NONE. (default: "sdk")
  --benchmarks-to-run, --benchmarks <benchmarks-to-run>  Comma-separated list of workflows tests to run. Valid options: mnist, camelyon or NONE. (default: "NONE")
  --orchestrator-mode, --mode <orchestrator-mode>        Comma-separated list of orchestrator modes to run tests for.


Please document what's needed for backups

Classification

[X] Documentation

Reproducibility

[X] Not applicable

Description
Would be nice to have documentation on what directories have to be backed up besides everything under the /mnt/docker mount?  I don't think some of the SQL index files need to be?
Also, one other thing worth documenting is you should do a docker-compose logs mariadb when you first start up the container up so you can write down the randomly generated root password.  I didn't do that and had to reinstall because I wanted to change my DB User password and didn't realize it had to be changed in the DB....took a while to figure out that dovecot couldn't talk to mariadb because postfixadmin had no such  problems....

Backup /mnt/docker entirely, a "minimal set" is not viable way when you setup a backup system. Backup everything in /mnt/docker to keep a consistent state and futur proof solution for all your containers. And use mysqldump to make MariaDB databases dump.

Also, one other thing worth documenting is you should do a docker-compose logs mariadb when you first start up the container up so you can write down the randomly generated root password.

Yes, you're right, I forgot to document that. Do you want to make a pull request ?

If you using PostgreSQL you can use this to backup all your databases:
docker exec -u postgres -w /tmp postgres pg_dumpall | gzip > ~/dump.sql.gz
This will save dump.sql.gz on your host home folder.
[]'s

Sounds good...I can do a first cut at a backup doc and you can tell me what I missed ;-)
It's sort of a question of minimal because stuff like the DB indexes don't need to be in your backup...and if there are any logs that end up in /mnt/docker, they shouldn't be backed up either.  You don't want to bloat your backups IMHO.  Postgresql DB dirs are also huge because they don't always get compacted.
I can do a PR to document the startup capture of the root password of mariadb as well...does that also have to be done for postgresql?


Added allowable optional honorific in \name

E.g., can now have:
\name[Dr.]{Virgil}{Griffith}
And have it display at the top but not at the bottom.  There may be a better way to do the spacing between the @honorific and the @firstname in the \headerfirstnamestyle .  I leave this to you great LaTeX gods.

For the sake of consistency, shouldn't there be an extra \honorificnamestyle and more importantly a conditional to print the space after the honorific only if needed?
If @posquit0 accepts the PR, i can take care of this.


Adding Algolia search

Hello,
I'm the author of the jekyll-algolia plugin. I've integrated the  plugin in a fork of your website and would like to know what you think of it. It replaces your current search bar with one that will display results instantly as you type your keywords (and handling typos). It also re-uses the styling that already exists on your website.

You can try it live on https://peaceful-stonebraker-517e10.netlify.com/
How it works:
I saw that you already gave the previous plugin (algoliasearch-jekyll) a try, so I assume you already are familiar with how Algolia works :)
In a nutshell, the plugin adds the jekyll algolia command. When executed, it will extract all relevant information from your website and push it to Algolia. Every time a user will type a character in the search bar, it will query the Algolia API and return the most relevant results. Those results will then be transformed into HTML elements and displayed in the page.
Why I am doing this:
I'm submitting this PR hoping that you will integrate it into your website.
I've been using your website as a test environment for the plugin, as this is one of the largest Jekyll website I've found. I also particularly enjoyed working on it, as your writing about technical documentation is very interesting. By having the plugin visible on your website, I'm also hoping that more developers / tech writers will know that it's possible to have fast and relevant search, even on static websites.
How to run it automatically:
Note that for this demo, I'm using Netlify to host a preview of the website and run the jekyll algolia command automatically each I push to GitHub. You're currently using GitHub Pages, and they don't allow running arbitrary third party plugins. There are ways to make it work on GitHub pages as well, using either Travis or some webhooks. If you think this PR is interesting, we can work out the details of making it run automatically on each push.
Last notes:
I had to change one of your post, as the inclusion of {{ content }} was causing it to re-render the page recursively, causing records sent to Algolia to fail, due to their large size.
I'm also currently pushing all data to my own Algolia account. We'll have to switch this to your own account if you accept the PR. It currently uses ~30k records (which is more than the 10k records included in our free plan), but I'd be happy to give you access to our Open-Source plan that can go as high as 100k records.
Anyway, enough talking, let me know what you think of this fork and any feedback is appreciated :)

Hi Tim, I appreciate the sample integration and demo. It looks sharp and has a quick response. However, excluding {{ content }} is problematic because without it, the search depends on keywords appearing in the frontmatter only. For example, if you search for "touchpoints," which is a keyword that appears in the body and not the frontmatter, you'll get different results between Algolia and Google.
Any thoughts on how to index {{ content }}? Isn't search going to deliver better results if all the content is actually indexed?
Also, I wasn't aware that my site could be indexed for free in Algolia. When I checked this previously, I found that I had too many records.

Oh wait, I think there's some confusion about what I meant about me modifying {{ content }} :)
I talked about this post that looks "broken" on your current blog (as it recursively including the content of the page inside itself). I updated it on my fork to wrap the example calls to {{ content }} into {% raw %} and {% endraw %} tags to avoid the issue.

Current blog
Fork

Any thoughts on how to index {{ content }}? Isn't search going to deliver better results if all the content is actually indexed?

So to be clear, the whole content of the page is actually indexed, not just what is in the front-matter :). But you're right that your "touchpoint" query gives weird results. I think there is some issue created by the fact you have a weight attribute in your front-matter and I do use a weight attribute as well. I'm going to have a look at that :)

Also, I wasn't aware that my site could be indexed for free in Algolia. When I checked this previously, I found that I had too many records.

If we follow the pricing by book, you have too many records to fit in the free plan. But we also have an Open-Source plan that we can give to people promoting the sharing of knowledge, and I would be totally ok giving you access to this one.

Ahh, so you are indexing content. In that case, I'm much more interested and will want to take a closer look. Also, knowing that I could qualify under the open source plan is awesome.
I like the power and control that Algolia provides. It's something I'd like to become much more knowledgeable about.
Re the issue you described earlier, I do have a weight frontmatter tag. It's arbitrary, though, and I could potentially change it.
Also note that I have two other microsites:

http://idratherbewriting.com/learnapidoc/
http://idratherbewriting.com/simplifying-complexity/

These other sites are separate Jekyll projects, each in their own repos. However, a search on one site will surface content from the others, because I just kick off a site search for the root domain (idratherbewriting.com). I'm not sure what you think the best approach there is.

Great! Happy to see this is something you would be interested in. Also, if you have any specific questions about how Algolia works, feel free to ask :)
For the weight key, I think it's more my responsability, as a plugin creator, to make sure I'm not messing with the users arbitrary front-matter options. I'll see how I can fix this problem directly from my side, but ideally you should not have to rename any keys on your side. I created an issue my the plugin and will handle it.
As for the two other websites, my suggestion would be to have 3 independent searches. As they are 3 different projects, they can each have their own jekyll-algolia configuration and push to 3 different indices (using the same Algolia account). A search on "Documenting APIs" will only surface results from that website.
If you want to have a "meta" search that allow searching into all 3 websites at once, it's doable, but I don't think that's the easiest path for your users.
In a second phase, I would also like to suggest adding DocSearch to your documentation theme. But let's not get too carried away ;)

Hello again,
I updated the plugin to v1.2 that should fix all the issues we mentioned in this thread. The weight you have in your pages does not interfere with the plugin anymore. It makes the touchpoint query discussed much more relevant now. You can check the updated demo here.
I also added the plugin to Documenting APIs (demo, repo) and Simplifying Complexity (demo, repo) so you can see how it would look like.
How would you like to move from here?
If you merge the PR, you will need to replace the application id with your own. You will need to create an Algolia account for that if you don't already have one. Once you have it, you can send me an email (tim at algolia dot com) with your application ID and I'll upgrade it to the Open-Source version.
Let me know if you have any questions, I'd be happy to answer them :)

I updated the PR to have all the changes in one commit only. You can check it live on https://idratherbewriting-search.netlify.com/.
I'm still using my application ID and search API key so the live demo correctly works but you should update them to your credentials when your merge it. I've also upgraded your account for I'd rather be writing to our Open-Source plan. You can now host up to 100k records (your website currently uses ~30k).
I've also submitted PR to your other two repositories, but those two already fit in the Community (free) plan as they only need ~2k records each (and the plan offers up to 10k).
Happy to help on any more questions you might have. I'm excited to see it live on your website soon :)

Thanks for making these updates! I really appreciate it. I merged the 3 PRs, updated the app ID and search key in the config files, and also ran the index in each of the projects. I looked at the searches on each site, and they seem to be working as designed. If you want to confirm that everything looks good, feel free. I still need to explore all the code changes to understand the implementation better in the theme, and I'll do that within a couple of weeks or so, but so far it looks great. Thanks again.

Tim, when I search for "Simplified Technical English", I don't get any results. I'm looking for this post: http://idratherbewriting.com/2017/01/25/hyperste-simplified-technical-english-asd-ste100/
Do you know why this post isn't indexed in the Algolia index?

I think the issue is that the plugin isn't indexing the content correctly. I get errors warning me that some entries are too large. For example:
The jekyll-algolia plugin could not push one of your records as it exceeds the
10 Kb size limit.

The plugin will create one record for each element matching your
`nodes_to_index` value (currently set to "p,.summary, li").
Each record should not weight more than 10 Kb.
One of your records weights 37.02 Kb and has been rejected.

objectID: 26d1ffabbabdf93b16076f5127d7100e
title:    Twenty Usability Tips for Your Blog — Condensed from Dozens of
Bloggers' Experiences
url:      /2007/04/09/twenty-usability-tips-for-your-blog-condensed-from-dozens-of-bloggers-experiences/

Most probable keys causing the issue:
   html (17.84 Kb), content (17.82 Kb), summary (0.24 Kb)

Complete log of the record has been extracted to:
   /Users/tomjoht/projects/idratherbewriting/jekyll-algolia-record-too-big-26d1ffabbabdf93b16076f5127d7100e.log

This issue can be caused by malformed HTML preventing the parser to correctly
grab the content of the nodes. Double check that the page actually renders
correctly with a regular `jekyll build`.

If you're having trouble solving this issue, feel free to file a bug on GitHub,
ideally with a link to a repository where we can reproduce the issue.
  https://github.com/algolia/jekyll-algolia/issues

Going to check that right now :)

I understand what is going on. It seems that a few of your posts generate records that are too big. I was doing my tests on a privileged account with higher quotas, so I didn't see this issue until now.
The good news is that the "records too big" issue is actually something I can fix from inside the plugin, without requiring you (or others) to upgrade their account. I have working version on my machine, I now need to test it properly before releasing a new version of the gem. I might not be able to do it before the week-end, but I'll finish it early next week.
In the meantime, you can revert to using my account for the search, it still contains everything.

Sweet, thanks for finding a solution. I'll wait for your update to the plugin. In the meantime, I did switch the app ID and search key back to yours so that the search results would show more results.

I noticed an issue with the UI integration. When the site loads, the "Search Results" title and "Search by Algolia" image first appear for a split second before being hidden. As a result, the site jumps a bit.
Also, if I open the JS Console, I see an error related to the join code. I made a short little video (my voice is hardly audible) here: https://www.screencast.com/t/hQhX0mBN508
Would it be possible for you to address these two issues? I explored the site jumping issue a bit but couldn't figure it out.

Hi, Tim. @pixelastic Is there a live site on which I can try the integration?


Maintain dialog alive without activate alexa every time

Hi, I want know if this scenario is feasible with alexa-app framework.
Alexa:"It's greater!"

Related? Dups? #289 #196

Ok thanks!
There is a way to execute an intent only if I'm in a particular context? I.e. if I ask to my skill info about my gas bill and after I ask if there are suggestions about cost saving, the cost saving intent can be provided only if I'm in the dialog mode and I've asked  previously about gas bill?

There's no way at the library level to only conditionally execute an intent.
What most folks do, and what Amazon themselves recommend, is do your own state management from within your own handlers. At a trivial level, this could be as simple as:
app.intent("DoAThingOnlyIfStateIsFoo", {}, (req, res) => {
  if (req.getSession().get('state') === "Foo") { // Set in a different intent
    // Do the thing!
  } else {
    res.say("I'm sorry, I didn't understand you")
  }
})

For more complex skills, I've personally found success modeling a more formal state machine, with intent handlers capable of causing state machine transitions and executing separate state machine handler functions as appropriate.

ok thank you! another thing. I've developed a skill that has slots mandatory, but although they are mandatory and of a specific type (AMAZON.Number or AMAZON.Month) if I say "CAT" for a month alexa go ahead!! There is a way to delegate the check of response to Alexa platform? It works also for custom slots?
Thanks

Yeah, there's sadly no good way to do that, it's a problem on Amazon's end that they'll happily trigger an intent with nonsense slot values.


Device: Calliope mini 3

This pull request will start the registration of your device in the Jacdac catalog.

@joernalraun please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.
@microsoft-github-policy-service agree [company="{Calliope gGmbH}"]

@joernalraun  Do you have more info about the USB connection? See the microbit device config.
https://github.com/microsoft/jacdac/blob/main/devices/microbit-educational-foundation/microbitv2.json

The picture looks a bit dark maybe? https://microsoft.github.io/jacdac-docs/devices/calliope-ggmbh/calliopeminiv3v30/

The picture looks a bit dark maybe? https://microsoft.github.io/jacdac-docs/devices/calliope-ggmbh/calliopeminiv3v30/

I think it is ok, the pcb is now black/dark gray matte. ;)

@joernalraun Do you have more info about the USB connection? See the microbit device config.
https://github.com/microsoft/jacdac/blob/main/devices/microbit-educational-foundation/microbitv2.json

USB is "Calliope mini V3 CMSIS-DAP".
So this would be:
"transport": {
"type": "usb",
"requestDescription": "Calliope mini V3 CMSIS-DAP"
}
I think! :)

It would be good to also have a build of the "jukebox" for calliope.

It would be good to also have a build of the "jukebox" for calliope.

Yes, sounds good to me as well. We will look into this!


Turn on -preview=markdown

This PR does two things:

Adds the -preview=markdown flag to the Phobos Ddoc build.
Fixes places in the docs that were unintentionally interpreted as markdown.

The fixes were generally for:

Lines that start with -, + or *, which in Markdown starts a list item.
Backslashes that aren't in code blocks, and so are treated as escaping the next character.
Words in brackets that aren't in code blocks, which Ddoc treats as reference links to symbols with the same name.

ping 3 days are up.


[Access Change Request] table 363 "Analysis View" - ValidateModify

Hi,
could you please change the access of the procedure ValidateModify from local to global?
Thanks.

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. Please do not reply to this, as we do not monitor closed issues. If you have follow-up questions or requests, please create a new issue where you reference this one.
Build ID: 10762.


Refactor: move model/ to server/model/

Shouldn't we try to replace drone-go types with models and therefore leave the models at some central point?

the drone-go is the web sdk and I think we should differ between database backend and rest api structs


Fix blog RSS/Atom feed validation

Fix validation errors for the generated RSS and Atom feeds, according to https://validator.w3.org/feed/.
I'd add tests, but there are no specific tests for the feeds, or I couldn't find them...

Merging this would be appreciated 👍

I can add more tests if required...

Thanks


Error Looking at Orders when logged in as member and user

Hi,
There is a bug with the legacy User access being used (so the same common code can be used for Umbraco 6 and 7).
When you use:
umbraco.BusinessLogic.User.GetCurrent()
To access the backend user in Umbraco 7+, if a Member is logged in on the front end of the site, this returns Null. This is because the BusinessLogic functionality is only to be used for Members in 7+.
For Users they should be accessed using the Services methods (e.g):
var us = Services.UserService;
us.GetUserById(Security.CurrentUser.Id);
I don't think this can be updated without changing a fair few things, as that is part of the common codebase used by both 6 and 7, however I wanted to identify this so it is a gotcha people can know about when using the system (and therefore know not to be logged in to the front end of the site, when working in the backend).
Cheers,
Tony

Intersting issue, will have a look at this, we can make umb7 only functionality (as we done with some others as well). Will put on todo!

We pushed a possible fix for this to our development branch. But it needs proper/better testing.
Also the 2.7 release does need an Package Install since it add a new database table (to make repeatable orders possible, work in progress!)


get rid of some output

@thieman this cuts down a bit.


docs(operators): write comprehensive JSDoc on window operators

Merge after PR #1473 is merged

I'll check in #1473 soon. (May require another rebase?)

No problem to rebase IMO

Coverage remained the same at 95.982% when pulling b93b3c27cf2e599a51e43b2f1ab2a43d0548ce69 on staltz:window-docs into 82cd6d998b50f67626b25f43fab925e6ec54a0eb on ReactiveX:master.

Coverage remained the same at 95.982% when pulling b93b3c27cf2e599a51e43b2f1ab2a43d0548ce69 on staltz:window-docs into 82cd6d998b50f67626b25f43fab925e6ec54a0eb on ReactiveX:master.

LGTM

Merged with 7d9ed39, thanks @staltz