Performance of Geod.fwd for large numpy arrays as input.

Hi.
I am a big fan of the pyproj implementation.
My issue is more a question rather than an issue.
I am working on a project where i need to make a large number of points along a route consisting of great circle lines.
I am using g.fwd to create the sample points using the initial azimuth and an array of partial distances from the waypoints to create the points. The code is fast, but I need between 25,000 and 100,000 points along a given route and the creation (g.fwd) of the points ends up being a bottleneck.
Do anyone have a suggestion to increase performance?
Can I somehow trick npts to take an array of waypoints as input? looping through npts for each part of the route is significantly slower.
Multiprocessing is not an option as the main program is already running calculations like this in parallel.
A simple example is included below:
#Finding the distances and directions for traveled route legs
import numpy as np
from pyproj import Geod
g = Geod(ellps='clrk66') # Geod definition

#Definition of waypoints
lon0 = np.array([-44., -42.25797664, -40.50913661, -37.,
                 -33.49086339, -31.74202336, -30.])
lat0 = np.array([ 56.5,  56.58622486,  56.64795118,  56.69741501,
                 56.64795118,  56.58622486,  56.5])
#Calculating forward and back azimuths and distances between waypoints
az12,az21,distances = g.inv(lon0[0:-1],lat0[0:-1],lon0[1:],lat0[1:])

#1000 samplepoints between each waypoint
points = np.ones(np.size(lon0)-1).astype(int) * 100

#Creating distance factor from a waypoint to a given samplepoint
linspace_array = np.linspace(0, 1. - 1./points[0],points[0])
#Repeating array to create distance factor for each samplepoint between waypoints
points_space = np.tile(linspace_array,len(points))
#Creating the longitude start array, excluding the end point
lon0_start_array = np.repeat(lon0[0:-1],points)
#Creating the latitude start array, excluding the end point
lat0_start_array = np.repeat(lat0[0:-1],points)

#Creating the array of dirrections - repeating the direction for
#the number of points you want between two points
#Creating arrys used to find the sampling points
az12_array = np.repeat(az12,points)
#Creating the distance array used to create the points.
dist_array_repeat = np.repeat(distances,points) * points_space
#Creating sampled points in between the given lat and lon coordinates for
#way points.
lon, lat, backaz = g.fwd(lon0_start_array, lat0_start_array,
                         az12_array, dist_array_repeat)
#The end point is not returned from the above calculation

Thanks for, from what I have seen, the most efficient package for what I want to do.

Try using the npts method for Geod.  This might speed things up.  However, I notice from
the code that the npts method itself could easily be speeded up.  I'll submit a separate issue
about this.

Hi cffk.
Thanks.
npts might be faster in itself compared to inv/fwd, but it does not handle an array input (a route with several waypoints) so I would have to loop through it, thus making the code significantly slower than the presented example.

OK.  FYI, the best-case timing for a calculation like this is about 0.35 us per waypoint.  This is based
on times for the C++ code (with all the looping done within compiled code) -- I expect the C timings to
be similar.

Ok, interesting.
I just tested the above code snip. It results in roughly 0.7 * 10^(-6) seconds per produced point, so there is still some performance gain to achieve. If it is possible with a route with waypoints not on one great circle line.
Using npts by looping through the waypoints (as shown below), I get an  increase in computation time close to 100 times that of the above inv/fwd method (0.65*10^(-4)).
for i in range(len(lon0)-1):
    lonlat_tuple = g.npts(lon0[i], lat[i], lon[i+1], lat[i+1], points[i], radians=False)

Thanks for all your help so fare.

I've submitted pull request #115 to give you that extra factor of 2.  Please try it.
There's another 20% improvement available, if you're willing to accept waypoints
which are only approximately evenly spaced (+/- 0.3%).  This involves specifying
the distance in terms of the arc length on the auxiliary sphere.  However, the
complexity of explaining what's going on and the modest performance gain (only
20%) means that it probably isn't appropriate for pyproj.  The C documentation
is here.

I have just updated my pyproj install. (checked it both using conda and pip)
I see an insignificant performance gain, but no where near a factor 2.
It is the first time i actively update a package for after a pull request, so it might just be an older update I have installed.

The pull request hasn't been merged into pyproj.  So you would need to check out
git@github.com:cffk/pyproj.git
branch = geod-npts-fix
and go through the steps to build and install it.  Alternatively, sit tight and maybe
the pull request will be merged and included in the next release.

PR #115 was just merged for Geod.npts performance improvements.


Kind:Issuer

Method: $cluster->fromTemplatedYamlFile just skipping kubernetes resource Issuer.
I didn't find information about Issuer in your documentation.
Would be great if you add possibility to create this resource type.

Issuer kind is a custom kind right? It isn't a Core Kubernetes resource.

Issuer kind is a custom kind right? It isn't a Core Kubernetes resource.

Yes it is cert-manager

@sandweel This package contains only kinds that are in the Kubernetes core, but you can create your own classes and make them interact with your $cluster variable: https://php-k8s.renoki.org/advanced/create-classes-for-crds


1.16.3

Here i am sending the json for wyrmroost heavy use of comments so it is easier to tweak stuff later.
wyrmroost.txt

This is not the place where this goes.
Addtionally this PR targets the wrong branch,

You will need to go to the GooForMods github repo and add your values there.

could i ask for a link

https://github.com/MercuriusXeno/GooForMods


More tests and coverage

Codecov Report

Merging #23 into master will increase coverage by 16.31%.
The diff coverage is 2.5%.

@@             Coverage Diff             @@
##           master      #23       +/-   ##
===========================================
+ Coverage     8.5%   24.82%   +16.31%
===========================================
  Files          29       30        +1
  Lines         811      850       +39
===========================================
+ Hits           69      211      +142
+ Misses        739      596      -143
- Partials        3       43       +40

Impacted Files
Coverage Δ

cmd/bindata.go
21.83% <ø> (+21.83%)
:arrow_up:

internal/testwrapper/vg/main.go
0% <0%> (ø)

cmd/initSettings.go
54.54% <33.33%> (+43.22%)
:arrow_up:

internal/workspace/localInstall.go
3.97% <0%> (+3.97%)
:arrow_up:

internal/utils/utils.go
46.15% <0%> (+17.3%)
:arrow_up:

cmd/root.go
40% <0%> (+20%)
:arrow_up:

cmd/activateReal.go
71.42% <0%> (+42.85%)
:arrow_up:

... and 7 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0445f63...29561bf. Read the comment docs.


change the object of deviceImageInfo to deviceImageSetInfo,

to fix a bug that we see duplicate image in group data
when devices with same image
related to THEEDGE-2485
Signed-off-by: mgold1234<EMAIL_ADDRESS>Description
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
Fixes # (issue)
Type of change
What is it?

[x] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Documentation update
[ ] Tests update
[ ] Refactor

Checklist:

[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes
[x] I run make pre-commit to check fmt/vet/lint/test-no-fdo

/retest

LGTM


fails to unmount bind mount

Occasionally when running acbuild I'll get an error like this:
umount: /home/jcollie/dev/rkt-logstash/.acbuild/target: target is busy
        (In some cases useful info about processes that
         use the device is found by lsof(8) or fuser(1).)

Once that happens and subsequent commands fail with:
run: remove .acbuild/target/tmp: directory not empty

It doesn't happen every time (maybe 5-10% of the time).

Once this happens and acbuild exits, are you able to run umount .acbuild/target without error?

Ok, I feel like there's some race condition resulting in umount running before systemd-nspawn is fully done in the directory. I'll poke around, thanks for reporting this.

I assume you're using acbuild run when you encounter this. What command are you calling? I can't seem to get this to happen with a while true; do sudo acbuild run ls; done.

Here's the script that I'm running to build my ACI:
http://fpaste.org/287028/75556144/
I put the sleeps in to help mitigate the problem but it doesn't always help.  I used to install all of the plugins as a separate acbuild run for each plugin before I realized that I could install them all in one shot.  I don't recall it breaking on anything but an acbuild run command.
I'm using Fedora 23 which comes with systemd 222.

I'm still having issues reproducing this. I switched from exec'ing the umount command to having the acbuild process send the syscall itself, which is something that I should've done in the first place. Maybe this will help fix your issue?
I've just submitted a PR for this (https://github.com/appc/acbuild/pull/94), would you be willing to give the changes a try and see if the error still occurs?
You can pull the new commit from here, if you'd prefer to avoid waiting for it to get into master.

I haven't had a chance to try the change, but I'm starting to suspect a 3rd party... I use syncthing to sync this directory between my work desktop and my laptop and I think that may be getting in the way here.  I'll be out on vacation for the next week so I don't know if I'll get a chance to do any more testing before I get back.

Ok, no rush. If this is an issue with acbuild, either someone else will come along in the interim with the issue who can test things for me, or it's not affecting very many people.

@dgonyeo Another option to explore is to use --overlay on modern versions of systemd-nspawn which would potentially mean we can avoid doing any of the mounting ourselves. (This could kind of go in hand with #86 in that we would provide a controlled/known-good version of nspawn)

OK, back from vacation and had some time to test.  The problem was definitely caused by the file sync software (Syncthing).  After shutting down Syncthing I can no longer reproduce.


chore: move repo to foundry template

In https://github.com/vacp2p/rln-contract/pull/31 the repo was ported to vacp2p/foundry-template. We need to patch the main branch with those changes so we may start working on the rln-v2 feature set.

Addressed in https://github.com/vacp2p/rln-contract/pull/34


Added BFS Graph traversal in C

Description
Added BFS Graph traversal in C.
Fixes #176
Type of change

[ ] New feature (non-breaking change which adds functionality)

Checklist:

[ ] My code follows the style guidelines(Clean Code) of this project
[ ] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] My changes generate no new warnings
[ ] I have added tests/screenshots(if any) that prove my fix is effective or that my feature works

@ravikr126 @Amit366 plz review this PR

@ravikr126 could u plz add label to this merged PR??


Fractional Zoom and SuperCluster

Hi Team,
Thanks for open sourcing / making your project public.
Not sure if this is a bug or if I am just having a Sunday Funday.
There appears to be an issue with the default cluster algorithm when applied to a map that is vector  based instead of raster based when updating clusters on click.
I am not 100% sure if I am debugging the issue correctly but it appears to be caused by/when zoom is not an integer value when tilted etc. Overriding the SuperClusterAlgorithm cluster function to round map.getZoom() to an integer seems to solve the issue more or less but not sure of any other issues it may cause or if it would be better to open an issue with the Super Cluster library team as the error seems to occur  in the getClusters function on const tree = this.trees[this._limitZoom(zoom)] line.

Yeah, fractional zoom is probably an issue since SuperCluster is going to require discrete dimensions for its kd tree.

Thanks for reporting. Will be published in the next release shortly!


How can I build caffeonsparkpythonapi.zip?

Sorry if this has been answered before, I am very new to spark and caffe and I have lots of questions.
Do I need to provide the caffe python api everytime I run a spark-submit, even though caffe is installed on each of the workers that will be participating in the cluster?
Is ssh communication required to ship these files?
What are all the jar files and why do I need them?
Any input would be greatly appreciated.

Hello,
I can answer a few of your queries:

SSH access: In almost all cluster setup, you will require bidirectional data flow. So all nodes will have to communicate with each other. What you should do is to set up the ssh keys in all nodes so that they can communicate without password. This is, in general, adding the public key in each nodes to the authorized keys.
About this: "Do I need to provide the caffe python api everytime I run a spark-submit"
What are you trying to do here ? Are you using the command line interface to do follow the GetStarted guides like GetStarted_yarn or trying Jupyter Notebooks like these DLDemo ?

Arun, Thanks for the reply I have an ansible role that I can run to help me achieve just that, redistribution of public keys on each of the nodes and the master. Is ssh required in the event that I am using a spark-submit without any additional parameters such as the pythonapi.zip and the jar files?

Sorry for the late reply. In my understanding, you need to make sure that ssh localhost works without a password in a single node cluster and that you can ssh back and forth from all nodes in a multi-node cluster, regardless of the parameters you include.
It is a hadoop/spark requirement I suppose. Namenode, Datanode in Hadoop should be configured for a passwordless ssh. Master and slaves in a Spark cluster too.

Thanks for the reply Arun, I'll try setting up key-based ssh authentication and see if anything changes in terms of the caffe module not being found as well.

"caffe module not being found as well"
-- did you mention that in your question ?

I did not mention that, not directly anyway. The issue that I am having with caffe is whenever I try to run a spark submit on a standalone cluster I get an error such as "caffe module not found" This does not seem to be an issue when I run a spark sumbit locally, the caffe module is found by spark and the job completes successfully.
I have posted this question on stackoverflow as well.
http://stackoverflow.com/q/39882281/6107394

I will have to ask a few questions in that case:

What is your OS ?
I suppose you want to use CaffeOnSpark in Python. Did you look into https://github.com/yahoo/CaffeOnSpark/wiki/GetStarted_python ?
What is the command that you run for the above mentioned "whenever I try to run a spark submit on a standalone cluster"? Paste it here.
What is the command that you run for "when I run a spark sumbit locally, the caffe module is found by spark and the job completes successfully".

Thanks !

Ubuntu 16.04
I am not currently using caffeonspark, I build caffe from scratch from official git repository
spark-submit --master spark://master-ip:7077 script.py
spark-submit --master local script.py (this works fine)

Oh I had this doubt when reading through the StackOverflow question.
@itsfootball , Please note that the CaffeOnSpark team ported Caffe to a distributed environment backed by Hadoop and Spark. You cannot, I am 99.99% sure, use Caffe alone in a Spark cluster or any distributed environment per se. (Caffe team is known to be working on this).
If you need distributed deep-learning using Caffe, please follow the building method mentioned here in https://github.com/yahoo/CaffeOnSpark/wiki/build to build CaffeOnSpark for that and use CaffeOnSpark instead of Caffe.
But, best bet will be to follow either this GetStarted_standalone wiki or GetStarted_yarn wiki to create you a distributed environment to carry out deep-learning.
Further, to add python, please go through GetStarted_python wiki.
Please let me know if it helped. I will post this answer to StackOverflow now.
Please close the issue and feel free to reopen it again if you have any doubts.

Ah got it, Thank you so much, this was very helpful.
I appreciate your patience on these questions.

No problem.


New session handling

@jread-usgs @aappling-usgs You guys want to check off this workflow before I do anymore?

Changes Unknown when pulling fc24c962b58423a5057651fa004b178e283999bd on lawinslow:master into ** on USGS-R:master**.

Ah, there we go. @aappling-usgs merge if you're cool with the changes. (also, Powstreams won't build correctly until this is merged).

huzzah! thanks, @lawinslow


Allow disabling holidays config

add functionality to be able disable holiday even when holidays are configured in the yaml file
This allows the workday? method to be able to not take into account the holidays configured if with_holidays: false is passed in.
I would love to see if other people have the same use case like we did.

No need for this with the block style of configuring.


When would I include a module in all?

I'm trying to improve the intellisense experience of using the Panel package by adding __all__.
I'm in doubt whether modules should be included in __all__ and why? I've tried to google an answer without luck.
Example Pydantic
Pydantic does it :-)

https://github.com/samuelcolvin/pydantic/blob/master/pydantic/__init__.py
Should Panel do it?

Bonus question
Is there a "best practice" package that I could look at if I am not able to understand or find what I need in the documentation?

The purpose of __all__ is simply to define which symbols should be imported as part of a wildcard import that targets that module. If a symbol is importable through a wildcard import, it makes sense that it should be considered public.
As I mentioned in this response, I wouldn't recommend adding __all__ to your library if it doesn't already support it because wildcard imports should really never be used to import from a library. I would recommend instead that you use redundant import forms if the intent is to re-export an imported symbol from the module.
from . import label as label
from . import links as links
...

I realize this probably seems convoluted and overly complicated. The problem is that Python, unlike most modern languages, doesn't have keywords to designate access control. Most languages have keywords like public, private and export so you can be explicit about access control. Python started as a scripting language, and access control is less important in that context. So now we're stuck with these convoluted conventions so library authors can specify which symbols are intended to be public versus private. :(

Using redundant form is a hard sell :-)

Isn't public, private and export a PEP away? :-)

To add a little bit more context. A part of the value proposition of Panel is that it should work

for beginning pythonistas and data exploration people in Jupyter Notebooks - They might be using from panel.widgets import *. For data exploration or an analysis situation on a trading floor this provides a short time to market.
for experienced software developers working in VS Code and using all kinds of tools and best practices to provide solid code.

Personally I'm trying my best to be the second persona. But my job is to support the first type of persona.
We are at version 0.13.0 :-). So early stage. But have +500.000 downloads per month.

Ahh. redundant form and __all__ are not just two ways of achieving the same thing. They have two different intents. The intent of __all__ is to enable using from XYZ import *. But pyright, pylance etc also uses them to determine which symbols should be considered public.
The intent of the redundant form is to help pyright, pylance etc to determine which symbols should be considered public.
??

Yes, that's correct. And to be clear, the "redundant form of import" wasn't our idea. It comes from PEP 484 and is the standard way to indicate in type stubs that an imported symbol should be re-exported. We're just reusing a convention that already exists rather than inventing a new one.

my job is to support the first type of persona.

To be clear, we're interested in both personas too! Pylance powers the programming experience in VS Code for both Python source (".py") files and Jupyter notebooks, and it caters to beginners who simply want completion suggestions all the way to power users who want refactoring tools and static type checking. So we understand where you're coming from!

Isn't public, private and export "just" a PEP away? :-)

This is highly unlikely. The Python steering council has not been receptive to adding features like this to the language.

@MarcSkovMadsen, I'm closing this issue as it seems that your questions have been answered. If you disagree, please let us know.