Apache Sedona translate from geotiff to COG

Expected Scenario
We're trying to translate tiff files to COG. Is it possible to use the Apache Sedona to do the translation?
If yes, is there any code reference that we can look at.
Settings
Sedona version = 1.3.1
Apache Spark version = 3.2
API type = Python (PySpark)?
Environment = Azure, Databricks, Synapse

In Sedona 1.4 we've added a new raster type. Its basically a serialized GridCoverage2D (from GeoTools).
You can use the new api to load the tiff files: https://sedona.apache.org/latest-snapshot/api/sql/Raster-loader/#rs_fromgeotiff
Sedona has no function to create COG files yet. You could implement it yourself as a udf. If you are willing to send a PR to Sedona - I'm sure the community would be very happy!

I did some more research. GDAL is able to create COGs and rasterio is a python wrapper around GDAL. You should be able to convert GeoTiffs to COGs in rasterio. If not there is a rasterio plugin to do that. https://github.com/cogeotiff/rio-cogeo
You can create a python udf that does the conversion. It will be painfully slow and consume lots of memory.
The other option is to implement the udf in a JVM language. Geosolutions has developed a GDAL wrapper in Java, which you can access at https://github.com/geosolutions-it/imageio-ext.

@umartin Hi Martin, I have been thinking about the COG format for a while. It is a bit hard to find a full-fledged java reader/writer for COG.
GeoTools does not have it. Image-io ext (https://github.com/geosolutions-it/imageio-ext/tree/master/plugin/cog) has a reader but no writer.
The only way to support both is GDAL java api. Do you think it is a good idea to include GDAL java as a dependency?
GDAL has comprehensive support for raster format: https://gdal.org/drivers/raster/index.html  If we can use it, this will open the entire raster world for Sedona. Raster data support could be the next big thing for Sedona.

@jiayuasu I think that would be a good idea. It’s not ideal to depend on a native LGPL library but I think it’s our only option right now. I guess you are thinking about adding something like this: https://postgis.net/docs/RT_ST_AsGDALRaster.html
If we can write a jvm implementation of COG at a later time we can simply add RS_AsCOG

We plan to write a COG writer in Apache SIS (it already has a COG reader). I can not promise when because it depends on external factors, but if we get selected in OGC Testbed-19 we may develop the COG writer this summer. The COG reader in Apache SIS has been developed as part of OGC Testbed-17, so the same thing may happen with Testbed-19 because a task item will be to develop a GeoTIFF writer for extraterrestrial use. This is an area where Apache SIS is well positioned, because it has one of the most advanced Referencing by Coordinates framework available in open source (Apache SIS was used as a proof of concept in Testbed-18 for referencing objects in space).
It brings back the topic of Sedona dependency to GeoTools. LGPL dependencies are normally not allowed in Apache projects unless they are optional. But given the importance of geospatial services for Sedona, it seems to me that a significant fraction of Sedona functionalities depends on GeoTools, and this fraction may be growing as new geospatial features are added in Sedona. GeoTools can be advantageously replaced by Apache SIS for metadata, referencing services and raster support. The GridCoverage2D class in GeoTools exists also in Apache SIS (actually I'm the original author of that class in GeoTools). The main drawback is the smaller amount of supported formats. But if bindings to GDAL are used as a temporary solution for getting a COG writer before it become supported in Apache SIS, that GDAL binding could be used for other formats as well.
I will be physically present at the joint OGC/OSGeo/ASF code sprint in Switzerland, April 25 to 27. If anyone from the Sedona community plan to be there, maybe it would be an opportunity to explore the feasibility of GeoTools replacement by Apache SIS?

@desruisseaux
That's really exciting news that you're considering implementing a COG writer! I'm curious, would this also mean that Apache SIS would support writing "regular" GeoTIFFs as well? If the COG writer is successfully implemented and backported to a Java 8 release of Apache SIS, I would love to hear about it. With write support for GeoTIFF, Apache SIS would be on feature parity with GeoTools, at least from Sedona's perspective. And with write support for COG, it would give it an advantage.
If you do release a Java 8 version with these features, I'd be happy to set up a Sedona branch that replaces GeoTools with Apache SIS, we could run some benchmarks and have a new discussion in the Sedona community.
Although I won't be able to attend the OGC/OSGeo/ASF code sprint in Switzerland myself, I appreciate you letting me know about it. Thanks!

@desruisseaux
For the testing purpose,  I think even a SNAPSHOT Java 8 release of Apache SIS is good enough. We can use that to confirm the integration between SIS and Sedona.

Hello @umartin and @jiayuasu, thanks for your reply!
Actually COG is not a file format, but rather a set of good practices in the way to encode GeoTIFF files. For example the TIFF format allows metadata to be located anywhere in the file, but COG restricts them to the beginning of the file. TIFF is very flexible on the way to organize tiles and overviews, but COG put some restrictions on the layout, etc. Consequently all COG files are also regular GeoTIFF files, only restricted to a subset of TIFF flexibility. For a first version, I see no reason to not produce COG files unconditionally.
In a future version, there is some reasons why we may sometime want a non-COG file. For example if we want to create a GeoTIFF file in append mode, i.e. add tiles to an existing file in random order, the COG layout is not suitable for that. But GeoTIFF stay well suited if we forget COG for that particular scenario.
For testing purpose, the SIS 1.3 release is Java 8 compatible and has the COG reader, optionally with Amazon S3 support. SIS 1.4-SNAPSHOT has some improvements and bug fixes, but requires Java 11. A port to Java 8 may be possible, but before to do that, if Sedona can be tested in Java 11, it would be useful for determining if the current set of SIS functionalities is suitable to Sedona.

Note: if testing with a large COG file, the following note may be useful: Handle rasters bigger than memory. In summary SIS supports immediate or deferred data loading. The former may be more intuitive and is the default for that reason, the latter scale better.


Make tests

I've added several tests with pytest for functions from geo_functions. HTML requests often yield slightly different results, so if there's a discrepancy, only a warning is shown. I added a GitHub action for PRs (main), but only for remote data. To run tests with only remote data use: pytest -m "not local_data". To run tests with only downloaded data use: pytest -m "not request".

Well it works now, and I merge it. I will open a couple of issues though.


Intellisense Not Working

Issue Type: Performance Issue
Hello,
Intellisense on my copy of Visual Studio Code is not working. I have tried several troubleshooting methods such as restarting the program, restarting my device, and re-installing my program. Nothing is working. Please advise. Thank you.
VS Code version: Code 1.51.1 (e5a624b788d92b8d34d1392e4c4d9789406efe8f, 2020-11-11T01:11:34.018Z)
OS version: Darwin x64 19.6.0

System Info

Item
Value

CPUs
Intel(R) Core(TM) i5-8257U CPU @ 1.40GHz (8 x 1400)

GPU Status
2d_canvas: enabledflash_3d: enabledflash_stage3d: enabledflash_stage3d_baseline: enabledgpu_compositing: enabledmetal: disabled_offmultiple_raster_threads: enabled_onoop_rasterization: enabledopengl: enabled_onprotected_video_decode: unavailable_offrasterization: enabledskia_renderer: disabled_off_okvideo_decode: enabledwebgl: enabledwebgl2: enabled

Load (avg)
3, 3, 3

Memory (System)
8.00GB (0.10GB free)

Process Argv
--crash-reporter-id f1278add-bcaa-4e07-a218-62d73d3dbb74

Screen Reader
no

VM
0%

Process Info
CPU %	Mem MB	   PID	Process
   16	   106	 77089	code main
    7	    25	 77092	   gpu-process
    0	     8	 77094	   utility
    4	    33	 77110	   shared-process

Workspace Info
|  Window (index.js — DayThree)
|    Folder (DayThree): 37 files
|      File types: js(8) json(8) gitignore(4) md(4)
|      Conf files: package.json(4);

Extensions (3)

Extension
Author (truncated)
Version

bracket-pair-colorizer-2
Coe
0.2.0

html-live-server
nav
0.1.0

LiveServer
rit
5.6.1

Can confirm, when i hover for the first time after reloading, it shows "loading..." indefinitely, if I move the cursor away from that I don't get hover tooltips anymore :(

@ItsDigitalEngine What language is this for. Please provide steps to reproduce the issue
/needsMoreInfo

@Bewinxed Please open a separate issue (make sure to fill in the issue template and include proper steps to reproduce)

@Bewinxed Please open a separate issue (make sure to fill in the issue template and include proper steps to reproduce)

Issues already exist for this, and it has been fixed with today's update, thanks!

Version: 1.52.0-insider (user setup)
Commit: 92192baf760315ac47fb2caaf3ce6d0778962352
Date: 2020-12-03T05:43:59.635Z
Electron: 9.3.5
Chrome: 83.0.4103.122
Node.js: 12.14.1
V8: <IP_ADDRESS>-electron.0
OS: Windows_NT x64 10.0.19042

Closing as more information is needed to investigate this


Update coding style with test method annotation

Update coding style with test method annotation, e.g. if we want to add a test method say 'foo' to an object, we will name it with prefix TEST, TEST_foo.

Is this PR still being worked on ?


Build lanelet2 wheel and publish in GH release

This PR implements a small GH Actions workflow which packages lanelet2 as a Python wheel. Whenever a tag is pushed, the wheel gets published as part of the release.
I won't get into the details about the build process and the failure to package lanelet2 using the standard tools provided by e.g. cibuildwheel. As a result, the wheel is not manylinux-compatible and as such, cannot be uploaded to pypi.org. Therefore, I chose to store the .whl as part of the GitHub release associated with each tag.
To use lanelet2 simply install via
$ pip install https://github.com/jdrudolph/Lanelet2/releases/download/1.2.0-test3/lanelet2-1.2.0-cp38-cp38-linux_x86_64.whl

Please feel free to make changes/extend this proposal.

I've tried out some of the suggested changes:

adding --plat manylinux_2_31_x86_64 worked nicely!
removing the cp38 dependency by renaming the wheel: I strongly advice against this approach. While it works for some of the non-3.8 versions in the tests you can see that is only due to the "coincidence" of the correct system libraries being available on Ubuntu 20.04.
the release build greatly reduced the wheel size, which is great

With the new --plat and the release build we can now upload to pypi.org. I already tested this with test-pypi.


Consider IntegerNode storing base and not a flag

If you look at functions like str2inum they work with base but IntegerNode requires examining boolean flags to determine base.  There are not many bases so this is a fairly minor enhancement.

I can't make this another field without adding a bunch of bytes because of alignment... ugh.
What I can do (which is a little hacky, but kind of funny) is that I could rearrange the flags so that they're:

binary
decimal
octal
hexadecimal

If we did that, then you could get the base by:
((flags ^ ~decimal) << 1) || 10

We could put tests in here to ensure that stays correct. What do you think?

Doesn't str2inum or so accept a base of 0 and then it auto-detects what to do? (and so flags are not necessary, since we're reusing the logic for Kernel#Integer)
I'm using ConvertBytes.bytesToInum() in TruffleRuby, which IIRC comes from JRuby.

@eregon yeah I suppose the overall cost of thoses ifs+switch is not a big deal but if we can just get it why not use it?
@kddnewton I like the idea.  As @eregon points out we have logic to examine the string but if we ever make the decision to use some binary representation for integers that will not work any more.  So I think this will future-proof things if we do decide to do that later.


Configuring OAuth2.0 endpoints of external identity provider is not getting reflected

Describe the issue:
Configuring OAuth2.0 endpoints of external identity provider is not getting reflected. Seems like it is always reading the hardcoded endpoints.
How to reproduce
Add below configs.
endpoints:{
    authorizationEndpoint:"/authorize",
    tokenEndpoint:"/oauth/token",
    userinfoEndpoint:"/userinfo",
    jwksUri: "/.well-known/jwks.json"
}

Even though I configured endpoints externally, it is taking the hardcoded endpoints in the class.
This was the authorization request that I could see in the network trace. (I configured /authorize only)
https://<idp_domain>/oauth2/authorize?response_type=code&client_id=<client_id>&scope=openid&redirect_uri=https%3A%2F%2Flocalhost%3A4200&response_mode=query&code_challenge_method=S256&code_challenge=ky83tKxV2DHV3GE984HCGOiF0KZSwqPRab1uk1JfAKc

https://github.com/asgardeo/asgardeo-auth-angular-sdk/issues/81

@thivi Is there any possibility to fix this any time soon

We will try to fix it within this week and release it.

@thivi what is the status of the fix

@thivi what is the status of the fix

A PR has been raised with the fix: https://github.com/asgardeo/asgardeo-auth-spa-sdk/pull/36.
A new release can be triggered once the PR is reviewed and merged.

Thanks @thivi

Resolved by https://github.com/asgardeo/asgardeo-auth-spa-sdk/issues/35 and https://github.com/asgardeo/asgardeo-auth-spa-sdk/pull/36


Particle animation

Adding feature #2725

To test
`var createScene = function () {
var scene = new BABYLON.Scene(engine);
            // Setup environment
            var light0 = new BABYLON.PointLight("Omni", new BABYLON.Vector3(0, 2, 8), scene);
            var camera = new BABYLON.ArcRotateCamera("ArcRotateCamera", 1, 0.8, 20, new BABYLON.Vector3(
                0, 0, 0), scene);
            camera.attachControl(canvas, true);

            // Fountain object
            var fountain = BABYLON.Mesh.CreateBox("foutain", 1.0, scene);

            // Ground
            var ground = BABYLON.Mesh.CreatePlane("ground", 50.0, scene);
            ground.position = new BABYLON.Vector3(0, -10, 0);
            ground.rotation = new BABYLON.Vector3(Math.PI / 2, 0, 0);

            ground.material = new BABYLON.StandardMaterial("groundMat", scene);
            ground.material.backFaceCulling = false;
            ground.material.diffuseColor = new BABYLON.Color3(0.3, 0.3, 1);

            // Create a particle system
            //var particleSystem = new BABYLON.ParticleSystem("particles", 2000, scene);
            //particleSystem.particleTexture = new BABYLON.Texture("textures/flare.png", scene, true, false, BABYLON.Texture.TRILINEAR_SAMPLINGMODE);;

             var particleSystem = new BABYLON.ParticleSystem("particles", 2000, scene, null, 64);
             particleSystem.particleTexture = new BABYLON.Texture("textures/player.png", scene, true, false, BABYLON.Texture.TRILINEAR_SAMPLINGMODE);;
            //particleSystem.playAnimation(0, 44, true, 100);
            //particleSystem.invertU=1;
            //particleSystem.invertV=1;
            particleSystem.cellIndex = 44;

            //Texture of each particle

            // Where the particles come from
            particleSystem.emitter = fountain; // the starting object, the emitter
            particleSystem.minEmitBox = new BABYLON.Vector3(-1, 0, 0); // Starting all from
            particleSystem.maxEmitBox = new BABYLON.Vector3(1, 0, 0); // To...

            // Colors of all particles
            // particleSystem.color1 = new BABYLON.Color4(0.7, 0.8, 1.0, 1.0);
            // particleSystem.color2 = new BABYLON.Color4(0.2, 0.5, 1.0, 1.0);
            // particleSystem.colorDead = new BABYLON.Color4(0, 0, 0.2, 0.0);

            // Size of each particle (random between...
            particleSystem.minSize = .5;
            particleSystem.maxSize = 1;

            // Life time of each particle (random between...
            particleSystem.minLifeTime = 0.3;
            particleSystem.maxLifeTime = 1.5;

            // Emission rate
            particleSystem.emitRate = 1500;

            // Blend mode : BLENDMODE_ONEONE, or BLENDMODE_STANDARD
            particleSystem.blendMode = BABYLON.ParticleSystem.BLENDMODE_ONEONE;

            // Set the gravity of all particles
            particleSystem.gravity = new BABYLON.Vector3(0, -9.81, 0);

            // Direction of each particle after it has been emitted
            particleSystem.direction1 = new BABYLON.Vector3(-7, 8, 3);
            //particleSystem.direction2 = new BABYLON.Vector3(7, 8, -3);

            // Angular speed, in radians
            particleSystem.minAngularSpeed = 0;
            particleSystem.maxAngularSpeed = Math.PI;

            // Speed
            particleSystem.minEmitPower = 1;
            particleSystem.maxEmitPower = 3;
            particleSystem.updateSpeed = 0.001;

            // Start the particle system
            particleSystem.start();

            // Fountain's animation
            var keys = [];
            var animation = new BABYLON.Animation("animation", "rotation.x", 30, BABYLON.Animation.ANIMATIONTYPE_FLOAT,
                BABYLON.Animation.ANIMATIONLOOPMODE_CYCLE);
            // At the animation key 0, the value of scaling is "1"
            keys.push({
                frame: 0,
                value: 0
            });

            // At the animation key 50, the value of scaling is "0.2"
            keys.push({
                frame: 50,
                value: Math.PI
            });

            // At the animation key 100, the value of scaling is "1"
            keys.push({
                frame: 100,
                value: 0
            });

            // Launch animation
            animation.setKeys(keys);
            fountain.animations.push(animation);
            scene.beginAnimation(fountain, 0, 100, true);

            return scene;
        }`

I changed a small logic than the SpitreManager because of the following issue
#2759

@deltakosh ping

another question why the deploy/netlify version in PR checks is referencing https://preview.babylonjs.com/babylon.js not the current PR babylon.js version ? :)

No worry for netlify it is more for the final merge :)
Regarding your PR, this is excellent.
Some comments:

The additional vertex data should be added only if particles are set to animate.
The sheet animation should be per particle. The particle should start with one cell and evolve during its lifetime to a final cell. We can then decide to loop or not. This should be done when updating a particle. The cell id should be incremented using a defined speed (like every x ticks). so that every particle could be on a different cell id.

I hope I'm clear and once again THANKS for your great contribution

You're clear my friend :)
Regarding second point, ParticleSystem doesn't expose it's particles like what spriteManager does which you can control every sprite to use which animation.
In this implementation I'm animating all at the same time not every particle alone to not repeat same operation on all particles.
Do you want to expose the particles ? if yes what would be the difference between the particles and sprites ? :)

you want each particle to animate differently than the other but randomly, so no need to expose the particles ?

Actually it is quite like the spritemanager where each sprite can have it's own cellID.
Regarding random I do not need random here actually :)
the particle system can define the following properties:

firstCell
LastCell
Loop
Speed

Based on that everytime a particle is generated its cellID is set to firstCell, then based on speed the cellID will be incremented until we reach the lastCell. Then depending on loop you restart or not :)

Alright :) make sense.

testing code
`// Create a particle system
// var particleSystem = new BABYLON.ParticleSystem("particles", 2000, scene);
// particleSystem.particleTexture = new BABYLON.Texture("textures/flare.png", scene, true, false, BABYLON.Texture.TRILINEAR_SAMPLINGMODE);;
            var particleSystem = new BABYLON.ParticleSystem("particles", 2000, scene, null, 64);
            particleSystem.particleTexture = new BABYLON.Texture("textures/player.png", scene, true,
                false, BABYLON.Texture.TRILINEAR_SAMPLINGMODE);;
            particleSystem.playAnimation(0, 44, false, 100);
            particleSystem.disposeParticlesWhenFinishedAnimating = true;
            //particleSystem.invertU=1;
            //particleSystem.invertV=1;
            //particleSystem.cellIndex = 44;`

I used the property delay instead of speed similar to SpriteManager

I still need to add an option to stop animation on particle

@deltakosh ping.

@deltakosh ping again :) feel free to comment

@deltakosh I absolutely have no problem with any comment as long as I'm learning or doing the right thing :) will start working on the comments .

I appreciate your good mindset

To test
` var particleSystem = new BABYLON.ParticleSystem("particles", 2000, scene, null, 64);
particleSystem.particleTexture = new BABYLON.Texture("textures/player.png", scene, true,
false, BABYLON.Texture.TRILINEAR_SAMPLINGMODE);
            particleSystem.startSpriteCellID = 0;
            particleSystem.endSpriteCellID = 44;
            particleSystem.spriteCellLoop = true;`

1)I didn't create spriteCellWidth\Height because I need this info at constructor level to know the length of the buffer.

I passed to vertex shader x (number of rows) y(number of columns) z(rowSize) not the cellWidth/Height as it was easier.

A question why you wanted to do more calculations on vertex shader rather than JS ?
Hope I hear LGTM :D @deltakosh

@deltakosh ping :)  you can test using
`
var particleSystem = new BABYLON.ParticleSystem("particles", 2000, scene);
particleSystem.particleTexture = new BABYLON.Texture("textures/flare.png", scene, true,
false, BABYLON.Texture.TRILINEAR_SAMPLINGMODE);;
            var particleSystem = new BABYLON.ParticleSystem("particles", 2000, scene, null, true);
            particleSystem.particleTexture = new BABYLON.Texture("textures/player.png", scene, true,
                false, BABYLON.Texture.TRILINEAR_SAMPLINGMODE);

            particleSystem.startSpriteCellID = 0;
            particleSystem.endSpriteCellID = 44;
            particleSystem.spriteCellHeight = 64;
            particleSystem.spriteCellWidth = 64;
            particleSystem.spriteCellLoop = true;

            //particleSystem.spriteCellChangeSpeed = 4;

`

@deltakosh done, will update the particle system overview page next weekend :) for the overview page I didn't update it because what's new.md at the repo root has v3.0 at the beginning so I didn't want to miss with it :D

Conflicts :( but LGTM :D

And this is the what's new I mentioned: https://github.com/BabylonJS/Babylon.js/blob/master/dist/preview release/what's new.md

@deltakosh done :)

Congrats  !!!! Now next step is Doc;)


jnp.ndarray unpacking yields onp.ndarray results

Example:
import jax.numpy as jnp
import numpy as np
a = jnp.arange(10).reshape((2, 5))

type(a)

jaxlib.xla_extension.DeviceArray

But if we do:
b, c = a

We get
type(b), type(c)

(numpy.ndarray, numpy.ndarray)

https://colab.research.google.com/gist/romanngg/bc4197d71a404f28224914a2b0dc4481/jnp-np-coversion-when-unpacking.ipynb

Hi - thanks for the issue! I think this is a duplicate of #1583, which has a potential fix in #3821
I can't quite recall why that fix stalled – but I recall there were some subtle aspects of this that made it a tricky fix.

I'll assign this to @mattjj as a light ping on the issue :grin:

This has been fixed


Design the unified suggestions menu

When the user inputs a query, they need to see:

a list of text fields where the query is found and in what amount (see #207)
a list of words often related to the query in the selected field (see #223)

This info has to be presented at the same time, so the idea is to have a menu that composites these 2 lists in a nice way.

Attaching the discussed designs.

Selecting a field by clicking on it or navigating with arrows shows suggestions.
Eventually thir can also be done by hovering fields, so that the user will have a preview of suggested words if they select a certain field, depending on the the speed required to make the query to get suggestions.
In there designs I have used colors to distinguisn functions:

black: typing a query, suggestions, search
blue: fields analytics

making it apparent how the two macro columns (fields & suggestions) seem to be swapped.
If we feel we want to visually associate the query with the sugested words, it's easy enough to swap them later after some internal user testing.
For now this order seems to be the most logical one.
Depending on this, later on we'll also adjust the widget colors.

Attaching the discussed designs.

Selecting a field by clicking on it or navigating with arrows shows suggestions.
Eventually thir can also be done by hovering fields, so that the user will have a preview of suggested words if they select a certain field, depending on the the speed required to make the query to get suggestions.
In there designs I have used colors to distinguisn functions:

black: typing a query, suggestions, search
blue: fields analytics

making it apparent how the two macro columns (fields & suggestions) seem to be swapped.
If we feel we want to visually associate the query with the sugested words, it's easy enough to swap them later after some internal user testing.
For now this order seems to be the most logical one.
Depending on this, later on we'll also adjust the widget colors.

Closing as I'm done with this task

Closing as I'm done with this task


Update version: Ruben2776.PicView version 2.0

Pull request has been created with WinGet Automation :rocket:
Microsoft Reviewers: Open in CodeFlow

Automatic Validation ended with:

Standard error:  2023-11-22 11:31:59.492   Exception message:
Internal error: Cannot expand "commonpf64" constant on this version of Windows.


Loop of All Sky Starting then failing

It seems that me, and Patriot Astro, have ran into a new issue where everything works, but in the GUI it will say "starting up" then give a "not running" error and do that cycle a few times.

@DamoclesSword, please check in the log file /var/log/allsky.log.  AllSky is starting, getting an error of some type, exiting, then restarting, getting the error, etc.
If you see ASI_ERROR_TIMEOUT messages in the log file, please make sure you have the latest capture.cpp and camera_options_ZWO.json files, and in the GUI, go to the bottom, click on "Show advanced options", and make sure "Version 0.8 Exposure" is UNchecked.

Try applying the patch from #469. It won't fix your problem, but maybe it'll tell you what's failing.

@DamoclesSword, please check in the log file /var/log/allsky.log. AllSky is starting, getting an error of some type, exiting, then restarting, getting the error, etc.
If you see ASI_ERROR_TIMEOUT messages in the log file, please make sure you have the latest capture.cpp and camera_options_ZWO.json files, and in the GUI, go to the bottom, click on "Show advanced options", and make sure "Version 0.8 Exposure" is UNchecked.

I finally got time to check. Here is the log:
Sep  7 20:08:26 allsky allsky.sh[2377]:      ***** Starting AllSky *****
Sep  7 20:08:26 allsky allsky.sh[2377]: Making sure allsky.sh is not already running...
Sep  7 20:08:26 allsky allsky.sh[2377]: Trying to automatically choose between ZWO and RPI camera
Sep  7 20:08:26 allsky allsky.sh[2377]: CAMERA: ZWO
Sep  7 20:08:26 allsky allsky.sh[2377]: CAMERA_SETTINGS: /home/pi/allsky/settings_ZWO.json
Sep  7 20:08:27 allsky allsky.sh[2377]: Starting allsky camera...
Sep  7 20:08:43 allsky allsky.sh[498]:      ***** Starting AllSky *****
Sep  7 20:08:43 allsky allsky.sh[498]: Making sure allsky.sh is not already running...
Sep  7 20:08:44 allsky allsky.sh[498]: Trying to automatically choose between ZWO and RPI camera
Sep  7 20:08:44 allsky allsky.sh[498]: None of RPI or ZWO Cameras were found. Exiting.
I dont see any clear error.
As for Ckuethe, I am not sure how to do what you suggest. I am new to all of this.

"None of RPI or ZWO Cameras were found. Exiting." You haven't said what kind of camera you're using.
If you are using a ZWO camera, does it show up when you run lsusb, eg. ID 03c3:120d ZWO ASI120MM-S? If you're using the rpi camera, does vcgencmd get_camera say supported=1 detected=1

Ive tried it with a 120MC-S and 178MM

Sorry didnt see the other half. As for that "run lsusb, eg. ID 03c3:120d ZWO ASI120MM-S? If you're using the rpi camera, does vcgencmd get_camera say supported=1 detected=1" I have no idea how to do that

In the same way that you installed the allsky software, you'll need to open up a shell to run those commands. Just type lsusb or vcgencmd get_camera and see what happens.

When running lsusb, I get:
Bus 001 Device 002: ID 03c3:178c
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
and with the get_camera it says zero for both. Granted, this is using my 178MM and not my 120MC-S

The new error log shows:
Sep 12 16:50:19 allsky allsky.sh[2265]: Stop the allsky service to end this process.
Sep 12 16:50:19 allsky allsky.sh[2265]: ==========
Sep 12 16:50:19 allsky allsky.sh[2265]: === Starting daytime capture ===
Sep 12 16:50:19 allsky allsky.sh[2265]: ==========
Sep 12 16:50:19 allsky allsky.sh[2265]: Saving images with delay of 5,000 ms (5 sec)
Sep 12 16:50:19 allsky allsky.sh[2265]:   > ERROR: Failed getting image, status = 11 (ASI_ERROR_TIMEOUT)
Sep 12 16:50:22 allsky allsky.sh[2265]:   > ERROR: Failed getting image, status = 11 (ASI_ERROR_TIMEOUT)
Sep 12 16:50:24 allsky allsky.sh[2265]:   > ERROR: Failed getting image, status = 11 (ASI_ERROR_TIMEOUT)
Sep 12 16:50:30 allsky allsky.sh[2265]:      ***** Stopping AllSky *****
Sep 12 16:50:30 allsky allsky.sh[2265]: capture exited with retcode=2

Progress, then. Your camera is detected. Now try Eric's suggestion and see if changing the 0.8 Exposure makes any difference.

It was already turned off. So no change besides the new error log

@DamoclesSword,
Please try the attached file. It's the version 0.7 capture.cpp that knows about the new option names (e.g., "daybrightness" instead of "brightness") that were added in version 0.8. It does not use the "Version 0.8 exposure" mode.
Unpack the file, rename the .cpp file to "capture.cpp", put in ~/allsky, then "cd allsky ; make capture" and restart AllSky. It won't give any errors in the log file but you'll know it's working if you get images.
Let me know how it goes.
Eric
capture0.7with0.8arguments.zip

@DamoclesSword,
what did you do to have the Pi recognize your camera?  Your first attempt didn't even find the camera but the second attempt did.
Eric

@DamoclesSword,
any update?  If you have things working with the newest code, please close this issue.

Closing issue, never heard back from user.


changes to properly get userstr when RBAC is enabled and set it in threadcontext

https://github.com/opensearch-project/alerting/issues/698
Description of changes:
userstr is properly obtained and set in threadcontext.
CheckList:
[ ] Commits are signed per the DCO using --signoff
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

PR #852 got merged and resolved the issue.


Wrong Heart Rate Calculation

Hello Pedro, congrats for your work!
I think there is something wrong with the time_domain.heart_rate function when I use NNI as input. When I use the biosppy.ecg function and calculate the mean of my input signal the result is 80.24485892889525. With your time_domain function the mean result is<PHONE_NUMBER>577905 (exactly twice).
Also, I think that the documentation of this function (heart_rate) is not right.
Currently return is: hr (array): Series of NN intervals in [ms].
While I think should be: hr (array): Series of Heart Rate in [bpm].
Thank you again Pedro!

I used the rpeaks parameter instead of the heart_rate_ts parameter of the ecg function from biosppy. The heart_rate_ts works when sample frequency is not 1kHz.


Unlisted package page should not allow downloading or provide the snippet to install it

Today, if a package is unlisted by the owner, they do not show up in nuget.org search. However, search engines and other nuget package indexes may continue to point to provide a link to the package page when you search for the package. Once on the page, you can either download the nupkg or install it using the PMC using the conveniently provided snippet.
By unlisting a package, the intent of the author is to stop more users from taking a dependency on it without breaking existing users. To that effect, I believe the download link and the PMC install snippet should be removed for unlisted packages.

@anangaur ?

Unlisting is meant to prevent discovery. I think hiding the download link or install snippet on unlisted packages is only going to create customer frustration because they've already found it.

Don't have very strong opinion here but I tend to agree with @karann-msft here. The package, though found states that its unlisted and should not be used but then we provide methods/ways to use it. Kind of conflicting info IMO.

this could become a part of the deprecate feature :)
https://github.com/NuGet/Home/issues/2867

@karann-msft Thats right. Its already part of my proposal.

closing as dupe of NuGet/Home#2867


matplotlib: RuntimeError: Could not find the matplotlib data files

Versions
$ python3 -m nuitka --version

<IP_ADDRESS>
Python: 3.6.8 (default, Apr 25 2019, 21:02:35)
Executable: /usr/bin/python3
OS: Linux
Arch: x86_64

>>> sys.version

'3.6.8 (default, Apr 25 2019, 21:02:35) \n[GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]'

$ pip3 show matplotlib

Name: matplotlib
Version: 3.0.3
Summary: Python plotting package
Home-page: http://matplotlib.org
Author: John D. Hunter, Michael Droettboom
Author-email<EMAIL_ADDRESS>License: PSF
Location: /usr/local/lib64/python3.6/site-packages
Requires: cycler, numpy, kiwisolver, python-dateutil, pyparsing
Required-by:

Installation
$ yum install python36
$ pip3 install Nuitka
$ pip3 install matplotlib
Issue
I had a much bigger script but boiled it down to this gem of a one-liner:
import matplotlib.pyplot as plt
It compiled, although I did get the following warnings just prior to scons:
$ python3 -m nuitka --standalone --show-progress --show-scons test_matplotlib.py

...
Nuitka:INFO:Optimizing module '_codecs_hk', 2 more modules to go after that.
Nuitka:INFO:Optimizing module '_codecs_cn', 1 more modules to go after that.
Nuitka:INFO:Optimizing module '_bz2', 0 more modules to go after that.
Nuitka:WARNING:Unresolved '__import__' call at '/usr/local/lib64/python3.6/site-packages/numpy/core/function_base.py:453' may require use of '--include-plugin-directory' or '--include-plugin-files'.
Nuitka:WARNING:Unresolved '__import__' call at '/usr/local/lib64/python3.6/site-packages/numpy/lib/utils.py:366' may require use of '--include-plugin-directory' or '--include-plugin-files'.
Nuitka:WARNING:Unresolved '__import__' call at '/usr/local/lib64/python3.6/site-packages/numpy/lib/utils.py:865' may require use of '--include-plugin-directory' or '--include-plugin-files'.
Nuitka:WARNING:Unresolved '__import__' call at '/usr/local/lib64/python3.6/site-packages/numpy/lib/utils.py:923' may require use of '--include-plugin-directory' or '--include-plugin-files'.
Nuitka:INFO:Total memory usage before running scons: 775.68 MB (813355008 bytes):
...

Upon trying to run I get:

Traceback (most recent call last):
File "/test/test_matplotlib.dist/test_matplotlib.py", line 1, in
File "", line 971, in _find_and_load
File "", line 955, in _find_and_load_unlocked
File "", line 656, in _load_unlocked
File "", line 626, in _load_backward_compatible
File "/test/test_matplotlib.dist/matplotlib/__init__.py", line 1099, in
File "/test/test_matplotlib.dist/matplotlib/__init__.py", line 940, in rc_params
File "/test/test_matplotlib.dist/matplotlib/__init__.py", line 746, in matplotlib_fname
File "/test/test_matplotlib.dist/matplotlib/__init__.py", line 744, in gen_candidates
File "/test/test_matplotlib.dist/matplotlib/__init__.py", line 413, in wrapper
File "/test/test_matplotlib.dist/matplotlib/__init__.py", line 690, in get_data_path
File "/test/test_matplotlib.dist/matplotlib/__init__.py", line 684, in _get_data_path
RuntimeError: Could not find the matplotlib data files

I will need to modify the numpy plugin somewhat so these data get copied.
No big deal.

Also getting this issue, any ideas on a fix?

As a temporary work around you can do the following:
cd /usr
find -name "mpl-data"
# mine shows ./local/lib64/python3.6/site-packages/matplotlib/mpl-data

in your python script add the following:
os.environ['MATPLOTLIBDATA'] = '/usr/local/lib64/python3.6/site-packages/matplotlib/mpl-data'`

Depending on what you're doing with matplotlib, you may have to add some more imports depending on what complains to you at runtime... I had to do the following as well:
import matplotlib.backends.backend_tkagg
import matplotlib.backends.backend_svg

(You don't have to actually do anything with those imports, they're just there to get nuitka to mark them as a needed import)
A little painful, but shrugs either that or wait around until it's fixed :)

Please give me a bit of more time:
The required changes to several plugins are underway.
I will then upload a ZIP file which you can use to (1) temporarily overwrite the plugins folder of your Nuitka installation, and (2) it will contain the required updated tools for standalone-compiling matplotlib, cv2 and tensorflow scripts.

please also refer to this.

@NathanVasquez - what is your current status?
Your issue should be addressed with the current nuitka version, resp. numpy plugin version ...

I ran into the same matplotlib data problem.
What version should address the problem?
I tried the current developer version (0.6.4rc6), but got the same result for the proposed minimal example from the first comment here.

@jakams - during compile: was there the message Copied 'mpl-data' from 'matplotlib' installation.?
If not, you were not using the correct compile command:
You should use python get-hints.py yourscript.py first. After that do python nuitka-hints.py yourscript.py.
The first step creates a logfile of all imports your script is doing. The second (the actual compile) is using this log to determine the required plugins and any additional plugin options to configure the appropriate compile parameters.

I didn't do it this way. I'll try it now.

I cant find much info on hinting...
Is there a hinting file missing in your comment: https://github.com/Nuitka/NUITKA-Utilities/issues/10#issuecomment-493712297
I found and used hinting.rar from 4 days ago (https://github.com/Nuitka/Nuitka/issues/364#issuecomment-498027642)
I get the following error:

D:\test>python get-hints.py test_matplotlib.py
Call cleaning has removed 496 items.

D:\test>python nuitka-hints.py test_matplotlib.py
NUITKA is compiling 'test_matplotlib.py' with these options:
--standalone
--remove-output
--experimental=use_pefile
--recurse-none
--disable-dll-dependency-cache
Nuitka:INFO: 'hinted-mods.py' is adding the following options:
Nuitka:INFO: --enable-plugin=numpy=matplotlib
Nuitka:INFO: --enable-plugin=tk-inter
Nuitka:INFO: --recurse-to for 248 imported modules.
Nuitka:INFO:
Nuitka:INFO:User plugin 'hinted-mods.py' loaded.
Nuitka:INFO:Injecting plug-in based pre load code for module 'tkinter':
Nuitka:INFO:    Need to make sure we set environment variables for TCL.
Nuitka:INFO: drop site
Nuitka:INFO: drop IPython
Problem with statement at C:\Users\Jaka\Anaconda3\lib\site-packages\matplotlib\p
yplot.py:122:
->             from IPython.core.pylabtools import backend2gui
Problem with statement at C:\Users\Jaka\Anaconda3\lib\site-packages\matplotlib\p
yplot.py:98:
->         if 'IPython' in sys.modules:
Problem with statement at C:\Users\Jaka\Anaconda3\lib\site-packages\matplotlib\p
yplot.py:97:
->     try:
Problem with statement at C:\Users\Jaka\Anaconda3\lib\site-packages\matplotlib\p
yplot.py:82:
-> def install_repl_displayhook():
Problem with statement at C:\Users\Jaka\Anaconda3\lib\site-packages\matplotlib\p
yplot.py:82:
-> def install_repl_displayhook():
Nuitka:INFO:Interrupted while working on '<Node 'COMPILED_PYTHON_MODULE' with {'
filename': 'C:\Users\Jaka\Anaconda3\lib\site-packages\matplotlib\pyplot.p
y', 'package': 'matplotlib', 'name': 'pyplot'}>'.
Traceback (most recent call last):
File "C:\Users\Jaka\Anaconda3\lib\site-packages\nuitka-0.6.4rc6-py3.6.egg\nuit
ka_main_.py", line 184, in
main()
File "C:\Users\Jaka\Anaconda3\lib\site-packages\nuitka-0.6.4rc6-py3.6.egg\nuit
ka_main_.py", line 177, in main
MainControl.main()
File "C:\Users\Jaka\Anaconda3\lib\site-packages\nuitka-0.6.4rc6-py3.6.egg\nuit
ka\MainControl.py", line 746, in main
main_module = createNodeTree(filename=filename)
File "C:\Users\Jaka\Anaconda3\lib\site-packages\nuitka-0.6.4rc6-py3.6.egg\nuit
ka\MainControl.py", line 141, in createNodeTree
Optimization.optimize(main_module.getOutputFilename())
File "C:\Users\Jaka\Anaconda3\lib\site-packages\nuitka-0.6.4rc6-py3.6.egg\nuit
ka\optimizations\Optimization.py", line 542, in optimize
makeOptimizationPass(initial_pass=True)
File "C:\Users\Jaka\Anaconda3\lib\site-packages\nuitka-0.6.4rc6-py3.6.egg\nuit
ka\optimizations\Optimization.py", line 456, in makeOptimizationPass
changed = optimizeModule(current_module)
File "C:\Users\Jaka\Anaconda3\lib\site-packages\nuitka-0.6.4rc6-py3.6.egg\nuit
ka\optimizations\Optimization.py", line 167, in optimizeModule
changed = optimizeCompiledPythonModule(module)
File "C:\Users\Jaka\Anaconda3\lib\site-packages\nuitka-0.6.4rc6-py3.6.egg\nuit
ka\optimizations\Optimization.py", line 100, in optimizeCompiledPythonModule
module.computeModule()
File "C:\Users\Jaka\Anaconda3\lib\site-packages\nuitka-0.6.4rc6-py3.6.egg\nuit
ka\nodes\ModuleNodes.py", line 489, in computeModule
trace_collection=self.trace_collection
File "C:\Users\Jaka\Anaconda3\lib\site-packages\nuitka-0.6.4rc6-py3.6.egg\nuit
ka\nodes\StatementNodes.py", line 165, in computeStatementsSequence
new_statement = statement.computeStatementsSequence(trace_collection)
File "C:\Users\Jaka\Anaconda3\lib\site-packages\nuitka-0.6.4rc6-py3.6.egg\nuit
ka\nodes\FrameNodes.py", line 182, in computeStatementsSequence
new_statement = trace_collection.onStatement(statement=statement)
File "C:\Users\Jaka\Anaconda3\lib\site-packages\nuitka-0.6.4rc6-py3.6.egg\nuit
ka\optimizations\TraceCollections.py", line 550, in onStatement
new_statement, change_tags, change_desc = statement.computeStatement(self)
File "C:\Users\Jaka\Anaconda3\lib\site-packages\nuitka-0.6.4rc6-py3.6.egg\nuit
ka\nodes\AssignNodes.py", line 267, in computeStatement
trace_collection.onExpression(self.getAssignSource())
File "C:\Users\Jaka\Anaconda3\lib\site-packages\nuitka-0.6.4rc6-py3.6.egg\nuit
ka\optimizations\TraceCollections.py", line 531, in onExpression
r = expression.computeExpressionRaw(trace_collection=self)
File "C:\Users\Jaka\Anaconda3\lib\site-packages\nuitka-0.6.4rc6-py3.6.egg\nuit
ka\nodes\ExpressionBases.py", line 980, in computeExpressionRaw
expression = trace_collection.onExpression(sub_expression)
File "C:\Users\Jaka\Anaconda3\lib\site-packages\nuitka-0.6.4rc6-py3.6.egg\nuit
ka\optimizations\TraceCollections.py", line 531, in onExpression
r = expression.computeExpressionRaw(trace_collection=self)
File "C:\Users\Jaka\Anaconda3\lib\site-packages\nuitka-0.6.4rc6-py3.6.egg\nuit
ka\nodes\FunctionNodes.py", line 830, in computeExpressionRaw
function_body.computeFunctionRaw(trace_collection)
File "C:\Users\Jaka\Anaconda3\lib\site-packages\nuitka-0.6.4rc6-py3.6.egg\nuit
ka\nodes\FunctionNodes.py", line 384, in computeFunctionRaw
self.computeFunction(trace_collection)
File "C:\Users\Jaka\Anaconda3\lib\site-packages\nuitka-0.6.4rc6-py3.6.egg\nuit
ka\nodes\FunctionNodes.py", line 397, in computeFunction
trace_collection=trace_collection
File "C:\Users\Jaka\Anaconda3\lib\site-packages\nuitka-0.6.4rc6-py3.6.egg\nuit
ka\nodes\StatementNodes.py", line 167, in computeStatementsSequence
new_statement = trace_collection.onStatement(statement=statement)
File "C:\Users\Jaka\Anaconda3\lib\site-packages\nuitka-0.6.4rc6-py3.6.egg\nuit
ka\optimizations\TraceCollections.py", line 550, in onStatement
new_statement, change_tags, change_desc = statement.computeStatement(self)
File "C:\Users\Jaka\Anaconda3\lib\site-packages\nuitka-0.6.4rc6-py3.6.egg\nuit
ka\nodes\TryNodes.py", line 117, in computeStatement
result = tried.computeStatementsSequence(trace_collection=trace_collection)
File "C:\Users\Jaka\Anaconda3\lib\site-packages\nuitka-0.6.4rc6-py3.6.egg\nuit
ka\nodes\StatementNodes.py", line 165, in computeStatementsSequence
new_statement = statement.computeStatementsSequence(trace_collection)
File "C:\Users\Jaka\Anaconda3\lib\site-packages\nuitka-0.6.4rc6-py3.6.egg\nuit
ka\nodes\FrameNodes.py", line 182, in computeStatementsSequence
new_statement = trace_collection.onStatement(statement=statement)
File "C:\Users\Jaka\Anaconda3\lib\site-packages\nuitka-0.6.4rc6-py3.6.egg\nuit
ka\optimizations\TraceCollections.py", line 550, in onStatement
new_statement, change_tags, change_desc = statement.computeStatement(self)
File "C:\Users\Jaka\Anaconda3\lib\site-packages\nuitka-0.6.4rc6-py3.6.egg\nuit
ka\nodes\TryNodes.py", line 117, in computeStatement
result = tried.computeStatementsSequence(trace_collection=trace_collection)
File "C:\Users\Jaka\Anaconda3\lib\site-packages\nuitka-0.6.4rc6-py3.6.egg\nuit
ka\nodes\StatementNodes.py", line 167, in computeStatementsSequence
new_statement = trace_collection.onStatement(statement=statement)
File "C:\Users\Jaka\Anaconda3\lib\site-packages\nuitka-0.6.4rc6-py3.6.egg\nuit
ka\optimizations\TraceCollections.py", line 550, in onStatement
new_statement, change_tags, change_desc = statement.computeStatement(self)
File "C:\Users\Jaka\Anaconda3\lib\site-packages\nuitka-0.6.4rc6-py3.6.egg\nuit
ka\nodes\ConditionalNodes.py", line 576, in computeStatement
branch_yes_collection.computeBranch(branch=yes_branch)
File "C:\Users\Jaka\Anaconda3\lib\site-packages\nuitka-0.6.4rc6-py3.6.egg\nuit
ka\optimizations\TraceCollections.py", line 730, in computeBranch
result = branch.computeStatementsSequence(trace_collection=self)
File "C:\Users\Jaka\Anaconda3\lib\site-packages\nuitka-0.6.4rc6-py3.6.egg\nuit
ka\nodes\StatementNodes.py", line 167, in computeStatementsSequence
new_statement = trace_collection.onStatement(statement=statement)
File "C:\Users\Jaka\Anaconda3\lib\site-packages\nuitka-0.6.4rc6-py3.6.egg\nuit
ka\optimizations\TraceCollections.py", line 550, in onStatement
new_statement, change_tags, change_desc = statement.computeStatement(self)
File "C:\Users\Jaka\Anaconda3\lib\site-packages\nuitka-0.6.4rc6-py3.6.egg\nuit
ka\nodes\AssignNodes.py", line 267, in computeStatement
trace_collection.onExpression(self.getAssignSource())
File "C:\Users\Jaka\Anaconda3\lib\site-packages\nuitka-0.6.4rc6-py3.6.egg\nuit
ka\optimizations\TraceCollections.py", line 531, in onExpression
r = expression.computeExpressionRaw(trace_collection=self)
File "C:\Users\Jaka\Anaconda3\lib\site-packages\nuitka-0.6.4rc6-py3.6.egg\nuit
ka\nodes\ExpressionBases.py", line 1050, in computeExpressionRaw
expression = trace_collection.onExpression(expression=sub_expression)
File "C:\Users\Jaka\Anaconda3\lib\site-packages\nuitka-0.6.4rc6-py3.6.egg\nuit
ka\optimizations\TraceCollections.py", line 531, in onExpression
r = expression.computeExpressionRaw(trace_collection=self)
File "C:\Users\Jaka\Anaconda3\lib\site-packages\nuitka-0.6.4rc6-py3.6.egg\nuit
ka\nodes\ExpressionBases.py", line 999, in computeExpressionRaw
return self.computeExpression(trace_collection=trace_collection)
File "C:\Users\Jaka\Anaconda3\lib\site-packages\nuitka-0.6.4rc6-py3.6.egg\nuit
ka\nodes\ImportNodes.py", line 401, in computeExpression
trace_collection=trace_collection, module_name=imported_module_name
File "C:\Users\Jaka\Anaconda3\lib\site-packages\nuitka-0.6.4rc6-py3.6.egg\nuit
ka\nodes\ImportNodes.py", line 280, in _attemptRecursion
module_package=module_package,
File "C:\Users\Jaka\Anaconda3\lib\site-packages\nuitka-0.6.4rc6-py3.6.egg\nuit
ka\nodes\ImportNodes.py", line 199, in _consider
module_kind=module_kind,
File "C:\Users\Jaka\Anaconda3\lib\site-packages\nuitka-0.6.4rc6-py3.6.egg\nuit
ka\importing\Recursion.py", line 181, in decideRecursion
module_filename, module_name, module_package, module_kind
File "C:\Users\Jaka\Anaconda3\lib\site-packages\nuitka-0.6.4rc6-py3.6.egg\nuit
ka\plugins\Plugins.py", line 240, in onModuleEncounter
module_filename, module_name, module_package, module_kind
File "hinted-mods.py", line 294, in onModuleEncounter
import_set = self.ImplicitImports.getImportsByFullname(module_package)
AttributeError: 'NuitkaPluginPopularImplicitImports' object has no attribute 'ge
tImportsByFullname'

Hm, your scripts are correctly configured, but one component in your Nuitka version is incorrect. Don't know why this is so. I recommend you take the plugins directory contained in the following ZIP and copy it over to your installation's Nuitka plugins folder - just to see if this solves your issue.
Independently we will look into what is happening in the Nuitka distribution.
hinting.zip

But as you can see from your (unsuccessful) compile log: the numpy plugin must be used with option matplotlib, and you also need plugin tk-inter.
If you compile your script in the "normal" way mimiking these options, you should be successful ok ...

What the hinting logic does:

exclude everything except stuff which your script really needs during its execution

Overwriting nuitka's plugins seems to have done it.
compiling via hints seems to work, as well as using:

python -m nuitka --standalone --remove-output --experimental=use_pefile --disable-dll-dependency-cache --enable-plugin=numpy=matplotlib --enable-plugin=tk-inter test_matplotlib.py

which is based on the hints.
Hints seem very useful :).
I am off to attempt my original compilation...
By the way, maybe this helps someone, so I am posting it...
 One of them was that I tried compiling on a different computer, that had an older matplotlib version (2.2.2).
It successfully compiled using hints, but running it produced:

ModuleNotFoundError: No module named 'matplotlib.backends.backend_qt5agg'

I found:

Nuitka:WARNING: Not recursing to unused 'matplotlib.backends.backend_qt5agg'

in the compilation warnings.
This seems to have gotten solved by upgrading matplotlib to the current 3.1.0. version.

Thanks for the feedback.

I think mpl-data and all it includes needs to be added to nuitka similarly to this: https://github.com/Nuitka/Nuitka/issues/215

I think mpl-data and all it includes needs to be added to nuitka similarly to this, because manual copying works: #215

The correct (!) version of numpy does that. Either you use hinted compiles or you specify --enable-plugin=numpy=matplotlib and it should work.
I sadly realized that release 0.0.4 and also the current development version did not pick up the most current plugins. If you want to try them out, copy the directory from this ZIP:
hinting.zip


Trust Chain Retrieval API Restrictions

Problem description
When using the /trust-chain api as highlighted in the swagger files, I noticed that only an Auditor role account is allowed to use it. This seems an odd choice as the trust chain is relevant for other user roles such as Project Proponents and VVB's as examples. Even the Standard Registries cannot use this API, and instead use other API's in order to see the blocks more directly from the policies api. These API's are locked to the Standard Registry role.
These restrictions don't necessarily reflect user flows as they would be expected with relation to trust chains. The Auditor role restriction for the /trust-chain API for example restricting access for the Project Proponent means that there are limited options available for a data producer to monitor the state of trust chains and minting processes that are a direct result of the data that they are producing. From the Demia end, as highlighted in this HIP proposal discussion, we are looking to connect the data producers into the submission process via our platform, but in order to retrieve the results of those submissions, our user instance has to be tied to the Registry or Auditor roles (for the blocks search or trust-chain api calls respectively), which doesn't really make sense as they are more aligned with a Project Proponent role.
A middle-ware solution can be used as a workaround process to navigate the trust chains within a guardian instance by tying into the Auditor or Registry roles via a third party API implementation, but this enforces a search through all trust chains within the connected Guardian instance to extract relevant information and return it to the end user, rather than the end user interact with the instance directly through their own guardian credentials. This extra step would not be required if the API restrictions are reduced, and Project Proponents can navigate either the policy blocks or the trust-chain api's themselves.
Requirements
The trust-chain api should not be restricted to only the Auditor role, and/or the policy block api's should not be restricted to only the Standard Registry role.
Definition of done

[ ] API's become accessible from other roles
[ ] A better documentation process of trust chain retrieval from different roles or documentation on how to provide those privileges within the policy generation for the specified roles

Acceptance criteria

[ ] Clear definition of which API's can be used by which user roles and why there are restrictions if there are any
[ ] Examples/Tests demonstrating the ability to access relevant policy and trust chain information from different user roles defined within a policy

@anvabr to check if this is possible via roles permissions functions to Demia's and others satisfaction. cc @DyrellC @sergmetelin @Neurone

@DyrellC @dubgeis This is a static end-point which is designed specifically for auditor access. The real (flexible) access to the trust chain that for all other policy users is managed by the policy itself. There are relevant blocks 1 and 2 which provide this information to which access is managed in the usual in-policy permissions/roles way.

@anvabr Yes I have come to learn this over time. The intention from your end was to make it so that the charts are accessible by querying blocks with appropriate permissions/roles assigned to the user account for retrieval correct as opposed to querying the trust_chain endpoint correct? Is there a reason the auditor's need access in this fashion as opposed to using the ReportBlock queries as well with an Auditor role?
I would perhaps suggest making that a bit more clear somewhere in the documentation that for project proponents or other non auditor role accounts, that retrieval of trust chains needs to be set up via a ReportBlock instead of the trust_chain endpoint, as it's pretty easy to confuse them.

The Auditor role access is a very old interface, it could be said to be obsolete now but we left it mostly for backward compatibility. We definitely need to update the docs, we'll do that for the upcoming release.  Thank you for noticing this @DyrellC !


Wireframe "messages"

Styling of basic UI messages

We have created page trying to show possible styling of basic status messages
https://www.loom.com/share/4aa338636b3a45289456cc1b7817b153
Figma file referenced
https://www.figma.com/proto/RRtoRLyr5Spf0KiyRfEclt/DMP-Tool-Wireframes?node-id=2295-1510&node-type=frame&t=eX1fgwI5TfzbQsgl-0&scaling=min-zoom&content-scaling=fixed&page-id=736%3A1383&starting-point-node-id=2295%3A1510&show-proto-sidebar=1


fix: add Goutam Nath

Changes proposed
added new profile Goutam Nath
Check List

[X] My code follows the code style of this project.
[ ] My change requires changes to the documentation.
[ ] I have updated the documentation accordingly.
[X] All new and existing tests passed.
[X] This PR does not contain plagiarized content.
[X] The title of my pull request is a short description of the requested changes.

@nathgoutam93 did you resolve the problem?

@nathgoutam93 did you resolve the problem?

yes, It is resolved, it was an issue with the git configuration for my forked repo. the author was set to a different Github handler instead of mine

ok thank you for letting me know 👍


Access to all diagrams

When I look at the readme for the lessons I think the images are brilliant! They really help me out and I'd love to be able to save them as a collection separate from the .mds and lessons
could there be a single folder containing all of these diagrams?
I think it would really help out myself and others

@Ben-Bromley I've made the change, and all the sketchnotes are now in their own /sketchnotes folder in the root. Let me know if you find any mistakes!


Help with installing into Nuxt.js

Nuxt.js Installation: Help Wanted
Hey 👋🏼
I know this is not the really a good kind of issue that I'm creating here, but since this amazing project is still relatively unknown and I couldn't find any help anywhere else on the web, I though I'd post it here.
But please close this issue if you don't want this type of "Help wanted" issue
I can't get your library installed with nuxt.js. I spend a few days already and thought, I'd see if anyone here can get it to work.
Problem
It is always saying
This dependency was not found:
* atropos/css
* atropos/vue

Problem Demonstration
https://codesandbox.io/s/broken-smoke-bjh6y?file=/nuxt.config.js
This is a sandbox demonstrating my issue.
Anyone help is much appreciated 🙏🏼

Hi @8BitJonny , how did you make it work with Nuxt3? Thank you in advance


Depend on emglken via npm, not git submodules

The web gulp task doesn't work unless you've inflated all of the git submodules with git submodule update --init --recursive; then you have to manually/separately run npm install in the src/upstream/emglken directory and npm run build. Then go grab a coffee… that build is gonna take a long time.
This is a very silly way to depend on emglken; emglken is available as an npm module. It should simply appear as an entry in package.json. (If you need to do local development of Parchment with emglken, developers can use npm link instead.)

I really like my current setup though. But I can see that it would be much more straightforward for anyone else to use it this way.
I'll look into npm link and see if it will work for me.
However even if it does, there could be times when I make breaking changes to emglken upstream and there may be a little delay before I publish a version to npm.

Thanks, you were right, this is a much better approach. Npm-linking is very easy.


Design a generic message format for all extensions to ignore unintended bytes.

Design a generic message format for all extensions to ignore unintended bytes.

So following up on #415 I was able to get a protobuf serialization on top of OpenSearch SteamInput/StreamOutput which was able to cleanly support backward compatible and forward compatible message.
I've pushed the code changes to my fork and a different branch: https://github.com/saratvemulapalli/OpenSearch/commit/3c506cb98faa27d8244c824c5b0f6f09e20c09e6
Here is an example sample message in OpenSearch:
message Extension {
  string name = 1;
  int32 id = 2;
  string description = 3;
}

Generate the proto serializer and deserializer:  protoc --java_out=server/src/main/java/ server/src/main/java/org/opensearch/extensions/proto/extension.proto
On the extension:

On same version as OpenSearch:

11:50:23.941 [opensearch[hello-world][generic][T#1]] INFO  org.opensearch.sdk.handlers.ExtensionsInitRequestHandler - ExtensionProto message: name: "FirstExtension"
id: 10
description: "Extension working with Proto"

OpenSearch in new version (added new field):

message Extension {
  string name = 1;
  int32 id = 2;
  string description = 3;
  int32 ephemeral_id = 4;
}

The extension could cleanly de-serialize the message though new extra bytes are sent, it knows how to ignore it.
11:50:23.941 [opensearch[hello-world][generic][T#1]] INFO  org.opensearch.sdk.handlers.ExtensionsInitRequestHandler - ExtensionProto message: name: "FirstExtension"
id: 10
description: "Extension working with Proto"
4:<PHONE_NUMBER>

OpenSearch is on older version and SDK is compiled with newer interfaces of OpenSearch:

message Extension {
  string name = 1;
}

Extension cleanly accepts partial information and can decide how to proceed.
11:50:23.941 [opensearch[hello-world][generic][T#1]] INFO  org.opensearch.sdk.handlers.ExtensionsInitRequestHandler - ExtensionProto message: name: "FirstExtension"


Create algolia-search-request and algolia-search-response packages

Context
instant-meilisearch is an adapter between instantsearch and algoliasearch.js.
The way it works is that instantsearch provides search requests compatible with the search method of algoliasearch.js. It then expects a search response in the format that is returned  by the search method of algoliasearch.js.
The role of instant-meilisearch is to transform the Algolia search requests into meilisearch search requests and then transform the meilisearch search response into an algoliasearch response format.
Problem
Since instant-meilisearch is the adapter of algolia, instant-meilisearch is used by autocomplete-client for the same purpose as described above: adapting the search request and response. This constitute the first problem, as instant-meilisearch also provides the possibility to send configurations specifically related to instant-meilisearch which are not relevant to autocomplete-client and autocomplete-client user agent always contains instant-meilisearch in its agents.
The second issue lies when the user wants to use instantsearch in a SSR environment. Typically, a user might want to create a first back-end search and build the website with the returned hits. By doing so, the first hits are referenced on the different search engines (SEO) and the initial load of the hits is way faster. This is a common implementation on e-commerce websites.
Currently, this is only possible if the user creates an algolia search request. Which means, using the search request parameters of algolia.
For example:
The user wants to build its website with an initial search. He needs to do the following:
const instantMeilisearchClient = instantMeilisearch('host', 'apiKey');

const algoliaSearchParams = {
    "indexName": "games",
    "params": {
      "facets": [
        "genres"
      ],
      "hitsPerPage": 6,
      "page": 0,
      "query": "",
    }
}

const response = instantMeilisearchClient.search([algoliaSearchParams]);

// use response as initial search in the instantSearch instance

As you can see, the search request was made using the algoliasearch format and not the meilisearch format. It would have been more intuitive if the user could make a request with Meilisearch search parameters:
const searchParams = {
   indexUid: "games"
   facets: [
        "genres"
      ],
   q: "",
   // etc...
}

Solution
To be able to provide this intuitive SSR experience, we need to extract two parts of instant-meilisearch. The algolia-search-request adapter and the algolia-search-response adapter.
By doing so, the user can now create a SSR initial search with Meilisearch search parameters:
import { algoliaSearchResponseClient} from '@meilisearch/algolia-search-response'

const searchClient = algoliaSearchResponseClient('host', 'apiKey')

const searchParams = {
   indexUid: "games"
   facets: [
        "genres"
      ],
   q: "",
   // etc...
}

const algoliaSearchResponse = await searchClient.search([searchParams]); // search response in an algoliasearch.js format

// use response as initial search in the instantSearch instance

By using the searchRequest adapter, autocomplete can also remove instant-meilisearch from its user-agents.
// in autocomplete sources
import { algoliaSearchRequestClient } from '@meilisearch/algolia-search-request'

const searchClient = algoliaSearchRequestClient('host', 'apiKey', { userAgent: ['autocomplete (vx.x.x)'] })

const algoliasearchParams = {
   indexName: "games",
   params: { query: "test" }
}

const meilisearchResponse = await searchClient.search([algoliaSearchParams])

In the above example, the user agent sends to Meilisearch the engine are now
meilisearch js (vx.x.x); autocomplete-client (v.x.x.x)

instead of
meilisearch js (vx.x.x); instant-meilisearch (vx.x.x) autocomplete-client (v.x.x.x)

Design choices.
How should we name both packages?

@meilisearch/algolia-search-request-adapter and @meilisearch/algolia-search-response-adapter ?

How should we name the methods that are served by these packages?

algoliaSearchRequestClient and algoliaSearchResponseClient ?

Todo

[ ] extract [search-request-adapter](https://github.com/meilisearch/meilisearch-js-plugins/tree/main/packages/instant-meilisearch/src/adapter/search-request-adapter] to its own @meilisearch/algolia-search-request-adapter package
[ ] extract search-response-adapter to its own @meilisearch/algolia-search-response-adapter package.
[ ] Handle pagination
[ ] Handle sorting adapter

@bidoubiwa, can you close it if there's no progress in half a year since this was created?


remove unused env vars in pilot

Signed-off-by: Shriram Rajagopalan<EMAIL_ADDRESS>

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: rshriram
If they are not already assigned, you can assign the PR to them by writing /assign @rshriram in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

pilot/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@rshriram: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-presubmit.sh
f94cf3d991805dd37e4010030f2fb4966ff2e24a
link
/test istio-presubmit

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@rshriram: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-presubmit.sh
f94cf3d991805dd37e4010030f2fb4966ff2e24a
link
/test istio-presubmit

prow/istio-unit-tests.sh
f94cf3d991805dd37e4010030f2fb4966ff2e24a
link
/test istio-unit-tests

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: costinm
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

pilot/OWNERS [costinm]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

New changes are detected. LGTM label has been removed.

@rshriram: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-presubmit.sh
f94cf3d991805dd37e4010030f2fb4966ff2e24a
link
/test istio-presubmit

prow/istio-unit-tests.sh
67d393601cf77338ef75f7ae4d257a3fc2cc7961
link
/test istio-unit-tests

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@rshriram: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-unit-tests.sh
67d393601cf77338ef75f7ae4d257a3fc2cc7961
link
/test istio-unit-tests

prow/istio-presubmit.sh
67d393601cf77338ef75f7ae4d257a3fc2cc7961
link
/test istio-presubmit

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/lgtm

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: costinm, hzxuzhonghu
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

pilot/OWNERS [costinm,hzxuzhonghu]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@rshriram: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/e2e-mixer-no_auth-mcp.sh
36af28d41eaa2463cab86eafd51578ba0a142a69
link
/test e2e-mixer-no_auth-mcp

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/test istio-unit-tests

@rshriram: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/e2e-mixer-no_auth-mcp.sh
36af28d41eaa2463cab86eafd51578ba0a142a69
link
/test e2e-mixer-no_auth-mcp

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.


Wall lockers can grab mobs

Plz no it's too jank.

as in sentient lockers can pull stuff or something?

what are the issues with it? it's predicted now, no?

what are the issues with it? it's predicted now, no?

When you leave you clip into the wall despite the offset and it doesn't handle stuff occupying the leaving tile properly.


Updating chef-provisioning-aws breaks chef-client -z functionality with ChefDK 0.4.0 (current version).

Hey guys,
I think there's an issue with the gem dependencies for chef-provisioning-aws that ran afoul of this issue with Cheffish while using the current Chef DK.
[2015-04-11T18:35:18-04:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)```

**To replicate on Mac OSX:**
* brew cask install chefdk - Installs the current chefdk package - chefdk-0.4.0-1.pkg
* chef verify
* Test a recipe using `chef-client -z simple.rb` or the like, as posted in the ReadME.md
* chef gem update chef-provisioning-aws
* Test a recipe using chef-client -z again.

**ChefDK 4.0 stock gems:**

$ chef gem list | grep chef
chef (12.0.3)
chef-dk (0.4.0)
chef-provisioning (0.18)
chef-provisioning-aws (0.2.1)
chef-provisioning-azure (0.1)
chef-provisioning-fog (0.12)
chef-provisioning-vagrant (0.8.1)
chef-vault (2.4.0)
chef-zero (3.2.1, 1.5.6)
cheffish (0.9.2)
chefspec (4.2.0)

**Update chef-provisioning-aws and dependencies.  List versions:**

[18:33:20] (2) jreid@JoesMacBookPro:~
$ chef gem update chef-provisioning-aws
Updating installed gems
Updating chef-provisioning-aws
Fetching: specinfra-2.28.1.gem (100%)
Successfully installed specinfra-2.28.1
Fetching: rspec-support-3.2.2.gem (100%)
Successfully installed rspec-support-3.2.2
Fetching: rspec-expectations-3.2.1.gem (100%)
Successfully installed rspec-expectations-3.2.1
Fetching: rspec-core-3.2.3.gem (100%)
Successfully installed rspec-core-3.2.3
Fetching: rspec-its-1.2.0.gem (100%)
Successfully installed rspec-its-1.2.0
Fetching: rspec-mocks-3.2.1.gem (100%)
Successfully installed rspec-mocks-3.2.1
Fetching: rspec-3.2.0.gem (100%)
Successfully installed rspec-3.2.0
Fetching: serverspec-2.14.1.gem (100%)
Successfully installed serverspec-2.14.1
Fetching: rspec_junit_formatter-0.2.0.gem (100%)
Successfully installed rspec_junit_formatter-0.2.0
Fetching: chef-zero-4.2.1.gem (100%)
Successfully installed chef-zero-4.2.1
Fetching: chef-12.2.1.gem (100%)
Successfully installed chef-12.2.1
Fetching: cheffish-1.1.2.gem (100%)
Successfully installed cheffish-1.1.2
Fetching: chef-provisioning-1.0.1.gem (100%)
Successfully installed chef-provisioning-1.0.1
Fetching: chef-provisioning-aws-1.0.4.gem (100%)
Successfully installed chef-provisioning-aws-1.0.4
Gems updated: chef chef-provisioning chef-provisioning-aws chef-zero cheffish rspec rspec-core rspec-expectations rspec-its rspec-mocks rspec-support rspec_junit_formatter serverspec specinfra
[18:34:01] (3) jreid@JoesMacBookPro:~
$ chef gem list | grep chef
chef (12.2.1, 12.0.3)
chef-dk (0.4.0)
chef-provisioning (1.0.1, 0.18)
chef-provisioning-aws (1.0.4, 0.2.1)
chef-provisioning-azure (0.1)
chef-provisioning-fog (0.12)
chef-provisioning-vagrant (0.8.1)
chef-vault (2.4.0)
chef-zero (4.2.1, 3.2.1, 1.5.6)
cheffish (1.1.2, 0.9.2)
chefspec (4.2.0)

**Impact:**
A relatively simple recipe like the following

require 'chef/provisioning/aws_driver'
machine 'chef_provisioning_test_node' do
tag 'itsa_me_mario'
recipe 'JO2O'
with_machine_options :bootstrap_options => {
:key_name => 'korrelate2012',
:image_id => 'ami-2c348844',
}
end

fails with `ERROR: undefined method `with_machine_options' for Chef::Resource::Machine` on chef-provisioning-aws (0.2.1), presumably do to the changes made since 0.2.1 was released:

[2015-04-11T18:31:20-04:00] FATAL: Stacktrace dumped to /Users/jreid/repo/chef-repo/.chef/local-mode-cache/cache/chef-stacktrace.out
Chef Client failed. 0 resources updated in 4.297599 seconds
[2015-04-11T18:31:20-04:00] ERROR: undefined method `with_machine_options' for Chef::Resource::Machine
[2015-04-11T18:31:20-04:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

But an update to chef-provisioning-aws 1.0.4 means a chef-client -z simple.rb fails with the following:

[2015-04-11T18:35:18-04:00] ERROR: Unable to activate cheffish-1.1.2, because chef-zero-3.2.1 conflicts with chef-zero (~> 4.0)
[2015-04-11T18:35:18-04:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

Regards,
Joe Reid

Realized this may be more appropriate in the chef-provisioning-aws project.  Created a duplicate ticket there.
Let me know, either way, and I will close the duplicate.

I am not sure I can speak to this officially but it was the same with the azure driver too when chef-provisioning-1.0 was released. Kept me up most of the night after ChefConf wondering why this went bad.
I have forgotten the specifics but it's something like the chef-provisioning releases are going to be tied to the latest client at all times because both require cheffish eventually. The other thing is you cannot just update the gems because of the way they are bundled.
This tool https://github.com/fnichol/chefdk-update-app provides the correct way of updating your chef client when using ChefDK
I haven't tested with recent cheffish updates though but this may be the same for you

@stuartpreston - Hey, thanks man!  That helped a lot.
I was able to use the chefdk-update-app to update the version of chef (and the chef-client) used by the ChefDK.
The steps I followed were:

Wipe gems and ChefDK install.
brew cask install chefdk
gem update chef-provisioning
use chef-dk-update-app to update ChefDK to chef 12.2.1.  Output posted here.

Posted the output here, and I'm now showing
$ chef-client -v
Chef: 12.2.1

Unfortunately I'm not all the way home.  The next run errors out with [2015-04-11T21:25:00-04:00] ERROR: Unable to activate chef-provisioning-aws-0.2.1, because chef-provisioning-1.0.1 conflicts with chef-provisioning (~> 0.9)
$ chef-client -z simple.rb
[2015-04-11T21:24:58-04:00] INFO: Started chef-zero at http://localhost:8889 with repository at /Users/jreid/repo/chef-repo
  One version per cookbook

[2015-04-11T21:24:58-04:00] INFO: Forking chef instance to converge...
Starting Chef Client, version 12.2.1
[2015-04-11T21:24:58-04:00] INFO: *** Chef 12.2.1 ***
[2015-04-11T21:24:58-04:00] INFO: Chef-client pid: 17270
[2015-04-11T21:25:00-04:00] INFO: Run List is []
[2015-04-11T21:25:00-04:00] INFO: Run List expands to []
[2015-04-11T21:25:00-04:00] INFO: Starting Chef Run for JoesMacBookPro.local
[2015-04-11T21:25:00-04:00] INFO: Running start handlers
[2015-04-11T21:25:00-04:00] INFO: Start handlers complete.
[2015-04-11T21:25:00-04:00] INFO: HTTP Request Returned 404 Not Found : Object not found: /reports/nodes/JoesMacBookPro.local/runs
resolving cookbooks for run list: []
[2015-04-11T21:25:00-04:00] INFO: Loading cookbooks []
Synchronizing Cookbooks:
Compiling Cookbooks...

Running handlers:
[2015-04-11T21:25:00-04:00] ERROR: Running exception handlers
Running handlers complete
[2015-04-11T21:25:00-04:00] ERROR: Exception handlers complete
[2015-04-11T21:25:00-04:00] FATAL: Stacktrace dumped to /Users/jreid/repo/chef-repo/.chef/local-mode-cache/cache/chef-stacktrace.out
Chef Client failed. 0 resources updated in 1.966828 seconds
[2015-04-11T21:25:00-04:00] ERROR: Unable to activate chef-provisioning-aws-0.2.1, because chef-provisioning-1.0.1 conflicts with chef-provisioning (~> 0.9)
[2015-04-11T21:25:00-04:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

Current gem list:

I'm not all the way home yet.  The next run errors out with [2015-04-11T21:25:00-04:00] ERROR: Unable to activate chef-provisioning-aws-0.2.1, because chef-provisioning-1.0.1 conflicts with chef-provisioning (~> 0.9)
$ chef-client -z simple.rb
[2015-04-11T21:24:58-04:00] INFO: Started chef-zero at http://localhost:8889 with repository at /Users/jreid/repo/chef-repo
  One version per cookbook

[2015-04-11T21:24:58-04:00] INFO: Forking chef instance to converge...
Starting Chef Client, version 12.2.1
[2015-04-11T21:24:58-04:00] INFO: *** Chef 12.2.1 ***
[2015-04-11T21:24:58-04:00] INFO: Chef-client pid: 17270
[2015-04-11T21:25:00-04:00] INFO: Run List is []
[2015-04-11T21:25:00-04:00] INFO: Run List expands to []
[2015-04-11T21:25:00-04:00] INFO: Starting Chef Run for JoesMacBookPro.local
[2015-04-11T21:25:00-04:00] INFO: Running start handlers
[2015-04-11T21:25:00-04:00] INFO: Start handlers complete.
[2015-04-11T21:25:00-04:00] INFO: HTTP Request Returned 404 Not Found : Object not found: /reports/nodes/JoesMacBookPro.local/runs
resolving cookbooks for run list: []
[2015-04-11T21:25:00-04:00] INFO: Loading cookbooks []
Synchronizing Cookbooks:
Compiling Cookbooks...

Running handlers:
[2015-04-11T21:25:00-04:00] ERROR: Running exception handlers
Running handlers complete
[2015-04-11T21:25:00-04:00] ERROR: Exception handlers complete
[2015-04-11T21:25:00-04:00] FATAL: Stacktrace dumped to /Users/jreid/repo/chef-repo/.chef/local-mode-cache/cache/chef-stacktrace.out
Chef Client failed. 0 resources updated in 1.966828 seconds
[2015-04-11T21:25:00-04:00] ERROR: Unable to activate chef-provisioning-aws-0.2.1, because chef-provisioning-1.0.1 conflicts with chef-provisioning (~> 0.9)
[2015-04-11T21:25:00-04:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

Current chef-provisioning gems:
$ chef gem list | grep chef-provisioning
chef-provisioning (1.0.1, 0.18)
chef-provisioning-aws (0.2.1)
chef-provisioning-azure (0.1)
chef-provisioning-fog (0.12)
chef-provisioning-vagrant (0.8.1)

I didn't think you could have a with_machine_options inside a machine block.
If you change '''with_machine_options''' to '''machine_options''' inside your machine block does that get you further?

@stuartpreston - Yep!  Thanks again.  That got me around this issue, and on to other things.
I'm going to leave this open for now.  Hopefully it can help other people.
As rapidly as chef-provisioning is developing, I would suggest throwing some documentation and troubleshooting tips on the ReadME.md until such time as this gets merged down to Chef core.  I could have used some updated examples, too.
I'm happy to submit a pull request and write this stuff up, if there's interest in my doing that?

@JoeReid-Korrelate Definitely - I'm sure PRs with documentation that helps others get started are very welcome.

@JoeReid-Korrelate Can you try again with the latest RC ChefDK I have been working on?  If you're on OSX, the download link is here

@tyler-ball Sure thing.  Might not get to it today, but some time tomorrow should work.

+1 having the same problem

@tyler-ball - Tough week at work, deploying constantly.  Haven't had a safe window to wipe out everything and try the RC yet.  I'll give it a shot this weekend.

I've used the latest RC ChefDK for OSX posted here and simply installed without wiping anything. Works.

I've used the latest RC ChefDK for OSX posted here and simply installed without wiping anything. Works.
Same.  Or, at least, I do not believe the error I get is related to this problem.
Thanks guys!

@tyler-ball - Is there a ubuubu (sic) version of the RC ChefDK I can grab?

There is an Ubuntu (I'm assuming thats what you mean?) package!  You can get it by searching our packagecloud repository or running the following:
curl https://packagecloud.io/install/repositories/chef/current/script.deb | sudo bash
sudo apt-get install chefdk=0.5.0~rc.5-1

Thanks Tyler! Yep, that's what I meant.

Chef DK is up to 0.5.1 now.  No more need to grab the release candidate.
https://github.com/chef/chef-dk/blob/master/CHANGELOG.md


Reduce the number of times we run apply

What does this PR do?
The current implementation of the install scripts runs terraform apply 2 times per terraform module using -target. Additionally apply is run
again 2 more times at the end of the script.
This means that we are running terraform apply twice as much as required.
This commit adds  | tee /dev/tty to all the apply steps so that the output can be captured and evaluated, as well as having the output print to screen.
More

[x] Yes, I have tested the PR using my local account setup (Provide any test evidence report under Additional Notes)
[ ] Mandatory for new blueprints. Yes, I have added a example to support my blueprint PR
[ ] Mandatory for new blueprints. Yes, I have updated the website/docs or website/blog section for this feature
[ ] Yes, I ran pre-commit run -a with this PR. Link for installing pre-commit locally

For Moderators

[ ] E2E Test successfully complete before merge?

The | tee /dev/tty is going to interfere with the $? check (tee will always return 0). Let me take another pass at this.


Improve Chat UX

The place I most commonly use Privly is in the chat box of gchat or Facebook chat -- even though the applications I am sharing in the chat box were never explicitly designed to target this use case. While it works remarkably well, the UX problem is that users must repeatedly post new content links by launching an application to generate the link. A more appropriate approach would be to send a singe application link whose associated contents can be updated by the user. A challenge in this project is to design a fully realized chat application that works well injected into any host page's chat application.

I am Kamdjou Duplex, a 4th year student of Computer Engineering at the
University of Buea, Cameroon. I have programming skills in JavaScript,
AngularJS, HTML5, CSS3, Firebase, Bootstrap. I am motivated to work on this project and with
over 2 years of Angular JS programming, I believe i have the skills to
work on this project with minimal supervision. I will like to get some
assistance with respect to resources and further information i may
need in order to write my proposal. I am currently checking out the
repository and will start working on my proposal ASAP. I think that I can aproached more easy with WebRTC
Thanks for the assistance.
Regards,
Duplex

I updated the labels to reflect the progress made for seamless posting, which improves chat functionality substantially.


Linking animal excretion to the litter model

Description
This pull request links animal excretion (that takes place in the animal module) to the metabolic litter pool. The treatment of animal excrement decay is extremely simplistic (it builds up and then all decays at once), so this will need further work in future.
I mainly want to check that my changes to the animal model are sensible and that the way that the models are setup to communicate is sensible.
There's another couple of processes within the animal model that should feed into the litter model, so this pull request is also a prototype for those (slightly more complex) exchanges.
Type of change

[x] New feature (non-breaking change which adds functionality)
[ ] Optimization (back-end change that speeds up the code)
[ ] Bug fix (non-breaking change which fixes an issue)

Key checklist

[x] Make sure you've run the pre-commit checks: $ pre-commit run -a
[x] All tests pass: $ poetry run pytest

Further checks

[x] Code is commented, particularly in hard-to-understand areas
[x] Tests added that prove fix is effective or that feature works

Codecov Report

Merging #287 (6502e1e) into develop (baba994) will increase coverage by 0.03%.
The diff coverage is 100.00%.

@@             Coverage Diff             @@
##           develop     #287      +/-   ##
===========================================
+ Coverage    95.14%   95.18%   +0.03%
===========================================
  Files           46       46
  Lines         2041     2054      +13
===========================================
+ Hits          1942     1955      +13
  Misses          99       99

Files Changed
Coverage Δ

virtual_rainforest/models/animals/__init__.py
100.00% <ø> (ø)

virtual_rainforest/models/litter/litter_model.py
100.00% <ø> (ø)

virtual_rainforest/models/litter/litter_pools.py
100.00% <ø> (ø)

...irtual_rainforest/models/animals/animal_cohorts.py
98.46% <100.00%> (ø)

...al_rainforest/models/animals/animal_communities.py
90.76% <100.00%> (-0.14%)
:arrow_down:

virtual_rainforest/models/animals/animal_model.py
100.00% <100.00%> (ø)

...ual_rainforest/models/animals/carcasses_and_poo.py
100.00% <100.00%> (ø)

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Support for "logs in context" for BOSH Release VMs

Description
I am investigating ways to have log data pulled from this nozzle to show up in the Logs UI view for a selected infrastructure host that is a VM deployed by Bosh hosting various Jobs. I work at new relic, so please reach out and I'm happy to show some screenshots.
Acceptance Criteria
I noticed that there's a JSON map in https://github.com/newrelic/newrelic-pcf-nozzle-tile/blob/bad926c060368a7c68b89ac1889bd4c283fe3f29/newrelic/metrics/marshaler.go
Is it possible to update the JSON Map so that "host.name" or "hostname" or "host" would be assigned the concatenated values of "Job" + "Index." So the New MAP might look like the following (Although I'm unfamiliar with how to concatenate in a map):
JSONDefaults = JSONMap{
    "EnvelopeType":     "eventType",
    "Origin":           "pcf.origin",
    "Deployement":      "pcf.deployment",
    "Job":              "pcf.job",
    "Index":            "pcf.index",
    "Job" + "Index":    "host.name",
    "IP":               "pcf.ip",
    "AppGUID":          "pcf.app.guid",
    "AppName":          "pcf.app.name",
    "AppSpaceName":     "pcf.app.space.name",
    "AppOrgName":       "pcf.app.org.name",
    "AppInstanceIndex": "pcf.app.instance.index",
    "AppInstanceState": "pcf.app.instance.state",
    "Name":             "pcf.metric",
    "Unit":             "pcf.metric.unit",
    "Min":              "pcf.metric.min",
    "Max":              "pcf.metric.max",
    "Total":            "pcf.metric.total",
    "Quota":            "pcf.metric.quota",
    "Count":            "pcf.metric.samples",
    "Average":          "pcf.metric.average",
    "QuotaUsed":        "pcf.metric.quota.used",
    "Drift":            "pcf.metric.drift",
}

Describe Alternatives
None known at this time. I am unable to find a way for customers to decorate firehose logs with their own attributes/tags. There may be another way to tackle this.
The goal is for the customer to be able to see firehose logs in the UI when looking at a host related to a specific BOSH VM (i.e. vms for diego, api, router, etc...)  where the NR Infrastructure Agent is deployed.
Based on the NRQL queries performed for those screens, there only needs to exist a "host", "host.name" or "hostname" field.
Dependencies
Possibly the infrastructure team?
For Maintainers Only or Hero Triaging this bug
Suggested Priority (P1,P2,P3,P4,P5):
Suggested T-Shirt size (S, M, L, XL, Unknown):

Or if there exists a "host" or "hostname" in the JSON obtained from the Loggregator, then please use that. So for example:
JSONDefaults = JSONMap{
    "EnvelopeType":     "eventType",
    "Origin":           "pcf.origin",
    "Deployement":      "pcf.deployment",
    "Job":              "pcf.job",
    "Index":            "pcf.index",
    "Host":             "host.name",
    "IP":               "pcf.ip",
    "AppGUID":          "pcf.app.guid",
    "AppName":          "pcf.app.name",
    "AppSpaceName":     "pcf.app.space.name",
    "AppOrgName":       "pcf.app.org.name",
    "AppInstanceIndex": "pcf.app.instance.index",
    "AppInstanceState": "pcf.app.instance.state",
    "Name":             "pcf.metric",
    "Unit":             "pcf.metric.unit",
    "Min":              "pcf.metric.min",
    "Max":              "pcf.metric.max",
    "Total":            "pcf.metric.total",
    "Quota":            "pcf.metric.quota",
    "Count":            "pcf.metric.samples",
    "Average":          "pcf.metric.average",
    "QuotaUsed":        "pcf.metric.quota.used",
    "Drift":            "pcf.metric.drift",
}

FYI: This should be pretty simple. Nothing needs to be done on the platform side from what I can tell. If we can just find a way to get the bosh name and guid (with exception to "bosh" and "jumpbox" vm) for the VMs it deploys added as "hostname", "host", or "host.name"... the "logs in context", for any BOSH VMs that have the infra agent, should work.


loader: make stylus-loader official

Fixes #3982

[x] Read and sign the CLA. PRs that haven't signed it won't be accepted.
[x] Make sure your PR complies with the writer's guide.
[x] Review the diff carefully as sometimes this can reveal issues.
[x] Do not abandon your Pull Request: Stale Pull Requests.
Remove these instructions from your PR as they are for your eyes only.

Thanks!