Find the inner and outer contour of a set of points

Hi again @marcomusy. I have a set of points that I want to find the inner and outer contour of these points. This means the inner and outer contour of the red points below. The points order aren't organized. This is so I can compute the deviation between the outer and inner contour to the blue lines.  I have tried to reconstruct a surface of the points with recoSurface(pts), and tried to find the boundaries of the mesh, but I didnt suceed since the mesh was so coarse for my points. Also creating the surface with delauney2d. I have tried to reconstruct the same issue with some code below.  In advance, thank you!

from vedo import *

cyl1  = Cylinder(pos=(0,0,0), r=2, height=4, axis=(1,0,0), alpha=.5, cap=0, res=100).triangulate()
cyl2 = Cylinder(pos=(0,0,2), r=1, height=3, axis=(0,0.3,1), alpha=.5, cap=0, res=100).triangulate()
cyl3 = Cylinder(pos=(0,0,2), r=1.1, height=3, axis=(0,0.3,1), alpha=.5, cap=0, res=100).triangulate()
intersect_1 = cyl1.intersectWith(cyl2).join(reset=True).c('b')
intersect_2 = cyl1.intersectWith(cyl3).join(reset=True).c('b')
#show(cyl1,cyl2,cyl3, intersect_1, intersect_2).close()

#Trying to cut out the the surface between the two intersect lines
surf = cyl1.clone()
surf.cutWithMesh(cyl3, invert= True) #These two lines doesn't work for me, to cut out the section between cyl2 and cyl3? Have I done it wrong?
surf.cutWithMesh(cyl2, invert= True) # I tried using the cutWithCylinder also, instead of mesh, but the cut did end up with the same as cutWithMesh. An empty mesh.
show(surf,cyl2, cyl3).close()

#when figuring what I have done wrong with cutWithMesh, extract the points of the surf and find the countour. Maybe randomize the order of the points.
pts = v.Points(surf.points())

#find a way to find the inner and outer contour?

Hi
It doesn't work because there are no vertices on the surface of the cylinder1 (only faces!). So I make i cylinder by extruding a closed line made from a circle (i might have extruded the circle but it would be capped).
Check out:
from vedo import *

# cyl1 = Cylinder(pos=(0,0,0), r=2, height=4, axis=(1,0,0), alpha=.5, cap=0, res=100).triangulate() # not good
cyl1 = Line(Circle(r=2), closed=True).extrude(4, res=50).triangulate().shift(0,0,-2).rotateY(90)

cyl2 = Cylinder(pos=(0,0,2), r=1,   height=3, axis=(0,0.3,1), alpha=.5, cap=0, res=100).triangulate()
cyl3 = Cylinder(pos=(0,0,2), r=1.1, height=3, axis=(0,0.3,1), alpha=.5, cap=0, res=100).triangulate()
intersect_1 = cyl1.intersectWith(cyl2).join(reset=True).c('b').lw(3)
intersect_2 = cyl1.intersectWith(cyl3).join(reset=True).c('v').lw(3)
show(cyl1,cyl2,cyl3, intersect_1, intersect_2, axes=1).close()

#Trying to cut out the the surface between the two intersect lines
surf = cyl1.clone()
surf.cutWithCylinder(center=(0,0,2), axis=(0,0.3,1), r=1.1, invert=False) # much faster than cutWithMesh
surf.cutWithCylinder(center=(0,0,2), axis=(0,0.3,1), r=1.0, invert=True)
surf.subdivide(3, method=1).lw(1) # make mesh denser in points
bnds = surf.boundaries().c("red") # extract boundaries

show(surf,cyl2, cyl3, bnds, axes=1).close()

Okay! Thank you !My problem is that the extracted surface, the variable surf above. I only have the points, and need the boundaries.  I'm not sure if this is right to say but I have only have the surf.points() in my issue and not having the mesh. So I need to create mesh from the points or extract the boundaries in another way? I tried to recreate the problem below using your code:

from vedo import *

# cyl1 = Cylinder(pos=(0,0,0), r=2, height=4, axis=(1,0,0), alpha=.5, cap=0, res=100).triangulate() # not good
cyl1 = Line(Circle(r=2), closed=True).extrude(4, res=50).triangulate().shift(0,0,-2).rotateY(90)

cyl2 = Cylinder(pos=(0,0,2), r=1,   height=3, axis=(0,0.3,1), alpha=.5, cap=0, res=100).triangulate()
cyl3 = Cylinder(pos=(0,0,2), r=1.1, height=3, axis=(0,0.3,1), alpha=.5, cap=0, res=100).triangulate()
intersect_1 = cyl1.intersectWith(cyl2).join(reset=True).c('b').lw(3)
intersect_2 = cyl1.intersectWith(cyl3).join(reset=True).c('v').lw(3)
#show(cyl1,cyl2,cyl3, intersect_1, intersect_2, axes=1).close()

#Trying to cut out the the surface between the two intersect lines
surf = cyl1.clone()
surf.cutWithCylinder(center=(0,0,2), axis=(0,0.3,1), r=1.1, invert=False) # much faster than cutWithMesh
surf.cutWithCylinder(center=(0,0,2), axis=(0,0.3,1), r=1.0, invert=True)
surf.subdivide(3, method=1).lw(1) # make mesh denser in points
bnds = surf.boundaries().c("red") # extract boundaries

#extract only the one region of surf points
pts = surf.points()
pts_2 = []
for p in pts:
    cpt = intersect_1.closestPoint(p)
    if mag2(p-cpt)<5:
        pts_2.append(p)

# I have only the points, pts_2
np.random.shuffle(pts_2) #shuffle for disorganized array
#somehow extract the boundias from pts_2?
pts_2 = Points(pts_2)
surf_2 = recoSurface(pts_2)
bnds_2  = surf_2.boundaries().c('g')
#this is the issue I have, the boundaries doesnt match with the contour on the points. Is there another way?
show(pts_2,bnds_2, axes=1).close()

If the cloud is a generic point cloud then this becomes a hard problem with no general solution I guess.
You can play with creating a "tube" around the line to erode the mesh and then use these new boundaries.
import numpy as np
from vedo import *

cyl1 = Line(Circle(r=2), closed=True).extrude(4, res=50).triangulate().shift(0,0,-2).rotateY(90)

cyl2 = Cylinder(pos=(0,0,2), r=1,   height=3, axis=(0,0.3,1), alpha=.5, cap=0, res=100).triangulate()
cyl3 = Cylinder(pos=(0,0,2), r=1.1, height=3, axis=(0,0.3,1), alpha=.5, cap=0, res=100).triangulate()
intersect_1 = cyl1.intersectWith(cyl2).join(reset=True).c('b').lw(3)
intersect_2 = cyl1.intersectWith(cyl3).join(reset=True).c('v').lw(3)

#Trying to cut out the the surface between the two intersect lines
surf = cyl1.clone()
surf.cutWithCylinder(center=(0,0,2), axis=(0,0.3,1), r=1.1, invert=False)
surf.cutWithCylinder(center=(0,0,2), axis=(0,0.3,1), r=1.0, invert=True)
surf.cutWithPlane(normal='z')
surf.subdivide(method=1).lw(1) # make mesh denser in points

# show(surf,cyl2, cyl3, axes=1).close()

#extract only the one region of surf points
pts = surf.points(copy=True)
np.random.shuffle(pts) #shuffle for disorganized array

#somehow extract the boundias from pts_2?
pts_2  = Points(pts)
surf_2 = recoSurface(pts, radius=0.04, dims=(200,200,50)).c('y')
# surf_2 = ConvexHull(pts).alpha(1) # this doesn't seem to work well in this case
bnds_2 = surf_2.boundaries().c('g')

modl = bnds_2.implicitModeller(distance=0.03, res=(150,150,50))
surf_2.cutWithMesh(modl)

show(pts_2, surf_2, bnds_2, axes=1).close()

Or you may try to identify the closest points to the center etc... but I think there is no "magic" solutions.

Nice ! I see ! This seems to work for my issue, after playing with some parameters. I can get the outer and inner boundaries of the points :) The code below isn't so fast, is it because of the implicitModeller?

Is it possible to extract the inner and the outer boundaries points without iterate with the closestpoint?
I want to plot the distance between the boundaries lines and the intersecion lines. So compute the distance between intersect_1 and the inner_bnds, and intersect_2  and the outer_bnds.
Is it possible to find point correspondece to compute the distance between the points. Cast a ray  360 degrees from the center point or something? Get average distance or an average transformation ?

Below is a started example. If you have a better suggestion or want me to look more into some of your examples! Feel free to say!
import numpy as np
from vedo import *

cyl1 = Line(Circle(r=2), closed=True).extrude(4, res=50).triangulate().shift(0,0,-2).rotateY(90)

cyl2 = Cylinder(pos=(0,0,2), r=1,   height=3, axis=(0,0.3,1), alpha=.5, cap=0, res=100).triangulate()
cyl3 = Cylinder(pos=(0,0,2), r=1.1, height=3, axis=(0,0.3,1), alpha=.5, cap=0, res=100).triangulate()
intersect_1 = cyl1.intersectWith(cyl2).join(reset=True).c('b').lw(3)
intersect_2 = cyl1.intersectWith(cyl3).join(reset=True).c('v').lw(3)

#Trying to cut out the the surface between the two intersect lines
surf = cyl1.clone()
surf.cutWithCylinder(center=(0,0,2), axis=(0,0.3,1), r=1.1, invert=False)
surf.cutWithCylinder(center=(0,0,2), axis=(0,0.3,1), r=1.0, invert=True)
surf.cutWithPlane(normal='z')
surf.subdivide(method=1).lw(1) # make mesh denser in points

# show(surf,cyl2, cyl3, axes=1).close()

#extract only the one region of surf points
pts = surf.points(copy=True)
np.random.shuffle(pts) #shuffle for disorganized array

#somehow extract the boundias from pts_2?
pts_2  = Points(pts)
surf_2 = recoSurface(pts, radius=0.04, dims=(200,200,50)).c('y')
# surf_2 = ConvexHull(pts).alpha(1) # this doesn't seem to work well in this case
bnds_2 = surf_2.boundaries().c('g')

modl = bnds_2.implicitModeller(distance=0.03, res=(150,150,50))
surf_2.cutWithMesh(modl)
bnds_3 = surf_2.boundaries().c('y')
bnds_3.shift(0.03, 0.02, 0.01) # to create a distance between

#extact the inner and outer boundaries, it is a better way for this?
inner_bnds = []
for p in bnds_3.points():
    cpt = intersect_1.closestPoint(p)
    if mag2(p-cpt)<0.003:
        inner_bnds.append(p)
outer_bnds = []
for p in bnds_3.points():
    cpt = intersect_2.closestPoint(p)
    if mag2(p-cpt)<0.003:
        outer_bnds.append(p)
outer_bnds = Points(outer_bnds).c('y')
inner_bnds=Points(inner_bnds).c('r')

#Smooth the points
outer_bnds_s = outer_bnds.clone().smoothMLS1D(f=0.4).c('y')
inner_bnds_s = inner_bnds.clone().smoothMLS1D(f=0.4).c('g')

# here I want to compute a distance between the inner_bnds and intersect_1, and outer_bnds and intersect_2.
show(inner_bnds_s, outer_bnds_s, intersect_1, intersect_2, axes=1).close()

The code below isn't so fast, is it because of the implicitModeller?

Yes, and recoSurface() is also slow.

maybe you can create another cylinder to cut off inner/outer ring?
you may use distanceToMesh() (see examples/basic/distance2mesh.py)
i guess you have no real point to point correspondence so you need to figure out a rule that makes sense for your specific problem.

the rest of the code looks fine to me!

Thank you ! Sorry for the late reply!

Yes, that could be a possibility!
I did look into that example, and i'm adding it in for visualizing.
True, I don't have that. I ended up using ICP to find the deviation within transformation between the surface surf and the surf_2 points.  I did the ICP with rigid = True. Is it possible to find the difference in scaling after ICP?
I did compare surf.area() with the surf_2.area(). But this doesen't seem right because the recosurface()` gives me a coarse mesh.

you can access the transformation object this way:
surf_2.transform.GetMatrix() # vtkMatrix4x4

True, I don't have that. I ended up using ICP to find the deviation within transformation between the surface surf and the surf_2 points.  I did the ICP with rigid = True. Is it possible to find the difference in scaling after ICP?

Yes, I have the transformation from surf_2.transform.GetMatrix(), but it is possible to get a measure of scaling between the two point clouds?

in principle you should extract the scaling from the matrix.

in principle you should extract the scaling from the matrix.

Okey, I thought this matrix gave me the rotation and translation in terms to move the one pointcloud to another?
Since the rigid = True is set.

..well it gives everything including the scaling, if you set rigid=True then this scaling should be one.

I understand. As always, gratitude for the fast response!


Painkillers are missing interaction menu interaction

Somebody said that the interaction is missing from the interaction menu.

was fixed with commit e928deec28be278dfde7f23103f75c2d0b6aefbb


Increase parallelism for the aarch64 RHEL and Fedora workers

Due to a hardware upgrade, each worker now boasts 40 cores
hence increasing the tests run in parallel here.

Yeah! Parallelize all the things!


Appstudio update test-component-pac-ahdc

Pipelines as Code configuration proposal

Pipelines as Code CI/test-component-pac-ahdc-on-pull-request has successfully validated your commit.

StatusDurationName

✅ Succeeded
6 seconds
init

✅ Succeeded
13 seconds
clone-repository

✅ Succeeded
31 seconds
build-container

✅ Succeeded
10 seconds
deprecated-base-image-check

✅ Succeeded
11 seconds
inspect-image

✅ Succeeded
8 seconds
clair-scan

✅ Succeeded
39 seconds
clamav-scan

✅ Succeeded
8 seconds
sbom-json-check

✅ Succeeded
22 seconds
optional-label-check

✅ Succeeded
21 seconds
label-check

✅ Succeeded
7 seconds
show-summary


[Module Proposal]: avm-res-web-appservice (res) (bicep)

Check for previous/existing GitHub issues/module proposals

[X] I have checked for previous/existing GitHub issues/module proposals

Check this module doesn't already exist in another project that is being evolved into AVM

[X] I have checked for that this module doesn't already exist in CARML or TFVM

Terraform or Bicep?
Bicep
Module Classification?
Resource Module
Module Name
avm-res-web-appservice
Module Details
After discussing with Mate Barabas, I will pick up the CARML conversion for App Service, App Service Plan, App Service Environment.
Do you want to be the owner of this module?
Yes
Module Owner's GitHub Username
tsc-buddy

@matebarabas - I also have a colleague who is happy to co-own these. So when we sync next, let me know the process to onboard them.

@matebarabas i will handle this together with other conversion request. @tsc-buddy i will share with you upgrade checklist that you can use to continuo your work.

@tsc-buddy, thanks for opening this request and volunteering! I'll shortly send you a request to officially confirm ownership.
Also, @prjelesi is going to reach out to you separately and help with the next steps, given this is a migrate from CARML scenario.

@matebarabas - I also have a colleague who is happy to co-own these. So, when we sync next, let me know the process to onboard them.

Please let me know who the other person would be; we can capture him/her as a secondary owner.
Thank you!

Hi @tsc-buddy ,
Thanks for requesting/proposing to be an AVM module owner.
We just want to confirm you agree to the below pages that define what module ownership means:

https://azure.github.io/Azure-Verified-Modules/specs/shared/team-definitions
https://azure.github.io/Azure-Verified-Modules/specs/shared
https://azure.github.io/Azure-Verified-Modules/help-support/module-support

Any questions or clarifications needed, let us know.
If you agree please just reply to this issue with the exact sentence below (as this helps with our automation 👍):
"I CONFIRM I WISH TO OWN THIS AVM MODULE AND UNDERSTAND THE REQUIREMENTS AND DEFINITION OF A MODULE OWNER"
Thanks,
The AVM Core Team

"I CONFIRM I WISH TO OWN THIS AVM MODULE AND UNDERSTAND THE REQUIREMENTS AND DEFINITION OF A MODULE OWNER"

Hi @tsc-buddy,
Thanks for confirming that you wish to own this AVM module and understand the related requirements and responsibilities!
We just want to ask you to double check a few important things before you start the development.
Please use the following values explicitly as provided in the module index page:

For your module:

ModuleName - to name your module
TelemetryIdPrefix - to be used in your module's telemetry

For your module's repository:

Repository name and folder path are defined in RepoURL
Create the GitHub teams defined in the ModuleOwnersGHTeam and ModuleContributorsGHTeam columns

You can now start the development of this module! ✅ Happy coding! 🎉
Please respond to this comment, once your module is ready to be published! 🙏
Any further questions or clarifications needed, let us know!
Thanks,
The AVM Core Team

NOTE:

This Module Proposal issue MUST remain open until the module is fully developed, tested and published to the relevant registry. Do NOT close the issue before the successful publication is confirmed!
Once the module is fully developed, tested and published to the relevant registry, and the Module Proposal issue was closed, it MUST remain closed.

Hi @tsc-buddy, I'm reaching out as having this issue revisited, I now believe that this one might have been given the wrong name.
If I'm not mistaken, the intent here is to cover Web Apps and Function apps (as these are the assets deployed into an App Service Plan. The correct name for this one is avm-res-web-site, and the donor module in CARML is here: https://github.com/Azure/ResourceModules/tree/main/modules/web/site
Can you please confirm that this GH issue is meant to cover this module?

Note: I don't want to further confuse you, but please expect the naming convention for Bicep modules to slightly change. I'll reach out to you once again when this happens.

@tsc-buddy,
Due to a recent change in the naming convention for AVM Bicep modules, this module is getting an updated name.
The title and description of this GH issue has been updated to reflect the name change. Please make sure you adhere to the new naming convention when publishing your module!
Please see the new naming convention in:

RMNFR1 for resource modules
PMNFR1 for pattern modules

Please note that the Telemetry ID and name of GitHub Teams haven't changed.
If you feel uncertain about any details, please check the Bicep modules indexes.

To be migrated from CARML: https://github.com/Azure/ResourceModules/tree/main/modules/web/site
Requested by the AZD team: #261
CC: @jongio

@matebarabas FYI: I will support @tsc-buddy with the migration of this module as I need it for a customer project.

(draft) PR: https://github.com/Azure/bicep-registry-modules/pull/779

@matebarabas @tsc-buddy PR has been merged into the BRM repo: https://github.com/Azure/bicep-registry-modules/pull/779

[x] PR merged; Module published.
[x] GitHub teams are available and assigned to their parents.
[x] CODEOWNERS file updated.

Thank you for your work @tsc-buddy and @krbar! I'm closing this issue now.


Distilled layoutlm ?

Any plan to release a mini layoutlm model?

yes, layoutlm model distillation is on the track. please stay tune.

So… is the model development plan stopped?


Bug: Missing jar files in Grails 4.0.4 install

I created a new Grails project today using Grails 4.0.4 from SDKMan and OpenJDK 11.0.8 from the Ubuntu 20.04 repository.  I used grails create-domain-class to make my (12) domain classes, put in the fields I wanted, then used grails generate-all to create the default CRUD.
When I tried to run the app using grails run-app or ./grailsw run-app I received two sets of very similar error messages explaining that I was missing the following .jar files:
javax.activation-api-1.2.0.jar
FastInfoset-1.2.15.jar
stax-ex-1.8.jar
istack-commons-runtime-3.0.7.jar
txw2-2.3.1.jar
jaxb-api-2.3.1.jar
from two separate locations:
~/.gradle/caches/modules-2/files-2.1/com.sun.xml.bind/jaxb-impl/2.3.1/ (a long string)
and
~/.gradle/caches/modules-2/files-2.1/org.glassfish.jaxb/jaxb-runtime/2.3.1/ (a different long string)
I retrieved the correct versions from the relevant repositories on
https://mvnrepository.com/artifact/
and was able to proceed.
Not sure if this is an SDKMAN build issue or something missing from the Grails end of things.

Hi!
This seems to be related to
https://blog.codefx.org/java/java-11-migration-guide/#Removal-Of-Java-EE-Modules
TLDR; JDK 11 no longer includes JAF (javax.activation), JAXB (java.xml.bind.),  JAXWS (javax.jws, javax.jws.soap, javax.xml.soap, and javax.xml.ws.).
If these dependencies are required, they need to be added separately (like you did).

Hi @monetschemist, in addition to that, the binaries that you install with SDKMAN come directly from the Grails download page. We also don't tamper with the contents of the binaries during the installation process. Hope that this and what @helpermethod posted helps!

Hi Marco, Oliver;
Thanks VERY MUCH for setting me straight here.  Speaking generally, your
always-responsive help is a big part of what makes SDKMAN so wonderfully
useful.
Two really important things I learned here:

you use the builds provided by Grails (I kind of thought you might
but I wasn't sure);
I may be premature in wanting to use Java 11.

Both of these things point me back to the good people at Grails.  I'm going
to make a suggestion there, based on all of this.
Again, thanks so much!
On Sun, Sep 20, 2020 at 8:48 AM Marco Vermeulen<EMAIL_ADDRESS>wrote:

Hi @monetschemist https://github.com/monetschemist, in addition to
that, the binaries that you install with SDKMAN come directly from the
Grails download page. We also don't tamper with the contents of the
binaries during the installation process. Hope that this and what
@helpermethod https://github.com/helpermethod posted helps!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/sdkman/sdkman-cli/issues/801#issuecomment-695802086,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AB4ABHW44WLCY5YACUJMNOTSGYP47ANCNFSM4RTHBKDQ
.

--
Chris Hermansen · clhermansen "at" gmail "dot" com
C'est ma façon de parler.


Using done() on request returns a null value

Hello, I recently started using your library within the context of an application powered by ReactPHP. I encountered a problem with the promises returned from the calls made via the client.
...
//function body
$client = new Browser($loop);

$request = $client->get(url, headers)->done(function ($response) {
  return (string) $response->getBody();
});

return $request;
...

The snippet of code above is a small part of a larger application. The problem I am encountering is that the request returns a null value and yet using the echo directive on the response prints a valid API response. I would like to know how to fix this.

Hi @ace411, welcome to ReactPHP! :tada:
It looks like you want to use then() instead of done() and your code should work :+1:
Let's break this down:
// broken
$request = $client->get(url, headers)->done(function ($response) {
  return (string) $response->getBody();
});

This is equivalent to this:
// still broken
$promise = $client->get(url, headers);
$request = $promise->done(function ($response) {
  return (string) $response->getBody();
});

The Promise::done() method is documented to return void, hence why this example will return a null value here.
I hope this helps :+1:


fix(DTFS2-6694): remove shared user token in gef-ui api calls

Codecov Report

Merging #2061 (756f875) into main (2390d16) will decrease coverage by 19.14%.
Report is 10 commits behind head on main.
The diff coverage is 2.24%.

@@            Coverage Diff             @@
##             main   #2061       +/-   ##
==========================================
- Coverage   21.01%   1.87%   -19.14%
==========================================
  Files        1030     851      -179
  Lines       25229   18727     -6502
  Branches     4081    2959     -1122
==========================================
- Hits         5303     352     -4951
+ Misses      17370   16049     -1321
+ Partials     2556    2326      -230

Flag
Coverage Δ

api-test
0.48% <0.00%> (?)

ui-test
?

unit-test
1.99% <2.66%> (+1.27%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed
Coverage Δ

...-function/acbs-amend-facility-loan-record/index.js
0.00% <0.00%> (ø)

...unction/acbs-amend-facility-master-record/index.js
0.00% <0.00%> (ø)

...nctions/acbs-function/acbs-amend-facility/index.js
0.00% <0.00%> (ø)

...unctions/acbs-function/acbs-facility-bond/index.js
0.00% <0.00%> (ø)

...unctions/acbs-function/acbs-facility-loan/index.js
0.00% <0.00%> (ø)

...ure-functions/acbs-function/acbs-facility/index.js
0.00% <0.00%> (ø)

...nctions/acbs-function/acbs-issue-facility/index.js
0.00% <0.00%> (ø)

azure-functions/acbs-function/acbs/index.js
0.00% <0.00%> (ø)

...on/activity-create-code-value-transaction/index.js
0.00% <0.00%> (ø)

...s-function/activity-create-deal-guarantee/index.js
0.00% <0.00%> (ø)

... and 85 more

... and 380 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


chore: lower sentry transaction sample rate #trivial

This PR resolves []
Description

Lower transaction sample rate for sentry.
PR Checklist

[ ] I have tested my changes on iOS and Android.
[x] I hid my changes behind a feature flag, or they don't need one.
[x] I have included screenshots or videos, or I have not changed the UI.
[x] I have added tests, or my changes don't require any.
[x] I added an app state migration, or my changes do not require one.
[x] I have documented any follow-up work that this PR will require, or it does not require any.
[x] I have added a changelog entry below, or my changes do not require one.

To the reviewers 👀

[ ] I would like at least one of the reviewers to run this PR on the simulator or device.

Changelog updates
Changelog updates

Cross-platform user-facing changes

iOS user-facing changes

Android user-facing changes

Dev changes

lower transaction sample rate - brian

Need help with something? Have a look at our docs, or get in touch with us.

This PR contains the following changes:

Dev changes (lower transaction sample rate - brian - brainbicycle)

  Generated by :no_entry_sign: dangerJS against 621d5de9bd70d610145891fb26403706255d9256


Fix type checking (mypy)

Currently mypy type checks are not passing - with a large number of easy to resolve errors.
poetry run mypy transformer_lens

Resolving these would help make the codebase more robust, and the corresponding checks in /.github/workflows/checks.yml can then be enabled (they're commented out).
https://github.com/neelnanda-io/TransformerLens/blob/72b9be00c12b5be37baaa16b57e5df5e2aff3aad/.github/workflows/checks.yml#L58

I tried a bunch to fix it with stub packages but mainly didn't really get anywhere

I tried a bunch to fix it with stub packages but mainly didn't really get anywhere

Hmm, when I run mypy, most of what I see seem to be it eg failing to deal with torchtyping?
Eg:
transformer_lens/components.py:16: error: Skipping analyzing "fancy_einsum": module is installed, but missing library stubs or py.typed marker  [import]
transformer_lens/components.py:23: error: Skipping analyzing "torchtyping": module is installed, but missing library stubs or py.typed marker  [import]
transformer_lens/components.py:37: error: Name "batch" is not defined  [name-defined]
transformer_lens/components.py:37: error: Name "position" is not defined  [name-defined]
transformer_lens/components.py:38: error: Name "batch" is not defined  [name-defined]
transformer_lens/components.py:38: error: Name "position" is not defined  [name-defined]
transformer_lens/components.py:38: error: Name "d_model" is not defined  [name-defined]
transformer_lens/components.py:57: error: Name "batch" is not defined  [name-defined]
transformer_lens/components.py:57: error: Name "position" is not defined  [name-defined]
transformer_lens/components.py:57: error: Name "d_model" is not defined  [name-defined]
transformer_lens/components.py:58: error: Name "batch" is not defined  [name-defined]
transformer_lens/components.py:58: error: Name "position" is not defined  [name-defined]
transformer_lens/components.py:58: error: Name "d_vocab_out" is not defined  [name-defined]
transformer_lens/components.py:79: error: Name "batch" is not defined  [name-defined]
transformer_lens/components.py:79: error: Name "position" is not defined  [name-defined]
transformer_lens/components.py:80: error: Name "batch" is not defined  [name-defined]
transformer_lens/components.py:80: error: Name "position" is not defined  [name-defined]
transformer_lens/components.py:80: error: Name "d_model" is not defined  [name-defined]
transformer_lens/components.py:118: error: Name "batch" is not defined  [name-defined]
transformer_lens/components.py:118: error: Name "position" is not defined  [name-defined]
transformer_lens/components.py:118: error: Name "length" is not defined  [name-defined]
transformer_lens/components.py:119: error: Name "batch" is not defined  [name-defined]
transformer_lens/components.py:119: error: Name "position" is not defined  [name-defined]
transformer_lens/components.py:119: error: Name "length" is not defined  [name-defined]

Hi guys - just wanted to add that it looks like this might be expected because of mypy limitations (see https://github.com/patrick-kidger/torchtyping/issues/35)
Patrick's first suggestion of defining an object with the strings might be the way to go.  It could help solidify the naming conventions and avoid potential confusion like using "num_components" vs "components", etc.
I tried this in components.py and it got rid of the errors

from enum import Enum
class T(Enum):
    BATCH = "batch"
    D_VOCAB = "d_vocab"
    POSITION = "position"
    D_MODEL = "d_model"

# Embed & Unembed
class Embed(nn.Module):

    ...

    def forward(
        self, tokens: TT[T.BATCH, T.POSITION]
    ) -> TT[T.BATCH, T.POSITION, T.D_MODEL]:
        # If A has shape [a, b] and B has shape [c, d], then A[:, B] has shape [a, c, d]
        # B acts as a tensor of indices into the second dimension (so >=0 and <b)
        return self.W_E[tokens, :]

@dkamm great idea! This is a far better approach

Yeah, that looks great @dkamm, I'd love for you to add that if you're
interested.
On Mon, 16 Jan 2023 at 10:18, Alan @.***> wrote:

@dkamm https://github.com/dkamm great idea! This is a far better
approach
—
Reply to this email directly, view it on GitHub
https://github.com/neelnanda-io/TransformerLens/issues/130#issuecomment-1383812247,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ASRPNKLNF33XI3KWMTVTWTLWSUN6TANCNFSM6AAAAAATJU6GPA
.
You are receiving this because you commented.Message ID:
@.***>

Sure! Will send a PR


Update README.rst #237

Added the year

Hey @cmougan,
Thank you for looking into this. This is indeed correct, there were a couple of issues with the documentation and typing that we have now fixed. I suggest you update (merge) the current branch with the latest updates made on the master branch. Then we will be able to merge your PR!


Verify case of headers

This might be a no-op, but please double check. Filing this issue just in case.
We need to ensure that all headers used for gRPC are written in lower-case. Please review this Pull Request for reference of what needs to be updated.
Again, perhaps this doesn't apply to your language. Feel free to comment and then close the issue (after double checking).

Thanks for the heads-up @omaray! Will do.

Hey folks, are we good here? Can we close?

I believe that in google-cloud-ruby, gRPC headers are never set directly. See this Pub/Sub stub initialization code for an example similar to that in the Python PR above.
Closing. (@blowmage: Please affirm or re-open, thanks!)

I'm not aware of any custom headers in the GRPC service implementations.


Dragon battleaxe spec boost is calculating higher than it should

screenshot below of my stats right after using special - notice strength is only boosted by 15 instead of the suggested 25

I'm inclined to believe the correct boost is actually 16, not 15, and that you decayed a level as you were taking the screenshot, but I've fixed it now in b412e4dc1b15c7f368b11c793e710f019dac3d42.


[Question]: Shutdown isolate when app quit by swiping from recent items in Android

Hi @simolus3
I am starting the moor isolate on main thread as described in your documentation. The steps I have followed are as follows:

Start isolate on main thread.
Shutdown isolate in onWillPop i.e user quits app by pressing back button.
Also manage shutdown of isolate when user quits app by swiping from recent items in android??

How to achieve step 3. If step 3 is not handled then the isolate continues running which is not good. Because if I open the app, it will start a brand new isolate.
The only method is onWillPop. Do you know of a method that can detect android app quit??
Also is there a different approach on handling the shutdown case for isolates or is this implementation the way to go??

I think Android terminates the whole app process after swiping it from recent items (unless you have a special setup with background processes / services). So to my best knowledge, an isolate started from the main thread will be terminated (so is the main thread). Since the process will be terminated, there is no reliant callback (and I don't think Flutter exposes onStop).
Something that's more reliable than onWillPop would be a WidgetsBindingObserver. You could listen for lifecycle events and terminate the isolate when the app reaches a paused state, and spawn the isolate when resumed (or as needed). Be aware that the app may reach a paused state without being terminated, for instance if a user temporarily switches to another app.
Note that a safe shutdown via shutdownAll or close is not required to maintain data integrity, just to clear up unnecessary resources. If the system shuts down your app, all resources will be freed either way. So while closing resources manually is definitely a good practice, it isn't strictly necessary here.

Something that's more reliable than onWillPop would be a WidgetsBindingObserver. You could listen for lifecycle events and terminate the isolate when the app reaches a paused state, and spawn the isolate when resumed (or as needed). Be aware that the app may reach a paused state without being terminated, for instance if a user temporarily switches to another app.

Unfortunately when user taps back button the paused state is not called. Thus I had to handle onWillPop to shutdown the isolate.
Also I am using WidgetsBindingObserver to detect lifecycle events such as paused and resumed.

unless you have a special setup with background processes / services
yes i do have a special setup. Can you provide a solution during this approach?? I would like to manually handle resources and don't want to depend on the system.

Can there be a feature that allows me to check whether moor isolate is already running. This would help in avoiding moor isolate to start again if user opens app again??
Can this be done??

Are you using the MoorIsolate from multiple isolates or just the main thread? If it's just the main thread, you could do something like
MoorIsolate _instance;

// call this whenever you need a moor isolate, don't create it anywhere else
Future<MoorIsolate> useIsolate() async {
  if (_instance != null) {
    return _instance;
  }
  return _instance = await MoorIsolate.spawn(...);
}

Future<void> shutdown() async {
  _instance?.shutdownAll();
  _instance = null;
}

Won't _instance be null when user swipes app from recent items and open's app again??

I can't think of any scenario where the Flutter engine would terminate the main isolate but keep a background isolate running... If that happened there wouldn't be any way to detect whether a background isolate is running on the second app start (at least not from Dart).
You mentioned that you have a special setup, can you share more details? If your app doesn't run operations independent of UI (say with WorkManager or some geofencing), this shouldn't affect you.

I am using audio_service package for background audio.
This runs it's own isolate. I was unable to send across my moor isolate instance to the audioservice isolate.
Thus I am trying to keep moor isolate running as long as the UI is visible and then clean up resources when UI is not visible.
Cleaning up moor isolate in applifecycle paused is wasteful because the isolate needs to be spawned again on resumed. This means if the user switches back and forth from the app I would be shutting down and again have to be starting the moor isolate.
I would like shutting down of moor isolate in two ways:

When user presses back button.
When user swipes the app from recent item.

I have achieved 1, but 2 seems to be unachievable. 2 calls onTaskRemoved in android which cannot be detected in Flutter dart side. Thus it is impossible to shutdown the isolate.
Also if possible could you provide a solution where I can send moor isolate instance specifically to the audioservice isolate.
Here is the link of audioservice
https://github.com/ryanheise/audio_service

Hi @simolus3
Can there be a mechanism where on app launch we first check if moor isolate is already running??
Is this possible??

@rohansohonee

I have achieved 1, but 2 seems to be unachievable. 2 calls onTaskRemoved in android which cannot be detected in Flutter dart side. Thus it is impossible to shutdown the isolate.

I faced a similar issue, to solve this, I listen to the onDetachedFromEngine on the native side (Java), and send a message through the MethodChannel to the dart side, so I have a chance to do some cleanup on the dart side or kill the isolate.
Here's my code https://github.com/AgoraIO-Extensions/iris_method_channel_flutter/pull/75/files

I'm closing question issues that are older than a year.
This is one of them.


Michaeljeffrey/docker local testing

I'm iffy about if we want to compile the LoRaMac-Node stuff when we build the image, but it cuts down on testing time.

Going to keep this as a local change


Getting an error while fastlane setup

Hi,
I've had fastlane already setup, but deleted all folders and files and decided to do the setup again.
After step with choosing team for my Apple ID - I get error:
Multiple teams found, please enter the number of the team you want to use:
1) *** "******" (Company/Organization)
2) *** "****" (Company/Organization)
1
[15:43:30]: Error occurred with the setup program! Reverting changes now!
[15:43:30]: Deleting the 'fastlane' folder
Successfully submitted a crash report. If this is a problem with one of the tools specifically,
please submit an issue on GitHub and attach the following number to it: '953b655e64d1bd92d643a956c09baba8'
The crash report has been stored locally '/tmp/sentry_953b655e64d1bd92d643a956c09baba8.json'

/Users/quarezz/.rvm/rubies/ruby-2.2.1/lib/ruby/gems/2.2.0/gems/spaceship-0.19.3/lib/spaceship/client.rb:268:in `parse_response': [!] {"data"=>nil, "messages"=>{"warn"=>nil, "error"=>["Unauthorized access"], "info"=>nil}, "statusCode"=>"ERROR"} (Spaceship::Client::UnexpectedResponse)
	from /Users/quarezz/.rvm/rubies/ruby-2.2.1/lib/ruby/gems/2.2.0/gems/spaceship-0.19.3/lib/spaceship/tunes/tunes_client.rb:45:in `teams'
	from /Users/quarezz/.rvm/rubies/ruby-2.2.1/lib/ruby/gems/2.2.0/gems/spaceship-0.19.3/lib/spaceship/tunes/tunes_client.rb:83:in `select_team'
	from /Users/quarezz/.rvm/rubies/ruby-2.2.1/lib/ruby/gems/2.2.0/gems/spaceship-0.19.3/lib/spaceship/tunes/spaceship.rb:30:in `select_team'
	from /Users/quarezz/.rvm/rubies/ruby-2.2.1/lib/ruby/gems/2.2.0/gems/fastlane-1.55.0/lib/fastlane/setup/setup_ios.rb:173:in `detect_if_app_is_available'
	from /Users/quarezz/.rvm/rubies/ruby-2.2.1/lib/ruby/gems/2.2.0/gems/fastlane-1.55.0/lib/fastlane/setup/setup_ios.rb:29:in `run'
	from /Users/quarezz/.rvm/rubies/ruby-2.2.1/lib/ruby/gems/2.2.0/gems/fastlane-1.55.0/lib/fastlane/setup/setup.rb:19:in `run'
	from /Users/quarezz/.rvm/rubies/ruby-2.2.1/lib/ruby/gems/2.2.0/gems/fastlane-1.55.0/bin/fastlane:41:in `block (2 levels) in run'
	from /Users/quarezz/.rvm/rubies/ruby-2.2.1/lib/ruby/gems/2.2.0/gems/commander-4.3.5/lib/commander/command.rb:178:in `call'
	from /Users/quarezz/.rvm/rubies/ruby-2.2.1/lib/ruby/gems/2.2.0/gems/commander-4.3.5/lib/commander/command.rb:178:in `call'
	from /Users/quarezz/.rvm/rubies/ruby-2.2.1/lib/ruby/gems/2.2.0/gems/commander-4.3.5/lib/commander/command.rb:153:in `run'
	from /Users/quarezz/.rvm/rubies/ruby-2.2.1/lib/ruby/gems/2.2.0/gems/commander-4.3.5/lib/commander/runner.rb:428:in `run_active_command'
	from /Users/quarezz/.rvm/rubies/ruby-2.2.1/lib/ruby/gems/2.2.0/gems/fastlane_core-0.36.1/lib/fastlane_core/ui/fastlane_runner.rb:23:in `run!'
	from /Users/quarezz/.rvm/rubies/ruby-2.2.1/lib/ruby/gems/2.2.0/gems/commander-4.3.5/lib/commander/delegates.rb:15:in `run!'
	from /Users/quarezz/.rvm/rubies/ruby-2.2.1/lib/ruby/gems/2.2.0/gems/fastlane-1.55.0/bin/fastlane:150:in `run'
	from /Users/quarezz/.rvm/rubies/ruby-2.2.1/lib/ruby/gems/2.2.0/gems/fastlane-1.55.0/bin/fastlane:156:in `<top (required)>'
	from /Users/quarezz/.rvm/rubies/ruby-2.2.1/bin/fastlane:23:in `load'
	from /Users/quarezz/.rvm/rubies/ruby-2.2.1/bin/fastlane:23:in `<main>'
	from /Users/quarezz/.rvm/gems/ruby-2.2.1/bin/ruby_executable_hooks:15:in `eval'
	from /Users/quarezz/.rvm/gems/ruby-2.2.1/bin/ruby_executable_hooks:15:in `<main>'

Also getting this. The Apple ID I'm using does not have multiple teams. Same stack trace.

Yeah, tried on several test projects - no luck. Maybe Apple changed something and spaceship doesn't work on setup right now...

I'm getting an error during setup too but mine is different
/Users/Ilter/.rvm/rubies/ruby-2.2.1/lib/ruby/2.2.0/net/http.rb:923:in `connect': [!] SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (Faraday::SSLError)
	from /Users/Ilter/.rvm/rubies/ruby-2.2.1/lib/ruby/2.2.0/net/http.rb:923:in `block in connect'
	from /Users/Ilter/.rvm/rubies/ruby-2.2.1/lib/ruby/2.2.0/timeout.rb:74:in `timeout'
	from /Users/Ilter/.rvm/rubies/ruby-2.2.1/lib/ruby/2.2.0/net/http.rb:923:in `connect'
	from /Users/Ilter/.rvm/rubies/ruby-2.2.1/lib/ruby/2.2.0/net/http.rb:863:in `do_start'
	from /Users/Ilter/.rvm/rubies/ruby-2.2.1/lib/ruby/2.2.0/net/http.rb:852:in `start'
	from /Users/Ilter/.rvm/rubies/ruby-2.2.1/lib/ruby/2.2.0/net/http.rb:1375:in `request'
	from /Users/Ilter/.rvm/rubies/ruby-2.2.1/lib/ruby/2.2.0/net/http.rb:1133:in `get'
	from /Users/Ilter/.rvm/gems/ruby-2.2.1/gems/faraday-0.9.2/lib/faraday/adapter/net_http.rb:80:in `perform_request'
	from /Users/Ilter/.rvm/gems/ruby-2.2.1/gems/faraday-0.9.2/lib/faraday/adapter/net_http.rb:40:in `block in call'
	from /Users/Ilter/.rvm/gems/ruby-2.2.1/gems/faraday-0.9.2/lib/faraday/adapter/net_http.rb:87:in `with_net_http_connection'
	from /Users/Ilter/.rvm/gems/ruby-2.2.1/gems/faraday-0.9.2/lib/faraday/adapter/net_http.rb:32:in `call'
	from /Users/Ilter/.rvm/gems/ruby-2.2.1/gems/faraday-cookie_jar-0.0.6/lib/faraday/cookie_jar.rb:22:in `call'
	from /Users/Ilter/.rvm/gems/ruby-2.2.1/gems/faraday_middleware-0.10.0/lib/faraday_middleware/response_middleware.rb:30:in `call'
	from /Users/Ilter/.rvm/gems/ruby-2.2.1/gems/faraday_middleware-0.10.0/lib/faraday_middleware/response_middleware.rb:30:in `call'
	from /Users/Ilter/.rvm/gems/ruby-2.2.1/gems/faraday_middleware-0.10.0/lib/faraday_middleware/response_middleware.rb:30:in `call'
	from /Users/Ilter/.rvm/gems/ruby-2.2.1/gems/faraday-0.9.2/lib/faraday/rack_builder.rb:139:in `build_response'
	from /Users/Ilter/.rvm/gems/ruby-2.2.1/gems/faraday-0.9.2/lib/faraday/connection.rb:377:in `run_request'
	from /Users/Ilter/.rvm/gems/ruby-2.2.1/gems/faraday-0.9.2/lib/faraday/connection.rb:140:in `get'
	from /Users/Ilter/.rvm/gems/ruby-2.2.1/gems/spaceship-0.19.3/lib/spaceship/portal/portal_client.rb:22:in `api_key'
	from /Users/Ilter/.rvm/gems/ruby-2.2.1/gems/spaceship-0.19.3/lib/spaceship/portal/portal_client.rb:37:in `send_login_request'
	from /Users/Ilter/.rvm/gems/ruby-2.2.1/gems/spaceship-0.19.3/lib/spaceship/client.rb:173:in `login'
	from /Users/Ilter/.rvm/gems/ruby-2.2.1/gems/spaceship-0.19.3/lib/spaceship/client.rb:57:in `login'
	from /Users/Ilter/.rvm/gems/ruby-2.2.1/gems/spaceship-0.19.3/lib/spaceship/portal/spaceship.rb:22:in `login'
	from /Users/Ilter/.rvm/gems/ruby-2.2.1/gems/spaceship-0.19.3/lib/spaceship/portal/spaceship.rb:77:in `login'
	from /Users/Ilter/.rvm/gems/ruby-2.2.1/gems/fastlane-1.57.0/lib/fastlane/setup/setup_ios.rb:168:in `detect_if_app_is_available'
	from /Users/Ilter/.rvm/gems/ruby-2.2.1/gems/fastlane-1.57.0/lib/fastlane/setup/setup_ios.rb:29:in `run'
	from /Users/Ilter/.rvm/gems/ruby-2.2.1/gems/fastlane-1.57.0/lib/fastlane/setup/setup.rb:19:in `run'
	from /Users/Ilter/.rvm/gems/ruby-2.2.1/gems/fastlane-1.57.0/bin/fastlane:51:in `block (2 levels) in run'
	from /Users/Ilter/.rvm/gems/ruby-2.2.1/gems/commander-4.3.5/lib/commander/command.rb:178:in `call'
	from /Users/Ilter/.rvm/gems/ruby-2.2.1/gems/commander-4.3.5/lib/commander/command.rb:178:in `call'
	from /Users/Ilter/.rvm/gems/ruby-2.2.1/gems/commander-4.3.5/lib/commander/command.rb:153:in `run'
	from /Users/Ilter/.rvm/gems/ruby-2.2.1/gems/commander-4.3.5/lib/commander/runner.rb:428:in `run_active_command'
	from /Users/Ilter/.rvm/gems/ruby-2.2.1/gems/fastlane_core-0.36.1/lib/fastlane_core/ui/fastlane_runner.rb:23:in `run!'
	from /Users/Ilter/.rvm/gems/ruby-2.2.1/gems/commander-4.3.5/lib/commander/delegates.rb:15:in `run!'
	from /Users/Ilter/.rvm/gems/ruby-2.2.1/gems/fastlane-1.57.0/bin/fastlane:150:in `run'
	from /Users/Ilter/.rvm/gems/ruby-2.2.1/gems/fastlane-1.57.0/bin/fastlane:156:in `<top (required)>'
	from /Users/Ilter/.rvm/gems/ruby-2.2.1/bin/fastlane:23:in `load'
	from /Users/Ilter/.rvm/gems/ruby-2.2.1/bin/fastlane:23:in `<main>'
	from /Users/Ilter/.rvm/gems/ruby-2.2.1/bin/ruby_executable_hooks:15:in `eval'
	from /Users/Ilter/.rvm/gems/ruby-2.2.1/bin/ruby_executable_hooks:15:in `<main>'

This occurs after I enter my Apple ID and password. Any info about that?

These are all different errors. @iltercengiz your issue is covered here https://github.com/fastlane/fastlane/blob/master/docs/FAQs.md#im-getting-an-ssl-error

@KrauseFx tried it but no chance. Still getting the same error. Also I've updated all gems.

Fixed the SSL error. A little research led me here:
https://github.com/rvm/rvm/issues/3330 and then here:
https://toadle.me/2015/04/16/fixing-failing-ssl-verification-with-rvm.html
TL;DR: If ruby is managed by rvm and the installed version is the binary version, ruby is searching the certs in the wrong directory. Instead, install ruby from source locally by doing
rvm install 2.2.1 --disable-binary
This way it will find certs.

It looks like there hasn’t been any recent activity on this issue in some time, so I’m closing it. If this is still a problem, feel free to re-open a new issue and I’d be happy to help :+1:


{"code":-32000,"message":"only replay-protected (EIP-155) transactions allowed over RPC"}

When i try to transfer funds from www.coinfy.com wallet i am getting this error
{"code":-32000,"message":"only replay-protected (EIP-155) transactions allowed over RPC"}
tried emailing the support team for www.coinfy.com but still haven't received any reply, i have deposited some btc to the same wallet but the BTC is still not deposited to my wallet and in the blockchain the transaction was completed. is this www.Coinfy.com a SCAM?
https://github.com/elevenyellow/coinfy

I believe this is similar to https://community.infura.io/t/polygon-rpc-only-replay-protected-eip-155-transactions-allowed-over-rpc/2997.
Try passing chainId in your transaction


Just single Pixel rendering

Hello @vye16, @liruilong940607, @kerrj,
Is it possible to render just single pixel which returns the color of each single ray from different colmap camera positions using gsplat?
I am interested in getting final color at particular location for different view directions, by providing GS pointcloud, colmap camera poses, "location of target pixel for tracking in first fully rasterized image using first colmap camera pose from colmap pose sequence" or "3D location in scene (how to find it?)".

Hey Hi again, are you able to achive this ?

Hey. I'm not sure what is the actual need here. If you want to get the rendering for a particular pixel, is there anything preventing you from rendering a image and get the pixel color from it?

Ah I see. The current master does not support that because everything is fused in CUDA.
However, we are working on an upgrade to the CUDA backend in this PR #172, where a pure python implementation is support so that you can easily get that information. In the code below we are accumulating all Gaussians that contribute to each pixel in Python:
https://github.com/nerfstudio-project/gsplat/blob/f4d455e63df0282612aa852a28c3803778f7bd08/gsplat/experimental/cuda/_torch_impl.py#L267-L282

Thank you so much ❤ I'll look into it.

Ah I see. The current master does not support that because everything is fused in CUDA.
However, we are working on an upgrade to the CUDA backend in this PR #172, where a pure python implementation is support so that you can easily get that information. In the code below we are accumulating all Gaussians that contribute to each pixel in Python:
https://github.com/nerfstudio-project/gsplat/blob/f4d455e63df0282612aa852a28c3803778f7bd08/gsplat/experimental/cuda/_torch_impl.py#L267-L282
If you are in a hurry need then you can just fetch that PR and use it. The PR itself is already seriously verified. Just we are thinking a best way of merging it without too much of interrupt on the master

I'm facing issues while I'm trying to run the example script alone
Traceback (most recent call last):
  File "example.py", line 33, in <module>
    from nerfdata.dataset.colmap.dataset import Dataset
ModuleNotFoundError: No module named 'nerfdata'

could you please help me solve this ?

That is a dependence we are not open-source yet.
So that means the script in this PR is not directly executable. As I mentioned above, we are figuring out the best way of merging it in without outsourced dependence.
However it should be fairly easy to swap our nerfdata with your own data parser -- it is simply just for parsing a colmap capture.

Thanks for your reply. I'm working on it, and in the meantime, I have a small doubt: is there any render.py available in Gsplat like in '''graphdeco-inria''' where I can render the splats directly once the model is trained entirely so I can debug and modify code easily?

If what you are looking for is a minimal script only to render a splat, we don't have it yet but we will add it together with that PR soon.
But it should not be very hard to create one quickly from that standalone example script in this PR.

Thanks for your help, but I'm new to the Nerfstudio codebase, so could you please help me with how I can pass transforms.json instead of nerfdata, or please tell me what K is from the dataset so I can modify it and pass it.
dataset = Dataset(data_dir=f"data/360_v2/{args.scene}/", factor=8, normalize=True)

    # preload
    images = np.stack(
        [imageio.imread(image_path)[..., :3] for image_path in dataset.image_paths]
    )
    images = torch.from_numpy(images).float().to(device) / 255.0
    c2ws = torch.from_numpy(dataset.camtoworlds).float().to(device)
    K = torch.from_numpy(dataset.K).float().to(device)
    points = torch.from_numpy(dataset.points).float()
    points_rgb = torch.from_numpy(dataset.points_rgb).float()
    n_images, height, width = images.shape[:3]
    ```

  what is ``` K ``` here ? Thank you.

Thank

Ah I see. The current master does not support that because everything is fused in CUDA.
However, we are working on an upgrade to the CUDA backend in this PR #172, where a pure python implementation is support so that you can easily get that information. In the code below we are accumulating all Gaussians that contribute to each pixel in Python:
https://github.com/nerfstudio-project/gsplat/blob/f4d455e63df0282612aa852a28c3803778f7bd08/gsplat/experimental/cuda/_torch_impl.py#L267-L282
If you are in a hurry need then you can just fetch that PR and use it. The PR itself is already seriously verified. Just we are thinking a best way of merging it without too much of interrupt on the master

Thank you for your work. I was successfully able to train the model and I have one doubt about where you are calling this accumulate() in your code. I read your code, but I don't think you are calling it anywhere, so could you please tell me where to use this function exactly?

That's used in this function:
https://github.com/nerfstudio-project/gsplat/blob/f4d455e63df0282612aa852a28c3803778f7bd08/gsplat/experimental/test.py#L303-L305
Which is a exact substitute of the rendering function.
Note that if you call _rendering function, you should expect much higher memory usages and much slower speed as it is running rasterization in pure python. That's just the price have to pay in exchange for flexibility

When the training is happening, I clearly see that rasterization is happening; however, when I try to run the rasterization alone, I keep ending up with a big, plain single-color image. Could you please help me?
rendering Script :
def test_rasterize_to_pixels():
      from gsplat.experimental.cuda import _rendering,
      data_path = "results/splatfacto_model_data.npz"

      data = np.load(data_path)  # 3 cameras
      height, width = data["height"].item(), data["width"].item()
      viewmats = torch.from_numpy(data["viewmats"]).to(device)
      Ks = torch.from_numpy(data["Ks"]).to(device)

      means = torch.from_numpy(data["means3d"]).to(device)

      scales = torch.from_numpy(data["scales"]).to(device) * 0.1 + 0.1

      quats = torch.from_numpy(data["quats"]).to(device)

      opacities = torch.from_numpy(data["opacities"]).to(device)
      opacities = opacities.squeeze()

      Ks = torch.tensor([[320.0, 0.0, 320.0],
                         [0.0, 320.0, 320.0],
                         [0.0, 0.0, 1.0]]).unsqueeze(0).to(device=device)  # Single camera matrix
      N = data["viewmats"].shape[0]  # Assuming viewmats is [C, 4, 4]
      Ks = Ks.repeat(N, 1, 1)

      colors = torch.from_numpy(data["colors"]).to(device)
      colors = colors.unsqueeze(0).repeat(N, 1, 1)

      print(colors.shape)
      C = len(Ks)

      viewmats.requires_grad = True
      quats.requires_grad = True
      scales.requires_grad = True
      means.requires_grad = True

      render_colors, render_alphas = _rendering(
          means, quats, scales, opacities, colors, viewmats, Ks, width, height
      )

      return render_colors, render_alphas

 render_colors, render_alphs = test_rasterize_to_pixels()

But when I plot the render_colors, I end up with the same color on all pixels.

You probably don't want *0.1+0.1 in this line if you have a trained model already.
scales = torch.from_numpy(data["scales"]).to(device) * 0.1 + 0.1

Yeah, I changed that, but still, I'm facing the same issue. Could you please tell me how I can pass my own camera extrinsic from transforms? json, when I try to load it, it shows grad_fn is required, so I tried gradient = true, false, and still I can't pass my own camera extrinsic. So far, I'm passing dummy.

Ah I see. The current master does not support that because everything is fused in CUDA.
However, we are working on an upgrade to the CUDA backend in this PR #172, where a pure python implementation is support so that you can easily get that information. In the code below we are accumulating all Gaussians that contribute to each pixel in Python:
https://github.com/nerfstudio-project/gsplat/blob/f4d455e63df0282612aa852a28c3803778f7bd08/gsplat/experimental/cuda/_torch_impl.py#L267-L282
If you are in a hurry need then you can just fetch that PR and use it. The PR itself is already seriously verified. Just we are thinking a best way of merging it without too much of interrupt on the master

@liruilong940607 Thank you for your great work. I am wondering if this something it will do, or information that will be possible to get? From another thread, I asked for this but I do not think the community knows this yet as it might yet to be implemented. But as I read this, it seems that this will be supported soon with this PR? See below comment if that will solve it;

With COLMAP, it is possible to extract data to determine which images and keypoints correspond to specific 3D points and to identify which images have been matched.
I am wondering if it is possible to track which Gaussian splats correspond to which keypoints, along with their associated image files and pixel coordinates via some sort of index? Is there a way to "tap" into the code to keep track of this when using Nerfstudio/GSplat?
Specifically, I want to identify the originating image and pixel coordinate for each Gaussian splat (with respect to the gaussian splatted point cloud). While COLMAP provides this information for 3D points, I would like to know if it is possible to extend this tracking to the exported PLY file for the splats.

I've created a script for COLMAP that consolidates the points3d.bin, camera.bin, and images.bin files into a single file. This simplifies the interpretation of data, clarifying contributions to pixel coordinates, image filenames, and their indices.
Example such as below;

POINT3D_ID 3D_X 3D_Y 3D_Z R G B POINT2D_IDX PIXEL_X PIXEL_Y IMAGE_ID IMAGE_NAME
1054 -2.2514710427743658 0.34472229191518061 -0.31922396259922448 169<PHONE_NUMBER> 2189.711181640625 2253.42578125 83 IMG_2692.JPG
1054 -2.2514710427743658 0.34472229191518061 -0.31922396259922448 169<PHONE_NUMBER> 2936.756103515625 2486.4677734375 85 IMG_2694.JPG
1054 -2.2514710427743658 0.34472229191518061 -0.31922396259922448 169<PHONE_NUMBER><PHONE_NUMBER>1015625 2102.93994140625 80 IMG_2689.JPG
446883 3.5806275984290243 0.52853412315400827 2.527234807406364 193<PHONE_NUMBER> 3242.96533203125 2536.799072265625 262 IMG_2871.JPG
446883 3.5806275984290243 0.52853412315400827 2.527234807406364 193<PHONE_NUMBER> 2603.289794921875 2463.16650390625 263 IMG_2872.JPG

I am looking to do the same with GSplat, if possible.

Hi @abrahamezzeddine, the splitting of the Gaussians are control on the python side, not CUDA, so it should be very doable to track the IDs of the GSs by hacking the python code.

Hi @abrahamezzeddine, the splitting of the Gaussians are control on the python side, not CUDA, so it should be very doable to track the IDs of the GSs by hacking the python code.

Hello @liruilong940607
Could you please guide me a little bit? I am not very familiar with the GSplat Code in general, so a little push in the right direction and I should be able to do the rest, hopefully. :)

Gsplat repo will have a big update very soon, in which we will put a standalone training script in. For now I could only point you to the nerfstudio repo.
The splitting & cloning & pruning is happening here in nerfstudio's implementation.
https://github.com/nerfstudio-project/nerfstudio/blob/742963e07369b70912b786a100e94c98dec1b742/nerfstudio/models/splatfacto.py#L439-L494
So I guess what you want is to maintain a tensor storing the initial IDs, and split/clone/prune that tensor together with all other attributes of GS.

@hariharan1412 are you able to achieve this? how to do it using original 3D-GS?

Could you explain your question more in detail? as you can see, we discussed lot of things here, so please be more specific about what you are asking

Could you explain your question more in detail? as you can see, we discussed lot of things here, so please be more specific about what you are asking
I am talking about this:
Is it possible to render just single pixel which returns the color of each single ray from different colmap camera positions using gsplat?
I am interested in getting final color at particular location for different view directions, by providing GS pointcloud, colmap camera poses, "location of target pixel for tracking in first fully rasterized image using first colmap camera pose from colmap pose sequence" or "3D location in scene (how to find it?)".

Yes I can,
refer this : https://github.com/nerfstudio-project/gsplat/issues/180#issue-2306505326

hello @liruilong940607,
thanks for the great update to gsplat.
how can I use accumulate() in your fork?
how can I update that code?
Thanks.

Hi @pknmax
I just added a function _rasterization() to the latest main branch, which aligns with the rasterization() function (with a few advanced arguments not supported) but uses the accumulate() function to render the image. So you can easily swap
between the two.
https://github.com/nerfstudio-project/gsplat/blob/56f186749edf500fbd023b4b73cd82beef17de3a/gsplat/rendering.py#L406

Hi @liruilong940607,
does this _rasterization() approach takes  20+hrs instead of 20/25 min for optimization?

yes because its fully pytorch implementation (no CUDA kernel fusion).

@liruilong940607 can we use rasterize_to_indices_in_range() and accumulate() with optimized gaussians ply file by rasterization()  approach and track a pixel for different camera poses?

Yes these functions are perfectly aligned. In fact we were using rasterize_to_indices_in_range() and accumulate() to test our CUDA implementation of rasterization()

@liruilong940607 I tied using _rasterize_to_pixels with optimized gaussians ply file by rasterization(), which in-turn uses rasterize_to_indices_in_range() and accumulate()
; but just getting black/empty image (..../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [7,0,0], thread: [16,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed...)
```

ply_file_path = "/work/3DGSoutput/struct_color_taylor_primaries_plyaug_gsplat_extended/point_cloud/taylor_swift_extended_cropped.ply"
colmap_path = "/work/3DGSdatasets/struct_color_taylor_primaries/"
print("Loading PLY file...")
 # Loading ply files
pc = GaussianModel(sh_degree=3)
pc.load_ply(ply_file_path)
bg_color = [0, 0, 0]
bg_color = torch.tensor(bg_color, dtype=torch.float32, device="cuda")
scaling_modifier = 1.0
means3D = pc.get_xyz
opacity = pc.get_opacity
scales = pc.get_scaling * scaling_modifier
rotations = pc.get_rotation
colors = pc.get_features # [N, K, 3]
sh_degree = pc.active_sh_degree

print("Loading colmap data...")
camera_params = dummyReadColmapSceneInfo(colmap_path)

for frame in camera_params[0]:
    print(f"Processing frame: {frame.name}")

    #########
    # test rasterization
    ########
    viewmat = frame.world_view_transform.transpose(0, 1) # [4, 4]

    tanfovx = math.tan(frame.FoVx * 0.5)
    tanfovy = math.tan(frame.FoVy * 0.5)
    focal_length_x = frame.image_width / (2 * tanfovx)
    focal_length_y = frame.image_height / (2 * tanfovy)
    K = torch.tensor(
        [
            [focal_length_x, 0, frame.image_width / 2.0],
            [0, focal_length_y, frame.image_height / 2.0],
            [0, 0, 1],
        ],
        device="cuda",
    )

    render_colors, render_alphas, info = rasterization(
                                                            means=means3D,  # [N, 3]
                                                            quats=rotations,  # [N, 4]
                                                            scales=scales,  # [N, 3]
                                                            opacities=opacity.squeeze(-1),  # [N,]
                                                            colors=colors,
                                                            viewmats=viewmat[None],  # [1, 4, 4]
                                                            Ks=K[None],  # [1, 3, 3]
                                                            backgrounds=bg_color[None],
                                                            width=int(frame.image_width),
                                                            height=int(frame.image_height),
                                                            packed=False,
                                                            sh_degree=sh_degree,
                                                            # render_mode = "RGB+D",
                                                            rasterize_mode = "antialiased",
                                                            absgrad = True,
                                                        )

    rendered_image = render_colors[0].permute(2, 0, 1)
    image_array = (rendered_image.detach().cpu().permute(1, 2, 0).numpy() * 255).astype('uint8')
    image_pil = Image.fromarray(image_array)
    image_pil.show()
    # image_pil.save("/HPS/ColorNeRF/work/gaussian-splatting-gsplat/output_test_5.png")

    #########
    # test _rasterize_to_pixels
    ########
    Ks = K[None]
    viewmats = viewmat[None]
    height = int(frame.image_width)
    width = int(frame.image_height)
    quats = rotations
    scales = scales * 0.1
    means = means3D
    opacities = opacity.squeeze(-1)
    backgrounds=bg_color[None]
    C = len(Ks)

    covars, _ = quat_scale_to_covar_preci(quats, scales, compute_preci=False, triu=True)

    # Project Gaussians to 2D
    radii, means2d, depths, conics, compensations = fully_fused_projection(
        means, covars, None, None, viewmats, Ks, width, height
    )
    opacities = opacities.repeat(C, 1)

    # Identify intersecting tiles
    tile_size = 16 #if channels <= 32 else 4
    tile_width = math.ceil(width / float(tile_size))
    tile_height = math.ceil(height / float(tile_size))

    tiles_per_gauss, isect_ids, flatten_ids = isect_tiles(
        means2d, radii, depths, tile_size, tile_width, tile_height
    )

    isect_offsets = isect_offset_encode(isect_ids, C, tile_width, tile_height)

    _render_colors, _render_alphas = _rasterize_to_pixels(
    means2d,
    conics,
    colors,
    opacities,
    width,
    height,
    tile_size,
    isect_offsets,
    flatten_ids,
    backgrounds=backgrounds,
    )

    rendered_image = _render_colors[0].permute(2, 0, 1)
    image_array = (rendered_image.detach().cpu().permute(1, 2, 0).numpy() * 255).astype('uint8')
    image_pil = Image.fromarray(image_array)
    image_pil.show()

    break

hi @liruilong940607 getting this kind of rendering, can you please tell why?

what is your sh_degree? If it is not None you would need this line:
https://github.com/nerfstudio-project/gsplat/blob/56f186749edf500fbd023b4b73cd82beef17de3a/gsplat/rendering.py#L532-L533
Btw, one tip I can give for debugging is that you could run
from gsplat.rendering import rasterization, _rasterization

render_colors, render_alphas, info = rasterization(
                                                                means=means3D,  # [N, 3]
                                                                quats=rotations,  # [N, 4]
                                                                scales=scales,  # [N, 3]
                                                                opacities=opacity.squeeze(-1),  # [N,]
                                                                colors=colors, # [N, K, 3]
                                                                viewmats=viewmat[None],  # [1, 4, 4]
                                                                Ks=K[None],  # [1, 3, 3]
                                                                backgrounds=bg_color[None],
                                                                width=int(frame.image_width),
                                                                height=int(frame.image_height),
                                                                packed=False,
                                                                sh_degree=sh_degree,
                                                                # render_mode = "RGB+D",
                                                                # rasterize_mode = "antialiased",
                                                                # absgrad = True,
                                                                )
 _render_colors, _render_alphas, _info = _rasterization(
                                                                means=means3D,  # [N, 3]
                                                                quats=rotations,  # [N, 4]
                                                                scales=scales,  # [N, 3]
                                                                opacities=opacity.squeeze(-1),  # [N,]
                                                                colors=colors, # [N, K, 3]
                                                                viewmats=viewmat[None],  # [1, 4, 4]
                                                                Ks=K[None],  # [1, 3, 3]
                                                                backgrounds=bg_color[None],
                                                                width=int(frame.image_width),
                                                                height=int(frame.image_height),
                                                                packed=False,
                                                                sh_degree=sh_degree,
                                                                # render_mode = "RGB+D",
                                                                # rasterize_mode = "antialiased",
                                                                # absgrad = True,
                                                                )

These two should produce the same image. Then you can open up the _rasterization function like you are doing now and see where went wrong.

hello @liruilong940607,
thanks for the help.
Suppose, if i want to just track and rasterize two pixel locations [(659, 776.), (718, 778)] from first rasterized frame w.r.t. first cam pose,
how do I pass/use these locations/ids in accumulate() to rasterize only those tracked locations for different cam poses?

If you are looking for reading out an optical flow, you don't need to open up the rasterization function. You could reproject the pixel to 3D using the rendered depth map, and then project it to another image plane to get the corresponded pixel location in the other frame.
Here is a relevant issue: https://github.com/nerfstudio-project/gsplat/issues/203
Is that what you need?

Hello @liruilong940607,
I tried the approach you suggested but the tracking is not good enough when camera moves a lot.
I have sent you some images to look at.
Is there anything I can do to improve the results?
I would really appreciate your help.
Thank you very much.

If the depth is not accurate, (which it can be because Gaussian splatting only optimize for rendered colors, not the geometry), then there is nothing we can do to get an accurate flow map from that.

hi @liruilong940607, can you please check the images of sample depth maps I sent to you via email? And tell me whether it is problem with depth map or something else?


Proposing CredentialProvider interface

Creating Interface in order to let custom credential provider register user's secret keys.
This PR only proposes an interface, and the next step will be to use this interface in HadoopSecurityManager_H_2_0 class.
Note: the changes in HadoopSecurityManager_H_2_0.java are automatically made by save action IntelliJ plugin.

What is the custom credential provider? Can you give a little more detail on what you are planning to do with this new interface?

Description Updated. Thanks. @jamiesjc


Bumping bigtable versions

Bumping bigtable versions

Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

It will help us expedite review of your Pull Request if you tag someone (e.g. @username) to look at it.
Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---
---
---
---
---

Java

Python

---

---
---
---
---

cc @lukecwik @sduskis this is ready to review


Integrate with Confluent schema registry (Avro)

Would be nice to have a non-JVM alternative to kafka-avro-console-consumer

can you elaborate what this would do for the CLI? e.g. displaying the avro payloads in JSON format?
i'm not actively using avro as our techstack uses mostly protobuf and JSON.

Sure!  The goal would be to publish or consume Avro-encoded messages from the CLI using the Confluent schema registry for encoding or decoding.
Currently to do so from the command line requires having the Confluent stack installed along with the JVM.  There would be an additional CLI option required to point to the schema registry's endpoint as a Kafka property.
Example for kafka-avro-console-consumer:
--property schema.registry.url=https://confluent-registry.foo.bar

okay so the console producer/consumer just reads/prints avro in JSON format, from what i read here: https://docs.confluent.io/1.0.1/quickstart.html
so this would entail fetching the schema from schema registry, and using an avro library to pretty convert to/from avro. could be interesting. i'd like to have the same for protobuf, but the tooling is rather difficult as there's no really good schema registry out there for proto.


dragging from retina to non-retina, or the other way around, doenst work

You have to blow away your glyph atlas and rerender at higher dpi for this to work properly, using the scale factor on that new display.

Looks like repeat of #71

agreed


Tighten Send and Sync requirements

This is a breaking change to rayon-core, necessary for soundness.

Scope::spawn now requires Send for the closure.  Fixes #430.

ThreadPool::install now requires Send for the return value.

(unstable) internal::Task now requires Send + Sync instead of just
Send, so its use of Arc properly implements Send and Sync too.

Added a fix for #446, since that also needed the change to install() here.


Ordering works only for a single page:

          Ordering works only for a single page:

Originally posted by @nulls in https://github.com/saveourtool/save-cloud/issues/2715#issuecomment-1764263548

Not a backlog actually, for tables framework it is very important for out next projects


Rumble chat embed

Rumble recently added popout chats, and even recently-er allowed those popout chats to be embedded on other domains.
So I've added functionality along with YT and Twitch chats.
This implementation is a little more complex, though, because I have to make a request to get the embedded stream's HTML to find the id of that stream's chat.
As far as I can tell there isn't an API to get this information, so I'm downloading the HTML and running a regex search on it.
This is somewhat fragile, and while it should fail gracefully, I would not be surprised if the code becomes outdated as the embedded stream's HTML changes.
Up to you if you want to add it :)

Heyo! Thanks for the PR, this seems like a good thing to add. Parsing the page for the ID is a little clunky and, luckily, I know a better way to get it (haven't tested it much, but I don't see any reason it shouldn't work all the time): the numerical ID is just the embed ID (without the 'v') converted from 36 base to decimal 😄
So, for example, for embed ID 'v2an734', where chat popup URL is 'https://rumble.com/chat/popup/138810784' (just picked some random rumble stream dont judge me PepeLaugh), you can just do parseInt('2an734', 36), which gets you 138810784 😎
Once you change that up I'll merge ofc!
Thanks again ❤

That's an impressive find; I just assumed those were different ids, not different bases of the same number 😮
That makes this a lot easier, and basically the same flow as YT and Twitch. Thanks for the help :heart:
I've updated the PR to just convert the embed id to the chat id, and there's no more additional query.

Yeah, I spent a day trying to figure out how they store their stuff after that Sneako stream that Steve was on got privated :smile: You can also get some other ID from the regular URL as well, which you can convert to some base64-ish string that you can use to get the mp4 files from their CDN. Made a lidl Go program for it, should prob make a repo for it.
Anyway, this looks good! Thanks for the PR as always :heart:


No need to namespace CSS anymore

Now that they're in an iframe it's no longer needed.
Takes some effort out of troubleshooting legacy gadgets.
Moves us one step closer to username independence.

@ThrivingKings You'll like this! (;

yesssss!


set new leftDrawerWidth animated

On iPad, I have a search bar in my left navigation. On focus I want to expand the left navigation width. This works as expected, but I want it smooth. Is that possible?

Hey,
I implemented this a while back:
mainWindow.setLeftDrawerWidth({
    width: 160,
    animated: true
});

Just have to read the docs... :camel:


Add support for Collections

Feature Request
Adding support for Collections
https://www.reddit.com/dev/api/#section_collections
https://www.reddit.com/r/changelog/comments/bgifzd/an_update_on_making_it_easier_to_host_events_on/
https://mods.reddithelp.com/hc/en-us/articles/360027311431
Currently, only the same mod who created the collection is allowed to add to the collection, making this perfect for a bot account. I would utilize flairs and keywords to automatically add items to a collection.
As someone who is not at all qualified or knowledgeable to make this change, thank you all so much for your help. You all have made my life infinitely easier.

One thing to note is that all the endpoints in the documentation you linked are currently tagged *** BETA ***. That said, I agree that it makes sense for PRAW to support this feature.
I'd like to take this on. I'll get started this weekend.

Another thing which should be added as part of this is updating the submit arguments for "submit to collection," which automatically adds a new post to a collection: https://www.reddit.com/dev/api/#POST_api_submit

@bboe I'd like your opinion and advice on something here:
Collections do not have fullnames (like t3_bgj1yr). They do, however, have UUIDs which can be used to fetch them individually. The RedditBase class provides all sorts of useful functionality, including lazy-loads when an unknown attribute is accessed and __str__ and __repr__ methods. It also makes the assumption that all of its subclasses are types that have fullnames.
My dilemma is that I would like to use the above-mentioned useful features of RedditBase, but I don't think Collections fully map on to the model that RedditBase provides because of the lack of fullnames.
How do you recommend I proceed?

@bboe seems to have gone AWOL.
Would returning "Collection_" + UUID not be appropriate as a fullname for a collection? I personally wouldn’t mind fabricating a fullname if Reddit doesn’t provide one. We could define the fullname property to be “return the fullname or the closest thing to one”.
Apart from the fullname, is there any other reason why a collection might not fit the RedditBase model? @bboe has expressed removing fullname as a model attribute in future. Maybe we can accelerate that goal if Reddit is no longer doing fullnames?

Thanks for weighing in @Pyprohly!
I would rather not have to fabricate a fullname when one doesn't exist. I think that it's somehow "misleading" or "lying" to do that.
Thanks for linking bboe's comment regarding removal of fullname. I think that it probably would make sense to turn fullname support into a Mixin, or perhaps to create a new class in the inheritance hierarchy between PRAWBase and RedditBase that provides all of the non-fullname-related helper methods.

is there any other reason why a collection might not fit the RedditBase model?

I don't think there is. RedditBase's _fetch method supports custom "info paths"  which would work with Collections.

Maybe we can accelerate that goal if Reddit is no longer doing fullnames?

That's a good observation. It does seem like Reddit may be moving away from fullnames. Or, perhaps fullnames were only ever meant to be used for the six "first class" types that they are currently used for (Comment, Account, Link, Message, Subreddit, Award).

Hi all. Still around, just not a ton of time to look too deeply into things. I will support whatever approach you think is best @jarhill0. If that means creating a new base class that doesn't include fullname, or hacking UUID onto fullname somehow, either can work.

Thanks for linking bboe's comment regarding removal of fullname. I think that it probably would make sense to turn fullname support into a Mixin

That’s just like the ID_MIXIN idea @bboe had. I don’t really like this plan. It doesn’t seem sensible to move only one (or maybe two if id is included) properties into a new mixin.
Why not just treat and write fullname as a regular attribute in like the rest?
If the new attribute system I’ve proposed gets approved then it can be neatly separated away from the other JSON ones.
I’ve designed the new attribute system (I mean the new, new one whose details I haven’t described anywhere here yet) to be able to generate custom attributes on demand, and fullname is the epitome of the use for this feature. fullname would need to eventually be integrated into this system anyway since it would not be very friendly to allow praw_object.fullname but not praw_object.a.fullname.

or perhaps to create a new class in the inheritance hierarchy between PRAWBase and RedditBase that provides all of the non-fullname-related helper methods.

What non-fullname-related helper methods are you referring to? I can only see you mean moving _fetch() out of RedditBase. That would only create another mixin.
The real problem here is that _fetch() needs to be written to not use fullname. Derived classes should explicitly define what attribute of theirs is to be used as their info param(s).

Why not just treat and write fullname as a regular attribute in like the rest?

Because for those six types mentioned above, the format and use is the same. For that reason, I don't think we should define fullname separately in six places.

and fullname is the epitome of the use for this feature

Awesome! I look forward to it 😁😁

What non-fullname-related helper methods are you referring to?

All the methods in RedditBase aside from fullname.

The real problem here is that _fetch() needs to be written to not use fullname

It does work without fullname. See the link in my previous comment.

Because for those six types mentioned above, the format and use is the same. For that reason, I don't think we should define fullname separately in six places.

Fair enough on that point. We should avoid duplication. And it’s clear that some grouping is going on there, so a mixin for fullname wouldn’t be completely unwise.
Rather, I guess a new mixin is necessary then.
One thing that bothers me is that you now have to come up with a mixin name that somehow includes all the RedditBase classes that have a fullname (which is, currently, all of them, not just the six) but doesn’t include Collections. FullnameMixin? That doesn’t sound too good.

It does work without fullname. See the link in my previous comment.

Of course. Just a poor choice of wording on my part…
What I would really like to point out is that fullname being mentioned in _fetch() makes it have more of a presence over RedditBase than we would likely want to keep now that we’re planning on moving it. If we’re going to remove the fullname property from RedditBase then we need to have it not be directly mentioned in _fetch(). The best way to do this is to just have derived classes explicitly define their info params.

or perhaps to create a new class in the inheritance hierarchy between PRAWBase and RedditBase that provides all of the non-fullname-related helper methods.

What non-fullname-related helper methods are you referring to?

All the methods in RedditBase aside from fullname.

Why? That would seem to defeat the purpose of having RedditBase.

If we’re going to remove the fullname property from RedditBase then we need to have it not be directly mentioned in _fetch().

I'm going to take a stab at this. I'll override the _fetch() method in FullnameMixin (or whatever I call it).

Why? That would seem to defeat the purpose of having RedditBase.

Yeah, you're right. That's a bad idea. A mixin is much better.

Hey, what about AttributeMixins.Fullname?
fullname is the only custom attribute on PRAW objects at the moment, but I can see myself adding more to AttributeMixins in future.

Hey, what about AttributeMixins.Fullname?
fullname is the only custom attribute on PRAW objects at the moment, but it might be useful to add more to AttributeMixins in future.

Pull submitted! Thanks for creating this issue @jonathanwall.


No Credentials error

I am getting below error , while I try to run the executor function
wrenexec = pywren.default_executor()
NoCredentialsError                        Traceback (most recent call last)
 in ()
----> 1 wrenexec = pywren.default_executor()
/usr/local/lib/python3.5/dist-packages/pywren/wren.py in default_executor(**kwargs)
19
20     if executor_str == 'lambda':
---> 21         return lambda_executor(**kwargs)
22     elif executor_str == 'remote' or executor_str == 'standalone':
23         return remote_executor(**kwargs)
/usr/local/lib/python3.5/dist-packages/pywren/wren.py in lambda_executor(config, job_max_runtime)
35     invoker = invokers.LambdaInvoker(AWS_REGION, FUNCTION_NAME)
36
---> 37     return Executor(invoker, config, job_max_runtime)
38
39
/usr/local/lib/python3.5/dist-packages/pywren/executor.py in init(self, invoker, config, job_max_runtime)
38         self.storage_config = wrenconfig.extract_storage_config(self.config)
39         self.storage = storage.Storage(self.storage_config)
---> 40         self.runtime_meta_info = runtime.get_runtime_info(config['runtime'])
41
42
/usr/local/lib/python3.5/dist-packages/pywren/runtime.py in get_runtime_info(runtime_config)
8     Download runtime information from storage at deserialize
9     """
---> 10     runtime_meta = storage.get_runtime_info(runtime_config)
11
12     if not runtime_valid(runtime_meta):
/usr/local/lib/python3.5/dist-packages/pywren/storage/storage.py in get_runtime_info(runtime_config)
110     if '.tar.gz' in key:
111         key = key.replace(".tar.gz", ".meta.json")
--> 112     json_str = handler.get_object(key)
113     runtime_meta = json.loads(json_str.decode("ascii"))
114     return runtime_meta
/usr/local/lib/python3.5/dist-packages/pywren/storage/s3_backend.py in get_object(self, key)
33         """
34         try:
---> 35             r = self.s3client.get_object(Bucket=self.s3_bucket, Key=key)
36             data = r['Body'].read()
37             return data
/usr/local/lib/python3.5/dist-packages/botocore/client.py in _api_call(self, *args, **kwargs)
312                     "%s() only accepts keyword arguments." % py_operation_name)
313             # The "self" in this scope is referring to the BaseClient.
--> 314             return self._make_api_call(operation_name, kwargs)
315
316         _api_call.name = str(py_operation_name)
/usr/local/lib/python3.5/dist-packages/botocore/client.py in _make_api_call(self, operation_name, api_params)
597         else:
598             http, parsed_response = self._endpoint.make_request(
--> 599                 operation_model, request_dict)
600
601         self.meta.events.emit(
/usr/local/lib/python3.5/dist-packages/botocore/endpoint.py in make_request(self, operation_model, request_dict)
141         logger.debug("Making request for %s (verify_ssl=%s) with params: %s",
142                      operation_model, self.verify, request_dict)
--> 143         return self._send_request(request_dict, operation_model)
144
145     def create_request(self, params, operation_model=None):
/usr/local/lib/python3.5/dist-packages/botocore/endpoint.py in _send_request(self, request_dict, operation_model)
166     def _send_request(self, request_dict, operation_model):
167         attempts = 1
--> 168         request = self.create_request(request_dict, operation_model)
169         success_response, exception = self._get_response(
170             request, operation_model, attempts)
/usr/local/lib/python3.5/dist-packages/botocore/endpoint.py in create_request(self, params, operation_model)
150                 op_name=operation_model.name)
151             self._event_emitter.emit(event_name, request=request,
--> 152                                      operation_name=operation_model.name)
153         prepared_request = self.prepare_request(request)
154         return prepared_request
/usr/local/lib/python3.5/dist-packages/botocore/hooks.py in emit(self, event_name, **kwargs)
225                  handlers.
226         """
--> 227         return self._emit(event_name, kwargs)
228
229     def emit_until_response(self, event_name, **kwargs):
/usr/local/lib/python3.5/dist-packages/botocore/hooks.py in _emit(self, event_name, kwargs, stop_on_response)
208         for handler in handlers_to_call:
209             logger.debug('Event %s: calling handler %s', event_name, handler)
--> 210             response = handler(**kwargs)
211             responses.append((handler, response))
212             if stop_on_response and response is not None:
/usr/local/lib/python3.5/dist-packages/botocore/signers.py in handler(self, operation_name, request, **kwargs)
88         # this method is invoked to sign the request.
89         # Don't call this method directly.
---> 90         return self.sign(operation_name, request)
91
92     def sign(self, operation_name, request, region_name=None,
/usr/local/lib/python3.5/dist-packages/botocore/signers.py in sign(self, operation_name, request, region_name, signing_type, expires_in, signing_name)
154                     raise e
155
--> 156             auth.add_auth(request)
157
158     def _choose_signer(self, operation_name, signing_type, context):
/usr/local/lib/python3.5/dist-packages/botocore/auth.py in add_auth(self, request)
418         self._region_name = signing_context.get(
419             'region', self._default_region_name)
--> 420         super(S3SigV4Auth, self).add_auth(request)
421
422     def _modify_request_before_signing(self, request):
/usr/local/lib/python3.5/dist-packages/botocore/auth.py in add_auth(self, request)
350     def add_auth(self, request):
351         if self.credentials is None:
--> 352             raise NoCredentialsError
353         datetime_now = datetime.datetime.utcnow()
354         request.context['timestamp'] = datetime_now.strftime(SIGV4_TIMESTAMP)
NoCredentialsError: Unable to locate credentials

Yeah Vaishaal, I have completed the setup on Ubuntu terminal including setting up the AWS credentials and after that when i am trying to run any program using pywren on jupyter notebook , it gives me this error.

did you run pywren-setup?

Yes, I run the whole setup on Ubuntu terminal

it is returning "Hello World"

It is giving me the same error
NameError: name 'pwex' is not defined
Does pywren support jupyter notebook ?
I guess this can be the only issue

ohh sorry , i got this error
NoCredentialsError: Unable to locate credentials

This is strange, if pywren test_function runs but this does not. It might be related to boto3.
Can you try this?
import boto3
s3 = boto3.resource('s3')

still getting the same error
Pywren_error.docx

@Amandeep-singh11 Have you tried it from the terminal instead of the notebook?

@Amandeep-singh11 This seems to be your python environment failing to find the correct AWS credentials.
It is somehow contradictory to the fact that pywren test_function works. So probably the python environment in which the notebook is started is different from the one you ran pywren test_function?


incUsage is not called when acquireConnection -> createConnection

incUsage is not called when acquireConnection -> createConnection
but decUsage is still called when releasrConnection, which lead usage to become -1
after this, PingConnectionHandler will behave incorrectly:

send ping when in usage (poll: -1 to 0)
not send ping when idle (release: 0 to -1)

Expected behavior
incUsage is called when acquireConnection
Actual behavior
incUsage is called when acquire exist free connection
incUsage is not called when acquire create new connection
Steps to reproduce or test case
freeConnections size < acquireConnection size
for example (pseudo-code):

setConnectionMinimumIdleSize(1)
setConnectionPoolSize(8)
Redisson.create(config)
Executor pool = Executors.newFixedThreadPool(8)
for (int i = 0; i < 8; i++) pool.execute(() -> redisson.getBucket("test").getAsync())

Redis version
6.2.6
Redisson version
3.17.4
Redisson configuration
connectionMinimumIdleSize: 1
connectionPoolSize: 8
pingConnectionInterval: 5000
idleConnectionTimeout: 10000
【acquire exist free connection】【release correctly：0 to 1 to 0】

【acquire create new connection】【release incorrectly：0 to -1】

【PingConnectionHandler behavior】

Fixed. Thanks for report

@mrniko
You fixed the message connection but neglected the pubsub connection.
PubSubConnectionPool overrides poll and releaseConnection, which will not call incUsage and decUsage.
When acquire create new pubsub connection, incUsage in ConnectionPool without checking PubSubConnectionPool will lead usage to become 1.
Connection pinging will also work incorrectly in this case.
【PubSubConnectionPool】

【pollConnection】

【releaseConnection】

【pollSubscribeConnection】

【releaseConnection】

@LYFKiririn
Fixed. Thanks for code review.


webadmin client cert auth

The webadmin already asks for a client certificate when you connect using SSL, so I thought it would be cool if you could use the same certificate as with IRC cert auth as a SSL client certificate in the webadmin panel, or alternatively changing it so it doesn't ask for one if not possible.

Want to back this issue? Place a bounty on it! We accept bounties via Bountysource.

You can.

How?

Ah well crap, seems I failed when testing it. Sorry for the noise.

For posterity:
openssl pkcs12 -export -in user.pem -out user.p12

Then, in Chrome, go to Settings, Show Advanced Settings, HTTPS/SSL, Manager Certificates. Import the newly-created PKCS#12 file.
When you visit your ZNC web interface it should prompt you to choose the certificate for identification. Then, you can enter your username with a blank password and log in (as you would specify a blank password for IRC authentication).
If someone else wents to pipe up regarding FireFox instructions, please do.

What I usually do is:
openssl req -nodes -newkey rsa:4096 -keyout Mikaela.pem -x509 -days 3650 -out Mikaela.pem -subj "/CN=Mikaela"

and if I recall correctly both Chrome and Firefox accept the pem file directly.


feat(client-go): add certs endpoint

After this PR, client-go will be feature complete. \o/

Code LGTM, my minor comments notwithstanding.

Done!

code LGTM

Running this as https://ci.deis.io/job/test-ec2/2242/


Ensure that Font.charToGlyph won't fail because String.fromCodePoint is given an invalid code point (issue 11768)

Please note: This patch on its own is not sufficient to address the underlying problem in the referenced issue, hence why no test-case is included since the actual bug still needs to be fixed.
As can be seen in the specification, https://tc39.es/ecma262/#sec-string.fromcodepoint, String.fromCodePoint will throw a RangeError for invalid code points.
In the event that a CMap, in a composite font, contains invalid data and/or we fail to parse it correctly, it's thus possible that the glyph mapping that we build end up with entires that cause String.fromCodePoint to throw and thus Font.charToGlyph to break.
If that happens, as is the case in issue #11768, significant portions of a page/document may fail to render which seems very unfortunate.
While this patch doesn't fix the underlying problem, it's hopefully deemed useful not only for the referenced issue but also to prevent similar bugs in the future.

/botio-linux preview

/botio test

Thanks!


add option to enable encrypted communication

NATS / event bus (see also https://github.com/owncloud/ocis-charts/pull/105)
GRCP
http

Hint: Apply the settings in https://doc.owncloud.com/ocis/next/deployment/services/tls.html#tls-for-grpc


Language Server Protocol support

Hi There,
Are you aware of anyone working on adding LSP support for terraform? Based on the list of LSP implementations this one looks to be actively developed: https://github.com/juliosueiras/terraform-lsp
Could this possibly remove any need to update the cached data at https://github.com/rafalcieslak/emacs-company-terraform/blob/master/company-terraform-data.el ?

It looks like Hashicorp is looking to support a Terraform language server:
https://github.com/hashicorp/terraform-ls
Announcement:
https://www.hashicorp.com/blog/supporting-the-hashicorp-terraform-extension-for-visual-studio-code/

Thanks for letting me know. This looks very promising, and will likely justify even a major rewrite of this package.

Company already supports pulling completions from LSP via completion-at-point and either lsp-mode or eglot.  I think this package could be safely deprecated.


 It works perfectly fine without configuring youtube.
spring.log attached. Let me know if you need more information.
Output of java -version:
java version "<IP_ADDRESS>" 2022-08-18
Java(TM) SE Runtime Environment (build <IP_ADDRESS>+1-1)
Java HotSpot(TM) 64-Bit Server VM (build <IP_ADDRESS>+1-1, mixed mode, sharing)

Output of 'java -jar Lavalink.jar --version':
Version:        3.6.1
Build time:     28.10.2022 09:12:10 UTC
Branch          HEAD
Commit:         ee42ec1
Commit time:    28.10.2022 09:10:11 UTC
JVM:            <IP_ADDRESS>
Lavaplayer      1.3.99-original

spring.log

Started Launcher in 1.376 seconds (JVM running for 2.804)
...
Lavalink is ready to accept connections.

Looks perfectly fine to me, but you're not connecting to Lavalink. What were you expecting?

Well, looks like it was just a case of me not getting enough sleep. My apologies for the confusion and thanks for checking it out! By the way, thank you so much for all your hard work in creating Lavalink, it's really helpful.


Update user-data with kube-aws

Hi,
I want to modify the worker user-data configuration in a cluster that it's already running.
I've modified some parameters in the cloud-config-worker file, and then I run kube-aws render, which shows:
Error: mkdir credentials: file exists

I don't want to modify the existing credentials. Moreover, I can't modify the userdata set in the cloudformation-template due to it's encrypted.
How can I modify the user-data without create new credentials / stack?

See the first item in #340. As of now, you can do this today. (Assume starting out with a freshly-created cluster)
vim userdata/cloud-config-controller #and/or worker
kube-aws up --export

Then take that JSON file that's produced by --export and use it to update your cloudformation stack via AWS api, cli or web interface.

@arkkanoid if your question has been answered, could you close out this issue?

Hello @colhom ,
How can I decrypt UserData?

I'd be interested in knowing how to update the cloud-config on an existing cluster. I think that's what his original question was.
Basically, I'm trying to add a runtime-config parameter to the apiserver pod defined in the controller cloud-config (to enable ScheduledJob on my kube 1.4.x cluster.) I was hoping updating that would handle the pod recreation accordingly.
Any thoughts?