[HUDI-1513] Introduce WriteClient#preWrite() and relocate metadata table syncing

Syncing to metadata table, setting operation type, starting async cleaner done in preWrite()
Fixes an issues where delete() was not starting async cleaner correctly
Fixed tests and enabled metadata table for TestAsyncCompaction

Tips

Thank you very much for contributing to Apache Hudi.
Please review https://hudi.apache.org/contributing.html before opening a pull request.

What is the purpose of the pull request
(For example: This pull request adds quick-start document.)
Brief change log
(for example:)

Modify AnnotationLocation checkstyle rule in checkstyle.xml

Verify this pull request
(Please pick either of the following options)
This pull request is a trivial rework / code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)

Added integration tests for end-to-end.
Added HoodieClientWriteTest to verify the change.
Manually verified the change by running a job locally.

Committer checklist

[ ] Has a corresponding JIRA in PR title & commit

[ ] Commit message is descriptive of the change

[ ] CI is green

[ ] Necessary doc changes done or have another open PR

[ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

cc @prashantwason @rmpifer FYI

cc @prashantwason @rmpifer FYI

Looks fine.
I checked and you can verify too, there is no API in HoodieWriteClient which can be called before preWrite() and  end up returning a stale version of metadata.

Looks fine.
I checked and you can verify too, there is no API in HoodieWriteClient which can be called before preWrite() and  end up returning a stale version of metadata.

Yes. Correct.

Yes. Correct.

Codecov Report

Merging #2413 (f9992fd) into master (698694a) will decrease coverage by 40.19%.
The diff coverage is n/a.

@@              Coverage Diff              @@
##             master    #2413       +/-   ##
=============================================
- Coverage     50.23%   10.04%   -40.20%
+ Complexity     2985       48     -2937
=============================================
  Files           410       52      -358
  Lines         18398     1852    -16546
  Branches       1884      223     -1661
=============================================
- Hits           9242      186     -9056
+ Misses         8398     1653     -6745
+ Partials        758       13      -745

Flag
Coverage Δ
Complexity Δ

hudicli
?
?

hudiclient
100.00% <ø> (ø)
0.00 <ø> (ø)

hudicommon
?
?

hudiflink
?
?

hudihadoopmr
?
?

hudisparkdatasource
?
?

hudisync
?
?

huditimelineservice
?
?

hudiutilities
10.04% <ø> (-59.62%)
0.00 <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

The diff coverage is n/a.

@@              Coverage Diff              @@
##             master    #2413       +/-   ##
=============================================
- Coverage     50.23%   10.04%   -40.20%
+ Complexity     2985       48     -2937
=============================================
  Files           410       52      -358
  Lines         18398     1852    -16546
  Branches       1884      223     -1661
=============================================
- Hits           9242      186     -9056
+ Misses         8398     1653     -6745
+ Partials        758       13      -745

Flag
Coverage Δ
Complexity Δ

hudicli
?
?

hudiclient
100.00% <ø> (ø)
0.00 <ø> (ø)

hudicommon
?
?

hudiflink
?
?

hudihadoopmr
?
?

hudisparkdatasource
?
?

hudisync
?
?

huditimelineservice
?
?

hudiutilities
10.04% <ø> (-59.62%)
0.00 <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.


the variance of  loss is very large when initial training.

I want to reproduce your result using your code. I download datasets excluding CC12M.  And I find the three losses vary dramaticly. I don't know whether normal or not?

The variation you observe between batches should be normal.


Remove Wallet service

fix #1209

Codecov Report

Merging #1211 (89616b0) into master (828364b) will decrease coverage by 1.73%.
The diff coverage is 52.50%.

@@            Coverage Diff             @@
##           master    #1211      +/-   ##
==========================================
- Coverage   50.51%   48.77%   -1.74%
==========================================
  Files         145      152       +7
  Lines        7544     9369    +1825
==========================================
+ Hits         3811     4570     +759
- Misses       3066     4128    +1062
- Partials      667      671       +4

Impacted Files
Coverage Δ

pkg/config/consts.go
0.00% <ø> (ø)

pkg/config/genesis/config.go
0.00% <0.00%> (ø)

pkg/core/candidate/validator.go
64.00% <ø> (+6.85%)
:arrow_up:

pkg/core/chain/legacy.go
0.00% <0.00%> (-28.58%)
:arrow_down:

pkg/core/chain/mock.go
9.09% <ø> (+0.75%)
:arrow_up:

...kg/core/consensus/blockgenerator/candidate/mock.go
0.00% <0.00%> (ø)

pkg/core/consensus/comms.go
0.00% <0.00%> (ø)

pkg/core/consensus/fixtures.go
0.00% <0.00%> (ø)

pkg/core/consensus/header/header.go
24.50% <0.00%> (+2.41%)
:arrow_up:

pkg/core/consensus/publisher.go
0.00% <0.00%> (ø)

... and 182 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e806534...89616b0. Read the comment docs.


--imgpkg-lock-output should tell user it only works with pushed images

Describe the problem/challenge you have
If a user tries to use the --imgpkg-lock-output flag without relocating the images, you'll get this cryptic error message:
kbld: Error: Validating images lock:
  Expected ref to be in digest form, got 'kbld:simple-app-sha256-e5425420531e8f8dac8fa6acafc0f3f3a7f43603c5ff2be5b83c291ca9c1d9a0'

The problem is that the imgpkg lock file format needs a resolved image digest, which the local docker image storage cannot provide.
Describe the solution you'd like
A better error message is a start, maybe some documentation on the flag too.
$ kbld -f examples/simple-app-build-local/build.yml --imgpkg-lock-output foo.yaml

kbld: Error: --imgpkg-lock-output can only be used with images that have been pushed to a registry

Anything else you would like to add:
https://kubernetes.slack.com/archives/CH8KCCKA5/p1613491718286700

I can reproduce the above error.
Note: we should do this check before we build the images since the feedback loop would be too big.
Note2: This error comes from the imgpkg package https://github.com/vmware-tanzu/carvel-kbld/blob/develop/vendor/github.com/k14s/imgpkg/pkg/imgpkg/lockconfig/images_lock.go#L63 nevertheless we should not change the error message there. This behavior should be in kbld logic that checks if you are building local and not pushing the imgpkg lock file cannot be generated


Intermittent panoptic_mapper crashing (segfaulting)

Hello @Schmluk
First of all you have a great package over here.  I was able to compile  your package on my ubuntu 18.04 and was successfully able to run the flatdataset demo files as well.
However when I tried to run the same node on my custom dataset I am facing intermittent segmentation fault issues from the panoptic_mapper side. These segfaults are happening at variable timestamps within the same rosbag file over multiple runs. I have had 1 or 2 successful instances in avoiding the crash over the same rosbag play duration.  This is what the stack trace showed:
I0218 00:26:42.016423 24927 single_tsdf_integrator.cpp:120] Allocate: 306ms, Integrate: 1208ms.
I0218 00:26:42.081941 24927 map_manager.cpp:101] Pruned active blocks in 65ms.
Pruned 131 blocks from submap 0 (Unknown) in 65ms.
W0218 00:26:42.095261 24927 single_tsdf_visualizer.cpp:63] No Map to visualize.
I0218 00:26:42.108364 24927 panoptic_mapper.cpp:315] Processed input data.
(tracking: 0 + integration: 1515 + management: 78 + visual: 0 = 1594, frame: 1627ms)
terminate called after throwing an instance of 'std::out_of_range'
  what():  _Map_base::at
*** Aborted at<PHONE_NUMBER> (unix time) try "date -d @1645136547" if you are using GNU date ***
PC: @     0x7f88cc039fb7 gsignal
*** SIGABRT (@0x4e7b) received by PID 20091 (TID 0x7f88697fa700) from PID 20091; stack trace: ***
    @     0x7f88cc03a040 (unknown)
    @     0x7f88cc039fb7 gsignal
    @     0x7f88cc03b921 abort
    @     0x7f88cc690957 (unknown)
    @     0x7f88cc696ae6 (unknown)
    @     0x7f88cc696b21 std::terminate()
    @     0x7f88cc696d54 __cxa_throw
    @     0x7f88cc692837 (unknown)
    @     0x7f88c9404ebe panoptic_mapping::SubmapCollection::getSubmapPtr()
    @     0x7f88c94823b7 panoptic_mapping::SingleTsdfIntegrator::processInput()
    @     0x7f88cc9d4aea panoptic_mapping::PanopticMapper::processInput()
    @     0x7f88cc9d57f4 panoptic_mapping::PanopticMapper::inputCallback()
    @     0x7f88cd7f9dd7 ros::TimerManager<>::TimerQueueCallback::call()
    @     0x7f88cd81e829 ros::CallbackQueue::callOneCB()
    @     0x7f88cd81fb15 ros::CallbackQueue::callOne()
    @     0x7f88cd876f44 ros::AsyncSpinnerImpl::threadFunc()
    @     0x7f88ca8c3bcd (unknown)
    @     0x7f88cb4aa6db start_thread
    @     0x7f88cc11c71f clone

This made me think that I probably have some wrong parameters set due to which the mapper is trying to access a submap which might have gotten already pruned. However I did come across 1-2 instances in which the same node did not crash here is the console ouput from that run. (Note: No parameters were changed for either of runs).
I0218 00:23:16.871966 21811 single_tsdf_integrator.cpp:120] Allocate: 6ms, Integrate: 65ms.
I0218 00:23:16.973881 21811 planning_visualizer.cpp:186] Map lookups based on 1 submaps took 0.3+/-0.2, max 14.3us.
I0218 00:23:17.072490 21811 panoptic_mapper.cpp:315] Processed input data.
(tracking: 0 + integration: 71 + management: 0 + visual: 102 = 174, frame: 456ms)
I0218 00:23:17.122951 21806 single_tsdf_integrator.cpp:120] Allocate: 2ms, Integrate: 44ms.
I0218 00:23:17.229359 21806 planning_visualizer.cpp:186] Map lookups based on 1 submaps took 0.3+/-0.2, max 17.8us.
I0218 00:23:17.317369 21806 panoptic_mapper.cpp:315] Processed input data.
(tracking: 0 + integration: 47 + management: 0 + visual: 107 = 154, frame: 244ms)
I0218 00:23:17.365311 21800 single_tsdf_integrator.cpp:120] Allocate: 2ms, Integrate: 42ms.
I0218 00:23:17.468127 21800 planning_visualizer.cpp:186] Map lookups based on 1 submaps took 0.3+/-0.2, max 24.1us.
I0218 00:23:17.552709 21800 panoptic_mapper.cpp:315] Processed input data.
(tracking: 0 + integration: 44 + management: 0 + visual: 103 = 148, frame: 235ms)
I0218 00:23:17.605499 21806 single_tsdf_integrator.cpp:120] Allocate: 2ms, Integrate: 40ms.
I0218 00:23:17.704886 21806 planning_visualizer.cpp:186] Map lookups based on 1 submaps took 0.3+/-0.6, max 77.6us.
I0218 00:23:17.791018 21806 panoptic_mapper.cpp:315] Processed input data.
(tracking: 0 + integration: 43 + management: 0 + visual: 100 = 143, frame: 238ms)

Here is the output from the Rviz TSDF pointcloud:

However even in this run I could not see any visual output from the voxblox mesh on RViz.
My voxel size is set to 0.05 which I don't think is the issue since I was able to run my same data on the Kimera-Semantic mapping Library (which also uses voxblox on its backend) successfully and got a decent output as well
EG(Output from Kimera mapping package):

Could you please provide me with any hint as to what parameter I should look into (or how should I try to debug this) to get a proper output with the visual generated mesh ? or do you think this issue is caused by something else entirely ?  I am attaching my custom config file along with its label csv file as well.  (Note: I build my code in Release Mode).
Any help is appreciat
custom.csv
ed ... thanks!!
custom (1).txt

Hi @abhileshborode
Thank you for your interest! Yes, there seem to be some issues with the config. This is a bit a blessing and a curse of modular frameworks, but not all modules are compatible with each other.

If you wish to use the single-TSDF version, then you have to use the single_tsdf tracker, integrator, and visualizer. I'd also disable map management in this case. Easiest would be to start from the single_tsdf config and change only the params (such as the labels, not the modules).
If you want to use the mutli-TSDF case, then I'd start from the flat_groundtruth config and play with the params.

For visualization I'd recommend the mesh rather than the TSDF pointcloud. Also worth knowing, panoptic mapping expects the input poses in optical frame, i.e. with z pointing forward, x right, y down.
I hope this helps, let us know how it goes!

Hi @Schmluk ,
Thanks for the advise.
I tried your recommended steps but still am facing the intermittent node crashes.  Initially I was getting a lot of
Unable to lookup transform between 'world' and 'camera4_infra1_optical_frame' at time '1637684095.711249113' over '0.1s', skipping inputs. Exception: 'Lookup would require extrapolation into the past.  Requested time<PHONE_NUMBER>.711249113 but the earliest data is at time<PHONE_NUMBER>.967064142, when looking up transform from frame [camera4_infra1_optical_frame] to frame [world]'.  (My bag is running with the --clock arg along with use_sim_time true)
Even though my /tf was publishing at 40hz and rosrun tf tf_echo frame1 frame2 was outputting successfully. I just bypassed the issue by setting the max_input_queue_length: 10.
After that I tried the base configs for single_tsdf and multi-maps but still experienced the crashes sometimes.
Even in the instances I had non-crash runs I did not see any understandable/meaningful  output from Rviz atleast. The mesh was being published (I checked it via rostopic hz) but almost negligible output showed up on Rviz. (I could only see the free TSDF PCL which by itself I could not tell if it was correct or not).
When you say input poses are expected in optical frame do you mean that the input poses has to be published in the optical frame ?  As of now I have my pose in the world frame and I have a /tf  which is publishing the transform from world frame to optical frame. (I assumed this package is following the same convention as voxblox)
I think I am trying to use this package incorrectly at this point considering the way I am using it which might be causing these crashes. I am inputting an  infrared image("mono8" ) instead of "bgr8" / depth image("16UC1") instead of "32FC1" and a segmented image ("bgr8") instead of  "32SC1", I made changes to the input_synchronizer.cpp for the change in datatype for their subscribers accordingly.  I did not go through how all the data structures are operated on the backend ... do you think these could lead to those crashes ?
Also I noticed the examples in the package expects the use_detecton flag to be true. Is it possible to run the package without using detectron or equivalent. My custom dataset is 80% static classes and the 20% dynamic classes usually only show up as a single instance so I don't really need to run detectron2 verison (which provides instance segmentation layer).  My csv file [attached]  just labels the panoptic labels accordingly. My semantic segmentation image is an output from the deeplab model.  Is it possible to integrate this package with the pure semantic segmented images alone ?  If yes could you please advise on how I might go on with it ? I am thinking because I have no instances as an input the id_tracker part of the stack might be causing problems. ??
Looking forward to yo
custom.csv
ur reply

Hi @Schmluk ,
Just wanted to follow up on this. It seems the accuracy issue and some of the segfaults were coming from the fact that my input depth image was of 16UC1 instead the expected 32FC1. I was able to generate somewhat understandable ouput at this point. here are some screenshots: ( I have detectron false for both instances).
Generated Via single_tsdf config
![mesh-single-tsdf](https://user-images.githubusercontent.com/33388291/155063580-51cde6b0-35ce-470e-8c4d-6d532086a3

34.png)
I am trying to investigate why are there so many holes in the mesh generated. Could it be voxel size related ? I tried increasing the voxel size for some instances but I am still unable to generate a continuous mesh, also there seems to be only 1 submap mesh generate (as you can see in the image). Is there a parameter for specifically enabling multiple submaps in the single-tsdf config Since I recall the package generating multiple submaps for the flatdataset example using the same config. (I thought I had my mapping for the csv file incorrect but my mulit-tsdf file was generating multiple submaps ) Is this result expected while using single_tsdf config ?
Multi-TSDF config result:

While keeping cadence to the segfaults I am still intermittently  getting this segfault:
terminate called after throwing an instance of 'std::out_of_range'   what():  _Map_base::at
It seems to be arising from  return submaps_[id_to_index_.at(id)].get();
And surprisingly if it happens it only happening at the start of the rosnode. Could this be relevant to any specific parameter which is causing node to access an unallocated block ?
Any help is appreciated !!

Hi @abhileshborode
Apologies for not coming back to you earlier. Regarding your previous questions, yes it is important that all your input is converted to the formats expected py Panoptic Mapping, otherwise all image lookups (which are templated on opencv Mats) will fail. Yes, setting detectron to false will just use the IDs in the input image.

For the single_tsdf case, the idea is to integrate all the data into a single TSDF = submap (thus the name). Instead of having one object, each voxel in the TSDF is labeled (similar to Voxblox++, Kimera, ...). You can show the lables by setting the visualization mode to classes or instances.
The segfault you are getting results from submaps not being allocated or your input IDs not being correct. Usually this suggests that the input tracker is not set up correctly, or rarely that the map is being modified (e.g. loading a different map).
Regarding the reconstruction, it can be tuned like any TSDF-based method, I'd recommend looking at the voxel size and truncation distance first.

I hope this helps, otherwise could you provide the complete settings you are using and the full failure log?

Hi @Schmluk ,
your suggestion worked. After some little tuning (setting truncation_distance:  0.4)I am able to generate continuous meshs:

I had a question regarding the Planning output generated ... Is it possible to mask out certain classes from showing up on the planning slice layer entirely ? (i.e they should not even be considered to begin with). I am asking this because I have many pixel points which are segmented (from the input segmentation image) as background pixels (class name is background)( eg: sky) these pixels usually have very erroneous depth measurements so they show up on the maps often and they get detected as occupied space on the planning slice: eg: The black dots in the image shown below:

Is it possible to semantically mask out  certain classes from the input segmentation image so that they don't get appended / disturb the planning layer or mapping layer (without changing the planning slice resolution)? Can we rewire the config in some way to get this behavior ?
I also wanted to ask if its possible to modify this package such that it could take in a pointcloud as an input ? (Similar to voxlblox / Kimera) this could enable passing pointclouds from multiple sources ( eg: 2 cameras for wider fov) . Assuming I am providing a /tf from that pointcloud to the world frame at all times ?. I know this would need some development but if possible could you point me to which file will I have to modify ?
Regarding the segfaults I think I have located the cause it came from another erroneous image conversion of the input segmentation image. I am still getting some memory leaks (on htop) while running this package for longer duration though ( on 8-10 min bags). the memory usage grows to 15G along with 300% CPU usage and more. I ran the same stack on the example flat dataset and got like 1G memory usage along with 35-40% CPU usage. So it looks like my input data is causing high CPU / memory usage (which grows over time).  Only noticeable difference in my dataset is that it is inputted at 6Hz and my images are 848x640 which by itself I believe should not cause such a drastic computation performance over time. What are your thoughts on this ?
The results from your package has been very great btw when I compared it with Kimera / voxblox. Looking forward to your reply

Hi @Schmluk ,
Just wanted to follow up on this.... I increased the slice height for the planning layer and it mostly filters out background class labels due to erroneous depth measurements occurring on the planning slice. But I happen to see a bit of inconsistent behavior with this.
Example 1: (In this case the planning layer is properly ignoring the background class voxels (Black / Blue voxels  )that show up on the map due to erroneous depth.

Example 2: (In this case it does not ignore background voxel (black / blue voxels )that show up and the planning slice shows that space as occupied
Do you have any idea why this must be happening ?

Hi @abhileshborode
Great to hear that it worked for you! Regarding using classes for planning, the planning lookup happens here. There you can add conditions to e.g. skip submaps of specific labels. The planning layer just queries the submap collection and nothing should be ignored, except its changestate is absent or unknown.
Regarding the multiple cameras, it is in principle possible to use any pointcloud, but would need to implement a projective integrator for general pointclouds. This is not yet implemented, but could be similar to the way voxblox does it. What is easy to add is support for multiple cameras. You could e.g. in the input or transform frame se which camera you are using and use the standard pipeline with multiple camera objects.
I hope there should be no memory leaks but if you happen to find one I'd be happy to hear about it!

Hi @Schmluk hope you are doing well. Just wanted to follow up on this thread.
I happened to wrap multiple camera objects inside the global class instance and did the corresponding camera object lookups based on the input set frame  and it worked like a charm. The tracker does a really got job of tracking objects within the overlapping field of view of multiple cameras as well.
As far as memory leaks go at this point I don't think there are any.  I was using images with 848x480 resolution ... which was just too much data for it to handle which lead to extremely high CPU and memory growth. Applying a decimation factor of 3 prevents this issue.
I did however happen to notice an increase in CPU and throttling in frequency of the planning layer when I happen to enable the generatemesh layer option. And there seems to be increase in the object reconstruction  performance alongside the tracking of multiple objects/instances as well .
Here is a snippet with the generate mesh option enabled (link)
Here is a snippet without the generate mesh option enabled (link)
If you notice the output with the mesh enable option is much more stable and the tracker is able to do the data association within its object instance submaps much more accurately. (the output visual is the surface pcl generated from the submaps)
It seems to be coming from submap.upatemesh() line of that function while it is iterating through all of its submap instances . I am trying to understand on how is this different from the regular tsdf integrator update thats within the panoptic_mapper processinput callback  ? and why is output so much more stable after the submap.updatemesh() method ?
I  went through your published  computational performance metrics. You mentioned that you guys got significantly faster results (almost 30Hz.... Wow!!) on the RIO 224x172 resolution with fewer object detections.   What did you mean by fewer object ? Were you guys just performing tracking/integration updates for the object instances of the submaps?

Hi again @Schmluk  ... If i understand correctly the multi-mesh viz update is trying to query all the submaps within its collection and trying to  perform full volumetric reconstruction at its full resolution which seems to be really expensive and it affects the output FPS as well. This update however seems to be key in  accurately tracking and constructing scenes with semantic consistency and has a much more gradual temporal decay  as well . I was wondering if it was possible to limit this cool behavior to specific semantic  classID submaps only ? so that specific class objects could be fully reconstructed with semantic-temporal consistency instead of doing it on the entire scene.
I was looking into leveraging the Hierarchical structure within the submap to do this for high speed object tracking and reconstruction . I tried looking up the submap by their ClassIds here and only performed an submap.meshupdate() step if it was the submap of semantic interest however it still seemed to update all of its submaps instead. Could you please point me to where I can make this change to reconstruct specific objects at its set resolution ? Should this step be added after filtering out your submaps based on the associations made based on the IoU for faster performance?  or did I misunderstand this and something else is responsible for doing this object level full reconstruction ?

Hi @abhileshborode
Great to hear that this worked for you. Updating the meshes has nothing to do with the TSDF update, i.e. first the TSDF is updated during integration, then the mesh (and therewith the ISO-surface points) are extracted using marching cubes. The iso-surface points are rendered during tracking, i.e. all submaps you want to track need to be meshed. The mesh updates are incremental per block, so should not be expensive if queried too often. Regarding the visualization in RVIZ, this takes the mesh and converts it into an OGRE compatible rviz message, which can be expensive I guess. Filtering by IDs for visualization should definitively be possible.
Since for every detected object a submap is allocated, fewer object detections lead to fewer submas being updated in parallel.

Hi @Schmluk  .... thanks that makes more sense to me now. I had a quick question regarding which module  updates the mesh. I was trying to debug the renderingApproximate projective_id tracking as I was getting 0 tracking matches while using this compared to the rendertrackingVertices which had fair number of tracking matches but  rendertrackingVertices  seemed to do ray-tracing into each submaps which is a bit inefficient.  From what I could see the mesh update step seems to happen here  .  Upon doing a backtrace while using rendertrackingVertices  the mesh update step seems to come from  manageSubmapActivity within the map_manager .  But I could not locate where did that call come from ? And from what I searched the updatemesh function call is only called from within submap_visualizer. Could you point me to where exactly the mesh update step happens ? I believe the mesh update step is not happening in my case when I am using renderingApproximate projective_id  which is  causing 0 tracking matches while using this compared to using rendertrackingVertices.  Since the submaps are not tracked they are also pruned by the maps_manager. Do you think this reasoning makes sense ?  Or do you think something else is causing the submap tracking match failures ?

Yes, if the meshes are not update the tracking does not work. It might actually be that this only happens during visualization, but it should happen also in the main loop. Since this is incremental and cashes the progress you can just add it to the main loop. The renderTrackingInfoVertices was just a test to directly lookup the values but it's less efficient and less accurate.


[BUG] [v5.12] AttributeError: module 'discord_typings' has no attribute 'GuildOnboardingData'

Library Version
5.12
Describe the Bug
After updating to v5.12, on startup my project gives the following:
Traceback (most recent call last):                                                                       File "/bots/proj/main.py", line 1, in <module>                                                        from interactions import AutoShardedClient, Intents                                                  File "/usr/local/lib/python3.11/site-packages/interactions/__init__.py", line 2, in <module>             from .client import (                                                                                File "/usr/local/lib/python3.11/site-packages/interactions/client/__init__.py", line 36, in <module>     from .client import Client                                                                           File "/usr/local/lib/python3.11/site-packages/interactions/client/client.py", line 39, in <module>     File "/usr/local/lib/python3.11/site-packages/interactions/api/http/http_client.py", line 19, in <module>     from interactions.api.http.http_requests import (   File "/usr/local/lib/python3.11/site-packages/interactions/api/http/http_requests/__init__.py", line 5, in <module>     from .guild import GuildRequests   File "/usr/local/lib/python3.11/site-packages/interactions/api/http/http_requests/guild.py", line 18, in <module>     class GuildRequests(CanRequest):   File "/usr/local/lib/python3.11/site-packages/interactions/api/http/http_requests/guild.py", line 1045, in GuildRequests     async def get_guild_onboarding(self, guild_id: "Snowflake_Type") -> discord_typings.GuildOnboardingData:                                                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: module 'discord_typings' has no attribute 'GuildOnboardingData'
The script does not exit from this traceback, but I'm not sure if some interactions.py functionalities are impacted or not. Nothing was changed in my code between the transition between v5.11.0 and v5.12.0
Steps to Reproduce
Run a project using python 3,11 and interactions.py 5.12.0
Expected Results
When using interactions.py 5.11.0, the mentioned traceback didn't appear, so the expected results would be for it to not appear in v5.12.0
Minimal Reproducible Code
No response
Traceback
No response
Checklist

[X] I have searched the open issues for duplicates.
[X] I have shown the entire traceback, if possible.
[X] I have removed my token from display, if visible.
[X] I have attempted to debug this myself, and I believe this issue is with the library

Additional Information
No response

realized pip installed 5.12 even tho that's not the most recent version, im guessing this issue was fixed in the more recent versions...


Feat Req.:  would be nice if keyrunner env vars injected in request body would be highlighted in a different colour than regular strings

Is your feature request related to a problem? Please describe.
It would be nice to see injected keyrunner environment variables into request body be highlighted in a different colour then the current colour used for regular strings, to be able to better locate them.
Describe the solution you'd like
There's already a nice mouse-over effect in place, which will show its actual value which is perfect, so imo we're just missing the colour highlight to make this feature complete.
Describe alternatives you've considered
n.a.
Additional context

We will definitely look into adding this feature once we address the current bugs.


Create a "processed SLEAP data" table in database

Cleans up raw SLEAP assignments, e.g. by fixing impossible assignments.

Proposed table design
@schema
class SubjectPosition(dj.Computed):
    definition = """  #
    -> acquisition.Chunk
    """

    class Subject(dj.Part):
        definition = """
        -> master
        subject_name:          varchar(16)
        ---
        sample_count: int      #
        x:          longblob   #
        y:          longblob
        likelihood: longblob
        timestamps: longblob
        """

    class Part(dj.Part):
        definition = """
        -> master.Subject
        part_name: varchar(16)
        ---
        x:          longblob
        y:          longblob
        likelihood: longblob
        """

    def make(self, key):
        # SLEAP.PoseIdentity - subjects detected by SLEAP (for multiple cameras)
        # SubjectVisits (for "environment") - ground truth for subjects in the arena
        # RFID

        pass

An update to this issue.
We have discussed to apply the fix and store the results not only in the DataJoint pipeline, but instead to write the corrected SLEAP data into harp file with the same name as the original file, but placed in the "processed" folder instead.
This correction will be done on a per chunk basis - triggered using DataJoint as the workflow management tool, but again writing the results as files instead of storing in database.
Subsequently, DataJoint will load/ingest this corrected SLEAP data (in tracking.SLEAPTracking table).

First prototype in this branch: https://github.com/ttngu207/aeon_mecha
This prototype utilizes SLEAP identity correction method implemented by Chang Huan see here.
Currently, this method only works for 2 animals, we discussed potentially replacing this with the correction approach (using Kalman filter) developed first by Joaquin
See this table

This is almost done, lastly I just need to add a way to write out the pandas dataframe to the new binary output format of Bonsai.Sleap0.3

This is done.


Navigator.push not supported

The class Navigator is marked as supported, however Navigator.push does not compile:

Cannot find static method Navigator.push

Same with Navigator.of(context).push:

Unknown method NavigatorState.push

pushNamed is supported, can you use that? Or do you need push specifically for your use case?

In fact I'd like to use flutter_eval or dart_eval for building anti-piracy protection.
My idea is to put some application code (for example navigation) in bytecode, along with some protection code like flutter_security_checker, to defeat cracking software.
I don't know if it can work?


Make PokemonGo-Bot Significantly Better

Short Description:
It makes it better
Fixes:

deletes everything and tells players to go outside.
ensures people get fresh air.
doesn't cheat.

What if they live in China? Not much fresh air there :P

Been waiting for this feature! Please merge!

Closed because people have the freedom to do whatever they want.

But no, if you don't do it his way, you are obviously doing it wrong.  His way is better, he made that point clear.

Would you consider reopening this PR? There's a lot of good changes here you might be able to cherry pick.


How to I make a hybrid app using google materialize?

I have designed my pages using google materialize. They are html pages.How do I make an apk file as well as an ios app from the same. Please provide steps.I have installed npm from the documentation given in materialcss.com . But bower is not being installed.

Take a look at Cordova or phonegap. There are plenty of guides available on the internet

thank you.  That worked

Thank you. That worked


[Bug]: ArgoCD Admin password set via AWS Secrets no longer works from v3.5.*

Welcome to Amazon SSP EKS Accelerator!

[X] Yes, I've searched similar issues on GitHub and didn't find any.

Amazon EKS Accelerator Release version
3.5.0
What is your environment, configuration and the example used?
terraform {
  required_version = ">= 1.0.1"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 4.6.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = ">= 2.9.0"
    }
    helm = {
      source  = "hashicorp/helm"
      version = ">= 2.4.1"
    }
    kubectl = {
      source  = "gavinbunney/kubectl"
      version = ">= 1.13.1"
    }
  }

  backend "s3" {
    bucket = "xyx"
    key    = "eks.tfstate"
    region = "eu-west-1"
  }
}

What did you do and What did you see instead?
aws-eks-accelerator-for-terraform modules setup
VPC created using separate  terraform VPC creation as per example in eks-cluster-with-import-vpc
module "aws-eks-accelerator-for-terraform" {
  source = "github.com/aws-samples/aws-eks-accelerator-for-terraform.git?ref=v3.5.2"

  tenant            = local.tenant
  environment       = local.environment
  zone              = local.zone
  terraform_version = local.terraform_version

  # EKS Cluster VPC and Subnets
  vpc_id             = local.vpc_id
  private_subnet_ids = local.private_subnet_ids

  # EKS CONTROL PLANE VARIABLES
  create_eks         = true
  kubernetes_version = local.kubernetes_version

  # EKS MANAGED NODE GROUPS
  managed_node_groups = {
    mg_4 = {
      node_group_name = "managed-ondemand"
      instance_types  = ["t3a.large"]
      subnet_ids      = local.private_subnet_ids

      desired_size = "3"
      max_size     = "5"
      min_size     = "2"
    }
  }
}

module "eks-ssp-kubernetes-addons" {
  source = "github.com/aws-samples/aws-eks-accelerator-for-terraform.git?ref=v3.5.2/modules/kubernetes-addons"

  eks_cluster_id               = module.aws-eks-accelerator-for-terraform.eks_cluster_id
  eks_worker_security_group_id = module.aws-eks-accelerator-for-terraform.worker_security_group_id

  # EKS Managed Add-ons
  enable_amazon_eks_vpc_cni            = true
  enable_amazon_eks_coredns            = true
  enable_amazon_eks_kube_proxy         = true
  enable_amazon_eks_aws_ebs_csi_driver = false

  # K8s Add-ons
  enable_metrics_server               = true
  enable_cluster_autoscaler           = true
  enable_vpa                          = false
  enable_prometheus                   = false
  enable_ingress_nginx                = false
  enable_kubernetes_dashboard         = false
  enable_aws_load_balancer_controller = true

  enable_aws_for_fluentbit            = true
  aws_for_fluentbit_helm_config = {
    values = [templatefile("${path.module}/fluentbit-values.yaml", {
      aws_region  = data.aws_region.current.id
      es_url      = var.es_url
    })]
  }
  aws_for_fluentbit_irsa_policies = [aws_iam_policy.sortd-opensearch.arn]

  enable_argocd                       = true
  enable_argo_rollouts                = true
  argocd_manage_add_ons               = false
  argocd_admin_password_secret_name   = "ArgoCDAdmin"
  argocd_helm_config = {
    name             = "argo-cd"
    chart            = "argo-cd"
    repository       = "https://argoproj.github.io/argo-helm"
    version          = "3.33.8"
    namespace        = "argocd"
    timeout          = "1200"
    create_namespace = true
    values           = [templatefile("${path.module}/argocd-values.yaml", {})]
  }

  argocd_applications = {
    web = {
      project             = "some-project"
      path                = "some-path"
      repo_url            = "some_repo_url"
      target_revision     = "master"
      namespace           = "some-namespace"
      ssh_key_secret_name = "some_ssh_key"
      values              = {}
    }

  }

  depends_on = [
    module.aws-eks-accelerator-for-terraform.managed_node_groups
  ]
}

argocd-values.yaml
redis-ha:
  enabled: true

controller:
  enableStatefulSet: true

server:
  # -- Additional command line arguments to pass to Argo CD server
  extraArgs:
   - --insecure
  autoscaling:
    enabled: true
    minReplicas: 2
  additionalProjects:
  - name: some-project
    finalizers:
    - resources-finalizer.argocd.argoproj.io
    description: Some Project
    sourceRepos:
    - '*'
    destinations:
    - namespace: '*'
      name: '*'
      server: '*'
    additionalLabels: {}
    additionalAnnotations: {}
    clusterResourceWhitelist: []

repoServer:
  autoscaling:
    enabled: true
    minReplicas: 2

ArgoCD password created in another terraform project before running the EKS cluster creation
resource "aws_secretsmanager_secret" "argocd" {
  name = "ArgoCDAdmin"
  description = "ArgoCD Admin Password"
  tags = {
    Env = title(local.zone)
    System = "SomeSystem"
    Type =	"Password"
    managed-by = "terraform"
  }
}

resource "aws_secretsmanager_secret_version" "argocd" {
  secret_id     = aws_secretsmanager_secret.argocd.id
  secret_string = var.argocd_password
}

Additional Information
In version 3.4.1 the argocd password is set correctly based on value stored in the AWS secret.
From version 3.5.0 onwards when running terraform apply, terraform complains saying

Error: Invalid dynamic for_each value
│
│   on .terraform/modules/eks-ssp-kubernetes-addons/modules/kubernetes-addons/helm-addon/main.tf line 51, in resource "helm_release" "addon":
│   51:     for_each = try(var.helm_config["set_sensitive"], null) != null ? distinct(concat(var.set_sensitive_values, var.helm_config["set_sensitive"])) : var.set_sensitive_values
│     ├────────────────
│     │ var.helm_config["set_sensitive"] is list of object with 1 element
│     │ var.set_sensitive_values is empty tuple
│
│ Cannot use a list of object value in for_each. An iterable collection is required.

"terraform plan" and "terraform validate" do not pick up the error.
I have tried this with terraform v1.1.3 and aws 3.66.0 as well but I get the same error.
I am new to terraform so I am unsure how to fix this correctly but I can HACK it to work for my use case by doing the following on v3.5.2 source code:
In eks-ssp-kubernetes-addons/modules/kubernetes-addons/helm-addon/main.tf line 51 in dynamic dynamic "set_sensitive" change to:
for_each   = {for secret in var.set_sensitive_values: secret.name => secret }
In eks-ssp-kubernetes-addons/modules/kubernetes-addons/argocd/main.tf line 6 add:
set_sensitive_values = local.default_helm_set_sensitive
In eks-ssp-kubernetes-addons/modules/kubernetes-addons/argocd/locals.tf on line 24 comment out
#set_sensitive    = local.default_helm_set_sensitive
I realise this is not equivalent and is a HACK but I cannot find any other examples in the code base of how this is supposed to work, but I suspect this may be a problem with how sensitive values are being set using the new helm_addon instead of the helm chart directly and may not therefore be unique to argocd.

@IrishAfrican thanks for reporting the issue. We will get this resolved soon.

@IrishAfrican this may be due to https://github.com/hashicorp/terraform/issues/29744 which points to a limitation. Let me discuss this internally and figure out the best approach to resolve this.

Thanks @askulkarni2 much appreciated.

@IrishAfrican I have a fix implemented in https://github.com/aws-samples/aws-eks-accelerator-for-terraform/tree/bug/fix-argocd-admin-password. If you don't mind, can you give it a go and see if that fixes your issue?

@askulkarni2 Thanks, your change does seem to fix the issue


D435i IMU Scale & Bias Calibration

D435i IMU scale & bias calibration whitepaper is now available:
RealSense_Depth_D435i_IMU_Calib.pdf

In addition to using the box as a way to keep the camera level, I got my best result using my 3D printer bed after re-leveling it relative to the world.  Turns out my whole house is off level  :laughing:


fix typo on line 97 of coursewwares.json

changed line 97 in freecodecamp/seed_data/courseware.json from "You an also comment on a link ..." to "You can also comment on a link ..."

Thanks for the typo catch. Please submit this PR against the staging branch.

@sageio Please submit against staging, as per @terakilobyte suggestion.

Done. I'm sorry about that!


Merge Check, CheckTier3, Docs and DocsTier3 CI jobs

And expand the amount of targets we check.
Instead of getting the target setup twice, one for the cargo check run and another for cargo doc, do it once and check both.

This basically copies Mio's check job to socket2.


fix incorrect return of last goal join

Thank you for submitting this fix! I think this logic is correct, but could you also submit a test case that passes with this fix but that would fail without it?

Actually I can do this myself, I'll add a quick test case...

Codecov Report

Merging #8 (f7fff8a) into main (5186647) will not change coverage.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##             main       #8   +/-   ##
=======================================
  Coverage   97.79%   97.79%
=======================================
  Files          16       16
  Lines         317      317
=======================================
  Hits          310      310
  Misses          7        7

Impacted Files
Coverage Δ

fuzzy_reasoner/__init__.py
100.00% <100.00%> (ø)

fuzzy_reasoner/prover/operations/recurse.py
98.07% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 214d676...f7fff8a. Read the comment docs.


Design a generic interpolation plugin

what
Create a generic interpolation plugin destined to live in if/plugins.
why
We prefer to maintain generic plugins to do arithmetic operations rather than plugins that are hardcoded with parameter values for specific use cases. Specific use cases are then served by combining arithmetic plugins, passing in appropriate values. This makes things much more flexible, configurable and maintainable. This specific plugin is intended to support the deprecation and removal of our teads-curve plugin. Instead of having a plugin just for that specific curve, we can provide a generic interpolation plugin and pass in the known x and y values representing the curve.
Note that to fully replicate the teads-curve plugin the interpolate plugin will have to be part of a pipeline with multiply and divide plugins too.
Acceptance criteria

[ ] The plugin should support linear, spline and polynomial interpolation methods.

Linear interpolation is a method of estimating values that fall between known values. It involves determining the value at a point on a straight line between two known points of curve fitting using linear polynomials to construct new data points within the range of a discrete set of known data points.

Polynomial - Polynomial interpolation involves fitting a polynomial function to a set of data points. We intend to use Lagrange's interpolation. Higher-order polynomials can capture more complex relationships but have over-fitting risks.

Spline - Spline interpolation uses piecewise-defined polynomial functions (usually low-degree) to interpolate between data points. A spline consists of multiple polynomial segments, often cubic, that are smoothly connected at specific points. Splines provide a balance between flexibility and smoothness, making them suitable for approximating functions with complex variations.

[ ] The plugin should accept the following global config:

method - Specifies the interpolation method for the data. Acceptable values are 'linear', 'spline', or 'polynomial'. The default method is linear. (optional)

x - Array of points. Numbers should be in ascending order (required). (In the old version the points are hardcoded: [0, 10, 50, 100]. Could it be optional to support the same hard-coded values?) (can it be in node-config ?)

y - Array of numbers. Numbers should be in ascending order (required)

input-parameter - a string defining the name to use its value to calculate the interpolation. It should match an existing key in the inputs array.

output-parameter - a string defining the name to use to add the result of interpolation with additional calculation

[ ] the plugin should validate that the lengths of x and y are equal

[ ] The plugin should accept the following data from inputs:

timestamp -  a timestamp for the input  (required)

duration - the amount of time, in seconds, that the input covers.  (required)

input-parameter - a field whose name matches the string provided to input-parameter in global config (i.e. if the input-parameter in global config is cpu/utilisation then cpu-utilisation must exist in the input data)

[ ] The plugin outputs a value z that is the result of looking up the y value at x = input-parameter using the user-defined interpolation method. z is assigned to a field with a name defined by output-parameter in the output data.

[ ] Unit tests have been added to the builtin repository

the unit tests accompanying the feature code should have 100% coverage.

[ ] Documentation has been added in the form of a feature README and linked out from if.greensoftware.foundation

Example Manifest
name: interpolation-demo
description: simple demo of interpolation plugin
tags:
initialize:
 plugins:
   interpolation:
     method: Interpolation
     path: "builtin"
     global-config:
       method: linear
       x: [0, 10, 50, 100]
       y: [0.12, 0.32, 0.75, 1.02]
       input-parameter: “cpu/utilization”
       output-parameter: “interpolation-result”
tree:
 children:
   child:
     pipeline:
       - interpolation
     inputs:
       - timestamp: 2023-07-06T00:00
         duration: 3600
         cpu/utilization: 10

Executing this manifest should yield the following result:
name: interpolation-demo
description: simple demo of interpolation plugin
tags:
initialize:
 plugins:
   interpolation:
     method: Interpolation
     path: "builtin"
     global-config:
       method: linear
       x: [0, 10, 50, 100]
       y: [0.12, 0.32, 0.75, 1.02]
       input-parameter: “cpu/utilization”
       output-parameter: “interpolation-result”
tree:
 children:
   child:
     pipeline:
       - interpolation
     inputs:
       - timestamp: 2023-07-06T00:00
         duration: 3600
         cpu/utilization: 10
    outputs:
       - timestamp: 2023-07-06T00:00
         duration: 3600
         cpu/utilization: 10
         interpolation-result: 0.12

Note on teads curve
The interpolation described in the example above gives the y value from the Tead's curve for any given x value. We pass in cpu/utilisation as an x value and receive the appropriate y. The y value is a scaling factor that should be multiplied by the cpu/thermal-design-power to yield cpu/energy. You may also want to scale the result by the ratio of allocated and total vcpus. Doing all these steps together replicate the existing teads-curve plugin, but only the interpolation part is in scope for this plugin.

Here is the specification doc of the plugin. I've attempted to compose a comprehensive document so anyone who can read it can implement it.
@jawache and @jmcook1186 could you please take a look to verify if this is the correct behavior of the plugin?
https://docs.google.com/document/d/1j6VqeB4ibhf0Hp0UxsLL_k8uSD6qXh4eCXzzkEa-ees/edit?usp=sharing
cc: @narekhovhannisyan @MariamKhalatova

Thanks @manushak
I forgot that the teads curve plugin does a lot more than just interpolation 😞
It also multiplies by the TDP and then scales by the VCPUs.
I think we need to solve this by breaking this up into  a pipeline like so

interpolation
total-cpu-energy (generic multiply by tdp)
used-cpu-energy (generic divide or multiply by VCPUs avail/used)???

We could call it the "teads-pipeline" or just CPUenergy pipeline.
That's the only way this interpolation plugin is generic enough to justify being a built-in.
If we do the above then the names of the input and output parameters needs to be configurable in global config and you could use interpolation with anything, not just utilisation etc ...
@manushak would the pipeline above work? I have a suspicion they don't work like I wrote above.

@jawache This new interpolation plugin performs the calculation you mentioned: it calculates interpolation using a specified method, multiplies by the cpu/thermal-design-power if provided, then multiplies the result by the duration and divides by 3600 to get seconds in an hour. After that, it divides by 1000 to get kilowatt-hours (kWh). If vcpus-total and vcpus-allocated are provided, the result is multiplied by their division. Does this calculation seem correct to you?
I'm considering making general input and output parameters. I'll update the specification doc and inform you to verify.

@jawache I've updated the spec doc, could you please verify it? (it is very massive, which is why I don't write here)
https://docs.google.com/document/d/1j6VqeB4ibhf0Hp0UxsLL_k8uSD6qXh4eCXzzkEa-ees/edit?usp=sharing

Hi @manushak I just read through the spec docs - this is pretty much what I had in mind for this - thanks!
I'll take your doc and use it to update this issue for you to work against when you are reallocated to IF (unless a contributor gets to it first ;o) ).
It makes sense to me that this plugin only does the interpolation and replicating teads requires some additional steps in a pipeline.

This is ready but will be parked for a while until @manushak can return to this project

@jmcook1186 i also suspect we can't replicate the rest of a teads-like pipeline without some more work on generics (multiply a parameter by the ratio of VCPUs total and reserved)

@manushak please check your PR still conforms to the issue requirements (they changes after your original PR was raised)

@manushak please check your PR still conforms to the issue requirements (they changes after your original PR was raised)

There are small changes to make. Currently, I'm working on it, after that I'll rise  a PR


Reporting Module

The purpose of this Epic is to capture items related to Reporting that don't fall into any other specific Epic/feature/user story.

Closed and will reopen assigned tickets in appropriate epics


MC: Pending orders modal + DAI approval/purchase flows

This is TBD, we will circle back to this later.

Done.

Woop wrong issue

Wait no this has been designed!
@cillianhunter0 Re-open if im wrong.


[.NET] Update XCeed number input control to support non-integer values

Related Issue
Fixes #5308
Description
Updated the XCeed number input renderer to use DoubleUpDown rather than IntegerUpDown in order to support non-integer values.
Also update the code to set the element max and min on the control in order to allow the control to enforce those properties.
Sample Card
Added sample card at https://github.com/microsoft/AdaptiveCards/blob/main/samples/v1.3/Tests/Input.Number.NonInteger.json
How Verified
Verified manually using the added sample card that decimal values are supported and that min and max are enforced as expected.
Microsoft Reviewers: Open in CodeFlow

@RebeccaAnne FWIW, iOS CI failed because the shared module failed to parse the card.

@RebeccaAnne FWIW, iOS CI failed because the shared module failed to parse the card.

Yeah, I think we need to wait for #5278 before adding a sample. Looks like @golddove has a sample card as part of his outstanding PR for that (#5323), so I'll just pull the sample card from here for now.


Clarify dev envs and docs

What changed?
Tiltfile and development instructions that helps make it clearer the modes of development we can offer people.
Moved development process out of the README. The readme should be helping our users using the tool.
Clarify contributing guide and maintainers.

Why was this change made?
Need to provider simpler guidance of just get going for a development environment that is a little more reliable and doesn't require local resources building which can give youome problems depending on your knowledge and OS system.
Simplifying docs and bringing clarity to contributing and borrowed from our flux guide.

How was this change implemented?
I like the film fast and furious so for an element of "fun" and also to make it clear that you need to consciously switch it on using the tiltfile for using local resources.
Changes to the tiltfile, co-authored with @ozamosi

How did you validate the change?
I've run through and setup a development environment for each mode of development.

@joshri I tested the frontend development setup, do you want to help me out and try it too in the instructions? You might need to delete what you have.

@JamWils @sympatheticmoose I've moved the development process out of the README. Can you have a read over please?

@joshri I tested the frontend development setup, do you want to help me out and try it too in the instructions? You might need to delete what you have.

ON IT 🌞

@jpellizzari can you test this out and tell me if your setup still works? I think you want to use the fun flag for the tiltfile, cos it keeps the speed optimisations you have for building things on your machine.

switching to draft as wonderful team have helped find some issues, so I'll dig in and get it fixed.


FR: YAML Timestamp section is wonderful, but it needs to ignore your templates.

Is Your Feature Request Related to a Problem? Please Describe.
The Linter YAML timestamp section is exactly what I've been looking for. Keep my timestamps updated or static as the case may be.
However, if after making use of YAML Timestamp very happily, you are unfortunate enough to load a template into Obsidian, your template fields are overwritten with the templates 'real' values. Not optimal.
Describe the Solution You'd Like
Avoid all directories nominated as Template directories.
Provide the user a field to provide the user's template directory [1] to avoid any template directory linting replacing template <% %> etc expressions with 'absolute values'.
[1] It is suggested that the Template core plugin for Obsidian is turned off if the Templater third-party plugin is being used. The template directory in core-plugin Templates is not reliable.
Describe Alternatives You've Considered
The alternative would be to scan the JSON created by Templater, Periodic Notes, and others looking to pick up their values. Not realistic and subject to change without notice.
Obsidian/.obsidian/plugins/templater-obsidian/data.json
3:  "templates_folder": "Templates",

Thank you for your excellent work.

@businessBoris , Thanks for the feedback. Could you explain how folders to ignore is not adequate for this already?

I'm sorry, @pjkaufman, I can't! I messed up.
I haven't seen that field before.
Profuse apologies.

No worries. It might help if you search the Linter's website when you are looking to see if a feature exists. Here is the link to the documentation website: https://platers.github.io/obsidian-linter/


Add ability to automatically complete any passed deadlines

Sometimes there are a bunch of old deadlines that passed long ago, and I have to manually hit "Complete" on each of them. It would be nice if there was a button which would remove all passed deadlines in one click, or at least all sufficiently old ones.

That's a good suggestion, I've realized that this is very frustrating too. I'm very short on time at the moment and therefore not sure when I would be able to fix this, I'll leave the general steps to fix this here if you'd like to take a look at this
This is a client-side fix since completed deadlines aren't stored on the backend as of now, I think it would be sufficient to add a button that would mark all (or all passed) deadlines as completed either in the workspace view, deadline view, or both.
The files related to this are:

src/components/screens/Subjects/components/DeadlineColumn.tsx (workspace page, right side column), I would suggest taking a look at the function removeDeadline (which is badly named, it should be "completeDeadline") in the same file to understand how this works. Then either extend the function or create a new one that simply marks all deadlines in the state as completed with a specific criteria.
There's also client/src/components/screens/Deadlines/Deadlines.tsx (deadline page)

For documentation on how to set up the environment etc, I would suggest taking a look at the docs page which you can find here. Sadly I don't have the page for the documentation running atm but that should be as easy as running, npm install followed by npm start OR if you don't want to bother about setting up the documentation locally you can just visit the raw .md files here.
The docs might be a bit spare on the setup (one thing that I don't think is mentioned is that you have to have both redisDB and mongoDB installed on your system to start the backend), if you actually do plan on implementing the feature and gets stuck on the environment setup don't hesitate to ask for more directions or help.


deploy fails on CDK 1.19

After setting up a Charmed Kubernetes 1.19 cluster and manually adding it to juju and bootstrapping a controller (bootstrapping with juju 2.8.2 seems broken btw) I tried to deploy charmed Kubeflow. When running the command python3 scripts/cli.py deploy-to juju-k8s --cloud k8s-cloud everything seems to go smoothly but it exits with the following error:
DEBUG:root:pytorch-operator is lead by pytorch-operator/0
DEBUG:root:seldon-core is lead by seldon-core/0
DEBUG:root:tf-job-operator is lead by tf-job-operator/0
+ juju kubectl apply -f /tmp/tmpelzkkd4i
Error: ErrorMessage { msg: "Couldn\'t determine cloud substrate." }
Command '('juju', 'kubectl', 'apply', '-f', '/tmp/tmpelzkkd4i')' returned non-zero exit status 1.

I did not run the python3 scripts/cli.py ck setup command as it seems this will deploy and bootstrap a CDK cluster, which I already have. Looking through the script quickly I didn't see the setup command doing more that just creating a cluster and bootstrapping a controller.

I doubt it is related but I am currently facing the following issue when trying to deploy Kubeflow with kfctl (which is why I decided to try charmed-kubeflow).
https://github.com/kubeflow/manifests/issues/1543

Are you using the same controller for both CK and Kubeflow? I ask because the juju kubectl command relies on scp'ing kubeconfig from the kubernetes-master/0 unit in the default model to work properly. If your setup looks different, it would be interesting to know what it looks like, so that juju kubectl could be made smarter.
At any rate, it should work for you if set up ~/.kube/config properly to talk to CK, then change these 3 references to run('kubectl', ... instead of juju('kubectl', ...:
https://github.com/juju-solutions/bundle-kubeflow/blob/master/scripts/cli.py#L347
https://github.com/juju-solutions/bundle-kubeflow/blob/master/scripts/cli.py#L349
https://github.com/juju-solutions/bundle-kubeflow/blob/master/scripts/cli.py#L360
And then update these lines to be 'kubectl', ... instead of 'juju', 'kubectl', ...:
https://github.com/juju-solutions/bundle-kubeflow/blob/master/scripts/cli.py#L144
https://github.com/juju-solutions/bundle-kubeflow/blob/master/scripts/cli.py#L186
https://github.com/juju-solutions/bundle-kubeflow/blob/master/scripts/cli.py#L193
https://github.com/juju-solutions/bundle-kubeflow/blob/master/scripts/cli.py#L200

I am not using the same controller for CK and Kubeflow. The model I am using for CK is not default but one I created (k8s). I did manually copy the config file with juju scp kubernetes-master/0:config ~/.kube/config. I will try with your mentioned edits in just a moment after my new kfctl deployment finishes failing.
FYI, I did manage to resolve the issue with deploying istio on CK which requires --service-account-issuer and --service-account-signing-key-file to be set on the kube-apiserver. Initially I tried to use juju to configure the api-extra-args without succes (althought I think it should be possible to change this way). I SSH'ed into the master nodes and ran sudo openssl rsa -in /root/cdk/serviceaccount.key -pubout -out /root/cdk/serviceaccount.pub to generate the public key. Then I edited /var/snap/kube-apiserver/current/args and added --service-account-issuer="kubernetes.default.svc" --service-account-signing-key-file="/root/cdk/serviceaccount.key" and changed --service-account-key-file="/root/cdk/serviceaccount.key" to --service-account-key-file="/root/cdk/serviceaccount.pub". Hopefully somebody can provide a proper or better way of setting the --service-account-issuer and --service-account-signing-key-file on CK (preferably something that can be configured in the deployment bundle).

The error I was facing has been resolved with the edits you mentioned above. However I am getting stuck at:
kubectl wait --namespace=kubeflow --for=condition=Ready pod --timeout=-1s --all
The pipelines-viewer pod and the ambassador pod are having trouble. The pipelinesviewer pod is in crashbackloop with the following log:
W0915 15:12:00.553509       1 client_config.go:552] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
runtime: mlock of signal stack failed: 12
runtime: increase the mlock limit (ulimit -l) or
runtime: update your kernel to 5.3.15+, 5.4.2+, or 5.5+
fatal error: mlock failed
runtime stack:
runtime.throw(0x12223db, 0xc)
	/usr/local/go/src/runtime/panic.go:1112 +0x72
runtime.mlockGsignal(0xc000602300)
	/usr/local/go/src/runtime/os_linux_x86.go:72 +0x107
runtime.mpreinit(0xc000700000)
	/usr/local/go/src/runtime/os_linux.go:341 +0x78
runtime.mcommoninit(0xc000700000)
	/usr/local/go/src/runtime/proc.go:630 +0x108
runtime.allocm(0xc00006e000, 0x12a4658, 0x0)
	/usr/local/go/src/runtime/proc.go:1390 +0x14e
runtime.newm(0x12a4658, 0xc00006e000)
	/usr/local/go/src/runtime/proc.go:1704 +0x39
runtime.startm(0x0, 0xc000426301)
	/usr/local/go/src/runtime/proc.go:1869 +0x12a
runtime.wakep(...)
	/usr/local/go/src/runtime/proc.go:1953
runtime.resetspinning()
	/usr/local/go/src/runtime/proc.go:2415 +0x93
runtime.schedule()
	/usr/local/go/src/runtime/proc.go:2527 +0x2de
runtime.park_m(0xc000102600)
	/usr/local/go/src/runtime/proc.go:2690 +0x9d
runtime.mcall(0x20000)
	/usr/local/go/src/runtime/asm_amd64.s:318 +0x5b
goroutine 1 [runnable]:
k8s.io/apimachinery/pkg/api/meta.NewDefaultRESTMapper(...)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/api/meta/restmapper.go:81
k8s.io/client-go/restmapper.NewDiscoveryRESTMapper(0xc00015b800, 0x1c, 0x20, 0x1c, 0x20)
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180718001006-59698c7d9724/restmapper/discovery.go:94 +0x62a
sigs.k8s.io/controller-runtime/pkg/client/apiutil.NewDiscoveryRESTMapper(0xc00016e000, 0xc, 0x1220956, 0xa, 0x12208ca)
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.0.0-20181121180216-5558165425ef/pkg/client/apiutil/apimachinery.go:40 +0x9f
sigs.k8s.io/controller-runtime/pkg/manager.New(0xc00016e000, 0xc0000baf60, 0x12a47c8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x7ffeea823f85, ...)
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.0.0-20181121180216-5558165425ef/pkg/manager/manager.go:156 +0x112
main.main()
	/src/github.com/kubeflow/pipelines/backend/src/crd/controller/viewer/main.go:74 +0x36e
goroutine 6 [chan receive]:
github.com/golang/glog.(*loggingT).flushDaemon(0x1bf3880)
	/go/pkg/mod/github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b/glog.go:882 +0x8b
created by github.com/golang/glog.init.0
	/go/pkg/mod/github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b/glog.go:410 +0x26f
goroutine 25 [IO wait]:
internal/poll.runtime_pollWait(0x7f60138cd008, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc000128718, 0x72, 0x2700, 0x271f, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000128700, 0xc000500000, 0x271f, 0x271f, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc000128700, 0xc000500000, 0x271f, 0x271f, 0x203000, 0x700c10, 0xc000188bb8)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc000186040, 0xc000500000, 0x271f, 0x271f, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x8e
crypto/tls.(*atLeastReader).Read(0xc0004c6740, 0xc000500000, 0x271f, 0x271f, 0x13e, 0x271a, 0xc00009f9c8)
	/usr/local/go/src/crypto/tls/conn.go:760 +0x60
bytes.(*Buffer).ReadFrom(0xc000188cd8, 0x139ede0, 0xc0004c6740, 0x40bc45, 0x10e3d80, 0x11dac60)
	/usr/local/go/src/bytes/buffer.go:204 +0xb1
crypto/tls.(*Conn).readFromUntil(0xc000188a80, 0x13a0260, 0xc000186040, 0x5, 0xc000186040, 0x8)
	/usr/local/go/src/crypto/tls/conn.go:782 +0xec
crypto/tls.(*Conn).readRecordOrCCS(0xc000188a80, 0x0, 0x0, 0xc00009fd38)
	/usr/local/go/src/crypto/tls/conn.go:589 +0x115
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:557
crypto/tls.(*Conn).Read(0xc000188a80, 0xc000494000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1233 +0x15b
bufio.(*Reader).Read(0xc000492360, 0xc000484118, 0x9, 0x9, 0xc00009fd38, 0x12a4800, 0x813b12)
	/usr/local/go/src/bufio/bufio.go:226 +0x24f
io.ReadAtLeast(0x139ec40, 0xc000492360, 0xc000484118, 0x9, 0x9, 0x9, 0xc0000a6060, 0x0, 0x139ef00)
	/usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
	/usr/local/go/src/io/io.go:329
golang.org/x/net/http2.readFrameHeader(0xc000484118, 0x9, 0x9, 0x139ec40, 0xc000492360, 0x0, 0x0, 0xc00059b080, 0x0)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20190827160401-ba9fcec4b297/http2/frame.go:237 +0x87
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0004840e0, 0xc00059b080, 0x0, 0x0, 0x0)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20190827160401-ba9fcec4b297/http2/frame.go:492 +0xa1
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc00009ffa8, 0xc00008c7b0, 0x7e41b2)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20190827160401-ba9fcec4b297/http2/transport.go:1686 +0x8d
golang.org/x/net/http2.(*ClientConn).readLoop(0xc00018a480)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20190827160401-ba9fcec4b297/http2/transport.go:1614 +0x6f
created by golang.org/x/net/http2.(*Transport).newClientConn
	/go/pkg/mod/golang.org/x/net@v0.0.0-20190827160401-ba9fcec4b297/http2/transport.go:673 +0x64a

The ambassador pod shows Readiness probe failed: Get "http://<IP_ADDRESS>:8877/ambassador/v0/check_ready": dial tcp <IP_ADDRESS>:8877: connect: connection refused with the following log:
2020-09-15 15:16:20 kubewatch [20 TMainThread] 0.50.0 INFO: kubewatch starting: mode 'sync' ambassador_config_dir '/ambassador/ambassador-config' envoy_config_file '/ambassador/envoy/envoy.json' debug 'False' delay '1.0' pid 'None'
2020-09-15 15:16:20 kubewatch [20 TMainThread] 0.50.0 INFO: namespace kubeflow, watching all namespaces
2020-09-15 15:16:20 kubewatch [20 TMainThread] 0.50.0 ERROR: couldn't read namespace kubeflow? (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'babf9967-a547-42bc-9e4b-1d5910260dff', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'Date': 'Tue, 15 Sep 2020 15:16:20 GMT', 'Content-Length': '329'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"namespaces \"default\" is forbidden: User \"system:serviceaccount:kubeflow:default\" cannot get resource \"namespaces\" in API group \"\" in the namespace \"default\"","reason":"Forbidden","details":{"name":"default","kind":"namespaces"},"code":403}
2020-09-15 15:16:20 kubewatch [20 TMainThread] 0.50.0 INFO: cluster ID is 07eb43c8-1166-5145-a060-45e4dd907e10 (from hardcoded ID)
2020-09-15 15:16:20 kubewatch [20 TMainThread] 0.50.0 WARNING: Kubernetes access failure! (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'fc78e7f2-be4d-40f2-bdda-c32534162832', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'Date': 'Tue, 15 Sep 2020 15:16:20 GMT', 'Content-Length': '287'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"services is forbidden: User \"system:serviceaccount:kubeflow:default\" cannot list resource \"services\" in API group \"\" at the cluster scope","reason":"Forbidden","details":{"kind":"services"},"code":403}

Then I edited /var/snap/kube-apiserver/current/args and added --service-account-issuer="kubernetes.default.svc" --service-account-signing-key-file="/root/cdk/serviceaccount.key" and changed --service-account-key-file="/root/cdk/serviceaccount.key" to --service-account-key-file="/root/cdk/serviceaccount.pub".

Careful here, the charm or the snap will revert this change sooner or later.
You should be able to set this with api-extra-args on the charm. Something like this:
juju config kubernetes-master api-extra-args="service-account-issuer=kubernetes.default.svc service-account-signing-key=/root/cdk/serviceaccount.key service-account-key-file=/root/cdk/serviceaccount.pub"

@Cynerva That would indeed make the changes persistent, however I doubt this setup will work from a bundle directly as serviceaccount.pub would need to be created.

@DavidSpek is this issue still relevant?


Create a snippet for static Vim caret

Instead of a blinking animated one, this one strips out the fanciness and we are left with the beautiful look of a static caret.

This is a great addition! Just add an entry to the CSS Snippets section that follows the conventions of the other entries and I'll merge the PR. Be sure to include a link in the TOC.


BUG Fix clicking "No items found" from causing ajax error

Attempted to fix https://github.com/silverstripe/silverstripe-cms/issues/1627 but found the original error had already fixed itself somehow, so I fixed this bug instead.

Rebased and fixed with your feedback.

@clarkepaul you're a committer now; Can you merge?

ohhh, just because I can doesn't mean I will unless I understand whats going on or reviewed so in this case its all good.
Such privileges


Find all references to symbol

Implement the following feature in CairoLS: https://code.visualstudio.com/api/language-extensions/programmatic-language-features#find-all-references-to-a-symbol

It would be really cool, to have it in LS 👍


getMediaDevices is not a function

i tried integrating the code of RMVideoChatConf project into my project.
and i get this.

"react": "16.9.0",
"react-native": "0.63.0"
help me on this

Our SDK uses react-native-webrtc lib. So you should manually link the lib.
From our docs: https://developers.connectycube.com/reactnative/videocalling?id=connect-webrtc-lib
Guides for iOS and Android: https://github.com/react-native-webrtc/react-native-webrtc#installation

fixed thank you!


fix: ignore Maximum call stack size exceeded in sentry

A bit worried about hiding messages like this that can easily be valid. Is there anyway we could detect the fact that it's coming from something injected?

I added e.stack?.includes('injectWalletGuard.js') as it worked in a similar error that I could reproduce. It should work here too 🤞


Hover effect on issue cards and search tags #14

Please verify and merge the PR.

Hey @aashay28 could you please restrict the hover effect to >= medium breakpoint


Remove Steve Issacs stream listing

Remove the Steve Issacs stream listing from the Available Streams. Throw in headings for Stream Schedule and Available Streams.
Image assets remain in case the stream is added back later.
Closes #6749

Probably should add the  in makeSchedule() and makeLessons().


FireDaemon OpenSSL 3.3.2

Checklist for Pull Requests

[ Y ] Have you signed the Contributor License Agreement?
[ N ] Is there a linked Issue?

Manifests

[ Y ] Have you checked that there aren't other open pull requests for the same manifest update/change?

[ Y ] Does your manifest conform to the 1.6 schema?

Note: <path> is the directory's name containing the manifest you're submitting.

Microsoft Reviewers: Open in CodeFlow

Thank's @stephengillie


[1.20] Update deps for CVE fixes

Please provide a description of this PR:
Update deps to pickup latest CVE fixes
golang.org/x/crypto CVE-2023-48795
github.com/go-jose/go-jose/v3 GHSA-2c7c-3mj9-8fqh

@sergii-ssh: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

release-notes_istio_release-1.20
5228ba081091c4ef5e0689748c177d3a3d263b3c
link
true
/test release-notes

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/retest


Social coding

Interesting. I agree that PR's aren't always followed up properly, and the temptation to brush minor comments aside can be felt. Especially when you moved on to the next change and don't really want to revisit 'the old one', or when corporate pressure (e.g. the desire for a bugfix release to production) is mounting.