Save pointcloud (RGB) without meshing?

Required Info

Camera Model
D435i

Firmware Version
<IP_ADDRESS>

Operating System & Version
Win10

Platform
PC

SDK Version
2.38.1

Language
python

Segment
Robot

Issue Description
Is there an example (python) to save the pointcloud (RGB) without meshing?
Basically, this functionality of the viewer:

I have the following code (adapted from examples in the sdk) but does not return the point cloud with RGB info (but the mesh):
import numpy as np
import pyrealsense2 as rs

import numpy as np

print("Environment Ready")

pc = rs.pointcloud()
points = rs.points()

pipe = rs.pipeline()

#Create a config and configure the pipeline to stream
#  different resolutions of color and depth streams
config = rs.config()
config.enable_stream(rs.stream.depth, 640, 480, rs.format.z16, 30)
config.enable_stream(rs.stream.color, 640, 480, rs.format.bgr8, 30)

# Start streaming
profile = pipe.start(config)

# Getting the depth sensor's depth scale (see rs-align example for explanation)
depth_sensor = profile.get_device().first_depth_sensor()
depth_scale = depth_sensor.get_depth_scale()
print("Depth Scale is: " , depth_scale)

# We will be removing the background of objects more than
#  clipping_distance_in_meters meters away
clipping_distance_in_meters = 1 #1 meter
clipping_distance = clipping_distance_in_meters / depth_scale

# Create an align object
# rs.align allows us to perform alignment of depth frames to others frames
# The "align_to" is the stream type to which we plan to align depth frames.
align_to = rs.stream.color
align = rs.align(align_to)

# Get frameset of color and depth
frames = pipe.wait_for_frames()
# frames.get_depth_frame() is a 640x360 depth image

# Align the depth frame to color frame
aligned_frames = align.process(frames)

# Get aligned frames
aligned_depth_frame = aligned_frames.get_depth_frame() # aligned_depth_frame is a 640x480 depth image
color_frame = aligned_frames.get_color_frame()

#depth_image = np.asanyarray(aligned_depth_frame.get_data())
#color_image = np.asanyarray(color_frame.get_data())

points = pc.calculate(aligned_depth_frame)
pc.map_to(color_frame)
print("Saving to 1.ply...")
points.export_to_ply("1.ply", color_frame)
print("Done")

Hi @philianeles   The SDK provides OPTION_PLY configuration commands for enabling / disabling mesh, normals and binary point cloud settings.
https://github.com/IntelRealSense/librealsense/pull/4906
A Python script in the opening comment of the discussion linked to below provides an example of defining the off / on state of these commands when exporting to a ply file.
https://github.com/IntelRealSense/librealsense/issues/6194

The settings that you sent if for rs2::save_to_ply processing block.
Which does not really work. In the code snipped that you sent, it is also written as "mesh is created by default". I am after getting the point cloud rather.
Could you you share some sample code that woks? I tried trace down the other issues (for saving pc with RGB) but many were closed before any solution was found.

Yes, I have this now:
    ply.set_option(rs.save_to_ply.option_ply_mesh, False)
    ply.set_option(rs.save_to_ply.option_ply_binary, False)
    ply.set_option(rs.save_to_ply.option_ply_normals, True)
    ply.set_option(rs.save_to_ply.option_ignore_color, False)

which gives me the pointcloud, without normals and without RGB info - how do I include them?

I went through your code carefully, checking it against other examples.  One trend I noticed is that pc_map_to preceded points:
pc.map_to(color_frame)
points = pc.calculate(aligned_depth_frame)

Whilst in your own script, you reverse that order:
points = pc.calculate(aligned_depth_frame)
pc.map_to(color_frame)

So it may be worth making that edit in your script just to eliminate it as a cause of your problems in getting RGB.
Also, my understanding is that the SDK's default is to export a ply in binary format, and binary = false is used if someone aims to export the data in textual / ASCII format.  So can you try setting binary to True please?
https://github.com/IntelRealSense/librealsense/issues/882#issuecomment-349212011

I reversed the order to your suggestion:
pc.map_to(color_frame)
points = pc.calculate(aligned_depth_frame)

However the result is still the mesh. No pointcloud with RGB info.
I tried also setting binary to True. But still no pointcloud with RGB info...

I compared your script to the SDK's export_ply_example Python example that can export color.
https://github.com/IntelRealSense/librealsense/blob/master/wrappers/python/examples/export_ply_example.py
It is possible that using points.export_to_ply to export the ply is not compatible with setting the ply options with ** ply.set_option(rs.save_to_ply ...)**.
In the export_to_ply example, which also supports setting the ply configuration options, it exports the ply with the save_to_ply method instead of export_to_ply:
https://github.com/IntelRealSense/librealsense/blob/master/wrappers/python/examples/export_ply_example.py#L36
Notably, it also sets the options after the export instruction.

I have a version with save_to_ply actually: (exactly like your suggestion - however does not return the RGB info. Only a point cloud that is missing the color info and the normals.
import numpy as np
import matplotlib.pyplot as plt
import pyrealsense2 as rs

import numpy as np

print("Environment Ready")

def export ():
    # Setup:
    pipe = rs.pipeline()
    cfg = rs.config()
    # cfg.enable_stream(rs.stream.depth, 640, 480, rs.format.z16, 30)
    cfg.enable_stream(rs.stream.depth, 848, 480, rs.format.z16, 30)
    # cfg.enable_stream(rs.stream.depth)
    profile = pipe.start(cfg)

    # Skip 5 first frames to give the Auto-Exposure time to adjust
    for x in range(5):
        pipe.wait_for_frames()

    frames = []
    for x in range(15):
        frameset = pipe.wait_for_frames()
        frames.append(frameset.get_depth_frame())
    # print (frames)

    # Cleanup:
    pipe.stop()
    print("Frames Captured")

    ### POST PROCESSING FILTERS ###
    ##DECLARE FILTERS
    #Threshold filter
    threshold = rs.threshold_filter(min_dist = 0.15, max_dist = 2)
    #Decimation filter
    decimation = rs.decimation_filter()
    decimation.set_option(rs.option.filter_magnitude, 2)
    #Spatial filter
    spatial = rs.spatial_filter()
    spatial.set_option(rs.option.filter_magnitude, 1)#Filtering itterations from 1.000 to 5.000
    spatial.set_option(rs.option.filter_smooth_alpha, 1)
    spatial.set_option(rs.option.filter_smooth_delta, 3)# 8 is good
    spatial.set_option(rs.option.holes_fill, 2)
    #Temporal Filter
    temporal = rs.temporal_filter()
    temporal.set_option(rs.option.filter_smooth_alpha, 0.2)
    temporal.set_option(rs.option.filter_smooth_delta, 100)
    #Hole filling
    hole_filling = rs.hole_filling_filter()
    #Colorize
    colorizer = rs.colorizer()
    colorizer.set_option(rs.option.color_scheme, 2)

    depth_to_disparity = rs.disparity_transform(True)
    disparity_to_depth = rs.disparity_transform(True)
    #Apply filters
    for i in range(15):
        depth_frame = frames[i]
        depth_frame = threshold.process(depth_frame)
        depth_frame = temporal.process(depth_frame)
        depth_frame = decimation.process(depth_frame)
        depth_frame = spatial.process(depth_frame)
        # depth_frame = hole_filling.process(depth_frame)
        # depth_frame = colorizer.process(depth_frame)
        # colorized = colorizer.process(depth_frame)

    # Create save_to_ply object
        ply = rs.save_to_ply("pc.ply")

        # Set options to the desired values
        # Generate a textual PLY with normals (mesh is already created by default)
        ply.set_option(rs.save_to_ply.option_ply_mesh, False)
        ply.set_option(rs.save_to_ply.option_ply_binary, False)
        ply.set_option(rs.save_to_ply.option_ply_normals, True)
        ply.set_option(rs.save_to_ply.option_ignore_color, False)

    print("Saving ply...")
    # Apply the processing block to the frameset which contains the depth frame and the texture
    ply.process(depth_frame)
    print("Done")

export()

The next action that I would advise trying is to move the entire ply export code block to one line above the pipe.stop() instruction so that the export instruction is run whilst the pipeline is still active.

Unfortunately if pipe.stop() comes after that block the code does not run without errors. No file saved at all.

Apologies for the delay in responding further.   I was carefully considering the next step.
I have the feeling that there may be an issue related to how alignment is being set up.  Looking at the align block setup in the code example below may give insights about something that may need to be changed / added in your own script.
https://github.com/IntelRealSense/librealsense/issues/2769#issuecomment-442035864

Hi @philianeles   Do you have progress with this case to report, please?   Thanks!

Hi @MartyG-RealSense, yes, I also tried this suggestion however did not help.

There was another case recently from a RealSense user trying to export a colored point cloud to a ply with Python and not succeeding.  It seems to be a common pattern of cases in which RealSense users try to achieve this in Python without success.
https://github.com/IntelRealSense/librealsense/issues/7770

Hi @philianeles   Do you require further assistance with this case, please?   Thanks!

Case closed due to no further cmmments received.

@MartyG-RealSense Hey I am working on cropping depth map that I get from Intel RealSense D435 to get the depth map of a particular object in the frame and then process that cropped depth map into a point cloud. I tried it with the inbuilt functions but it only saved the entire frame as point cloud and not the cropped depth map. How do you think we can do that?

Hi @sanjaiiv04   The D435 and D455 camera model types support an 848x100 resolution where the depth data is cropped to a wide but thin horizontal strip at the center of the camera's field of view.
2D mode

3D point cloud mode

It is possible to define your own horizontal and vertical cropping, though it is a complex procedure.  https://github.com/IntelRealSense/librealsense/issues/2016#issuecomment-403804157 offers advice from a RealSense user about a method for doing so.


Link task list guidance to task list code

Fixes #1320.
This PR adds links from our task list guidance to 2 places where users can get the necessary code for a task list example:

HTML from the task list page's HTML file
custom styles from the task list page's SCSS file

We've added these links because a user reported being unable to find the code.

Thanks, @lfdebrux!


INCEpTION HelmChart

What's in the PR

This PR contains a HelmChart allowing INCEpTION's deployment in a cloud-native & vendor-agnostic way

How to test manually

Deploy a local K8S cluster using kind with a deployed Ingress Controller
You must have Helm 3 installed
From the root of the git repo, just run helm upgrade --install inception ./chart/inception

Once the INCEpTION pod will be ready, you can access the application from http://localhost/

Automatic testing

There's no auto-test for Helm, but we can talk about including documentation generation, schema validation and helm linting in the CI

Documentation

All the Chart's documentation is inside chart/inception/README.md file
That documentation can be re-generated with helm-doc, from the root of the git repo :docker run --rm --volume "$(pwd)/chart/inception:/helm-docs" -u $(id -u) jnorwood/helm-docs:latest --sort-values-order=file

The values schema file can be re-generated with the Helm values-schema plugin, from the root of the git repo :helm schema -input chart/inception/values.yaml

If you're OK with the merge request, we can discuss about updating the official INCEpTION documentation regarding the Kubernete's deployment section

PS @reckart : The CLAs are on their way, the holidays are adding a bit of delay. I think it's best not merging this MR until you have them, but I'll happily exchange about the feature until then :)

TODO :

Update INCEpTION's documentation
Using gh pages as a chart repository


agent-basic-js: add agentVersion plugin to list of default plugins

This change is

Review status: 0 of 1 files reviewed at latest revision, all discussions resolved.

Comments from Reviewable

Reviewed 1 of 1 files at r1.
Review status: all files reviewed at latest revision, all discussions resolved.

Comments from Reviewable


Add TIFF decoding support

libtiff was compiled with support for all compression schemes defined by the TIFF 6.0 spec, in addition to schemes that Wikipedia considered "Common".
Closes #1092.
  Libtiff build version:              4.6.0
  Libtiff library version:            6:2:0
  Libtiff release date:               20230908

  Installation directory:             /usr/local
  Documentation directory:            ${prefix}/share/doc/tiff-4.6.0
  C compiler:                         clang-17 --target=wasm32-wasi --sysroot=/wasi-sysroot
  C++ compiler:                       clang++-17 --target=wasm32-wasi --sysroot=/wasi-sysroot
  Build tools:                        no
  Build tools-unsupported:            no
  Build tests:                        no
  Build contrib:                      no
  Build docs:                         no
  Build deprecated features:          no
  Enable runtime linker paths:        no
  Enable linker symbol versioning:    no
  Support Microsoft Document Imaging: no
  Use Win32 IO:                       no

 Support for internal codecs:
  CCITT Group 3 & 4 algorithms:       yes
  Macintosh PackBits algorithm:       yes
  LZW algorithm:                      yes
  ThunderScan 4-bit RLE algorithm:    no
  NeXT 2-bit RLE algorithm:           no
  LogLuv high dynamic range encoding: yes

 Support for external codecs:
  ZLIB support:                       yes
  libdeflate support:                 no
  Pixar log-format algorithm:         yes
  JPEG support:                       yes
  Old JPEG support:                   no
  JPEG 8/12 bit dual mode:            yes (libjpeg turbo >= 2.2 dual mode)
  ISO JBIG support:                   no
  LERC support:                       no
  LZMA2 support:                      no
  ZSTD support:                       no
  WEBP support:                       no

  C++ support:                        no

It was compiled using WASI toolchain instead of Emscripten because I vaguely recall last time I tried this with Emscripten I encountered a lot of issues. But it was a long time ago and I can't remember what those issues were...

Thanks so much for whipping up a PR. I really appreciate the effort. I’d love to merge and ship this, however I haven’t really  reviewed any code because to the last of my knowledge, the problem is that TIFF the file format is patented. This is not about the code that contains the library, but the intellectual property. I do not think we can ship this.
I’d love to be proven wrong, tho...

I've been looking around and wasn't able to find any information referring to patents or licensing requirements on the TIFF format. According to this Library of Congress page, the "Licensing and patents" sustainability factor for TIFF is described as:

Not exploited for the TIFF wrapper format. The only widely used compression scheme for the embedded image data that has been subject to patent claims in recent years is LZW (Lempel-Ziv-Welch) compression, i.e., in TIFF_LZW. The LZW patent is generally reported as having expired in 2003 (U.S.) and 2004 (Europe and Japan).

Besides libtiff, other libraries don't seem to have concerns with it either, such as Google's TIFF library written in Go, and another Rust library used in the popular image processing crate image-rs.

@surma Hi, could you take a look when you get a chance? Thanks!


chore: update go and golanci lint

This PR updates go to 1.23.1 and updates golanci-lint to v1.61.1, while fixing new lint errors.

@loosebazooka

+1 to jku's comment, but LGTM

removed, please +1 again.


Add recycle-bin

@Ash258 @rasa This shouldn't be put under bucket directory?

@iquiw You're right. Misclicked folder.


Create PR description template

Typically reminding which details to remember to include, a checklist of testing steps etc

Some other examples of the types of prompts that big teams at companies sometimes include:
This PR resolves Issue #
Change Summary

Add component _a
Update feature _b to do _c when _d
Fix bug when _e

I Had Difficulty With
I initially tried to implement this using our existing components, but ran into , so I decided to add a third party component for this task.


[PhpStan-Rules] symfony formView field not recognised in twig

Greetings !
using the latest version of symplify/phpstan-rules (in this context precisely :

php : 8.0.7
phpstan : 0.12.96
symplify/phpstan-rules : 9.4.48
symfony : 5.3.6
) and in continuity with the issue #3525, I have this error seemingly false positive output :

 ------ --------------------------------------------------------------------
  Line   src/Controller/DefaultController.php
 ------ --------------------------------------------------------------------
  27     Variable "form" of type "Symfony\Component\Form\FormView" does not
         have "fieldOne()" method
 ------ --------------------------------------------------------------------

when calling the field for example like this :  {{ form_row(form.fieldOne) }}
cf. >here< for more info.
regards !

Hi, thanks for reporting.
We'll need failing test fixture for that rule to be added via PR. The we can fix it.

Hello, the formView being mainly an iterator with an implementation of the offsetGet method on its children array (where it seems to be getting stuck), here I propose you those fixtures :  #3546


Fix inputNumber display

Fixes InputNumber component issue when using w-full class.
Before

After

Hi, Thank you for your PR 💚


Screenshots from firefox are blank

When updating a card with ctrl+shift+u the screenshot that ends up in anki is completely black when done in firefox, The screenshot is also completely black if you download it from mining history

Firefox isn't supported.
You'll have to use a Chromium-based browser.
Out of curiosity, you are seeing this with a local file or streaming video?

local file, everything else seems to work fine.

When updating a card with ctrl+shift+u the screenshot that ends up in anki is completely black when done in firefox, The screenshot is also completely black if you download it from mining history

What type of screenshot file is it?

Also, Have you tried Chrome or Edge?

When updating a card with ctrl+shift+u the screenshot that ends up in anki is completely black when done in firefox, The screenshot is also completely black if you download it from mining history

@ChimneyBiscuits, May I please see your code? I might be able to suggest something.

@ChimneyBiscuits, May I please see your code? I might be able to suggest something.

I don't code, i'm just a user of the site. I switched to using a chromium-based browser months ago

Oh, Sorry to bother you about coding.


Un-hardcoded genital type genes

Merged genital type gene classes as it's been bugging me since 1.4. Should probably be merged after jaaldabaoth's update.
Reasoning:

Reduces code repetition, likelihood of bugs, etc.
Gives genital genes something in common other than defName, making it easier to write code that works with them as a group.
Reduces amount of work involved in adding or changing genital genes.
Makes it possible or other mods to add similar genes without trying to access an internal class, or even using C# in the first place.

Other changes

Stopped tracking .csproj.user file. This allows people to define their own project settings without affecting the repo's .csproj (read: allows me to build in my own banjaxed environment without breaking it for others).
Refactored GenitaliaUtility to mostly not depend on defNames. In particular, the GetXForGene methods were checking against an incomplete list of hardcoded defNames, so this might fix some issues in places where these are used.

I left GetBreastsForGene alone for now since they don't get

Tweaked GenitalChanger to not overwrite parts if the new part would be of the same def, and so it can be used to replace just one or two parts instead of all three.

Included in #81.


Hotfix v1.0.1

Description:

Fixed bug introduced in last production release - set game background back to black

@ezequielgar

@ezequielgar


Triggers are not working

[Enter feedback here]

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: b14972b7-883c-37c8-96f9-114327f15e05
Version Independent ID: 00baf3e0-f225-dc97-a15d-f9d97dac28d2
Content: Build code from GitHub Enterprise Server - Azure Pipelines
Content Source: docs/pipelines/repos/github-enterprise.md
Service: azure-devops-pipelines
Sub-service: azure-devops-pipelines-repos
GitHub Login: @steved0x
Microsoft Alias: sdanie

@Radhika108 -- thank you for your feedback.  Please consider these resources:

Azure DevOps Services
Azure DevOps on Stack Overflow

I am closing this issue now.  You are welcome to @ mention me for any followup.  We hope to hear from you again.


RPC Raw Stream Errors (distinguish between transport errors and application errors)

Specification
We need to review how errors are handled for RPC raw handlers. The transport level error handling needs to be reviewed as well. Error handling between websocket streams and QUIC streams need to be stadardized between each other as well.
There needs to be separation between transport level errors and RPC level ones. I think this is already the case.
Additional context

Related: https://github.com/MatrixAI/Polykey/pull/525#discussion_r1272020257
Related MatrixAI/Polykey#525
Related: MatrixAI/Polykey#521
https://github.com/MatrixAI/js-ws/issues/1

Tasks

...
...
...

This can only be done after https://github.com/MatrixAI/js-ws/issues/1 and https://github.com/MatrixAI/js-ws/issues/2 is done first. Because in order to separate 3 levels of errors:

Connection Errors
Stream Errors
RPC Call Errors

The concept of a "stream" needs to exist above a "WebSocket" connection.
For js-quic this is already done due to QUICStream.

[x] Re-introduce codes which was refactored out and replace them with RPC specific codes instead of PK codes.
[ ] Write specific errors for RPC -> -32000 - -32099
[ ] Implement codes in server.

(Cope concurrent jests get fixed.)

[ ] Implement with toError and fromError.
[ ] Jests?


pull request for e2e test at 01/06/2017 23:26:13

:white_check_mark: Validation status: passed
For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Update workflow to point to 1.1

Signed-off-by: bowenlan-amzn<EMAIL_ADDRESS>Issue #, if available:
Description of changes:
CheckList:

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #146 (5c06312) into 1.1 (1ec0bc6) will increase coverage by 0.20%.
The diff coverage is n/a.

:exclamation: Current head 5c06312 differs from pull request most recent head d31c9be. Consider uploading reports for the commit d31c9be to get more accurate results

@@             Coverage Diff              @@
##                1.1     #146      +/-   ##
============================================
+ Coverage     75.79%   75.99%   +0.20%
- Complexity     1919     1921       +2
============================================
  Files           260      260
  Lines         10992    10992
  Branches       1720     1720
============================================
+ Hits           8331     8353      +22
+ Misses         1710     1683      -27
- Partials        951      956       +5

Impacted Files
Coverage Δ

...nt/indexstatemanagement/model/destination/Slack.kt
40.90% <0.00%> (-13.64%)
:arrow_down:

...anagement/indexstatemanagement/model/Transition.kt
63.07% <0.00%> (-1.54%)
:arrow_down:

.../opensearch/indexmanagement/rollup/model/Rollup.kt
86.56% <0.00%> (+0.49%)
:arrow_up:

...earch/indexmanagement/transform/model/Transform.kt
87.00% <0.00%> (+1.00%)
:arrow_up:

...ent/indexstatemanagement/util/ManagedIndexUtils.kt
79.22% <0.00%> (+1.29%)
:arrow_up:

...ransform/action/get/TransportGetTransformAction.kt
57.50% <0.00%> (+2.50%)
:arrow_up:

...statemanagement/model/destination/CustomWebhook.kt
92.75% <0.00%> (+27.53%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1ec0bc6...d31c9be. Read the comment docs.


Implement SepWriterOptions.Escape (true)

Currently SepWriter does not support automatic escaping (adding quotes) upon writing. Implement that similar to reader unescape. Default to false.

This coming? or maybe just a helper?

No progress sadly, the little time I have is spent on other issues currently. When you say helper what do you imagine?

@nietras perhaps an extension method on the writer, such as SetEscaped(string?)


[1.12] Mob Crusher causing ticking block entity crashes?

Playing on Sky Factory 4 modpack. Using the Mob Crusher to kill cows with a +5 Range Addon.
Guess I will have to find another way to get essence. =(
Industrial Foregoing Version: industrialforegoing-1.12.2-1.12.13-237.jar
TeslaCoreLib Version: tesla-core-lib-1.12.2-1.0.17.jar
Crashlog If Applicable (Upload to pastebin/gist): Crashlog

:wave: Sorry but Industrial Foregoing has ended support for version 1.12. I'm closing this issue as it will no longer be updated for that version. Thanks for playing with the mod.


Zabbix configuration steps in detail

Hi Shamil,
could you please explain in detail how to configure zabbix on puppet enterprise 2015.3 master.
And what configurations do i need to do on Zabbix server.
--Sagar

All relevant information is in README. Closing the issue.
If this still relevant, please re-open and I'll try to help you...


Adds mise to image for user-space package management

Closes https://github.com/spacelift-io/runner-terraform/issues/55

hey @lorengordon - thanks for the contribution. we're reluctant to add further dependencies into the image, especially ones that are only maintained by a single person. 🙂 if the community is very interested in it, we'll reconsider our stance but for now I'd recommend you to build your image on top of this one.

@peterdeme There is also asdf, to address the package availability and maintenance concern.

for now I'd recommend you to build your image on top of this one.

Please see the linked issue. The whole point of adding a user-space package management tool is to avoid the need to build and maintain an image publishing pipeline and associated infrastructure. That is not a trivial suggestion; there is significant operational burden there for private projects, and little to no business value. Whereas adding a tiny package to the spacelift image is quite trivial.
I definitely understand the reluctance to add packages. But this image is used as a CI service, and CI images quite commonly have rather large manifests to address the large variety of use cases and tool versions. Using a package manager like mise or asdf helps maintain the slimness of the image, exactly because users can use the tool to install packages instead of baking them into the image!

But this image is used as a CI service, and CI images quite commonly have rather large manifests to address the large variety of use cases and tool versions.

To be fair, we don't really see Spacelift as a generic CI service like Jenkins or TeamCity. I think this assumption makes a lot of sense there but not necessarily for plain Terraform deployments.

Using a package manager like mise or asdf helps maintain the slimness of the image, exactly because users can use the tool to install packages instead of baking them into the image!

How would you use it by the way? Would you install some tooling in as a pre-init hook?
In any case, I reopened the GitHub issue to see if anyone else feels strongly about this.

How would you use it by the way? Would you install some tooling in as a pre-init hook?

Yes, exactly, probably added to a stack using a context. I see the ability to manage tool versions as a key component of the package manager. Including terraform, opentofu, terragrunt, tflint, aws-cli, opa, infracost, etc. Even python and other languages.
Personally, I just don't like relying on image versions for that kind of consistency and control. Too many layers, too much abstraction, too many potential combinatorics for requirements across different projects. I much prefer managing it at the project level. I mean, Spacelift could replace the several different images they maintain with a single one, and run asdf/mise themselves in the "preparation" step of a stack run...
I know Spacelift offers some capability for managing at least the terraform/opentofu/terragrunt versions at the stack-level. But I think it's only those tools? Where it's just a builtin capability of asdf/mise for any tool in their registry. The support from renovatebot to update the tool versions is also quite useful.


Make hang and indexer speeds tunable

This will allow for fine-tuning of these mechanisms without constantly deploying code.

I've now also made the indexer's state tunable as well, so we can easily toggle these on the dashboard.


fix: README updates

Codecov Report

Merging #15 (5c99d81) into main (af5ce2b) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##              main       #15   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files            1         1
  Lines           80        80
  Branches        10        10
=========================================
  Hits            80        80

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update af5ce2b...5c99d81. Read the comment docs.


It seems that JWT filter can not handle correctly with multiple tokens and allow missing open

Title: It seems that JWT filter can not handle correctly with multiple tokens and allow missing open
Description:
https://github.com/envoyproxy/envoy/blob/8fc7f550edb6d42c70143cdb1f1be78f47af6388/source/extensions/filters/http/jwt_authn/authenticator.cc#L461
i am very confused about the logic here, what is expected if a request has multiple tokens and allow missing is true at the same time? according to code above, when allow missing is true, the verify result just depends on the last token. Is this as expected?

cc @TAOXUY  @lizan as code owner, thanks.

friendly ping @lizan @kyessenov

friendly ping @lizan @kyessenov


On the third time navigation, the window pops back. not work perfectly

On the third time navigation, the window pops back. not work perfectly

Solved


Admin: Switching admin_preview_as inside admin between text/html/python

I don't see it necessary by default, but with mixins.AdminPreviewMixin user can add one if needed.


rdpdr not working in debian jessie

Hi folks,
just to let you know that in debian jessie i cannot seem to get the rdpdr channel working (though it did work in wheezy). Funny enough the clipboard works fine both ways, but i do not see disks on the remote side. Tested in Cord ,  Microsoft Remote Desktop and RDP on windows.
From the linux side i do not see the channelsever being mounted by fuse.
Any ideas?

Is clipboard file working?

There is no .clipboard file in the folder normally showing local drives, however the clipboard works both ways.

do you have fuse installed on your server ?
I'm installing some jessie, and have no issues with drives redirections

Thanks for geeting back to me...
Fuse is installed indeed, for example sshfs (fuse module) works without any issues.
Which version did you install?
I have testest 0.9.0 (master) and 0.9.X - (dev) both not doing the drive redirection.

On 24 jul. 2015, at 09:30, Blindauer Emmanuel<EMAIL_ADDRESS>wrote:
do you have fuse installed on your server ?
I'm installing some jessie, and have no issues with drives redirections
—
Reply to this email directly or view it on GitHub https://github.com/neutrinolabs/xrdp/issues/266#issuecomment-124390181.

I've used the current "devel" branch.
Can you start "xrdp -ns" and "xrdp-sesman -ns" and save the ouput of these while connecting ?

Currently not, since these machines are in production.
next wednesday there is a window for that.
I remember though that the 'rdprd channel' was not enabled ( is strted the daemons like that indeed)
only the clipboard channel and the sound channel (which choked since i did not put the pulse stuff in the proper place).
As soon as i can I will post the output here - just not quite sure why the channel is not activated at all. (It used to work in the previous debian, with some hacks for LDAP , see https://github.com/neutrinolabs/xrdp/issues/198 https://github.com/neutrinolabs/xrdp/issues/198 )

On 24 jul. 2015, at 09:58, Blindauer Emmanuel<EMAIL_ADDRESS>wrote:
I've used the current "devel" branch.
Can you start "xrdp -ns" and "xrdp-sesman -ns" and save the ouput of these while connecting ?
—
Reply to this email directly or view it on GitHub https://github.com/neutrinolabs/xrdp/issues/266#issuecomment-124411235.

Ok, I just dragged what i could from journald:
Jul 24 15:19:11 blue XRDP-sesman[52153]: (52153)(140139386357504)[INFO ] An established connection closed to endpoint: <IP_ADDRESS>:39065 - socket: 8
Jul 24 15:19:11 blue XRDP-sesman[52155]: (52155)(140139386357504)[INFO ] X11rdp :53 -geometry 1280x1024 -depth 24 -bs -ac -nolisten tcp -uds
Jul 24 15:19:11 blue XRDP-sesman[52153]: (52153)(140139386357504)[INFO ] starting xrdp-sessvc - xpid=52155 - wmpid=52154
Jul 24 15:19:11 blue XRDP-sesman[52154]: pam_unix(xrdp-sesman:session): session opened for user vincent by (uid=0)
Jul 24 15:19:11 blue systemd-logind[1504]: New session c29 of user vincent.
Jul 24 15:19:11 blue systemd[52157]: pam_unix(systemd-user:session): session opened for user vincent by (uid=0)
Jul 24 15:19:11 blue systemd[52157]: Starting Paths.
Jul 24 15:19:11 blue systemd[52157]: Reached target Paths.
Jul 24 15:19:11 blue systemd[52157]: Starting Timers.
Jul 24 15:19:11 blue systemd[52157]: Reached target Timers.
Jul 24 15:19:11 blue systemd[52157]: Starting Sockets.
Jul 24 15:19:11 blue systemd[52157]: Reached target Sockets.
Jul 24 15:19:11 blue systemd[52157]: Starting Basic System.
Jul 24 15:19:11 blue systemd[52157]: Reached target Basic System.
Jul 24 15:19:11 blue systemd[52157]: Starting Default.
Jul 24 15:19:11 blue systemd[52157]: Reached target Default.
Jul 24 15:19:11 blue systemd[52157]: Startup finished in 12ms.
Jul 24 15:19:12 blue XRDP[52066]: (52066)(140575971706624)[INFO ] lib_mod_log_peer: xrdp_pid=52066 connected to X11rdp_pid=52155 X11rdp_uid=1539 X11rdp_gid=3000 client_ip=<IP_ADDRESS> client_port=64347
Jul 24 15:19:12 blue XRDP[52066]: (52066)(140575971706624)[DEBUG] xrdp_mm_connect_chansrv: chansrvconnect successful
Jul 24 15:19:12 blue XRDP[52066]: (52066)(140575971706624)[INFO ] An established connection closed to endpoint: <IP_ADDRESS>:3350 - socket: 11
Jul 24 15:19:12 blue XRDP[52066]: (52066)(140575971706624)[INFO ] The following channel is allowed: rdpdr (0)
Jul 24 15:19:12 blue XRDP[52066]: (52066)(140575971706624)[INFO ] The following channel is allowed: cliprdr (1)
Jul 24 15:19:12 blue XRDP[52066]: (52066)(140575971706624)[DEBUG] The allow channel list now initialized for this session
=-=- xrdp.ini =-=-=-
[globals]
xrdp.ini file version number
ini_version=1
bitmap_cache=yes
bitmap_compression=yes
port=3389
allow_channels=true
max_bpp=32
fork=yes
minimum security level allowed for client
can be 'none', 'low', 'medium', 'high', 'fips'
crypt_level=high
security layer can be 'tls', 'rdp' or 'negotiate'
for client compatible layer
security_layer=rdp
X.509 certificate and private key
openssl req -x509 -newkey rsa:2048 -nodes -keyout key.pem -out cert.pem -days 365
certificate=/etc/xrdp/cert.pem
key_file=/etc/xrdp/key.pem
regulate if the listening socket use socket option tcp_nodelay
no buffering will be performed in the TCP stack
tcp_nodelay=yes
regulate if the listening socket use socket option keepalive
if the network connection disappear without close messages the connection will be closed
tcp_keepalive=yes
#tcp_send_buffer_bytes=32768
#tcp_recv_buffer_bytes=32768

colors used by windows in RGB format

blue=009cb5
grey=dedede
#black=000000
#dark_grey=808080
#blue=08246b
#dark_blue=08246b
#white=ffffff
#red=ff0000
#green=00ff00
#background=626c72
#autorun=xrdp1
#hidelogwindow=yes
when true, userid/password must be passed on cmd line
require_credentials=yes
set a default entry for autorun if the client send login and pass directly
autorun=xrdp1
bulk_compression=yes
You can set the PAM error text in a gateway setup (MAX 256 chars)
#pamerrortxt=change your password according to policy at http://url
new_cursors=yes
allow_multimon=true
fastpath - can be set to input / output / both / none
use_fastpath=both

configure login screen

Login Screen Window Title
#ls_title=My Login Title
top level window background color in RGB format
ls_top_window_bg_color=009cb5
width and height of login screen
ls_width=350
ls_height=430
login screen background color in RGB format
ls_bg_color=dedede
optional background image filename (bmp format).
#ls_background_image=
logo
full path to bmp-file or file in shared folder
ls_logo_filename=
ls_logo_x_pos=55
ls_logo_y_pos=50
for positioning labels such as username, password etc
ls_label_x_pos=30
ls_label_width=60
for positioning text and combo boxes next to above labels
ls_input_x_pos=110
ls_input_width=210
y pos for first label and combo box
ls_input_y_pos=220
OK button
ls_btn_ok_x_pos=142
ls_btn_ok_y_pos=370
ls_btn_ok_width=85
ls_btn_ok_height=30
Cancel button
ls_btn_cancel_x_pos=237
ls_btn_cancel_y_pos=370
ls_btn_cancel_width=85
ls_btn_cancel_height=30
[Logging]
LogFile=xrdp.log
LogLevel=DEBUG
EnableSyslog=1
SyslogLevel=DEBUG
LogLevel and SysLogLevel could by any of: core, error, warning, info or debug
[channels]
Channel names not listed here will be blocked by XRDP.
You can block any channel by setting its value to false.
IMPORTANT! All channels are not supported in all use
cases even if you set all values to true.
You can override these settings on each session type
These settings are only used if allow_channels=true
rdpdr=true
rdpsnd=true
drdynvc=true
cliprdr=true
rail=true
xrdpvr=true
tcutils=true
for debugging xrdp, in section xrdp1, change port=-1 to this:
port=/tmp/.xrdp/xrdp_display_10
for debugging xrdp, add following line to section xrdp1
chansrvport=/tmp/.xrdp/xrdp_chansrv_socket_7210
[xrdp1]
name=sesman-X11rdp
lib=libxup.so
username=ask
password=ask
ip=<IP_ADDRESS>
port=-1
xserverbpp=24
code=10
[xrdp2]
name=sesman-Xvnc
lib=libvnc.so
username=ask
password=ask
ip=<IP_ADDRESS>
port=-1
#delay_ms=2000
[xrdp3]
name=console
lib=libvnc.so
ip=<IP_ADDRESS>
port=5900
username=na
password=ask
#delay_ms=2000
[xrdp4]
name=vnc-any
lib=libvnc.so
ip=ask
port=ask5900
username=na
password=ask
#pamusername=asksame
#pampassword=asksame
#pamsessionmng=<IP_ADDRESS>
#delay_ms=2000
[xrdp5]
name=sesman-any
lib=libvnc.so
ip=ask
port=-1
username=ask
password=ask
#delay_ms=2000
[xrdp6]
name=rdp-any
lib=librdp.so
ip=ask
port=ask3389
[xrdp7]
name=neutrinordp-any
lib=libxrdpneutrinordp.so
ip=ask
port=ask3389
username=ask
password=ask
[Session manager]
name=Session manager
lib=libxup.so
username=ask
password=ask
ip=<IP_ADDRESS>
port=-1
xserverbpp=24
code=20
You can override the common channel settings for each session type
#channel.rdpdr=true
#channel.rdpsnd=true
#channel.drdynvc=true
#channel.cliprdr=true
#channel.rail=true
#channel.xrdpvr=true
=-=-==- Sesman.ini =-=-=-=-=-=
[Globals]
ListenAddress=<IP_ADDRESS>
ListenPort=3350
EnableUserWindowManager=1
UserWindowManager=startwm.sh
DefaultWindowManager=startwm.sh
[Security]
AllowRootLogin=0
MaxLoginRetry=4
TerminalServerUsers=tsusers
TerminalServerAdmins=tsadmins
When AlwaysGroupCheck = false access will be permitted
if the group TerminalServerUsers is not defined.
AlwaysGroupCheck = false
[Sessions]
X11DisplayOffset - x11 display number offset
Type: integer
Default: 10
X11DisplayOffset=40
MaxSessions - maximum number of connections to an xrdp server
Type: integer
Default: 0
MaxSessions=50
KillDisconnected - kill disconnected sessions
Type: integer
Default: 0
if 1, true, or yes, kill session after 60 seconds
KillDisconnected=0
IdleTimeLimit - when to disconnect idle sessions
Type: integer
Default: 0
if not zero, the seconds without mouse or keyboard input before disconnect
not complete yet
IdleTimeLimit=0
DisconnectedTimeLimit - when to kill idle sessions
Type: integer
Default: 0
if not zero, the seconds before a disconnected session is killed
min 60 seconds
DisconnectedTimeLimit=0
Policy - session allocation policy
Type: enum [ "Default" | "UBD" | "UBI" | "UBC" | "UBDI" | "UBDC" ]
Default: Xrdp:<User,BitPerPixel> and Xvnc:<User,BitPerPixel,DisplaySize>
"UBD" session per <User,BitPerPixel,DisplaySize>
"UBI" session per <User,BitPerPixel,IPAddr>
"UBC" session per <User,BitPerPixel,Connection>
"UBDI" session per <User,BitPerPixel,DisplaySize,IPAddr>
"UBDC" session per <User,BitPerPixel,DisplaySize,Connection>
Policy=Default
[Logging]
LogFile=xrdp-sesman.log
LogLevel=DEBUG
EnableSyslog=1
SyslogLevel=DEBUG
[X11rdp]
param1=-bs
param2=-ac
param3=-nolisten
param4=tcp
param5=-uds
[Xvnc]
param1=-bs
param2=-ac
param3=-nolisten
param4=tcp
param5=-localhost
param6=-dpi
param7=96
[Xorg]
param1=-config
param2=xrdp/xorg.conf
param3=-logfile
param4=/dev/null
param5=-noreset
param6=-ac
param7=-nolisten
param8=tcp
[Chansrv]
drive redirection, defaults to xrdp_client if not set
FuseMountName=local_drives
[SessionVariables]
PULSE_SCRIPT=/etc/xrdp/pulse/default.pa

@vincentkersten Debian has xrdp v0.9.1 in their repository now. Can you test again with v0.9.1?

Wow, this is good news!
I'll get back to this , thank you for letting me know!
Vincent

On 19 jan. 2017, at 02:04, metalefty<EMAIL_ADDRESS>wrote:
@vincentkersten https://github.com/vincentkersten Debian has xrdp v0.9.1 in their repository now. Can you test again with v0.9.1?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/neutrinolabs/xrdp/issues/266#issuecomment-273651617, or mute the thread https://github.com/notifications/unsubscribe-auth/AFLYmXMoC18xLNkePfYkvPsiuHqMfMtJks5rTraagaJpZM4FbvKd.

Closing. If this issue is not fixed for you, please feel free to reopen.

Debian package works quite well, i've deployed on a few places.

Even really old drawing methods

the install script could chown /var/run/xrdp
(but that only really is because it took me so long to find it :) )

I have a freeze since two days on the login window on two servers. Not able to type name and password. Funny enough if my rdp client sends my login credentials automatically, i am connected.
Still looking into the issue, but i am not sure what to restart, actually, without affecting the running sessions.

Regards,
Vincent


watch-sources will trigger on hidden .swp files (vim swap files) by default

Vim by default creates hidden swap files for temporary storage and recoverability. It creates these  by default in the file's directory with the name .{filename}.swp ; e.g. Foo.scala will have a .Foo.scala.swp .
sbt's test watcher will trigger on these files by default, so for example everytime I switch buffers a compile/test is triggered - this may be several times in a second if I page through a couple buffers - after awhile bizarre things happen and SBT crashes with an OOM exception. We can work around telling vim to use another directory or just stuff the files, but It might be nice if sbt would ignore hidden files by default, or ignore files with extended name extensions.
Thanks!

@pfn 's solution fails in sbt 1.1.5 (and probably 1.x, based on the docs)
build.sbt:74: error: value isDirectory is not a member of sbt.Watched.WatchSource
watchSources ~= { .filterNot(.isDirectory) }


Cop.java:36-39: Handle suppressed rules. We have to...

The puzzle 110-ea4c930c from #110 has to be resolved:
https://github.com/volodya-lombrozo/jtcop/blob/dc6cd30f3ab35cd898c6c961150b4501b96b3496/src/main/java/com/github/lombrozo/testnames/Cop.java#L36-L39
The puzzle was created by @volodya-lombrozo on 27-Apr-23.
Estimate: 30 minutes,  role: DEV.
If you have any technical questions, don't ask me, submit new tickets instead. The task will be "done" when the problem is fixed and the text of the puzzle is removed from the source code. Here is more about PDD and about me.

@0pdd the puzzle #117 is still not solved.

@0pdd 2 puzzles #125, #126 are still not solved; solved: #117.

@0pdd the puzzle #125 is still not solved; solved: #117, #126.

@0pdd all 3 puzzles are solved here: #117, #125, #126.


New keyword in buildin lib: Repeat Keyword And Ignore Errors

Hello,
We need a keyword which allows us to repeat another keyword in the loop multiple times, let's say 500, and then create small statistics in Robot log (how many failures occurred, how many iterations were made).
It should pass only if all iterations passed
Sample usage:
Repeat Keyword And Ignore Errors    Another Keyword   x500
I think it may be useful also for other test teams, so I don't want to make it only for our own - it's relatively easy to do task.
Do you think that it is possible to provide us such kw?
Regards,
Paweł

Sounds like a conglomeration of the Repeat Keyword and Run Keyword And Ignore Error keywords.  I've never had a need for something like that, but obviously you've found a situation where it's useful.
I'd suggest you create a user-defined keyword that gathers your statistics and loops the keyword, likely easier if you use a 'for' loop internally along with the "Ignore Errors" keyword.  Just a thought.
If other folks would find that useful, they'll weigh in here and a decision can be made about incorporating that via Pull Request.
Good luck with that.

Sorry for not commenting earlier. I've been mainly working with SeleniumLibrary lately and haven't paid too close attention to Robot's tracker. Anyway, I'm not certain is this a generic enough functionality that should be added to the core framework, especially when there are already related features that ought to solve more or less the same use case.
Could you clarify why you couldn't use test templates instead? Or if they won't work for some reason, why couldn't you then use something like this:
Repeat Keyword    500x    Run Keyword And Continue On Failure
...    My Keyword    arg1    arg2

Now new info lately. Closing.


Fix charm building to work with new WheelhouseTactic updates

This fork of the WheelhouseTactic got significantly out of date from the base, and charm-tools PR #585 broke the tactic override mechanism (see https://github.com/juju/charm-tools/issues/596). This addresses both by working around the tactic override issue and bringing the custom tactic into line with the base by making it a subclass.
Fixes #27

Thanks for fixing this!

Thanks for fixing this!


Destination check contains unnecessary duplicate check

see: https://github.com/onelogin/php-saml/blob/570a5d9cc42a1a205355b9212bd182fbe35404d9/lib/Saml2/Response.php#L170-L181
If the check for whether the destination starts with current url fails, destination is checked with the OneLogin_Saml2_Utils::getSelfRoutedURLNoQuery() - however this is the initial value of  the first check (see https://github.com/onelogin/php-saml/blob/570a5d9cc42a1a205355b9212bd182fbe35404d9/lib/Saml2/Response.php#L133)
The repeated check does not appear to be needed.

The second one should be getSelfURLNoQuery


Flow incorrectly(?) assumes static properties are inherited

Flow disallows this program, even though static properties in JS aren't inherited (in contrast with Python class properties):
class Parent {
  static staticProperty: string = '';
}

class Child extends Parent {
  static staticProperty: number = 0;
}

https://flowtype.org/try/#0PTAEAEDMBsHsHcBQBjaBDAzh0AFNAnAUwDsAXUAb0VFA1LVIEtlb6nkd9YAHQ-UgJ4AuVvkbEA5qAC8oAORyA3IgC+iFOiygAwgAtG0ACahCAD1IlD2PETKVqrBs0ftOPPoJHEArgFsARnwyoAAMyipAA
Fixing this however is kind of non-trivial because Flow would need to reject this program:
class Parent {
  static staticProperty: string = '';
  printStaticProperty() {
    const cls = this.constructor;
    console.log(cls.staticProperty.toUpperCase());
  }
}

class Child extends Parent {
  static staticProperty: number = 0;
}

new Child().printStaticProperty();

static properties in JS aren't inherited

Hm... It seems that they are inherited, that's what Chrome console gave to me:

Static fields are inherited because a static field is set on a class object as the last step of class declaration, and a class's __proto__ is === it's parent class object.

That answers this, sorry for the noise.


✨ Use SuperClusterLabelFilter feature gate for all objects if enabled

What this PR does / why we need it:
#279 introduced labelling for all objects, so this PR adds filtering to all checkers where it is possible.
There are couple checkers (priority class, storage class, persistent volume) that are used for back populating only and have nothing to do with virtual clusters initially, so they still have labels.Everything() selector. All other checkers are used for synchronisation and garbage collecting, so labelling should help to reduce control-plane operations and overall controller memory consumption in clusters, where super-cluster is used outside of nested scope.

Thanks for working on this.
/lgtm
/approve


what is the data_name_or_path parameter for?

In the CNN inference code, you set up the bart object with
bart = BARTModel.from_pretrained(
'checkpoints/',
checkpoint_file='checkpoint_best.pt',
data_name_or_path='cnn_dm-bin'
)
what is the data_name_or_path parameter for? It is pointing to the cnn_dm-bin directory. I understand that's where the binarized training data is stored. Why would you need them for inference?
It matters because it is not clear how to translate this for the XSum case. There is no instruction for creating equivalent binary files for XSum.

It's for loading the dictionary files. It's not necessary for that directory to actually contain training data.


Find comment by id

Thank you for the wonderful action, it helps me a lot!
Currently I use it to fetch the body of a comment to be used in my workflow, but I find that I can't get the comment by comment id.
Would you please add the feature so that I can get a comment by comment id?

Hi @plazum
The point of this action is to find the comment ID of a comment that you only know other details about. If you already know the comment ID then you can just use it to get the body.
For example, you can use github-script. I haven't tested the following steps, but something close to this should work.
      - uses: actions/github-script@v5
        id: get-comment
        with:
          script: |
            return github.rest.issues.getComment({
              owner: context.repo.owner,
              repo: context.repo.repo,
              comment_id: ${{ steps.my-step.outputs.comment-id }}
            }).body
          result-encoding: string

      - run: echo "${{ steps.get-comment.outputs.result }}"

Hi @peter-evans, thank you for you kindly reply.
The code you give will work with a little modification as follows:
      - uses: actions/github-script@v5
        id: get-comment
        with:
          script: |
            return (await github.rest.issues.getComment({
              owner: context.repo.owner,
              repo: context.repo.repo,
              comment_id: ${{ steps.my-step.outputs.comment-id }}
            })).data.body;
          result-encoding: string


[C_CPP]: Randomised Quicksort

Description

Probabilistic Analysis
Randomized Quicksort

Domain
C/CPP
Type of Contribution
Documentation
Code of Conduct

[X] I follow Contributing Guidelines & Code of conduct of this project.

/assign


raiseskill command is broken

When trying to use the raiseskill <skill> <amount> command on the current stable release with Server Devcommands v1.36.0, I get the following error:
[Error  : Unity Log] MissingMethodException: void Skills.CheatRaiseSkill(string,single,bool)
Stack trace:
ServerDevcommands.Helper+<>c__DisplayClass20_0.<Catch>b__0 (Terminal+ConsoleEventArgs args) (at <b083ff1669544cb98d5a064b0c1a519b>:0)
(wrapper dynamic-method) Terminal+ConsoleCommand.DMD<Terminal+ConsoleCommand::RunAction>(Terminal/ConsoleCommand,Terminal/ConsoleEventArgs)
(wrapper dynamic-method) Terminal.DMD<Terminal::TryRunCommand>(Terminal,string,bool,bool)
(wrapper dynamic-method) Terminal.DMD<Terminal::InputText>(Terminal)
Terminal.SendInput () (at <cdb5bbec33f94182b860ca208c683122>:0)
(wrapper dynamic-method) Terminal.DMD<Terminal::UpdateInput>(Terminal)
Terminal.Update () (at <cdb5bbec33f94182b860ca208c683122>:0)
Console.Update () (at <cdb5bbec33f94182b860ca208c683122>:0)

It's not just this command. Had the same issue with spawn command as well. I'd guess it's pretty much anything with a string for an argument. Version 1.34 seems to work, but not 1.36. Haven't tested the version for Mistlands yet.

I have tried but I'm not able to get any issues with the one from Thunderstore.

Probably fixed now.


fix sbt script

cc @falaki
The typesafe repositories changed.

Current coverage is 85.01%

Merging #126 into master will not affect coverage as of 82d22fd

@@            master    #126   diff @@
======================================
  Files           10      10
  Stmts          407     407
  Branches       125     125
  Methods          0       0
======================================
  Hit            346     346
  Partial          0       0
  Missed          61      61

Review entire Coverage Diff as of 82d22fd

Powered by Codecov. Updated on successful CI builds.


Add endpoint to list all consumers using key-auth plugin and their keys

Summary
Adding an endpoint to list all the keys of all consumers.
This is an initial commit to get better feedback from the community.

The endpoint could be changed to something else.
Grouping of multiple keys belonging to the same consumer is something I'm considering
A similar thing could be done for the ACL plugin.

Tests are missing right now.
Full changelog

Implement an endpoint GET /plugins/key-auth/consumers to list all the consumers with their key(s).

Grouping of multiple keys belonging to the same consumer is something I'm considering

That'd be great too! The endpoint could relate to the one I suggested in my review, which would get us 3 new endpoints:
/key-auth/
/consumers/key-auth/
/consumers/key-auth/:credential_key_or_id

A similar thing could be done for the ACL plugin.

We are also interested similar endpoints for the ACL, basic-auth, hmac and JWT plugins if they do not exist yet! But let's process step-by-step. We should keep the scope of this PR limited to the key-auth plugin only :) We will gladly accept more contributions for other credentials-based plugins once we're square on this first plugin.

Tests are missing right now.

Great!

@thibaultcha Thanks a lot for the feedback.

/key-auth/ endpoint to list all the key-auth credential makes a lot of sense really since what it returns is key-auths. Other auth plugins could have similar GET all endpoints. I'll get this one done first.
The endpoint you metion/consumers/key-auth/:credential_key_or_id to retrieve a consumer for a given key-credential is a very useful endpoint to have.
However, consider /consumers/:consumer_id_or_name/key-auth, which returns all the key-auths for the specific consumer. To maintain consistency, we can instead have an endpoint /key-auth/: credential_key_or_id/consumer to retrieve an consumer for a key or key_id.

Thoughts?
I'll try to get this into a good shape and once we're done here, we can take on other plugins as well.

@hbagdi Yes, sorry, I meant /consumers/:consumer/key-auth/. I updated my comment to reflect this. Your proposal resonates to me. If I understand correctly, this would add the following two endpoints to our key-auth plugin:
1. /key-auth/
2. /key-auth/:credential_key_or_id/consumer

List all key-auth credentials in a paginated fashion. It's also a way to retrieve all credentials belonging to a Consumer, via filtering (/key-auth?consumer_id=...).
In a single call, retrieve the Consumer associated with a given key-auth credential.

Am I missing anything?

Sounds good to me.
So, this PR will add the following:

/key-auth/
/key-auth/:credential_key_or_id/consumer
Tests for 1 and 2

I'm done with 2 and half of 1 (need filtering).
I'll add tests and let you know.
Thanks for being prompt!

@thibaultcha I've updated the PR to reflect our discussion.

@thibaultcha I've added and corrected the tests.

@thibaultcha having the endpoint as /key-auths makes more sense?

having the endpoint as /key-auths makes more sense?

Hmmm yeah, probably. This would stay consistent with the /acls endpoint proposed in #2371. Unfortunately our existing key-auth endpoints use a singular form (/consumers/:username_or_id/key-auth/), but let's try to avoid this mistake again.
If you update this PR to use the plural form you suggested, I don't see any further blockers to merge it. Thanks!

@thibaultcha I'll send PRs for similar endpoints for JWT and hmac soon then.

Manually merged to master with an updated commit message and some minor edits, including a new test ensuring the /key-auths endpoint is paginated :)
This is awesome, thank you very much for giving a stab at those! Onwards to JWT, basic-auth, and acls :wink:

@hbagdi Oh, I was almost forgetting: will you also contribute the appropriate documentation on our open source docs repository: https://github.com/Kong/getkong.org/ ? Would be greatly appreciated!
You can find the key-auth plugin documentation here: https://getkong.org/plugins/key-authentication/ with, in the sidebar, some anchors that lead to various sections of the page describing the additional endpoints provided by this plugin. Maybe 2 more sections would be appropriate here:

Browse all key-auth credentials
Retrieve the Consumer associated with a key-auth credential

We do value complete contributions including code, tests and documentation :)

@thibaultcha I did have documentation on mind. I'll send out a PR soon.


Practice session: Choose from one of our suggested open source examples, run through it, and relate it to the material in this course.

Closed by #183


setChoices() : populate choices without callback system of ajax()

First, thanks for 'choices' option
I made a function to do the same with ajax() but without callback.
Can you watch it ?
Here it is : https://github.com/chymz/Choices/commit/02a71448cab38364f5c0555675507ad6a945f80c
Feel free to close if you don't want it ;)

This has now been implemented into version 1.0.3 (https://github.com/jshjohnson/Choices#setchoiceschoices-value-label)
Thanks!


Fix Issue 15405 - FormatSpec.writeUpToNextSpec() not documented

The style checker gives the following error, that I don't understand, because the normal unittests can compile this. Can anyone help me here?
generated/linux/release/64/publictests/std_format.d(1242): Error: template std.format.FormatSpec cannot deduce function from argument types !()(string), candidates are:
std/format.d(1011):        FormatSpec(Char)
  with Char = char
  must satisfy the following constraint:
       !is(Unqual!Char == Char)
std/format.d(1021):        FormatSpec(Char)

Meanwhile I found out, that I had to add the template parameter dchar explicitly in the example. But I still don't understand, why I have to do so in a public example, while the other unittests work without...

There are still linker errors when I run the style checker - similar to buildkite/phobos: style. I have no idea, what goes wrong here...

I noticed, that the public unittest already existed outside of the template. So I just added a link in the docs and changed that unittest a little bit to show how the return value works. (And I replaced %2.4f by %6.4f, because 2.4 is a typical fallacy when using format.)

Meanwhile I found out, that I had to add the template parameter dchar explicitly in the example. But I still don't understand, why I have to do so in a public example, while the other unittests work without...

Probably because there is no template interference for constructors IIRC (albeit it should probably result in a better error message)

Probably because there is no template interference for constructors

It's because of the documented unit test being inside the template, in that case FormatSpec is the instantiated struct.
But when it gets copied outside of the template to make sure the public example works, it needs the instantiating parameter.

The test seemed to have failed for an environmental issue. I'm re-running it.

Question: should writeUpToNextSpec even be a public function in the first place?  It looks more like implementation detail to me than a public function that must be documented, even though apparently it has never been marked private.

It was in the original docs for FormatSpec (in fact, this PR barely changes any unittests at all, just verifies the value is correct). I'm not 100% familiar with the type, but I'm guessing if you wanted to use FormatSpec outside of std.format, you would need this function, right?

Question: should writeUpToNextSpec even be a public function in the first place? It looks more like implementation detail to me than a public function that must be documented, even though apparently it has never been marked private.

That was my first thought too. But there might be usecases for this struct (and than you need these function as it is somewhat the heart of this struct): Think of an export function, where the user can specify the order and the appearence of the exported items. This function could be used to check the userinput by parsing it.

I'm also guessing that if DIP 1027 is approved, this will become an important struct for 3rd party libraries.


Are the status labels still needed?

What feature would you like to have?
We have some status:<state> labels across the repo which are automatically assigned when certain actions happen (issue created, PR created, etc).
Since we have the same info in GitHub Projects, are those labels still necessary? It seems like an info duplication to me, but maybe there is a reason to keep them that I'm missing.
/cc @HonkingGoose
What problem will this feature solve?
N/A
Do you want to help make the feature?
Yes
Is there anything else we need to know?
No response

I basically copied the status:* labels from Renovate. But now that we're using the project boards, we can drop them if you want. 😉

I'm tracking the projects and tickets state through GH Projects, so the labels are not that useful for me anymore.  If that's the case for you too, I'll proceed to delete such labels. Can you confirm this?

Yes, you can delete the labels you don't like anymore. 😉


candmachinehelper problem

This library really drove me crazy. It says publicsale:undifined and aptos shows as NaN and the mint key is not active. Where does the candymachine address in CandyMachineInfo.js come from?

This library really drove me crazy. It says publicsale:undifined and aptos shows as NaN and the mint key is not active. Where does the candymachine address in CandyMachineInfo.js come from?

As soon as you run python3 cli.py it says output like:
=== Preparing Candy Machine account ===
Public key: xxx

The public key is your candy machine address


add extract-cache and save-cache commands

Initial implementation of the cache commands for npm-ci.
Assumes AWS credentials exist on the build agent (works the same for other CI builds that use s3).
Design decisions:

Based on .ci_config. We can come up with some heuristic on how to get the paths that need to be cached. I'd rather start with explicitness and if we are building upon the .ci_config file, I guess that is a good place to put it. (under cache.paths).
The default cache key is going to be ORG_REPO_BRANCH. That means the cache is shared per branch build inside a repo. In pull requests, that means the first build of the branch will not have a cache, but subsequent builds will. (Maybe take the first cache from master? Check if the cache for the branch exists and fallback to master (and have master hard-coded as a base branch)? need to think about it). On long-lived branches, most of the builds will have the cache.

I understand you plan to use these commands from the npmBuild.sh file?
Note that you will need to add it in multiple places. Also a long term goal, in my opinion, is to reduce the amount logic inside the shell script.
A better option in my opinion would be to put it as part of the install command in npm-ci-scripts. WDYT?

At first I thought about just wrapping the install command. But looking at the implementation of other CI providers, and a bit common sense, I think we need to save the cache only if the build succeeded.
I think the ideal solution would have to put in the build step in the CI, as an after success or similar (not sure if it's available). But since we don't want to require a change to the CI template and have control over it, I think I'll just do it in the build command, which seems to be the one that runs the main process of install -> build -> test, and I'll change it to extract-cache -> install -> build -> test -> save-cache.


adding name to contributors list

adding elaine wong to contributors  list for pandas mentoring.

Hi Elaine! This looks great.
Would you mind to add the link to our wiki somewhere in the README (perhaps right after "the list is not yet complete...") in a separate PR, so our wiki can get more visibility? Maybe you can say something along the line, "if you want to know how to contribute, check out our wiki" or something else - it's up to you :) Thank you!

Thanks @elainewong good job so far.
Please take a look at the diff of your PR: https://github.com/python-sprints/pandas-mentoring/pull/52/files
What would be ideal is to only have your username being added in this PR. And then in a different PR the link to the wiki. Thanks!


date range queries return wrong number of records in the database

I am using Mongodb 2.4.8, Mongoid 4.0.2 and Rails 4.2.1
I have been trying to get a date range query ran through mongoid to return the right set of results on a test data with mongodb. I created just 6 documents spanning over 5 - 7 days period. For instance, the query below, ought to return 3 records for the days between 22nd - 27th, instead it returns one record :
Person.where(:person_email_clicks.elem_match => {:date.gte =>  'Wed, 22 Apr 2015 22:12:00 +0000'}, :person_email_clicks.elem_match => {:date.lte => 'Mon, 27 Apr 2015 22:12:00 +0000'})

This is the entire documents for test purposes:
{"_id"=>BSON::ObjectId('55364a416461760bf1000000'),
   "status"=>"premium"
   "person_email_clicks"=>[
      {"_id"=>BSON::ObjectId('55381d256461760b6c000000'), "name"=>"buyer", "date"=>2015-04-22 22:12:00 UTC},

      {"_id"=>BSON::ObjectId('55381d536461760b6c010000'), "name"=>"seller", "date"=>2015-04-23 01:12:00 UTC},

      {"_id"=>BSON::ObjectId('55381d916461760b6c020000'), "name"=>"giver", "date"=>2015-04-24 22:12:00 UTC},

      {"_id"=>BSON::ObjectId('55381dac6461760b6c030000'), "name"=>"help", "date"=>2015-04-27 22:12:00 UTC},

      {"_id"=>BSON::ObjectId('553824ba6461760b6c040000'), "name"=>"tt", "date"=>2015-04-22 22:12:00 UTC},

      {"_id"=>BSON::ObjectId('553824bf6461760b6c050000'), "name"=>"tt", "date"=>2015-04-22 22:12:00 UTC},

      {"_id"=>BSON::ObjectId('55382ad46461760b6c060000'), "name"=>"yyy", "date"=>2015-04-25 22:12:00 UTC}
    ]
}

Here are the models:
class Person
   include Mongoid::Document
   field :status,  type: String
   embeds_many :person_email_clicks
end

class PersonEmailClick
    include Mongoid::Document

    field :name, type: String
    field :date, type: DateTime
    embedded_in :person
end

These are the queries, I have tried so far and they all return 1 record instead of 3 records between the dates 22nd and 27th
 u = Person.first.person_email_clicks.first.date
 We store the datetime which is Wed, 22 Apr 2015 22:12:00 +0000

 e = Person.first.person_email_clicks.last.date
 We store the the 2nd date which is Mon, 27 Apr 2015 22:12:00 +0000

Now I use those variables holding the dates in my query and all 3 query below returns one record instead of 3 records:
 f = Person.where(:person_email_clicks.elem_match => {:date.gte =>  u}, :person_email_clicks.elem_match => {:date.lte => e})

 f = Person.where(:person_email_clicks.elem_match => {:date => {"lte" =>  u}}, :person_email_clicks.elem_match => {:date => {"gte" => e}})

 t = Person.where('person_email_clicks.date' => u..e)

Any suggestions on how to tweak this to return the 3 records between the date range of 22nd - 27th ?

This has finally been resolved. I needed to stored the date in a variable and then query it this way:
Person.first.person_email_clicks.gt(date: u).lt(date: e).

This is the full step to get the desired results:
u = Person.first.person_email_clicks.first.date
   => Wed, 22 Apr 2015 22:12:00 +0000

e = Person.first.person_email_clicks.last.date
   => Sat, 25 Apr 2015 22:12:00 +0000

h = Person.first.person_email_clicks.gt(date: u).lt(date: e).to_a

Which returned 2 records as shown below:
 => [
      #<PersonEmailClick _id: 55381d536461760b6c010000, name: "seller", date: 2015-04-23 01:12:00 UTC, daily_total: 0, email_open: 0, page_views: 0, clicks: 0, visits: 0>,

      #<PersonEmailClick _id: 55381d916461760b6c020000, name: "giver", date: 2015-04-24 22:12:00 UTC, daily_total: 0, email_open: 0, page_views: 0, clicks: 0, visits: 0>
 ]


Solana: add deps info

❓ Context

Impacted projects: []

Linked resource(s):

Add explanation why solana web3.js library is fixed at a version.
✅ Checklist

[ ] Test coverage: Did you write any tests to cover the changes introduced by this pull request?
[ ] Atomic delivery: Is this pull request standalone? In order words, does it depend on nothing else?
[ ] No breaking changes: Does this pull request contain breaking changes of any kind? If so, please explain why.

📸 Demo

🚀 Expectations to reach

@haammar-ledger


Fixed issue #1511

Pull Request Type
What kind of change does this PR introduce?
[x] Solved an issue/task
[ ] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Build related changes
[ ] CI related changes
[ ] Documentation content changes
[ ] Other... Please describe:
What is the current behavior?

Reading a csv file using C.
Issue Number: [Insert Issue Number (if applicable)]
Issue (C) #1511.
What is the new behavior?

Reading the csv file and printing on humam readable language.
Other information

Merged

This is an automated message from Fork, Commit, Merge [BOT].

Thank you for your contribution! Your pull request has been merged. The files have been reset for the next contributor.
What's next?
If you're looking for more ways to contribute, I invite you to check out our forkcommitmerge.io website repository. The repository contain real issues that you can help to resolve or create some of your own if you see any bugs or want to add new features to the site, etc. You can also check out the Influences section in the README to find more projects similar to this one.
Also please leave a star to this project if you feel it helped you, i would really appreciate it.
I look forward to seeing your contributions!


tag: add .ping

Add .ping, that's .tagged, except without name mangling.

Nah. Let's just use remember for that if people want to spam.


feat: make pandas and NumPy optional dependencies, don't require PyArrow for plotting with Polars/Modin/cuDF

Suggested in https://github.com/vega/altair/issues/3445#issuecomment-2194585068
closes #3456
closes #3445
Demo
Polars can work without PyArrow nor pandas installed (EDIT: and without NumPy either!):

Pyarrow table is still supported
(notice the .to_arrow)

Ibis is still supported

Performance
There's also a perf improvement to not repeatedly converting Polars to PyArrow - for example, if I run
import polars as pl
import altair as alt
from datetime import date

# dataset: https://github.com/Mcompetitions/M6-methods/blob/main/assets_m6.csv
df = pl.read_csv('../scratch/assets.csv', try_parse_dates=True)
df = df.filter(pl.col('date')>date(2022, 11, 15))
def func():
    alt.Chart(df).mark_line().encode(
        x='date', y='price', color='symbol',
    ).to_json()
results = %timeit -o -q func()
results.best

in IPython, then I see:

Altair main branch: 0.033584889399935494
Altair MarcoGorelli:narwhalify branch: 0.024283270300657023

That's for a (4475, 3)-shaped dataframe - for dataframe with many columns (esp. with strings / categoricals) I'd expect the difference to grow

@MarcoGorelli if all the tests are passing locally for you, try running these two:
hatch run generate-schema-wrapper
hatch run update-init-file

If there are any changes other than whitespace, you'll get a more informative error than https://github.com/vega/altair/actions/runs/9733301918/job/26860053484?pr=3452
Otherwise those two should clean everything up

https://github.com/vega/altair/actions/runs/9734437115/job/26862450764?pr=3452
@MarcoGorelli try this to run the tests on the matrix
hatch test --all

fix: add missing pandas import to doctest
519     Examples 520     -------- 521     >>> data = pd.DataFrame({'foo': ['A', 'B', 'A', 'B'], UNEXPECTED EXCEPTION: NameError("name 'pd' is not defined") Traceback (most recent call last):   File "..\Local\hatch\env\virtual\.pythons\3.12\python\Lib\doctest.py", line 1361, in __run           exec(compile(example.source, filename, "single",   File "<doctest altair.utils.core.parse_shorthand[0]>", line 1, in <module> NameError: name 'pd' is not defined C:\Users\danie\Documents\GitHub\altair\altair\utils\core.py:521: UnexpectedException ============================================== short test summary info ==============================================  FAILED altair/utils/core.py::altair.utils.core.parse_shorthand ========================= 1 failed, 1250 passed, 1 skipped, 2 xfailed, 1 xpassed in 30.64s ==========================

diff --git a/altair/utils/core.py b/altair/utils/core.py
index e6620146..d5d753db 100644
--- a/altair/utils/core.py
+++ b/altair/utils/core.py
@@ -520,6 +520,7 @@ def parse_shorthand(

     Examples
     --------
+    >>> import pandas as pd
     >>> data = pd.DataFrame({'foo': ['A', 'B', 'A', 'B'],
     ...                      'bar': [1, 2, 3, 4]})

refactor(typing): Reuse _typing alias as InferredVegaLiteType
diff --git a/altair/utils/core.py b/altair/utils/core.py
index e6620146..f3a5469a 100644
--- a/altair/utils/core.py
+++ b/altair/utils/core.py
@@ -42,6 +42,7 @@ else:
 if TYPE_CHECKING:
     from types import ModuleType
     import typing as t
+    from altair.vegalite.v5.schema._typing import StandardType_T as InferredVegaLiteType
     from altair.utils._dfi_types import DataFrame as DfiDataFrame
     from altair.utils.data import DataType
     from narwhals.typing import IntoExpr
@@ -199,9 +200,6 @@ TIMEUNITS = [
 ]

-InferredVegaLiteType = Literal["ordinal", "nominal", "quantitative", "temporal"]
-
-
 def infer_vegalite_type_for_pandas(
     data: object,
 ) -> InferredVegaLiteType | tuple[InferredVegaLiteType, list[Any]]:

@MarcoGorelli these were two minor things. I couldn't figure out a way to request these another way.
fix: add missing pandas import to doctest
519     Examples 520     -------- 521     >>> data = pd.DataFrame({'foo': ['A', 'B', 'A', 'B'], UNEXPECTED EXCEPTION: NameError("name 'pd' is not defined") Traceback (most recent call last):   File "..\Local\hatch\env\virtual\.pythons\3.12\python\Lib\doctest.py", line 1361, in __run           exec(compile(example.source, filename, "single",   File "<doctest altair.utils.core.parse_shorthand[0]>", line 1, in <module> NameError: name 'pd' is not defined C:\Users\danie\Documents\GitHub\altair\altair\utils\core.py:521: UnexpectedException ============================================== short test summary info ==============================================  FAILED altair/utils/core.py::altair.utils.core.parse_shorthand ========================= 1 failed, 1250 passed, 1 skipped, 2 xfailed, 1 xpassed in 30.64s ==========================

diff --git a/altair/utils/core.py b/altair/utils/core.py
index e6620146..d5d753db 100644
--- a/altair/utils/core.py
+++ b/altair/utils/core.py
@@ -520,6 +520,7 @@ def parse_shorthand(

     Examples
     --------
+    >>> import pandas as pd
     >>> data = pd.DataFrame({'foo': ['A', 'B', 'A', 'B'],
     ...                      'bar': [1, 2, 3, 4]})

refactor(typing): Reuse _typing alias as InferredVegaLiteType
diff --git a/altair/utils/core.py b/altair/utils/core.py
index e6620146..f3a5469a 100644
--- a/altair/utils/core.py
+++ b/altair/utils/core.py
@@ -42,6 +42,7 @@ else:
 if TYPE_CHECKING:
     from types import ModuleType
     import typing as t
+    from altair.vegalite.v5.schema._typing import StandardType_T as InferredVegaLiteType
     from altair.utils._dfi_types import DataFrame as DfiDataFrame
     from altair.utils.data import DataType
     from narwhals.typing import IntoExpr
@@ -199,9 +200,6 @@ TIMEUNITS = [
 ]

-InferredVegaLiteType = Literal["ordinal", "nominal", "quantitative", "temporal"]
-
-
 def infer_vegalite_type_for_pandas(
     data: object,
 ) -> InferredVegaLiteType | tuple[InferredVegaLiteType, list[Any]]:

Thank you very much for making this Pull Request. That is very much appreciated!
This PR introduces a new dependency on narwhals. Narwhals is described as a lightweight and extensible compatibility layer between multiple dataframe libraries.
Narwhals can replace our hard dependency on pandas, the size of pandas is ~11 MB (without mentioning its subdependencies). This by itself is a very interesting development already.
Next to this it enables support for native dataframes (pandas still included) without converting these to a pyarrow Tables through the dataframe interchange protocol.
The latest release of Altair is ~850 kB and after this PR, Altair will have the following hard dependencies:

jinja2 : ~130 kB
jsonschema : ~90 kB
typing_extensions: ~35 kB
packaging : ~55 kB
narwhals : ~80 kB
numpy : ~18 MB

When using Altair with only remote endpoints (like APIs that are returning JSON-data), I've the feeling that we do not need to rely on numpy. Therefor I'm still hopefull that we can reduce this hard dependency on numpy as well one day.
I've a few questions regarding this PR and narwhals.

Just curious, why is the lower limit of pandas in narwhals version 1.1.5? As the supported lower limit of pandas in Altair before this PR was version 0.25.

You serialize the nw.Date type to:

if dtype == nw.Date:
	columns.append(nw.col(name).dt.to_string("%Y-%m-%d"))

And change a test from:
"a": {"date": "2019-01-01T00:00:00"},

To:
"a": {"date": "2019-01-01"},

Upon first sight, I think we cannot introduce this breaking change. Does this change still adhere to full ISO-8601 dates representation as is required by Javascript? See also past issues like this: https://github.com/vega/altair/issues/1027.

Can you explain the behavior of nw.from_native()?
When I look to this code:

data = nw.from_native(data, eager_only=True, strict=False)
if isinstance(data, nw.DataFrame):
	return data
if isinstance(data, DataFrameLike):
	from altair.utils.data import arrow_table_from_dfi_dataframe

	pa_table = arrow_table_from_dfi_dataframe(data)
	return nw.from_native(pa_table, eager_only=True)

I'm a bit surprised that the data object, as a result of passing it into nw.from_native can be a non-narwhals dataframe. If it is not supported by narwhals it returns the original data object?

Maybe a naive question, but do you have any intention to support dataframes that support the dataframe interface protocol within narwhals?

Not being a type expert, but I cannot really understand why there needs to be quotes around the type "IntoDataFrame", like in:

DataType: TypeAlias = Union[
    Dict[Any, Any], "IntoDataFrame", SupportsGeoInterface, DataFrameLike
]

I remember I have discussed this with @dangotbanned recently in https://github.com/vega/altair/pull/3431, see https://github.com/vega/altair/pull/3431#issuecomment-2162877321 onwards, but I can't find a clear answer anymore on this topic.
Thanks again for this PR! Really promising! (also this PR will supersede https://github.com/vega/altair/pull/3384)

5. Not being a type expert, but I cannot really understand why there needs to be quotes around the type `"IntoDataFrame"`, like in:

DataType: TypeAlias = Union[
    Dict[Any, Any], "IntoDataFrame", SupportsGeoInterface, DataFrameLike
]

I remember I have discussed this with @dangotbanned recently in #3431, see #3431 (comment) onwards, but I can't find a clear answer anymore on this topic.

Thanks for the ping @mattijn
The good news is you can trust the ruff config now for managing cases like this (particularly TCH), what @MarcoGorelli has written is correct.
https://github.com/MarcoGorelli/altair/blob/2df9422dca75a26eb16fd0c8030b6e89b4d4fa0f/altair/utils/data.py#L44-L66
All of the imports within the TYPE_CHECKING block must be quoted when used in a runtime alias definition. IntoDataFrame is used in: DataType, TIntoDataFrame, SampleReturnType.
AFAIK, this should be fine to move to a regular import though

alt.utils.data requires explicit exports in alt.utils.__init__.__all__
narwhals is already imported at runtime in https://github.com/MarcoGorelli/altair/blob/2df9422dca75a26eb16fd0c8030b6e89b4d4fa0f/altair/utils/data.py#L26

Thanks so much for your quick and careful reviews @mattijn and @dangotbanned  🙏 ! Much appreciated

No reason :) I just lowered the minimum version of pandas in Narwhals - I set it to 0.25.3 rather than 0.25 though, as when I tried using 0.25 in CI it just froze indefinitely. 0.25.3 is also the one that Altair is currently testing against in CI - is it OK to set that as the minimum?

Interesting, thanks, and I'm sorry for not having picked up on that! Quoting the issue

If you were east of London when running this code, you wouldn't see this issue 😄.

that might be why I didn't see it when testing it out 😳 .
Polars doesn't let us format Dates with time directives (because the underlying Rust crate chrono doesn't either) - but, we can just cast to datetime, I've added a comment about why this is done. On my laptop, pyarrow.compute.strftime is about 200 times slower than casting Date to Datetime - so, the extra cast to ensure consistency, in the grand scheme of the things, is quite negligible, and not the bottleneck of the sanitize function

That's because of strict=False in the first line of the function https://narwhals-dev.github.io/narwhals/api-reference/narwhals/#narwhals.from_native, the idea is:

strict=False: if the object isn't supported by Narwhals, then it just gets passed through as-is
strict=True: if the object isn't supported by Narwhals, then raise

We'd like to have 2 levels of supported dataframes in Narwhals:

full API
basic. I.e. an extended and more user-friendly version of the dataframe interchange protocol. Anything which already supports the interchange protocol would be in scope. My main gripe with the interchange protocol is that it's bundled with dataframe libraries themselves, which means is super slow to change - for example, Jon reported https://github.com/pandas-dev/pandas/issues/54239, and it got fixed, but it's going to be several years until the minimum pandas version gets bumped up enough that it can be used... 😭

thanks @dangotbanned for helping out with this one!

I've the feeling that we do not need to rely on numpy

"in for a penny, in for a pound" - I've added a commit to do this. NumPy is used very little, mostly isinstance checks and np.ceil, so I think it can already be made optional. Then Altair would truly be the lightest visualisation library in the ecosystem?
I've caught up with Polars devs, and they're on board with using Altair in polars.DataFrame.plot if the plots can be done directly without going via pandas, so we may want to have a conversation later about how that should look (personally, I think df.plot.line(x='x', y='y') as a shorthand for alt.Chart(df).mark_line().encode(x='x', y='y').interactive() would be really nice, and users would ❤ love ❤  it)

Thanks @MarcoGorelli
When I originally suggested narwhals I didn't expect you'd take an active role - and this PR is certainly far beyond that!
I get the impression that this is beneficial for both projects, as any issues you've encountered have been quickly resolved in subsequent narwhals releases

I've caught up with Polars devs, and they're on board with using Altair in polars.DataFrame.plot if the plots can be done directly without going via pandas, so we may want to have a conversation later about how that should look (personally, I think df.plot.line(x='x', y='y') as a shorthand for alt.Chart(df).mark_line().encode(x='x', y='y').interactive() would be really nice, and users would ❤ love ❤ it)

I think most people would agree on the value of this, but probably best for another issue.
@joelostblom would probably be able to chime in, there was also this discussion https://github.com/vega/altair/discussions/2789#discussioncomment-4579799
I would expect there would be more support if the implementation was different to the existing hvPlot accessor.
Currently, the burden would be on altair to define what kinds of charts (e.g. line, scatter, etc), but this isn't a 1-to-1 with how many of the other plotting libraries work.
My suggestion would be for polars to define a protocol with some chart methods, with altair implementing those in an accessor.
That way, these methods/functions may have less of a chance of leaking out into the rest of the altair api.

Thanks @dangotbanned !

I would expect there would be more support if the implementation was different to the existing hvPlot (polars) / hvPlot accessor.

Definitely - plot is marked as unstable in Polars 😎 What I had in mind was just something like
class PlotNameSpace:
    def __init__(self, df: DataFrame) -> None:
        self.df = df
    def line(self, *args, **kwargs):
        return alt.Chart(self.df).mark_line().encode(*args, **kwargs).interactive()
    def point(self, *args, **kwargs):
        return alt.Chart(self.df).mark_point().encode(*args, **kwargs).interactive()
    ...

Anyway, we can figure out the details later, whether or not Altair decides to go ahead with Narwhals
(for the record, I think hvplot is amazing and its developers are really friendly, so I'd still advocating for it to be advertised in the Polars docs - but if Altair could provide a more lightweight alternative which doesn't require other heavy libraries, I think it'd be a more suitable default)

Really awesome to see you were able to remove numpy as a hard dependency too! Its great to see this happening!
A few more comments while you might still have this PR open.
In my opinion, there is no need to serialize microseconds for type Date (also reducing characters in the vega-lite specification). If we define local_iso_fmt_string = "%Y-%m-%dT%H:%M:%S" and use that when nw.Date applies and when nw.Datetime applies we can use f"{local_iso_fmt_string}%.f" or something similar.
A single comment like # using `strict=False` will return the `data` as-is if the object cannot be converted. would be appreciated for my future-self or other people who will be reading the following line:
https://github.com/vega/altair/blob/7312f25eea31b6c86e273d591641b6c2fb494e3b/altair/utils/core.py#L477
This line can be removed:
https://github.com/vega/altair/blob/7312f25eea31b6c86e273d591641b6c2fb494e3b/altair/utils/core.py#L451
Thanks again for pushing this!

A bit off topic, but regarding polars.DataFrame.plot functionality: (1) you also might learn some lessons from this archived repo: https://github.com/altair-viz/pdvega, (2) there is also some work underway to extend the .interactive() method within altair https://github.com/vega/altair/pull/3394.

This is exciting! :) I'd also like to have a quick look. Hopefully get to it this weekend. @jonmmease if you have a moment you might want to have a look as well as you've worked on the interchange protocol implementation and hence might be aware of edge cases we need to consider.

I'm very interested in this PR, but I'm on holiday and won't have time to look at it until mid next week

thanks for your lovely reviews!
I've updated to:

use the narwhals.stable.v1 (see here for the stable api policy)
Polars is in the dev section of pyproject.toml, whilst pandas and NumPy in all (but pandas is also in dev as per https://github.com/vega/altair/pull/3452#discussion_r1664492208)

@dangotbanned, I totally agree agree that you have been going very deep in the altair code recently! Which is very much appreciated. If being a maintainer would make it more easy for you to contribute more to altair, I would like to propose you to become an official maintainer.

@MarcoGorelli
FYI, I think you should hold off on updating this branch until https://github.com/vega/altair/pull/3475 is resolved, to avoid getting an unrelated mypy error

Thanks @jonmmease for your careful review, really appreciate it! 🙏

But when the "vegafusion" data transformer is enabled, VegaFusion is able to detect which columns are needed and it will request only the required columns when loading into pyarrow (See here).

Aah, that's amazing! I'd missed that. Maybe polars.LazyFrame should also implement the interchange protocol just at the metadata level. That's one for a separate discussion anyway
To address this here, we've made a new release of Narwhals which includes interchange-level support - that is to say, if the input object implements the interchange protocol, then we let you inspect the schema
Like this, the code complexity goes down, and we fix the bug Ibis users would have encountered whereby Date dtype wasn't supported (unless they use the vegafusion transformer, in which case, things are the same for them as they are with the latest Altair release). So, this PR improves things for Ibis users using the default data transformers, and preserves the status-quo for those using the vegafusion data transformer 😇

Thanks for the changes @MarcoGorelli!
I still have one question which I cannot really understand. Given the following example:
import datetime
import polars as pl
import pyarrow as pa
import narwhals as nw

data_type_date = [{"date": datetime.date(2024,7,15)}]
local_iso_fmt_string = "%Y-%m-%dT%H:%M:%S"
nw_selector = [nw.col('date').cast(nw.Datetime).dt.to_string(local_iso_fmt_string)]

pl_df = pl.DataFrame(data_type_date)
nw_pl_df = nw.from_native(pl_df, eager_only=True)
nw_pl_df.select(nw_selector).rows(named=True)

This returns

[{'date': '2024-07-15T00:00:00'}]

But if I do:
py_tb = pa.Table.from_pylist(data_type_date)
nw_py_tb = nw.from_native(py_tb, eager_only=True)
nw_py_tb.select(nw_selector).rows(named=True)

It returns

[{'date': '2024-07-15T00:00:00.000000'}]

I expected these output to be equal.

Yes for type nw.Datetime it works correctly. The question I have is about the nw.Date  type.
One change with your latest comment compared to my comment above is that for me the local_iso_fmt_string does not contain a %.f.
I thought it was an issue with pyarrow, but if I do:
import pyarrow.compute as pc
pc.strftime(py_tb['date'], format=local_iso_fmt_string).to_pylist()

It returns

['2024-07-15T00:00:00']

That seems right. And in the narwhals docs you link to (thanks for the link!) it is mentioned that:

for PyArrow, we replace "%S.%f" with "%S".

That makes me even more confused with your statement:

Practically speaking, if '2024-07-15' gets transformed to

'2024-07-15T00:00:00' for Polars and pandas-like libraries, and
'2024-07-15T00:00:00.000000' for PyArrow

Thanks for answering so promptly! Appreciated.

The difference is the .cast(nw.Datetime) 😉 If you start with a datetime, then PyArrow's strftime outputs fractional seconds even if we just specify '%S':

In [1]: import pyarrow as pa

In [2]: from datetime import date, datetime

In [3]: import pyarrow.compute as pc

In [4]: pc.strftime(pa.array([datetime(2020, 1, 1)]), '%Y-%m-%dT%H%:%M%:%S')
Out[4]:
<pyarrow.lib.StringArray object at 0x7fd513149cc0>
[
  "2020-01-01T00%:00%:00.000000"
]

Interesting... Let me see if I can find something on this at the pyarrow library. Thanks!

We have to cast the nw.Date type to nw.Datetime only because polars does not support time directives isn't it? Maybe we check upfront if we have to cast or not. @jonmmease, how important is this? Do you have some suggestions here. Its about this piece:
https://github.com/vega/altair/blob/8b4b3dbeadbe4459eecbe0ad4459e6fb701dcdfe/altair/utils/core.py#L451-L456

Sure - if you're OK with a teeny-tiny bit of Polars-specific logic, this can work 👍
        if dtype == nw.Date and nw.get_native_namespace(data) is get_polars():
            # Polars doesn't allow formatting `Date` with time directives.
            # The date -> datetime cast is extremely fast compared with `to_string`
            columns.append(
                nw.col(name).cast(nw.Datetime).dt.to_string(local_iso_fmt_string)
            )
        elif dtype == nw.Date:
            columns.append(nw.col(name).dt.to_string(local_iso_fmt_string))
        elif dtype == nw.Datetime:
            columns.append(nw.col(name).dt.to_string(f"{local_iso_fmt_string}%.f"))

thanks @jonmmease ! review comments look great to me, much appreciated 🙏 ✅

Thanks to everyone involved! This is very exciting and we should cut a release soon :) I'll open a new discussion for it.


Support for Django 5

Is this project actively maintained ?

Last release was just two months ago. I would say yes, but there are not many commits or new features.

@suavesav,
Package supports only till Django 4.2
Will there be are support for Django 5.0 in the near future ?


Textfield: X, C, V inserted during Cut, Copy and Paste

When I use one of the copy and paste actions on a text field, it inserts the corresponding shortcut key as well. For example if I press Command + V (I'm on a Mac, see below) it pastes my clipboard content and then inserts a 'v' behind the pasted content. Similarly, a 'c' is inserted after copying some text and an 'x' is inserted after cutting.
To reproduce the issue you can use the UITest (one of the LibGDX tests / examples), which showcases a text field among other controls.
I tested this on OS X Yosemite, Java 8, using the latest source code.
If you can test this on your system, please leave a comment with your system specification and what behavior you experienced.

I might be doing something wrong but I cannot reproduce this issue on Windows 8 with the latest master from Libgdx.
Sanity check, are you sure you're not leaving the V key pressed after you release Ctrl? It shouldn't trigger the repeat key but still.

Yeah, I am absolutely sure. It works like this: press command, press v, clipboard content and v appear immediately, release v, release command.

I can reproduce on OS X Yosemite.

I think it might be OS X specific, but that is only speculation. Would be nice if someone could try and reproduce it on OS X.
On the other hand, I figured that the copy and paste actions are handled in the keyDown method of TextFieldClickListener and the text insertion is handled in keyTyped. However, nothing in keyTyped cancels the insertion if command / control is pressed.
@saltares: Could you add the following line as the first line to the keyTyped method in TextFieldClickListener and see what it prints?
System.out.printf("typed '%c'%n", character);

For me it prints:
typed '' // command pressed
typed 'v' // v pressed

Also, for me, pasting seems to stop working correctly after the first paste. Afterwards, it takes several tries and very quick timing on the Command and V pressed to get it to work (they have to have been pressed nearly at the same time).

Adding the following line to keyTyped fixes it for me, however pasting works fine for me, if you ignore the inserted characters.
Stage stage = getStage();
if (stage == null || stage.getKeyboardFocus() != TextField.this) return false;

if (UIUtils.isMac && Gdx.input.isKeyPressed(Keys.SYM)) return true; // This line is new...

if ((character == TAB || character == ENTER_ANDROID) && focusTraversal) {
    next(UIUtils.shift());
}

Should I create a PR or does anyone have a different solution?

I added if (UIUtils.ctrl()) return false; on my end and it works, not sure if it affects things on the other platforms.

If you check for isMac explicitly it won't affect other platforms. The condition that I've posted is just the part of the UIUtils.ctrl() method that handles OS X.

Yeah, but I guess I have to wonder... under what conditions would you actually want to append text while Ctrol (or Command) is pressed?

This does only happen if you use the lwjgl test starter. It works fine on mac using the jglfw test starter...

I've also found this issue on OSX Yosemite. It also happens with Cmd+A to select everything (so it selects everything, then it types an A, so you get a field with only an A on it).
Also, and maybe related to what @MobiDevelop comments about it not working properly after a while, sometimes just pressing A clears the whole textfield and adds an A there, without me pressing Cmd


wrong swa install command

hello it should be:
npm -g install @azure/static-web-apps-cli
today its without the "-g" which makes it fail.
BR,
Rasmus

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: e4071ee2-abad-4d21-7f38-346985c152ae
Version Independent ID: 4c6f16f1-b40f-0933-9305-de14ea7b9f59
Content: Set up local development for Azure Static Web Apps
Content Source: articles/static-web-apps/local-development.md
Service: static-web-apps
GitHub Login: @craigshoemaker
Microsoft Alias: cshoe

@reduards Thanks for the feedback! We are currently investigating and will update you shortly.

@reduards Thanks! The feedback has been raised once reviewed by author will be visible.
We will now proceed to close this thread. If there are further questions regarding this matter, please let us know. We will gladly continue the discussion.