fix memory leak

Thanks for the contribution @thechampagne!
Seeing no downsides in freeing the mem. When you're already on this, can you provide a short snip on how you detected the leaks?

reading the code that's how i found them.
there is also another use of C.CString without explicit "free" but i couldn't free it because i don't know how it's being handled between C and Go.

The process of memory debugging is clear.
Upon seeing the changes in the PR I didn't doubt that mem would leak there.
Just hoped to safe some time creating a snip for this exact case to see how Go really behaves here. Personally I think it's good when confidence comes something simplify reproducible directly testable and verifiable.


cargo plugin sub commands no longer work in nightly version

All subcommands that come from external crates stopped working with the latest nightly cargo.
rustc --version
rustc 1.26.0-nightly (a04b88d19 2018-03-19)
rustup --version
rustup 1.11.0 (e751ff9f8 2018-02-13)
cargo --version
cargo 1.26.0-nightly (d6c3983fe 2018-03-16)
Installing /home/travis/.cargo/bin/cargo-make
The command "cargo install --debug cargo-make" exited with 0.
0.04s$ cargo make ci-flow
error: Found argument 'ci-flow' which wasn't expected, or isn't valid in this context
USAGE:
    cargo [SUBCOMMAND]
For more information try --help

See build example: https://travis-ci.org/sagiegurari/cargo-make/jobs/355826819#L530

Yep, that's a known issue that I've broken all subcommands on nightly :-(
The issue is already fixed in Cargo, the PR with Cargo update to rust-lang/rust is https://github.com/rust-lang/rust/pull/49182

Looks like https://github.com/rust-lang/rust/pull/49182 merged a couple of days ago, although I'm not sure which nightly it'll ride -- with nightly-2018-03-21 from rustup I'm seeing cargo 1.26.0-nightly (d6c3983fe 2018-03-16).

@matklad when can we expect a fix for this? I tried the latest nightly and its still failing.

@sagiegurari It looks like a new nightly has been pushed with the update to cargo.

Thanks @ehuss you are right. Works great now.


update: update lark to match feishu keyword

Description
set this command to match feishu keyword.
Screencast

Checklist

[x] I read the extension guidelines
[x] I read the documentation about publishing
[x] I ran npm run build and tested this distribution build in Raycast
[x] I checked that files in the assets folder are used by the extension itself
[x] I checked that assets used by the README are placed outside of the metadata folder

LGTM

@pernielsentikaer hi, thanks for your review, very thanks.


Revert "Fix Blob Quick Query XML elements to align with service and documentation."

Reverts Azure/azure-rest-api-specs#14688
There's either a bug in the service or the documentation.

Closing PR, the changes made from that this PR is trying to revert are fine and correct.

Hi, @amnguye. The PR has be closed for a long time and it's related branch still exist. Please tell me if you still need this branch or i will delete it in 14 days.

Hi, @amnguye. The PR has be closed for a long time and it's related branch still exist. Please tell me if you still need this branch or i will delete it in 14 days.

Hi, @amnguye. The PR has be closed for a long time and it's related branch still exist. Please tell me if you still need this branch or i will delete it in 14 days.

@JackTn Feel free to the delete the branch. Thanks.


Logo icon missing, unable to reopen sidebar

With the logo icon missing, there's nothing to click to reopen the sidebar.

PR pending: https://github.com/grommet/grommet-ferret/pull/33

fixed


execute under current uid|gid

containers should run under the privileges of current user.  e.g.
docker run -u $(id -u):$(id -g) ...
# docker run -u 1000:1000 ...

closed in 6154a5042263900fa8d65c06afa8d33fac7a8871


Feature request: Creation or deletion initiated by a LWM2M server, client notifies its application.

We would like to have the possibility to get a notification (callback or subscription) from the mbed-client when a object instance is created or deleted via a LWM2M server. This to inform the application to cope with the (dis)appearance of an instance.

ARM Internal Ref: IOTCLT-880

Hi,
Can you please elaborate your use case ? Which application needs to be informed about this ? There is already a callback which gets called when object instance is created
value_updated( , )
For deleted resources we are not sending any updates because those objects dont exist anymore.

Hello yogpan01,
A typical use case for our project is that we want to assign an application function to handle/process a object instance.

The relation, 1 object instance is processed by 1 application function.
Each object ID has a specific application function to do its job.

This way be we can dis/enabled specific features in our device via LWM2M commissioning. So this means when creating an object instance via a LWM2M server, a task is dynamically created to run the corresponding application function. But also visa versa, deleting the instance must kill it to free unused resources.
I'm aware of the value_updated callbacks, but how should this work? I cannot set the callback via the LWM2M server creation proces.

Closing the issue as inactive and due to the repository being deprecated.
Please take a look at our current offering at https://cloud.mbed.com/docs/current. The actively maintained version of this component can be found in http://github.com/armmbed/mbed-cloud-client.
Please open a new issue if this is still valid. We strongly encourage you to migrate to the latest product. New issues in this repository will most likely remain unfixed.


PriorityQueue tests failing seemingly at random

The functionality tests for the priority queue have each been failing seemingly at random from one run to the next.
Sometimes MemoryTest fails, sometimes PriorityQueueTest fails, and sometimes they both pass...
This should be really made consistent, as it either shows that the functionality of the PriorityQueue is wrong, or that the test is wrong.
@tjarker Please look into this.

will do!

Why was this closed? The issue still persists and it doesn't look great to keep having random red crosses in PRs and on the main branch.

I see that now, sorry I thought this had been solved


Am I missing something with Coordinate scaling?

I think I am missing a big piece here.
I subclass MacawView (class MacawSalesChartView: MacawView).
The chart I draw end up being different depending on what device I run it on. I can lock down height so it's the same across any device, but the width will change depending on the device.
So if I am drawing out a bar graph for example, I assume I need to know the width of the chart (frame/rect/whatever) so I can calculate spacing and bar width and all that.
So my question is 2 fold :

Is there some kind of automatic scaling system (I seem to recall someone saying there was in a youtube video) where I simply set a "virtual size" and do all the work and it redraws it correctly on the various devices

or

Do I need to modify the drawing code based on the width of the frame running on that particular device. And if so, where can I get the frame width/height from?

Thanks in advance.

Hi David,

There is content layout you can use: https://github.com/exyte/Macaw/wiki/Understanding-MacawView-content-layout

You don't need to modify your drawing code if you're using content layout. However you can always access node width/height using node.bounds.


Implement BatchGet

Would be really nice to have BatchGet available as a feature when reading multiple keys at once.
This is the API they're using at dynamite (for reference)
client.newBatchGetBuilder()
.requestItems('user', [{'userId': 'userA', 'column': '@'}, {'userId': 'userB', 'column': '@'}])
.requestItems('phones', [{'userId': 'userA', 'column': 'phone1'}, {'userId': 'userB', 'column': 'phone1'}])
.execute()

@blakebyrnes - It's been on my todo list for a long time, but I just haven't had the time.
@ricardopinto has added the functionallity in his fork:
https://github.com/ricardopinto/dynamoose#modelbatchgetkeys-options-callback
If @ricardopinto does a PR, I would be happy to merge it.

Hi! I've sent a PR with batch operations implemented. https://github.com/automategreen/dynamoose/pull/14

Thanks -- I'll merge it in shortly

I published npm version 0.4.0 with batch functionallity.
Special thanks to @ricardopinto & @khamasaki. They did all the work.


Bring back MinGW CI

With latest MinGW fix and this workaround applied build and tests work but it takes about 16 minutes for x86_64-gnu toolchain to finish. One of the gnu toolchains can be dropped to save time.
Nightly x86_64-gnu doesn't work at all so it's temporary using beta to see what you think about it.

Thanks for the PR! I'm hesitant to add this though as MinGW has historically been incredibly difficult to keep working on CI with very little benefit. It would have caught a regression like the MinGW toolchain not working, but that was altogether a pretty simple regression to fix and one that happens vanishing rarely (and was discovered quite quickly).
To that end I'd prefer to not add this for now, but it's good to have the knowledge here of how to get it working for other projects!

No problem, I'll keep the branch around so it's easier to access it.


Fix video annotation/detection

This PR does two main things:

Updates the event listener to add the annotate metadata when uploading a video, as well as an image
Changes the metadata on folders/items, to work in a more sensible way.

Now, the metadata structure is as follows:

Folders have a metadata structure of:

annotate: true | false
fps: Int
type: "video" | "image-sequence"
viame: true

Result detection items have a metadata structure of:

detection: <folderId>

The value of the detection field represents the folder to which the detections belong. Ideally, I think to keep track of videos more appropriately, we should find a way to set a "video" field on the folder metadata, with the item ID of the video for that folder. However, that's not possible with our current event listeners, as the event we receive is triggered before the item is created.

The only part I don't understand is why detection items (The CSV file, I'm assuming?) need a reference back to their parent folder.

The only part I don't understand is why detection items (The CSV file, I'm assuming?) need a reference back to their parent folder.

It's not necessary for the item itself to have this reference (the folder could have it), but in general it's used to associate the csv file with the video folder, as a detection. As far as why I did it this specific way, it was just a smaller change than having that live in the folder instead.
I'd actually say it makes more sense to have detection live on the folder, as I think it conveys the semantics better, however we'd have to make sure we don't lose functionality. Currently, if there's several csv files with this detection field set to the folderId, the most recently created one will be used. We could replicate this on the folder itself by just changing the value of detection every time a save was done (and a new csv file was uploaded). If we did it this way, any time a new save is done, the old csv files wouldn't have a link to the folder. I'm not sure if this is a problem or not, just laying out the scenario. Curious to hear what you think @subdavis.

If you upload a video without a subfolder, you still don't get an annotate button.

This is because currently, when creating a folder, we're using the a girder web component, and the viame metadata field is not set. So when uploading the video, it doesn't mark the folder for annotation, as the viame field is not set to true. I'm not sure if we're able to pass metadata to the call being made in the GWC.

This caused the annotations I already had to no longer get discovered.  I think some kind of migration will be necessary, to check if annotations exist in the old format and add whatever new metadata is required for them to get picked up.  What do you think?

Yes, I agree. I think merging should be held off until I have a working script in place to do this.

This is because currently, when creating a folder, we're using the a girder web component, and the viame metadata field is not set. So when uploading the video, it doesn't mark the folder for annotation, as the viame field is not set to true. I'm not sure if we're able to pass metadata to the call being made in the GWC.

Not entirely sure, but is this allowing you to set metadata on upload?

(in Upload.vue?)

Not entirely sure, but is this allowing you to set metadata on upload?

(in Upload.vue?)

That allows you to set metadata on the folder created when also uploading a video, but the scenario I'm referring to is just using the Create Folder button to create a folder, and then uploading a video into that, without creating a folder (so in your example, createFolder = false).

understood

I've got a working script that will make the necessary migrations. If everything else is approved, I can merge this, take the server down, run the migration, pull in master with these changes, and start it up again.

Data migration went successfully :+1:


cachetools conflict with firebase-admin

You require:
google-endpoints-api-management
which is also requiring:
google-auth
but firebase-admin depends upon:
google-api-core which requires :
google-auth<2.0dev,>=0.4.0
so I'm getting this error:
google-auth 1.6.3 has requirement cachetools>=2.0.0, but you'll have cachetools 1.1.6 which is incompatible.
Also can you please make a note on the README.MD that using endpoints FORCES you to ignore advice elsewhere on the net to use virtualenv (instead of vendoring) with GAE-standard

author of firebase-admin has kicked this over to you
https://github.com/firebase/firebase-admin-python/issues/269#issuecomment-472594642

google-endpoints-api-management does not require google-auth. It does require cachetools < 2. The most common reason for an upward bound on versions is that the new version does not function on App Engine. The second most common reason is that we have a dependency which requires an older version. I don't know the exact reason for this upward bound.
This issue is properly tracked in https://github.com/cloudendpoints/endpoints-management-python/issues/72, so I'm closing it here.

This issue is properly tracked in cloudendpoints/endpoints-management-python#72, so I'm closing it here.

There has been no movement on this bug over there since it was filed, despite a trivial PR that was submitted to no response for over 4 months.
Given that this library is the one I actually use, and it currently isn't compatible with the google-cloud library, is there any overlap in the teams that might help shake the tree a bit?  Or have the cloudendpoints libraries been completely abandoned?


Next.js < Image >

Hi nice work!
I'm trying to pass Next.js: custom < Image > render component to
customComponents={{
customImageRenderer={imageProps => <Image {...imageProps} />}
}}
(...replacing <img with <Image)
Get Error:
Error: failed to process internal error: entered unreachable code: assign property in object literal is invalid.
Not sure if the issue is on my side, or if it is a bug? :-)

customComponents={{
	customImageRenderer: (imageProps) => {
		return (
			<Image
			src={imageProps.src}
			height={imageProps.style.height}
			width={imageProps.style.width}
			/>
		)
     },}}`

customComponents={{
	customImageRenderer: (imageProps) => {
		return (
			<Image
			src={imageProps.src}
			height={imageProps.style.height}
			width={imageProps.style.width}
			/>
		)
     },}}`

Hi @kentare
Sorry for our late response.
Can you clarify about the metatags? Maybe even a link so we can read more and see if we can help.

The error message you're encountering suggests that there is a syntax issue in the way you're defining the customImageRenderer property in the customComponents object. It appears that there might be a syntax error or incorrect structure in your code.
In JavaScript object literals, you should define properties without the assignment operator (=). Instead, you should specify the property name followed by a colon (:) and then provide the property value.


Interconnect function strange behavior

Hello,
I am testing the interconnect function to integrate it within nodedge editor, but I don't understand the behavior in the example_with_connections.
In particular, I have the following warning:
UserWarning: Unused input(s) in InterconnectedSystem: (0, 1)=sum.p.y[0]

And the step response is not the same as for the example_with_explicit_signals.
Can you please explain me what I am doing wrong?
Best regards.
Anthony.

import control as ct
import numpy as np

def example_with_explicit_signals():
    P = ct.tf2io([1], [1, 1], inputs="u", outputs="y")
    C = ct.tf2io([1], [1, 0], inputs="e", outputs="u")
    sumblk = ct.summing_junction(inputs=["ref", "-y"], output="e", name="sum")
    io_closed = ct.interconnect([sumblk, C, P], inplist=["ref"], outlist="y")
    print(io_closed)

    time = np.linspace(0, 10, 100)
    t, y = ct.input_output_response(io_closed, time, U=30, X0=[0, 15])
    print(y)

    return y

def example_with_connections():
    P = ct.tf2io([1], [1, 1], name="p", inputs=1, outputs=1, states="X0")
    C = ct.tf2io([1], [1, 0], name="c", inputs=1, outputs=1, states="X1")
    sumblk = ct.summing_junction(inputs=["ref", "p.y[0]"], output="e", name="sum")
    io_closed = ct.interconnect(
        [sumblk, C, P],
        inplist=["ref"],
        outlist="p.y[0]",
        connections=[
            ("p.u[0]", "c.y[0]"),
            ("c.u[0]", "sum.e"),
        ],
    )
    print(io_closed)

    time = np.linspace(0, 10, 100)
    t, y = ct.input_output_response(io_closed, time, U=30, X0=[0, 15])
    print(y)

    return y

if __name__ == "__main__":
    np.testing.assert_array_equal(
        example_with_explicit_signals(), example_with_connections()
    )

The error is happening because of the way that signals are named.  In the command
sumblk = ct.summing_junction(inputs=["ref", "p.y[0]"], output="e", name="sum")

you are creating a system "sumblk" with inputs "ref" and "p.y[0]".  When you list the connections in the interconnect function, you are not connecting the second signal (which is named "sum.p.y[0]") to anything, so you are getting a warning message.
The seemingly correct command to getting everything up like you want would be
connections=[
            ("p.u[0]", "c.y[0]"),
            ("c.u[0]", "sum.e"),
			("sum.p.y[0]", "p.y[0]")
]

However, if you try that in the current main branch, you will get an error because signal names are not allowed to include '.' (not sure what version you are running, but apparently something earlier than did not have this check).
To get what (I think) you want, use
sumblk = ct.summing_junction(inputs=["ref", "p_y[0]"], output="e", name="sum")

and
io_closed = ct.interconnect(
        [sumblk, C, P],
        inplist=["ref"],
        outlist="p.y[0]",
        connections=[
            ["p.u[0]", "c.y[0]"],
            ["c.u[0]", "sum.e"],
			["sum.p_y[0]", "-p.y[0]"]
        ],
    )

Note that in adddition to changing the signal names, I also changed the connections to be a list of lists, as required in the current main branch (that was a change when vector signals were introduced).

Thank you, Richard, it works now.
I am curious about why there is an asymmetry between the signals ref and p_y[0].
For the signal ref, there is an implicit and automatic connection between the signal coming from inplist and sum.ref.
Instead, for the signal p.y[0], we need to declare an explicit connection between p.y[0] and sum.p_y[0]. It may be because signal names in the summing junction cannot have dots. Nevertheless, we can name signals like system.signal, and the latter is also the signal name by default when the user does not specify one.
This leads to the following ambiguous case:

the user does not explicitly name the output signals of a system that is linked to a summing junction,
the summing junction cannot directly and implicitly link the signals coming from the source system to the internal signals of the summing junction (because it would imply two dots, which is forbidden).

I might be missing the reason that led you to remove the possibility of naming signals with a dot (and hence having double dots in summing junctions).

The reason for now allowing the dot was because systems can be nested and so "a.b.y[0]" could mean the signal "b.y[0]" coming out of the block "a" or "a" could be an interconnected system with subsystem "b" and you could mean "y[0]" from that (internal) subsystem.  This "nested" functionality is not actually implemented, so perhaps the better solution would just be to allow arbitrary signal names.
In terms of the asymmetry with the system input signal names (eg, "ref") and the internal connections, I agree that this should be generalized so that if you specify a signal name that is unique, then the system need not be specified.
I'm tagging this issue as an enhancement as a flag for someone (perhaps me, someday) to implement that functionality.


Misc. changes to Livebook page

List of changes:

bump and add Kino related deps
update keyboard shortcut to format the code

:partying_face: :partying_face: :partying_face: :partying_face: :partying_face:


Consistent functions and method docstring style

HoloViews is built on param which has the great advantage of providing a neat way of documenting the parameters of classes. While this is a great way to structure class level parameters, it does not provide anything that documents methods and functions. The current state of the docstrings on methods and functions in HoloViews is therefore what I'd call highly inconsistent and therefore quite frustrating for users.
While a large paragraph explaining what the method does is often sensible, docstrings are far more commonly used to quickly figure out what that one particular argument was called, what options it supports and what type it expects. I therefore think there is a pretty desperate need for us to standardize on some standard docstring format at least for functions and methods.
I'd be strongly in favor of moving to simple NumPy-style docstrings with sections for Parameters and Return values:
Some description of the function or method.

Parameters
----------
x : type
    Description of parameter `x`.
y
    Description of parameter `y` (with type not specified)

Returns
-------
err_code : int
    Non-zero value indicates error code, or zero on success.
err_msg : str or None
    Human readable error message, or None on success.

Readability of this is infinitely better than having to scan a large paragraph for the argument that I care about. After 1.10 is released I'd like to start a systematic review of all docstrings (many of which are hopelessly outdated) and upgrade them to a consistent style. I have no strong preference for any specific format but think that explicitly listing the arguments and return values will result in a much improved user experience and help us consider and address current API inconsistencies.

I support the goal of standardizing docstrings to make them consistent (and describing the arguments) though I will note:

I want a format that is succinct. I hope there is a format that is slightly more succinct than that Numpy style.
This is more important for user code than developer code.
Going through all the docstrings will be a big job.

I agree this is a discussion item we should return to after 1.10 is released.

This is more important for user code than developer code.

Yes, I'd suggest we do it incrementally starting with some of the core classes like Dimensioned, LabelledData, NdMapping, HoloMap, DynamicMap, Element and Dataset.

Sorry, I wasn't aware of this issue when I made
https://github.com/pyviz/nbsite/issues/67. I'm mentioning that issue now
just to say that using an existing format supported by existing tooling
(e.g. numpydoc) will have benefits for things like api documentation
generation. I realize there are other concerns though.
On Wed, Sep 12, 2018 at 5:26 PM Philipp Rudiger<EMAIL_ADDRESS>wrote:

Could we decide on a format now so I can make sure any new docstrings that
get written conform to the standard? @jlstevens
https://github.com/jlstevens You say you don't like the verbosity of
NumPy style docstrings, could you make a suggestion on how to shorten it,
otherwise that is my strong vote.
@jbednar https://github.com/jbednar Any opinions?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/ioam/holoviews/issues/2322#issuecomment-420711871,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAHiaNG36333KWozKL8VqOBgYFo52dLks5uaTWlgaJpZM4R_RCx
.

Thanks @ceball, somehow missed your issue even though I was pinged in it. I strongly agree that supporting existing tooling is a strong argument to go with a standard format and I have a strong preference for numpydoc.

How will we handle the fact that huge groups of all HoloViews Elements accept the same argument, without duplicating that info across every Element's docstring (and thus making it impossible to change the wording over time)?

My primary concern is method and function level docstrings. Class-level docstrings need improvements too but not as urgently as methods and functions, which are currently, imo, unusable. The class level docstrings are already covered reasonably well because parameters have docstrings and can refer back to the docs for a baseclass.

How will we handle the fact that huge groups of all HoloViews Elements
accept the same argument, without duplicating that info across every
Element's docstring (and thus making it impossible to change the wording
over time)?

Do inheritance for docstrings ;) If using a style like numpydoc, it should
be straightforward...
On Wed, Sep 12, 2018 at 6:04 PM Philipp Rudiger<EMAIL_ADDRESS>wrote:

My primary concern is method and function level docstrings. Class-level
docstrings need improvements too but not as urgently as methods and
functions, which are currently, imo, unusable. The class level docstrings
are already covered reasonably well because parameters have docstrings and
can refer back to the docs for a baseclass.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ioam/holoviews/issues/2322#issuecomment-420723886,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAHiQVirg8Hm3wNdZd_-ybiCskW3a9Gks5uaT6EgaJpZM4R_RCx
.

Do inheritance for docstrings

I'm probably not thinking very clearly about this specific situation, but what I'm concerned about is that  inheritance is all or nothing.  For Parameters that's handled by associating the docstring with each parameter, so that as parameters are gradually built up via inheritance, all of that can be provided for hv.help.  But for positional arguments, e.g. of the constructor, we can't handle that by class-level docstring inheritance, because each class needs its own documentation, even though all the classes accept the same first positional argument, and large groups of them treat that data the same.  Can that be handled by docstring inheritance of the constructors (not the class)?  Yes, if we don't use those constructor docstrings already for subclass-specific stuff, in which case we can use numpydoc docstrings in a few key base class constructors?

Not quite following, constructors (i.e. __init__) generally don't have a docstring, the class level docstring is usually used instead.

I was joking about parsing the numpy docstring for parameters, return
values, etc, and doing inheritance.
On Wed, Sep 12, 2018 at 7:00 PM Philipp Rudiger<EMAIL_ADDRESS>wrote:

Not quite following, constructors (i.e. init) generally don't have a
docstring, the class level docstring is usually used instead.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ioam/holoviews/issues/2322#issuecomment-420742146,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAHiZ1kyGz213GJisQV6USWzD2tLr9Aks5uaUusgaJpZM4R_RCx
.

Not quite following, constructors (i.e. init) generally don't have a docstring, the class level docstring is usually used instead.

That's my point.  The constructor arguments are shared across many classes, but each class has its own class docstring, and so I don't see a way to inherit the documentation of positional parameters of constructors.  Chris was joking about inheritance for docstrings, but I was just talking about inheritance of docstrings, which can work for methods but not classes, and if we don't generally document the constructor, then it won't work.  So I'm still not sure what's proposed be solved by just using numpydoc.

So I'm still not sure what's proposed be solved by just using numpydoc.

As I mentioned above I'm primarily concerned about method and function docstrings which are huge walls of text which make it incredibly difficult to figure out what each argument means. Class level docstrings also need improvement but unlike methods they generally have only one argument that needs explaining (i.e. data), because everything else is a parameter.

I can't really read it on my phone, but seems like someone already had th e
idea: https://github.com/meowklaski/custom_inherit
Anyway I don't mean to distract. I'm with Philipp (i.e. seems like a good
idea to agree on numpy style and start using it).
On Wed, Sep 12, 2018 at 7:32 PM Philipp Rudiger<EMAIL_ADDRESS>wrote:

So I'm still not sure what's proposed be solved by just using numpydoc.
As I mentioned above I'm primarily concerned about method and function
docstrings which are huge walls of text which make it incredibly difficult
to figure out what each argument means. Class level docstrings also need
improvement but unlike methods they generally have only one argument that
needs explaining (i.e. data), because everything else is a parameter.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ioam/holoviews/issues/2322#issuecomment-420752443,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAHiR-F5YzTT0e2aF-meeuXnnjNRyXEks5uaVNFgaJpZM4R_RCx
.

Ah; sorry! I didn't read far enough back to see that indeed, this issue is just about improving existing walls of text.
That said, I don't think we should make any decision about docstring formats without also addressing the issues about positional constructor arguments that users have raised (e.g. https://github.com/ioam/holoviews/issues/2925 and https://github.com/ioam/holoviews/issues/2947), because it would be very confusing to have multiple competing conventions for our docstrings, each addressing specific issues.
So if we use numpydoc format to avoid wall-of-text function and method docstrings, then we should also presumably be using numpydoc format to document positional parameters for our constructors, right?  And then I don't know how to avoid duplicating all that numpydoc-format documentation for the positional arguments across all our classes.

then we should also presumably be using numpydoc format to document positional parameters for our constructors, right?

Sure I'm not necessarily against that, it's just that in the majority of cases it'd be something like this:
data: array or dataframe or dict
    ...
**params
    Dictionary of parameters for the class

where the parameters aren't documented a second time.

I don't have much to say other than that the numpy style looks fairly readable to me, at least from a user perspective. From a developer perspective, I'll need to find a way to get emacs to fold the docstrings away: applying this format consistently will hugely increase the number of lines taken up by docstrings which will be distracting when trying to read and write code.

Personally, I'm ok with docstrings dominating code files; that's what people should be reading first (what is this code meant to do?) anyway. :-)

Parameters
x : type
Description of parameter x.
y
Description of parameter y (with type not specified)

This seems like it will cause untold confusion.  For some reason, numpydoc calls function arguments "Parameters" and not arguments.  Won't this make users think it's about our Param-based Parameters that are listed in hv.help(), particularly when ParameterizedFunctions blur the distinction between functions and objects (which can have param.Parameters)?

data: array or dataframe or dict
...

It doesn't seem to me that array or dataframe or dict will work as documentation for the data parameter.  First, it doesn't have to be any of those things, right?  Can't it also be a list of lists, a tuple of lists, etc.?  Second, only some of those types of objects are accepted, right?  E.g. only certain dicts would work, and presumably we should have some descriptions of what's required.  So while the words array or dataframe or dict can well be copied to every class, it doesn't seem like that would be feasible for the actual documentation that is needed for users to figure out what can be passed for the data argument.

Returns
err_code : int
Non-zero value indicates error code, or zero on success.
err_msg : str or None
Human readable error message, or None on success.

Given that one of Jean-Luc's next tasks is to add an output decorator to Param, should we not see if that will also let us declare the return type of a method or function using Param?  Putting that information in the docstring allows docs to be made but doesn't allow any other automated processing of it like would be needed for boxflow.

This seems like it will cause untold confusion. For some reason, numpydoc calls function arguments "Parameters" and not arguments. Won't this make users think it's about our Param-based Parameters that are listed in hv.help(), particularly when ParameterizedFunctions blur the distinction between functions and objects (which can have param.Parameters)?

I think it's not a numpy thing, but a python thing:

Parameters are defined by the names that appear in a function definition, whereas arguments are the values actually passed to a function when calling it

(https://docs.python.org/2.7/faq/programming.html#faq-argument-vs-parameter)
I've noticed people in general gradually changing how they speak about stuff, but it's been happening slowly. Anyway, however it might have been in the past, only the situation now matters really. I.e. when we say things like "the parameters of a parameterized function", we are going to cause confusion :(

Not just a Python thing: https://codeburst.io/parameters-arguments-in-javascript-eb1d8bd0ef04
So maybe the confusion is inescapable.

It doesn't seem to me that array or dataframe or dict will work as documentation for the data parameter.

My example was not meant to be exhaustive it was just meant to highlight that class level descriptions of the constructor would generally only describe the data attribute while the **params would describe themselves.

Given that one of Jean-Luc's next tasks is to add an output decorator to Param, should we not see if that will also let us declare the return type of a method or function using Param?

I see this as a very specific mechanism to declare pipelines and data flow not for general return type annotations and documentation. Param is too limited to describe the return types in enough detail and I definitely do not want to litter all our code with these annotations.

My example was not meant to be exhaustive it was just meant to highlight that class level descriptions of the constructor would generally only describe the data attribute while the **params would describe themselves.

I'm still not hearing any answer to the underlying problem: how will we document the data argument, given that it is shared across many classes, not identical for every class, and very complicated to describe?

Param is too limited to describe the return types in enough detail and I definitely do not want to litter all our code with these annotations.

I don't mean that to get documentation we'd always have to be able to express the return types fully in Param.  I mean that,  (a), if return types are specified, then that information can be used (where present), (b) a similar mechanism can be used to make human-readable docstrings describing the return type, where the annotation is not present and/or not sufficient to describe it, and (c) it would be odd to have a completely separate, independent convention for documenting return types that is not aware of the annotations for return types.
It seems just like a Parameter -- typically we try to add a very specific type programmatically, e.g. Param.Integer(bounds=(0,10)), and if we do, then very little docstring is needed (nothing about types or values, just about intention).  But we can still handle the general case, using param.Parameter, in which case one needs to put very detailed information in the docstring about what's supported.
Similarly, for return values, one could add annotations when it's very simple what's returned (True, False), but human-written docs can be added instead or as well, which together form the documentation for the return type, some of which is generated (when the information is declared), and the rest of which comes from typing it in.
I'm not arguing strongly that it has to be done this way, but I think it's something more in keeping with how Param works than adopting a convention with specially formatted big docstrings.  And I'm not seeing how the code would get littered with the annotations any more than the code would get littered with the same information put in a docstring.

I'm still not hearing any answer to the underlying problem: how will we document the data argument, given that it is shared across many classes, not identical for every class, and very complicated to describe?

Every class will have to describe exactly what it supports and refer back to the Dataset docstring to explain each format in detail.

Not just a Python thing

I just meant that everyone (including the official docs) in the python world used to talk only about "arguments", but that's changed. At least, that's how I remember it (I might be wrong).

I agree with Jim on these two points:

For some reason, numpydoc calls function arguments "Parameters" and not arguments. Won't this make users think it's about our Param-based Parameters that are listed in hv.help(),....

This would bother me as well.

And I'm not seeing how the code would get littered with the annotations any more than the code would get littered with the same information put in a docstring.

Specially formatted strings in code are a nightmare to manage. That said, I don't want to be arguing that we should have betters docs/docstrings which means even though I'm not enthusiastic about this idea, I'm not objecting to it either.

I also agree that we should call arguments Arguments not Parameters.

Just my two cents as a relatively new user who's unfamiliar with the codebase:
I've definitely suffered because the documentation is really tough to use. At several points I considered abandoning Holoviews completely, despite all the amazing functionality it offers, just because of this one weakness.
Also I just wanted to throw Google style docstrings out there as an alternative to NumPy style. These can be parsed by Sphinx with an extension.

I've definitely suffered because the documentation is really tough to use.

I really sympathize, and kudos for sticking with it. This needs to be the main priority as soon as we push 1.11.0 out the door and if possible we should at least tackle some of the core classes and methods before then.

I don't mean that to get documentation we'd always have to be able to express the return types fully in Param. I mean that, (a), if return types are specified, then that information can be used (where present), (b) a similar mechanism can be used to make human-readable docstrings describing the return type, where the annotation is not present and/or not sufficient to describe it, and (c) it would be odd to have a completely separate, independent convention for documenting return types that is not aware of the annotations for return types.

Dynamically generated docstrings sound good in theory but I see no reasonable way to make them interleave sensible with actual text written by a developer and to ensure that they are presented in a readable format. I'd prefer to refer the user to some more exhaustive reference than automatically interpolate some potentially complex (list of) type(s) that may have been declared as part of the output. The output should be readable and so I'd much prefer on coming up with conventions to refer users to actual docs when discussing accepted data arguments and so on.
I also do not want to annotate return types using some special param only syntax and litter all or even a significant fraction of our code with it and if we only do it in some cases it is as good as useless for consistent documentation. We would then also have to have a separate decorator to annotate the input types. Basically I'm pretty strongly convinced that annotating input and return types with param is a horrible way to auto-generate docs.
It sounds like you're envisioning as the param decorators as some general type system that we will apply to all our methods and functions, which I also think is a really bad idea. We're not going to put param decorators around every public function and method in holoviews, which is what would be needed to generate a consistent set of documentation.
My understanding was that param output declaration would be very useful for declaring pipelines on things like operations but not that they would become our own hand-rolled typing system, which we're going to use throughout the codebase.
So I would strongly argue we adopt either a NumPy or Google docstring style very soon, rather than waiting on param decorators, which imo are not well suited to the task anyway.

annotating input and return types with param is a horrible way to auto-generate docs

Here you're talking about methods?  Seems like there is a continuum where the tradeoffs differ.  If you have an arbitrary method with several complex positional arguments, there's no point to having some separate way to annotate the return type, because you're going to have a long docstring already and it's annoying to go to some special place and syntax to add one more bit of information.  Plus such a method can never be automated in any meaningful way; it's for a human to call.
But the other end of the extreme is what I would have thought is a common case, with a method that only at most accepts parameters, with no positional or special keyword arguments.  Such a method is worth annotating the return type with Param, because that one last bit of information will let it be handled automatically by things like boxflow -- on top of the parameter declarations, the output type is just one tiny bit more information needed, at which point it can be used to chain pipelines while doing full validation of outputs to inputs.  So for __call__(**params), I'd argue that annotating the output type is worthwhile and adds very important affordances for validation, automation, and so on; much bang for the buck!
So you and I are probably thinking of different scenarios when you're imagining using these tools; I'm not imagining hand-rolling a complete typing system to cover arbitrary methods, because there would be no use for such a thing, but I do expect that simple parameterized pipeline stages can be captured well and have important usages.  For HoloViews you are the one who has the set of methods in mind and knows whether most methods that need documentation fall in one or the other of those categories (i.e. lots of positional arguments vs none or only a standard widely shared positional argument), so it's your call to make.

Okay, I'm glad we were mostly talking past each other here then. Yes, I'm only talking about methods and functions here, I was hoping that was made clear by the opening paragraph of this issue.

So for call(**params), I'd argue that annotating the output type is worthwhile and adds very important affordances for validation, automation, and so on; much bang for the buck!

This really only applies to operations, of which there are only a handful and where I'd be very happy to do this. What I've been talking about in this issue is every single other public method and function in holoviews, of which there are 100x more than there are operations.

The point I do agree with is that the constructors for all the core datastructures will need to be given some thought, but that should be fairly straightforward as in most cases the positional arguments are simply: data, kdims and vdims.

Sounds good.

every single other public method and function in holoviews, of which there are 100x more than there are operations

And yet operations are mainly what I use, perhaps because the rest aren't well documented. :-)

Since we've not made a decision and started the conversion I don't think we need this issue anymore.


Remove setup section from book

It makes more sense to me that this just lives in the readme.

I thought that the idea was to move more of this stuff out of the readme and into the book?

I think "installing dependencies" is something that makes sense in the readme?

I'm of the opinion that as much of the information should be stored in one place, and if that place is the book then I think that this information should be included in the book instead of the readme. The readme can "sell" the concept of druid but the book is all the information you need to use it as well as how it works under the hood.


Composer autoload is not wotking

I am using a framework called cakephp. The installation for composer completed successfully, but the SimpleXLSXGen class does not load automatically. I am attaching my composer.json file
composer.json.txt
.

Re: [shuchkin/simplexlsxgen] Composer autoload is not wotking (Issue #49)
Hello, Gonzalo.
there conflict
please remove line 40
    "classmap": ["vendor/shuchkin/simplexlsxgen/src"]
I am using a framework called cakephp. The installation for composer completed successfully, but the SimpleXLSXGen class does not load automatically. I am attaching my composer.json file
composer.json.txt
.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
--
Thank You
Sergey
http://www.sibvision.ru/
http://www.smspilot.ru/
https://twitter.com/shuchkin
https://www.facebook.com/shuchkin

Hi Sergey,
Thanks for your time. I affair I have same mistake (after remove the line)...

I attach mi file in current state.
composer.json.txt
Thanks again,
Gonzalo

Re: [shuchkin/simplexlsxgen] Composer autoload is not wotking (Issue #49)
Gonzalo,
you have to figure out your autoloading problems yourself
anytime u can use
require_once "path/to/SimpleXLSXGen.php";
Hi Sergey,
Thanks for your time. I affair I have same mistake (after remove the line)...
I attach mi file in current state.
composer.json.txt
Thanks again,
Gonzalo
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
--
Thank You
Sergey
http://www.sibvision.ru/
http://www.smspilot.ru/
https://twitter.com/shuchkin
https://www.facebook.com/shuchkin


Add prettier config

I made this purely to seperate the formatting changes from the feature changes in the other PR's and to simplify the somewhat complex PR situation my multiple additions have caused due to formatting.
This branch is purely for adding the a .prettier config and formatting project files for consistency.
Pull requests https://github.com/alvgaona/cloudflare-discord-bot/pull/14, https://github.com/alvgaona/cloudflare-discord-bot/pull/13 and https://github.com/alvgaona/cloudflare-discord-bot/pull/12 are upstream from the commits on this branch.
Changes to styling requested can either be implemented directly here and then manually refactored into the above PR branches or this can be merged and styling changes can be handled as a seperate PR.

I beleive this PR is ready to be merged.

I beleive this PR is ready to be merged.

Now you can pull for the other PRs, right?

Now you can pull for the other PRs, right?

Yes I'll refractor those PR branches to be based off the merged changes to here. Then after those two are merged can discuss doing the same for components.


Azure functions not working

See https://github.com/unjs/nitro-deploys/issues/48 and https://github.com/unjs/nitro/pull/2092 for some context.
Unfrounatly i don't have any more information what exactly is the issue nor had a chance to investigate. So here is a placeholder.🤷♂

(ref) all magic we have for azure support:
azure swa:

https://github.com/unjs/nitro/blob/main/src/runtime/entries/azure.ts
https://github.com/unjs/nitro/blob/main/src/presets/azure.ts

azure fns:

https://github.com/unjs/nitro/blob/main/src/runtime/entries/azure-functions.ts
https://github.com/unjs/nitro/blob/main/src/presets/azure-functions.ts

I have done a bit of investigation and this appears to be perhaps at least partially related to using pnpm or anything but npm.
Reading the Azure Static Web Apps docs carefully it says this:

Also notably it says in the Next.JS hybrid rendering docs the following:

I'm going to take a small leap here and suggest that this note should also be on the nuxt docs. It appears that Microsoft's GitHub and Azure Pipelines does more than just build our applications it also appears to modify parts of it.
I have two built outputs one from the SWA CLI from the GitHub task and one I built locally. They have some significant differences in various files e.g.:

I'm not totally sure but I presume these are transforms added by the SWA CLI. Therefore if you are not using the SWA CLI to build your application you do not get these changes applied to the built output.
At present Microsoft's Oryx builder does not support pnpm or other package managers and it doesn't look like support will be coming any time soon 😞 (see https://github.com/microsoft/Oryx/issues/2340 https://github.com/microsoft/Oryx/issues/1150#issuecomment-1886067023)
I believe the Nitro docs should be updated to clarify that at present the Azure preset only works with npm as a package manager. It also makes it quite tricky to support any monorepos (my solution is a mono repo) as you have limited control over the build steps in the SWA cli.

Thanks for all your investigations! Just to double check, have you tried the shamefully-hoist: true flag in .npmrc?

Yep this is set in the .npmrc file. Also I can replicate the same issues using npm by building locally with npm and then deploying the outpuit straight to Azure without using the SWA CLI and the same 404 behaviour occurs.
The outputs.zip file above are both built using npm as the package manager. The only difference is that SWA CLI built one of them. And the differences are shown in the screenshots.
I probably shouldn't have said its related to pnpm because its not really. Its just that you must let the SWA CLI build your project. The reason this may be pertinent to pnpm users is that SWA CLI can only build projects that use npm as a package manager.

The other thing to mention is that as it stands the docs for the average nuxt application will work. Except it does potentially cause issues as users may not realise that when it deploys the pacakage manager being used to build is npm which may not be the one they expected.
This means their lock files aren't neccesarily respected so could cause some unexpected behaviour when deployed.
It probably just needs a similar warning on the Azure preset to note that this only works with npm at present. Shall I raise a small PR?

Okay, i think we are on a much better track. If you can confirm both presets work only with npm only, sure that would be more helpful to people I guess. Feel free to make PR 👍🏼
At the same time, this seems interesting. I am suspicious we might be doing something in nitro, external tracer that they don't like or support. Do you have a chance to try noExternals: true option if it makes any different?

With noExternals and a vanialla Docus site built with NPM I seem to get this error:

@GerryWilko Would you please use only Nitro or Nuxt? Docus is not maintained and could be source of false positive results in our case.

Have you tried adding this to your Build and Deploy action if you are using pnpm?

env:
          PRE_BUILD_COMMAND: npm install -g pnpm
          CUSTOM_BUILD_COMMAND: pnpm install && pnpm run build
          NODE_VERSION: 18.17.1 (or any other one)

(corepack enable might also do the trick)

Testing, Azure SWA works fine as documented on currently released nitro version but fails with either h3 or nitropack nightly releases.

Update: the underlying code works fine; the issue is that the code includes imports from nitropack/dist/runtime/plugin which is not present as the traced files only contain nitropack-nightly. Not clear why this affects azure more than any other preset.


Animated background

Adding animation in the background
https://user-images.githubusercontent.com/60808292/194712295-06636558-544f-4711-93d0-674d0e4d5db3.mp4

@Jaskaran2 Thank you!
First, I want to approve the dark theme pr.

@Jaskaran2 Thank you!
First, I want to approve the dark theme pr.

Sorry it's not very clear. Are you talking about someone else's PR?

@Jaskaran2 Thank you!
First, I want to approve the dark theme pr.

Sorry it's not very clear. Are you talking about someone else's PR?

We need to migrate into front framework, than we can reopen this PR.

@Jaskaran2 Thank you!
First, I want to approve the dark theme pr.

Sorry it's not very clear. Are you talking about someone else's PR?

We need to migrate into front framework, than we can reopen this PR.

Ok, let me know when to again raise the PR 😊


Create Scripture editor extension

User Story
As a user, I want to be able to edit Scripture so that I can do one of the most basic things of this software.
Description
Now that the resource viewer is definitively not an editor with #542, we need an actual editor extension.

Closed - #542 was reworked to cover this work.


Add Debian and Ubuntu OQS ci-containers.

Keep circleci-openjdk-9 container as well. Jobs run in parallel.
Address #4.

@baentsch Thanks for the feedback!
I added nightly builds only for the master branch here, and that's why I included the git_no_checkin_in_last_day.sh script. Should I modify it (before merge) to be similar with the one from the openssl repository.

Should I modify it (before merge) to be similar with the one from the openssl repository?

I'd say Yes - it saves us a turn-around.

@jimouris Thanks very much! Merged. Let's see how this runs tonight.


Add OpenCTD

Insert URLs to the project here: https://github.com/OceanographyforEveryone/OpenCTD

[x] The projects is active, documented, open source licensed, shows usage from external parties and is directly targeting environmental sustainability. Find more details in the Contribution Guide.

All issues labeled as Good First Issue of the project listed on OpenSustain.tech will be visible on ClimateTriage.com. This is a great way to welcome new community members to your project.
All projects listed will be posted on our Mastodon channel

@DrAndrewThaler


[SPARK-48049][BUILD] Upgrade Scala to 2.13.14

What changes were proposed in this pull request?
The pr aims to upgrade scala from 2.13.13 to 2.13.14.
Why are the changes needed?
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Pass GA.
Was this patch authored or co-authored using generative AI tooling?
No.

Let's wait for it
https://contributors.scala-lang.org/t/scala-2-13-14-release-planning/6581/12

Like we observed before, I guess we need to wait new compatible plugins and Ammonite shell additionally. Isn't it?
cc @LuciferYang , too

Like we observed before, I guess we need to wait new compatible plugins and Ammonite shell additionally. Isn't it?
cc @LuciferYang , too

Yeah, that's right. 😄
I guess it should be able to catch up with spark4.0 release.

Thank you so much, @SethTisue !

Although we are waiting for Ammonite still, could you base this PR once more, @panbingkun ?

https://github.com/com-lihaoyi/Ammonite/pull/1482

Although we are waiting for Ammonite still, could you base this PR once more, @panbingkun ?

Add support for Scala 2.13.14 com-lihaoyi/Ammonite#1482

Sure, done.

https://repo1.maven.org/maven2/com/typesafe/genjavadoc/genjavadoc-plugin_2.13.14/0.19/

I have updated the version of genjavadoc in the file project/SparkBuild.scala.

Thanks!

jline

I guess it may be related to the fact that the Ammonite supporting scala 2.13.14 has not been released.

Although we already use ammonite_2.13.13 without this jline issue, let's wait and see~

I guess it may be related to the fact that the Ammonite supporting scala 2.13.14 has not been released.

@SethTisue
Just right, you are here too
https://github.com/scala/scala/pull/10717
https://github.com/scala/scala/pull/10717/files#diff-87e6fe6a086cec9825e72357099f0ae6b84800ea025c5aee97b6cc7088c6cd4d
I found that in scala-2.13.14, the version of org.jline:jline was upgraded from 3.14.1 to 3.15.1 because:
https://github.com/scala/bug/issues/12957

But I found that org.jline:jline:3.15.1 is compiled based on JDK 21.

So the maven plugin  maven-enforcer-plugin helped us find this issue when Spark upgraded the scala version from 2.13.13 to 2.13.14.
Because our current Spark master branch supports JDK 17 and JDK 21.
Based on the above reasons, I would like to ask if it is also reasonable and feasible for us to exclude org.jline:jline:3.15.1 from org.scala-lang:scala-compiler:jar:2.13.14,  and explicitly let it use org.jline:jline:3.14.1?

https://github.com/jline/jline3/blob/c75301facc8716b59c1d57d3e3c5943358022560/pom.xml#L93-L95
https://github.com/jline/jline3/blob/c75301facc8716b59c1d57d3e3c5943358022560/pom.xml#L422-L434
The release version of jline 3.25.1 is still Java 8, this should not affect Spark's use of it.

[INFO] Restricted to JDK 17 yet org.jline:jline:jar:3.25.1:compile contains org/jline/terminal/impl/ffm/CLibrary$termios.class targeted to 65.-257
Oh... It seems that CLibrary$termios.class is not compatible with Java 8.

Thank you for the analysis and sharing, @panbingkun and @LuciferYang .

Thank you for the analysis and sharing, @panbingkun and @LuciferYang .

[INFO] Restricted to JDK 17 yet org.jline:jline:jar:3.25.1:compile contains org/jline/terminal/impl/ffm/CLibrary$termios.class targeted to 65.-257
Oh... It seems that CLibrary$termios.class is not compatible with Java 8.
In jline 3.25.1, CLibrary references some implementations that were moved to the java.lang.foreign package in Java 21, but in Java 17, they were still in the jdk.incubator.foreign package.
However, in jline 3.24.1, CLibrary also import classes in java.lang.foreign package, it's just that the final jline jar does not include the terminal-ffm module, so it won't trigger a compilation error from the maven-enforcer-plugin.
So it seems that using jline 3.25.1 should not pose any issues. It's unclear whether it's possible to workaround this by modifying the rules of the maven-enforcer-plugin. @panbingkun

Let me try, We should be able to refer to the following:
https://github.com/mojohaus/extra-enforcer-rules/issues/46

But I'm not sure we really want to do this?
We have 2 options:

Use org.jline:jline:3.14.1 (exclude org.jline:jline:3.15.1 from org.scala-lang:scala-compiler:jar:2.13.14)
Use org.jline:jline:3.15.1 (Modify the rule enforceBytecodeVersion by configuring ignoreClasses  and bypass it )
WDYT @dongjoon-hyun

Currently, this GA has turned green.

Do we have a corresponding issue in the Scala community or JLine community (or GitHub issue)?
IMO,
a. We can ignore Scala 2.13.14 completely due to this issue.
b. We can cut the transitive dependency as you mentioned as Option 1. However, it's Scala's dependency. So, the downstream projects or users (who are not aware of this) will complain again and again.
c. We cannot choose Option 2 because the downstream projects or users will be affected and complain again.
For Apache Spark 4.0.0-preview, let's choose (a) because we didn't have enough time to play around with this new Scala version.
Until 4.0.0 release, we had better keep tracking the upstream community (Scala/JLine) activity about this topic. The best case is for them to fix it for all downstream (including us).

Thank you for the analysis and sharing, @panbingkun and @LuciferYang .

[INFO] Restricted to JDK 17 yet org.jline:jline:jar:3.25.1:compile contains org/jline/terminal/impl/ffm/CLibrary$termios.class targeted to 65.-257
Oh... It seems that CLibrary$termios.class is not compatible with Java 8.
In jline 3.25.1, CLibrary references some implementations that were moved to the java.lang.foreign package in Java 21, but in Java 17, they were still in the jdk.incubator.foreign package.
However, in jline 3.24.1, CLibrary also import classes in java.lang.foreign package, it's just that the final jline jar does not include the terminal-ffm module, so it won't trigger a compilation error from the maven-enforcer-plugin.
So it seems that using jline 3.25.1 should not pose any issues. It's unclear whether it's possible to workaround this by modifying the rules of the maven-enforcer-plugin. @panbingkun

Let me try, We should be able to refer to the following: mojohaus/extra-enforcer-rules#46
But I'm not sure we really want to do this? We have 2 options:

Use org.jline:jline:3.14.1 (exclude org.jline:jline:3.15.1 from org.scala-lang:scala-compiler:jar:2.13.14)
Use org.jline:jline:3.15.1 (Modify the rule enforceBytecodeVersion by configuring ignoreClasses  and bypass it)

WDYT @dongjoon-hyun

If it can be confirmed that Scala 2.13.14 does not use this class(CLibrary$termios), I think it can be done this way because , but is this the only issue?

Do we have a corresponding issue in the Scala community or JLine community (or GitHub issue)?
IMO, a. We can ignore Scala 2.13.14 completely due to this issue. b. We can cut the transitive dependency as you mentioned as Option 1. However, it's Scala's dependency. So, the downstream projects or users (who are not aware of this) will complain again and again. c. We cannot choose Option 2 because the downstream projects or users will be affected and complain again.
For Apache Spark 4.0.0-preview, let's choose (a) because we didn't have enough time to play around with this new Scala version.
Until 4.0.0 release, we had better keep tracking the upstream community (Scala/JLine) activity about this topic. The best case is for them to fix it for all downstream (including us).

+1 for choice  Option a if there are no critical issues that need to be fixed by Scala 2.13.14
one issue I can see is Fix ArrayBuilder regression in Scala 2.13.13 (OutOfMemoryError when adding empty arrays)
Let's create an issue in the Scala community? @panbingkun

Do we have a corresponding issue in the Scala community or JLine community (or GitHub issue)?
IMO, a. We can ignore Scala 2.13.14 completely due to this issue. b. We can cut the transitive dependency as you mentioned as Option 1. However, it's Scala's dependency. So, the downstream projects or users (who are not aware of this) will complain again and again. c. We cannot choose Option 2 because the downstream projects or users will be affected and complain again.
For Apache Spark 4.0.0-preview, let's choose (a) because we didn't have enough time to play around with this new Scala version.
Until 4.0.0 release, we had better keep tracking the upstream community (Scala/JLine) activity about this topic. The best case is for them to fix it for all downstream (including us).

@dongjoon-hyun @LuciferYang
Let's keep an eye on it. I have submitted this issue to the scala community
https://github.com/scala/bug/issues/12994
Thank all!

@dongjoon-hyun @LuciferYang
Based on the detailed explanation of https://github.com/scala/bug/issues/12994,

and double-check by debug jline on local env. The verification process is as follows:
1.Based on this pr upgrade (the version of scala is 2.13.14 , the version of jline is 3.25.1)
compile spark by commands:
./build/mvn -Pyarn -Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Pspark-ganglia-lgpl -Pkinesis-asl clean package -DskipTests

2.When open spark-shell with the debug mode on local , and we will see the following run thread stack
export SPARK_SUBMIT_OPTS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=8888

3.The code logic of org.jline:jline:3.25.1 is as follows:
https://github.com/jline/jline3/blob/jline-parent-3.25.1/terminal/src/main/java/org/jline/terminal/TerminalBuilder.java#L654-L675

When loading org.jline.terminal.ffm, it will automatically skip because of the exception
java.lang.UnsupportedClassVersionError: org/jline/terminal/impl/ffm/FfmTerminalProvider has been compiled by a more recent version of the Java Runtime (class file version 65.65535), this version of the Java Runtime only recognizes class file versions up to 61.0

so we can completely ignore it in the maven plugin maven-enforcer-plugin#enforceBytecodeVersion, because it will automatically skip because of loading failure in the lower version of JDK (JDK < 21).

Thank you, @panbingkun .

Is the PR ready?
ammonite 3.0.0-M1` is enough?

https://ammonite.io/#3.0.0-M1

I think we  need wait a new Ammonite release version, as 3.0-M1 does not support Scala 2.13.14.

Is the PR ready?

ammonite 3.0.0-M1` is enough?

https://ammonite.io/#3.0.0-M1

I use this version [3.0.0-M1-19-a7973e17](https://mvnrepository.com/artifact/com.lihaoyi/ammonite_3.3.3/3.0.0-M1-19-a7973e17), which currently supports scala 2.13.14 https://mvnrepository.com/artifact/com.lihaoyi/ammonite

This seems to be a snapshot or test version, not an official release？

Is the PR ready?

ammonite 3.0.0-M1` is enough?

https://ammonite.io/#3.0.0-M1

I use this version [3.0.0-M1-19-a7973e17](https://mvnrepository.com/artifact/com.lihaoyi/ammonite_3.3.3/3.0.0-M1-19-a7973e17), which currently supports scala 2.13.14 https://mvnrepository.com/artifact/com.lihaoyi/ammonite

This seems to be a snapshot or test version, not an official release？

https://github.com/com-lihaoyi/Ammonite/issues/1481#issuecomment-2097731736

Is the PR ready?

ammonite 3.0.0-M1` is enough?

https://ammonite.io/#3.0.0-M1

I use this version [3.0.0-M1-19-a7973e17](https://mvnrepository.com/artifact/com.lihaoyi/ammonite_3.3.3/3.0.0-M1-19-a7973e17), which currently supports scala 2.13.14 https://mvnrepository.com/artifact/com.lihaoyi/ammonite

This seems to be a snapshot or test version, not an official release？

I have asked the community to release an official version for us at his convenience
https://github.com/com-lihaoyi/Ammonite/issues/1481#issuecomment-2097731736

https://github.com/com-lihaoyi/Ammonite/releases/tag/3.0.0-M2
3.0.0-M2 released ~ @panbingkun

https://github.com/com-lihaoyi/Ammonite/releases/tag/3.0.0-M2
3.0.0-M2 released ~ @panbingkun

Thanks~ ❤
Updated.


Autogenerated .DS_Store files on macOS are included in detect test suite

Summary
macOS occasionally generates .DS_Store files to store metadata about a directory for its builtin Finder application. Unfortunately, this can sometimes happen in a test/detect directory and generate erroneous detect test cases. Once the file exists, it's pretty hard to permanently remove it, as macOS will almost instantly regenerate the file if it is manually removed by rm -f .DS_Store or by other means.
Reproduction Steps
Unfortunately, I cannot seem to reliably trigger the heuristic that generates the .DS_Store file, so instead one can emulate the behavior by doing the following:

Clone the highlight.js repository
Run echo '[hello]' > test/detect/1c/.DS_Store from the repository root
Build with node ./tools/build.js -t node
Run the full test suite with npm run test
Get a failing test reporting AssertionError: .DS_Store should be detected as 1c, but was angelscript

Expected behavior
Since the user has little control over .DS_Store files, skip .DS_Store files when iterating through the directory here: https://github.com/highlightjs/highlight.js/blob/e964bec6c8d72d263042ab37ae196baa79f49c47/test/detect/index.js#L20-L22
Additional context
There are several ways to approach the problem:
1. Indiscriminately skip all .DS_Store files
  const filenames = await fs.readdir(languagePath);
  await Promise.all(filenames
+   .filter(filename => filename !== '.DS_Store')
    .map(async function(example) {

This is the easiest to implement, effectively hardcoding a blocklist of .DS_Store files.
If someone decided to make a language that used the .DS_Store extension, they can still just name the detect test file something like testcase.txt, since the current test runner does not care about file extensions.
2. Only skip all .DS_Store files on macOS
  const filenames = await fs.readdir(languagePath);
  await Promise.all(filenames
+   .filter(filename => process.platform !== "darwin" || filename !== '.DS_Store')
    .map(async function(example) {

This does not skip .DS_Store on other platforms.
You could also optimize it into a one-time check for other platforms:
+ const checkDsStoreOrNoOp =
+   process.platform !== 'darwin'
+    : (_) => true
+    ? (filename) => filename !== '.DS_Store';
...
  const filenames = await fs.readdir(languagePath);
  await Promise.all(filenames
+   .filter(checkDsStoreOrNoOp)
    .map(async function(example) {

The main drawback is that it technically causes inconsistent behavior between platforms (where the test only ignores the file on macOS), so the indiscriminate block suggested in (1) may be preferable.
3. Do nothing
This kinda sucks but since nobody reported this before I guess it's not a very common issue.

Never seen this [wrt Highlight.js] in all my years of maintainer-ship (on Mac OS X).  That said an indiscriminate block would be fine.  It would also need to be applied to the checkAutoDetect.js script... a PR would be welcome.

I usually put .DS_Store in my .gitignore file and delete all the files in the repo. They won't come back

Would anyone here like to pick up this issue and build a fix?

I would like to work on this issue, is it open?

Sure.