Deprecate non-distributable layers

Microsoft has announced their intention to change their image redistribution policy so that Windows images no longer make use of non-distributable layers: https://techcommunity.microsoft.com/t5/containers/announcing-windows-container-base-image-redistribution-rights/ba-p/3645201
@justincormack says:

Foreign layers in OCI images can now be deprecated

https://twitter.com/justincormack/status/1580536342114492417

This PR updates the spec to note that clients SHOULD NOT create new images with non-distributable layers. They may still need to handle them for old images created pre-this-deprecation however.
Please feel free to suggest better wording. 🚲🏠

Thanks @imjasonh.
I'm seeing a few things worth calling out on this thread since we are waiting from @jonjohnsonjr to reach out to some folks in Red Hat as what I recollect from the call. To me this is -

Deprecating/"Recommending against" using 3 media types going fowrard-

     application/vnd.oci.image.layer.nondistributable.v1.tar
     application/vnd.oci.image.layer.nondistributable.v1.tar+gzip
     application/vnd.oci.image.layer.nondistributable.v1.tar+zstd

Discouraging use of URLs which is typically associated with these mediaTypes but is not strictly tied to non-distributable layers

Descriptors referencing non-distributable layers MAY include urls for downloading these layers directly; however, the presence of the urls field SHOULD NOT be used to determine whether or not a layer is non-distributable.

https://github.com/opencontainers/image-spec/blob/8159c8264e2efaacc151612358f1c0adfebac9ad/specs-go/v1/descriptor.go#L32-L33

For new client implementations should they continue honoring these? Asking because there might be TOS associated with these and these definitions are kept only for backwards compatability only.

Lastly should we update  the comments to the descriptor fields as well or are we continuing to support this but only recommending not using the mediaTypes?

Discouraging use of URLs which is typically associated with these mediaTypes but is not strictly tied to non-distributable layers

I'm not sure if I'd go this far.

cc @dmesser -- Hey Daniel, we're hoping to signal the deprecation of non-distributable layers now that Windows images aren't going to be using them anymore. When we discussed yesterday @jonjohnsonjr mentioned he thought he might have seen some Red Hat images using them. Do you happen to know of any such cases?
If not we'd like to proceed with deprecation, but just in case there are still real world users of them it would be helpful to know.

I don't think we are in a state to move forward with a depreciation just yet. When we started evaluating that internally we found a number of concerns (mostly around developers pushing images) that I think should be discussed in the community before we commit to a path.

I don't think we are in a state to move forward with a depreciation just yet. When we started evaluating that internally we found a number of concerns (mostly around developers pushing images) that I think should be discussed in the community before we commit to a path.

Thanks, that's really helpful to hear! Can you enumerate the concerns?
FWIW I don't think the language as written should be interpreted as "clients SHOULD NOT pass along images with non-distributable layers", only that new builder tools SHOULD NOT produce new images with them.
Even that weak advice is unenforceable -- the OCI Police will knock down your door if you do it -- but it gives folks a gentle nudge away from them if they're considering using them in new images, or supporting them in new tools that don't expect to handle old pre-deprecation layers.
But obviously if you've got insights from actual users then that would be really helpful to include here as well.

cc @dmesser -- Hey Daniel, we're hoping to signal the deprecation of non-distributable layers now that Windows images aren't going to be using them anymore. When we discussed yesterday @jonjohnsonjr mentioned he thought he might have seen some Red Hat images using them. Do you happen to know of any such cases?

Thanks for reaching out @imjasonh - I don't know of such cases, as an open source company Red Hat doesn't use licenses that prohibit permanent binary storage.

@thecloudtaylor gentle ping

Thanks, that's really helpful to hear! Can you enumerate the concerns?

@NAWhitehead from our team is driving a discussion on the windows-container repo.
https://github.com/microsoft/Windows-Containers/discussions/287


iPhone heating up

Hey!
Thank you for starting this project, this has been the only package that actually worked for me so far.
While the streaming is working perfectly, my iPhone 14 heats up after some usage. Not sure if it's an issue from my code or the package. Have you had this issue ?
Thanks!

Hey! I have exactly the same issue here. It's the only library that works for my streaming (so great job) but my phone gets very hot.
What i have inspected so far is the following behavior. The CPU usage is fine for a while (it's low and constant. At some point it will have more and more CPU usage and it keeps accumulating until it hits 80-100% and the phone turns into a heating boiler.
The threads which are doing a lot of work when the CPU usage is that high are always caught up in the function "provideMediaDataIfNeeded". So maybe something is wrong here.
I couldnt find a pattern though of the point in which it starts accumulating...
Thanks a lot!

Same issue here, which is a shame because this is the only repo that works for my use case (that I've found), and the battery hit is going to be too high to use it, unfortunately.

Ok the issue seems to happen in the "provideMediaDataIfNeeded" - more concrete in the "requestMediaDataWhenReady" closure. At some point (and i couldnt find a pattern) it seems to execute indefinitely wihtout ever stopping.
So in the invalidate function you can add the following lines of code which will stop the closure from running.
audioRenderer?.stopRequestingMediaData() if let audioRendererFixed = audioRenderer {     audioSynchronizer?.removeRenderer(audioRendererFixed, at: CMTime(value: 0, timescale: 0)) // If you're dynamically adding/removing renderers         }
    This should resolve the issue i think. At least my phone didnt get hot after having an 1 hour phone call.

You mean like this?
func invalidate() {
        audioRenderer?.stopRequestingMediaData()
        if let audioRendererFixed = audioRenderer {
            audioSynchronizer?.removeRenderer(audioRendererFixed, at: CMTime(value: 0, timescale: 0))
        }
        removeBuffers()
        closeFileStream()
        cancelObservation()
        receiveComplete = false
        audioSynchronizer = nil
        audioRenderer = nil
    }

Yes exactly. However I think this alone makes the error just less likely, from what I see it can still loop in the provideMediaDataIfNeeded while loop.
I added now an additional bool in this while loop that breaks out of this loop.
while audioRenderer.isReadyForMoreMediaData && shouldProvidingData {
            if let buffer = audioBuffersQueue?.dequeue() {
                print("dequeue and enque buffer")
                audioRenderer.enqueue(buffer)
                startPlaybackIfCan(
                    audioRenderer: audioRenderer,
                    audioSynchronizer: audioSynchronizer
                )
            } else if receiveComplete {
                audioRenderer.stopRequestingMediaData()
                break
            }
        }
        ````

    Now it works for me I have 5 phones running simultaneously and they all stay cold. It's more a hacky solution but it works :))

Awesome, where are you managing the state of shouldProvidingData to toggle between true and false

Awesome, where are you managing the state of shouldProvidingData to toggle between true and false?

Thanks, looks like you might have named shouldProvidingData  incorrectly as shouldProvideDataGlobal in the invalidate() function

Ah you're right I renamed everything to global in my code but renamed it back to the original name here to stay consistent with the chat.
thanks!

Before I open a new ticket, do you get any audible clipping sounds on your playback when on the device? Like short popping noises and the voice skips a little, I guess between data packets? They don't appear on the simulator, only when on the device itself.

No I dont experience that. At least not as a problem of the audio player. I once had that my voice generator was too slow in generating the next chunks such that there was an audible gap. But I guess that is not a problem of the audio player

Thanks all for the provided info, this has been very useful.
Would any of you be able to test a fix from this branch and let me know if it helped? - https://github.com/mihai8804858/swift-chunked-audio-player/tree/fix-infinite-loop

Seems to work fine, i tested it for 1 hour and the phone didnt heat up. I'll let you know if things change.
Thanks!

I've integrated these changes. Will let you know but so far so good!

Thanks, changes have been released in 1.0.1


Model joint torque noise in Bullet simulation

Usage of the new feature is documented in the simulate_joint_friction_and_noise.py example.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>0
Details

25 of 34 (73.53%)  changed or added relevant lines in 5 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.09%) to 90.084%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

upkie/cpp/actuation/BulletInterface.h
2
6
33.33%

upkie/cpp/actuation/BulletJointProperties.h
8
13
61.54%

Totals

Change from base Build<PHONE_NUMBER>8:
-0.09%

Covered Lines:
2689

Relevant Lines:
2985

💛  - Coveralls

Running the example with torque noises of 0.1, 10 and 100 Nm in the left leg, and looking at the resulting wheel velocities (not torques):

Whoops, I mixed in some kinetic joint friction (0.1–0.4 N m) in the above plot. Here is the same with pure torque white noise in [10.0, 0.0, 0.1, 0.2, 0.5, 1.0, 5.0] N m:

Same log, joint torques:


Command not working

I followed the wiki trying to create a custom command that switch between spectator and creative but its not working when i add the unsuccesful part
`schemaVersion = 1
commandMode = "COMMAND_CUSTOM"
command = "gmode"
[[actions]]
id = "generic"
command = "gamemode spectator $executor_name()"
commandType = "SERVER"
requireSuccess = false
messageIfUnsuccessful = "Test"
[[actionsIfUnsuccessful]]
command = "gamemode creative $executor_name()"
commandType = "SERVER"
requireSuccess = false
messageIfUnsuccessful = "Test"This is the .toml file i got but i just get a error that it not load:[12:29:55] [Server thread/ERROR]: Failed to load file at "/home/container/./config/commandaliases/g-mode.toml" throws com.fasterxml.jackson.dataformat.toml.TomlStreamReadException: Newline not permitted here
at [Source: (File); line: 12, column: 22]`

Sorry for the late response, you can achieve the follow with
schemaVersion = 1
commandMode = "COMMAND_CUSTOM"
command = "gmode"

[[actions]]
command = "gamemode spectator $executor_name()"
commandType = "SERVER"

[[actions.actionsIfUnsuccessful]]
command = "gamemode creative $executor_name()"
commandType = "SERVER"

I removed the extra page. It's not suppose to be there.


Update package.json main property

Adding main property for resolving in node. I'm using only npm in my project and there is the "main" property which is missing. I can't use "require('ng-bs-daterangepicker');" without this property.

@luisfarzati What's the holdup here? This module can't be required without this key.


Remove left-over requirements on OpenSSL >= 1.1.0 for cert name matching

When bug #4206 was resolved, some bits remained that still indicate that OpenSSL 1.1.0 or later is required:

Invalid error logging when name verification is enabled with older OpenSSL (or LibreSSL).
Outdated note in the unbound.conf(5) man page.

Thanks, doc updated.


Provide a more comprehensive guide

Or examples which cover most of the exposed APIs

At this time this project is in maintenance mode. Adding documentation is not a priority, but a PR with documentation and / or examples would be welcome.


Added support for VPCs that don't have hostnames

The example config doesn't work in VPCs that have DNS hostnames disabled. This change uses instance metadata + the aws cli to discover the EFS mount point's ip address without using a hostname.

Calling EFS with the AWS CLI requires adding permission to the instance profile. This requires additional configuration steps and would be a breaking change, but I could see putting it in a branch for custom VPCs. Do you have a use case for not enabling DNS resolution and DNS host names?
Thanks,
-Michael

I personally don't have a desire to turn off DNS host names, but the IT department at my company does.

Merged into custom-vpc branch.
Thanks,
-Michael


Add css repeat() support

First, thanks for creating this website.
It's very nice to have this utility and I'll probably use it a lot when creating layouts.
Motivation
When using css grid, one of the css features that can be used is the function repeat()
when you have lots of columns or rows.
Adding support to this function would help the developer know the number of columns / rows
only by first looking at the repeat first argument (the number of repetitions).
For example:
/* I'm CoNfUsEd(???) How many columns are there??? */
.selector {
  grid-template-columns: 1fr 1fr 1fr 1fr 1fr 1fr 100px;
}

would be translated to
/* YEEAH now I get it! there're 6 fraction columns + 1 100px column */
.selector {
  grid-template-columns: repeat(6, 1fr) 100px;
}

What is included in this PR?
This PR adds two functions:

groupRepeatedUnits (templateArray = ['1fr'])
this function iterates over the template array and creates a matrix in a format like
[['1fr', '1fr'], ['100px']] when passed ['1fr', '1fr', '100px'] as argument.

createRepetition (groups, maxRepetition = 1)
this function expects a data model like the one groupRepeatedUnits returned and
maps and joins it in a final format like 'repeat(2, '1fr') 100px'. Note: it expects a second arg called maxRepetition, which controls the max number of repetitions before adding a repeat() in the array. If you think we should only add repeat() for 2+ repetitions just increase this argument default value.

I chose the 2-functions implementation over a simple function that creates only one array in favor of cleaner code. This because a single function would add complexity and nested ifs. If you know a better implementation, please share with me.

Ahhh awesome! You know I thought about implementing this but way further down the line. You saved me a lot of work! I'm in the middle of something and can't review the PR just yet but hot dang, I'm excited!

I think that maybe it would make more sense to externalize the repetition functions to another file instead of just letting it be in the middle of the store.
What do you think @sdras ?

@sdras done

Looks great! Thank you!


SiblingSubgraph does not handle Copies at output boundary

The following test
    #[test]
    fn edge_both_output_and_copy() {
        let one_bit = type_row![BOOL_T];
        let two_bit = type_row![BOOL_T, BOOL_T];

        let mut builder =
            DFGBuilder::new(FunctionType::new(one_bit.clone(), two_bit.clone())).unwrap();
        let inw = builder.input_wires().exactly_one().unwrap();
        let outw1 = builder.add_dataflow_op(not_op(), [inw]).unwrap().out_wire(0);
        let outw2 = builder
            .add_dataflow_op(and_op(), [inw, outw1])
            .unwrap()
            .outputs();
        let outw = [outw1]
            .into_iter()
            .chain(outw2);
        let h = builder.finish_hugr_with_outputs(outw, &EMPTY_REG).unwrap();
        let view = SiblingGraph::<DfgID>::new(&h, h.root());
        let subg = SiblingSubgraph::try_new_dataflow_subgraph(&view).unwrap();
        assert_eq!(subg.nodes().len(), 1);
    }

encodes the computation
graph TD
IN --> NOT
IN --> AND
NOT --> AND
NOT --> OUT
AND --> OUT

This currently fails with the error NotConvex as the output of the NOT is taken to be an output of the subgraph, even though one copy is actually still within the subgraph and used for the AND gate.

@lmondada I think the final assertion in that test should be
        assert_eq!(subg.nodes().len(), 2);

Am I right?

Yes, absolutely!


LibertyDev error Cannot get property 'baseName' on null object with spring boot app

Using pugins:

io.openliberty.tools:liberty-gradle-plugin:3.0 - in buildscript block
"org.springframework.boot" version "1.5.10.RELEASE"
groovy
war

I have a spring boot app (with no embedded server) that we assemble into a war and run on liberty with the tag <webApplication location="spring-boot.war"/> using the libertyStart task successfully.
I want to run this same spring-boot.war with the libertyDev task. What configuration do I need to make this happen? Configuration currently:

server.xml: <webApplication location="spring-boot.war.xml"/>
build.gradle:

liberty {
    server {
        name = "LibertyProjectServer"
        looseApplication = true
        stripVersion = true

        features {
            acceptLicense = true
        }
    }
    install {
        runtimeUrl = "some custom url to com/ibm/websphere/appserver/runtime/wlp-javaee7/<IP_ADDRESS>/wlp-javaee7-<IP_ADDRESS>.zip"
    }
}

Currently if I let the libertyDev task run with no configuration it fails the build with an exception in in the deploy task:
Execution failed for task ':deploy'.
> Cannot get property 'baseName' on null object

Running with stacktrace:
Execution failed for task ':libertyDev'.
> Could not execute build using Gradle distribution 'gradle-6.2.2-all.zip'.

* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':libertyDev'.
	[This part of stack trace omitted because of irrelevance]
Caused by: java.lang.NullPointerException: Cannot get property 'baseName' on null object
	at io.openliberty.tools.gradle.tasks.DeployTask.installProjectArchive(DeployTask.groovy:119)
	at io.openliberty.tools.gradle.tasks.DeployTask.installLooseApplication(DeployTask.groovy:255)
	at io.openliberty.tools.gradle.tasks.DeployTask.installProject(DeployTask.groovy:147)
	at io.openliberty.tools.gradle.tasks.DeployTask$_installMultipleApps_closure5.doCall(DeployTask.groovy:111)
	at io.openliberty.tools.gradle.tasks.DeployTask.installMultipleApps(DeployTask.groovy:110)
	at io.openliberty.tools.gradle.tasks.DeployTask.deploy(DeployTask.groovy:75)
	[Next part of stack trace omitted because of irrelevance]

Looking at the source code of the plugin, it seems it fails to set springBootTask properly in this findSpringBootTask method resulting in this line of code archiveBaseName = springBootTask.baseName having a null pointer exception.
Any help is greatly appreciated!

@hearit-danny libertyDev and the loose application configuration currently do not support Spring Boot applications.
However, this particular error occurs when the Spring Boot plugin is specified using plugins DSL.  Could you try specifying the plugin in the buildscript section instead?  For example:
buildscript {
    repositories {
        mavenCentral()
        maven {
            url "https://plugins.gradle.org/m2/"
        }
    }
    dependencies {
        classpath 'io.openliberty.tools:liberty-gradle-plugin:3.0'
        classpath "org.springframework.boot:spring-boot-gradle-plugin:1.5.10.RELEASE"
    }
}
apply plugin: "org.springframework.boot"
apply plugin: 'liberty'

Thanks for the reply and a solution to the error. If a spring boot application is not supported for loose application configuration, what does the spring boot code in the deploy task indicate? What supported spring boot wise?
Thanks!

To clarify, the libertyDev task currently does not support Spring Boot applications because libertyDev requires loose application configuration.  For Spring Boot applications, the looseApplication flag will be ignored and will be set to false.  This can be seen in the output if using -i with the gradle command:
Loose application configuration is not supported for packaging type springboot. The project artifact will be installed as an archive file.
You can use the deploy and libertyStart tasks with Spring Boot application war files.


How to call the get_entity or ResolveUsername?

I need to get User object from user_id
Could find and methods by seing repo
Pls help.
Im new to Golang was previously using python based telethon lib.

Hello. You may use Bot.ChatByID(123456)

thkns


fix(deps): update aws-sdk-go-v2 monorepo

This PR contains the following updates:

Package
Type
Update
Change

github.com/aws/aws-sdk-go-v2/config
require
patch
v1.27.17 -> v1.27.18

github.com/aws/aws-sdk-go-v2/credentials
require
patch
v1.17.17 -> v1.17.18

github.com/aws/aws-sdk-go-v2/service/ecr
require
patch
v1.28.4 -> v1.28.5

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

ℹ Artifact update notice
File name: go.mod
In order to perform the update(s) described in the table above, Renovate ran the go get command, which resulted in the following additional change(s):

8 additional dependencies were updated

Details:

Package
Change

github.com/aws/aws-sdk-go-v2
v1.27.1 -> v1.27.2

github.com/aws/aws-sdk-go-v2/feature/ec2/imds
v1.16.4 -> v1.16.5

github.com/aws/aws-sdk-go-v2/internal/configsources
v1.3.8 -> v1.3.9

github.com/aws/aws-sdk-go-v2/internal/endpoints/v2
v2.6.8 -> v2.6.9

github.com/aws/aws-sdk-go-v2/service/internal/presigned-url
v1.11.10 -> v1.11.11

github.com/aws/aws-sdk-go-v2/service/sso
v1.20.10 -> v1.20.11

github.com/aws/aws-sdk-go-v2/service/ssooidc
v1.24.4 -> v1.24.5

github.com/aws/aws-sdk-go-v2/service/sts
v1.28.11 -> v1.28.12


--ignore / -i doesn't work

Using the ignore option does nothing at all, no directory specified is ignored, and no error is thrown either. Is this functionality implemented?
C:\wamp\bin\php\php5.3.1>php.exe "C:\docblox\bin\docblox.php" project:run -d "C:\test" -t "C:\docbloxoutput2" --force -v --ignore "c:\test\test"

2011-04-19T13:32:14+00:00 INFO (6): [2.26mb]: Starting to process 2 files
2011-04-19T13:32:14+00:00 INFO (6): [2.26mb]: Starting to parse file: C:\test\test\newtest.php
2011-04-19T13:32:14+00:00 INFO (6): [2.81mb]: Starting to parse file: C:\test\test.php
*snip*
2011-04-19T13:32:14+00:00 INFO (6): [3.29mb]: Processing the file: \test\newtest.php as C:\docbloxoutput2\test_newtest.html
2011-04-19T13:32:14+00:00 INFO (6): [3.29mb]: Processing the file: \test.php as C:\docbloxoutput2\test.html

@dpkirchner That helped me, thanks! (File was vendor/symfony/finder/Finder.php.)


fix: Changed admin.database to use database-compat package

The @firebase/database package now exposes the modular API since the v9 JS SDK release. Admin SDK should upgrade to the latest dependencies, using the @firebase/database-compat package as the new intermediary. @Feiyang1 is working on some changes in the JS SDK side to make this work smoothly.

Resolves #1434


Error with df.ta.strategy("All")

Which version are you running? The lastest version is on Github. Pip is for major releases.
Python: 3.8.6
pandas-ta: 0.2.23b0

Upgrade.
done

Describe the bug
A clear and concise description of what the bug is.
These examples fails on the latest version:
The Default Strategy is the ta.AllStrategy. The following are equivalent:
df.ta.strategy()
df.ta.strategy("All")
df.ta.strategy(ta.AllStrategy)
Specifying specific categories work (['candles', 'momentum', 'overlap', 'performance', 'statistics', 'trend', 'volatility', 'volume'])
These have no effect on the result:
dataframe.ta.mp = True
dataframe.ta.mp = False
Error:
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/opt/tradebot/freqtrade/.env/lib/python3.8/site-packages/pandas_ta/core.py", line 432, in _mp_worker
    return getattr(self, method)(*args, **kwargs)
TypeError: crossed() got an unexpected keyword argument 'append'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "./.env/lib/python3.8/site-packages/pandas_ta/core.py", line 682, in strategy
    [self._post_process(r, **kwargs) for r in results]
  File "./.env/lib/python3.8/site-packages/pandas_ta/core.py", line 682, in <listcomp>
    [self._post_process(r, **kwargs) for r in results]
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 448, in <genexpr>
    return (item for chunk in result for item in chunk)
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 868, in next
    raise value
TypeError: crossed() got an unexpected keyword argument 'append'

To Reproduce
Download sample data and try the above examples.
Expected behavior
All indicators added to dataframe.
Thanks for using Pandas TA!
Thanks for a great TA library!

Hello @Fredrik81,
The Latest Version is the Github version 0.2.42b0. You have the last stable version 0.2.23b0 from pypi.
Please upgrade and give it another shot. If it still does not work, we can go from there.
$ pip install -U git+https://github.com/twopirllc/pandas-ta

Kind Regards,
KJ

Same issue :-(
pip install -U git+https://github.com/twopirllc/pandas-ta
Collecting git+https://github.com/twopirllc/pandas-ta
  Cloning https://github.com/twopirllc/pandas-ta to /tmp/pip-req-build-72f7uy_2
  Running command git clone -q https://github.com/twopirllc/pandas-ta /tmp/pip-req-build-72f7uy_2
Requirement already satisfied: pandas in ./.env/lib/python3.8/site-packages (from pandas-ta==0.2.42b0) (1.2.2)
Requirement already satisfied: pytz>=2017.3 in ./.env/lib/python3.8/site-packages (from pandas->pandas-ta==0.2.42b0) (2020.1)
Requirement already satisfied: numpy>=1.16.5 in ./.env/lib/python3.8/site-packages (from pandas->pandas-ta==0.2.42b0) (1.20.1)
Requirement already satisfied: python-dateutil>=2.7.3 in ./.env/lib/python3.8/site-packages (from pandas->pandas-ta==0.2.42b0) (2.8.1)
Requirement already satisfied: six>=1.5 in ./.env/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas->pandas-ta==0.2.42b0) (1.15.0)

$ python
Python 3.8.6 (default, Sep 25 2020, 09:36:53)
[GCC 10.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas_ta as ta
>>> ta.version
'0.2.42b0'

@Fredrik81,
Hmmm... can you provide more reproducible code with sample data?
Thanks,
KJ

@twopirllc
Hmm i did a smaller manual script and it worked:
import ccxt
import pandas_ta as ta
import pandas as pd

exchange = ccxt.binance()

candles = exchange.fetch_ohlcv('BTC/USDT', timeframe='5m')

df = pd.DataFrame(candles, columns=['date', 'open', 'high', 'low', 'close', 'volume'])
df['date'] = pd.to_datetime(df['date'], unit='ms')
df = df.set_index('date')

df.ta.strategy()
print(df.tail(5))

So i could use some help at least for troubleshooting :-) thinking the secret is in that one of the indicators might not be in the categories?
['candles', 'momentum', 'overlap', 'performance', 'statistics', 'trend', 'volatility', 'volume']
As all of these are working if i add them one by one.. what indicator am i missing?

@Fredrik81,
Thanks for giving me something to work with. It is weird because you are getting a TypeError: crossed() got an unexpected keyword argument 'append'. There is no crossed() in core.py that can be called. Anyhow, I will take a look at the code.
Thanks,
KJ

@Fredrik81,
I had no problem with that script nor using df.ta.strategy("all"). I need some code that breaks.
Thanks,
KJ

I use it in combination to freqtrade platform.
        print(type(dataframe))
        print('-------------------')
        print(dataframe.dtypes)
        print('-------------------')
        print(dataframe.tail(1))
        print('-------------------')

Return:
<class 'pandas.core.frame.DataFrame'>
-------------------
date      datetime64[ns, UTC]
open                  float64
high                  float64
low                   float64
close                 float64
volume                float64
dtype: object
-------------------
                           date     open     high      low    close    volume
12116 2021-02-12 00:00:00+00:00  27.8645  27.9951  27.7551  27.7933  51591.83

@Fredrik81,
That is weird. There is no crossed() in Pandas TA but somehow FreqTrade might be injecting into the dataframe before dataframe.ta.strategy() is called. 🤷🏼♂
Could you add the following print statement to your local Pandas TA ./freqtrade/.env/lib/python3.8/site-packages/pandas_ta/core.py. I want to see if crossed() shows up in the multiprocessing pool somehow like the error suggests. If it does, please give me the full printed list.
    def _mp_worker(self, arguments: tuple):
        """Multiprocessing Worker to handle different Methods."""
        method, args, kwargs = arguments

        print(f"{method}({args}, {kwargs})") # Add this line
        if method != "ichimoku":
            return getattr(self, method)(*args, **kwargs)
        else:
            return getattr(self, method)(*args, **kwargs)[0]

Also can you share more of the file: "./freqtrade/user_data/strategies/Indicator.py" please? Perhaps something else is occurring that we are not seeing. Are you also importing qtpylib or talib in Indicator.py?
Thanks,
KJ

Here are the output from the print:
-------------------
aberration((), {'append': True})
apo((), {'append': True})
cci((), {'append': True})
cmf((), {'append': True})
dema((), {'append': True})
eri((), {'append': True})
aroon((), {'append': True})
hma((), {'append': True})
donchian((), {'append': True})
kc((), {'append': True})
cmo((), {'append': True})
coppock((), {'append': True})
dpo((), {'append': True})
fisher((), {'append': True})
efi((), {'append': True})
ema((), {'append': True})
entropy((), {'append': True})
eom((), {'append': True})
er((), {'append': True})
atr((), {'append': True})
macd((), {'append': True})
mad((), {'append': True})
fwma((), {'append': True})
ha((), {'append': True})
cdl_doji((), {'append': True})
cdl_inside((), {'append': True})
crossed((), {'append': True})
midprice((), {'append': True})
mom((), {'append': True})
accbands((), {'append': True})
natr((), {'append': True})
ad((), {'append': True})
adosc((), {'append': True})
massi((), {'append': True})
adx((), {'append': True})
mcgd((), {'append': True})
kdj((), {'append': True})
keltner_channel((), {'append': True})
pgo((), {'append': True})
bbands((), {'append': True})
bias((), {'append': True})
bollinger_bands((), {'append': True})
pvt((), {'append': True})
pwma((), {'append': True})
qqe((), {'append': True})
cfo((), {'append': True})
nvi((), {'append': True})
amat((), {'append': True})
ppo((), {'append': True})
psar((), {'append': True})
ao((), {'append': True})
aobv((), {'append': True})
obv((), {'append': True})
hull_moving_average((), {'append': True})
rolling_max((), {'append': True})
rvgi((), {'append': True})
squeeze((), {'append': True})
rvi((), {'append': True})
ohlc4((), {'append': True})
pdist((), {'append': True})
percent_return((), {'append': True})
tdi((), {'append': True})
typical_price((), {'append': True})
wcp((), {'append': True})
weighted_bollinger_bands((), {'append': True})
zlwma((), {'append': True})
heikinashi((), {'append': True})
psl((), {'append': True})
ssf((), {'append': True})
pvi((), {'append': True})
pvo((), {'append': True})
pvol((), {'append': True})
pvr((), {'append': True})
session((), {'append': True})
median((), {'append': True})
mfi((), {'append': True})
mid_price((), {'append': True})
stdev((), {'append': True})
stoch((), {'append': True})
stochrsi((), {'append': True})
supertrend((), {'append': True})
swma((), {'append': True})
t3((), {'append': True})
cg((), {'append': True})
chop((), {'append': True})
chopiness((), {'append': True})
qstick((), {'append': True})
quantile((), {'append': True})
returns((), {'append': True})
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/opt/tradebot/freqtrade/.env/lib/python3.8/site-packages/pandas_ta/core.py", line 432, in _mp_worker
    return getattr(self, method)(*args, **kwargs)
TypeError: crossed() got an unexpected keyword argument 'append'
"""

I will send more of the file tomorrow. And yes i'm importing both qtpylib and talib

@Fredrik81,

Yes i'm importing both qtpylib and talib but did a quick test to remove them and it did not change anything.

That's odd. Well somehow qtpylib, though not imported, is appending it's indicators to the DataFrame.
Qtpylib Indicators

In your Indicator.py code, could you add:
    print(type(dataframe))
    print('-------------------')
    print(dataframe.dtypes)
    print('-------------------')
    print(dataframe.tail(1))
    print('-------------------')
    ta_indicators = list((x for x in dir(dataframe.ta) if not x.startswith("_") and not x.endswith("_")))
    print(", ".join(ta_indicators))
    # dataframe.ta.strategy()

When you get a chance, could you please share the output for both cases:

importing both qtpylib and pandas_ta
only importing pandas_ta.

Thanks,
KJ

@twopirllc
Sure :-)
Here is the code:
        dataframe.ta.mp = True
        #print(dataframe.columns)
        #print(dataframe.ta.adx().tail(2))
        print(type(dataframe))
        print('-------------------')
        print(dataframe.dtypes)
        print('-------------------')
        print(dataframe.tail(1))
        print('-------------------')
        ta_indicators = list((x for x in dir(dataframe.ta) if not x.startswith("_") and not x.endswith("_")))
        print(", ".join(ta_indicators))

Without qtpylib:
# --- Do not remove these libs ---
import numpy as np
#from numpy.lib.function_base import append  # noqa
import pandas as pd  # noqa
from pandas import DataFrame

from freqtrade.strategy.interface import IStrategy

# --------------------------------
# Add your lib to import here
#import talib.abstract as ta
import pandas_ta as pta
#import freqtrade.vendor.qtpylib.indicators as qtpylib

Output:
<class 'pandas.core.frame.DataFrame'>
-------------------
date      datetime64[ns, UTC]
open                  float64
high                  float64
low                   float64
close                 float64
volume                float64
dtype: object
-------------------
                           date     open     high      low    close    volume
12116 2021-02-12 00:00:00+00:00  27.8645  27.9951  27.7551  27.7933  51591.83
-------------------
aberration, above, above_value, accbands, ad, adjusted, adosc, adx, amat, ao, aobv, apo, aroon, atr, bbands, below, below_value, bias, bollinger_bands, bop, brar, categories, cci, cdl_doji, cdl_inside, cfo, cg, chop, chopiness, cksp, cmf, cmo, constants, coppock, cores, cross, cross_value, crossed, crossed_above, crossed_below, datetime_ordered, decay, decreasing, dema, donchian, dpo, efi, ema, entropy, eom, er, eri, fisher, fwma, ha, heikinashi, hilo, hl2, hlc3, hma, hull_moving_average, ibs, ichimoku, implied_volatility, increasing, indicators, inertia, kama, kc, kdj, keltner_channel, kst, kurtosis, linreg, log_return, log_returns, long_run, macd, mad, massi, mcgd, median, mfi, mid_price, midpoint, midprice, mom, mp, natr, nvi, obv, ohlc4, pdist, percent_return, pgo, ppo, psar, psl, pvi, pvo, pvol, pvr, pvt, pwma, qqe, qstick, quantile, returns, reverse, rma, roc, rolling_max, rolling_mean, rolling_min, rolling_std, rolling_vwap, rolling_weighted_mean, rsi, rsx, rvgi, rvi, session, short_run, sinwma, skew, slope, sma, smi, squeeze, ssf, stdev, stoch, stochrsi, strategy, supertrend, swma, t3, tdi, tema, thermo, trend_return, trima, trix, true_range, tsi, ttm_trend, typical_price, ui, uo, variance, version, vidya, vortex, vp, vwap, vwma, wcp, weighted_bollinger_bands, willr, wma, zlema, zlhma, zlma, zlsma, zlwma, zscore

With qtpylib:
# --- Do not remove these libs ---
import numpy as np
#from numpy.lib.function_base import append  # noqa
import pandas as pd
from pandas import DataFrame

from freqtrade.strategy.interface import IStrategy

# --------------------------------
# Add your lib to import here
#import talib.abstract as ta
import pandas_ta as pta
import freqtrade.vendor.qtpylib.indicators as qtpylib

<class 'pandas.core.frame.DataFrame'>
-------------------
date      datetime64[ns, UTC]
open                  float64
high                  float64
low                   float64
close                 float64
volume                float64
dtype: object
-------------------
                           date     open     high      low    close    volume
12116 2021-02-12 00:00:00+00:00  27.8645  27.9951  27.7551  27.7933  51591.83
-------------------
aberration, above, above_value, accbands, ad, adjusted, adosc, adx, amat, ao, aobv, apo, aroon, atr, bbands, below, below_value, bias, bollinger_bands, bop, brar, categories, cci, cdl_doji, cdl_inside, cfo, cg, chop, chopiness, cksp, cmf, cmo, constants, coppock, cores, cross, cross_value, crossed, crossed_above, crossed_below, datetime_ordered, decay, decreasing, dema, donchian, dpo, efi, ema, entropy, eom, er, eri, fisher, fwma, ha, heikinashi, hilo, hl2, hlc3, hma, hull_moving_average, ibs, ichimoku, implied_volatility, increasing, indicators, inertia, kama, kc, kdj, keltner_channel, kst, kurtosis, linreg, log_return, log_returns, long_run, macd, mad, massi, mcgd, median, mfi, mid_price, midpoint, midprice, mom, mp, natr, nvi, obv, ohlc4, pdist, percent_return, pgo, ppo, psar, psl, pvi, pvo, pvol, pvr, pvt, pwma, qqe, qstick, quantile, returns, reverse, rma, roc, rolling_max, rolling_mean, rolling_min, rolling_std, rolling_vwap, rolling_weighted_mean, rsi, rsx, rvgi, rvi, session, short_run, sinwma, skew, slope, sma, smi, squeeze, ssf, stdev, stoch, stochrsi, strategy, supertrend, swma, t3, tdi, tema, thermo, trend_return, trima, trix, true_range, tsi, ttm_trend, typical_price, ui, uo, variance, version, vidya, vortex, vp, vwap, vwma, wcp, weighted_bollinger_bands, willr, wma, zlema, zlhma, zlma, zlsma, zlwma, zscore

@Fredrik81,
Thanks!
Are there other indicators in the same directory as "./freqtrade/user_data/strategies/Indicator.py" that import freqtrade.vendor.qtpylib.indicators as qtpylib? Or any file in "./freqtrade/"?
It could be stuck in the import cache. Does it still happen after clearing it?
Thanks,
KJ

@twopirllc
That's it :-) I remove all other strategy files.
It got past that error and now has a new error in the "dataframe.ta.strategy()". But I guess I should not ask for more assistance and take more time here.
Thanks a lot!

@Fredrik81,

That's it :-) I remove all other strategy files.

Excellent! Thanks for your assistance and patience with debugging why the DataFrame was getting extended prior to ta.strategy() executing.

AttributeError: 'RangeIndex' object has no attribute 'to_period'

But I guess I should not ask for more assistance and take more time here.

No worries. I recommend changing your DataFrame index to a DatatimeIndex so Pandas can anchor the period correctly. See help(ta.vwap).
import pandas as pd
import pandas_ta as ta

dataframe = # your ohlcv
dataframe.set_index(pd.DatetimeIndex(dataframe["date"]), inplace=True, drop=True)
dataframe.drop(columns="date", axis=1, inplace=True)

dataframe.ta.strategy()
print(dataframe)

Let me know if that works.
Thanks,
KJ

yeah that did the tick now it's working perfectly!

@Fredrik81,
Excellent! Glad we were able to get it to work with freqtrade. Please let me know how it performs with it.
Kind Regards,
KJ

@twopirllc
Well why i'm switching is because pandas-ta seems to have better performance and feels a bit more light weight than ta-lib.
What i'm working on at the moment is combining freqtrade, tensorflow and stable_baselines to see how i can get re-enforced learning to work with it.
If i could wish for one thing it would be namings of the pandas columns like "[indicator]_[settings]". I see the point of the way it's implemented in pandas-ta but it can be acting strange in atomization etc. perhaps there is already a way to make it take away the _[settings] part in the name creation but at the moment i just do this to take out some of the characters that become a problem for me:
dataframe.columns = dataframe.columns.str.replace('[#,@,&,.]', '')

@Fredrik81,

Well why i'm switching is because pandas-ta seems to have better performance and feels a bit more light weight than ta-lib. What i'm working on at the moment is combining freqtrade, tensorflow and stable_baselines to see how i can get re-enforced learning to work with it.

Gotcha

If i could wish for one thing it would be namings of the pandas columns like "[indicator]_[settings]". I see the point of the way it's implemented in pandas-ta but it can be acting strange in atomization etc. perhaps there is already a way to make it take away the _[settings] part in the name creation but at the moment i just do this to take out some of the characters that become a problem for me:

Yeah I understand. For the resultant column names, "[indicator]_[settings]" seemed to be the least problematic and yet descriptive way to identify them. Obviously the "." column names from indicators with float parameters makes the process less smooth in the AI/ML pipeline. I am open to suggestions for better column naming.
Yes, this works and is a simple command to adjust column names in the feature generation post-processing  phase of the pipeline. I have thought about adding something similar to Pandas TA but unsure if it is needed as post-processing requirements may vary.
dataframe.columns = dataframe.columns.str.replace('[#,@,&,.]', '')

Kind Regards,
KJ


Non-ascii character in doc causing issues file_bundle_utils.py

Bug Report
Incorrect Functionality and General Questions
I noticed a non-ascii char in "file_bundle_utils.py"
To Reproduce

otiopluginfo
...
SyntaxError: Non-ASCII character '\xc2' in file /scratch/opentimelineio-0.14dev1_pixar.02/lib/python2.7/site-packages/opentimelineio/adapters/file_bundle_utils.py on line 113, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details

Closed by #939


Radix Sorting

Added Radix sorting with readme File

Please do the following changes @D3v3sh5ingh

I am Creating a new pr with the required changes.


multiple stream_cmds

would be awesome

Trying to get something like stream_cmd = tee >(python sinks/foobar.py) | python sinks/graphite.py <IP_ADDRESS> 2003 '' working, but it seems like statsite doesn't support subshell syntax here. Are subshells possible w/ different syntax? Or is the only option here to write a wrapper sink?
Our goal is to manage sink configuration externally (via chef) w/o requiring code changes to add, remove, or reconfigure sinks. That said, we could always write the sink dynamically via chef, but that isn't ideal.

You may be interested in the twitter-forks/statsite version which features multiple sink support.
I haven't upstreamed any changes yet pending doc cleanups.

@dpkp That is also a feature of shells, while statsite just does a fork/exec. You could try to prefix that whole command with "/bin/bash -c" and run it under a shell.

@theatrus: wasn't aware of twitter-forks/statsite, will take a look.
@armon: of course! thanks for the hint.


Language dropdown display issues after #28

Looks like the language dropdown needs an update after #28. Sometimes it shows only English.

I believe this is actually related to some of the pages not being translated yet.

Correct: If a post does not have a translation, it will only show the English dropdown.

Can't we just put the English text as placeholder for a translation?
I mean, it is really confusing to see two different websites when switching the language:

It just needs the translations to be made in _data/navigation.yml (and other navs to be sync'd up with the en:  tree).

I have pushed the current en: structure to the other languages.

Nice, c40eebb2c3084c045d3fbc25623a3da9bfa98ff9 fixed my issue.

Would you mind doing the German translation for those menus?


Feat/field database apps bind name

Hello @ylgnerbecton! Thanks for updating this PR.

In the file dbaas/logical/migrations/0036_auto__add_field_databasehistory_apps_bind_name.py:

Line 17:5: E303 too many blank lines (2)
Line 22:5: E303 too many blank lines (2)
Line 25:101: E501 line too long (115 > 100 characters)
Line 27:101: E501 line too long (131 > 100 characters)
Line 28:101: E501 line too long (130 > 100 characters)
Line 29:101: E501 line too long (130 > 100 characters)
Line 30:101: E501 line too long (126 > 100 characters)
Line 33:101: E501 line too long (110 > 100 characters)
Line 37:101: E501 line too long (101 > 100 characters)
Line 38:101: E501 line too long (115 > 100 characters)
Line 39:101: E501 line too long (116 > 100 characters)
Line 42:101: E501 line too long (102 > 100 characters)
Line 43:101: E501 line too long (190 > 100 characters)
Line 44:101: E501 line too long (101 > 100 characters)
Line 45:101: E501 line too long (119 > 100 characters)
Line 46:101: E501 line too long (111 > 100 characters)
Line 47:101: E501 line too long (129 > 100 characters)
Line 52:101: E501 line too long (102 > 100 characters)
Line 53:101: E501 line too long (158 > 100 characters)
Line 56:101: E501 line too long (187 > 100 characters)
Line 58:101: E501 line too long (123 > 100 characters)
Line 64:101: E501 line too long (111 > 100 characters)
Line 65:101: E501 line too long (103 > 100 characters)
Line 66:101: E501 line too long (107 > 100 characters)
Line 67:101: E501 line too long (180 > 100 characters)
Line 72:101: E501 line too long (110 > 100 characters)
Line 73:101: E501 line too long (106 > 100 characters)
Line 75:101: E501 line too long (195 > 100 characters)
Line 76:101: E501 line too long (105 > 100 characters)
Line 79:101: E501 line too long (161 > 100 characters)
Line 86:101: E501 line too long (136 > 100 characters)
Line 87:101: E501 line too long (115 > 100 characters)
Line 88:101: E501 line too long (145 > 100 characters)
Line 92:101: E501 line too long (113 > 100 characters)
Line 93:101: E501 line too long (111 > 100 characters)
Line 97:101: E501 line too long (125 > 100 characters)
Line 98:101: E501 line too long (125 > 100 characters)
Line 99:101: E501 line too long (115 > 100 characters)
Line 100:101: E501 line too long (185 > 100 characters)
Line 101:101: E501 line too long (104 > 100 characters)
Line 103:101: E501 line too long (181 > 100 characters)
Line 107:101: E501 line too long (105 > 100 characters)
Line 108:101: E501 line too long (205 > 100 characters)
Line 109:101: E501 line too long (106 > 100 characters)
Line 110:101: E501 line too long (187 > 100 characters)
Line 112:101: E501 line too long (107 > 100 characters)
Line 113:101: E501 line too long (168 > 100 characters)
Line 114:101: E501 line too long (111 > 100 characters)
Line 119:101: E501 line too long (107 > 100 characters)
Line 123:101: E501 line too long (115 > 100 characters)
Line 124:101: E501 line too long (104 > 100 characters)
Line 137:101: E501 line too long (115 > 100 characters)
Line 138:101: E501 line too long (156 > 100 characters)
Line 140:101: E501 line too long (142 > 100 characters)
Line 141:101: E501 line too long (110 > 100 characters)
Line 145:101: E501 line too long (115 > 100 characters)
Line 146:101: E501 line too long (104 > 100 characters)
Line 149:101: E501 line too long (103 > 100 characters)
Line 151:101: E501 line too long (110 > 100 characters)
Line 155:101: E501 line too long (102 > 100 characters)
Line 157:101: E501 line too long (115 > 100 characters)
Line 158:101: E501 line too long (147 > 100 characters)
Line 159:101: E501 line too long (101 > 100 characters)
Line 161:101: E501 line too long (105 > 100 characters)
Line 163:101: E501 line too long (126 > 100 characters)
Line 164:101: E501 line too long (105 > 100 characters)
Line 165:101: E501 line too long (107 > 100 characters)
Line 166:101: E501 line too long (121 > 100 characters)
Line 167:101: E501 line too long (123 > 100 characters)
Line 168:101: E501 line too long (137 > 100 characters)
Line 169:101: E501 line too long (111 > 100 characters)
Line 170:101: E501 line too long (137 > 100 characters)
Line 174:101: E501 line too long (115 > 100 characters)
Line 177:101: E501 line too long (110 > 100 characters)
Line 183:101: E501 line too long (115 > 100 characters)
Line 184:101: E501 line too long (126 > 100 characters)
Line 185:101: E501 line too long (205 > 100 characters)
Line 186:101: E501 line too long (217 > 100 characters)
Line 187:101: E501 line too long (122 > 100 characters)
Line 188:101: E501 line too long (126 > 100 characters)
Line 189:101: E501 line too long (176 > 100 characters)
Line 190:101: E501 line too long (203 > 100 characters)
Line 191:101: E501 line too long (186 > 100 characters)
Line 193:101: E501 line too long (111 > 100 characters)
Line 197:101: E501 line too long (103 > 100 characters)
Line 198:101: E501 line too long (124 > 100 characters)
Line 199:101: E501 line too long (123 > 100 characters)
Line 200:101: E501 line too long (106 > 100 characters)
Line 201:101: E501 line too long (103 > 100 characters)
Line 202:101: E501 line too long (172 > 100 characters)
Line 203:101: E501 line too long (165 > 100 characters)
Line 204:101: E501 line too long (129 > 100 characters)
Line 206:101: E501 line too long (123 > 100 characters)
Line 207:101: E501 line too long (111 > 100 characters)
Line 208:101: E501 line too long (101 > 100 characters)
Line 212:101: E501 line too long (115 > 100 characters)
Line 213:101: E501 line too long (184 > 100 characters)
Line 215:101: E501 line too long (103 > 100 characters)
Line 217:101: E501 line too long (110 > 100 characters)
Line 221:101: E501 line too long (115 > 100 characters)
Line 222:101: E501 line too long (189 > 100 characters)
Line 226:101: E501 line too long (103 > 100 characters)
Line 227:101: E501 line too long (103 > 100 characters)
Line 228:101: E501 line too long (110 > 100 characters)
Line 231:101: E501 line too long (150 > 100 characters)
Line 232:101: E501 line too long (115 > 100 characters)
Line 233:101: E501 line too long (178 > 100 characters)
Line 234:101: E501 line too long (227 > 100 characters)
Line 238:101: E501 line too long (117 > 100 characters)
Line 239:101: E501 line too long (117 > 100 characters)
Line 240:101: E501 line too long (118 > 100 characters)
Line 241:101: E501 line too long (153 > 100 characters)
Line 242:101: E501 line too long (127 > 100 characters)
Line 243:101: E501 line too long (111 > 100 characters)
Line 244:101: E501 line too long (109 > 100 characters)
Line 246:101: E501 line too long (153 > 100 characters)
Line 250:101: E501 line too long (115 > 100 characters)
Line 251:101: E501 line too long (137 > 100 characters)
Line 254:101: E501 line too long (142 > 100 characters)
Line 255:101: E501 line too long (146 > 100 characters)
Line 257:101: E501 line too long (115 > 100 characters)
Line 258:101: E501 line too long (110 > 100 characters)
Line 262:101: E501 line too long (115 > 100 characters)
Line 265:101: E501 line too long (103 > 100 characters)
Line 266:101: E501 line too long (110 > 100 characters)
Line 270:101: E501 line too long (177 > 100 characters)
Line 271:101: E501 line too long (115 > 100 characters)
Line 274:101: E501 line too long (153 > 100 characters)
Line 275:101: E501 line too long (192 > 100 characters)
Line 277:101: E501 line too long (103 > 100 characters)
Line 281:101: E501 line too long (110 > 100 characters)
Line 286:101: E501 line too long (115 > 100 characters)
Line 287:101: E501 line too long (180 > 100 characters)
Line 291:101: E501 line too long (110 > 100 characters)
Line 294:101: E501 line too long (147 > 100 characters)
Line 295:101: E501 line too long (146 > 100 characters)
Line 296:101: E501 line too long (115 > 100 characters)
Line 297:101: E501 line too long (127 > 100 characters)
Line 298:101: E501 line too long (125 > 100 characters)
Line 300:101: E501 line too long (181 > 100 characters)
Line 303:101: E501 line too long (113 > 100 characters)
Line 304:101: E501 line too long (110 > 100 characters)
Line 308:101: E501 line too long (115 > 100 characters)
Line 309:101: E501 line too long (104 > 100 characters)
Line 310:101: E501 line too long (213 > 100 characters)
Line 311:101: E501 line too long (208 > 100 characters)
Line 312:101: E501 line too long (136 > 100 characters)
Line 313:101: E501 line too long (224 > 100 characters)
Line 314:101: E501 line too long (176 > 100 characters)
Line 320:101: E501 line too long (239 > 100 characters)
Line 321:101: E501 line too long (178 > 100 characters)
Line 322:101: E501 line too long (103 > 100 characters)
Line 323:101: E501 line too long (239 > 100 characters)
Line 325:101: E501 line too long (194 > 100 characters)
Line 326:101: E501 line too long (191 > 100 characters)
Line 327:101: E501 line too long (111 > 100 characters)
Line 328:101: E501 line too long (190 > 100 characters)
Line 335:101: E501 line too long (115 > 100 characters)
Line 338:101: E501 line too long (146 > 100 characters)
Line 340:101: E501 line too long (103 > 100 characters)
Line 341:101: E501 line too long (109 > 100 characters)
Line 345:101: E501 line too long (161 > 100 characters)
Line 346:101: E501 line too long (111 > 100 characters)
Line 351:101: E501 line too long (103 > 100 characters)
Line 353:101: E501 line too long (101 > 100 characters)
Line 360:101: E501 line too long (115 > 100 characters)
Line 361:101: E501 line too long (121 > 100 characters)
Line 362:101: E501 line too long (182 > 100 characters)
Line 363:101: E501 line too long (109 > 100 characters)
Line 366:101: E501 line too long (205 > 100 characters)
Line 367:101: E501 line too long (186 > 100 characters)
Line 368:101: E501 line too long (110 > 100 characters)
Line 373:101: E501 line too long (127 > 100 characters)
Line 374:101: E501 line too long (115 > 100 characters)
Line 377:101: E501 line too long (130 > 100 characters)
Line 380:101: E501 line too long (131 > 100 characters)
Line 381:101: E501 line too long (110 > 100 characters)
Line 385:32: W292 no newline at end of file


Words not stretched in Firefox

In Firefox, horizontal stretching of the SVG text does not seem to be working, as noticed by @mbennett-uoe on the IIIF Slack. The reason for this is that we use spacingAndGlyphs for the lengthAdjust parameter, which seems to be bugged in Firefox (although it's supposed to be supported: https://developer.mozilla.org/en-US/docs/Web/SVG/Attribute/lengthAdjust).

The reason seems to be that Firefox only supports the lengthAdjust parameter on <text> Elements, not <tspan> that we are using: https://bugzilla.mozilla.org/show_bug.cgi?id=890692
The current reasoning for using the text > tspan layout is the assumption that this is neccessary to ensure a correct reading/selection-order, especially with more non-straightforward layouts. We should probably verify that assumption and maybe only use text Elements for rendering, at least for Firefox/Gecko-based browsers.


Circle means for variables

I guess it would be useful to store the circle mean data for the variables in Level 4 as well. However, if we do, it would make sense to somehow weight the sondes depending on there distance to other sondes (or something similar).
Maybe we can brainstorm a bit how to best do that

Hi Helene, I agree. Indeed the circle mean data are very useful. I haven't yet looked at details of how L4 is being adopted from JOANNE, but there it is already included and the weighting is inherent in the regression. So, at every altitude, we will have an oversolved system, which means that for a least-squares fit, we get an estimate of the mean and gradients along x and y, which due to how the regression works already accounts for the differences in distances between sondes.

Ah nice,
we don't have a real level 4 yet, I just do some calculation (code is here), but I did not do the regression yet.
Also, in this I changed the structure a bit, because to me it does not really make sense to change the dimensions for everything; i.e. right now, level4 is just a extension of level3 with circle data added (and two new dimensions for the variables in circles being flight_id and position; which probably should be changed to circle_name). So currently there is
ta[sonde_id, alt] as well as omega[flight_id, position, alt] in the same dataset. There are also variables added flight[sonde_id] and c_name[sonde_id], which hold the information for each sonde which flight and circle it belongs to
To me that made sense, because sonde_id is still a unique identifier and a circle should definitely be defined by the circle name and flight_id. All other information can be derived as well (and the regressed mean variables would also get dimensions flight_id and position).

@hgloeckner the way I would do it is to have only (circle, alt) as dimensions in the level 4 data, since one still has access to the level3 which contains the sonde information, and the flight segmentation can be used to find out which sondes belong to which circle. That way the level 4 contains only the circle products, and if someone wants the sonde-by-sonde information they would have a look at the level 3.

it does not really make sense to change the dimensions for everything; i.e. right now, level4 is just a extension of level3 with circle data added (and two new dimensions for the variables in circles being flight_id and position; which probably should be changed to circle_name).

The idea behind L4 is supposed to be data that contains only area-averaged properties (and relevant ancillary variables). Therefore, the dimension would have to change from sonde_id to circle (or something else). In fact, the dimensions always change going from L2 to L3 to L4 (see Tables 7,8 and 9 in JOANNE), because for L2 it is the original independent dimension, i.e. time, for the gridded L3, it will be sonde_id and alt and for L4 it will be circle and alt. I would try and keep the dimensions names also the same as EUREC4A unless there is really a need for it. It is best to have consistency among datasets as much as possible.

@hgloeckner the way I would do it is to have only (circle, alt) as dimensions in the level 4 data, since one still has access to the level3 which contains the sonde information, and the flight segmentation can be used to find out which sondes belong to which circle. That way the level 4 contains only the circle products, and if someone wants the sonde-by-sonde information they would have a look at the level 3.

I agree with this. In fact, one can simplify it even further for the user to not have to go back to the flight segmentation files and add a variable in L4 itself. This is what we did in JOANNE. Quoting from the JOANNE paper, "The list of sonde IDs included in every circle is included as a variable along dimension sonde_id, making it easier to retrieve data (from L3) for the individual soundings in the circle."

Hm.. I am not so sure. A circle is uniquely defined by its flight id and name. Merging that to one dimension seems a bit odd for this campaign, as one major thing is to have those center, south and north circles in many flights.
Also, from talking with Julia it is a common use case to select all individual sondes for a circle after looking at the mean. I also don't really see a disadvantage of having the level 3 data in the same dataset (plus a flight_id and c_name variable, which could be combined with a single circle-id) the dataset is small enough that it would be feasible

I am not completely sure I understand what merging the circles to one dimension means. All the center, south and north circles will be part of Level-4. Each circle will be a coordinate in the circle dimension. Therefore every circle will have its area-averaged (by regression) values with its coordinate being the circle-id. Could you outline the structure of how you envision Level-4?

Also, from talking with Julia it is a common use case to select all individual sondes for a circle after looking at the mean.

Yes, of course, that is exactly why having a variable that just lists all sonde_ids in a circle is useful, so that it is easy to access from L3, with something like:
sondes_from_my_circle = l4.sel(circle = my_circle_id.sonde_id)
l3.sel(sonde_id = sondes_from_my_circle)

One could also think of having the L3 data in L4 itself, but that just is redundancy of data and creates potential points of error during development. I don't see the use of duplicating L3 in L4.

Also, from talking with Julia it is a common use case to select all individual sondes for a circle after looking at the mean.

Yes, of course, that is exactly why having a variable that just lists all sonde_ids in a circle is useful, so that it is easy to access from L3, with something like:
sondes_from_my_circle = l4.sel(circle = my_circle_id.sonde_id)
l3.sel(sonde_id = sondes_from_my_circle)

One could also think of having the L3 data in L4 itself, but that just is redundancy of data and creates potential points of error during development. I don't see the use of duplicating L3 in L4.

I think from a users-perspective it's nice to have all data together, especially the circle products and individual sondes. And using an ds = ds.where((ds.c_name==<name>) & (ds.flight==<flight>)) is a easier to select the sondes for a circle than opening another dataset and combining them (and also enables people to get all sondes for the circles labeled 'south', which might not have been a usecase in EURECA). We sometimes forget that other people are not deeply into the datastructure and what happens where. Judging from the questions here, people open level4, look at the circle products and wonder what individual sondes look like.
I would wager the benefits of easier usability higher than the risk of errors (especially since we are only adding circle data and not meddling with the original sonde data between lev3 and lev4). The redundancy would be a point if our circle+sonde data was bigger than 100MB, but as it is I think this duplication is not too bad.

I am not completely sure I understand what merging the circles to one dimension means. All the center, south and north circles will be part of Level-4. Each circle will be a coordinate in the circle dimension. Therefore every circle will have its area-averaged (by regression) values with its coordinate being the circle-id. Could you outline the structure of how you envision Level-4?

In my first lev4 I had two circle dimensions (flight_id and position), but you are right - just having a circle_id and maybe having the other two as variables is better


R-package isFalse function doesn't exists

https://github.com/Microsoft/LightGBM/blob/a5f43c9d559848aed71a870321d2729fe64beab1/R-package/R/callback.R#L362
I had an error with this specific line stating isFalse function didn't exist.
Replaced isFalse function with ! as done in other part of the script. Re-installed the package and now it's working fine.
I'm on Mac OS, followed these instructions
http://lightgbm.readthedocs.io/en/latest/Installation-Guide.html#macos
Then these ones
https://github.com/Microsoft/LightGBM/tree/master/R-package#install (only the export and R CMD INSTALL parts).

@Laurae2

I can confirm the isFalse issue and the fix by @jacquespeeters on Windows also.


LCFS - Transactions page & table

Ticket Completion: LCFS - Transactions Page & Table #190
Description:
Develop a new Transactions page featuring a detailed transactions table. This page should be accessible to both IDIR and BCeID users.  The table should include the following columns, arranged from left to right: ID, Compliance Period, Type, Units From, Units To, Number of Units, Value per Unit, Status, and Last Updated On. The default sorting of the table should be by the latest ID in descending order. Ensure that the column widths are adequately formatted to display transaction information clearly. Implement selectable and searchable drop-downs for columns associated with a look-up table. The table should be server-side to optimize performance. Additionally, disable search/filtering options for the 'Units From' and 'Units To' columns. A reference screenshot of a react table from TFRS can be found in a private Zenhub image.
Screenshot of react table from TFRS: Private Zenhub Image
Purpose and Benefit to User:
This feature enables users to efficiently view and manage transactions related to their organization, enhancing the user experience by providing a comprehensive and easy-to-navigate interface.
Acceptance Criteria:

Given I am a BCeID user, When I access the Transactions page, Then I should only be able to view transactions related to my organization.
Given I am an IDIR user, When I access the Transactions page, Then I should be able to view all transactions.

Development Checklist:

Implement the Transactions table using ag-grid for a dynamic and responsive design.
Utilize BCDataGridServer or equivalent for server-side pagination and data handling to ensure efficient performance and scalability.
Ensure that the table is responsive and accessible, with a focus on readability and ease of use.
Conduct thorough testing across various devices and browsers to ensure compatibility and functionality.

Notes:

Prioritize user experience by ensuring the interface is intuitive and user-friendly.
Implement appropriate security measures to ensure that users can only access data relevant to their organization.
Coordinate with the design team to ensure consistency with the overall application aesthetic and usability standards.

Please add your planning poker estimate with Zenhub @hamed-valiollahi

Currently the default sorting of the table is by the latest ID in ascending order (newest transaction goes to the end of the table).
The default sorting of the table should be by the latest ID in descending order.


removing-deprecated-cache-code

Fixes: https://github.com/mit-cml/appinventor-sources/issues/1830
Removed depricated DrawingCache functions
Updated variables accordingly

Can one of the admins verify this patch?

@AppInventorWorkerBee ok to test

Fixed Indentation and Typo


Add unix dgram support, custom type support (for e.g. seqpacket), fix docs

unix dgram sockets are common in e.g. syslog servers. Since we have std::os::unix::net::UnixDatagram it's easy to add support in the usual way.
unix seqpacket sockets are used by e.g. FreeBSD's devd. There is no support in the standard library, and no one leading crate for this (uds is very popluar but not the most actively maintained, e.g. supports tokio 0.2 still, while tokio-seqpacket is for 1.0 but only for tokio). Instead of per-crate features I think just adding support for custom types makes sense.

This makes sense to me.


[Bug]: tlsVerify does not work for sync

zot version
v2.1.0 (docker/helm)
Describe the bug
Hello!
It seems like the sync plugin does not respect the tlsVerify setting.
To reproduce

Configuration

{
  "storage": {
    "rootDirectory": "/var/lib/registry",
    "gc": true
  },
  "http": {
    "address": "<IP_ADDRESS>",
    "port": "5000"
  },
  "log": {
    "level": "debug"
  },
  "extensions": {
    "search": {
      "enable": true
    },
    "ui": {
      "enable": true
    },
    "sync": {
      "enable": true,
      "credentialsFile": "/etc/zot/auth.json",
      "registries": [
        {
          "urls": [
            "https://registry.p1ng.link/"
          ],
          "content": [
            {
              "prefix": "**",
              "destination": "/p1nglink"
            }
          ],
          "onDemand": true,
          "tlsVerify": false,
          "certDir": "/etc/zot/ca/",
          "onlySigned": false
        }
      ]
    }
  }
}

docker pull from the zot registry (goes to the p1nglink registry)
Log reads: {"level":"error","error":"Get \"https://registry.p1ng.link/v2/\": tls: failed to verify certificate: x509: certificate signed by unknown authority","url":"https://registry.p1ng.link/v2/","component":"sync","errorType":"*url.Error","goroutine":1,"caller":"zotregistry.dev/zot/pkg/extensions/sync/httpclient/client.go:272","time":"2024-07-21T14:47:22.930557924Z","message":"failed to make request"}

Expected behavior
Pull should succeed.
Screenshots
No response
Additional context
I also haven't been able to import the certificate into the container, any help in that direction would also be greatly appreciated (what file goes where)?

Hello @Jauchi. Thanks for trying zot!
I'm not sure what is the problem but I took a guess and pushed a patch for this: https://github.com/project-zot/zot/pull/2558
Can you try it please?
Also, can you post the logs please?
Thank you!

Hi there!
Sorry for the late response - I actually tried running your commit and didn't realize that your forked was a lot older than expected, so I was hitting errors I couldn't explain before (all good now and I learned a couple of things about docker as well ;).
Right, I ran 1.2.0 with your patch applied:

tlsVerify still has no effect
certDir also does nothing

Not quite sure what you mean by log, is this what you're looking for?
log.txt
helm_values.txt

Right, I ran 1.2.0 with your patch applied

Sorry, meant 2.1.0 - everything else still stands.

{"level":"error","error":"Get "https://registry.p1ng.link/v2/\": tls: failed to verify certificate: x509: certificate signed by unknown

@Jauchi the host above has an invalid certificate - the issuer is unknown and hence unsafe. Is this really what you want?
If so, would just download the CA cert and launch zot from a container.

Hi there!
Correct, that's exactly what I tried to do using certDir. When that failed, I set tlsVerify to false, which also did not work - hence the issue.
According to my understanding, the helm_values.txt should be set correctly, that's why I'm assuming it's a bug with zot.

https://github.com/project-zot/zot/pull/2558
^ does this fix your issue?

#2558 ^ does this fix your issue?
No, I don't think it does, doesn't seem to have any effect. Maybe I messed something up. Could you check whether or not you get an error with an invalid SSL certificate? https://untrusted-root.badssl.com/ as URL should work when skipping CA checks (but will fail because it's not a registry)

https://github.com/project-zot/zot/issues/2557#issuecomment-2243551441

I think the problem is. the http.client config for sync sets tls to disable, if tlsVerify is set to false.
https://github.com/project-zot/zot/blob/f735680ce253face0cc66c9abcd6687516a21c4c/pkg/extensions/sync/httpclient/client.go#L118-L123

@mariusbertram is right, I noticed the same thing. TLSEnabled is disabled if tlsVerify is set to false, which means that further downstream it'll return a HTTP client without SSL verification disabled:
https://github.com/project-zot/zot/blob/f735680ce253face0cc66c9abcd6687516a21c4c/pkg/common/http_client.go#L87-L102
I have proposed a fix in https://github.com/project-zot/zot/pull/2747, which will determine whether TLS should be enabled based on whether the upstream registry is a HTTPS URL.

The option with the custom ca doesnt work because it tries to add clientCertandclientKey. Which should only be set on mTLS` (authentication via cert) connections.
https://github.com/project-zot/zot/blob/51e779fab4a6bfb79987b6ae9bab32d3249c1b4e/pkg/common/http_client.go#L11-L33
A better aproach would be to set a path to a ca-bundle and add this bundle to the caCertPool.
I will try my best on an PR but go is not my best "friend"

https://github.com/project-zot/zot/pull/2628 is merged. @Jauchi, can you please try it out, can we close this issue?

#2747 is merged. @Jauchi, can you please try it out, can we close this issue?

Thanks for addressing the issue, however I'm quite busy atm, I'll check it out on the weekend.

#2747 is merged. @Jauchi, can you please try it out, can we close this issue?

Hi there!
Sorry for the late reply, I was trying to get the latest version built using docker, and for some reason, it always failed.
After about an hour, I noticed that there already a ready to use release candidate for me to try.
So I gave that a spin and now I don't get TLS errors anymore - just authentication issues.
But that seems to be a seperate problem.
Thanks for fixing, have a great week!


AfxIsValidAddress / AfxIsValidString do nothing

The code only checks for NULL. The documentation is false.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: bcb51741-60c9-fc00-3cd2-55f09698dae4
Version Independent ID: 0ee1bf38-0cee-ca78-a6f5-101529878ff9
Content: Diagnostic Services
Content Source: docs/mfc/reference/diagnostic-services.md
Product: visual-cpp
GitHub Login: @mikeblome
Microsoft Alias: mblome

@OfekShilon  Isn't this info in the Return Value section of each function? I.e. -- In non-debug builds, nonzero if lpsz is not NULL; otherwise 0.

Topic updated. The checks for valid memory space are performed by SAL annotations at compile-time.


Update docs to mention Win x86 and Linux ARM support

Advertise that the agent now targets .NET Core 2.1
Update the supported Linux distributions and versions
Advertise that we now support two new environments: Windows x86, and Linux ARM (e.g Raspberry Pi)

Hey @TingluoHuang, can you take a look at this PR whenever you have a chance? Thanks in advance! 😄


Query failing to return any results when filter is an equality check on strings (bad statistics in parquet)

With the attached file, running the query:
CREATE EXTERNAL TABLE t STORED AS PARQUET LOCATION 'test.parquet';
SELECT "adt" FROM t  WHERE "direction" = 'Two Way';

returns 0 rows, but:
SELECT "adt" FROM t  WHERE "direction" LIKE 'Two Way';

returns the expected 101 rows.
I noticed that if I disable the FilterPushDown optimization pass in Datafusion I also get the expected result.
test.zip

This seems to be something in the pruning logic:
❯ explain analyze SELECT "adt" FROM t  WHERE "direction" = 'Two Way';
+-------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| plan_type         | plan                                                                                |
+-------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Plan with Metrics | CoalescePartitionsExec, metrics=[output_rows=0, elapsed_compute=32.249µs]                                                                                |
|                   |   ProjectionExec: expr=[adt@1 as adt], metrics=[output_rows=0, elapsed_compute=NOT RECORDED]                                                                                |
|                   |     CoalesceBatchesExec: target_batch_size=4096, metrics=[output_rows=0, elapsed_compute=26.051µs]                                                                                |
|                   |       FilterExec: direction@0 = Two Way, metrics=[output_rows=0, elapsed_compute=NOT RECORDED]                                                                                |
|                   |         RepartitionExec: partitioning=RoundRobinBatch(16), metrics=[repart_time{inputPartition=0}=NOT RECORDED, send_time{inputPartition=0}=NOT RECORDED, fetch_time{inputPartition=0}=509.572µs]                          |
|                   |           ParquetExec: batch_size=8192, limit=None, partitions=[test.parquet], metrics=[row_groups_pruned{filename=test.parquet}=1, num_predicate_creation_errors=0, predicate_evaluation_errors{filename=test.parquet}=0] |
|                   |                                                                                |
+-------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set. Query took 0.002 seconds.

Already in ParquetExec there is no output (RepartitionExec doesn't hae to repartition any input row).

This sounds similar to something we hit in IOx (https://github.com/influxdata/influxdb_iox/issues/2153) which I ultimately tracked down to a bug in the parquet statistics generation: https://github.com/apache/arrow-rs/issues/641
So in this case, the statistics embedded in the parquet file for the  direction column are T:[min: Merged, max: Outgoing, num_nulls not defined], namely that the minimum value is "Merged" and the maximum value is "Outgoing" which I do not think is correct
$ parquet-tools meta test.parquet
file:        file:/Users/alamb/Downloads/test.parquet
creator:     UrbanLogiq
extra:       ARROW:schema = /////+gAAAAQAAAAAAAKAA4ADAALAAQACgAAABQAAAAAAAABBAAKAAwAAAAIAAQACgAAAAgAAAAIAAAAAAAAAAMAAAB8AAAAPAAAAAQAAACg////GAAAACAAAAAAAAACHAAAAAgADAAEAAsACAAAACAAAAAAAAABAAAAAAMAAABhZHQA1P///xQAAAAMAAAAAAAABQwAAAAAAAAAxP///wkAAABkaXJlY3Rpb24AAAAQABQAEAAAAA8ABAAAAAgAEAAAABgAAAAMAAAAAAAABRAAAAAAAAAABAAEAAQAAAAKAAAAdWxfbm9kZV9pZAAA

file schema: arrow_schema
--------------------------------------------------------------------------------
ul_node_id:  REQUIRED BINARY L:STRING R:0 D:0
direction:   REQUIRED BINARY L:STRING R:0 D:0
adt:         REQUIRED INT32 R:0 D:0

row group 1: RC:301 TS:3384 OFFSET:4
--------------------------------------------------------------------------------
ul_node_id:   BINARY ZSTD DO:4 FPO:1796 SZ:2143/3187/1.49 VC:301 ENC:RLE_DICTIONARY,PLAIN,RLE ST:[min: /ehIvdei+UGfkQ4Gy5fr1w==, max: zThqpswvY6fa3VHF4BKWfw==, num_nulls not defined]
direction:    BINARY ZSTD DO:2243 FPO:2311 SZ:195/177/0.91 VC:301 ENC:RLE_DICTIONARY,PLAIN,RLE ST:[min: Merged, max: Outgoing, num_nulls not defined]
adt:          INT32 ZSTD DO:2500 FPO:3159 SZ:1046/1503/1.44 VC:301 ENC:RLE_DICTIONARY,PLAIN,RLE ST:[min: 15, max: 23116, num_nulls not defined]

Which appears to be incorrect  for the data in test.parquet:
❯ select distinct direction from t order by direction;
+-----------+
| direction |
+-----------+
| Incoming  |
| Merged    |
| Outgoing  |
| Two Way   |
+-----------+

If you can't rewrite your parquet files, we could potentially add a "don't do parquet pruning" flag to ExecContext

That's really interesting; thanks for finding that @alamb !
We aren't doing anything special with our Parquet generation; we basically write them straight from Arrow (via the Rust libraries), so perhaps there's an underlying bug in the Rust Parquet statistics generation code that we can start investigating.
Being able to skip parquet pruning would be useful as well.
From a curiosity point of view, what should the min/max values be for string columns?

@maxburke  -- I fixed at least one bug in the rust parquet statistics generation ( PR was https://github.com/apache/arrow-rs/pull/643, which was first released in 5.2.0 it appears according to the changelog.
There may still be other bugs and if so I would love to help fix them

From a curiosity point of view, what should the min/max values be for string columns?

In this case, I would expect the min value to be "Incoming" and the max value to be "Two Way" (namely the min and max values if the data was sorted via unicode codepoint)

I think this one is resolved. Closing for now