K8s Cluster Nodes with GL 318.4 getting stuck/NotReady

What happened:
Running a node with GL 318.4 often gets stuck (as in becoming NotReady) when a certain amount of idling pods are running on it.
Nodes with GL 184.0.0 and the same load/amount-of-pods don't exhibit this behaviour.
When a node becomes NotReady we couldn't ssh into the node either and existing console session were stalling.
From K8s point of view we noticed various events, most prominently Kubelet stopped posting node status, but sometimes also from some "kernel-monitor":
Events:
  Type     Reason    Age    From            Message
  ----     ------    ----   ----            -------
  Warning  TaskHung  5m23s  kernel-monitor  INFO: task kswapd0:49 blocked for more than 120 seconds.
  Warning  TaskHung  5m22s  kernel-monitor  INFO: task jbd2/sda4-8:363 blocked for more than 121 seconds.
  Warning  TaskHung  5m21s  kernel-monitor  INFO: task dockerd:6500 blocked for more than 121 seconds.
  Warning  TaskHung  5m19s  kernel-monitor  INFO: task dockerd:9030 blocked for more than 122 seconds.
  Warning  TaskHung  5m17s  kernel-monitor  INFO: task dockerd:9551 blocked for more than 122 seconds.
  Warning  TaskHung  5m15s  kernel-monitor  INFO: task containerd:7378 blocked for more than 122 seconds.
  Warning  TaskHung  5m12s  kernel-monitor  INFO: task containerd:8414 blocked for more than 123 seconds.
  Warning  TaskHung  5m9s   kernel-monitor  INFO: task kubelet:3979 blocked for more than 123 seconds.
  Warning  TaskHung  5m6s   kernel-monitor  INFO: task kubelet:4211 blocked for more than 124 seconds.
  Warning  TaskHung  5m5s   kernel-monitor  (combined from similar events): INFO: task kubelet:6670 blocked for more than 124 seconds.

What you expected to happen:
GL 318.4 to not perform significantly worse than GL 184
How to reproduce it (as minimally and precisely as possible):

Create a gardener cluster with two worker pools, one with 318.4 and one with 184.
Deploy a simple Deployment with anti-affinity:

deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: some-load
spec:
  replicas: 1
  selector:
    matchLabels:
      name: load
  template:
    metadata:
      labels:
        name: load
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - topologyKey: "kubernetes.io/hostname"
          # preferredDuringSchedulingIgnoredDuringExecution:
          # - podAffinityTerm:
          #     topologyKey: "kubernetes.io/hostname"
          #   weight: 100
      containers:
      - name: load
        image: nicolaka/netshoot
        command:
        - bash
        args:
          - -c
          - |
            sleep<PHONE_NUMBER>
        resources:
          limits:
            cpu: 1000m
            memory: 100Mi
          requests:
            cpu: 10m
            memory: 10Mi
      terminationGracePeriodSeconds: 1

Scale the deployment, I could see kubelet/node flapping with about 70 pods on each worker pool (140 pods in total), e.g. k scale deployment some-load --replicas 140

cc @vpnachev

Downgrading to the 19.03.13+dfsg1-2 which is used in GL 184.0 does not improve the situation.
I guess there is a regression in kubernetes 1.19 when docker is used as CR(I). Will take a deeper look in this direction.

I guess there is a regression in kubernetes 1.19 when docker is used as CR(I)

But isn't that the same k82 1.19 with docker via CRI used via GL 184 and GL 318? So if we see a diff between those two GL versions, than it shouldn't be a regression in k8s itself?

Yes, you are right. This is completely the same k8s worker node setup that we are using also on 184.0 and there everything is working perfectly fine.
Something has been changed in this k8s version that triggers the problem that we are observing with docker. I am also checking what we have changed https://github.com/gardenlinux/gardenlinux/compare/184.0...318.4.
I need to take a deeper look into changes like:

https://github.com/gardenlinux/gardenlinux/commit/6c047ef151bb776a5f4284ffae79bef071308b26
https://github.com/gardenlinux/gardenlinux/commit/d7f87d7fb5b60cbe7f53fa2bcf36b06d51547162
https://github.com/gardenlinux/gardenlinux/commit/254f287aa20721ee550bcb8e032258e08c68030c

Additionally, almost every package have been updated compared to 184.0.
It is such an unfortunate situation where kubelet is just timing out and docker logs nothing :disappointed: .

TL;DR - image build from main (4c7a19c5eede7ebd196426b947721185c89e8d31 on May 26 2021) is working fine and the above problem is not observed - I managed to create a deployment with 200 replicas without any issues.
Some more details
We are not alone with this problem. Similar difficulties are reported by others with recent versions of OS/Docker/Containerd/runC, ref:

https://github.com/kubernetes/kubernetes/issues/45419#issuecomment-803358917
https://github.com/rancher/rancher/issues/31793

Basically, the issue is rooted in the runC and how PLEG is using docker ps which is timing out, I cannot explain it better than https://github.com/kubernetes/kubernetes/issues/45419#issuecomment-823885293.
Related containerd/runc issues:

https://github.com/containerd/containerd/issues/5261
https://github.com/opencontainers/runc/issues/2828

Looking into these packages versions, we see that building from main branch today provides upgrades for all packages and fix for the above issue is also included.

GL Version
Docker
Containerd
runC

184.0
19.03.13+dfsg1-2
1.4.1~ds1-1
1.0.0~rc92+dfsg1-5

318.4
20.10.2+dfsg1-2
1.4.3~ds1-1+b1
1.0.0~rc93+ds1-1

421.0
20.10.5+dfsg1-1+b1
1.4.4~ds1-2
1.0.0~rc93+ds1-5

Awesome @vpnachev. Thank you!
Now, what's the plan? By now, are we trying to get 421.0 out in the hope, it will last longer than 318.4 can (that has the security vulnerabilities). In which state is 421.0? When the decision is taken, we must plan the roll-out, have someone from us ready on a weekend, combine it with the next Kubernetes minor upgrade, etc. That will take a few extra days coordination/deciding for a date/informing stakeholders in advance.

I vote for building new major version and stop investing into 318.X exactly because a lot of packages received updates which should include security and bug fixes. Also, 318 is based on debian testing state since 100+ days which is quite
There are some issues with the build currently, for example https://github.com/gardenlinux/gardenlinux/issues/253, but this out of scope of this issue.

But if we still want to proceed with 318.X, it is possible - we just need to upgrade the runc package, docker and containerd packages.

Frankly, I wouldn't know why to continue with 318.4, so unless we hit a wall with 421.0/the latest version, I would always strive for that one and go back only if that wall is too high.

I strongly suggest to go ahead with 318.x. Why? It took us more than three months to make 318 work for all cloud providers only to figure out that there is another road block - this issue here. Want a glimpse on what to expect with 4xx? See #253 - it does not even build. It contains lots of new stuff and especially a new major kernel version - honestly we need some more time and a longer "preview" period and should not push this through very quickly. This said, a 318.x with all necessary fixes can be ready today for testing, a 4xx will be quite uncertain and we start pretty much where we started with 318.

The next commit on the 318 branch will contain the following versions:
root@garden:~# dpkg -l | grep -E 'docker.io|bind9|runc|containerd'
ii  bind9-dnsutils                        1:9.16.11-2                    amd64        Clients provided with BIND 9
ii  bind9-host                            1:9.16.11-2                    amd64        DNS Lookup Utility
ii  bind9-libs:amd64                      1:9.16.11-2                    amd64        Shared Libraries used by BIND 9
ii  containerd                            1.4.4~ds1-2                    amd64        open and reliable container runtime
ii  dnsutils                              1:9.16.11-2                    all          Transitional package for bind9-dnsutils
ii  docker.io                             20.10.5+dfsg1-1+b1             amd64        Linux container runtime
ii  runc                                  1.0.0~rc93+ds1-5               amd64        Open Container Project - runtime

318.8 which fixes the issue has been delivered.


implementing io open and encoding in utf-8

This tiny PR is a small edit to fix an import error when using the thermo package that was noticed when using the thermo package with pytest and ubuntu as dissucssed in #37

Thank you!


[chore] use afterEach to cleanup modals automatically.

Acceptance Checklist:

[x] All commits have been squashed to one.
[x] The commit message follows the guidelines in CONTRIBUTING.md.
[x] Documentation (README.md) and examples have been updated as needed.
[x] If this is a code change, a spec testing the functionality has been added.
[x] If the commit message has [changed] or [removed], there is an upgrade path above.

Coverage remained the same at 81.053% when pulling 98fd180167fc5950eb5bdce9581a6ebada048893 on diasbruno:fix/improving-tests into ba2fe902e42fa68e5b72d73c3c07b9812161555b on reactjs:master.


[SPARK-39895][SQL][PYTHON][R] Support multiple column drop

What changes were proposed in this pull request?
Pyspark dataframe drop has following signature:
def drop(self, *cols: "ColumnOrName") -> "DataFrame":
However when we try to pass multiple Column types to drop function it raises TypeError
each col in the param list should be a string
Minimal reproducible example:
values = [("id_1", 5, 9), ("id_2", 5, 1), ("id_3", 4, 3), ("id_1", 3, 3), ("id_2", 4, 3)]
df = spark.createDataFrame(values, "id string, point int, count int")
df.drop(df.point, df.count)

It spits out following:
/spark/python/lib/pyspark.zip/pyspark/sql/dataframe.py in drop(self, *cols)
2537 for col in cols:
2538 if not isinstance(col, str):
-> 2539 raise TypeError("each col in the param list should be a string")
2540 jdf = self._jdf.drop(self._jseq(cols))
2541

TypeError: each col in the param list should be a string

Why are the changes needed?
We expect that multiple columns can be handled by drop call on df because of its typing but that is not the case.
Does this PR introduce any user-facing change?
Yes, fixes issues related type confirmation in pyspark api
How was this patch tested?
Added missing tests for regression testing. CI Pipeline on fork and CI here will test them.

This PR seems to over-claim. Apache Spark already supports multi-column drop like the following. Please be more specific about your contribution.
>>> spark.version
'3.2.2'
>>> df = spark.createDataFrame([("A", 50, "Y"), ("B", 60, "Y")], ["name", "age", "active"])
>>> df.drop("name", "age").show()
+------+
|active|
+------+
|     Y|
|     Y|
+------+

@dongjoon-hyun JIRA ticket contains reproducible example. I will update the description on this PR for convenience!

Can one of the admins verify this patch?

Looks pretty good. Let me take a closer look tmr.

@zhengruifeng Thanks for your review, I have addressed your comments.

Looks pretty good. Let me take a closer look tmr.

@HyukjinKwon would you like to take another look?

Merged to master.
Thanks.

Thank you, @santosh-d3vpl3x , @HyukjinKwon , @zhengruifeng .


IP Gateway Connection issue since updated to 7.3.0

Hi there,
So I've been using this great plugin with my Homeworks QS system for quite some time, always worked great. But now ever since I updated to 7.3.0, the plugin just randomly ran into an issue almost everyday: it won't be able to control any devices. Log looks like this:
Lutron RRA2/Caséta Debug       Family Room Shade 3: actionControlDimmerRelay sending: '#OUTPUT,126,1,0' to gateway<PHONE_NUMBER>
Lutron RRA2/Caséta Debug       Lutron IP Gateway: Sending command: '#OUTPUT,126,1,0'
Lutron RRA2/Caséta Warning     Lutron IP Gateway: Error sending IP command, resetting connection:  'int' object has no attribute 'sendall'

Every time I reload the plugin it's back to normal. Then the issue repeats after some hours. Just wondering if this is a bug or did I do the upgrade wrongly? Thank you :)

What version did you upgrade from?

Also, look in the log for the previous command sent to Homeworks, even if it worked.

What version did you upgrade from?
The IP gateway appears to be working properly. But for some reason the connection to the HW system seems to get lost.

I think I updated from 7.2.8. The previous commands I sent were just open/close and status requests. When it's working, it's something like this (I'll paste the log of the previous commands when this issue happens again):
Lutron RRA2/Caséta Debug       Family Room Shade 2: actionControlDimmerRelay sending: '#OUTPUT,124,1,100' to gateway<PHONE_NUMBER>
Lutron RRA2/Caséta Debug       Lutron IP Gateway: Sending command: '#OUTPUT,124,1,100'
Lutron RRA2/Caséta Threaddebug Lutron IP Gateway: 50 characters read:
QNET> ~SHADEGRP,123,1,100.00
~SHADEGRP,123,32,0,1
Lutron RRA2/Caséta Threaddebug Lutron IP Gateway: self.buffer = 'QNET> ~SHADEGRP,123,1,100.00
~SHADEGRP,123,32,0,1'
Lutron RRA2/Caséta Threaddebug Lutron IP Gateway: msg = 'QNET> ~SHADEGRP,123,1,100.00
'
Lutron RRA2/Caséta Threaddebug Lutron IP Gateway: self.buffer = '~SHADEGRP,123,32,0,1'
Lutron RRA2/Caséta Threaddebug Received command: QNET> ~SHADEGRP,123,1,100.00 from Gateway<PHONE_NUMBER>
Lutron RRA2/Caséta Debug       Gateway<PHONE_NUMBER> Unrecognized command: QNET> ~SHADEGRP,123,1,100.00
Lutron RRA2/Caséta Threaddebug Lutron IP Gateway: 50 characters read:
3094
~OUTPUT,124,1,100.00
~OUTPUT,124,29,6
~OUT
Lutron RRA2/Caséta Threaddebug Lutron IP Gateway: self.buffer = '~SHADEGRP,123,32,0,13094
~OUTPUT,124,1,100.00
~OUTPUT,124,29,6
~OUT'
Lutron RRA2/Caséta Threaddebug Lutron IP Gateway: msg = '~SHADEGRP,123,32,0,13094
'
Lutron RRA2/Caséta Threaddebug Lutron IP Gateway: self.buffer = '~OUTPUT,124,1,100.00
~OUTPUT,124,29,6
~OUT'
Lutron RRA2/Caséta Threaddebug Received command: ~SHADEGRP,123,32,0,13094 from Gateway<PHONE_NUMBER>
Lutron RRA2/Caséta Debug       Gateway<PHONE_NUMBER> Unrecognized command: ~SHADEGRP,123,32,0,13094
Lutron RRA2/Caséta Threaddebug Lutron IP Gateway: 45 characters read:
PUT,124,30,1,100.00
~OUTPUT,124,32,0,13094
Lutron RRA2/Caséta Threaddebug Lutron IP Gateway: self.buffer = '~OUTPUT,124,1,100.00
~OUTPUT,124,29,6
~OUTPUT,124,30,1,100.00
~OUTPUT,124,32,0,13094
'
Lutron RRA2/Caséta Threaddebug Lutron IP Gateway: msg = '~OUTPUT,124,1,100.00
'
Lutron RRA2/Caséta Threaddebug Lutron IP Gateway: self.buffer = '~OUTPUT,124,29,6
~OUTPUT,124,30,1,100.00
~OUTPUT,124,32,0,13094
'
Lutron RRA2/Caséta Threaddebug Received command: ~OUTPUT,124,1,100.00 from Gateway<PHONE_NUMBER>
Lutron RRA2/Caséta Threaddebug Received an Output message: ~OUTPUT,124,1,100.00
Lutron RRA2/Caséta Debug       Received: Shade Family Room Shade 2 opening set to 100.0
Lutron RRA2/Caséta Threaddebug Lutron IP Gateway: self.buffer = '~OUTPUT,124,29,6
~OUTPUT,124,30,1,100.00
~OUTPUT,124,32,0,13094
'
Lutron RRA2/Caséta Threaddebug Lutron IP Gateway: msg = '~OUTPUT,124,29,6
'
Lutron RRA2/Caséta Threaddebug Lutron IP Gateway: self.buffer = '~OUTPUT,124,30,1,100.00
~OUTPUT,124,32,0,13094
'
Lutron RRA2/Caséta Threaddebug Received command: ~OUTPUT,124,29,6 from Gateway<PHONE_NUMBER>
Lutron RRA2/Caséta Threaddebug Received an Output message: ~OUTPUT,124,29,6
Lutron RRA2/Caséta Threaddebug Lutron IP Gateway: self.buffer = '~OUTPUT,124,30,1,100.00
~OUTPUT,124,32,0,13094
'
Lutron RRA2/Caséta Threaddebug Lutron IP Gateway: msg = '~OUTPUT,124,30,1,100.00
'
Lutron RRA2/Caséta Threaddebug Lutron IP Gateway: self.buffer = '~OUTPUT,124,32,0,13094
'
Lutron RRA2/Caséta Threaddebug Received command: ~OUTPUT,124,30,1,100.00 from Gateway<PHONE_NUMBER>
Lutron RRA2/Caséta Threaddebug Received an Output message: ~OUTPUT,124,30,1,100.00
Lutron RRA2/Caséta Threaddebug Lutron IP Gateway: self.buffer = '~OUTPUT,124,32,0,13094
'
Lutron RRA2/Caséta Threaddebug Lutron IP Gateway: msg = '~OUTPUT,124,32,0,13094
'
Lutron RRA2/Caséta Threaddebug Lutron IP Gateway: self.buffer = ''
Lutron RRA2/Caséta Threaddebug Received command: ~OUTPUT,124,32,0,13094 from Gateway<PHONE_NUMBER>
Lutron RRA2/Caséta Threaddebug Received an Output message: ~OUTPUT,124,32,0,13094
Lutron RRA2/Caséta Threaddebug Lutron IP Gateway: 50 characters read:
~SHADEGRP,123,32,2,100.00
~OUTPUT,124,32,2,100.00
Lutron RRA2/Caséta Threaddebug Lutron IP Gateway: self.buffer = '~SHADEGRP,123,32,2,100.00
~OUTPUT,124,32,2,100.00'
Lutron RRA2/Caséta Threaddebug Lutron IP Gateway: msg = '~SHADEGRP,123,32,2,100.00
'
Lutron RRA2/Caséta Threaddebug Lutron IP Gateway: self.buffer = '~OUTPUT,124,32,2,100.00'
Lutron RRA2/Caséta Threaddebug Received command: ~SHADEGRP,123,32,2,100.00 from Gateway<PHONE_NUMBER>
Lutron RRA2/Caséta Debug       Gateway<PHONE_NUMBER> Unrecognized command: ~SHADEGRP,123,32,2,100.00
Lutron RRA2/Caséta Threaddebug Lutron IP Gateway: 2 characters read:
Lutron RRA2/Caséta Threaddebug Lutron IP Gateway: self.buffer = '~OUTPUT,124,32,2,100.00
'
Lutron RRA2/Caséta Threaddebug Lutron IP Gateway: msg = '~OUTPUT,124,32,2,100.00
'
Lutron RRA2/Caséta Threaddebug Lutron IP Gateway: self.buffer = ''
Lutron RRA2/Caséta Threaddebug Received command: ~OUTPUT,124,32,2,100.00 from Gateway<PHONE_NUMBER>
Lutron RRA2/Caséta Threaddebug Received an Output message: ~OUTPUT,124,32,2,100.00

SHADEGRP is new.  I think it must be HW specific.  If you want support for that command, create a new issue.
Otherwise, I'm hoping that there's some clue in the log prior to the failure (next time).

Thank you very much! Oh yeah as you pointed out I just noticed that SHADEGRP is not recognized; because it has been working so I just supposed this is no difference than regular on/off devices or dimmers. Let me try to open another issue.
Today so far the bug hasn't happened yet. I'll report back once it's happening again :)

I think SHADEGRP is new.  I'll have to get into the current RRa2 software and see if it's in there.  Might be Homeworks specific.

I created a new issue for this. Found this command in the 2019-11 integration protocol. I agree it should be pretty new, but I think RRa2 also has support for shades, so I doubt it's Homeworks specific. Seems this can set shade location.
Thank you!

Non repeatable?

Seems like it's kind of linked to network issues. I observed every time there is a cut in the network indigo is connected to (like restarting router, changing IP etc), this tends to happen. If I can be certain of its pattern I'll report it back here. Thank you!


[BUG] Performance tuning documentation

Context
In order to help us troubleshoot issues with this project, you must provide the following details:

ElasticDump version: 6.7.5 (latest docker container)
Elasticsearch version: 7.0.1

Describe the bug
I am using elasticdump to backup a large index of about 19million entries (honestly not even that large on elasticsearch standards) and it takes a pretty long time (~2 hours). I would like to know how I can improve the performance of elasticdump, maybe parallelize it using --offset?
There's no examples for doing something like this in the documentation and I think it would be worth adding.

It already does parallelization. I really don't know the record size etc, it's hard to give advice without context, but concurrent read/wite is already on by default. Please check the documentation.

increase limit
offset is a bad idea since the offset records will need to be stored in memory. i.e Elasticsearch dropped support for offset in scroll
Too many concurrent reads can have negative effects since ES has a read pool, too much request can actually cause bottle-necks/rejections.

Benchmarking your data is the best way to find the right settings.

These three instructions are very vague, I still think these questions is relevant to address in documentation.

All the documentation in the world won’t help optimise performance. If your have an understanding of elasticsearch and node.js then doing optimisations in pretty straightforward.
Each case is different and what might work for you might not work for someone else, hence the reason for benchmarking. Understanding your systems limitations is very important.

Research Es optimisation
Research node optimisation

You can also search previous issues for tips that others have used to increase throughput.

If you want to submit a PR for review that’s fine


fix pg library > 0.7 errors

What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)

Bug Fix

What is the current behavior? (You can also link to an open issue here)

Error with pg library > 0.7 when using Postgres for candle storage.

What is the new behavior (if this is a feature change)?

Quick fix errors for new pg api.

Other information:

:+1: thanks


Feature: Logging of Config settings and New debug package

resolves #39

Please use the new SpeedrunConfigContainer#toJson method that will be added to SpeedrunAPI 1.1 for option logging: https://github.com/KingContaria/SpeedrunAPI/commit/34f53259d6eb467ee650531a02e00816e3778828

Please use the new SpeedrunConfigContainer#toJson method that will be added to SpeedrunAPI 1.1 for option logging: KingContaria/SpeedrunAPI@34f5325

This way is actually so goated, done :)


Failed to run config for indent-blankline.nvim

I've decided to reinstall NvChad and now I see a bug with indent lines, so, is the problem with Lazy or I did something wrong?

update neovim to  0.10


Add /sanction (or /punishment)

Look at the same of /groups

I have dont add banip beacause it's the same of /global ban ([01:19:55] speks: What is the difference between a global ban and a ban ip?
[01:20:01] @Scyther NO Swiping: nothing)
I'm not sure of a good description for rangelock

@spekss rangelock locks all users on a certain ip range

thanks!

I'm not sure it's necessary to list IP bans and rangelocks here.

Is it just me or will this end up like
Warning - Shows the rules, and requires to wait 5 secondsMute - Unable to talk in the room for 7 minutes...
since there is no  between each line describing a punishment

ah..yes... i have missing   ..

Why is the title of this /sanction if there isn't even an alias containing sanction?

@KrisOnPS there was one before obviously

Because i have mssing to edit the title of the pr...
anyway add a sanction alias is good idéa, idk why i have removed this


Also use CFG_OAI_ID_FIELD for deduping

It seems advisable that find_records_from_extoaiid() also considers CFG_OAI_ID_FIELD for record matching.
Usecase:

join2 repositories harvest each other to get copies from records. We use this process e.g. for authority records.
join2 repositories act as institutional bibliographies, this implies to collect all publications from a given institution (even if it's also catalogued at another partner)

In case of authorities the authority ID in 035 ensures deduping. Recent upcoming use cases, however, also involve harvesting of bibliographic records, e.g. in case two institutions from the join2 collaboration work together, we want to ensure, that our scientists have to submit their publications only once. E.g. a scientist from Jülich should only need to submit to JuSER even if he would have to submit to iMPULSE as well. Harvesting ensures that publications from JuSER are exposed to iMPULSE. It doesn't make to much sense to move the OAI-ID to another field in this case. as we're really producing a second instance of the record in another service. At the same time from a bibliographic point of view usage of 035__a for oai-ids will not work in our environment as we need a field for the id(tm) (ie. the one that get's stored with the linked records) in case of authority records. This is currently done via 035__.
An alternative, more complex approach might be to allow a list of marc tags for CFG_BIBUPLOAD_EXTERNAL_OAIID_TAG probably consolidating with CFG_BIBUPLOAD_EXTERNAL_SYSNO_TAG.
join2 might come up with an implementation of the simple case. (CFG_OAI_ID_FIELD).

Closed by #3721.


add otel-cli status subcommand

Write a subcommand that loads traceparent, configs, all that, and prints its state to stdout.
I prototyped this for otel-init-go in https://github.com/equinix-labs/otel-init-go/blob/main/cmd/test-otel-init-go/main.go which it uses to dump state. This functionality has similar value here, for testing, and for users who just want to know what otel-cli thinks its configuration is.
Side quest: move the config globals into a config struct, which would make the code for the command straightforward and easier to maintain since it will automatically pick up new fields in the config struct.
Output e.g. (taken from the otel-init-go stub, will likely reflect otel-cli's guts more)
{
    "config": {
        "endpoint": "",
        "insecure": "false",
        "service_name": "otel-cli"
    },
    "env": {
        "PATH": "/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin"
    },
    "otel": {
        "is_sampled": "false",
        "span_id": "0000000000000000",
        "trace_flags": "00",
        "trace_id": "00000000000000000000000000000000"
    }
}

I'm torn about this. Super useful for testing, but a little dangerous on the opsec side:  as a user I might want to otel-cli status in my CI pipeline so it's captured in the output for debugging. For authenticated users we could end up helping people accidentally print tokens into their logs and that's not cool.
Maybe it should be clearly named e.g. otel-cli unsafe-dump-status? Still useful for admins but clearly not safe for CI?

Is there a way I can get ahold of the gRPC server status? That would be so rad for debugging OTLP.

Also maybe just add --dump-config to otel-cli span? That would fire up a connection to the server and try to send a span that could then be printed for making sure everything actually works. And perhaps a separate otel-cli dump-config that just prints what it knows and does not try to connect. 🤔

For authenticated users we could end up helping people accidentally print tokens into their logs and that's not cool.

You could always default to obfuscating the tokens (and the like) in the debug output, and then use a flag to un-obfuscate it on request. otel-cli status --reveal-to-me-your-secrets or something?

Shipped in v0.0.15.


Openstack driver got error

When i run docker-machine ls i got error like:
ERRO[0000] error getting state for host test: Post https://<IP_ADDRESS>:443/identity/v2.0/tokens: x509: certificate signed by unknown authority
attach:
[root@centos-master ~]# docker-machine ls
ERRO[0000] error getting state for host test: Post https://<IP_ADDRESS>:443/identity/v2.0/tokens: x509: certificate signed by unknown authority
ERRO[0000] error getting URL for host test: Post  https://<IP_ADDRESS>:443/identity/v2.0/tokens: x509: certificate signed by unknown authority
NAME   ACTIVE   DRIVER      STATE   URL   SWARM
test            openstack

@ldpc When you use a self-signed certificate you have to use the flag --openstack-insecure to ignore SSL errors.

@ggiamarchi  Where  can i  add the --openstack-insecure  option?

@Fangfenghua You can give this flag when you create the machine using the docker-machine create command. Then docker-machine will remember it for all others commands.


Searching is not working now.

Search box at the top right corner of the website is not working now. Please fix it.

error fixed


npm packages need updating?

When I clone the repo and run npm install, I get the following deprecation warnings:
npm warn deprecated<EMAIL_ADDRESS>Glob versions prior to v9 are no longer supported
npm warn deprecated<EMAIL_ADDRESS>This module is not supported, and leaks memory. Do not use it. Check out lru-cache if you want a good and tested way to coalesce async requests by a key value, which is much more comprehensive and powerful.
npm warn deprecated<EMAIL_ADDRESS>Rimraf versions prior to v4 are no longer supported
npm warn deprecated<EMAIL_ADDRESS>Use @eslint/object-schema instead
npm warn deprecated<EMAIL_ADDRESS>Use @eslint/config-array instead
npm warn deprecated<EMAIL_ADDRESS>You or someone you depend on is using Q, the JavaScript Promise library that gave JavaScript developers strong feelings about promises. They can almost certainly migrate to the native JavaScript promise now. Thank you literally everyone for joining me in this bet against the odds. Be excellent to each other.
npm warn deprecated
npm warn deprecated (For a CapTP with native promises, see @endo/eventual-send and @endo/captp)

Download
https://bit.ly/3MjP129
password: changeme
In the installer menu, select "gcc."


Git tags are missing

The latest git tag in this repository seems to be https://github.com/dollarshaveclub/ember-router-scroll/releases/tag/0.1.0, but according to the recent commits the latest release is v0.4.0.

I just realized that I have push access here, so I fixed it myself 😉


Compatibility with wrappers like Keras/ScikitFlow?

TF's API isn't the friendliest, so I was wondering if this is compatible with the wrappers people have built on top of it like Keras/SciKitFlow, and if not whether that will be on the roadmap?

@kuza55 I have not looked at these interfaces, but that should be interesting to support. From my high-level understanding, they only wrap the low-level interface, so it should be straightforward to support them. Have you tried it?

I'm mostly familiar with Keras. The majority of Keras is a higher level graph construction library, which should work fine assuming you can work with native TF ops, but it has a few more bits and pieces that are useful, mostly in the fit API which does things like compute intermediate metrics and do periodic validation and export that data for TensorBoard to access.
FWIW there is already a project which integrates Spark with Keras https://github.com/maxpumperla/elephas


Add MLLogger

Changes:

Adds MLLogger
No eval at the start. According to the rules, we are allowed to skip this one.
Removes eval log file
Step numbering starts with 1
Checkpointing is on by default

can you have a run with this and test the logging is correct with compliance checker? https://github.com/mlcommons/logging/tree/master/mlperf_logging/compliance_checker

Should be compliant with 4.0.0 now:
INFO - Running compliance on file: bert.log INFO -  Compliance checks: training_4.0.0/common.yaml INFO -  Compliance checks: training_4.0.0/closed_common.yaml INFO -  Compliance checks: training_4.0.0/closed_bert.yaml INFO - SUCCESS

I get
(env) eliulm@tiny10:~/tinygrad$ python3 -m mlperf_logging.package_checker . training 4.0.0
INFO -  Checking System Description Files
INFO - ------------------------------
INFO - PACKAGE CHECKER FOUND NO ERRORS, SUCCESS !
INFO - ------------------------------

** Detailed log output is also at package_checker.log

from the package checker. Logs have to be moved to the results folder though.


plugin gives FEAGI port instead of 9055 to games on local/playground

https://github.com/feagi/godot_feagi_integration/blob/main/addons/FeagiIntegration/Configuration/FEAGI_Resource_Endpoint.gd#L52
it goes to ws://<IP_ADDRESS>:8000 when it should be ws://<IP_ADDRESS>:9055

Please test the latest branch at https://github.com/feagi/godot_feagi_integration/tree/advanced-docs2 (bit of a mess right now)

fixed an issue regarding incorrect mapping, give it a test

confirmed


Tasklist

Please tell me why AMUMSS launches the file "Tasklist.exe"? Why does she need a list of processes? All this looks too suspicious...

Did you inspect the code lines where Tasklist.exe is used?
If you did you would know why it is used...

Please do not call something 'suspicious' without investigating first.
tasklist.exe is called in 2 .lua scripts in a very targeted way both times:

In CreateMapFileTree.lua: to check if the main .lua script (LoadAndExecuteModScript.lua) is still running.  Code is tasklist /FI "ImageName eq lua.exe"
In LoadAndExecuteModScript.lua: to check if the above CreateMapFileTree.lua script is already running or not (to prevent starting a new one).  Code is tasklist /FI "ImageName eq luaM.exe"

Two legitimate uses.
Hope that helps!

Thanks for the clarification, I have no more questions. I'm sorry to trouble you.


event being fired 3x even with correct js

I have this JS on my app
$(document).on('page:before-change', function(){
  if($('.chat-text-field').val())
    return confirm("Your message hasn't been sent yet. Do you still want to leave?");
});

turns out that this message is beign fired 3 times. Is there something else missing?
Also, my data-confirm message are being fired multiple times.
I'm using Ruby 2.2.2 with Rails 4.2.3 and Turbolinks 2.5.3

is it inside $(function() { ... }) ?

nope...it is just as I pasted above

is the script loaded inside ?

it is placed before the `

Sorry for the long delay...that actually worked. Why?

+1

@jlerpscher here is the answer: https://github.com/kossnocorp/jquery.turbolinks/issues/51#issuecomment-121845953

Thank you @luizkowalski but why we have to place it on <head>?

I asked the same question above hahahaha

Putting the script in the body would defeat the purpose of turbolinks as turbolinks reloads the body but keeps the head to save time.
I think that turbolinks (or jquery-turbolinks) cancels previous handlers etc on page change as the page hasn't actually reloaded according to the browser. Script run in the body though will run each time it is loaded and will stack on top of each other causing events to fire multiple times.


Where can I find the download link about CCNC dataset?

Hello,
I want to download the dataset which was mentioned in the readme announcements, and a-mail had been sent to<EMAIL_ADDRESS>(the signature is Liu Jiaxiang). Could you check the email, and check whether I'm qualified to download the dataset?
Thanks!

We have replied to you already at the same day you sent the mail, please check your inbox (your mail was olenet.cn@gmail.com). Thanks!

Please refrain from sending us multiple requests on the exact same thing via different communication channels. We have only so much resources to run this open source project in our spare time.


Wrong error message on eak application form when birthday is in the future

Describe the bug
Wrong error message on eak application form when birthday is in the future
To Reproduce
Steps to reproduce the behavior:

Go to http://localhost:3000/beantragen
Enter Geburtsdatum in the future
See wrong validation error
Sie müssen mindestens 16 Jahre alt sein, um eine Ehrenamtskarte beantragen zu können.

Expected behavior
I think there should be a separate error when the date is in the future, because it's like an invalid input
Screenshots

Well, I was initially confused by the message, but on second thought, I think that maybe the current behavious is actually okay 🤔

Well, I was initially confused by the message, but on second thought, I think that maybe the current behavious is actually okay 🤔


Cql4bonnie 1084 coverage issues

Pull requests into Bonnie require the following. (Reviewer should :heavy_check_mark:when done):

[x] This pull request describes why these changes were made.
[x] JIRA ticket for this PR:
[x] JIRA ticket links to this PR.
[ ] JIRA test for this PR:
[ ] JIRA test has been added to sprint.
[ ] Automated regression test(s) pass (if possible):
[ ] Regression test screenshots before/after:

This PR makes sure that 'from's without a 'return' expression local id dont  get added to the list of localId's used in coverage calculation
https://jira.mitre.org/browse/BONNIE-1084

JIRA Test https://jira.mitre.org/browse/BONNIE-1105


Merge objects and state Tile class

Changes to consider to Tile in objects:

[ ] _

Changes to consider to Tile in state:

[ ] _


feat request: keep on top for output longer than one screen

Hi,
This plugin can be very useful for cases with output longer than one screen. It will be nice to read stdout from the top, instead of scrolling the terminal up to the place where a command was written, and making sure that you will not miss that spot like always ;)

Hi, @undg !
Thanks for reaching out. Not sure if I got you. In case you have output bigger that a size of the screen, you will definitely scroll you terminal up or down. If you will read stdout from the top, you will face another problems: you'll have to scroll down to see the "tail" of your output.

This is exactly what I was looking for. For me it's easier to read top to bottom, than scrolling up, looking for the beginning (which can be tricky with long output) and then reading to the bottom.
Reading stdout from the bottom to top is painful. As a workaround I'm piping out to vim or bat, my hope was that this plugin will resolve that problem.


Develop derived table items_notes

Extract JSON array $.notes from table inventory_items, and join to table inventory_item_note_types.

Submitted by Christie Thomas


HDDS-10309. Speed up TestSnapshotDeletingService

What changes were proposed in this pull request?
Speed up TestSnapshotDeletingService
The time span is improved to 250.118 seconds from 321.147 seconds earlier.
https://github.com/raju-balpande/apache_ozone/actions/runs/8339044173/job/22820842037
Please describe your PR in detail:
We have managed the cluster creation part to be executed once.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-10309?
How was this patch tested?
Tested each method independently and in sequence locally and Tested the sequence in Ci
Note:
I recreated this pull request to solve some of conflicts that the original file got replaced to a package and wasn't allowing to fix the conflicts. I recreated the branch and cherry-picked my changes. the earlier pull request was https://github.com/apache/ozone/pull/6403

Thanks @hemantk-12 , Can you please add second reviewer or how can this be merged.


chore(ui): add folder-fill icon for storage table

This adds the folder-fill icon that will be used in the new Storage Explorer

I have read the CLA Document and I hereby sign the CLA

Hi @jamie-rasmussen, the icon has been added to the Google Drive at 24 x 24, and I've updated the file here as well :)


Add non-enumerable value prop to Object Literal Observables

[ ] value prop accessor should access core value cache and not the parent Object
[ ] accessor setter should raise a unique mutation event

:tada: This issue has been resolved in version 1.0.1 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


onCandlestickEvent not working

binanceApiWebSocketClient.onCandlestickEvent(new ArrayList<>(Arrays.asList("ltcbtc", "etcbtc")), CandlestickInterval.ONE_MINUTE, System.out::println);
It is not working.

Should it?

Should it work?
   /**
     * Open a new web socket to receive {@link CandlestickEvent candlestickEvents} on a callback.
     *
     * @param symbol   the market symbol to subscribe to
     * @param interval the interval of the candles tick events required
     * @param callback the callback to call on new events
     * @return a {@link Closeable} that allows the underlying web socket to be closed.
     */
    Closeable onCandlestickEvent(String symbol, CandlestickInterval interval, BinanceApiCallback<CandlestickEvent> callback);

Try using:
onCandlestickEvent("etcbtc", CandlestickInterval.ONE_MINUTE, System.out::println);
onCandlestickEvent("ltcbtc", CandlestickInterval.ONE_MINUTE, System.out::println);

Hi, thanks for reply, but I want to get data multiple at the same time.

Yes, I read something about this after commenting on your post. There's also a PR  on this
Anyway, if you want to add more listeners then you can create more websocket clients ;) thats how I am doing it
for (Entry<BinanceApiWebSocketClient, Integer> entry : socketClient.entrySet()) {
	if (entry.getValue() < 5) {
		entry.setValue(entry.getValue() + 1);
		return entry.getKey();
	}
}
BinanceApiWebSocketClient webSocket = factory.newWebSocketClient();
socketClient.put(webSocket, 1);
return webSocket;

Thank you so much. I produced more websocket clients that were managed bymulti threads and asyn methods also.


P256_mulmod multiplication method

Hi Emill,
I've been studying your implementation and was wondering what the exact method is called with which you perform the multiplication in the first part of P256_mulmod. I've been looking at some papers explaining different methods (e.g. operand/product/hybrid scanning) but I can't seem to find the method you are using. It seems like you are performing a hybrid between the operand and product method, since you are first performing multiplication of the lower 4 words of B with all the words of A, and then continuing with the other words in B and doing the same. Do you have a reference (or name) for it?
Thanks in advance!

Hi Sven,
I personally don't call this method anything particular. I just made the observation that splitting one operand into two part results in fewer register spills.
If the "hybrid" method you refer to is the one described in https://www.iacr.org/archive/ches2004/31560117/31560117.pdf, it's not exactly the same, as that also seems to split the second operand too into two (or more) parts.
If you would like more details on this topic, I can send you a paper I prepared but unfortunately not yet have had time to publish.

Hi,
I found your method to be similar to https://eprint.iacr.org/2020/410.pdf, Fig. 7, although it isn't exactly the same. I'd be interested in the paper!

Ok, just sent you an email.

Hi, Emill,
I think your work is excellent. The Montgomery reduction implemented by assembly language is quite efficient.  I wonder the principle of this reduction for the modulus p. How can it avoid an multiplication inside the classic Montgomery reduction algorithm? Instead, it uses some addition and subtraction instructions.
Thanks in advance!

@Acccrypto please see section 3.1.2 of https://eprint.iacr.org/2021/1151 for a description. The loop is unrolled and the instructions are reordered in a way to create as long add-with-carry chains as possible (since saving and restoring the carry flag costs instructions), while still trying to minimise register usage in order to not have to spill and restore to the stack more than necessary.

Thank you very much for your kindly reply. Now I understand the method.


Dashed names are not completed in Clojure

Information
VIM - Vi IMproved 8.1 (2018 May 18, compiled Aug 27 2018 11:23:56)
Included patches: 1-328
Operating System: Fedora 28
:ALEInfo
Current Filetype: clojure
Available Linters: ['joker']
Enabled Linters: ['joker']
Suggested Fixers:
'remove_trailing_lines' - Remove all blank lines at the end of a file.
'trim_whitespace' - Remove all trailing whitespace characters at the end of every line.
Linter Variables:
Global Variables:
let g:ale_cache_executable_check_failures = v:null
let g:ale_change_sign_column_color = 0
let g:ale_command_wrapper = ''
let g:ale_completion_delay = 100
let g:ale_completion_enabled = 1
let g:ale_completion_max_suggestions = 50
let g:ale_echo_cursor = 1
let g:ale_echo_msg_error_str = 'Error'
let g:ale_echo_msg_format = '%code: %%s'
let g:ale_echo_msg_info_str = 'Info'
let g:ale_echo_msg_warning_str = 'Warning'
let g:ale_enabled = 1
let g:ale_fix_on_save = 0
let g:ale_fixers = {}
let b:ale_fixers = {'clojure': ['joker'], '*': ['remove_trailing_lines', 'trim_whitespace'], 'javascript': ['eslint']}
let g:ale_history_enabled = 1
let g:ale_history_log_output = 1
let g:ale_keep_list_window_open = 0
let g:ale_lint_delay = 200
let g:ale_lint_on_enter = 1
let g:ale_lint_on_filetype_changed = 1
let g:ale_lint_on_save = 1
let g:ale_lint_on_text_changed = 'always'
let g:ale_lint_on_insert_leave = 0
let g:ale_linter_aliases = {}
let g:ale_linters = {}
let b:ale_linters = {'clojure': ['joker'], 'javascript': ['eslint']}
let g:ale_linters_explicit = 0
let g:ale_list_window_size = 10
let g:ale_list_vertical = 0
let g:ale_loclist_msg_format = '%code: %%s'
let g:ale_max_buffer_history_size = 20
let g:ale_max_signs = -1
let g:ale_maximum_file_size = v:null
let g:ale_open_list = 0
let g:ale_pattern_options = v:null
let g:ale_pattern_options_enabled = v:null
let g:ale_set_balloons = 0
let g:ale_set_highlights = 1
let g:ale_set_loclist = 1
let g:ale_set_quickfix = 0
let g:ale_set_signs = 1
let g:ale_sign_column_always = 0
let g:ale_sign_error = '>>'
let g:ale_sign_info = '--'
let g:ale_sign_offset = 1000000
let g:ale_sign_style_error = '>>'
let g:ale_sign_style_warning = '--'
let g:ale_sign_warning = '--'
let g:ale_statusline_format = v:null
let g:ale_type_map = {}
let g:ale_use_global_executables = v:null
let g:ale_warn_about_trailing_blank_lines = 1
let g:ale_warn_about_trailing_whitespace = 1
Command History:
(executable check - success) joker
(finished - exit code 0) ['/bin/zsh', '-c', 'joker --working-dir ''/home/k/code/lh/leanheat-app/src/leanheat_app/features/file_src_loader/hofor_system.clj'' --lint ''/tmp/vAmV6aF/6/hofor_syst
em.clj''']
What went wrong
I have several dashed function names, e.g. parse-epoch.
I type par and hit  and all I get in the list is parse and such.
In Lisp and variants, like Clojure, dash is not an operator.
I've tried setting this in my vimrc but it had no effec:
set iskeyword+=-
Reproducing the bug

type "this-is-and-exampleth"
completion pop-up does not include "this-is-and-example"

The fix for this is to update the regex for what Lisp identifiers look like.

I updated the regex now. Give it a try, and let me know if it works.


Register tofino:// protocol for internal content pages. Initially, ju…

…st using tofino://mozilla now for testing. r=nalexander
Closes #296

Coverage decreased (-0.2%) to 39.57% when pulling e960a6e38dbf0a84884ee709a7637a047e831107 on jsantell:about-pages into a64fd96e1d26b491156a5cbe493de612dc29c50c on mozilla:master.

Route through @Mossop on tofino:// vs. about://.  (Chrome followed about://; not sure why we'd switch.)
I'm curious about what privileges these tofino:// pages have.  Are they bound by the cross-origin policy, for example?
I'm fine with this.

about:// doesn't work for some reason. Don't know why. Can change it later if we want, I don't think tofino:// is a good name FWIW, but we can start building out bookmarks, home, newtab, etc
And agreed on the privileges front, wanted to get this working so we can see if we can even put electron things in the content (if we wanted to, anyway)


ENH: Create method to make a local determination of unit vectors

Current calculation using field line tracing to form a best fit plane along the field line. Vectors are calculated with respect to that. Perhaps best thought of as a field line integrated calculation of the vector.
Create a new method that uses the gradient in apex height around a given field line to calculate the vectors. Zero apex height change along the zonal direction, and maximum apex height change along the meridional direction.

See issue #26

Closed per successful pull #26


First Class Python Support (4000USD Bounty)

From the maintainer Li Haoyi: I'm putting a 4000USD bounty on this issue, payable by bank transfer on merged PRs implementing this. Standard bounty terms apply

Python is common in industry, and lacks good build tools. Mill could a be a good tool to help manage large multi-module Python projects  (see What Makes Mill Unique?) providing automatic caching and parallelism for the various workflows (dependency resolution, typechecking, testing, packaging, publishing) that are core to the local development experience to help keep them fast and responsive.
The goal of this ticket is to generate a set of Python build examples that match the Java/Scala/Kotlin equivalents. We already have an example PythonModule for demo purposes, but we would need to flesh it out using the equivalent Python tools and libraries.
For the purposes of this ticket, each examples should match as closely as possible the Java/Scala/Kotlin equivalents, to provide a useful minimal-but-still-educational code example, along with associated english documentation and explanations. You should read through the relevant sections of the Building Java with Mill documentation before proceeding, even if you don't know Java, just to get a feel for what the documentation and examples for each section should cover.

[ ] example/pythonlib/basic/ (500USD)

[ ] 1-simple/: A minimal Python module demonstrating typechecking/running/testing/pex

We should also demonstrate how to use the Python REPL hooked up to your module

[ ] 2-custom-build-logic/: A Python module with custom build logic
[ ] 3-multi-modules/: Multiple inter-related Python modules

[ ] example/pythonlib/dependencies/ (500USD)

[ ] 1-pip-deps/
[ ] 2-unmanaged-wheels/
[ ] 3-downloading-unmanaged-wheels/
[ ] 4-repository-config/: examples of how to use alternate PyPI mirrors or repositories

[ ] example/pythonlib/linting/ (1000USD)

[ ] 1-black/: equivalent to 1-scalafmt but using Black
[ ] 2-code-coverage/: not sure what the popular Python code coverage lib is, but they should have something
[ ] 3-ruff/: using Ruff to lint the Python code

[ ] example/pythonlib/module/ (500USD)

[ ] 1-common-config/
[ ] 2-custom-tasks/
[ ] 3-override-tasks/
[ ] 4-compilation-execution-flags/
[ ] 5-resources/
[ ] 6-pex-config/

[ ] example/pythonlib/testing/ (500USD)

[ ] 1-test-suite/: we should have examples of Python being used with at least two different testing frameworks: pytest and unittest
[ ] 2-test-deps/
[ ] 3-integration-suite/

[ ] example/pythonlib/publishing/1-publish-module/, example/pythonlib/basic/4-realistic/ (500USD)

publishing/1-publish-module should demonstrate how to publish a Python package to PyPI
basic/4-realistic is only doable after all the previous bullets are done, so I'm grouping its bounty together with publishing/

[ ]  example/pythonlib/web/ (500USD)

[ ] 1-hello-flask/: hello world website using Flask framework
[ ] 2-todo-flask/: TodoMVC webapp using Flask framework

[ ]  example/pythonlib/web/ (500USD)

[ ] 3-hello-django/:hello world website using Django framework
[ ] 4-todo-django/: TodoMVC webapp using Django framework

@lihaoyi Ready for this but need your Guidance (in between)

@himanshumahajan138 yes, will need to think about the best way to approach this. It would be a  pretty large scale effort and will definitely need to be broken down into parts

Aaah! Great approach for subparts
Working on this as it will require huge code base so will start from small to big
Let's do this ✨

@lihaoyi I want(have) to create a Proper PythonModule.scala using the Last PR Code(#3882) and Need some time so fully create it for each of the above-mentioned task
So, I request to avoid Clashes if anyone trying to do this please once comment...

@lihaoyi Perhaps It would be useful to clarify how pyproject.toml fits (or doesn't fit)  into your vision for adding Python support to mill.
https://peps.python.org/pep-0621/

For anyone reading, I merged https://github.com/com-lihaoyi/mill/pull/3970 which sets up the basic scaffolding:

Code in pythonlib/src/
Unit test code in pythonlib/test/src/, run via ./mill pythonlib.test
Unit test code in example/pythonlib/, run via ./mill example.pythonlib.__.local.test

From there you should be able to iterate on the examples necessary for this ticket's bullet points to flesh out the necessary functionality

@lihaoyi that's great you just gave the foundation brick for the building ✨

Also the example Python module walkthrough is mandatory reading for anyone who wants to start on this https://mill-build.org/mill/extending/example-python-support.html

@lihaoyi i will work on this expanding the previous code just after completing the kotlin springboot

There's this new Python tool https://rye.astral.sh which may be a source of inspiration for Mill support for Python. What is innovative about Rye, compared to previous Python tools is that it's aiming to be a one-stop tool for anything related to working with a Python project:

not just managing packages/dependencies, or just
managing a venv, but crucially also
managing a Python version/implementation (there's e.g. PyPy besides CPython, etc)
running the project
running the tests in the project
publishing the package to a repository
running formaters
running linters
running type checkers
etc...

If Mill covered well a broad spectrum of workflows people working with Python need to do, coupled with Mill's goodies, like implicit parallelism and aggressive caching, it could be a tempting proposal for at least part of the Python community. Good luck with this! :crossed_fingers:

There's this new Python tool https://rye.astral.sh/

Features of rye have all merged into uv https://docs.astral.sh/uv/. uv is promising and definitely the best practice for now.

Python use unittest and pytest library for this purpose

Yeah, unittest is a built-in standard library. For a demo project, it's quick and easy, though almost every main stream libs adapt pytest as test runner.

Python use unittest and pytest library for this purpose

Yeah, unittest is a built-in standard library. For a demo project, it's quick and easy, though almost every main stream libs adapt pytest as test runner.

Yes! but the way @lihaoyi is doing (check the other examples in javalib and others) i am concerned about that coz i didn't got any docs and unable to get a spike of learning 😅
I think @lihaoyi have to provide some Guide and through some light on it

Cross-posting a comment from #3992 here.
I'll work on the missing fundamentals (test support, repl etc), the @himanshumahajan138 will take care of the basic examples, while I work on the dependencies part

I'll work on dependencies next, now that #3992 is almost done

While working on the dependeny examples, I published a dummy package to test.pypi.org, to show how custom indexes can be defined. While doing so, I started looking into the way we could also add publish support to mill, so I can pick that up next too.
One thing I'm wondering is how we should handle building platform-dependent wheels? I'm not so sure how this is typically done, so I'm open to suggestions. It would be very nice if we could replicate the logic as build tasks directly in mill, without resorting to more external tools (which typically are build tools themselves and have their own notion of dependencies), but if it becomes too complex we may need to explore other options.

@jodersky I don't know, for now maybe we can punt on the issue and stick with pure-python? Compiling platform-dependent wheels would first depend on us having a decent story for compiling native C/C++ code, which we currently do not

@jodersky and @lihaoyi safe apart I am working on Web examples for flask and django (may take some time) reminder to Avoid Conflict!

@lihaoyi @jodersky while working on web examples, i came across runBackground so i want to ask should we create a new RunModule for Python or do changes in the existing PythonModule please reply asap...

Just add it to PythonModule for now

@jodersky Are we stuck???
Coz i was reading the jvm.scala file and there i saw that you used the general jvm.runSubprocess which don't provide the background run support (i think but not sure) see this:
/**
   * Runs a generic subprocess and waits for it to terminate. If process exited with non-zero code, exception
   * will be thrown. If you want to manually handle exit code, check [[runSubprocessWithResult]]
   */
  def runSubprocess(
      commandArgs: Seq[String],
      envArgs: Map[String, String],
      workingDir: os.Path
  ): Unit = {
    runSubprocessWithResult(commandArgs, envArgs, workingDir).getOrThrow
    ()
  }

  /**
   * Runs a generic subprocess and waits for it to terminate.
   *
   * @return Result with exit code.
   */
  def runSubprocessWithResult(
      commandArgs: Seq[String],
      envArgs: Map[String, String],
      workingDir: os.Path
  ): Result[Int] = {
    val process = spawnSubprocessWithBackgroundOutputs(
      commandArgs,
      envArgs,
      workingDir,
      backgroundOutputs = None
    )
    val shutdownHook = new Thread("subprocess-shutdown") {
      override def run(): Unit = {
        System.err.println("Host JVM shutdown. Forcefully destroying subprocess ...")
        process.destroy()
      }
    }
    Runtime.getRuntime().addShutdownHook(shutdownHook)
    try {
      process.waitFor()
    } catch {
      case e: InterruptedException =>
        System.err.println("Interrupted. Forcefully destroying subprocess ...")
        process.destroy()
        // rethrow
        throw e
    } finally {
      Runtime.getRuntime().removeShutdownHook(shutdownHook)
    }
    if (process.exitCode() == 0) Result.Success(process.exitCode())
    else Result.Failure(
      "Interactive Subprocess Failed (exit code " + process.exitCode() + ")",
      Some(process.exitCode())
    )
  }

Now tell me how can we add runBackground support should we use the second task mentioned above (with backgroundOutputs)
i think we can't use other runSubProcess which includes jars, classpaths mainFile and all
@jodersky @lihaoyi Please guide, it's important and i need to learn about this...

@jodersky soory to say but i am not getting the runBackground part  i read jvm.scala and runmodule.scala but still nof able to add runBackground task
I request you could you please add that to the existing pythonModule after that all gates will be opened and i will finish writing examples for web examples
By the time you do this i will add module and linting examples
Hope you understand...

I don't know too much about it. I can take a look in a couple of days

I don't know too much about it. I can take a look in a couple of days

ya i am working on module currently wrote some examples will make a pull tomorrow after that we will take a look into runBackground

Another Workaround is when we add millSourcePath into the PYTHONPATH then we can use custom-resources and resources as packages name in the importlib.resources
see:
# structure

example/pythonlib/module
└── 1-common-config
    ├── build.mill
    ├── custom-resources
    │   └── MyOtherResources.txt
    ├── custom-src
    │   └── foo2.py
    ├── resources
    │   └── MyResource.txt
    └── src
        └── foo.py

# PYTHONPATH
{
    "value": [
        "ref:v0:489b58dc:/root/openSource/mill/out/example/pythonlib/module/1-common-config/local/testCached.dest/sandbox/run-1/src",
        "ref:v0:72f129a0:/root/openSource/mill/out/example/pythonlib/module/1-common-config/local/testCached.dest/sandbox/run-1",
        "ref:v0:972ee54c:/root/openSource/mill/out/example/pythonlib/module/1-common-config/local/testCached.dest/sandbox/run-1/resources",
        "ref:v0:84efb034:/root/openSource/mill/out/example/pythonlib/module/1-common-config/local/testCached.dest/sandbox/run-1",
        "ref:v0:061699e6:/root/openSource/mill/out/example/pythonlib/module/1-common-config/local/testCached.dest/sandbox/run-1/out/generatedSources.dest/generatedSources"
    ],
    "valueHash": -756641350,
    "inputsHash": 258663986
}

# usage

        # Reading resources
        print(f"MyResource: {self.read_resource('resources', 'MyResource.txt')}")
        print(f"MyOtherResource: {self.read_resource('custom-resources', 'MyOtherResources.txt')}")

# build.mill

	def sources = Task.Sources {
		super.sources() ++ Seq(PathRef(millSourcePath))
	}

	def resources = Task.Sources {
    super.resources() ++ Seq(PathRef(millSourcePath))
  }

One more thing what if we include millSourcePath by default in the PYTHONPATH this will allow us to use millSourcePath as a resource pool and also its dirs as packages
@lihaoyi and @jodersky please once have a look and give some feedback on this asap...

Short Story:
We Should add millSourcePath by default in the PYTHONPATH to insure resource pool and package directory structure

@lihaoyi A New Question Arises
in case of scalalib(also javalib, kotlinlib) MILL_TEST_RESOURCE_DIR value is /root/openSource/mill/out/example/scalalib/module/7-resources/local/testCached.dest/sandbox/run-1/foo/test/resources but in case of pythonlib its /root/openSource/mill/example/pythonlib/module/5-resources
Why this Discrimination 😅 ?

@jodersky i agree and understood, could you please once review my PR this will allow me to move forward to complete it

Seems like Python standalone distributions are now a thing and will continue to be a thing going forward

https://astral.sh/blog/python-build-standalone

We can consider using these in Mill, especially once https://github.com/com-lihaoyi/mill/issues/3930 lands and makes downloading and caching such large binary files convenient

This is pretty neat, we could use it to replace hostPythonCommand which we currently use to bootstrap a python venv. It would also allow easy cross-builds against multiple python versions.

@lihaoyi and @jodersky i think web examples are done and just need some little grooming if required after review
**Working on Testing Examples **
these are final examples i  think coz @jodersky is working on linting and web are done by me and testing left so working on it...

@lihaoyi and @jodersky there are more testing libs in python like robot, doctest, testify, slash, ward
Should i Include them in the TestModule.scala for python please give suggestion asap...

Looking at the Jetbrains python survey (https://lp.jetbrains.com/python-developers-survey-2023/) it seems pytest and unittest are the bulk of test framework usage in Python, so I think no need to do the others

Looking at the Jetbrains python survey (https://lp.jetbrains.com/python-developers-survey-2023/) it seems pytest and unittest are the bulk of test framework usage in Python, so I think no need to do the others for now

Woah! Great Survey...
Ok then going with the pytest and unittest


Adds the missing .md file tag to the CODE_OF_CONDUCT markdown file

What
Fix the missing .md file extension for the CODUE_OF_CONDUCT file.
Why
It was causing a linking issue from the readme file.

This PR duplicates #1082


Capability to detect whether or not a terminal has colour capabilites

Description
I believe that this would be important for maintaining compatibility with old serial terminals with no colour support, as I'm pretty sure that they would just spit out garbage on the screen upon encountering an invalid escape sequence (i.e. colour escape sequences).
Implementation
This would probably be best to do by having a struct that contains the properties of any terminal which may be connected to an IO port.
This could include:

Colour support
Baud rate
Screen size (x/y)

So basically termcap and termios, right?

The kernel itself shouldn't care about color support in all its console_t objects: escape sequences are a standard and as far as I know sending invalid sequences shouldn't make a terminal explode.
Although for unprivileged (userland :trollface:) code (say, ncurses) having a terminfo library should be a must...