Roadmap for migrating existing Pangeo hubs to 2i2c operation

The Pangeo Cloud Operations Working Group met today (myself, @jhamman, @consideRatio, and @scottyhq) discussed our future plans. We have another impending sustainability crisis: @salvis2 @tomaugspurger and Scott have been doing lots of the heavy lifting in terms of operations. They have either already left (Sebastian) or are soon leaving the project. So we need to plan a transition. I'd like to use this thread to discuss that plan.
Key questions:

What infrastructure are we currently operating?
What is special / custom about Pangeo hubs / binders compared to vanilla deployments?
What do we want Pangeo Cloud Jupyter infrastructure to look like in 4 months?
What effort is required to migrate our existing deployments to 2i2c-managed hubs?
What effort is required to maintain / operate the hubs post migration?

I will try to answer some of these in subsequent replies, but for now I'll just open the issue with those questions. Anyone else should feel free to chime in.

Another key question: what kind of time frame are we working with? You mentioned that some folks are leaving "soon" but how soon are we talking and for which people?

I’m still around, just off this week :)
I’m very interested in as smooth as possible a transition.

On Nov 9, 2020, at 11:47 AM, Chris Holdgraf<EMAIL_ADDRESS>wrote:
Another key question: what kind of time frame are we working with? You mentioned that some folks are leaving "soon" but how soon are we talking and for which people?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/2i2c-org/pangeo-hubs/issues/12#issuecomment-724169348, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKAOIXRPL4WQJX2KXOH5M3SPATKPANCNFSM4TPUP2MA.

Thanks for starting the convo @rabernat . I also won't be disappearing, but do plan to scale back significantly on infrastructure-related work ;)

What effort is required to maintain / operate the hubs post migration?

pangeo infrastructure efforts have generally been flush with credits from cloud providers, but short on dedicated personnel to to develop and maintain infrastructure.  I've been lucky enough to have ~30% FTE from a NASA award over the last two years to dedicate time to keeping pangeo infrastructure running on AWS. We also had @salvis2 working ~75% on this for the last year. That award is spinning down in the next 6 months, so we're trying our best to simply keep development frozen and focus on research applications.
a lot of time is required when you want new things (dask-gateway integration, scratch buckets for all, helm chart version bumps, shiny new things like spawner modifications, custom NFS solutions, metrics, cost-tracking, etc).

What infrastructure are we currently operating?

aws-uswest2.pangeo.io
aws-uswest2-binder.pangeo.io
(2 EKS clusters running on a UW eScience-managed account)

What is special / custom about Pangeo hubs / binders compared to vanilla deployments?

automated admission to anyone with a github account who fills out a google form
object storage scratch space that wipes every 24hrs / X days
integrated with dask-gateway
generous home directory quotas ;)
curated geoscience-focused docker images
multicore notebook server comparable to 'standard laptop' specs in the cloud
runs on AWS and Azure in addition to GCP
pangeo gallery is powered by equivalent binderhubs running on GCP and AWS

What do we want Pangeo Cloud Jupyter infrastructure to look like in 4 months?

I absolutely love having access to a capable environment on AWS us-west-2, and the binderhub in AWS us-west-2 has been instrumental for demos and workshops. The hubs and binderhubs are a rallying point for the pangeo community and the most frictionless way to get people experimenting with cloud-based workflows and dask distributed computing in my opinion. Their fully open nature is also extremely unsustainable!
At the very least I'd like to see an open-access binderhub on AWS us-west-2 maintained. I'd love for 2i2c to figure out a pay-per-team pangeo hub solution, where usage costs are tracked at the research group or grant team level. And in the meantime, if we are flush with credits and have a person dedicated to maintenance, I'd selfishly love for the AWS us-west-2 jupyterhub to continue as is!

I have a couple thoughts...
If Pangeo pivots into a more operational service its probably worth talking to CloudBank: For back office stuff; account management. Still leaves a tech-ops gap but it might lead to something.
I suggest set aside 2+ days for an internal "messaging workshop". All the comments above suggest this to me. Maybe the goal of that time spent would be a draft proposal. I use the term 'messaging' because Pangeo is among other things a testbed and this -- I claim! -- raises some impedence matching questions. Someone might ask "Isn't Pangeo too much horsepower / too experimental for the 2i2c democratization plan?' and so on.

@robfatland could you go into more detail on this comment:

"Isn't Pangeo too much horsepower / too experimental for the 2i2c democratization plan?'

And just as extra explanation - the goal of 2i2c is not just to democratize access to straightforward hub infrastructure, but also to be a place for cutting-edge-style development in collaboration with research/education partners. Those two kinds of models won't be the same hub (or the same group of collaborators), but I think the vision for 2i2c is that it can help amplify the kinds of experimental work done in Pangeo in a more sustainable manner.

What infrastructure are we currently operating?

Just want to mention that our main deployment is still the GCP-based cluster https://us-central1-b.gcp.pangeo.io/.

Just cross referencing an issue on the pangeo-cloud-federation repo that we basically have no capacity right now to handle: Action Required: Suspicious Activity Observed on Google Cloud Project pangeo . This highlights the need for someone to monitor these hubs on a regular basis.

I just posted https://github.com/pangeo-data/pangeo-cloud-federation/issues/874, which tries to document some of the special things about our GCP deployment.
I'm mostly offline for the next few days still, but I want to make a transition to 2i2c as smooth as possible.

Thinking of two kinds of deployments, mostly from the perspective of who / what pays for them.

'community' deployments, open for a wide array of folks to use. This would be all the hubs in pangeo-cloud-federation right now, primarily funded by grants to 'further the mission', as such. Would include all the binderhubs too.
'Organizational' deployments, restricted to a particular community of users from a distinct 'organization'. 2i2c run a hub for Farallon Institute for use by folks there. These are almost 'internal' hubs, but run exactly like PANGEO hubs - just different funding source and support setups.

Funding source and support expectations differentiate (1) and (2), and we should aim to keep everything else exactly the same. This eases support burden.
Operationally, I think the next important thing is probably to make sure that the community deployments can keep going. Given that credits seem to exist, it looks like to me the issue is one of onboarding 'maintainers' - folks who can take care of the infrastructure and work on improvements where needed - the jobs that @TomAugspurger, @scottyhq, @salvis2 and others have been graciously doing for now. Does this sound right?

Given that credits seem to exist, it looks like to me the issue is one of onboarding 'maintainers' - folks who can take care of the infrastructure and work on improvements where needed - the jobs that @TomAugspurger, @scottyhq, @salvis2 and others have been graciously doing for now. Does this sound right?

Correct. But we are not expecting anyone to do this for free. Our Moore Foundation and NSF EarthCube grants have funding for this type of work. (Just money flowed to Anaconda and UW for this purpose.)

Once https://github.com/2i2c-org/pilot-hubs/pull/134 is merged, we can create many Pangeo style hubs (notebook + dask) by editing hubs.yaml trivially. Eventually, I want to be able to trivially spin up / down hubs with minimal setup cost, so we can do ephemeral clusters for specific events without putting a lot of pressure on long-running hubs.

@yuvipanda and I had a great meeting today with the Pangeo cloud infrastructure  working group. (@TomAugspurger and @jhamman were the other attendees). We did a deep dive on the compatibility between Pangeo's existing hubs and what 2i2c is doing. The good news is that it looks like the migration should be pretty straightforward.
Notes from the meeting are here:
https://docs.google.com/document/d/1I-2VNNHoAjjeYvlCezQhFLmiu2OevqGDS5nUAP-6Hfw/edit
We can work on translating these to some specific action items as issues in this repo.

@yuvipanda and I had a great meeting today with the Pangeo cloud infrastructure  working group. (@TomAugspurger and @jhamman were the other attendees). We did a deep dive on the compatibility between Pangeo's existing hubs and what 2i2c is doing. The good news is that it looks like the migration should be pretty straightforward.
Notes from the meeting are here:
https://docs.google.com/document/d/1I-2VNNHoAjjeYvlCezQhFLmiu2OevqGDS5nUAP-6Hfw/edit
We can work on translating these to some specific action items as issues in this repo.

https://github.com/berkeley-dsep-infra/datahub/pull/2124#issuecomment-767948309 from @rabernat as the kinda thing that'd be nice in these hubs :)

https://github.com/berkeley-dsep-infra/datahub/pull/2124#issuecomment-767948309 from @rabernat as the kinda thing that'd be nice in these hubs :)


[Service bug] Change CopyAuthorization.expiresOn from long to OffsetDateTime

The service currently has a bug where the return value can't be parsed into a DateTimeOffset.
Once they fixed it, we need to update our type from long -> DateTimeOffset

Closed with  #12320


Bigger batch size with less memory consumption

Hello,
I do fine tuning on a custom dataset and I want to try, if a bigger batch size results in a better performance. But my graphics card has not enough memory to do a batch size bigger than 1. So I looked up a solution, but the final performance drops and the network converges slower with higher batch sizes (the higher the batch size, the slower the convergence).
My code is printed below. I know that the training time will be nearly the same with my code, but with performance I mean better Average Precision. I read that this code is exactly the same as real bigger batch sizes, except for batch normalization layers. Could the problem be the batch normalization layers in the ResNet backbone or would that have another impact?
I changed these lines:
https://github.com/fundamentalvision/Deformable-DETR/blob/dc33c7de943647fb306baed46105293dbefb589b/engine.py#L40-L68
To:
optimizer.zero_grad()
# for samples, targets in metric_logger.log_every(data_loader, print_freq, header):
for i in metric_logger.log_every(range(len(data_loader)), print_freq, header):
    outputs = model(samples)
    loss_dict = criterion(outputs, targets)
    weight_dict = criterion.weight_dict
    losses = sum(loss_dict[k] * weight_dict[k] for k in loss_dict.keys() if k in weight_dict)

    # reduce losses over all GPUs for logging purposes
    loss_dict_reduced = utils.reduce_dict(loss_dict)
    loss_dict_reduced_unscaled = {f'{k}_unscaled': v
                                  for k, v in loss_dict_reduced.items()}
    loss_dict_reduced_scaled = {k: v * weight_dict[k]
                                for k, v in loss_dict_reduced.items() if k in weight_dict}
    losses_reduced_scaled = sum(loss_dict_reduced_scaled.values())

    loss_value = losses_reduced_scaled.item()

    if not math.isfinite(loss_value):
        print("Loss is {}, stopping training".format(loss_value))
        print(loss_dict_reduced)
        sys.exit(1)

    # optimizer.zero_grad()
    losses.backward()

    if (i+1)%mini_batchsize == 0:
        if max_norm > 0:
            grad_total_norm = torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm)
        else:
            grad_total_norm = utils.get_total_grad_norm(model.parameters(), max_norm)
        optimizer.step()
        optimizer.zero_grad()

Thank you in advance!

In fact, a bigger batch size not always produces a better result at the same epochs, especially when batch normalization is frozen, since bigger batch size means less iterations to update the parameters, which may harm the performance when the model doesn't sufficiently converge.


Multiple StreamListeners errors when connected to Kafka

Apologies if this isn't the right forum to post this, as I'm not sure if it's a bug or (more likely) something I'm doing wrong.  However I tried posting a question on StackOverflow and had no luck.
I have two methods annotated with @StreamListener. One takes in two KTables, and the other takes in a KStream. The Spring Boot app starts correctly, but after a few seconds this error is shown:
Exception in thread "spring-cloud-stream-kafka-12427e22-82cc-4368-b07c-09573ada6670-StreamThread-2" java.lang.IllegalArgumentException: Assigned partition t3-0 for non-subscribed topic regex pattern; subscription pattern is t1|t2
	at org.apache.kafka.clients.consumer.internals.SubscriptionState.assignFromSubscribed(SubscriptionState.java:195)
	at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:225)
	at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:367)
	at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:316)
	at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:295)
	at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1146)
	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1111)
	at org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:848)
	at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:805)
	at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:771)
	at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:741)

The code is as follows (in Kotlin):
@SpringBootApplication
@EnableBinding(T1T2Binding::class, T3Binding::class)
class Application {

    @StreamListener
    fun t1Handler(@Input("t1") t1: KTable<String, String>, @Input("t2") t2: KTable<String, String>) {

    }

    @StreamListener
    fun t2Handler(@Input("t3") t3: KStream<String, String>) {
        t3.map { key, value ->
            KeyValue(key, value)
        }
    }
}

interface T1T2Binding {
    @Input
    fun t1(): KTable<String, String>

    @Input
    fun t2(): KTable<String, String>
}

interface T3Binding {
    @Input
    fun t3(): KStream<String, String>
}

And the application.yml file is:
spring.cloud.stream:
  bindings:
    t1:
      destination: t1
      consumer:
        headerMode: raw
    t2:
      destination: t2
      consumer:
        headerMode: raw
    t3:
      destination: t3
      consumer:
        headerMode: raw
  kafka.streams:
    binder:
      brokers: localhost
      zk-nodes: localhost
      configuration.commit.interval.ms: 1000
      application-id: "spring-cloud-stream-kafka"

A github repo where this problem can be replicated is available here.
Thanks in advance for any pointers on this.

@yonigibbs Noticed the issue on Stack Overflow. We will get to it as soon as possible (within a day or two). Thanks!

Great stuff, thanks @sobychacko.

@yonigibbs When you have multiple processors (StreamListener methods) in the same application, you need to use different application id for each of the input bindings. Here is a sample application that demonstrates multiple processors. Look at the configuration yaml to see how a unique application id is provided for each input binding. Based on your config above, you need to provide, these 3 properties to the configuration.
spring.cloud.stream.kafka.streams.bindings.t1.consumer.application-id=processor1-application-id
spring.cloud.stream.kafka.streams.bindings.t2.consumer.application-id=processor1-application-id
spring.cloud.stream.kafka.streams.bindings.t3.consumer.application-id=processor2-application-id

You probably want to test with the latest snapshots (2.1.0) as we made some recent changes in the way applicaion id is processed by the binder.

Thanks @sobychacko, that did indeed sort it, and you're right that I had to upgrade to the latest snapshot release, otherwise the changes had no impact.  I guess that's maybe related to issue 448.
I was going to suggest that the documentation might need to be clarified about this, but I've just spotted you've already done this as part of this commit.
So, thanks again!


Try building with Docker

Hi Paul,
I'm trying to get Unison to run in Docker which would make it super-easy for people to try it out by just doing a
docker run unisonweb/platform

Unfortunately I'm completely new to nix and cabal and compilation fails with changing reasons, the last one being
Preprocessing library base64-bytestring-<IP_ADDRESS>...
libtool: link: gcc -std=gnu99 -DUSE_POSIX_THREADS -g -O2 -o .libs/gettext-6-prg gettext_6_prg-gettext-6-prg.o  ../gnulib-lib/.libs/libgettextlib.so ../intl/.libs/libgnuintl.so -lc -lpthread -fopenmp -Wl,-rpath -Wl,/nix/store/lsgh3rjy7mipimjrgi5cnjz6nvq7rahy-gettext-0.18.2/lib
gcc -std=gnu99 -DLOCALEDIR=\"/nix/store/lsgh3rjy7mipimjrgi5cnjz6nvq7rahy-gettext-0.18.2/share/locale\" -DHAVE_CONFIG_H -I. -I..  -I.. -I../gnulib-lib -I../gnulib-lib -I../intl  -DUSE_POSIX_THREADS -g -O2 -c -o gettext_7_prg-gettext-7-prg.o `test -f 'gettext-7-prg.c' || echo './'`gettext-7-prg.c
/nix/store/0fmbypn0w29r3lykg5w1glbphz1pdbkz-bash-4.3-p33/bin/bash ../libtool  --tag=CC   --mode=link gcc -std=gnu99 -DUSE_POSIX_THREADS -g -O2   -o gettext-7-prg gettext_7_prg-gettext-7-prg.o ../gnulib-lib/libgettextlib.la ../intl/libgnuintl.la    -lpthread

malloc: dispose_cmd.c:249: assertion botched
free: called with already freed block argument
Aborting...Makefile:1810: recipe for target 'gettext-7-prg' failed
make[3]: *** [gettext-7-prg] Aborted
make[3]: Leaving directory '/tmp/nix-build-gettext-0.18.2.drv-0/gettext-0.18.2/gettext-tools/tests'
Makefile:1660: recipe for target 'all-recursive' failed
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory '/tmp/nix-build-gettext-0.18.2.drv-0/gettext-0.18.2/gettext-tools'
Makefile:1525: recipe for target 'all' failed
make[1]: *** [all] Error 2
make[1]: Leaving directory '/tmp/nix-build-gettext-0.18.2.drv-0/gettext-0.18.2/gettext-tools'
Makefile:315: recipe for target 'all-recursive' failed
make: *** [all-recursive] Error 1
builder for ‘/nix/store/p0b9lzdxi598zq8i9fp4p6qciixxkazd-gettext-0.18.2.drv’ failed with exit code 2
cannot build derivation ‘/nix/store/dscyq920pg351zi72wqg1sg0cbp14qlc-git-2.3.6.drv’: 1 dependencies couldn't be built

Could you maybe try if a
docker build -t unison .

in this branch works on your machine?
Best
Martin
BTW: I hope it's OK that I already created a PR with this although it's not yet running correctly. I find PRs very useful for tracking code and comments in the same place.

Cool. I don't know much about docker at all, but I'd have no objections to
merging this if you can get it working. Maybe look at the vagrantfile and
provisioning script, those are known to work, I'm guessing docker
provisioning will be similar? See the info on this in the README.
I'll also try running that on my machine.
On Thu, Sep 17, 2015 at 5:16 PM Martin Mauch<EMAIL_ADDRESS>wrote:

Hi Paul,
I'm trying to get Unison to run in Docker which would make it super-easy
for people to try it out by just doing a
docker run unisonweb/platform
Unfortunately I'm completely new to nix and cabal and compilation fails
with changing reasons, the last one being
Preprocessing library base64-bytestring-<IP_ADDRESS>...
libtool: link: gcc -std=gnu99 -DUSE_POSIX_THREADS -g -O2 -o .libs/gettext-6-prg gettext_6_prg-gettext-6-prg.o  ../gnulib-lib/.libs/libgettextlib.so ../intl/.libs/libgnuintl.so -lc -lpthread -fopenmp -Wl,-rpath -Wl,/nix/store/lsgh3rjy7mipimjrgi5cnjz6nvq7rahy-gettext-0.18.2/lib
gcc -std=gnu99 -DLOCALEDIR="/nix/store/lsgh3rjy7mipimjrgi5cnjz6nvq7rahy-gettext-0.18.2/share/locale" -DHAVE_CONFIG_H -I. -I..  -I.. -I../gnulib-lib -I../gnulib-lib -I../intl  -DUSE_POSIX_THREADS -g -O2 -c -o gettext_7_prg-gettext-7-prg.o test -f 'gettext-7-prg.c' || echo './'gettext-7-prg.c
/nix/store/0fmbypn0w29r3lykg5w1glbphz1pdbkz-bash-4.3-p33/bin/bash ../libtool  --tag=CC   --mode=link gcc -std=gnu99 -DUSE_POSIX_THREADS -g -O2   -o gettext-7-prg gettext_7_prg-gettext-7-prg.o ../gnulib-lib/libgettextlib.la ../intl/libgnuintl.la    -lpthread
malloc: dispose_cmd.c:249: assertion botched
free: called with already freed block argument
Aborting...Makefile:1810: recipe for target 'gettext-7-prg' failed
make[3]: *** [gettext-7-prg] Aborted
make[3]: Leaving directory '/tmp/nix-build-gettext-0.18.2.drv-0/gettext-0.18.2/gettext-tools/tests'
Makefile:1660: recipe for target 'all-recursive' failed
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory '/tmp/nix-build-gettext-0.18.2.drv-0/gettext-0.18.2/gettext-tools'
Makefile:1525: recipe for target 'all' failed
make[1]: *** [all] Error 2
make[1]: Leaving directory '/tmp/nix-build-gettext-0.18.2.drv-0/gettext-0.18.2/gettext-tools'
Makefile:315: recipe for target 'all-recursive' failed
make: *** [all-recursive] Error 1
builder for ‘/nix/store/p0b9lzdxi598zq8i9fp4p6qciixxkazd-gettext-0.18.2.drv’ failed with exit code 2
cannot build derivation ‘/nix/store/dscyq920pg351zi72wqg1sg0cbp14qlc-git-2.3.6.drv’: 1 dependencies couldn't be built
Could you maybe try if a
docker build -t unison .
in this branch works on your machine?
Best
Martin
You can view, comment on, or merge this pull request online at:
https://github.com/unisonweb/platform/pull/32
Commit Summary

Try building with Docker

File Changes

A .dockerignore
https://github.com/unisonweb/platform/pull/32/files#diff-0 (1)
A Dockerfile
https://github.com/unisonweb/platform/pull/32/files#diff-1 (4)
A docker-provision.sh
https://github.com/unisonweb/platform/pull/32/files#diff-2 (14)

Patch Links:

https://github.com/unisonweb/platform/pull/32.patch
https://github.com/unisonweb/platform/pull/32.diff

—
Reply to this email directly or view it on GitHub
https://github.com/unisonweb/platform/pull/32.

Yes, I've tried to more or less replicate the Vagrant setup.
I've managed to manually get unison running by logging into the container and starting shell.sh multiple times.
Currently I'm trying to do the same in the Docker build with shell.sh || shell.sh || shell.sh
Is it normal that there are so many things that need to be compiled? Are those packages not provided in the extra-binary-caches?
Ideally, we would configure an automated build on the official Docker registry which would allow people to always get the newest version of unison.
But they have limits wrt to build time (and probably also memory) which the package compilation definitely breaks.

Compilation takes 10-15 minutes I think the first time, but it's a one time
cost. Ryan's cache just has all reflex related deps and it's missing some
dependencies of the unison node I think.
I'm sure it is possible to build a nix cache that has everything we need,
but I haven't looked into that and we'd have to run a server somewhere.
We'd want the cache to have all Mac and Linux binaries for all deps used by
the editor and node.
On Fri, Sep 18, 2015 at 8:19 AM Martin Mauch<EMAIL_ADDRESS>wrote:

Yes, I've tried to more or less replicate the Vagrant setup.
I've managed to manually get unison running by logging into the container
and starting shell.sh multiple times.
Currently I'm trying to do the same in the Docker build with shell.sh ||
shell.sh || shell.sh
Is it normal that there are so many things that need to be compiled? Are
those packages not provided in the extra-binary-caches?
Ideally, we would configure an automated build on the official Docker
registry which would allow people to always get the newest version of
unison.
But they have limits wrt to build time (and probably also memory) which
the package compilation definitely breaks.
—
Reply to this email directly or view it on GitHub
https://github.com/unisonweb/platform/pull/32#issuecomment-141434468.

https://github.com/commercialhaskell/stack/issues/337
https://github.com/commercialhaskell/stack/issues/749
Show some work getting GHCJS buildable with stack. Hopefully this means we could use a more standard set of packages than we do now, and thereby get by with the official cache.

@nightscape if you are still interested, you could take another look at this now that the build has been converted over to Stack (see #34) and does not use Nix anymore. It might be that this is now a lot simpler. This page in the Stack guide might be useful, not sure.

I'm still interested, although maybe not the best person to do it due to lack of any Cabal/Stack knowledge. I've asked in the repo of the official Haskell Docker image if they can provide an example on how to best build Stack based projects. If they comment on that I'll try and apply it to Unison.

Cool.
On Sat, Jan 30, 2016 at 3:40 PM Martin Mauch<EMAIL_ADDRESS>wrote:

I'm still interested, although maybe not the best person to do it due to
lack of any Cabal/Stack knowledge. I've asked in the repo of the official
Haskell Docker image if they can provide an example on how to best build
Stack based projects. If they comment on that I'll try and apply it to
Unison.
—
Reply to this email directly or view it on GitHub
https://github.com/unisonweb/platform/pull/32#issuecomment-177295773.

@pchiusano Do you happen to know if one can tell stack to just download and compile the dependencies, not the code itself?
This is essential for the Docker build to get to an acceptable speed, otherwise every single change in any file will bust the Docker cache and require starting all over with the dependencies...

Try stack build --only-dependencies
On Tue, Feb 2, 2016 at 7:59 PM Martin Mauch<EMAIL_ADDRESS>wrote:

@pchiusano https://github.com/pchiusano Do you happen to know if one
can tell stack to just download and compile the dependencies, not the code
itself?
This is essential for the Docker build to get to an acceptable speed,
otherwise every single change in any file will bust the Docker cache and
require starting all over with the dependencies...
—
Reply to this email directly or view it on GitHub
https://github.com/unisonweb/platform/pull/32#issuecomment-178929371.

Ok, I've got everything built inside Docker. It takes ages the first time, but it should be quite fast after that until you need to touch the .cabal or (even worse) the stack.yaml files ;)
Unfortunately, the generated editor.html assumes that one can access the compiled .js files via a path on disk which won't work with Docker. Could you change that so that the .js files get served by Scotty?
I think once that is done, we should have a working Docker build :+1:

Forgot to mention: The editor.html file itself also needs to be served by Scotty (or some other way via TCP port).

Ok, almost got it :D

And here we go! I'm just uploading my locally built image to docker hub, once the upload is finished (it's a hefty 5.6 GB) you can try it using the following command
docker run -it -p 8080:8080 --name nightscape/unison --rm unison

and then browsing to http://localhost:8080/.
Once this PR is merged you (or I) could create an automated build on Docker Hub so that people can always get the latest version (you can also select to build only dedicated branches).
I just hope that the automated build is not detected as DoS and killed :grin:
These are the limits to the automated build:

2 hours maximum build time
2 GB RAM
1 CPU
30 GB Disk Space

Hey, let's just get rid of the make-editor-html script now, the only reason for that was to lookup those system dependent paths. Instead let's just check in the editor.html file that refers to localhost in the path. Also please update the README accordingly.

Ok, got rid of the make-editor-script and adapted the README.
Any preferences on how the Docker image should be called? I used unisonweb/platform like the Github project in the README, but I can change it if you prefer another name.
It would be good to use the same name when building yourself and when pulling the official image from Docker Hub.

I would just call it 'unison' if that is available, otherwise 'unisonweb'. The 'platform' is not necessary. The GitHub org uses that just to distinguish from the unisonweb.org site, which is also under the unisonweb GitHub org.

You have to give it a organization/name tag though. The name builds are reserved for "official" Docker images, i.e. the ones listed here.

And I'm not a 100% sure if the name of an automated build can deviate from the Github repo name, checking that now...


Kafka producers throwing TimeoutExceptions

Problem description
Spring Kafka SSL producers are throwing timeouts when producing to a Kafka cluster with 3 brokers.

The SSL configuration is correct and messages are sent correctly for large periods of time (e.g. hours) before showing intermittent problems.
Both the server and clients have plenty of resources left.
The network is stable and powerful enough (200 mbit, no downtime in the last year).
The errors do not appear in a PLAINTEXT setup.
Configurations for the three brokers are identical apart from their broker ID and listeners.
No errors appear in the broker log or Spring Kafka consumers.

2018-06-06 11:15:44.103 ERROR 1 --- [ad | producer-1] o.s.k.support.LoggingProducerListener    : Exception thrown when sending a message with key='null' and payload='[...redacted...]': org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for topicname-0: 30001 ms has passed since last append
Detailed description
https://stackoverflow.com/questions/50725643/spring-kafka-producers-throwing-timeoutexceptions
Question
How can I determine if this is a problem on my end, or a bug?

Most likely a question best directed to the Apache Kafka folks rather than here, since we're just using their client.


syslogng: 4.7.1 -> 4.8.0

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/syslog-ng/syslog-ng/releases.
meta.description for syslogng is: Next-generation syslogd with advanced networking and filtering capabilities
meta.homepage for syslogng is: https://www.syslog-ng.com
Updates performed

Version update

To inspect upstream changes
Impact
Checks done

built on NixOS
The tests defined in passthru.tests, if any, passed
found 4.8.0 with grep in /nix/store/r6ppc9w335v7dnyi6x2wmshkgmsr4xa3-syslog-ng-4.8.0
found 4.8.0 in filename of file in /nix/store/r6ppc9w335v7dnyi6x2wmshkgmsr4xa3-syslog-ng-4.8.0

Rebuild report (if merged into master) (click to expand)

2 total rebuild path(s)

2 package rebuild(s)

First fifty rebuilds by attrpath

syslogng

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/r6ppc9w335v7dnyi6x2wmshkgmsr4xa3-syslog-ng-4.8.0 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A syslogng https://github.com/r-ryantm/nixpkgs/archive/432044471635f115ad0335fd8cbc14590063d58f.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/432044471635f115ad0335fd8cbc14590063d58f#syslogng

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/r6ppc9w335v7dnyi6x2wmshkgmsr4xa3-syslog-ng-4.8.0
ls -la /nix/store/r6ppc9w335v7dnyi6x2wmshkgmsr4xa3-syslog-ng-4.8.0/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

2 packages built:

syslogng
syslogng.man

Maintainer pings
cc @vifino for testing.

Result of nixpkgs-review pr 329532 run on x86_64-linux 1

2 packages built:

syslogng
syslogng.man


Collate and add project portfolio

Changes Unknown when pulling 43e1799e397ed08eaf181c9f74381f3e298be5a1 on Valerieyue:ChartEnhancement into ** on CS2103AUG2017-T16-B3:master**.

Coverage remained the same at 92.485% when pulling 43e1799e397ed08eaf181c9f74381f3e298be5a1 on Valerieyue:ChartEnhancement into f96d801ddb28119625d4d0caa9ca2014cb00d76e on CS2103AUG2017-T16-B3:master.

Coverage remained the same at 92.485% when pulling 43e1799e397ed08eaf181c9f74381f3e298be5a1 on Valerieyue:ChartEnhancement into f96d801ddb28119625d4d0caa9ca2014cb00d76e on CS2103AUG2017-T16-B3:master.


Develop

See Issue #14

Coverage decreased (-0.001%) to 99.299% when pulling a7a2c63fcde5411ee09beca15ad3fa45754cc9e1 on develop into 208f9234fe57aaabc54b70b43b4be2df2226e04a on master.

Coverage decreased (-0.001%) to 99.299% when pulling a7a2c63fcde5411ee09beca15ad3fa45754cc9e1 on develop into 208f9234fe57aaabc54b70b43b4be2df2226e04a on master.

Coverage decreased (-0.001%) to 99.299% when pulling a7a2c63fcde5411ee09beca15ad3fa45754cc9e1 on develop into 208f9234fe57aaabc54b70b43b4be2df2226e04a on master.

Coverage decreased (-0.001%) to 99.299% when pulling a7a2c63fcde5411ee09beca15ad3fa45754cc9e1 on develop into 208f9234fe57aaabc54b70b43b4be2df2226e04a on master.


Assist with image change and figure out credentials OpenEMR Collective

Matthew has this very cool project (https://opencollective.com/OpenEMR) he created an open collective using Github account and he is trying to figure out what his credentials are.
Also he requested if we could change the logo of his collective because he just got one of his first backers, so that's his No. 1 priority: change logo.
Here it is:

done


"Value cannot be null" Error When Installing "markdig" from Nuget

Hello. I tried to install Markdig from the Nuget package and am getting the following NULL error (see below, copied from terminal). Have tried it on two different computers and both act the same. Also, I renamed the Nuget cache folder (%appdata%\Nuget) on one of them to refresh it, but that didn't make a difference either. Lastly, I tried installing the next latest version (0.36.2) and it didn't work. Guess that means it's related to my two computers. What am I doing wrong? Thanks.
*PS C:\Users\me\source\repos\Markdig\Markdig> dotnet add "c:\Users\me\source\repos\Markdig\Markdig\Markdig.csproj" package "Markdig" -v "0.37.0"
  Determining projects to restore...
  Writing C:\Users\me\AppData\Local\Temp\tmpts3rid.tmp
info : X.509 certificate chain validation will use the default trust store selected by .NET for code signing.
info : X.509 certificate chain validation will use the default trust store selected by .NET for timestamping.
info : Adding PackageReference for package 'Markdig' into project 'c:\Users\me\source\repos\Markdig\Markdig\Markdig.csproj'.
info : Restoring packages for c:\Users\me\source\repos\Markdig\Markdig\Markdig.csproj...
error: NU1108: Cycle detected.
error:   Markdig -> Markdig (>= 0.37.0).
info : Package 'Markdig' is compatible with all the specified frameworks in project 'c:\Users\me\source\repos\Markdig\Markdig\Markdig.csproj'.
error: Value cannot be null. (Parameter 'path1')*

Definitely an install/setup problem.
Not sure adding a Markdig PackageReference to a Markdig.csproj is a good idea to start with. The dll of your project will override the one from Markdig. Use different folder names. Nothing will correct correctly.

Changing the folder/project name fixed it. I was just doing some testing, not an actual project yet, so I didn't come up with a better name. Live and learn I guess. At any rate, it is working now. Thank you for the quick response.


None is not correctly serialized for certain optional query params

Describe the bug
In https://github.com/triaxtec/openapi-python-client/pull/331, we decided to allow None to be passed in optional query params and treated identically to UNSET.
However, a few cases weren't supported:

model properties
enum properties
datetime properties

I haven't done an exhaustive search so there may be others as well; potentially any types that require non-trivial JSON serialization.
To Reproduce
This can already be seen in end_to_end_tests/golden-record/my_test_api_client/api/tests/defaults_tests_defaults_post.py:90, which will crash if None is passed in for the model.
Expected behavior
For all types, if the query param is optional, None is treated as UNSET.
OpenAPI Spec File
The existing openapi.json
Desktop (please complete the following information):

OS: [e.g. macOS 10.15.1]
Python Version: [e.g. 3.8.0]
openapi-python-client version [e.g. 0.1.0]

Additional context

cc @dtkav @bowenwr @packyg

Resolved in 0.10.2


Call to undefined method Fitbit::parsedResponse()

Trying to get the example from the readme to work I succeded pretty far until  $response = $provider->getParsedResponse($request); to which I get the output:
Fatal error: Call to undefined method djchen\OAuth2\Client\Provider\Fitbit::parsedResponse()

Which seems to tell me that a function I want to invoke is missing. Looked through the sourcecode and could find no reference to said function either.
At a loss as to how I can make this work.

I think you have the 0.3.x version of the library installed.
If thats the case, follow the README from here: https://github.com/djchen/oauth2-fitbit/tree/0.3.1
Or you can alternatively install the 1.0.0-beta2 version.

How could I check which version I am using and update?
Totally new to composer and just ran the composer require djchen/oauth2-fitbit I found in the README.

composer show will print a list of dependent libraries and their versions.

Thanks for the help. Got it to work now. Think I was confused at the end why composer would install the up-to-date version, not realizing that the main README is for a beta version.

Trying to use refresh token to get new access_token but it giving and error Call to undefined method stdClass::hasExpired(). Following your example code but having no luck can u please help.

Please open a new issue and provide more information on how to reproduce the same setup. Thanks!


Reword comment to not use "require"

Longshot I know...
browserify does a quick regexp match for \brequire\b before deciding on whether to build an AST to get the require calls. By rewording this comment so that the word "require" is nowhere in the source, you'd be improving everyones build times. Browserify provides a noparse escape hatch but this is an easy win.

Cool stuff and good to know


[BUG] Extension badge is not showing on Linux

The extension badge is not showing on Linux for me:

It's not hidden:

On Windows it's ok. Only on Linux this happens. Do you think it could be an issue in the extension?
Environment

VS Code version: 1.83.1
Commit: f1b07bd25dfad64b0167beb15359ae573aecd2cc
Date: 2023-10-10T23:45:31.402Z
Electron: 25.8.4
ElectronBuildId: 24154031
Chromium: 114.0.5735.289
Node.js: 18.15.0
V8: <IP_ADDRESS>-electron.0
OS: Linux x64 3.10.0-1160.92.1.el7.x86_64 (CentOS Linux release 7.9.2009 (Core))
Config: no json config edited for this extension; I'm using slang-6.0 and uctags-2024.06.20-linux-x86_64 added to Linux PATH

Logs:
2024-07-31 13:33:14.726 [info] [index] index activating
2024-07-31 13:33:14.726 [info] [ctags] activating
2024-07-31 13:33:14.726 [info] [lint] activating lint manager
2024-07-31 13:33:14.726 [info] [lint] [slang] linter config updated
2024-07-31 13:33:14.726 [info] [lint] [modelsim] linter config updated
2024-07-31 13:33:14.726 [info] [lint] [iverilog] linter config updated
2024-07-31 13:33:14.726 [info] [lint] [verilator] linter config updated
2024-07-31 13:33:14.726 [info] [lint] [xvlog] linter config updated
2024-07-31 13:33:14.726 [warning] [lint] skipping lint for non verilog file
2024-07-31 13:33:15.357 [info] [ctags] indexing 0 .svh files
2024-07-31 13:33:15.358 [info] [ctags] indexed 0 .svh files
2024-07-31 13:33:15.358 [info] index files finished
2024-07-31 13:33:15.358 [info] AndrewNolte.vscode-system-verilog activation finished.
2024-07-31 13:33:15.358 [info] ctags index includes finished
2024-07-31 13:33:26.354 [warning] [lint] skipping lint for non verilog file

I think this is most likely an issue with Vscode. The chip icon is a standard vscode icon that the extension references. Do you want to open this issue on the vscode repo? https://github.com/microsoft/vscode/issues


Link to OpenWeave from nest.com website broken.

from nest.com link to openweave.io ends in google.com

Sorry, it was temporarily down while we were in the process of launching our new site at openweave.io.  Everything should be working now.


Export all asciicasts available in my profile

I saw that #159 was closed so I'll ask a related issue, but a bit more specific.
I have almost 100 asciicasts on my account. I use them in my own website for educational purpose. So I have "invested" in this service. I'm not really interested in setting up my own server, I'm rather happy with the excellent service as it is.
But, to avoid disaster I would like to get a backup of all my asiicasts, I would like to export all my asciicasts I have uploaded to asciinema.org. This is just in case something happens, like asciinema.org gets sold, license is changed and original website gets closed down and I loose all my asciicasts and my own site gets broken until I record it all again...
Can it be done today - exporting all uploaded asiicasts - easy or hard way? Could it be integrated into the website? And/Or could the commandline utility automatically save a copy of any uploaded asciicast (this is really a question for the recorder repo - I'll ask it there depending on the answer I get here).

First, I don't have a plan to sell it, neither to change the license nor shut it down. It's not a product, and I don't plan to turn it into business (because it would make it more serious and less fun, and fun is more precious than money).
But nobody knows the future and I fully understand your concern. Independence of asciicasts from asciinema.org service was something that was discussed by several people. Actually, if I was in your position (having "invested" this much in the service) I'd be equally concerned about it.
Because it's not a product/business I don't have a reason to create "vendor lock-in". That's why my focus now is on decoupling the recordings from the service and allowing people to self-host them in an easy way. asciinema.org is of course going to stay for those who don't want to self-host (most?).
First part of the process of decoupling was creating an official asciicast format and allowing recording to local files (also see 1.0 annoucement). Next part is to update the JS player so it's able to consume asciicasts directly without a need for pre-processing step, which is currently required, and done by asciinema.org. Once that's done you'll be able to embed asciicasts on your pages by downloading player (1 JS file + 1 CSS file + optionally a theme CSS file) and the actual asciicast file, and adding them to your HTML. The other self-hosting option I plan to add is ability to run your own "mini asciinema.org" site with asciinema server /path/to/your/asciicasts -p 8080, and embed asciicasts in your pages directly from there (yourdomain:8080).
Re exporting: It's not trivial but possible to export the asciicasts now, so if you can't wait for the described work to be finished I can give you a list of manual steps for exporting. Anyway, this topic is my focus for the following couple of months. I hope this makes you more comfortable.

Yep, makes me comfortable. Seems like you have it all under control. I will sit tight and continue investing. I like the plan you present above.

@sickill High five for this!
Looking forward to 1.0 then :-)

@AVGP 1.0 is already out there. The plan described is for 1.1 or 1.2.

 Soooo, I'm looking forward to exporting / locally recording asciicasts soon, then ^^


Add support for Kubernetes 1.12.0

What this PR does / why we need it:
See https://github.com/kubernetes/kubernetes/releases/tag/v1.12.0
and https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.12.md#v1120
TODO:

[x] upload Windows binary

If applicable:

[ ] documentation
[ ] unit tests
[ ] tested backward compatibility (ie. deploy with previous version, upgrade with this branch)

Release note:

Add support for Kubernetes 1.12.0

HPA is failing after 1.11-->1.12 upgrades.
$ kubectl describe hpa php-apache-kubernetes-uksouth-30786-843
Name:                                                  php-apache-kubernetes-uksouth-30786-843
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           <none>
CreationTimestamp:                                     Tue, 02 Oct 2018 16:04:08 +0000
Reference:                                             Deployment/php-apache-kubernetes-uksouth-30786-843
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  0% (0) / 5%
Min replicas:                                          1
Max replicas:                                          10
Deployment pods:                                       1 current / 1 desired
Conditions:
  Type            Status  Reason            Message
  ----            ------  ------            -------
  AbleToScale     True    ReadyForNewScale  recommended size matches current size
  ScalingActive   True    ValidMetricFound  the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
  ScalingLimited  True    TooFewReplicas    the desired replica count is increasing faster than the maximum scale rate
Events:
  Type     Reason                        Age                From                       Message
  ----     ------                        ----               ----                       -------
  Warning  FailedGetResourceMetric       38m (x3 over 39m)  horizontal-pod-autoscaler  unable to get metrics for resource cpu: no metrics returned from resource metrics API
  Warning  FailedComputeMetricsReplicas  38m (x3 over 39m)  horizontal-pod-autoscaler  failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

Confirmed that upgrading clusters does not deliver updated component versions:
$ kubectl get nodes
NAME                        STATUS   ROLES    AGE   VERSION
k8s-agentpool1-16834821-1   Ready    agent    55m   v1.12.0
k8s-master-16834821-0       Ready    master   77m   v1.12.0
$ kubectl describe pod metrics-server-789c47657d-jc6lh -n kube-system | grep 'Image\:'
    Image:         k8s.gcr.io/metrics-server-amd64:v0.2.1
$ kubectl describe pod heapster-855c6c8b5b-6jsn7 -n kube-system | grep 'Image\:'
    Image:         k8s.gcr.io/heapster-amd64:v1.5.3
    Image:         k8s.gcr.io/addon-resizer:1.8.1
$ kubectl describe pod kube-addon-manager-k8s-master-16834821-0 -n kube-system | grep 'Image\:'
    Image:          k8s.gcr.io/kube-addon-manager-amd64:v8.7

Also, cluster-autoscaler is not present in /etc/kubernetes/addons/.

Actually k8s.gcr.io/kube-addon-manager-amd64:v8.7 is correct post-upgrade. So it looks like only addons-related components failed to be updated.

/lgtm

1.12 upgrade and scale tests passed

/lgtm


Diagnostics message misplaced

Problem

The error which is thrown in the assertion, is showing beside the [Fact] attribute.
Should be on the assertion line, where the exception occurred.
versions
neovim - 0.9.1
dotnet - 7.0.107
Thank you for your great work 🙏

Hi @veyxov - Thanks for opening an issue. I'm afraid this isn't something that the neotest-dotnet adapter can control, and it's rather a feature of neotest (core).
The way neotest determines the position in the file that the test is located, is by taking the beginning of the treesitter range where the query matches the test definition. In this case, it's where the Fact attribute starts. I can't modify where this position is, and if I did it in the adapter, it would break how neotest (core) works.
I know in other IDEs, that the position where the error was thrown is given in the output, and it may be possible to parse this position from the stack trace, find the point in the test itself that originated the call further up in the stack, fetch the error output for that particular test via neotest, create a new diagnostic message, and insert it into the diagnostic error list in neovim. However, to do that as part of the neotest adapter isn't something on my roadmap for the adapter itself. That said, I would happily consider a PR for that feature.


Blink cursor after leap

How can I set up my cursor to blink after leaping for a few seconds?

I'm interested in this too, if someone has a complete solution, feel free to share the snippet.
Currently, I think you need to do something like:

Register the cursor position on LeapEnter.
Create a function (blink) that sets guicursor to achieve blinking effect, and then restores it after a while (defer_fn(restore_guicursor, timeout)...)
Trigger blink on LeapSelectPre and on LeapLeave if the cursor has moved. (On LeapSelectPre, you need to register the position again after an autojump.)


Enable manual deploy of wiki

Added workflow_dispatch event trigger to allow users to manually run the workflow to deploy the changes.
More info: https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#workflow_dispatch

@U8NWXD PTAL


zsh: command not found: nvm ?

Operating system and version:

Debian GNU/Linux 10 (buster)

nvm debug output:

nvm ls output:

How did you install nvm?

Readme -> curl . . .

What steps did you perform?
What happened?
What did you expect to happen?
Is there anything in any of your profile files that modifies the PATH?

If you are having installation issues, or getting "N/A", what does curl -I --compressed -v https://nodejs.org/dist/ print out?

Any command I execute after installation, the output on the screen is always : command not found: nvm

It's likely the installation script isn't properly detecting your profile file.
Do you have a ~/.zprofile or ~/.zshrc file?

I checked, but none of the files exist, what should I do ?

It's likely the installation script isn't properly detecting your profile file.
Do you have a ~/.zprofile or ~/.zshrc file?

I checked, but none of the files exist, what should I do ?

touch ~/.zprofile, and then add the lines indicated here to it. Then relaunch your shell.

I added the following line to the .zshrc file plugins=(... nvm)

Great, did that fix it?

Great, did that fix it?

Yes.


Add better type inference

This PR only changes some types, but I think that it will have a huge impact in the developer experience, as the types will be inferred based in the StateCreator type.
One caveat with this PR is that the generic of create has been changed from TState extends State to TStateCreator extends StateCreator<State> to allow better inference (see this line). Another "generic argument" was added to, but that one should not be touched by the end user. In this last case, the second "generic argument" could be avoided with a little code duplication. What do you think?
This should be treated as a breaking change, as the generic constraint have been changed.

Some screenshots of VSCode showing the new type inference

Now I realized that the set type is not "safe", I will close and reopen if I manage to solve this problem.


Multiple security issues

As subprocess is used to encrypt / decrypt the safe with GPG, one can easily intercept the password, as it is passed as a flag to that subprocess (dumb way would be to call ps aux | grep gpg thousands of times a second and hope it will catch the subprocess call). Fix for this: Do not use --passphrase flag for gpg.

pick suggest reading the password from "PICK_TOKEN" env var: https://github.com/bndw/pick/blob/732ecce6593115df342c154adf74b3924be3b375/pick#L130
This should clearly not be done.

Why use gpg anyway? Why not symmetrically encrypt using openssl / derivate, which even offers nice ciphers like aes-gcm / chacha20 with AEAD to provide integrity? Also pick should probably not save the master password in memory. This then transfers all sensitive tasks (i.e. encryption, decryption & asking for "master" password) to external libraries.
Bash:
SSLLIB="openssl"
if type libressl > /dev/null 2>&1; then
	SSLLIB="libressl"
else
	# openssl doesn't support chacha \o/
	exit 1
fi

# ChaCha

# $ chacha-enc file.zip
chacha-enc() {
	$SSLLIB enc -chacha -e -in "$1" -out "$1.chacha"
}

# $ chacha-dec file.zip.chacha
chacha-dec() {
	$SSLLIB enc -chacha -d -in "$1" -out "${1%.*}"
}

Hi @leonklingele, thanks for taking the time to review the code and offer some feedback. I've addressed your points below-

reading the password from "PICK_TOKEN" env var

Agreed. This was something I added early on when I still wasn't sure where this project was going. At this point I agree, it should be removed.

pick should probably not save the master password in memory. This then transfers all sensitive tasks (i.e. encryption, decryption & asking for "master" password) to external libraries.

As subprocess is used to encrypt / decrypt the safe with GPG, one can easily intercept the password...

Why use gpg anyway?

I used GPG initially because it was what I'd previously been using in bash. So from a design standpoint, that was my starting point. However, this project has evolved and it maybe time to reevaluate.
Regarding switching to openssl/etc and taking advantage of new ciphers like chacha20- absolutely. I'd be more than willing to accept a pull request with more robust encryption.
Do you have any recommendations on how to we could talk to openssl, leaving the password entering tasks to the external library (not passing it to the subprocess), while taking a minimal hit on dependencies?

One could use https://github.com/dlitz/pycrypto or https://github.com/Legrandin/pycryptodome
The former seems to be a dead project, no commits made since 2014.
The latter is too new to use in production imo.
As the code base for this project is rather small, one could work on a golang implementation.
Go even supports AES-GCM right out of the box.
I could work on a PR if wanted :)

I wasn't able to find a suitable library for newer ciphers, either. Given this constraint, I don't see decent way to avoid passing the --passphrase to gpg.
To that point, the issue I see with collecting and passing the passphrase is a malicious process could intercept the passphrase from the gpg invocation (`ps aux | grep gpg)
Given the current Python implementation, I don't see a way around this.
Regarding a rewrite in Golang-
I actually did a rewrite in Go about a week ago. I was not sure if I wanted to replace the Python code, but I've been wanting to do something in Go and this project seemed like a good fit. My apprehension to replacing Python with a compiled language is the loss of simplicity- but providing precompiled binary "releases" may be a decent option.
I'll create a branch and drop in the Go project after a quick once over. It's already feature complete and has tests around the "safe" module. I'd love some help with the crypto- is that something you'd be interested in @leonklingele? Right now it's using crypto/openpgp to symmetrically encrypt.

Sure, just push it. :)

Sure, just push it. :)

@leonklingele https://github.com/bndw/pick/tree/f/go
It's currently using GB for builds- I'm definitely open to ditching that and using the standard Go build tools (this was my first project in Go)

I will take a look at it. Might take some time though :unamused:
Here are some things I noticed after a quick first look at source:

For additional security do not use openpgp package. Use crypto/aes for AES-GCM support instead. Example: https://play.golang.org/p/5ziXIL_avB
Don't use user's master password directly to encrypt the safe. Derive a key from that password using pbkdf2.Key with preferably more than 10k rounds and at least 8 bytes long salt read with crypto/rand.
I recommend to create a folder in ~ again for the safe (~/.pick/). Makes it much easier to implement features like "backup". The salt from previous step should be stored in a config file in that folder. Things I would store in the config file: Number of PBKDF iterations, salt, maybe cipher used (aes-gcm). This makes it easier to later (automatically) upgrade the safe once we increase – say the iterations – to 20k, without destroying the safe.

Will update this list as a todo for me / us.


create-or-reset-admin should reset admin role to 'admin'

Pull Request Test Coverage Report for Build 78d6f15b83e63cd1c2af44f1b0ce47b1df547755-PR-922

2 of 2 (100.0%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.04%) to 67.437%

Totals

Change from base Build 42923c3ebe8d27426e0955726deb1c6f9c0172f4:
0.04%

Covered Lines:
1050

Relevant Lines:
1557

💛  - Coveralls


getItem should return null for not-existing keys

See https://developer.mozilla.org/en-US/docs/Web/API/Storage/getItem

Good catch!
And... ehh.. sorry I missed this PR for soooo looong!!
Thank you so much for contributing!


Inconsistent behaviour in "closeOnSelect: false" with mouse and keyboard

jquery 3.4.1
select2 4.0.9
Trying to use "closeOnSelect: false" with multiselect select2 box
Problem:
Behaviour of flag is not consistent with keyboard and mouse access.
With mouse:

open select dropdown
select value from dropdown list with mouse
click selected value from dropdown with mouse
Value is deselected from list

With keyboard:

open select dropdown
select value from dropdown list with ENTER
try to deselect value from dropdown with ENTER
=> dropdown closes and values is not delected
=> Thus it seems that it's not possible to do similar "select/deselect" logic in dropdown as with mouse

Can be seen in https://jsfiddle.net/kikonen/pc50daj2/8/

The keyboard command used to select/deselect options is CTRL + SPACE, not ENTER.
For more information on why this was done, see https://github.com/select2/select2/commit/017c20109471fa5b835603faf5dc37f7c2c2ea45.

CTRL + SPACE appears to somewhat unfortunate default since it's shortcut key used by my web browser for searching open tabs.


updated quick_overview docs

First comment would be to add a description

Hello @kristiankyvik,
I'm very sorry, but I cannot approve the PR in its current state. I've thought about this long & hard (hard in like "thinking hard") and following the OSS ethos, this PR is simply not ready to be merged yet.
This must be disappointing to hear for you, but you could also see this as a chance to further grow and improve. What's really important here in my opinion is to take this negative message and turn it into a positive one (this could also help: https://www.amazon.com/Positively-Negative-turn-Messages-Positive/dp/B099BYQNXL#:~:text=Positively Negative%3A How to Turn Negative Message into Positive Ones,you need to be positive.)
Open sourcely,
Patrick


fix: remove useMediaQuery which breaks ssr

related: #137 #171
closes: #171

can we delete this branch now?


Create Domain Model

Create diagram showing how the components of the project work together.
Include the Domain Model in the README.md file.

Useful resources on Domain Models
Brief Introduction to Domain Modeling
Domain Modeling
Domain Driven Architecture Diagram

Completed in prep #4. See: domain model


Broken @material-ui dependency?

Hi, I am trying to include your Component in a React project of mine.
Running "@material-ui/core": "^4.3.1",
Getting the following errors:
./node_modules/react-image-annotate/Theme/index.js
Module not found: Can't resolve '@material-ui/core/styles/MuiThemeProvider' in '.../node_modules/react-image-annotate/Theme'

When I go back to a Version 3.9.3 of @material-ui the MuiThemeProvider is found.
But the @material-ui/lab/Slider dependency from material-survey will be lost.
Can't resolve '@material-ui/lab/Slider' in '.../node_modules/material-survey/components/RatingQuestion'
Or it may be related to this
Any suggestions?

Sorry for the trouble, does the import come from the auto-import feature of your IDE?
I would encourage the following:
-import { makeStyles } from "@material-ui/core/styles"
-import createMuiTheme from "@material-ui/core/styles/createMuiTheme"
-import MuiThemeProvider from "@material-ui/core/styles/MuiThemeProvider"
+import { makeStyles, createMuiTheme, ThemeProvider as MuiThemeProvider } from "@material-ui/core/styles"

What do you mean by

does the import come from the auto-import feature of your IDE?

I used the npm package and tried to use your Annotator Component
import {Annotator} from "react-image-annotate"
I guess the fix for your Theme would work.

@ArcticTee Sorry, it was a question for the author of the package.

Sorry, confused you with the author or contributors of this package😂

Your best workaround is to freeze @material-ui/core to v4.5.0.

I solved it by using
    "@material-ui/core": "4.4.0",
    "@material-ui/lab": "4.0.0-alpha.22",

in my package.json

@ArcticTee thanks for reporting, assigning myself to fix this issue.

Notice that Material-UI v4.5.2 will soon come out with a warning instead of an exception for the import of this private module.

fixed in 0.1.28


[glib] update to 2.70.1

@BillyONeal  what happend to x64-osx? Never saw that error.
Attempting to fetch 134 packages from HTTP servers.
Exception: /Users/vagrant/Data/work/1/s/scripts/azure-pipelines/test-modified-ports.ps1:199
Line |
 199 |          throw "vcpkg ci failed"
     |          ~~~~~~~~~~~~~~~~~~~~~~~
     | vcpkg ci failed

##[error]PowerShell exited with code '1'.
##[error]PowerShell wrote one or more lines to the standard error stream.
##[error]Exception: /Users/vagrant/Data/work/1/s/scripts/azure-pipelines/test-modified-ports.ps1:199
Line |
 199 |          throw "vcpkg ci failed"
     |          ~~~~~~~~~~~~~~~~~~~~~~~
     | vcpkg ci failed

This osx error already appeared randomly in the last weeks.

@Neumann-A ,Thanks for your pr, I tested features locally on x64-linux triplet but failed :
CMake Error at scripts/cmake/vcpkg_execute_required_process.cmake:127 (message):
    Command failed: /usr/bin/python3 /home/test1/vcpkg/downloads/tools/meson/meson-aeda7f249c4a5dbbecc52e44f382246a2377b5b0/meson.py -Dinstalled_tests=false -Dselinux=enabled -Dlibmount=enabled -Dtests=false -Dxattr=false -Dlibelf=disabled --buildtype plain --backend ninja --wrap-mode nodownload --native /home/test1/vcpkg/buildtrees/glib/meson-native-x64-linux.log --default-library static --libdir lib --native /home/test1/vcpkg/buildtrees/glib/meson-native-x64-linux-debug.log -Ddebug=true --prefix /home/test1/vcpkg/packages/glib_x64-linux/debug --includedir ../include -Dcmake_prefix_path=['/home/test1/vcpkg/installed/x64-linux/debug','/home/test1/vcpkg/installed/x64-linux'] /home/test1/vcpkg/buildtrees/glib/src/glib-2-3e22263b95.clean
    Working Directory: /home/vzhli17/test1/vcpkg/buildtrees/glib/x64-linux-dbg
    Error code: 1
    See logs for more information:
      /home/test1/vcpkg/buildtrees/glib/config-x64-linux-dbg-out.log

Call Stack (most recent call first):
  scripts/cmake/vcpkg_configure_meson.cmake:512 (vcpkg_execute_required_process)
  ports/glib/portfile.cmake:46 (vcpkg_configure_meson)
  scripts/ports.cmake:142 (include)

/home/test1/vcpkg/buildtrees/glib/x64-linux-dbg:
../src/glib-2-3e22263b95.clean/meson.build:2077:2: ERROR: Dependency "libselinux" not found, tried pkgconfig and cmake

A full log can be found at /home/test1/vcpkg/buildtrees/glib/x64-linux-dbg/meson-logs/meson-log.txt

/home/test1/vcpkg/buildtrees/glib/x64-linux-dbg/meson-logs/meson-log.txt:
Compiler stderr:
 /home/test1/vcpkg/buildtrees/glib/x64-linux-dbg/meson-private/tmprk8e3u5r/testfile.c:4:12: error: #error "Header 'mach/mach_time.h' could not be found"
           #error "Header 'mach/mach_time.h' could not be found"
            ^~~~~

Has header "mach/mach_time.h" : NO
Running compile:
Working directory:  /home/test1/vcpkg/buildtrees/glib/x64-linux-dbg/meson-private/tmppavpsk6g
Command line:  /usr/bin/cc -I/home/test1/vcpkg/installed/x64-linux/include /home/test1/vcpkg/buildtrees/glib/x64-linux-dbg/meson-private/tmppavpsk6g/testfile.c -E -P -fPIC -g -D_FILE_OFFSET_BITS=64 -P -O0 -std=gnu99

@JonLiu1993: I assume you are missing a system dependency if you want to enable feature selinux (ERROR: Dependency "libselinux" not found, tried pkgconfig)
mach/mach_time.h is a osx header according to a quick google search so it is no wonder that it is missing on linux.

@JonLiu1993: I assume you are missing a system dependency if you want to enable feature selinux (ERROR: Dependency "libselinux" not found, tried pkgconfig)
mach/mach_time.h is a osx header according to a quick google search so it is no wonder that it is missing on linux.

Have you tested all features locally?

Have you tested all features locally?

Have you tested the current baseline with the same features?
No I cannot test everything locally, especially not osx.
I can tell you that feature selinux works if the required system packages are installed

These mean that the vcpkg binary returned a nonzero exit code for some reason:
    & "./vcpkg$executableExtension" ci $Triplet --x-xunit=$xmlFile --exclude=$skipList --failure-logs=$failureLogs @hostArgs @commonArgs @cachingArgs @parentHashes

    $failureLogsEmpty = (-Not (Test-Path $failureLogs) -Or ((Get-ChildItem $failureLogs).count -eq 0))
    Write-Host "##vso[task.setvariable variable=FAILURE_LOGS_EMPTY]$failureLogsEmpty"

    if ($LASTEXITCODE -ne 0)
    {
        throw "vcpkg ci failed"
    }

This osx error already appeared randomly in the last weeks.

Hmmm I have not seen yet. Is it always the same build node or something else?

Thanks for the update!

Hmmm I have not seen yet. Is it always the same build node or something else?

@BillyONeal  Where do I see the node? Yesterday:
https://dev.azure.com/vcpkg/public/_build/results?buildId=63849&view=logs&j=7b75bd19-17d3-53d4-00fd-23f1a49a8ba4&t=a23dba4e-3a62-59dc-a73c-a7222676d7a4
https://dev.azure.com/vcpkg/public/_build/results?buildId=63847&view=logs&j=7b75bd19-17d3-53d4-00fd-23f1a49a8ba4&t=a23dba4e-3a62-59dc-a73c-a7222676d7a4

@BillyONeal Where do I see the node?

If you click on 'x64-osx' on those links it shows the node in the output window:

Both those examples ran on vcpkg-eg-mac-06 which suggests that that one is unhealthy. I disabled it for now

I vagrant destroy && vagrant up'd -06 so it should be back.


Don't try to use nonexistent global attribute id constants on Darwin.

We have some global attributes that are marked provisional.  New clusters being added would try to generate a constant for the global attribute inside the cluster, until we had a chance to set it provisional, but set its value to the nonexistent global constant, which would fail to compile.
This just skips generating the relevant cluster constants and the methods that use them.

Fast-tracking platform-specific change with platform owner review.


Restore Wallet

Balance is zero in desktop wallet which I restored for the first time whereas the correct balance should be positive and as reflected in my mobile wallet.

On chain or what?

Thanks for your response.
One was a swept force close of 39,152 sats on chain and one lightning tx of 100,000 sats.
Since spent buy have a balance of approx 84,000.

Wait for it to sync and also double check the seed words. Other than that, you'll need to provide logs on both ends.

Ok.  Seedwords checked but will do so again.
What do you mean logs both ends?
I think I know where to find the logs option.
Thanks Tony.
Will come back in the coming days.

It's in the emergency toolkit in the settings page. By both ends, I mean the device that has the correct balances and the browser where it's showing up blank.

Unable to upload either of the logs her ie destroy or mobile. Mobile image attached.
I'm using an iPhone 13 Pro Max which I hate! 😬
Back to android I go...

It's in the emergency toolkit in the settings page. By both ends, I mean the device that has the correct balances and the browser where it's showing up blank.

What OS does Mutiny need for use on a MacBook Pro?
Just realised that wallet features are missing having watched Ben'stutorial...For example, I do not have the option to add install app.

Ok, I've got it to work on my iMac.  Must be the OS as it's an older laptop.
Thanks for the assistance, Tony.
#BTC for the win. 🔥🧡


Multi node slow training

Hi, i'm using multi node training from your example in scripts/train/
n_gpus=16
Starting training...
******************************
Config:
enabled_algorithms/GradientClipping: true
node_name: unknown because NODENAME environment variable not set
num_gpus_per_node: 8

composer --world_size 16 --node_rank 0 --master_addr <MY_IP> --master_port <MY_PORT> train.py yamls/pretrain/mpt-1b-rulm-cluster.yaml train_loader.dataset.split=train eval_loader.dataset.split=val on 0 node
composer --world_size 16 --node_rank 1 --master_addr <MY_IP> --master_port <MY_PORT> train.py yamls/pretrain/mpt-1b-rulm-cluster.yaml train_loader.dataset.split=train eval_loader.dataset.split=val on 1 node
Training starting but it's going really slow, like in 8 times slower than on single node with 8 gpu
Train batch size now 2x bigger than single node, and micro batch sizes are same, so I guess everything fine
all 16 gpus are working in 100% with nvidia-smi
do you have same issues maybe?

problem was no infiniband, with ib everything is ok


MEASUREMENTS fail for all slot id except for 0 (GROUP 7 TC)

In GROUP 7 TC (spdm_responder_test_7_measurements.c),
only one certificate chain with slot 0 is set on setup phase.
This causes MEASUREMENTS fail for all slot id except for 0.

@chanss-park, all slots are using same key. I am not sure why it fails.
Are you observing the failure? If so, please let me know how to reproduce.
or just guessing that it will failure?

(Following the comment on #121)
Is it also intended to use only slot 0 in TC 6.2 ? (VCA then CHALLENGE without GET_DIGEST and GET_CERTIFICATE)
In the TC above,
"spdm_context->connection_info.peer_used_cert_chain_slot_id" is always 0 at line 568
as the first argument of libspdm_verify_challenge_auth_signature().
Please confirm.


Add 'PreserveTitleAndBody' configuration

When enabled the pull request title and body will not be overwritten
when a pull request is updated.
This also has the side effect of not outputing the stack into in the
pull request body. Ammending the current latest message is beyond the
scope of this change.
fixes: #236

Stack:

#245
#246
#244
#243
#242 ⬅
#239

⚠ Part of a stack created by spr. Do not merge manually using the UI - doing so may have unexpected results.

✓ Commit merged in pull request #246


CI Solution

Need to look into a good Continuous Integration solution. Possibly Travis CI?

netlify is another CI solution to maybe research.

We should run the tools discussed in the issues below on each pull request via our CI environment.
vuln checker: https://github.com/DigitalRiver/react-atlas/issues/264
license checker: https://github.com/DigitalRiver/react-atlas/issues/263
flow type checking: https://github.com/DigitalRiver/react-atlas/issues/261
prop comment checking: https://github.com/DigitalRiver/react-atlas/issues/258
eslint
prettier
accessibility checker

Fixed by https://github.com/DigitalRiver/react-atlas/pull/367


HDDS-8906. Avoid stream when getting in-service healthy nodes

What changes were proposed in this pull request?
Try to avoid the stream created for NodeStateManager#getNodes(IN_SERVICE, HEALTHY) (invoked by totalHealthyVolumeCount).
https://issues.apache.org/jira/browse/HDDS-8906
How was this patch tested?
Refactoring, covered by existing tests.
CI:
https://github.com/adoroszlai/hadoop-ozone/actions/runs/5346100498

Thanks @adoroszlai for working on this, LGTM

Thanks @Galsza, @sodonnel for the review.


about the newsfeed

my XML file is use encoding gb2312 , so it display garbled
how to fix it
thanks

Read the newsfeed module documentation. You can specify the encoding.


Wildcard and OR operator support for ?contains parameter

Example:
https://jokeapi.dev/joke/Any?contains=foo*|*baz

Jokes:
foo (match)
foobar (match)
foo bar baz (match)
bar foo (no match)
bar baz (match)
bar baz bar (no match)

Important:
Make sure the users can't ReDoS the API

Still not done, see #331


4 read a customer resource

Made the following changes:
Added read customer endpoint in 'routes.py'
Added test case for get customer endpoint

Changes look good to me!


It s possible to use uptrace service in k8s like Deployment?

It s possible to use Uptrace service in k8s like Deployment? Not as StatefulSet as present in current Helm Chart.
For me its better for configuration and scalling.

Since all Uptrace nodes are equal and don't have state, we probably should use Deployment for Uptrace.
@mk-raven I wonder if it is the only change you needed to make. Could we briefly chat at Telegram what was your experience with K8s and Uptrace chart?


Adds sort to every config option that represents a range of values

Modified/simplified check for agentMaxAge
Added agentSelfishnessFactor verification to check if it is disabled with debugMode statement.
Modified logic of agentDecisionModelTribalFactor to check if it is disabled with debugMode statement.
Added a check to make sure the number of unique ethical decision models is less than or equal to the total number of colors in the GUI.

I was not sure if I should have added the debugMode statements to the selfishnessFactor and tribalFactor.

Good work!
@colinhanrahan please test for correctness.
I will tidy up some debug message wording then merge once I get the green light.

Numerical correctness looks good, but I'll do a little more testing.
If we're committing to proper config verification—and this would be a separate feature—there are plenty of other config settings that cannot take negative values. We should either disclose this in the README or verify them beforehand, or both.

@amuller26 Ready to merge?

@nkremerh yes.

Numerical correctness looks good, but I'll do a little more testing.
If we're committing to proper config verification—and this would be a separate feature—there are plenty of other config settings that cannot take negative values. We should either disclose this in the README or verify them beforehand, or both.

New item: would you like to add this more thorough checking?

@nkremerh I created the Kanban card.


Can you update fluentd-kubernetes-daemonset:v1.14-debian-opensearch-1 to include latest opensearch plugin?

Can you update the fluentd-kubernetes-daemonset:v1.14-debian-opensearch-1 image to include latest OpenSearch plugin?
Thanks!
References:

https://github.com/fluent/fluent-plugin-opensearch/releases/tag/v1.0.7
https://github.com/fluent/fluent-plugin-opensearch/pull/52

Updated to fluent-plugin-opensearch 1.0.7


issue-2: adjust "now" to avoid floating point issues; fix a typo

fix a problem where converting the epoch (date/time and microseconds)
borders on the limits of a 64-bit double-precision floating point number;
the resulting value is adjusted relative to January 1, 2017.
correct a typo.

:+1::+1::+1:
Thanks!


Update reset-the-audio-conferencing-pin-in-teams.md

https://github.com/MicrosoftDocs/OfficeDocs-SkypeForBusiness/issues/5852
Changed wording to gender neutral.

@JohanFreelancer9 Copy editing is needed for this PR. Thanks.

@tonysmit PR has been copyedited and is ready for final review, could you please check and merge? Thanks!


Bugfix/set isrelease for installer releases

This merge request has code that sets -PisRelease, along with -Pfinal or -Prc, for the .base-release-installer Gitlab job.
By setting -PisRelease, build.gradle files can leverage the BSP's isRelease variable to conditionally publish to specific custom snapshot or release repos. For example the below would be in a build.gradle file.
buildSupport {
    repoManagement {
        custom name: 'example-Release', url: 'https://example.com:443/artifactory/example-releases', credentialName: 'EXAMPLE', publish: isRelease
        custom name: 'example-Snapshot', url: 'https://example.com:443/artifactory/example-snapshots', credentialName: 'EXAMPLE', publish: !isRelease
   }
}

Note, this merge request heavily borrows from the merge request at https://github.com/chesapeaketechnology/gitlab-templates/pull/74, thanks @AndrewTrow! I went ahead and closed https://github.com/chesapeaketechnology/gitlab-templates/pull/74 since it has the same purpose as this merge request.

The  -Pfinal and -Prc is for @JRhoades22 project.
The thought behind this merge request was that by adding  -Pfinal and Prc to the .base-release-installer: Gitlab job, that @JRhoades22 team could choose to release a final installer or release candidate installer via the Gitlab web UI.
However thinking about it more, I could just trash this merge request and instead have @JRhoades22 team set their release candidate installer version via their installer's projectVersion.txt. Let me talk to @JRhoades22 and see what works best for them.

That is the intended design.  If you specify -Prelease=<anything>, then the installer is build using the version in the projectVersion.txt file.  The intent being that you modify that file according to the type of release you want (RC or FINAL).  If you look at the TAKX CI build, when we do a release build, it updates the raptorVersion.txt file in takx-installer accordingly, and kicks off a release build.

Makes sense, I decided to not modify the installer pipeline Gitlab template and go another route.


Updated vscode dependency version. Fixes #157.

Fixed #157 by updating VSCode dependency version. The relevant API was added in November 2018, in part for this: https://github.com/microsoft/vscode/issues/62483

Thanks :)


Create desktop and UI version

PRB: command line install and operation.
Create a desktop application which can use the same sources as a web server version.
Python windowed - not portable, not installable, not scalable

 AT MVC Data


3D Pose Matrix

Hi there! Is it possible to get / generate somehow a realtime 3D pose matrix from the 68 face landmarks? Thanks!

It is possible in general yeah, but face-api.js currently doesn't provide a model to do so.


Experiment with MaybeCompareFn

Currently an array needs to statically decide if it supports a compute function, prior to seeing the RHS or other arguments.
This change allows the array's compute function to see the RHS, and then decide it doesn't support the operation, enabling easier fallback.
This is only implemented for compare.
It has also highlighted the need for storage DType information to be available to an ExtScalar. Either by adding it to ExtDType, or adding it directly into that scalar variant. @a10y keen for your thoughts

I can start building on top of this PR, I think the general approach is good and we can start focusing on combinations we can specialize

@gatesn IIRC we had talked a couple months back about adding the storage DType to ExtensionDType. It makes sense to me that an ExtensionDType is just a logical type with some associated metadata. This still preserves flexibility of the physical encoding. I'd be a fan of adding the storage type into the ExtensionDType.

The problem is, that means we would have to use e.g. int64 as the storage type for all date times, regardless of the unit.
In our Python prototype with had an IntWidthArray that allowed you to "reinterpret cast" a narrower int type into a wider one, e.g. you could zero-copy from Arrow an int32 array but pretend that it's int64 in Vortex.
It would be kind of weird to canonicalize back to Arrow by somehow introspecting this though...


Updated node-pre-gyp: Lots of npm audit fixes

Would much appreciate if you could merge this as we're using js-gardener and it fails on this dependency due to npm audit issues.

Thanks! There were issues in the past getting newer node-pre-gyp versions to work, so I’ll release this as a semver-major change today.


Compromised password; cannot change or close acccount

My Google account has notified me that arxiv-sanity had a data-breach and that my password is no longer secure. Since I have not used the site in years, I decided I should just delete my account rather than changing my password. It seems that I cannot do either. How do I go about removing my account?
Thanks!

agree

@karpathy any update on this?

3 years later still getting notifications that this password is compromised :(


addition of ruler? or numbers on top and left edge of pattern

typically each 10 (10, 20, 30, etc.) is labeled across the top and bottom of the pattern, this could also be the thing that we make show hovering over the pattern when you zoom in and out @travisdoesmath

numbers added, but font size is an issue