Feature request: Influence build order (build large ports and their dependencies early)

We have noticed that very often, after building 1500-or-so packages, the remaining few are llvm, gcc, openjdk*, etc., and stuff that depends on them. It would be useful to be able to influence the order in which ports are built, much like we can give ports priority boosts.
E.g. we'd like to be able to have poudriere start building the openjdk and complier ports as early as possible, which means any dependencies would get a corresponding bump too. We have 48 CPU cores on the build system so building 1500+ smaller ports is reasonably fast, but the larger ports are often single-threaded builds and take a long time - so doing them in parallel with the smaller ones would save a great deal of time.

Example output towards end of build run:
[103amd64-default] [2016-05-12_15h03m40s] [parallel_build:] Queued: 1755 Built: 1698 Failed: 2    Skipped: 0    Ignored: 1    Tobuild: 54    Time: 01:29:22         [01]: devel/libclc                     build           (00:07:37)         [02]: lang/gcc                         package         (00:38:40)         [03]: java/openjdk6                    build           (00:27:22)         [05]: java/openjdk7                    build           (00:41:23)         [06]: devel/boost-libs                 package         (00:36:35)         [09]: graphics/opencv                  build           (00:23:35)         [15]: databases/xtrabackup             build           (00:24:59)
During the one-and-a-half hour so far, almost 1700 ports have been built. Few of those are required for the current build jobs, so the ones listed above should/could have been started earlier, saving time.

Hi Eirik,
We have something that will help a little.  It is called PRIORITY_BOOST and can be set in your poudriere.conf.  Unfortunately, it does not boost the priority of the depends.. Just the package you specify.  So it needs some work.
Regards,
Brad Davis

Hi,
thanks - I'm doing just that in the hope that it might help a little, but in reality it doesn't.
We'd be interested in sponsoring the work needed to improve this functionality. What do you think it would take, and how to go about it?
I understand (from previous experience...don't ask) that this can be a bit voodoo, but it can hardly be as bad as back in the old 4.x/5.x days..
/Eirik

Agreed that PRIORITY_BOOST isn't enough since it is not recursive.  Making it recursive would be too onerous on the startup time I think.  Right now we use a modified tsort(1) on the entire dependency tree to determine how "deep" a package is in the tree and then order the queue based on the depth.  The current PRIORITY_BOOST increases that depth to 99 which basically is just making that port build before any other ports of the same depth.  We do all of this when building the queue.  The queue is static; it is not just a list that is ordered at the point of "want should I build next?".
I'm happy with the current ordering overall but would love to see it work more as expected.  The problem is that to boost something like lang/gcc we really need to also boost all of its dependencies too. I believe that if for every boosted thing we also boost its dependencies then the goal will be achieved.  I think we just need to see a prototype of this to see if it is worth the extra overhead.

I recently setup 3 poudriere build servers and also noticed ports that take a long time to build and have dependencies take a long time to build. So at the end I have 3 out of 24 builders doing work and everything left to build depends on these.
Ideally you'd like to to have all builders busy until you have fewer ports to build than builders left.
After thinking about the problem for a few weeks the best idea I had was to build the ports with the longest dependency chains first. I don't believe there is any reason to start building a port without dependents until all ports with dependents are being or have been built. I think this will improve the average number of busy builders and that will shorten the total build time.
What is the current algorithm?

What is the current algorithm?

Duh, the post before mine says "we use a modified tsort(1)"
I found the source for it; is there an easy way to generate tsort input from a ports-list?

I just wrote ptsort in hopes that it might help address this issue. With ptsort instead of tsort, you would:

Start with the same input (list of pairs of package names) that you feed tsort today.
I'm not sure whether you already do this today, but all packages should probably also be listed as successors to pkg.
For each boosted package, append one line with the name of the package followed by a large number (basically, anything larger than the total number of packages in the list).
Feed all of that to ptsort.

In the output from ptsort, every package you boosted and all of their predecessors (build dependencies) will be listed ahead of everything else.
You can also use ptsort to detect cycles, but the semantics and output aren't precisely identical to tsort's, so deadlock_detected() will need a few tweaks.  Although I don't understand why you only check for cycles after a deadlock has occurred, rather than before starting the build—or did I miss something?
Anyway, please give ptsort a go, let me know of any changes you require, and once you're happy with it, I'll roll a release and create a port.

@dag-erling thanks!

As for deadlock_detected, it is poorly named. It should be detect_deadlock.  It is called before the build starts as well as during it due to many historical deadlocks and hangs in the queue handling.

This definitely should help, but there is another part.
All big ports can be built with some concurrency, as opposed to just single-threaded. But enabling concurency within a single build poses the problem of overloading the build host.
To that end I propose the following trivial & likely good enough approach: instead of tracking the number of builds in progress, you can track concurrency. That is, a single-threaded build adds 1. But bigger packages can be built with, say, -j 4 and that would add 4.
Let's say n == 40. When one build finishes, n == 39. You can temporarily oversubscribe the machine by adding a -j 4 build. Now n == 43. You don't start any new builds until n reaches 39 or less.
The thing is, I'm playing with port builds on an 80-way box and get 20k packages in little under 7 hours or so, and then the rest is trailing single-threades for 20h.

I don't see how that could actually work. How many jobs do you give to the first port to build?

You start builds until the last added package bumps the counter to the limit or above.

The first build would raise the counter to the limit since we'd give it all available jobs.  Then there's nothing to do until that build is done.

I think you're thinking that there's a way to tell how many CPU a package build needs or uses, both of which are impossible or infeasible.

Most jobs would remain -j1.  Only "big" ports would get high -j.  This is using the same manual hints already provided.

I'm not suggesting any heuristics. Instead, build logs clearly show which ports take a long time to build single-theaded. So they can be annotated in their makefile.

@cemeyer  We already have something like that:

# List of packages that will always be allowed to use MAKE_JOBS
# regardless of ALLOW_MAKE_JOBS. This is useful for allowing ports
# which holdup the rest of the queue to build more quickly.
#ALLOW_MAKE_JOBS_PACKAGES="pkg ccache py*"

@bdrewery mjg pointed out a problem with the existing mechanism and a proposal to mitigate the problem.

Again, how do you know how many CPU a build is using? How do you "bump the counter"? A job may not even use more than 1 CPU for several minutes, and then it may use all that you gave it.  It may also only use 1 even though you gave it 99.  Then you're still wasting those job slots.

Because all of the information is explicitly annotated in Makefiles.  Don't give small ports 99 slots.  It's probably fine to oversubscribe cores by a factor of two or so.
There are packages we know take 10s of hours to build.  Single pkg improvements have the potential to reduce full builds by a long time here.

There are packages which take a long time to build single threaded, there are logs on official builders and other places. Realistically all of them should support high -j builds. An annotation can be added to relevant makefiles, e.g. PACKAGE_BUILD= BIG. Upon seeing such an annotation, poudriere will do the build with -j 4 (or higher/lower, depending on the machine) as described previously. Sometimes this will oversubscribe the machine a little bit, and sometimes it will be left slightly underutilized. Either way, the end result should be a net win in terms of total build time.
From time to time one will have to check for packages building very long to see if they also should be annotated. A big port is not going to shrink though.
Here are top offenders from few weeks ago:
./modula3-5.10.0_2.log:build time: 24:02:33
./gcc-arm-embedded-5.4.20160622.log:build time: 22:52:54
./chromium-51.0.2704.106_2.log:build time: 16:40:55
./libreoffice-5.0.6_2.log:build time: 13:02:12
./erlang-riak-16.b.02.log:build time: 12:21:09
./libreoffice4-4.3.7_5.log:build time: 11:53:22
./ghc-7.10.2.log:build time: 10:48:03
./webkit2-gtk3-2.8.5_5.log:build time: 10:23:54
./gcc7-devel-7.0.0.s20160724.log:build time: 09:57:07

We already support annotating which ports should build with jobs https://github.com/freebsd/poudriere/issues/387#issuecomment-252095545.  Annotating it in Makefiles is a separate problem.  This feature doesn't scale the amount of jobs given to the build, it just allows -jNCPU (really whatever the user has MAKE_JOBS_NUMBER set to in their make.conf).  I still think that it is not feasible to scale the jobs dynamically rather than just give a fixed number.  The feature could be improved to allow specifying a number of jobs to allow for specific packages.
The original proposal here doesn't seem to match where this discussion has gone.
Consider that if we built 1 package at a time and let each use -jNCPU, that the whole build would definitely take longer.  The builds are not 100% CPU bound. There is a lot of overhead such as fetching a package, extracting it, writing the build logs to disk, running QA checks, installing dependencies, staging the files to disk, creating a package from the stagedir, deleting the files, moving the package, running tests in some cases.  So clearly we cannot just make a bucket of CPU and risk building ports serially and letting each have the entire bucket of CPUs.
So back to the original proposal, how do we split up the known number of CPUs between normal and long packages?   Say we reserve 10% of CPU threads for these long builds and know in the queue all choices are not the long builds, do we just let the reserved threads be idle? That's not ideal at all.  It's realistic too as these long builds take several hours until they are eligible to build as their dependencies need to build first. Boosted dependencies, or late in the build,it could be possible that among all choices in the queue to build all are considered "long" or in need of a boost.  What then? That 24 hour modula port build really needs to get started ASAP, we cannot just start gcc7-devel with the 10% reserved CPUs and not let modula start until 5 hours from now.  Even if we split that 10% reserved between 2 builds, we're still starving other long builds from starting and progressing.  Don't forget that that modula build of 24 hours may not even support >1 CPU.
We also have dirty ports like java that will use all cpus and memory on the system even if we tell it to use N CPUs and N memory.
The only way I see this realistically working is with dynamic cpusets/groups with floating CPUs.  We always start all package builds with -jNCPU, but we limit it to 1 CPU by cpuset.   I have no idea what the impact of -j24 would be with only 1 cpu in the set. Once the build slows down and we actually have some threads to spare, then we add them into these long builds to help them out. This idea of mine would need a huge overall to our cpuset support though.  This idea seems similar to if we just ran all builds in their own individual VMs and let the hypervisor schedule the virtual CPUs as needed.
Anyway, this is a hard problem.  The simplest answer is to limit your PARALLEL_JOBS to something like 60% or 70% or your CPUs and then set MAKE_JOBS to 2 or 3.  Or something like that. Tuning to your specific system and package needs.

I don't see how what I wrote later differs from what I proposed in the first comment. It lacked the information how poudriere would know to increase -j for given package, which I later clarified - known big ports would be annotated in their makefiles.
Blindly adding -j 4 is of course suboptimal, but also should be significantly better than what is seen right now.
I completely disagree with idea like doing a high -j and then balancing the load with cpusets. Induced context switches will kill performance.
Instead, the hack proposed by me automagically balances the load with bound oversubscription, which should clear itself most of the time.
Devising an optimal solution is definitely a non-trivial task, hence the hack.

So ptsort doesn't do exactly what I need, but combining with my tsort -D modification with ptsort, I do get what we need.
What tsort -D does is report the "depth" of each node, so given the list:
ca_root_nss-3.28.1 ccache-3.3.3
ca_root_nss-3.28.1 perl5-5.24.1
ca_root_nss-3.28.1 pkg-1.9.4_1
ccache-3.3.3 pkg-1.9.4_1
freebsd-release-manifests-20161010 pkg-1.9.4_1
perl5-5.24.1 ccache-3.3.3
perl5-5.24.1 pkg-1.9.4_1
portlint-2.17.6 perl5-5.24.1
portlint-2.17.6 pkg-1.9.4_1
portmaster-3.17.10 ccache-3.3.3
portmaster-3.17.10 pkg-1.9.4_1
poudriere-3.1.14_2 ca_root_nss-3.28.1
poudriere-3.1.14_2 ccache-3.3.3
poudriere-3.1.14_2 freebsd-release-manifests-20161010
poudriere-3.1.14_2 pkg-1.9.4_1
poudriere-devel-<IP_ADDRESS>60624_2 ca_root_nss-3.28.1
poudriere-devel-<IP_ADDRESS>60624_2 ccache-3.3.3
poudriere-devel-<IP_ADDRESS>60624_2 freebsd-release-manifests-20161010
poudriere-devel-<IP_ADDRESS>60624_2 pkg-1.9.4_1

We get:
0 poudriere-devel-<IP_ADDRESS>60624_2
0 poudriere-3.1.14_2
0 portmaster-3.17.10
0 portlint-2.17.6
1 freebsd-release-manifests-20161010
1 ca_root_nss-3.28.1
2 perl5-5.24.1
3 ccache-3.3.3
4 pkg-1.9.4_1

This is used to build the queue directory, sorted by priority. So the queue directory has 4 directories in this case with each package inside of it along with each missing dependency in that directory:
0/poudriere-devel-<IP_ADDRESS>60624_2/pkg-1.9.4_1/
0/poudriere-devel-<IP_ADDRESS>60624_2/ca_root_nss-3.28.1/
0/poudriere-devel-<IP_ADDRESS>60624_2/ccache-3.3.3/
0/poudriere-3.1.14_2/pkg-1.9.4_1/
0/poudriere-3.1.14_2/ca_root_nss-3.28.1/
0/poudriere-3.1.14_2/ccache-3.3.3/
0/portmaster-3.17.10/pkg-1.9.4_1/
0/portlint-2.17.6/pkg-1.9.4_1/
1 freebsd-release-manifests-20161010/pkg-1.9.4_1/
1 ca_root_nss-3.28.1/pkg-1.9.4_1/
1 ca_root_nss-3.28.1/pkg-1.9.4_1/perl5-5.24.1
2 perl5-5.24.1/pkg-1.9.4_1/
2 perl5-5.24.1/ccache-3.3.3/
3 ccache-3.3.3/pkg-1.9.4_1/
4 pkg-1.9.4_1/

The queue lookup looks in reverse order using find 4 3 2 1 0 -type d -depth 1 -empty -print -quit, which in this case would find 4/pkg-1.9.4_1.  As packages are built, their directory is removed from children in the queue. So empty top-level directories are "ready to build".
Now PRIORITY_BOOST is implemented such that if it were to contain perl5*, it would move perl5 to the 99/ directory rather than the 2/ directory:
0/poudriere-devel-<IP_ADDRESS>60624_2/pkg-1.9.4_1/
0/poudriere-devel-<IP_ADDRESS>60624_2/ca_root_nss-3.28.1/
0/poudriere-devel-<IP_ADDRESS>60624_2/ccache-3.3.3/
0/poudriere-3.1.14_2/pkg-1.9.4_1/
0/poudriere-3.1.14_2/ca_root_nss-3.28.1/
0/poudriere-3.1.14_2/ccache-3.3.3/
0/portmaster-3.17.10/pkg-1.9.4_1/
0/portlint-2.17.6/pkg-1.9.4_1/
1 freebsd-release-manifests-20161010/pkg-1.9.4_1/
1 ca_root_nss-3.28.1/pkg-1.9.4_1/
1 ca_root_nss-3.28.1/pkg-1.9.4_1/perl5-5.24.1
3 ccache-3.3.3/pkg-1.9.4_1/
4 pkg-1.9.4_1/
99 perl5-5.24.1/pkg-1.9.4_1/
99 perl5-5.24.1/ccache-3.3.3/

The bug here is that, none of the dependencies of perl were bumped up in this case, such as pkg or ccache.  So we eventually try to build perl5 as a priority, but we didn't prioritize it as we built up to its dependencies.  The find would constantly look in the 99/ directory first, but never consider perl ready to build until its dependencies were satisified. So it's also wasting time in the queue looking there.
With a proper sorting I would expect this:
0/poudriere-devel-<IP_ADDRESS>60624_2/pkg-1.9.4_1/
0/poudriere-devel-<IP_ADDRESS>60624_2/ca_root_nss-3.28.1/
0/poudriere-devel-<IP_ADDRESS>60624_2/ccache-3.3.3/
0/poudriere-3.1.14_2/pkg-1.9.4_1/
0/poudriere-3.1.14_2/ca_root_nss-3.28.1/
0/poudriere-3.1.14_2/ccache-3.3.3/
0/portmaster-3.17.10/pkg-1.9.4_1/
0/portlint-2.17.6/pkg-1.9.4_1/
1 freebsd-release-manifests-20161010/pkg-1.9.4_1/
1 ca_root_nss-3.28.1/pkg-1.9.4_1/
1 ca_root_nss-3.28.1/pkg-1.9.4_1/perl5-5.24.1
99 perl5-5.24.1/pkg-1.9.4_1/
99 perl5-5.24.1/ccache-3.3.3/
102 ccache-3.3.3/pkg-1.9.4_1/
103 pkg-1.9.4_1/

Adding 99 to each existing depth from tsort -D. So I took the output from tsort -D, boosted
the packages from PRIORITY_LIST with 99, then added all packages with their new priority
(whether 0, 1, ..., or 99) back into the tsort file and then fed that into ptsort -p and it gives
what I expect:
    101 pkg-1.9.4_1
    100 ccache-3.3.3
     99 perl5-5.24.1
      1 freebsd-release-manifests-20161010
      1 ca_root_nss-3.28.1
      0 poudriere-3.1.14_2
      0 poudriere-devel-<IP_ADDRESS>60624_2
      0 portlint-2.17.6
      0 portmaster-3.17.10

The only problem I had was that I had to reverse the columns from what tsort was using.
So instead of this:
ca_root_nss-3.28.1 ccache-3.3.3
ca_root_nss-3.28.1 perl5-5.24.1
ca_root_nss-3.28.1 pkg-1.9.4_1
...

I had to use this:
ccache-3.3.3 ca_root_nss-3.28.1
perl5-5.24.1 ca_root_nss-3.28.1
pkg-1.9.4_1 ca_root_nss-3.28.1

Otherwise the final output was boosting reverse dependencies:
    101 poudriere-devel-<IP_ADDRESS>60624_2
    101 poudriere-3.1.14_2
    100 portlint-2.17.6
    100 ca_root_nss-3.28.1
     99 perl5-5.24.1
      6 portmaster-3.17.10
      5 freebsd-release-manifests-20161010
      5 ccache-3.3.3
      4 pkg-1.9.4_1

Anyway, if ptsort could use this "depth" priority by default and allow boosting specific packages by
a given number, that would eliminate having to run tsort -D and ptsort -p. Meaning this input:
ca_root_nss-3.28.1 ccache-3.3.3
ca_root_nss-3.28.1 perl5-5.24.1
ca_root_nss-3.28.1 pkg-1.9.4_1
ccache-3.3.3 pkg-1.9.4_1
freebsd-release-manifests-20161010 pkg-1.9.4_1
perl5-5.24.1 ccache-3.3.3
perl5-5.24.1 pkg-1.9.4_1
portlint-2.17.6 perl5-5.24.1
portlint-2.17.6 pkg-1.9.4_1
portmaster-3.17.10 ccache-3.3.3
portmaster-3.17.10 pkg-1.9.4_1
poudriere-3.1.14_2 ca_root_nss-3.28.1
poudriere-3.1.14_2 ccache-3.3.3
poudriere-3.1.14_2 freebsd-release-manifests-20161010
poudriere-3.1.14_2 pkg-1.9.4_1
poudriere-devel-<IP_ADDRESS>60624_2 ca_root_nss-3.28.1
poudriere-devel-<IP_ADDRESS>60624_2 ccache-3.3.3
poudriere-devel-<IP_ADDRESS>60624_2 freebsd-release-manifests-20161010
poudriere-devel-<IP_ADDRESS>60624_2 pkg-1.9.4_1
perl5-5.24.1 99

Should produce this output (I ignored the column ordering difference here):
    101 pkg-1.9.4_1
    100 ccache-3.3.3
     99 perl5-5.24.1
      1 freebsd-release-manifests-20161010
      1 ca_root_nss-3.28.1
      0 poudriere-3.1.14_2
      0 poudriere-devel-<IP_ADDRESS>60624_2
      0 portlint-2.17.6
      0 portmaster-3.17.10

Your sample input is inverted. I'm not sure what poudriere's tsort expects, but both ptsort and regular tsort expect is predecessor successor, not successor predecessor. With the correct input, ptsort gives exactly what you asked for:
% ./bin/ptsort -p
ccache-3.3.3 ca_root_nss-3.28.1
perl5-5.24.1 ca_root_nss-3.28.1
pkg-1.9.4_1 ca_root_nss-3.28.1
pkg-1.9.4_1 ccache-3.3.3
pkg-1.9.4_1 freebsd-release-manifests-20161010
ccache-3.3.3 perl5-5.24.1
pkg-1.9.4_1 perl5-5.24.1
perl5-5.24.1 portlint-2.17.6
pkg-1.9.4_1 portlint-2.17.6
ccache-3.3.3 portmaster-3.17.10
pkg-1.9.4_1 portmaster-3.17.10
ca_root_nss-3.28.1 poudriere-3.1.14_2
ccache-3.3.3 poudriere-3.1.14_2
freebsd-release-manifests-20161010 poudriere-3.1.14_2
pkg-1.9.4_1 poudriere-3.1.14_2
ca_root_nss-3.28.1 poudriere-devel-<IP_ADDRESS>60624_2
ccache-3.3.3 poudriere-devel-<IP_ADDRESS>60624_2
freebsd-release-manifests-20161010 poudriere-devel-<IP_ADDRESS>60624_2
pkg-1.9.4_1 poudriere-devel-<IP_ADDRESS>60624_2
perl5-5.24.1 99
    101 pkg-1.9.4_1
    100 ccache-3.3.3
     99 perl5-5.24.1
      1 freebsd-release-manifests-20161010
      1 ca_root_nss-3.28.1
      0 poudriere-3.1.14_2
      0 poudriere-devel-<IP_ADDRESS>60624_2
      0 portlint-2.17.6
      0 portmaster-3.17.10

Something seemed off (more than just the reversed input), so I modified ptsort to also print the depth and ran some more tests. It turns out that you can not sort packages strictly by depth:
% ./bin/ptsort -dp
a b
b c
c d
b e
e 99
      2     101 a
      1     100 b
      0      99 e
      1       1 c
      0       0 d
% ./bin/ptsort -Ddp
a b
b c
c d
b e
e 99
      2     101 a
      1       1 c
      1     100 b
      0      99 e
      0       0 d

If you sort these packages by depth, you nullify e's priority boost.
BTW, unless you know the depth of the tree in advance, the best boost value to use is simply the total number of packages plus one.

I played around with this more. I had misunderstood from @dag-erling when he suggested "a large number (basically, anything larger than the total number of packages in the list)" to mean that the priorities would reach up to 26000. However that's not the case. Running just a normal "pkg dep" list through ptsort -p, I get something that works (from bulk -a):
     38 pkg-1.9.4_1
     37 ccache-3.3.3_1
     36 indexinfo-0.2.6
     35 gettext-runtime-<IP_ADDRESS>_1
     34 gettext-tools-<IP_ADDRESS>
     34 perl5-5.24.1
     33 p5-Locale-gettext-1.06
     33 libffi-3.2.1
     33 pkgconf-1.2.2_1
     33 gmake-4.2.1_1
     32 m4-1.4.18,1
     32 autoconf-wrapper-20131203
     32 python27-2.7.13_1
     32 help2man-1.47.4
     31 automake-wrapper-20131203
     31 libgpg-error-1.26
     31 py27-setuptools27-32.1.0
     31 xorg-macros-1.19.1
     31 autoconf-2.69_1
     30 libgcrypt-1.7.6
     30 xproto-7.0.31
     30 libtool-2.4.6
     30 automake-1.15_1
     30 py27-pytz-2016.10,1
     30 libxml2-2.9.4
     29 libpthread-stubs-0.3_6
     29 xcb-proto-1.12
     29 python2-2_3
     29 py27-Babel-2.3.4
     29 libxslt-1.1.29_1
     29 libXdmcp-1.1.2
     29 xmlcatmgr-2.2_2
     29 py27-pystemmer-1.3.0_1
     29 py27-MarkupSafe-0.23_1
     29 libXau-1.0.8_3
     29 check-0.11.0
     28 bigreqsproto-1.1.2
     28 liblz4-1.7.5,1
     28 py27-docutils-0.13.1
     28 py27-sphinx_rtd_theme-0.1.9
     28 xmlcharent-0.3_2
     28 scons-2.5.1
     28 libnghttp2-1.19.0
     28 xcmiscproto-1.2.2
     28 py27-pygments-2.1.3_1
     28 kbproto-1.0.7
     28 lzo2-2.09
     28 ca_root_nss-3.29
     28 inputproto-2.3.2
     28 py27-alabaster-0.7.6
     28 py27-imagesize-0.7.1
     28 libxcb-1.12
     28 py27-Jinja2-2.8
     28 expat-2.2.0_1
     28 xtrans-1.3.5
     28 py27-snowballstemmer-1.2.0_1
     28 py27-six-1.10.0
     28 xextproto-7.3.0
     28 gmp-6.1.2
     28 iso8879-1986_3
     28 xf86bigfontproto-1.2.0
     27 bison-3.0.4,1
     27 docbook-xml-5.0_3
     27 libuv-1.11.0
     27 jsoncpp-1.8.0_1
     27 libarchive-3.2.2,1
     27 sdocbook-xml-1.1_2,2
     27 curl-7.52.1_1
     27 cmake-modules-3.7.2
     27 docbook-sgml-4.5_1
     27 py27-sphinx-1.4.8,1
     27 libX11-1.6.4,1
     27 mpfr-3.1.5_1
     26 binutils-2.27_5,1
     26 docbook-1.5
     26 libedit-3.1.20150325_2,1
     26 swig13-1.3.40_1
     26 fixesproto-5.0
     26 py27-enum34-1.1.6
     26 videoproto-2.3.3
     26 pciids-20170208
     26 ninja-1.7.2,2
     26 libXext-1.3.3_1,1
     26 cmake-3.7.2
     25 damageproto-1.2.1
     25 libdevq-0.0.4
     25 docbook-xsl-1.76.1_3
     25 libXv-1.0.11,1
     25 libpciaccess-0.13.4
     25 llvm39-3.9.1_1
     25 libXfixes-5.0.3
     24 libdrm-2.4.75,1
     24 libxshmfence-1.2
     24 libclc-<IP_ADDRESS>60915
     24 dri2proto-2.8
     24 makedepend-1.0.5,1
     24 libXvMC-1.0.10
     24 glproto-1.4.17
     24 libXdamage-1.1.4_3
     24 presentproto-1.1
     24 dri3proto-1.0
     23 libICE-1.0.9_1,1
     23 libglapi-13.0.4
     23 xf86vidmodeproto-2.3.1
     22 libXxf86vm-1.1.4_1
     22 renderproto-0.11.1
     22 pcre-8.40
     22 html2text-1.3.2a
     22 xcb-util-0.4.0_2,1
     22 fontsproto-2.1.3,1
     22 gbm-13.0.4
     22 freetype2-2.6.3
     22 libSM-1.2.2_3,1
     22 libfontenc-1.1.3_1
     22 libiconv-1.14_10
     21 nasm-2.11.08_1,1
     21 fontcacheproto-0.1.3
     21 libGL-13.0.4_1
     21 libXfont-1.5.2,2
     21 pixman-0.34.0
     21 libEGL-13.0.4_1
     21 qt5-qmake-5.6.2_1
     21 libXrender-0.9.10
     21 minixmlto-0.0.2
     21 libXt-1.1.5,1
     21 fontconfig-2.12.1,1
     21 mkfontscale-1.1.2
     21 png-1.6.28
     21 glib-2.46.2_4
     21 xcb-util-renderutil-0.3.9_1
     20 font-util-1.3.1
...

 The number on the left is the ordering. It creates a directory for the number and puts the package in there. So however it is sorted in stdout is not important.
What I was saying before was that using the "depth" as the initial priority (plus boosts) was what I wanted for -p. But having retried today, what you have for -p works fine. There's no need for a change to how -p works right now.

The ptsort -D feature is interesting in showing the root issue in this bug though, that the current method is not boosting all of these dependencies buried in the queue.

I hadn't noticed the ptsort -Dd thing you added. It matches what I had before, but I think your priority system works better. Btw there's no boosting support for depth it seems, which makes sense from a design standpoint, but I'm not sure if you intended that or not.

No, there's no boosting for depth, by definition. Priority is depth + boost. And please read my comments and maybe also the updated man page before replying. You do not want to sort by depth. In fact, I added the -D and -d options mostly to be able to prove that. You want to sort by final priority. You'll get a few more bins and they won't be consecutive, but cut -c1-8 | sort -urn will give you a sorted list that you can traverse instead of trying to count down from the maximum value.

Likewise, I said the sorting order is unimportant, thus -D is interesting but not useful. I don't use the sorted order of the output for anything. The output of [p]tsort is not the direct order that poudriere is building. It doesn't care what the order is in the output, only that things have some kind of priority next to them, and that value is what is used for ordering later.  So even if I used -D with -p, the end result to Poudriere is the same.  The depth idea was a simple thing I implemented that worked well enough but didn't recursively boost the depth of things, which your priority essentially is doing because it allows setting an initial value for an item, rather than the initial value being its depth.

Getting an exp-run with this at https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=217123 to ensure it does the right thing, before enabling by default for 3.1.

I just found a bug in ptsort which may result in incorrect ordering. I haven't yet come up with a test case where it makes a difference, but I won't rule it out. Here's the patch—I'll commit it once I have a regression test for it.
diff --git a/bin/ptsort.c b/bin/ptsort.c
index 177e1bc..d0bc36d 100644
--- a/bin/ptsort.c
+++ b/bin/ptsort.c
@@ -182,14 +182,16 @@ pnode_recalc(pnode *n, unsigned long depth, unsigned long prio)

 	if (n->depth >= depth && n->prio >= prio)
 		return;
-	if (depth > n->depth)
+	if (depth > n->depth) {
 		verbose("increasing the depth of node %s from %lu to %lu",
 		    n->name, n->depth, depth);
-	if (prio > n->prio)
+		n->depth = depth;
+	}
+	if (prio > n->prio) {
 		verbose("raising the priority of node %s from %lu to %lu",
 		    n->name, n->prio, prio);
-	n->depth = depth;
-	n->prio = prio;
+		n->prio = prio;
+	}
 	n->name[NAMELEN] = '*';
 	for (p = aa_first(&n->pred, &nit); p != NULL; p = aa_next(&nit)) {
 		if (p->name[NAMELEN] != '\0') {


feat: @farcaster/js should run in browsers

What is the feature you would like to implement?
Get @farcaster/js compiling in browsers
Why is this feature important?
Single-page and stand alone client applications will be able to interface directly with Hubs without a server.
Will the protocol spec need to be updated??
No
Additional context

Can @farcaster/js just use grpc-web instead?
Can @farcaster/hubble run a server that runs both grpc-js and grpc-web?
Can @farcaster/js compile in browsers end to end?

Related other tickets to consolidate:

https://github.com/farcasterxyz/hubble/issues/424
https://github.com/farcasterxyz/hubble/issues/497
https://github.com/farcasterxyz/hubble/issues/473
https://github.com/farcasterxyz/hubble/issues/494

Supporting web browsers isn't an immediate problem, but in the medium term it lowers the barrier to entry for community members looking to make something using farcaster, which is good. I imagine some devs will want very basic access and choose to use the hub RPC for reading/writing and others will use more feature-rich APIs or build their own backends for indexing data. There's certainly an opportunity for community members to make their own indexing services and APIs on top of hubs for many use cases, similar to Alchemy's various APIs for ethereum.
We should support both grpc and grpc-web simultaneously using a proxy (I made it work with Envoy, as you mentioned, though there may be other options too). So hub-to-hub communication will continue to use gRPC directly.

we should see whether farcaster/js works by default in serverless platforms like vercel. if it does, we can punt this to a later milestone

High-level goal:
Create a new package called @farcaster/hub-web that exports the same methods as @farcaster/hub-nodejs except with a grpc-web client instead of a grpc one that works in browsers.
Work to be done:

Create new @farcaster/hub-web package in the monorepo
Generate grpc-web compatible client from the existing rpc.proto schema and export it from the new package. If using ts-proto, we can use the outputClientImpl=grpc-web option (see documentation).
Wrap the grpc-web client implementation in promises and results (see client.ts in utils for reference)
We want to re-export all of @farcaster/utils from the new package, but utils right now depends on @farcaster/protobufs which depends on @grpc/grpc-js which is not supported in the browser. So we need to refactor utils to not depend on grpc-js. One option would be to remove any services generated code from the protobufs package and add generated grpc-js code to the @farcaster/hub-nodejs package. That way the nodejs package has the grpc-js client and the web package has the grpc-web client. Additionally, the hubble app will have to generate its own server implementation from the rpc.proto schema, since right now that's imported from the protobufs package (see server.ts).
Bonus points for merging @farcaster/utils and @farcaster/protobufs into a single @farcaster/core repo, since they'll always be used together and should support all environments.
Add a grpc-web proxy to the hub deployment. See grpc-web docs for envoy example.


The Lossnay switching procedure is not correct

I have a problem with the Lossnays. When I turn off a Lossnay device, the button becomes grayed out, then after about 1 minute the button turns back on, and only after another minute does the button become grayed out again. This behavior is not noticeable with the air conditioners. Everything is perfect here.
Uploading homebridge.log (17).txt…

try 0.18.0-beta and let me know

homebridge.log (18).txt
The on and off sequence is not working correctly. Unfortunately, I am not at the location where the Lossnays are situated. I can only check the status of the Lossnays using the Melcloud app. It seems that the Lossnays are being switched immediately, but the plugin buttons are switching back. During this process, the Lossnays are not being switched. After 1-3 minutes, the status of the button is corrected again. I hope my description is understandable.

Please  first remove old log instance because is very big now, (trash button in config ui x).
Install beta.2, save and restart plugin, set power off your lossnay with Home app, wait 3 min and get the log please.

homebridge.log (19).txt
Sorry i can't get the Home App working because the app don't find the Bridge on the remote site. But i can activate the Home Integration von Home Assistant. Here i activate the Lossnay wait 3 Minutes. Then i deactivate the Lossnay and wait 3 minutes. And now it seems that the on and off sequence is working correctly!


APIClient stub

Having an empty implementation of APIClient (and possibly some of the other engine-api interfaces as well) is very useful for testing. @vdemeester wrote an implementation that we use in libcompose. This might be something to consider adding to engine-api.

Note that it's the same for swarm, and several libraries that use engine-api (including mine 🐹), I also think it would make sence to have somewhere a "reference" 👼
/cc @calavera


helm template command failing

Hey everyone, I have a question about an issue I am having with nautobot helm chart v1.3.13 and v1.13.14
When i am running it with helm template command i am getting the following error:
nikolaos.a@MBP-176136 [15:44:52] [~/testing/Projects/nautobot-helm-config] [master *]
-> % helm template nautobot nautobot/nautobot -n nautobot --values helm-config-prd.yml --debug
install.go:194: [debug] Original chart version: ""
install.go:211: [debug] CHART PATH: /Users/nikolaos.a/Library/Caches/helm/repository/nautobot-1.3.14.tgz

Error: execution error at (nautobot/templates/secret.yaml:17:27): Existing secret postgres-secret not found!
helm.go:84: [debug] execution error at (nautobot/templates/secret.yaml:17:27): Existing secret postgres-secret not found!
nikolaos.a@MBP-176136 [15:58:16] [~/testing/Projects/nautobot-helm-config] [master *]

This happens both in the cli as well via our deployment tool. Commands helm install and helm upgrade work fine and the secret that supposedly does not exist, does in fact exist. I am a bit confused why this is happening. Any ideas?

This was fixed in #254 and will be released shortly in version 2.0.1

It seems like this is actually expected behavior since helm template can't use the lookup to access external cluster resources like secrets when using helm template.
If you are aware of a way to make this work locally and with ArgoCD I'd appreciate it a lot. Installing the chart with helm install works just fine.
According to this blog (see section More on the lookup Function) helm template always returns an empty response.


intercepting the file before download

great library!
forgive me if this is obvious but in the case that I wanted to let people review the video before downloading, would there be an elegant way to interrupt the process so that the file isn't downloaded directly?   at the moment, the file just downloads but in some cases, it may be helpful to preview the recorded video as the user may want to discard it (this is for an interactive work).  My understanding / memory is a little hazy, but I guess it's a blob before it gets downloaded.
thanks much in advance!

Yes, you are right, you can get a hold of the blob before downloading. The blob is generated inside each recorder:
https://github.com/tapioca24/p5.capture/blob/0cf2f2e4359c45e99944a30412c1518928c78a7f/src/recorders/webm-recorder.ts#L66
The event listener which receives the blob and downloads it:
https://github.com/tapioca24/p5.capture/blob/8b63af83179473939b89b13ea9882038a1b68cd1/src/p5.capture.ts#L285
Depending on your config you might want a segmented event handler too.
The easiest path is to add an option for a customized onFinish callback, which gets called with (blob, filename, defaultHandler) instead of the download. To keep it uniform, perhaps allow for all event handlers to be configured through the options (onFinished, onProgress, onAdded, onSegmented, onError, onStop, onStart).
It would look like:
P5Capture.setDefaultOptions({
  onFinished: (blob, filename, defaultHandler) => {
    // do your preview and then invoke defaultHandler later if you want to
  }
});

Hi there!
Thank you @ofZach for your request.
Thank you @guilhermesilveira for pointing me in the right direction.
The callback option to hook into events is flexible and powerful and I agree with this direction.
I plan to reorganize the interface a bit and add the following option.
P5Capture.setDefaultOptions({
  beforeDownload: (blob, context, next) => {
    // do something...
    next()
  }
})

type is as follows.
type P5CaptureOptions = {
  // ...
  beforeDownload?: (
    blob: Blob,
    context: {
      filename: string;
      format: P5CaptureFormat;
      isSegmented: boolean;
    },
    next: () => void,
  ) => Promise<void> | void;
};

The context object provides a source of information for additional processing.
Calling the next callback will start the file download, but can be omitted if not needed.
If you have any concerns, we would love to hear your feedback.

Released in v1.4.0. Thank you for all your contributions!


Ability to upload translations only for a certain language

Describe the solution you'd like
I'd like to upload translations for only one language (french). I tried with language option but it seems it is only supported for downloading, not uploading.
Describe alternatives you've considered
Use the button "upload translations" in the github integration settings.
Additional context
When developing, developers write both in english (source) and french (one of our translated languages)

Hi @philippeauriach thank you for your suggestion!
We added option upload_language
Now you can set to one of your translated languages and upload translations only for a certain language.


Add advisory for symfony/polyfill

Ref: https://github.com/symfony/polyfill/pull/155

I think this should also contain symfony/polyfill-php55

Should I create two YAML files for this? Or is there a better way to manage that?

Two yaml please yes.


1.18.2 0.7 forge crash at startup

The new version does not start.
Latest: https://0x0.st/oYjl.txt
Crash: https://0x0.st/oYjU.txt

That's interesting... It works fine on its own.
I can't tell you what's going on right now. I'll look closer later.
Sorry for the inconvenience 😅

No problem, take your time and keep up your good work!

same for forge 1.19.2 and 0.7, but version 0.6 already didn't work (crash at startup):
crash: https://pastebin.com/xt4nh9BV
debug.log

No problem, take your time and keep up your good work!

@agentwolf88 sooo, I think I fixed the 1.18 version.
Could you please check out if this version works for you?

Works fine now. :D THX!

Should I create a separate issue for 1.19.2?

Yeah.
Also, I still don't know what the problem with 1.19 is. Seems like forge changed more things internally. I'm not sure how soon (or if at all) I can fix this.


Call a callback on When, Then, And...

Hi!, I would love to know how can I call a callback for asynchronous operations? With Jest the it/test method includes a callback if we need it, and I need it because async/await doesn't help me right now, for example, I am uploading big files and they are handled by streams so, I need to call req.end manually with the callback, example:
  test('User gets OK after file is uploaded', ({ given, when, then }) => {
    let stream: Readable;
    let req: any;
    given('a dataset file', () => {
      stream = fs.createReadStream(`${process.cwd()}/files/autopromo.csv`);
    });
    when('the file is uploaded', () => {
      req = supertest.agent(app)
      .post('/v1/upload')
      .set('Content-Type', 'application/octet-stream')
      .set('Accept', 'application/json')
      .set('Transfer-Encoding', 'chunked')
      .set('x-csv-provider', 'autopromo')
      .set('x-csv-columns', 'vin,uuid,make,model,year');
      stream.pipe(req as any, { end: false });
      stream.on('end', () => req.end());
    });
    then('the server responds with status 200', () => {
      req.on('response', (data: any) => {
        expect(data.body).toEqual({ msg: 'ok' });
      });
    });
  });

This line: stream.on('end', () => req.end());
needs to be something like this:
    when('the file is uploaded', (done) => {
      req = supertest.agent(app)
      .post('/v1/upload')
      .set('Content-Type', 'application/octet-stream')
      .set('Accept', 'application/json')
      .set('Transfer-Encoding', 'chunked')
      .set('x-csv-provider', 'autopromo')
      .set('x-csv-columns', 'vin,uuid,make,model,year');
      stream.pipe(req as any, { end: false });
      stream.on('end', () => req.end(done));
    });

whereas done is the callback, but when I set it, it throws an error, I mean, the test passes, but there's a complain.
Any help?

Callbacks aren't supported, but promises are, so something like this should work for you:
    when('the file is uploaded', (done) => {
      return new Promise((resolve) => {
        req = supertest.agent(app)
          .post('/v1/upload')
          .set('Content-Type', 'application/octet-stream')
          .set('Accept', 'application/json')
          .set('Transfer-Encoding', 'chunked')
          .set('x-csv-provider', 'autopromo')
          .set('x-csv-columns', 'vin,uuid,make,model,year');
          stream.pipe(req as any, { end: false });
          stream.on('end', () => req.end(resolve));
        });
    });

Thank you @bencompton!,
Hope this help somebody else:
test('User gets OK after file is uploaded', ({ given, when, then }) => {
  let stream: Readable;
  let req: supertest.Test;
  let res: supertest.Response;

  given('a dataset file', () => {
    stream = fs.createReadStream(`${process.cwd()}/files/autopromo.csv`);
  });
  when('the file is uploaded', () => {
    return new Promise((resolve) => {
      req = request
        .post('/v1/upload')
        .set('Content-Type', 'application/octet-stream')
        .set('Accept', 'application/json')
        .set('Transfer-Encoding', 'chunked')
        .set('x-csv-provider', 'autopromo')
        .set('x-csv-columns', 'vin,uuid,make,model,year')
        .on('response', (response) => { res = response; })
      stream.pipe(req as any, { end: false });
      stream.on('end', () => req.end(resolve));
    });
  });
  then('the server responds with status 200', () => {
    expect(res.body.msg).toEqual('OK');
  });
});


Importance of zstack-size

I'm trying to understand what zstack-size does and why it's important.
Reason I'm trying to investigate his is that I'm almost certain that it causes a stack overflow in my code but I can't say for sure.
I've taken the wasm module and run it in the browser I'm getting Maximum call stack size exceeded error.
If I'm removing zstack-size then in the browser it runs to infinity but on arduino/wasm3 it doesn't even start.
Any pointers would be greatly appreciated!

I'm not sure, wasm3 is handled by a different repository.
Also, it would be great to have a minimal example that  highlights the issue.


Question about entropy

@kdomino I have such a reflection - if we want to compare entropy for 1-3 and 1-4 angles, shouldn't we take only 23000 for both? If we take 24k for one and 23k for the second the entropy always will be greater for the first, it is then no related with chemical structure. And we see this in the CS6 case. But now we don't know if there is part of the entropy which is related to the specific angles: if 1-3 have different entropy than 1-4. We see only that we have greater probe.

We may, but this is not the large difference. Entropy is a bit more dependent on the number of bins. By def (Eq. 2) sum goes over non-zero bins. So mind the time we have, for now I would only mention that these numbers are similar but not the same

I'm just not sure if difference presented on this plot:

is not related only to the size of the probe. The difference is not lagre - it is only 2.

If you think it is important, let drop an article for now or ask @soycapitan , I do not have time either full idea how to encode it

https://pl.wikipedia.org/wiki/Entropia_metryczna
What about some normalization? Can we safely divide by our vector length?
Taking 23k out of 24k would mean discarding 1k data points - we should then rationalize why we chose one 1k points over others to discard, this may be tricky.
If we normalize by length we should get some closer values:
57 x 23 = 1311
55 x 24 = 1320
1320/1311 = 1.007
57/55 = 1.04

In my opinion it is enough to just comment it in caption or somewhere, but if you believe it needs more elaboration ....

In your code there is parameter self.no_mers - I change 24 on 23 and I compare the results for 1-4.
This has some further implications. This number means that we have a structure of no_mers mers. Playing with this means that we discard angles in both metrics - when we drop last mer (this is what effectively happens) we drop not only its internal angles but also the inter-mer angles related to the dropped last mer.

@soycapitan I saw this, that's why I comparing only 1-4 before and after. But having 22k 1-3 also show us that difference in 1k points it is not much.
@kdomino As I was worried only that difference in entropy can be about 1-2, and we just see that it is not true, I will only add one sentence in the manuscript.


[Receive bitcoin] Address label is not visible anywhere

App version: v0.1.92

Goto Receive bitcoin screen
Set address label
Send sats to that address

Expected:

Address label to be shown in txn page next to Recipient bitcoin address
Wallet -> Settings -> Addresses. Respective address to display the given address label.
Wallet -> Settings -> Addresses -> Tap on that address. The address label to be displayed in the address popup.

Observed:

The label is not seen anywhere

Note: Same behavior when you do 'Edit label' in Address popup, save the label, go back to home and check in wallet again.

Fixed on main. Address label was being overwritten by empty tx label.


Improve nak pdu api

Codecov Report
Attention: 17 lines in your changes are missing coverage. Please review.

Comparison is base (da17922) 96.29% compared to head (68f911f) 96.04%.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #49      +/-   ##
==========================================
- Coverage   96.29%   96.04%   -0.26%
==========================================
  Files          84       84
  Lines        6365     6391      +26
==========================================
+ Hits         6129     6138       +9
- Misses        236      253      +17

Flag
Coverage Δ

unittests
96.04% <39.28%> (-0.26%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Files
Coverage Δ

spacepackets/util.py
91.04% <100.00%> (+0.09%)
:arrow_up:

tests/test_util.py
97.61% <100.00%> (+0.08%)
:arrow_up:

spacepackets/cfdp/conf.py
96.66% <50.00%> (-3.34%)
:arrow_down:

spacepackets/cfdp/pdu/nak.py
84.29% <15.78%> (-12.77%)
:arrow_down:

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Lower case on --workDir command

It's more readable and easier to type command to --workdir than using camel case, don't you think if all the command can update to lower case?
I would rather call it --rootfolder or --rootdir to make sense for other developers from PHP, Java, etc.

This seems like a good idea to me.

This was added in 0.11.


Add support for foreign keys

The basic pattern for creating a foreign key relationship is adding this to the model class:
foreign_key_1 = foreign_key_relationship.ForeignKeyRelationship(
  'SmallTestModel', {'referencing_key_1': 'key'})

In this case, the generated DDL is as follows: CONSTRAINT foreign_key_1 FOREIGN KEY (referencing_key_1) REFERENCES SmallTestModel (key).
As demonstrated in the tests, this PR allows for more complex foreign key relationships as well, such as multiple columns in one relationship and circular references.
This PR does not yet add support for adding or removing foreign key relationships, but since we require foreign keys to have names, we can reference that for deletion.

have you looked at IncludesCondition? if this meant to replace "Relationship", it should probably cover conditions too

Good catch. That does require quite a bit of changes so starting a separate PR: https://github.com/google/python-spanner-orm/pull/88/files. That PR is not yet complete but seems like ForeignKeyRelation should be able to be used with IncludesCondition.


Default Folder wont allow folder under "Bookmark Toolbar"

Add-on version
Add-on version: 2.8
Describe the bug
Under more options of the Livemarks add-on I try to select a Default folder that is in the "Bookmarks Toolbar" folder - i.e I have the Live RSS feeds as a drop down menu off a Folder on the Bookmarks Toolbar. It only shows me a list of Folders at the top level of Bookmarks in Firefox and it is also not the full list at that level.
To Reproduce
Steps to reproduce the behavior:

Go to Firefox settings menu, select Add-Ons, select Livemarks, select Options and then More Options and try to select a folder in the Default folder area.
I have a folder under Bookmarks Toolbar called RSS-feeds which I would like as the Default Folder for my RSS feeds. Livemarks does not allow me to traverse the Bookmarks Toolbar folder so I can't select it.
In the past I had added the RSS feeds to another, high level, folder and then copied them from there to the other folder but recently they feeds have stopped updating and now show as broken livemarks - again I can't select a subfolder to assign it to.
Error is no update and broken livemark status

Expected behavior
Selecting a subfolder should be acceptable to Livemarks options.

I'm able to set the default folder to a sub folder fine. I think we do remove the default folders from the broken feeds dialog (I can't remember why), but their subfolders should still be there.

I'm having this issue also.  The problem is not setting the default folder to a sub-folder, but rather setting the default folder to any sub-folder on the Bookmarks Toolbar.  ie The Bookmarks 'Toolbar' is not a selectable option.  But this is exactly where I want to put livemarks that I access all the time.  Thanks, cs

That's a different issue and basically intended behavior. You can only have one feed per bookmark folder. Some of the built-in default bookmarks folder like Toolbar, Other Bookmarks etc. are intentionally not allowed.

Thanks for the quick response although I don't think I understand your explanation!  I also don't think I communicated the problem very well either, so here is a more long-winded description.
This was working as I expected before, but I corrupted my Firefox bookmarks and was forced to create a new profile and load a bookmarks backup file to get my bookmarks back.  However this meant I needed to reload my Firefox extensions including LiveMarks.  My bookmarks were restored correctly including my previous LiveMarks, which I stored in a sub-folder on the Bookmarks Toolbar called "RSS Feeds".   They are stilll getting updated correctly from the RSS feeds but when I attempted to use the same folder for new Livemarks, it would only allow me to a specify a location under "Bookmarks Menu" and Livemarks appeared to have no knowledge of the previously created livemarks (in my original profile)!
Here are screenshots I hope illustrate the problem;

Image of previous RSS subfolder on toolbar in old profile
Image of previous Default Folder choices in old profile (which included Bookmarks Toolbar)
Image of current Default Folder choices in new profile under Bookmarks Manager

I hope this makes sense (Images 1, 2 & 3 follow) - Thanks!

I can confirm this error !!!
In the config is no folger available (earlier already), which is in the "Bookmark Toolbar".
In my toolbar i have the folder "News" ...
It used to work. Probably because of a new firefox version...

For me this bug was a showstopper.  I.e. I created a folder named 'rss' on the toolbar which allowed easy access and organization to my RSS feeds.  Now I am unable to access the toolbar or any of its sub-folders, which means that I have to mix my RSS live marks in with the 'regular' bookmarks - which pretty much negates the convenience!
I'm tempted to try and fix it but hesitant due to my lack of experience working on Firefox extensions ...

I honestly still don't understand the problem fully, but I have a theory:
I see that you use your "rss" folder as both the parent of other folders, but also a RSS folder itself? That is not supported and will break thing. You have to change the livemarks that uses the folder for "rss" to something else.

Tom,
Tom,
Thanks for your quick response.  I will try to explain more clearly below your comment.
On Wednesday, July 28, 2021, 1:12:05 PM MDT, Tom Schuster<EMAIL_ADDRESS>wrote:

I honestly still don't understand the problem fully, but I have a theory:  I see that you use your "rss" folder as
both the parent of other folders, but also a RSS folder itself?
No, the RSS folder was simply an ordinary folder on the Toolbar.  Each rss feed was a separate sub-folder under the "RSS' folder.  This worked as desired for several months, since I initially added the livemarks extension.  Suddenly it stopped working (after a Firefox update I believe).  I tried to fix it by creating a new profile and loading a backup copy of my Firefox bookmarks and then adding Livemarks again.  However when I tried to add 'RSS;' (or any folder on the toolbar for that matter) as the location for my Rss feeds, the toolbar and its sub-menus were no longer visible, i.e. it was not visible/selectable as an option in the menu.    Only the "Bookmarks Menu" and all of its sub-menus were visilble.
I hope that is a clearer explanation.  I think there is something about the structure of my Firefox bookmarks that Livemarks doesn't like, but I have no clue as to what, because they seem very ordinary and there is no feedback as to what the problem is, other than Livemarks just not "seeing" the Toolbar folder.  I am very perplexed!

Craig Steury
On Wednesday, July 28, 2021, 1:12:05 PM MDT, Tom Schuster @.***> wrote:

That is not supported and will break thing. You have to change the livemarks that uses the folder for "rss" to
  something else.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.

I suggest you start by investigating the getAllBookmarkFolders method. You can also run this from the console and set breakpoint.

Ok, thanks I'll have a look.
cs
On Wednesday, July 28, 2021, 2:22:03 PM MDT, Tom Schuster<EMAIL_ADDRESS>wrote:

I suggest you start by investigating the getAllBookmarkFolders method. You can also run this from the console and set breakpoint.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.


Add vue 2 support

ready () is deprecated in vue 2.0 - so it doesnt work now

I tried changing ready() to mounted(), but I'm getting >> Uncaught ReferenceError: CountUp is not defined.
Did you try this successfully?

Ahh, figured.. countup.js is added right in the html.. I'd think it should be imported into the plugin..?

Thanks for your messages guys.  I will look into making this vue-2 compatible.


Update friends.md

We've been using this in testing mode for a few months, good enough for production now, so adding to the friends list.

Thanks!


docs: added Hetzner Cloud machine driver

Proposed changes
added an external Docker Machine Driver to the docs.

Deploy preview for docsdocker ready!
Built with commit 73a4560673221d93688ace0fc5be810cd5813930
https://deploy-preview-9062--docsdocker.netlify.com


Add image encryption support and ctr support

This series of patches adds support for the asymmetric and symmetric layer encryption code as well as support for ctr's images subcommand to support encryption (encrypt) and decryption (decrypt) of images as well as the retrieval of information about the layers of an image (layerinfo). The latter shows the encryption scheme used for each layer. An extensions to the documentation is also added.

This PR is a superset of PR #3001. It adds patches on top of the branch created for PR #3001, so the patches from PR #3001 are also visible here. The patch "Address Phil's comments to the asymmetric crypto code" is the last patch from #3001 .

Codecov Report

Merging #3134 into master will decrease coverage by 0.76%.
The diff coverage is 34.08%.

@@            Coverage Diff             @@
##           master    #3134      +/-   ##
==========================================
- Coverage   43.58%   42.81%   -0.77%
==========================================
  Files         104      112       +8
  Lines       11156    12133     +977
==========================================
+ Hits         4862     5195     +333
- Misses       5555     6134     +579
- Partials      739      804      +65

Flag
Coverage Δ

#linux
46.69% <37.1%> (-0.88%)
:arrow_down:

#windows
39.9% <34.08%> (-0.6%)
:arrow_down:

Impacted Files
Coverage Δ

content/content.go
0% <ø> (ø)
:arrow_up:

images/encryption/gpg.go
0% <0%> (ø)

images/encryption/gpgvault.go
0% <0%> (ø)

platforms/platforms.go
71.56% <0%> (-7.78%)
:arrow_down:

content/helpers.go
17.91% <0%> (-4.74%)
:arrow_down:

images/encryption/keywrap/pgp/keywrapper_gpg.go
35% <35%> (ø)

...mages/encryption/keywrap/pkcs7/keywrapper_pkcs7.go
44.61% <44.61%> (ø)

images/encryption/keywrap/jwe/keywrapper_jwe.go
52.77% <52.77%> (ø)

images/encryption/encryption.go
54.7% <54.7%> (ø)

images/encryption/blockcipher/blockcipher.go
69.44% <69.44%> (ø)

... and 9 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 90a7da8...afac701. Read the comment docs.

Thanks for the review @stevvooe ! Looks like most of the changes are straightforward and we will start working on those. The only comment we need some clarity on is on the layer filters. @stefanberger and I had a discussion to understand the comments there and put together a summary of our discussion. I'll put that under on of the inline comments!

vndr is now reporting a mismatch between vendor.conf settings and actual files committed; maybe a recent commit as this used to pass? CI Log:
These files were modified:
 M vendor/golang.org/x/crypto/cast5/cast5.go
 M vendor/golang.org/x/crypto/ed25519/ed25519.go
 M vendor/golang.org/x/crypto/ed25519/internal/edwards25519/edwards25519.go
 M vendor/golang.org/x/crypto/openpgp/keys.go
 M vendor/golang.org/x/crypto/openpgp/packet/encrypted_key.go
 M vendor/golang.org/x/crypto/openpgp/packet/packet.go
 M vendor/golang.org/x/crypto/openpgp/packet/private_key.go
 M vendor/golang.org/x/crypto/openpgp/packet/public_key.go
 M vendor/golang.org/x/crypto/openpgp/packet/signature.go
 M vendor/golang.org/x/crypto/openpgp/packet/userattribute.go
 M vendor/golang.org/x/crypto/openpgp/write.go
The command "../project/script/validate/vendor" exited with 1.

@estesp Thanks for pointing this out. We didn't do any forced updates since this error happened but somehow the master branch has seen updates that affected this now. I am not sure how these are connected. I added a patch now on top that fixes the issue.

rebased

@dmcgowan From my understanding of the lease resource management PR, it seems like I will need to change the encryption APIs to add a leases.Manager input to the API call? And use the manager to add resources from that lease manager. Does this sound okay?
func EncryptImage(ctx context.Context, cs content.Store, ls leases.Manager, desc ocispec.Descriptor, cc *encconfig.CryptoConfig, lf *LayerFilter) (ocispec.Descriptor, bool, error)

We are aiming for kube 1.16 for https://github.com/kubernetes/enhancements/pull/1066. It would be great if we could restart the conversations around this.

@lumjjb i'm good with it as longs as the majority of this is implemented client side

@crosbymichael Yup! Most of it is client side :) :) :)

Thanks, I'm going to review and play with this today

How do I test this without gpg?  I just wanna use simple rsa keys or something like that

@crosbymichael
# genkeys
openssl genpkey -algorithm RSA -out private_key.pem -pkeyopt rsa_keygen_bits:2048
openssl rsa -pubout -in private_key.pem -out public_key.pem

# enc
sudo ctr images encrypt --recipient public_key.pem  docker.io/library/ubuntu:16.04 enc_img

# dec
sudo ctr images decrypt --recipient private_key.pem  enc_img dec_img

@lumjjb I get the error ctr: unable to determine GPG version

I only bring this up because I'm not sure gpg should be a requirement.  Isn't there simpler crypt methods that we can support out of the box as well?

After installing gnupg I get the following errors:
sudo ctr images encrypt --recipient public_key.pem  docker.io/library/ubuntu:16.04 enc_img
Encrypting docker.io/library/ubuntu:16.04 to enc_img
ctr: content digest sha256:2a522207f17dad6487398e666217b842511b6d0aef5d5d3b8a6c92e28581d60e: not found

@crosbymichael I just ran into the same problem. Try to only encrypt for amd64:
./bin/ctr images encrypt --recipient mypubkey.pem  --platform linux/amd64 docker.io/library/ubuntu:16.04 ubuntu.enc:latest

For you command line to work you would need to download all layers:
./bin/ctr images pull --all-platforms docker.io/library/ubuntu:16.04

What it was missing was an arm/v7 layer:
$ ./bin/ctr images layerinfo enc_img
   #                                                                    DIGEST         PLATFORM       SIZE   ENCRYPTION   RECIPIENTS
   0   sha256:8c3dd606e321b1b12909d9da52d1f3ef8cfa6970e2efd93cf7ec1e189710bb08      linux/amd64   43838430          jwe        [jwe]
   1   sha256:2601873ef2c570747d703c1773e206129017927b935c40f1b790963e807ecf2e      linux/amd64        545          jwe        [jwe]
   2   sha256:9ec1df30185ea93d6171c49e3661e74a2827c0c37699882d09e28173df06bd84      linux/amd64        866          jwe        [jwe]
   3   sha256:b9117b45a266fefb02754a2cd778f5ae1848a91cd22e71a30801ea097ac5abfe      linux/amd64        185          jwe        [jwe]
   0   sha256:2a522207f17dad6487398e666217b842511b6d0aef5d5d3b8a6c92e28581d60e     linux/arm/v7   38710547
   1   sha256:85e59c2ace1d7baaf22d49477f12e41743f37497fbdeed7092b5b847be1d05e9     linux/arm/v7        510
   2   sha256:23423fd3acc94a25767511b76b252e7fc469334e588905cf449583baab12fcf5     linux/arm/v7        852
   3   sha256:20fb32d384c9241b858083119d34b653c651bda61a0a79679511a3e3ee22ad90     linux/arm/v7        170
   0   sha256:6d4d0d1f37de0d4417055235ff4a4ce52439ec078835b8d6c91cecd64d192029   linux/arm64/v8   39790979
   1   sha256:b2b10db5fd500822d1d74c2d604ce345b50cbea31db8f3813f5b166633b8042d   linux/arm64/v8        470
   2   sha256:45503012c89a34b797c44068bce23b360a1212b5582d4483553249eba85ce075   linux/arm64/v8        850
   3   sha256:078966f8f5cec9611dad9af01d4b52e898763e22fd0facf8a5dd72d48cbe568f   linux/arm64/v8        169
   0   sha256:856ae43f63ed6657b450c1d9c8e4cb587d875421e485fe008070949c85478d95        linux/386   43979421
   1   sha256:fb11583701fe0ca24f56e54f018637031575222c3e805aef8d79d17d47d26386        linux/386        515
   2   sha256:f15f4b605d91eafa00b250a75f7557fdec29c3f0435c255edb984caee21cabb6        linux/386        849
   3   sha256:1f07c26ededbd5bb64a792e097f441fb697d2ffe42776a621885522339045517        linux/386        169
   0   sha256:1b0f236ed1ddec38971bf51256ce70e6672ebc41b9d7544e3f278adf3bda5373    linux/ppc64le   45976401
   1   sha256:d122a84ecd3a18e9b734692549cc4536a3bd6c4ef8ea05527403a57c1232d1cc    linux/ppc64le        472
   2   sha256:cf57bfa9cc2bbbd897299e6df30a2374ffcbd35a3bbde6ab9130c888321c4663    linux/ppc64le        855
   3   sha256:fb32e0af70bba6cc9b0b95d580545ed84bebf7dcdae99d34ca6a0cd9acc41f5f    linux/ppc64le        171
   0   sha256:13164bfcbd4c81a114d0c20e22b11bbc14d1e6c6c1406525bd49ea9b80196704      linux/s390x   42663919
   1   sha256:5ca6e9f3028d16f9753abf51f7e631b78cbdffb436a2288a6365615e21fdd19b      linux/s390x        470
   2   sha256:84b454fecd10a241beb8ca744cdd20dca457837201c9078fec4a4ea4ec85f56b      linux/s390x        851
   3   sha256:6915070f22f8931076f003becb0eeef15eb5c5b52d6539abb9e2d37afd6cd7a7      linux/s390x        170

@stefanberger thanks! That did the trick

I only bring this up because I'm not sure gpg should be a requirement. Isn't there simpler crypt methods that we can support out of the box as well?

Gotcha, we'll make the change so that it will only require gpg only if it is provided a GPG key.

Overall I think it looks good for an initial merge.
What I would like to see in a follow up is more of a client API designed for containerd.  We use the opts pattern on many of our client functions and we should have things like client.Pull(ctx, ref, containerd.WithImageDecryption) and same with push.  Clients like CRI and such all use the Go client API to interact with containerd so ctr is good for debugging but users need a solid Go API.

What I would like to see in a follow up is more of a client API designed for containerd. We use the opts pattern on many of our client functions and we should have things like client.Pull(ctx, ref, containerd.WithImageDecryption) and same with push. Clients like CRI and such all use the Go client API to interact with containerd so ctr is good for debugging but users need a solid Go API.

Awesome - yea, agreed on this. The next step of this would be to have a PR for the entire pull flow. And after that cri would be able to use it pretty easily as well after the KEP gets merged.

Overall I think it looks good for an initial merge.
What I would like to see in a follow up is more of a client API designed for containerd. We use the opts pattern on many of our client functions and we should have things like client.Pull(ctx, ref, containerd.WithImageDecryption) and same with push. Clients like CRI and such all use the Go client API to interact with containerd so ctr is good for debugging but users need a solid Go API.

@crosbymichael We do have an implementation for Pull which uses the pattern you mentioned. We have a function WithDecryptionKeys, https://github.com/harche/containerd/blob/v5%2B%2B/client_opts.go#L208  which is consumed by CRI while trying to pull the image, https://github.com/harche/containerd/blob/v5++/vendor/github.com/containerd/cri/pkg/server/image_pull.go#L103
That code is not part of this PR though, we thought changes related to CRI could be part of a separate PR.

Got caught with upstream change from gometalinter to new CI lint; these should be fairly easy to fix:
$ make check
🇩 proto-fmt
🇩 check
golangci-lint run
images/encryption/blockcipher/blockcipher.go:38:2: structtag: struct field tag `json:'symkey'` not compatible with reflect.StructTag.Get: bad syntax for struct tag value (govet)
	SymmetricKey  []byte            `json:'symkey'`
	^
images/encryption/blockcipher/blockcipher.go:39:2: structtag: struct field tag `json:'cipheroptions'` not compatible with reflect.StructTag.Get: bad syntax for struct tag value (govet)
	CipherOptions map[string][]byte `json:'cipheroptions'`
	^
images/encryption/utils/testing.go:131:20: `tempalte` is a misspelling of `template` (misspell)
// The certificate tempalte for the public key is optional
                   ^
images/enc_utils.go:198:71: `encyrption` is a misspelling of `encryption` (misspell)
		// we only encrypt child layers and have to update their parents if encyrption happened
		                                                                    ^
images/encryption/gpgvault.go:43:39: `originall` is a misspelling of `original` (misspell)
	keyDataList [][]byte // the raw data originall passed in
	                                     ^

Addressed all of the comments except the one on WriteBlindBlobLeased because the issue here is that the digest is not known until later on. We could change the interface to return a finalizer, after leases are initialized... Or return the content.Writer. @dmcgowan any thoughts on this?

@lumjjb The lease rides the context during Commit and gets added then. There is no need to manually mess with the lease in these normal flows.

@dmcgowan I've removed the regular WriteBlobLeased, but am still having an issue with the blind write since we can't AddResource before knowing the digest, which happens after the write, but before the commit. I hope I am not misunderstanding something.

Do we want to squash this before merge? At 53 commits over a period of months, git bisect would be troublesome for a lot of the ones that needed rebase, are not compilable, etc.

I don't think it is worth squashing. Mostly new code so bisect isn't important here.

Let me know if you'd like me to squash this and break down the changes into a single commit message.

Lets squash it then get it in

Squashed to 3 commits!

Thanks so much @dmcgowan and @estesp for following along the long journey of this gigantic PR! Am really glad to see this get merged!!! Thanks @crosbymichael , @AkihiroSuda for the reviews and comments too!

@dmcgowan @estesp @crosbymichael @AkihiroSuda Thanks a lot for you help!

Thanks @lumjjb and @stefanberger for your patience and perseverance, this was one of the trickier ones.


Add back button to BCP Schema Reference View

Is your feature request related to a problem?
Navigating the schema reference in BCP is quite clunky, because there is no way to go back while in type view.
So if I want to check the type of a field and then return to the object/query that contains that field, I have to start from the top (list of queries) all over again, which is quite infuriating tbh. 😅 At first I thought that this was just a "me" problem, but now two of my colleagues ranted about the exact same issues, so I thought I'd open a feature request.
Other than that I love your product. Keep it up!
The solution you'd like
Add a small back button to navigate back to the latest view:

I don't care about the exact placement. The idea counts.
Product
Banana Cake Pop

@DerZade Thanks for opening this issue. We've added some improvements to the "Schema Reference" in the next release, however it won't cover this part of the navigation, which will most likely come in the next one.
CC @rstaib

In the mean time you can use the column view.

I would still love this 🙈

Is this still the case?
Back button doesn't work for me...


Update armeria, armeria-grpc, ... to 1.13.2

Updates

com.linecorp.armeria:armeria
com.linecorp.armeria:armeria-grpc
com.linecorp.armeria:armeria-scalapb

from 1.11.0 to 1.13.2.
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "com.linecorp.armeria" } ]

labels: library-update, early-semver-minor, semver-spec-minor

Superseded by #124.


Bump generated keypairs and begin many_stakers

bumps generated keys up from 40 to 120
creates a new postake genesis ledger option with 20 large balance accounts

@bkase I want to repeat values in the list in a simple way..
let rec helper acc n l =
   if n = 0 then acc else helper (l @ acc) (n-1) l ;;

lets me do things like
helper [] 10 [ 10 ]
;;
- : int list = [10; 10; 10; 10; 10; 10; 10; 10; 10; 10]

I can initialize
helper [ 50; 50; 50; ] 10 [10];;
- : int list = [10; 10; 10; 10; 10; 10; 10; 10; 10; 10; 50; 50; 50]

but I can't nest them.
helper [ helper [] 10 [10] ] 5  [ 50 ]
;;
Error: This expression has type int but an expression was expected of type
         int list

use two declarations?  One for the small numbers, and then one for the joined list?

ok. @bkase
include Make (struct
  let accounts =
    let high_balances = List.init 20 ~f:(Fn.const 5_000_000) in
    let low_balances = List.init 10 ~f:(Fn.const 1_000) in
    let balances = List.append high_balances low_balances in
    List.mapi balances ~f:(fun i b ->
        { balance= b
        ; pk= fst Coda_base.Sample_keypairs.keypairs.(i)
        ; sk= snd Coda_base.Sample_keypairs.keypairs.(i) } )
end)

Good pattern?  Use on other files too?


Add Troubleshooting doc from solutions builder

I'll suggest thinking more on how we can keep consistent on the docs between here and SB repo. Since this troubleshoot covers generic issues, you can just add a section in the root README here, and point to the SB's troubleshoot.


Support ability to add/remove TConstruct Materials (e.g. Bow Limb, etc.)

It looks like its possible to modify existing materials via ToolStats, but not to add or remove materials.

Take a look at my other mod :)
http://minecraft.curseforge.com/mc-mods/228458-jsonables


support custom mobile capabilities logic on carina level: fullResetBeforeSuite

there is standard fullReset capability which guarantee that app is uninstalled completely and cache is cleaned. Using this capability we can successfully override custom install/uninstall logic using direct adb calls (already removed from carina).
But negative aspect in this capability is that based on driver_mode we always reinstall application on the same device.
To minimize copy operation I proposed to add new capability:
-fullResetBeforeSuite
if it is true MobileCapabilityMatcher:

when we request device at first and capabilities matchers are satisfied then we convert this capability to standard
fullReset=true
remember device udid into local storage
During next attempt to open session on the same device do not transfer fullReset=true.

As result we will clean app/cache only at once during test suite start.

done
https://github.com/qaprosoft/carina/pull/712/files

closing as next changes expected to be delivered on carina-grid independent project


TagBot trigger issue

This issue is used to trigger TagBot; feel free to unsubscribe.
If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.
If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

Triggering TagBot for merged registry pull request: https://github.com/JuliaRegistries/General/pull/73590

Triggering TagBot for merged registry pull request: https://github.com/JuliaRegistries/General/pull/106711


scoop uninstall blocked when app is running

Example:
$ scoop uninstall openssh
uninstalling openssh (5.4p1-1)
removing shim for scp
removing shim for sftp
removing shim for slogin
removing shim for ssh-add
removing shim for ssh-agent
removing shim for ssh-keygen
removing shim for ssh-keyscan
removing shim for ssh
couldn't remove ~/appdata/local/scoop\apps\openssh\5.4p1-1: it may be in use

I'm thinking that users should be able to uninstall apps regardless of whether it is running or not - which means that we need some way of stopping services before uninstalling, so that the user doesn't need to figure it out themselves.

Agreed! But then there should be a way to add services in the first place not ?

First thing to do when uninstalling should be check for directory is in use.
I could think only using Get-Process. Something like:
if ((Get-Process -FileVersionInfo -ErrorAction SilentlyContinue | ? { $_.FileName -like '$dir`*'}).Count -gt 0) {
     abort 'Folder is in use. Make sure to close program'
}

https://github.com/lukesampson/scoop/blob/99b5afafa086246aac65d63160218c38b014650d/libexec/scoop-uninstall.ps1#L41-L46
$dir should be evaluated before writing information about uninstalling (L43)
@r15ch13 Do you have better solution?

Result of get-process for runing Steam

PS C:\Windows\system32> (Get-Process -FileVersionInfo -ErrorAction SilentlyContinue | ? { $_.FileName -like 'D:\scoop\ap
ps\Steam*'})

ProductVersion   FileVersion      FileName
--------------   -----------      --------
2<IP_ADDRESS>4... 2<IP_ADDRESS>4... D:\SCOOP\apps\Steam\current\steamapps\common\God Awe-full Clicker\GAC.exe
<IP_ADDRESS>      <IP_ADDRESS>      D:\SCOOP\apps\Steam\current\GameOverlayUI.exe
<IP_ADDRESS>      <IP_ADDRESS>      D:\SCOOP\apps\Steam\current\Steam.exe
<IP_ADDRESS>      <IP_ADDRESS>      D:\SCOOP\apps\Steam\nightly-20181111\bin\cef\cef.win7\steamwebhelper.exe
<IP_ADDRESS>      <IP_ADDRESS>      D:\SCOOP\apps\Steam\nightly-20181111\bin\cef\cef.win7\steamwebhelper.exe
<IP_ADDRESS>      <IP_ADDRESS>      D:\SCOOP\apps\Steam\nightly-20181111\bin\cef\cef.win7\steamwebhelper.exe
<IP_ADDRESS>      <IP_ADDRESS>      D:\SCOOP\apps\Steam\nightly-20181111\bin\cef\cef.win7\steamwebhelper.exe
<IP_ADDRESS>      <IP_ADDRESS>      D:\SCOOP\apps\Steam\nightly-20181111\bin\cef\cef.win7\steamwebhelper.exe
<IP_ADDRESS>      <IP_ADDRESS>      D:\SCOOP\apps\Steam\nightly-20181111\bin\cef\cef.win7\steamwebhelper.exe
                                  D:\SCOOP\apps\Steam\current\steamapps\common\God Awe-full Clicker\UnityCrashHandle...

PS C:\Windows\system32> (Get-Process -FileVersionInfo -ErrorAction SilentlyContinue | ? { $_.FileName -like 'D:\scoop\ap
ps\Steam*'}).Count
10

Good idea 👍
It's a bit faster when the -FileVersionInfo isn't used. We don't need the file version for this. The Path property is enough.
Warning the user and do nothing would be the best way. This should also be implemented in the update process to skip running programs.
(Get-Process | ? { $_.Path -like 'D:\scoop\apps\*'} )
(Get-Process | ? { $_.Path -like 'D:\scoop\apps\<app>\*'} )

λ (Get-Process | ? { $_.Path -like 'D:\scoop\apps\*'} )

Handles  NPM(K)    PM(K)      WS(K)     CPU(s)     Id  SI ProcessName
-------  ------    -----      -----     ------     --  -- -----------
   2238     176   322672     451496     746,63   3832   8 firefox
   1654     198   868320     820768     701,23  17776   8 firefox
   1506     150   454816     436532     107,84  19904   8 firefox
    823      33   153628     170696     387,61  20420   8 firefox
    809     138   390228     475888      79,34  27380   8 firefox
   1524     166   481256     425892     300,42  30668   8 firefox
   1544     165   553188     494080     229,72  30920   8 firefox
    216      15     4828      83404       0,16  32460   8 firefox
    448      25    53392      58904       1,31  34272   8 QuickLook
    126       9     1320       6180       0,03  18816   8 QuickLook.WoW64HookHelper
    355      29    44316      68388       0,92   7316   8 Wavebox
    328      32    56140      82212      28,56  13692   8 Wavebox
    551      49    87400     117192       2,64  17148   8 Wavebox
    361      53   154552     172872      28,86  20368   8 Wavebox
   1646      83   128120     166200      74,45  20576   8 Wavebox
    738      43   369040     146876      21,42  20764   8 Wavebox
    305      24    26568      35984       0,11  22416   8 Wavebox
    370      75   252996     286688      52,16  24092   8 Wavebox
    357      43    88072     117712      32,67  34092   8 Wavebox
    211      14     6776      12616       0,03  34792   8 Wavebox

λ (Get-Process | ? { $_.Path -like 'D:\scoop\apps\wavebox\*'} )

Handles  NPM(K)    PM(K)      WS(K)     CPU(s)     Id  SI ProcessName
-------  ------    -----      -----     ------     --  -- -----------
    355      29    44316      68400       0,92   7316   8 Wavebox
    325      32    56116      82576      28,70  13692   8 Wavebox
    551      49    87400     117428       2,64  17148   8 Wavebox
    361      53   154552     172944      28,86  20368   8 Wavebox
   1661      85   128088     165180      74,70  20576   8 Wavebox
    738      43   369040     146876      21,42  20764   8 Wavebox
    302      24    26144      36320       0,11  22416   8 Wavebox
    379      76   246384     281988      52,16  24092   8 Wavebox
    359      43    87176     117444      32,81  34092   8 Wavebox
    211      14     6776      12616       0,03  34792   8 Wavebox

Fixed as soon as #3608 arrive in master
Proper implementation will follow in #2952

Implemented in https://github.com/lukesampson/scoop/pull/3608


[Feature Request]: Dark Mode for Improved Accessibility and User Comfort.

Is there an existing issue for this?

[X] I have searched the existing issues

Feature Description
As a GSSoC '24 extended contributor, I propose adding a dark mode to enhance user experience. Dark mode reduces eye strain in low-light environments and offers a modern user interface feature. I would like to contribute to implementing this feature.
Use Case
Adding a dark mode would enhance the user experience by providing an option for users to switch to a visually comfortable interface, particularly in low-light environments. It reduces eye strain, conserves device battery life on OLED screens, and caters to user preferences for aesthetic customization. Additionally, having this option would align the platform with modern web design trends, making it more user-friendly and accessible to a wider audience, which can ultimately improve engagement with the project.
Benefits
No response
Add ScreenShots
No response
Priority
Medium
Record

[X] I have read the Contributing Guidelines
[X] I'm a GSSOC'24 contributor
[X] I have starred the repository

Could you assign this to me?

Check issue #127 ..it is already raised by others..


Sorting models

Hello.
First, thank you for the development and maintenance of the module, I've been using it for some quite time and like it a lot.
This is more a question than an issue:
Is there any option to sort the list of models? The same way you can with the list of tags, and endpoints.
If not, would a PR adding this be welcomed?

I think that would be welcome feature. Any PR must be backed by corresponding unit test.

I'll work on it, then. My guess is that I'll have some time for this between the 7 to the 14th of January in the worst case.

no rush. let me know if you need help.

Please reopen was you get the feedback.

Hi, is this actually supported? If not, I am happy to help with a PR but would be great to have an idea of where to start.
Thanks :)

@ianaya89  never got a response back if this was better suited for ui to sort.

Hello.
My apologies.
I tried asking on the repo of the ui at the time I opened the issue, but never got a response on their side.
Then, pandemic happened and I completely forgot about this.


Memory store is not memory efficient

Here's some stats on real-world usage of Nativelink:

Since rust is not a garbage collected language it is not allowed to pack memory, so we need to deal with memory fragmentation ourselves. Normally programs don't need to think much about this but in this case we do.
Details on how memory works:
Linux allocates memory in pages, often 4k. The memory manager will pack the memory for you by slotting the returned points from malloc usually from a pool of already allocated pages. This allows code to request (lets say) 100 bytes of data and that 100 bytes may share a page with other pointers in memory.
The down side is that small allocations that live for a long time might end up keeping the entire page (4k) allocated by the kernel. It is rare for an allocator to recycle memory pages that were full at one point; this means if a page was ever full, it will release the page back to the kernel once no data is allocated in the page instead of keep trying to pack new allocations into it.
The problem
Looking at the graph above, you can see that roughly 50% of all of the CAS is ~128bytes or less. This means that the vast majority of items are super tiny and only use a little bit of memory. Since memory store frequently evicts items that have not been used recently from the store, it means that it's very common for an item to be put into it and shortly after to be evicted because the object is rarely used.
So, if a 4k page was allocated, then 32 x 128byte objects were loaded into it then an hour later 31 of the items were evicted, but 1 item is used very frequently, it means that we are now using 4k of memory for 128 bytes of data. Multiply this out and you can see how inefficient this can become.
Possible solutions

A very simple solution to this problem is to simply have a timer or trigger that runs every once in a while and copies some of the data from the top of the LRU-Map (least recently used map) to a new place in memory then frees the old pointer. This is inefficient, but is extremely simple and will have very negligible impact on performance.
We could try to find a smarter Allocator that can handle this case better. This would be great if there's an allocator we can ask how compact our memory is or how packed our memory page is from a given pointer.
Similar to 1, We could store references to the data in a sorted order by age of insertion in addition to age of access (LRU), then periodically make full copies of data that has lived a long time in the LRU.

Some things to keep in mind
Lots of items are very small, if 50% are 128bytes and a sha256/blake3 is 40 byte, this means just copying the hash + a pointer is about +37% for the lower 50%.

This one seems overly complex. The design burden and maintenance overhead of the memory store feels high. When users hit this bottleneck today, it seems it might be more efficient and safer to use a caching mechanism from a database.

The first proposed solution is only ~30 lines of code. It is extremely simple and nearly no down side.

SGTM.

Merged with #289


Order list of evm chains in graph init

make ethereum-mainnet the first and then alphabetically order rest of the chains.
Discussed in https://github.com/graphprotocol/graph-tooling/discussions/1616

Can I pick this issue if it isn't being worked on?

@joshuanazareth97 go for it! Feel free to ping me here for any questions navigating the codebase.

@joshuanazareth97 Fork the repository to your own version and push your branch to your own fork. You can then open a PR here that ask to merge your's branch fork into this repository.

@joshuanazareth97 Here some useful links if you never done this:

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/fork-a-repo
https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request-from-a-fork

@saihaj @maoueh Have done as requested above and created the PR, do let me know if anything is missed


Issue on page /configure.html

Hi,
great template. I'm trying out the various options, but I get the impression that extra_sidebar does not seem to do anything.
cheers,
Maarten

Hey @maartenbreddels 👋 - ah I think this one is a mismatch between the docs and master...it has been a little while since we released. Can you try with master and we'll get a new release soon? (need to wait for the pydata theme to make a release first)

Hi ✋ . Yeah, I tried master and also that didn't do much. I don't need the feature, but the 'report issue' was so easy to click 👍

dang it! what did you try? could you share the conf.py config that didn't work?

yep - was a docs bug. Use extra_navbar instead of extra_sidebar and see if that works


After deleting a report from a folder, the remaining reports are not shown

Do the following in the Test tab to see the error:

Have a single report that exists in the root folder.
Copy the report to a new foler "My folder".
Select folder "My folder".
Select the report in that folder.
Press the delete button.
In the confirmation dialog, the list of reports being deleted should have length one. This works fine.
Press "Yes".
The folder "My folder" is gone and the root folder is selected.
See the error: no reports are shown anymore.
Press refresh.
See another error: still no reports are shown.
Press the refresh button of the browser.
Ok, the report that I expected is back.

I dont see how this is a problem, in your steps you describe having only one report in the test tab. You move that report to its own folder, after which you delete the report. This causes the folder to be removed as well.
After that you describe that refreshing the test tab does not show any reports, which makes sense because you just deleted the only report that was in the test tab.
Could you clarify what the problem is exactly?


Bug: 在fedora 40 操作系统下，AppImage安装完点击无法启动，没有响应

Chat2DB Version
Chat2DB Pro
Describe the bug
fedora 40 操作系统下，AppImage安装完点击无法启动，没有响应

Did you install AppImage Manager on your system? Could you provide a screenshot when you right click on the image.

Did you install AppImage Manager on your system? Could you provide a screenshot when you right click on the image.

I use Gear Lever to install APPImage, can install successful, but nothing happens when clicking the chat2DB icon

Could you provide the whole client log? Refer below screenshot to find log.
By the way, do you have the community version installed too? If you have community version installed, then it might be a known issue. It will be fixed soon.


Vulnerability roundup 85: ameba-0.12.1: 1 advisory

search, files

[ ] CVE-2014-3902 (nixos-unstable)

Scanned versions: nixos-unstable: 22c98819ccd. May contain false positives.

Dup of #92029 - false positive.


Merge build infrastructure changes from rel/2.0.3 to dev

This change brings back the infrastructure changes we made in rel/2.0.3 to the dev branch, along with the bower.json removal.
Changes:

Replaces PackageLineup with the dependencies.props approach to manage package versions. - https://github.com/aspnet/templating/pull/58
Generate the csproj/fsproj files that go into template packages - https://github.com/aspnet/templating/pull/129
Remove the DependencyUpdater and PullRequestSubmitter tasks
Remove bower.json and bowerrc from Microsoft.DotNet.Web.ProjectTemplates - https://github.com/aspnet/templating/pull/55
Set MyGet feeds via MSBuild instead of NuGet.config
Upgrades to the functional tests

Update Selenium - https://github.com/aspnet/templating/pull/136
Make it possible to run tests on macOS/linux
Disable test parallelization

@natemcmaster,
Thanks for having already signed the Contribution License Agreement. Your agreement was validated by .NET Foundation. We will now review your pull request.
Thanks,
.NET Foundation Pull Request Bot

Resolves https://github.com/aspnet/templating/issues/143
Resolves https://github.com/aspnet/templating/issues/131