WebUI 2.0.8 Prototype Suggestion

WebUI 2.0.8 Prototype Suggestion
WebUI is working fine, but if you try to open multiple windows and update the window's content, you will find that the API names need to be clarified. So I'm suggesting new APIs/Rename to make it simpler.
Current
Open a static HTML window, change the content, and close the window.
webui_show(my_window, "MyHTML", webui.browser.any);
webui_refresh(my_window, "MyNewHTML");
webui_close(my_window);

Now let's do the same thing but using HTML files.
const char* url = webui_new_server(my_window, "PATH_TO_ROOT_FOLDER");
webui_open(my_window, url, webui.browser.any); // index.html
webui_open(my_window, "second_page.html", webui.browser.any);
webui_close(my_window);

So, webui_show() shows a window using static HTML code, and webui_open() also shows a window, but for files.
Suggestion
webui_show_file(my_window, "PATH_TO_ROOT_FOLDER", webui.browser.any);
webui_refresh_file(my_window, "second_page.html");
webui_refresh_file(my_window, "/foo/bar.html");
webui_close(my_window);

Options
I suggest to keep old functions webui_new_server() and webui_open(), so If you would like a WebUI server only and open the link in the standard browser to interact with your backend language, then use webui_new_server(). And if you already have a web server let's say Deno, Python.HTTP, or NodeJS... and you would like to use WebUI to open a nice app mode window, then use webui_open().

Implemented in https://github.com/alifcommunity/webui/commit/b7bf6082401302713822259e3ec05704e9364342.


Playwright Installation

Thank you for your contribution to the KodaDot - One Stop Shop for Polkadot NFTs.
👇 __ Let's make a quick check before the contribution.
Thanks for all the support, next step: rewriting old cypress tests
Who tests the tests added from the testers automated tests? 🐛
PR Type

[ ] Bugfix
[x] Feature
[ ] Refactoring

Context

[x] Closes #6598
[ ] Requires deployment <snek/rubick/worker>

Before submitting pull request, please make sure:

[x] My contribution builds clean without any errors or warnings
[x] I've merged recent default branch -- main and I've no conflicts
[x] I've tried to respect high code quality standards
[x] I've didn't break any original functionality

Optional

[ ] I've tested it at </ksm/collection>
[ ] I've tested PR on mobile
[ ] I've written unit tests 🧪
[ ] I've found edge cases

Did your issue had any of the "$" label on it?
@stephenjason89 and @preschian did all the heavy lifting
Community participation

[ ] Are you at KodaDot Discord?

Screenshot 📸

Copilot Summary

🤖 Generated by Copilot at 41de705
This pull request introduces Playwright tests for the NFT gallery project. It adds the @playwright/test package and its configuration, a GitHub Actions workflow to run the tests, and a basic test case for the Create Page. It also updates some dependencies and excludes the Playwright tests from Vitest.

🤖 Generated by Copilot at 41de705

To test the NFT gallery site
We added Playwright tests to write
With @playwright/test
And config file set
We run them on GitHub with workflow.yml

Hello @prury, I believe the problem is with the playwright local webServer
so first we don't need this anymore
playwright.config.js

second
playwright.config.js

We need to point it on our dev server
third to execute the test we need to run our local server
pnpm dev
finally run test ❤
pnpm exec playwright test
npx playwright show-report

cool, next PR, time to migrate and adjust cypress test to the playwright

yes! Thank you so much @preschian

Amazing ❤ ❤

What is current Playwright test coverage?

What is current Playwright test coverage?

basically test if the page exists, can you open issue for me to work on test old cypress tests conversion?


AppRTC turnserver NO TURN SERVER

Hello everyone, I wrote in this topic, but it is closed.
https://github.com/webrtc/apprtc/issues/266
I'm trying to use this app on a VPS, so I'm not using localhost, and I do not even https, this is the address:
http://<IP_ADDRESS>:8083?wstls=false
The collider is working properly, the log shows me the room opened, closed, connections, etc
To start the turnserver I also use the simple command
sudo -v -a -o -n --no-DTLS --no-tls -u hello:hello -r someRealm
Open ports are the 3478 and 3479, so I have included this variable in call.js file
var servers = [{credentials: "hello", urls: [ "turn: <IP_ADDRESS>: 3478? transport = udp", "turn: <IP_ADDRESS>: 3478? transport = tcp"], username: "hello"}, {urls: [ "stun: stun.l.google.com: 19302"]}];
When I try to connect with my smartphone it does not work.
"No TURN server; unlikely That average will traverse networks"
From console
http://pastebin.com/eVwVLD7f

Have you tested your turn server using https://webrtc.github.io/samples/src/content/peerconnection/trickle-ice/ ?

Hello, I tried it, it works, this is the result
0.170	1	relay<PHONE_NUMBER>	udp	<IP_ADDRESS>	53601	2 | 30 | 255
0.175	2	relay<PHONE_NUMBER>	udp	<IP_ADDRESS>	59968	2 | 30 | 254
0.175	Done

Its done.

I figured out the problem, I was naively think that it was enough to replace the server array in calls.js but in fact the requestIceServers function in utils.js fails causing the TURN logic to fail. I've updated the instructions at https://github.com/webrtc/apprtc/blob/master/README.md#deployment.
Please try them out and let me know if it works.

So i have to delete the servers array in calls.js? I dont think so,  If I delete, nothing changes the problem persists.
In calls.js:
 var servers = [{credentials:"hello", urls:["turn:<IP_ADDRESS>:3478?transport=udp", "turn:<IP_ADDRESS>:3478?transport=tcp"], username:"hello"}, {urls:["stun:stun.l.google.com:19302"]}];
In utilis.js add:
function requestIceServers (iceServerRequestUrl, iceTransports) {
   return new Promise (function (resolve, reject) {
     var servers = [{
         credentials: "hello",
         username: "hello"
         urls: [
           "Turn:<PHONE_NUMBER>9: 3478? Transport = udp",
           "Turn:<PHONE_NUMBER>9: 3478? Transport = tdp"
         ]
     }];
     resolve (servers);
   });
}

You wrote? "Transport = TDP" and not TCP, correct?
However comes a popup that says the following error:
Can not create RTCPeerConnection: RTCPeerConnection constructor passed invalid RTCConfiguration - missing credential: turn:<PHONE_NUMBER>9: 3478? Transport = udp
Missing credential? its strange
Here the complete log:
http://pastebin.com/MtwqJZf0

Sorry for the typo, tdp should obviously be tcp.
The instructions assume no other changes, hence you can restore call.js to the original.
Also to clarify, you should replace the requestIceServers function already there.

Ok i've restored constants.py to original and changed only WSS_INSTANCE_HOST_KEY
Collider working good.
Restored calls.js to original from the git
Restored utils.js and replaced function to:
function requestIceServers (iceServerRequestUrl, iceTransports) {
   return new Promise (function (resolve, reject) {
     var servers = [{
         credentials: "hello",
         username: "hello"
         urls: [
           "Turn:<IP_ADDRESS>:3478?Transport=udp",
           "Turn:<IP_ADDRESS>:3478?Transport=tdp"
         ]
     }];
     resolve (servers);
   });
}

Tested with localhost too
Same problem as before:
Can not create RTCPeerConnection: RTCPeerConnection constructor passed invalid RTCConfiguration - missing credential:turn:51254115249:3478?Transport=udp
Log:
http://pastebin.com/7Z1jCPGd
Ofc credentials are corrects
https://webrtc.github.io/samples/src/content/peerconnection/trickle-ice/
Still return Done
I'm trying to figure out the problem

credentials should be credential.

It works,
There are errors, thank you.
There is one thing I do not understand,
I can connect from my smartphone ANDROID disabling the wireless and using 3G and 4G connections,
But if I try to use an iphone, it does not work, there is a strange, gray circle at the center of the screen.

If i remove
  <video id="local-video" autoplay muted></video>
from index_template.html,the circle disappear, but of course does not resolve the problem.
it is as if the app does not communicate with the iphone, when I click on Join the collider log does not report anything, no connection and the screen is gray

Chrome on iOS does not have WebRTC  hence you cannot use it, details to why can be found in this post. You need to build the AppRTCDemo iOS (there is one for Android as well) app part of the webrtc.org code base. Instructions here.
As an alternative you could try http://www.openwebrtc.org/bowser/ and see if it works with the web version of appr.tc, I've not tried it myself.

I must ask you the last thing,
The problem is not only chrome, even using safari and firefox can not connect to appr.tc using the iphone.
I would understand if there is a way to connect directly to the web site, without using the app brower.
If I implement apprtc on my website, I can connect only using Android smartphone?
The only way I think is to implement AppRTCDemo-iOS in my APP and connect it to my server on VPS, thats right?

Safari is not a WebRTC capable browser. Please to https://webrtc.org/ to see the officially supported WebRTC browsers.

I know about safari , but also firefox does not work using the iphone

Yes, no browser with the exception of Bowser has WebRTC since they are all just a wrapper around the built in WebKit engine used by safari due to Apple policy. This is outlined in the post in comment https://github.com/webrtc/apprtc/issues/341#issuecomment-237495234 I added yesterday..


[Fix](Top-N opt) evicting quering rowsets in prior to correct use_cou…

…nt (#102)
This addresses the scenario where a rowset cannot be removed.
Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run buildall

run buildall


Attempt 2  of merge of Webpage installer & latest database branch

Attempt to the merge the webinstaller with the sql lite db made by @karltrowbridge1

Closing pull request, changes were made on pre-playlist-db this will be the NEW branch for the webinstaller & db to be integrated into


[Icon Request] Add (meson) wrap

[X] I'm sure this issue is not a duplicate.

[X] I'm submitting an icon request:

Type: filename
Icon Name: meson/meson-wrap
Sample original Icon:
Extensions: subprojects/*.wrap
Language ids: ``

Language Extensions: https://marketplace.visualstudio.com/items?itemName=asabil.meson

More info: http://mesonbuild.com/Wrap-dependency-system-manual.html

Should probably just be an alias for meson

Not sure that the wrap file extension is Meson specific.

Possibly not but for meson they are always located in a subprojects folder so perhaps that's specific enough?

We cannot bind the parent folder name. vscode has no such capability.

That's unfortunate
A quick search on the various "extension info" websites only seems to pull up some obsure imaging software for *.wrap


Sourcegenerator first pull request

Based on fullstackhero  version of  june 13th

Good Job ! Do you write some "Get Started"? :-)


Support network isolation between distinct YARN clusters using overlay networks

Enable creation of a overlay networks per tenant YARN cluster using a virtual switch (like Open vSwitch)
Connect different docker containers belonging to a cluster to the same overlay network.

Really interested in this! I was going to start adding docker support into the nodemanager in the next couple days (basic to start, just provide dockerfile and options), I've already been in TaskFactory lately so that shouldn't be bad.  If you've got an idea of what this would look like can you post to the mailing list?

Sure, I'll post what I have been trying, later today or early tomorrow.
Regards
Swapnil
On Thu, May 7, 2015 at 9:32 AM, Darin<EMAIL_ADDRESS>wrote:

Really interested in this! I was going to start adding docker support into
the nodemanager in the next couple days (basic to start, just provide
dockerfile and options), I've already been in TaskFactory lately so that
shouldn't be bad. If you've got an idea of what this would look like can
you post to the mailing list?
—
Reply to this email directly or view it on GitHub
https://github.com/mesos/myriad/issues/96#issuecomment-99930873.

Hi All,
Sorry for the delay. Here is a first draft.
Regards
Swapnil
On Thu, May 7, 2015 at 10:23 AM, Swapnil Daingade<EMAIL_ADDRESS>wrote:

Sure, I'll post what I have been trying, later today or early tomorrow.
Regards
Swapnil
On Thu, May 7, 2015 at 9:32 AM, Darin<EMAIL_ADDRESS>wrote:

Really interested in this! I was going to start adding docker support
into the nodemanager in the next couple days (basic to start, just provide
dockerfile and options), I've already been in TaskFactory lately so that
shouldn't be bad. If you've got an idea of what this would look like can
you post to the mailing list?
—
Reply to this email directly or view it on GitHub
https://github.com/mesos/myriad/issues/96#issuecomment-99930873.

For some reason the attachment did not go through. Sharing using google
docs.
https://docs.google.com/document/d/1uV2V0cSTngVfWs-5pYm2b9gOCYF4WSNkyzj2dm3bRnw/pub
Regards
Swapnil
On Fri, May 8, 2015 at 4:38 PM, Swapnil Daingade<EMAIL_ADDRESS>wrote:

Hi All,
Sorry for the delay. Here is a first draft.
Regards
Swapnil
On Thu, May 7, 2015 at 10:23 AM, Swapnil Daingade<EMAIL_ADDRESS>wrote:

Sure, I'll post what I have been trying, later today or early tomorrow.
Regards
Swapnil
On Thu, May 7, 2015 at 9:32 AM, Darin<EMAIL_ADDRESS>wrote:

Really interested in this! I was going to start adding docker support
into the nodemanager in the next couple days (basic to start, just
provide
dockerfile and options), I've already been in TaskFactory lately so that
shouldn't be bad. If you've got an idea of what this would look like can
you post to the mailing list?
—
Reply to this email directly or view it on GitHub
https://github.com/mesos/myriad/issues/96#issuecomment-99930873.

—
Reply to this email directly or view it on GitHub
https://github.com/mesos/myriad/issues/96#issuecomment-100394455.

IMHO overlay networks to isolate network between Myriad frameworks is not a good idea. This causes a considerable amount of network performance drop that it won't actually be used in production. One of the reasons we totally abandoned OpenStack was excessive use of overlay techs in Neutron. I myself would gladly go through the burden of manually setting NAT for each Myriad deployment as long as native network throughput is guaranteed.

I understand there could be performance concern and not all users would be interested in having complete network isolation between Myriad clusters. Perhaps this could be made configurable with out of box default not using overlay networks.
I believe the NATing approach is very similar to the Bridge Mode that docker currently supports, but it would be great if you could elaborate more.

Swapnil, how do you propose possible interframework communication/overlay network say hdfs/Cassandra/HBase etc?
Also, worried about the performance hit, but if it works I've got applications.

So the way socketplane seems to be doing it is that every time we create a
network segment we do so using an id (e.g. blue). For every id there is a
corresponding vlan tag reserved for that network segment.
For e.g. socketplane network create blue
http://www.google.com/url?q=http%3A%2F%2F<IP_ADDRESS>%2F16&sa=D&sntz=1&usg=AFQjCNGCK60U8_Dh4wLJBZVTmS2ducR2SQ
<IP_ADDRESS>/16
http://www.google.com/url?q=http%3A%2F%2F<IP_ADDRESS>%2F16&sa=D&sntz=1&usg=AFQjCNGCK60U8_Dh4wLJBZVTmS2ducR2SQ
{
    "gateway": "<IP_ADDRESS>",

    *"id": "blue",*

    "subnet": "<IP_ADDRESS>/16

http://www.google.com/url?q=http%3A%2F%2F<IP_ADDRESS>%2F16&sa=D&sntz=1&usg=AFQjCNGCK60U8_Dh4wLJBZVTmS2ducR2SQ
",
    *"vlan": 2*

}
vlan tag 2 corresponds id blue.
Any docker container that is launched using that id will create a port on
the local OVS switch. The port will have the corresponding vlan tag.
The container will be attached to this newly created port.
e.g. socketplane run -n blue -itd centos:centos7 /bin/bash (Notice the -n
blue option).
The socketplane script is just a wrapper around the docker script.
For ovs it will translate into a command like.
ovs-vsctl add-port docker0-ovs  port1  tag=2 -- set Interface port1
type=internal
where docker0-ovs is the name of the ovs switch on the host and port1 is
the port being newly added.
Any docker container launched (as part of any framework task) with the same
id will be on the same network segment.
So you can have 2 or more frameworks running as part of the same network
segment as long as they launch their containers
using the same id.
socketplane run
-n blue ...
On Wed, May 20, 2015 at 6:52 PM, Darin<EMAIL_ADDRESS>wrote:

Swapnil, how do you propose possible interframework communication/overlay
network say hdfs/Cassandra/HBase etc?
Also, worried about the performance hit, but if it works I've got
applications.
—
Reply to this email directly or view it on GitHub
https://github.com/mesos/myriad/issues/96#issuecomment-104092200.

Docker enhancing support for multi host networking http://blog.docker.com/2015/06/networking-receives-an-upgrade/


wish to provide ios Objective-C custom AppDelegate sample code

Is your feature request related to a problem? Please describe.
wish to provide ios Objective-C custom AppDelegate sample code
Describe the solution you'd like
I don't know much about ios development，wish to provide ios Objective-C custom AppDelegate sample code.
thank you!

I've found a way
1.Create a Swift file that imports app_links and encapsulates the relevant methods
`
//  AppLinksHelper.swift
import Foundation
import app_links
@objcMembers class AppLinksHelper: NSObject{
@objc static public func needHandleLink(launchOptions: [AnyHashable : Any]?) -> Bool{
if let url = AppLinks.shared.getLink(launchOptions: launchOptions) {
// We have a link, propagate it to your Flutter app
AppLinks.shared.handleLink(url: url)
return true // Returning true will stop the propagation to other packages
}
return false
}
}
2.In AppDelegate.m, import "Runner-Swift.h" and use the methods
// AppDelegate.m
#import "AppDelegate.h"
#import "GeneratedPluginRegistrant.h"
#import <AppTrackingTransparency/AppTrackingTransparency.h>
#import "Runner-Swift.h"

(BOOL)application:(UIApplication *)application
didFinishLaunchingWithOptions:(NSDictionary *)launchOptions {
[GeneratedPluginRegistrant registerWithRegistry:self];
self.launchOptions = launchOptions;
BOOL isNeedHandle = [AppLinksHelper needHandleLinkWithLaunchOptions:launchOptions];
if(isNeedHandle){
return YES;
}
// Override point for customization after application launch.
return [super application:application didFinishLaunchingWithOptions:launchOptions];
}
`

Closing this since you found how to call this from objC langage.
I don't know this langage, so I wouldn't have been of any help.


about shutdown

hi @leewyang
we used tfos running well. thanks for contribution.
after inspecting the code, some points puzzled me.
L160 each worker/executor records its executor_id in file on local working directory.
nodeRDD = sc.parallelize(range(num_executors), num_executors)
using nodeRDD to run the tf cluster.
while shutdown
L497
here start
workerRDD = self.sc.parallelize(range(workers), workers)

how does it make sure that the workerRDD will read file in the same directory as nodeRDD? each partition in workerRDD will read its own local executor_id, but it might be a new executor which is different from all executors of nodeRDD, and no executor_id file exist.
shutdown the ps is reasonable, however, since the worker will finished no matter in mode.SPARK or mode.TENSORFLOW, why need to shutdown the workers?
could ps nodes run as one thread in the executor? right now, no matter in what mode, the ps run in a subprocess.
thanks

A nodeRDD represents all the nodes in a TF cluster, whereas workerRDD only represents the workers.  The reason we do this is because the PS nodes usually just run forever in TF.  So in our design, they would continue to "occupy" a Spark task slot, so the remaining slots would be "completed" workers.  On any given worker, the Spark executor should have a consistent "working directory".  Again, note that we require one executor == one task == one TF worker.
This is mostly due to the InputMode.SPARK.  In that mode, it hard to tell if the end of a partition also signifies the end of the entire dataset, so we needed to send an EOF signal to all the input queues to tell the TF process that there's no more data arriving.
It might be possible to just run the PS node as a thread in the executor's python worker.  My main motivation for using a subprocess was to try to avoid any issues with the global interpreter lock (between Spark and TensorFlow "processes").

Note: I'm still not absolutely happy with the shutdown sequence, so I'd like to revisit it someday (when time allows).  The main challenge is shutting down the cluster cleanly while also handling error conditions and retries.


Update build image to Go version 1.8

Just did an update based on our discussion.  PTAL.

Another update per feedback from zombiezen@ regarding checksum validation.


🍒 @preconcurrency import downgrades globals concurrency errors to warnings

Explanation: completes SE-0412 implementation @preconcurrency import specification
Scope: the changes are guarded behind an upcoming feature flag
Issue: none
Risk: low risk given that functionality is guarded behind an upcoming feature flag
Testing: lit.py tests included
Reviewer: @hborla
includes PRs:
https://github.com/apple/swift/pull/70421
https://github.com/apple/swift/pull/70725

@swift-ci please test

Thanks! We should enable GlobalConcurrency when -strict-concurrency=complete is set as well (not the CompleteConcurrency upcoming feature flag that exists on main; I already regret adding yet another way to enable complete concurrency checking).
Can you please also link the main branch PR(s) in the description?

@swift-ci please test

@swift-ci please test


Minimap Button "Clear Selected" is Not Working

After selecting an active set from the Minimap Button, I am unable to change the selection. It says the middle mouse button should clear the selection, but it does not. The only way I can change the selection is by reloading.

Couldn't reproduce


fabricId is hard coded as 0 in ChipController

Problem
fabricId is hard coded as 0 in ChipController. Not possible for a commissioner to designate its own fabricId.
Proposed Solution
a way to vend fabricId.

We extract the fabric ID from the NOC provided at controller init time now.


Catalog: maintenance

remove obsolete code
minor refactorings
update dependencies

rtm

idk why but npm i gives me a new package-lock.json

@akarve could you please shoe me npm ioutput and package-lock.json diff?


⚠ Bofrost.de has degraded performance

In 4977310, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 1177 ms

Resolved: Bofrost.de performance has improved in ac444b0 after 3 minutes.


Update project status

The readme states:

The work on depsolver is temporarily handed over to the enstaller project (in enstaller.new_solver), to ease the integration with our needs at Enthought. Once the implementation will be finished, we will extract back the solver into depsolver. We expect a fully functional solver for end of Q1 2015.

May you please update this to reflect the current status of the project? AFAICS, there was no merge from the enstaller project, yet.

@noraderam ?

@fbender the plan changed quite a bit. To make a long story short:

currently, the solver lives in https://github.com/enthought/sat-solver
that code is almost self-contained, and does not depend on any enthought-specific package manager
it may be renamed into the new depsolver

Cool, thanks for the update! Would you mind updating at least the description of the project (and maybe the README) to point to the enthought/sat-solver repo, then?


chore(npm): sync package.json from npm-package-template

engines, version, release scripts for semantic-release, mocha, jsdoc and istanbul

[Current coverage][cc-pull] is 82.89%

Merging [#35][cc-pull] into [master][cc-base-branch] will increase coverage by +0.12%

2 files (not in diff) in lib/util were modified. more

Hits +4

5 files (not in diff) in lib/recordCheck were modified. more

Hits +5

3 files (not in diff) in lib were modified. more

Misses +2
Hits +6

@@           master        #35   diff @@
========================================
  Files          11         11
  Lines         579        596    +17
  Methods         0         60    +60
  Branches      181        181
========================================
+ Hits          479        494    +15
- Misses        100        102     +2
  Partials        0          0

Powered by Codecov. Last updated by [d4841da...19553e3][cc-compare]
[cc-base-branch]: https://codecov.io/gh/Kronos-Integration/kronos-interceptor-object-data-processor-row/branch/master?src=pr
[cc-compare]: https://codecov.io/gh/Kronos-Integration/kronos-interceptor-object-data-processor-row/compare/d4841da1073524f7131e7ca35669e634de0810a6...19553e3d1c2e8af2ffb390340da850d41159b542
[cc-pull]: https://codecov.io/gh/Kronos-Integration/kronos-interceptor-object-data-processor-row/pull/35?src=pr


Rollback - remove last version

Hello, i have recently learned about this amazing program and testing it successfully for storing my mysqldumps. I would like to ask if it is possible to rollback an archive, just like zpaq seems to do. I accidentally added some files for the last version and would like to remove them and continue as if that did not happen.

AFAIK zpaq does NOT allow to remove.
With zpaqfranz you can use the command crop
C:\zpaqfranz>zpaqfranz h crop
zpaqfranz v60.6k-JIT-GUI-L,HW SHA1/2,SFX64 v55.1,(2024-08-11)
CMD   crop        Discard latest version(s)
<>:               Delete queued versions from the archive
                  up to the specified position or -until
                  By default DRY RUN (only test)
<>: -kill         Do a 'wet' (effective) run
<>: -to tiny.zpaq Reduce to tiny.zpaq (safer)
<>: -until X      Discard every versions >X
+ : -maxsize  X   Manually cut at X (RISKY)
+ : -force        Crop in-place (no backup: VERY RISKY!)
    Examples:
Reduce file (dry run, just infos):   crop z:\1.zpaq
Reduce up to version 100:            crop z:\1.zpaq -to d:\2.zpaq -until 100 -kill
Reduce to first 100.000:             crop z:\1.zpaq -to d:\2.zpaq -maxsize 100k -kill
Crop in place (NO BACKUP! RISKY!):   crop z:\1.zpaq -until 2 -kill -force

BTW zpaqfranz can get directly piped mysqldump (zpaq cannot) with -stdin
You can even pre-order the block for restoring to mysql (-stdout), but this will makes way bigger archives

This is an example
C:\zpaqfranz>zpaqfranz crop versioni.zpaq
zpaqfranz v60.6k-JIT-GUI-L,HW SHA1/2,SFX64 v55.1,(2024-08-11)
franz:-hw

versioni.zpaq:
4 versions, 20 files, 23.669 bytes (23.11 KB)
---------------------------------------------------------------------------
<  Ver  > <  date  > < time > <    version size    > < Offset (w/out encr)>
V00000001 2022-10-03 15:05:37 [                 948] @                  948
V00000002 2022-10-03 15:06:00 [               1.192] @                2.140
V00000003 2022-10-03 15:06:19 [                 897] @                3.037
V00000004 2022-10-09 15:12:25 [              20.632] @               23.669
no -kill, this is just a dry run

We want to discard the version 4
The safest way is
C:\zpaqfranz>zpaqfranz crop versioni.zpaq -to z:\cropped.zpaq -until 3 -kill
zpaqfranz v60.6k-JIT-GUI-L,HW SHA1/2,SFX64 v55.1,(2024-08-11)
franz:-to                   <<z:/cropped.zpaq>>
franz:-hw -kill

versioni.zpaq -until 3:
3 versions, 8 files, 3.037 bytes (2.97 KB)
---------------------------------------------------------------------------
<  Ver  > <  date  > < time > <    version size    > < Offset (w/out encr)>
V00000001 2022-10-03 15:05:37 [                 948] @                  948
V00000002 2022-10-03 15:06:00 [               1.192] @                2.140
V00000003 2022-10-03 15:06:19 [                 897] @                3.037
Writing data...
DONE on z:/cropped.zpaq (3.037)
0.079 seconds (00:00:00) (all OK)

And then a good test, just to be sure
C:\zpaqfranz>zpaqfranz t z:\cropped.zpaq -all
zpaqfranz v60.6k-JIT-GUI-L,HW SHA1/2,SFX64 v55.1,(2024-08-11)
franz:-all                                      4
franz:-hw

z:/cropped.zpaq:
3 versions, 8 files, 3.037 bytes (2.97 KB)
To be checked 27 in 4 files (4 threads)
7.15 stage time       0.02 no error detected (RAM ~4.26 MB), try CRC-32 (if any)
Checking                 4 blocks with CRC-32 (27 not-0 bytes)

CRC-32 time           0.00s
Blocks                  27 (           4)
Zeros                    0 (           0) 0.000000 s
Total                   27 speed 1.687/s (1.65 KB/s)
GOOD            : 00000004 of 00000004 (stored=decompressed)
VERDICT         : OK                   (CRC-32 stored vs decompressed)
0.031 seconds (00:00:00) (all OK)

Thanks for your answer and great examples! i am now cropping my backup and testing it.
about mysqldumps, i would like to save all dumps in a folder named by the date and save a file for each database. can this be done with pipes?

Finally, for mysqldump, you can try something like that
mysqldump -uroot -pthepassword thedatabase |c:\zpaqfranz\zpaqfranz a z:\thedump.zpaq thebackup.sql -stdin

Inside the archive "z:\thedump.zpaq" the dumps will be stored in thebackup.sql file
Then you can extract using -until something

Therefore you can use a single .zpaq (to store multiple dumps) with pipe
mysqldump -uroot -pthepassword firstdb |c:\zpaqfranz\zpaqfranz a z:\thedump.zpaq first.sql -stdin
mysqldump -uroot -pthepassword seconddb |c:\zpaqfranz\zpaqfranz a z:\thedump.zpaq second.sql -stdin
mysqldump -uroot -pthepassword thirddb |c:\zpaqfranz\zpaqfranz a z:\thedump.zpaq third.sql -stdin

Or use a folder tree
Suppose you make a ugo folder, with 3 folders inside
Z:\>tree ugo
Elenco del percorso delle cartelle per il volume RamDisk
Numero di serie del volume: EEEB-F5AE
Z:\UGO
├───folderdb1
├───folderdb2
└───folderdb3

Then you write something like that
mysqldump -uroot -pthepassword firstdb >z:\ugo\folderdb1\first.sql
mysqldump -uroot -pthepassword seconddb >z:\ugo\folderdb2\second.sql
mysqldump -uroot -pthepassword thirddb >z:\ugo\folderdb3\third.sql

Now you can make 3 different zpaqs with
c:\zpaqfranz\zpaqfranz a z:\thebackups z:\ugo -home
getting THREE different zpaqs

00001: 0      OK 00:00:00                 1.366 z:/thebackups_folderdb1.zpaq
00002: 0      OK 00:00:00                 1.366 z:/thebackups_folderdb2.zpaq
00003: 0      OK 00:00:00                 1.371 z:/thebackups_folderdb3.zpaq

Keeping different zpaqs (-home option) typically makes sense when working with data from different clients, where when the “foo” client is no longer a client, you just delete the foo.zpaq file and that's it.

On Windows I usually takes separated db dump, and a --all-databases too, just in case of a bare-metal restore
Of course this really depends on how many db do you have
Restoring one-by-one 3 or 5 is a thing, 200 different dbs is just another work
In this case I usually restore --all-database (in an empty virtual machine) then pick whatever I really need
I am really lazy and sometimes I do not like to make a lot of awk-kung-fu to extract lines from huge mysql dumps
Dealing with 1GB dump is easy, with a 1TB not so much
Short version: depends on your requirements

Thanks again for your detailed answers! I will what works better with my workflow.
I guess it isnt possible to make one version with one folder containing many dumps using pipes?

I forgot to specify that calling dump files differently (i.e., with date) is useless if you store them inside a zpaq file
The version already has the date stored in it, so it is easy to go back to it
For example with the i (info) command to quickly identify the version number
Or using PAKKA (which was originally born just for mysql backups)
And the answer is yes you can
(https://github.com/fcorbelli/zpaqfranz/issues/122#issuecomment-2298613241)

Oh great, nice! Thanks again

Please, if you want, do not forget to "star" and/or leave a review on https://sourceforge.net/projects/zpaqfranz/

Hi, 2 questions about -stdin:
mysqldump dbname | zpaqfranz a thedump.zpaq folder/dbname.sql -stdin

the db is saved as dbname.sql inside the archive, instead of folder/dbname.sql
dedupe does not seem to work

please tell me if i should open a new issue

You cannot use a folder name, just a file name. I'll make auto in new release
The deduplicator is active

C:\zpaqfranz>mysqldump -uroot -p1 zarc |c:\zpaqfranz\zpaqfranz a z:\archived dump.sql -stdin
zpaqfranz v60.6q-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2024-08-20)
franz:-hw -stdin
Creating z:/archived.zpaq at offset 0 + 0
Stdin Add 2024-08-21 19:47:07         1                  4 (   4.00  B) 32T (0 dirs)
1 +added, 0 -removed.

0 + (860.875.603 -> 860.875.599 ->  155.885.098) = 155.885.098  @ 138.63 MB/s
Files  added +1
5.938 seconds (00:00:05) (all OK)

C:\zpaqfranz>mysqldump -uroot -p1 zarc |c:\zpaqfranz\zpaqfranz a z:\archived dump.sql -stdin
zpaqfranz v60.6q-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2024-08-20)
franz:-hw -stdin

z:/archived.zpaq:
1 versions, 1 files, 155.885.098 bytes (148.66 MB)
Updating z:/archived.zpaq at offset 155.885.098 + 0
Stdin Add 2024-08-21 19:47:16         1                  4 (   4.00  B) 32T (0 dirs)
1 +added, 0 -removed.

155.885.098 + (860.875.603 -> 289.615 ->  155.959.855) = 155.959.855  @ 232.51 MB/s
Files  updated #1
3.531 seconds (00:00:03) (all OK)

Then do something on the db
C:\zpaqfranz>mysql -uroot -p1 zarc
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 7
Server version: 11.1.2-MariaDB mariadb.org binary distribution

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [zarc]> delete from zarcper limit 1;
Query OK, 1 row affected (0.002 sec)

MariaDB [zarc]> bye
    -> quit
    -> Bye
Ctrl-C -- exit!

C:\zpaqfranz>mysqldump -uroot -p1 zarc |c:\zpaqfranz\zpaqfranz a z:\archived dump.sql -stdin
zpaqfranz v60.6q-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2024-08-20)
franz:-hw -stdin

z:/archived.zpaq:
2 versions, 2 files, 155.959.855 bytes (148.73 MB)
Updating z:/archived.zpaq at offset 155.959.855 + 0
Stdin Add 2024-08-21 19:49:17         1                  4 (   4.00  B) 32T (0 dirs)
1 +added, 0 -removed.

155.959.855 + (860.875.094 -> 5.491.026 ->  156.999.099) = 156.999.099  @ 229.46 MB/s
Files  updated #1
3.593 seconds (00:00:03) (all OK)

Thanks for your answer. I think it would be useful to have a foldername included as then i could group my files using that.
About dedup, my zpaq file has about 342 versions of my mysqldumps (daily over about one year).
V00000338 2024-08-18 07:03:00  +00000039 -00000001 ->           13.739.782
V00000339 2024-08-19 07:03:00  +00000039 -00000001 ->           19.408.517
V00000340 2024-08-20 07:03:00  +00000039 -00000001 ->           15.503.969
V00000341 2024-08-21 07:03:00  +00000040 -00000001 ->           17.508.378
V00000342 2024-08-21 20:29:11  +00000039 -00000001 ->           10.977.342
when i add another day normaly (add a folder of mysqldumps) then it uses 10-20MB.
but when i add for example the large database using -stdin i get
V00000339 2024-08-19 07:03:00  +00000039 -00000001 ->           19.408.517
V00000340 2024-08-20 07:03:00  +00000039 -00000001 ->           15.503.969
V00000341 2024-08-21 07:03:00  +00000040 -00000001 ->           17.508.378
V00000342 2024-08-21 20:29:11  +00000039 -00000001 ->           10.977.342
V00000343 2024-08-21 20:29:12  +00000001 -00000000 ->        5.341.016.752

Windows or Linux for a pre-release test build?

Linux

60_6s.zip
The attached pre-release should (not tested at all)

No restriction on file name (aka: folders) in -stdin
Better deduplication of -stdin (just as good as zpaq's, only faster)

I make no tests, more urgent jobs to complete
Please let me know

Thanks Franco! Just compiled and I am testing right now!

Thanks again Franco! One small fix:
2024-08-22T16_21_43/homeassistant.sql: No such file or directory should not be printed for files coming from stdin.

60-6t.zip
Be careful to check the extracted file

thanks, i think it works perfectly good.
full logs:
[130] $ cat 2024-08-22T16_21_43/homeassistant.sql| /root/zpaqfranz/zpaqfranz a mysqlbackups.zpaq 2024-08-22T16_21_43/homeassistant.sql -comment 2024-08-22T16_21_43 -timestamp 2024-08-22T16_21_43 -stdin
zpaqfranz v60.6t-JIT-L(2024-08-22)
franz:-comment              <<2024-08-22T16_21_43>>
franz: -timestamp change from 2024-08-22 16:21:43 => 2024-08-22 16:21:43
franz:-timestamp              <<2024-08-22 16:21:43>>
Unknown option ignored: 2024-08-22T16_21_43
franz:-comment -stdin
mysqlbackups.zpaq:
342 versions, 13.075 files, 10.851.416.557 bytes (10.11 GB)
Updating mysqlbackups.zpaq at offset 10.851.416.557 + 0
Stdin <<2024-08-22T16_21_43>>
10.851.416.557 + (22.579.990.744 -> 29.542.763 ->  10.859.405.898) = 10.859.405.898  @ 112.87 MB/s
Files  added +1
190.793 seconds (00:03:10) (all OK)
$ zpaqfranz l mysqlbackups.zpaq|tail
2024-08-21 04:22:27       6.151.117   12% + 2024-08-21T07_03_00/information_schema.sql
2024-08-21 04:22:28       3.623.817   14% + 2024-08-21T07_03_00/mysql.sql
2024-08-21 04:22:29       1.200.801   14% + 2024-08-21T07_03_00/mythconverg.sql
2024-08-21 04:22:29         207.394   10% + 2024-08-21T07_03_00/performance_schema.sql
2024-08-21 04:22:30         558.597   12% + 2024-08-21T07_03_00/sys.sql
2024-08-22 21:42:42  22.579.990.740   24% + 2024-08-22T16_21_43/homeassistant.sql
$ zpaqfranz i mysqlbackups.zpaq|tail
V00000335 2024-08-15 07:03:00  +00000039 -00000001 ->           15.201.874
V00000336 2024-08-16 07:03:00  +00000039 -00000001 ->           15.467.466
V00000337 2024-08-17 07:03:00  +00000039 -00000001 ->           15.919.177
V00000338 2024-08-18 07:03:00  +00000039 -00000001 ->           13.739.782
V00000339 2024-08-19 07:03:00  +00000039 -00000001 ->           19.408.517
V00000340 2024-08-20 07:03:00  +00000039 -00000001 ->           15.503.969
V00000341 2024-08-21 07:03:00  +00000040 -00000001 ->           17.508.378
V00000342 2024-08-22 07:03:00  +00000039 -00000001 ->           17.267.099
V00000343 2024-08-22 16:21:43  +00000001 -00000001 ->            7.989.341
5.914 seconds (00:00:05) (all OK)
$ /root/zpaqfranz/zpaqfranz  t mysqlbackups.zpaq
zpaqfranz v60.6t-JIT-L(2024-08-22)
mysqlbackups.zpaq:
343 versions, 13.076 files, 10.859.405.898 bytes (10.11 GB)
To be checked 4.691.578.957.372 in 12.735 files (8 threads)
7.15 stage time    1798.47 no error detected (RAM ~128.52 MB), try CRC-32 (if any)
Checking         6.367.284 blocks with CRC-32 (4.691.578.957.372 not-0 bytes)
Block 05965K          3.84 TB
CRC-32 time         309.87s
Blocks   4.691.578.957.372 (   6.367.284)
Zeros                    0 (           0) 0.000000 s
Total    4.691.578.957.372 speed 15.140.474.900/s (14.10 GB/s)
GOOD            : 00012735 of 00012735 (stored=decompressed)
VERDICT         : OK                   (CRC-32 stored vs decompressed)
2108.343 seconds (00:35:08) (all OK)
Only one question: i am using timestamp like this:
-timestamp 2024-08-22T16_21_43
timestamps are set, but i get:
Unknown option ignored: 2024-08-22T16_21_43

60_6u.zip
You can try this untested pre-release

I think timestamp now works perfectly!
zpaqfranz a /_databases/mysqlbackups.zpaq 2024-08-23T19_46_29/* -comment 2024-08-23T19_46_29 -timestamp 2024-08-23T19_46_29
zpaqfranz v60.6u-JIT-L(2024-08-23)
franz:-comment              <<2024-08-23T19_46_29>>
franz:-timestamp              <<2024-08-23 19:46:29>>
franz:-comment
/_databases/mysqlbackups.zpaq:
281 versions, 465 files, 6.402.773.446 bytes (5.96 GB)
Updating /_databases/mysqlbackups.zpaq at offset 6.402.773.446 + 0
Add 2024-08-23 19:46:29        93     79.025.430.401 (  73.60 GB) 32T (0 dirs)<<2024-08-23T19_46_29>>
93 +added, 0 -removed.
6.402.773.446 + (79.025.430.401 -> 1.121.444.426 ->  6.459.617.189) = 6.459.617.189  @ 198.85 MB/s
Files  added +93
379.005 seconds (00:06:19) (all OK)
question: is there a way to see how much space a file takes inside an archive, like zpaqfranz i does for versions?
zpaqfranz l does not have such an option, it only shows the pre-compressed size

Newer zpaqfranz already shows extimated compression ratio with the l list command for every file
Even Red green Yellow
BTW the -stat switch for i command show a good extimation, albeit slow (for huge archives and many version)

I think it would be quite helpful if the -stat output for i switch would work the same for the l switch.
Also, the l switch only shows the first 10.000 versions, is there a way to list them all?
╰─ zpaqfranz i fileserver_recompress.zpaq
zpaqfranz v60.6u-JIT-L(2024-08-23)
franz:-nocolor
fileserver_recompress.zpaq:
12275 versions, 12.275 files, 10.669.410.908 bytes (9.94 GB)
<  Ver  > <  date  > < time >  < added >     <    bytes added   >
V00000001 2023-07-10 07:03:01  +00000001 -00000001 ->                9.616
V00000002 2023-07-10 07:03:02  +00000001 -00000001 ->                9.633
.
.
.
V00009997 2024-05-20 07:03:37  +00000001 -00000001 ->               13.890
V00009998 2024-05-21 07:03:00  +00000001 -00000001 ->                9.696
V00009999 2024-05-21 07:03:01  +00000001 -00000001 ->                9.593
4.696 seconds (00:00:04) (all OK)
╰─ zpaqfranz i fileserver_recompress.zpaq -range ::50
zpaqfranz v60.6u-JIT-L(2024-08-23)
franz:-range                                 ::50
fileserver_recompress.zpaq:
Incomplete transaction ignored4 (block/s)
12252 versions, 12.252 files, 10.653.011.660 bytes (9.92 GB)
<  Ver  > <  date  > < time >  < added >     <    bytes added   >
4.206 seconds (00:00:04) (all OK)
╰─ zpaqfranz i fileserver_recompress.zpaq -range 10050
zpaqfranz v60.6u-JIT-L(2024-08-23)
franz:-range                                10050
franz:rangefrom (version)                  10.050
franz:rangeto   (version)                  10.050
fileserver_recompress.zpaq:
12255 versions, 12.255 files, 10.669.153.957 bytes (9.94 GB)
<  Ver  > <  date  > < time >  < added >     <    bytes added   >
4.479 seconds (00:00:04) (all OK)
also, about dates:
when i add a file with stdin,i get this warning and the version timestamp is not set to what i want.
Stdin <<2023-07-11T07_03_01/sys.sql>>
Warning: adjusting date from 2023-07-11 07:03:01 to 2023-07-11 07:03:37
1.588.239.586 + (4 -> 63.673 ->  1.588.247.490) = 1.588.247.490  @ 102.00  B/s
Files  added +1
0.039 seconds (00:00:00) (all OK)
2023-07-12T07_03_01
AnnasVideos116.sql.gz
zpaqfranz v60.6u-JIT-L(2024-08-23)
franz:-comment              <<2023-07-12T07_03_01/AnnasVideos116.sql>>
franz:-timestamp              <<2023-07-12 07:03:01>>
franz:-summary                                  1
franz:-comment -nocolor -noeta -stdin
/backups/fileserver_recompress.zpaq:
74 versions, 74 files, 1.588.247.490 bytes (1.48 GB)
╰─ zpaqfranz i fileserver_recompress.zpaq -stat |grep 2023-07-11T07_03_01/sys.sql
V00000074 2023-07-11 07:03:37  +00000001 -00000001              281.562 [               7.904] <<2023-07-11T07_03_01/sys.sql>>
also, the file does not get that timestamp assigned to it, is there a way to set a custom timestamp for a file (not a version)?
╰─ zpaqfranz l fileserver_recompress.zpaq -stat |grep 2023-07-11T07_03_01/sys.sql
2024-08-24 02:10:04         281.562    8% + 2023-07-11T07_03_01/sys.sql

I think it would be quite helpful if the -stat output for i switch would work the same for the l switch

Mmmhhh... This will take a lot of space in output, without a really useful information (it is an EXTIMATION, due to very complex zpaq archiving method)

Also, the l switch only shows the first 10.000 versions, is there a way to list them all?

It is an hardcoded all=4. I can upscale to 5 (99.999)

also, about dates: when i add a file with stdin,i get this warning and the version timestamp is not set to what i want (where does it find the other time?)

You cannot make a transaction IN THE PAST, but only IN THE FUTURE

also, the file does not get that timestamp assigned to it, is there a way to set a custom timestamp for a file (not a version)?

Yes, but... why?

I think it would be quite helpful if the -stat output for i switch would work the same for the l switch

Mmmhhh... This will take a lot of space in output, without a really useful information (it is an EXTIMATION, due to very complex zpaq archiving method)

I just want to know how much space my backups take! I am trying to decide which is the most efficient and closer to the way i work way for me to store my mysqldumps. (will write more about that in a bit)

Also, the l switch only shows the first 10.000 versions, is there a way to list them all?

It is an hardcoded all=4. I can upscale to 5 (99.999)
OK. Things like this should be documented in the h section, and in my opinion, they should be able to be changed by the user
(also a nitpick, i would also like to have an option to trim an archive without writing the captcha)

also, about dates: when i add a file with stdin,i get this warning and the version timestamp is not set to what i want (where does it find the other time?)

You cannot make a transaction IN THE PAST, but only IN THE FUTURE

then what is the use of -timestamp? and in any case it changed the
2023-07-11 07:03:01 to 2023-07-11 07:03:37, where both are in the past.

also, the file does not get that timestamp assigned to it, is there a way to set a custom timestamp for a file (not a version)?

Yes, but... why?

I am storing the whole history of my mysqldumps in one zpaq file and would like to have the original file date stored.

I just want to know how much space my backups take! I am trying to decide which is the most efficient and closer to the way i work way for me to store my mysqldumps. (will write more about that in a bit)

It is just the filesize of the archive

Also, the l switch only shows the first 10.000 versions, is there a way to list them all?
Setting on 8 is enough

int Jidac::info()
{
	///flagcomment=true;
	versioncomment="";
	all=8;
	return enumeratecomments();
}

(also a nitpick, i would also like to have an option to trim an archive without writing the captcha)

Surely NOT
Altering an archive is quite a risky thing to do.
zpaq works only ADDING data to the archive
Trimming is way too dangerous, you can lose all of your data very quickly
If you KNOW what you are doing, you can do anything

also, the file does not get that timestamp assigned to it, is there a way to set a custom timestamp for a file (not a version)?

Past and future is... the timestamp of the very last version is "current". Everything BEFORE is past (not good). Everything after is FUTURE (that's good)
The versions MUST BE monotonically increasing
You cannot have version X timestamp 27 AND version X+1 timestamp 27 (or 26, or 3)
Must be (at least) 27+1

I am storing the whole history of my mysqldumps in one zpaq file and would like to have the original file date stored.
Put the filedate in the filename
Or extract them all, touch them all, insert them all
Altering the timestamp of an already archived file is impossible
If you have the file "pippo.txt" archived with datetime 27, you cannot change to datetime 28
A brand new transaction is needed to write down newer i-blocks
There is NOT an "index" of archived files in zpaq

I just want to know how much space my backups take! I am trying to decide which is the most efficient and closer to the way i work way for me to store my mysqldumps. (will write more about that in a bit)

It is just the filesize of the archive

i mean how much space each file takes :)

Also, the l switch only shows the first 10.000 versions, is there a way to list them all?
Setting on 8 is enough

int Jidac::info()
{
	///flagcomment=true;
	versioncomment="";
	all=8;
	return enumeratecomments();
}

-all gets accepted, but does not change anything:
╰─ zpaqfranz i fileserver_recompress.zpaq -stat -all 8 | head
zpaqfranz v60.6u-JIT-L(2024-08-23)
franz:-all                                      8
franz:-nocolor -stat
fileserver_recompress.zpaq:
12581 versions, 12.581 files, 10.877.950.693 bytes (10.13 GB)
<  Ver  > <  date  > < time >  < added >  <    uncompressed   > <    compressed     >
V00000001 2023-07-10 07:03:01  +00000001 -00000001               72.156 [               9.616] <<2023-07-10T07_03_01/AnnasVideos116.sql>>
...
V00009999 2024-05-21 07:03:01  +00000001 -00000001               72.377 [               9.593] <<2024-05-21T07_03_00/AnnasVideos119.sql>>
4.778 seconds (00:00:04) (all OK)

(also a nitpick, i would also like to have an option to trim an archive without writing the captcha)

Surely NOT Altering an archive is quite a risky thing to do. zpaq works only ADDING data to the archive Trimming is way too dangerous, you can lose all of your data very quickly If you KNOW what you are doing, you can do anything

Yes i know, i just wanted to use if for scripting and do not want to pipe the captcha. Maybe a --force-i-am-sure-i-will-lose-data switch :)

also, the file does not get that timestamp assigned to it, is there a way to set a custom timestamp for a file (not a version)?

Past and future is... the timestamp of the very last version is "current". Everything BEFORE is past (not good). Everything after is FUTURE (that's good) The versions MUST BE monotonically increasing You cannot have version X timestamp 27 AND version X+1 timestamp 27 (or 26, or 3) Must be (at least) 27+1

aah ok i understand now.

I am storing the whole history of my mysqldumps in one zpaq file and would like to have the original file date stored.
Put the filedate in the filename
Or extract them all, touch them all, insert them all
Altering the timestamp of an already archived file is impossible
If you have the file "pippo.txt" archived with datetime 27, you cannot change to datetime 28
A brand new transaction is needed to write down newer i-blocks
There is NOT an "index" of archived files in zpaq

no, i just want to set the creation date for a piped file before it is saved in the archive.

i mean how much space each file takes :)

You can't
Because this information does not exists in zpaq
You have to run something very slow, very lengthy, to take this information

-all gets accepted, but does not change anything:

It is hardcoded to 4. Changing the source code is necessary

no, i just want to set the creation date for a piped file before it is saved in the archive.

It is doable

i mean how much space each file takes :)

You can't Because this information does not exists in zpaq You have to run something very slow, very lengthy, to take this information In previous post I already explained "why". But the short version is: there is NOT I can store an exact number, but this will require a LOT of complex work => not gonna happen

ok i understand

-all gets accepted, but does not change anything:

It is hardcoded to 4. Changing the source code is necessary

Ok will do.

no, i just want to set the creation date for a piped file before it is saved in the archive.

It is doable, but not really a thing

Ok. its just a nice to have thing, not really needed or anything, so no worries.
In any case zpaqfranz is amazing, thank you again for your dedication and excellent work here!

60_6w.zip
In this pre-release you can use -touch
type 1.txt|zpaqfranz a z:\2.zpaq 1.sql -stdin -touch 2024_02_03

It works perfectly!
thanks for changing to all=8 :)
btw crop doesnt work for over 9.999
zpaqfranz crop fileserver_recompress.zpaq -until 12581 -kill -force
..
-until 12.581 too big (> maxversion 9.999) => exit

Another hardcoded 4
	vector<uint64_t> version_position;
	version_position.push_back(0);
	all					=4;
	int64_t wheretotrim	=0;
	int errors			=0;
	int64_t csize=read_archive(NULL,archive.c_str(),&errors,1);

thanks, just recompiled and it works perfectly.

About my mysqldumps.
I backup to one file per database and one file containing all databases via mariabackup --backup --stream=xbstream
I have discovered 2 different ways to use zpaqfranz:

mysqldump all db files to a folder, (named by date, ie 2024-08-24T19_15_51) and then add this folder to zpaqfranz.
this minimizes the number of versions, adding to the archive is pretty quick and i can store everything neatly in one place. The problem with this method is that the dumped databases cannot use compression, so they take tons of space and make a huge amount of writes to the SSD (50-100GB each time). This can be minimized by using a compressed ramdisk, but it still needs quite a bit of RAM.

use pipes
Using -stdin is very fast, does not need large amounts of RAM and in general is pretty neat (thanks again for fixing the minor issues i found!)
The only problem for my use is that every db must be stored as a different version. This quickly adds to > 10.000 versions, so adding a new file takes a long time (that keeps increasing) and i think this will be more and more problematic as i add databases.

Is there a way to pipe multiple files to a single version?
I have no idea about how zpaq works, but maybe start zpaqfranz, create a new version, store the first file via pipe but leave the version "open", then add as many files are needed to the same version, and when finised "close/lock" that version. Or maybe some other way.
If that is possible then it will be a dream come true! Thanks again for your time and for creating this amazing program!

another possible way is to allow multiple files to be added via -stdin, and then use named pipes, or process substitution.

can something like this be made to work? (i havent managed to make anything yet)
zpaqfranz a mysqlbackups.zpaq <(mysqldump db1) <(mysqldump db2)
or
mkfifo db1.sql
mkfifo db2.sql
mysqldump db1 > db1.sql
mysqldump db2 > db2.sql
zpaqfranz a mysqlbackups.zpaq db1.sql db2.sql

Using a giant .zpaq to store everything it is not a good idea, much better to store every database in a different .zpaq
with 2 methods for each db, plus a "grand total" (aka: the doomsday zpaq)

mydumper
This will make different files for different tables.
Much easy to restore for a single table

the mysqldump |

mariabackup --backup

mysqldump --all-databases

Because remember: you cannot DELETE something. Then yours db will live forever.
With different files you can quickly delete the db "myvideoarchive" by removing "myvideoarchive.zpaq"
You can even use a zfs-deduplicated storage for temporary backups, the put inside zpaqfranz. This will greatly reduce the write tear
I do this way for full .vmdk backups

And no, you cannot discriminate stdin into files
Because stdin is not a file, it is just a stream of bytes, one after another
Yes it is called stdin, seems a file, have a file descriptor, a file handle
But this is not a "file" (for example, does not have a length)

BTW you can -freeze a zpaq and start again, automagically, if you really want a single big archive

Using a giant .zpaq to store everything it is not a good idea, much better to store every database in a different .zpaq with 2 methods for each db, plus a "grand total" (aka: the doomsday zpaq)
1. mydumper
   This will make different files for different tables.
   Much easy to restore for a single table

easy to restore tables is interesting, thanks for bringing it to my attention!

2. the mysqldump |

3. mariabackup --backup

4. mysqldump --all-databases

so you make 1 zpaq with 1 and 2 for each db, and one zpaq with 3 and 4 for everything? so for n databases you have n+1 zpaqs?

Because remember: you cannot DELETE something. Then yours db will live forever. With different files you can quickly delete the db "myvideoarchive" by removing "myvideoarchive.zpaq"

In my use case i dont care about deleting a single db. making everything take less space (and have fast backups) is a bigger priority.
with one file for every database i also get the dedup gain when many dbs are the same or similar.

You can even use a zfs-deduplicated storage for temporary backups, the put inside zpaqfranz. This will greatly reduce the write tear I do this way for full .vmdk backups

I havent yet explored the zfs stuff of zpaqfranz, i will take a look when i can.
for my msqldumps, a btrfs fs with zstd:10 mounted on a zram seems to work quite well.

And no, you cannot discriminate stdin into files Because stdin is not a file, it is just a stream of bytes, one after another Yes it is called stdin, seems a file, have a file descriptor, a file handle But this is not a "file" (for example, does not have a length)

maybe multiple stdin then?
something like:
zpaqfranz a myfile.zpaq -filename db1.sql <(mysqldump db1) -filename db2.sql <(mysqldump db2) - stdinmultiple


[TRI-1364] feat: Switch to Sonner toasts #555

Closes #555
✅ Checklist

[x] I have followed every step in the contributing guide
[x] The PR title follows the convention.
[x] I ran and tested the code works

Testing

Created new stories for UIs and toasts

Changelog

Removes react-hot-toast based Toast
Adds Sonner based toast

Screenshots

💯

I've recorded a video of some issues with this PR. As you can see in the video the toasts aren't right-aligned against the bottom of the window. They are also transparent and don't work very well when there are multiple toasts:
https://github.com/triggerdotdev/trigger.dev/assets/534/b8494410-708b-41a7-8dda-b7918b9736be

@ericallam  can you check the latest revision?

Will do! In the meantime can you make a similar screencast as mine so I can visually inspect the changes?

https://github.com/triggerdotdev/trigger.dev/assets/37960163/aacae2e4-915b-4c8c-93af-1378eb87c9d8
Here you go

Pulled this down and looks like there is an issue with the pnpm-lock.yaml file as I'm getting this error when running pnpm install:
Ignoring broken lockfile at /Users/eric/code/triggerdotdev/trigger.dev: Lockfile /Users/eric/code/triggerdotdev/trigger.dev/pnpm-lock.yaml not compatible with current pnpm

Can you make sure you are using<EMAIL_ADDRESS>

Also ran into this same issue in another PR and left a comment just now about how to recover from this:
https://github.com/triggerdotdev/trigger.dev/pull/581#issuecomment-1753869773

I seem to have messed up :(


Feature/add option to return rows as named tuples

With this change, users can optionally request that rows are returned as namedtuples, with column names being the namedtuple field names

@jabadia, ok, it's an interesting idea. I did a quick research about namedtuple.
It appears that creation of namedtuple is significantly slower than creation of both tuple and dict. Access time by index is almost identical to tuple, but access time by column name is much slower compared to dict.
You may find some tests on recent Python versions here: https://gist.github.com/zed/cc530d8d8c2f149aa6670b68c97c0f9c#gistcomment-2583111
Since Exasol is meant to process large volumes of data, I am not sure if namedtuple can be recommended from performance standpoint. Also, mixed access both by index and name might add some confusion to the code.
In my opinion, normal tuples and dicts are better choices both in terms of performance and code clarity.
However, I see there is a new feature available in Python 3.7 called "data classes". Maybe it can be used efficiently instead of named tuples.

Thanks for your answer and research.

It appears that creation of namedtuple is significantly slower than creation of both tuple and dict. Access time by index is almost identical to tuple, but access time by column name is much slower compared to dict.

I agree that namedtuples introduce a performance overhead. That's why it should be optional.

In my opinion, normal tuples and dicts are better choices both in terms of performance and code clarity.

The reason behind this change is to make rows behave similarly to rows returned by pyodbc (https://github.com/mkleehammer/pyodbc/wiki/Row).

I did some more research on this. I can share some interesting findings:

Row object in PyODBC is not a named tuple. They use different structure instead, which is mutable.

Some database drivers actually implemented NamedTupleCursor. For example, this is true for PostgreSQL database driver. However, please see some limitations mentioned in code. There are problems with column names starting with underscore or number. Also, they have to maintain LRU cache for efficiency.

Named tuples can only have maximum 255 arguments in Python 3.6. It was fixed in Python 3.7, but it won't be possible to read wide dataset in Python 3.6.

Performance implications are dire.

I don't see any real use case when having named tuples can be beneficial for quality of the code.
However, I've extended 25_overload example to show how you can still use named tuples in your code if you wish to do so.
class NamedTupleExaStatement(pyexasol.ExaStatement):
    def __next__(self):
        row = super().__next__()
        return self.cls_row(*row)

    def _init_result_set(self, res):
        super()._init_result_set(res)
        self.cls_row = collections.namedtuple('Row', self.col_names)

class NamedTupleExaConnection(pyexasol.ExaConnection):
    cls_statement = NamedTupleExaStatement

C = NamedTupleExaConnection(dsn=config.dsn, user=config.user, password=config.password, schema=config.schema)
stmt = C.execute("SELECT * FROM users ORDER BY user_id LIMIT 5")
print(stmt.fetchone())
print(stmt.fetchone())

Hope it helps!

By the way, please note that you did not need option replace=True. There is an extra check in PyEXASOL to ensure column name uniqueness in result sets. It helps to eliminate potential ambiguous behaviour and helps to maintain integrity of fetch_dict=True option.

However, I've extended 25_overload example to show how you can still use named tuples in your code if you wish to do so.

This is really cool. It absolutely works for me.

By the way, please note that you did not need option replace=True

I needed it for my specific case, because I had some column names starting with underscore, that made namedtuples fail in some cases (as you already identified as a limitation)
I think that doing this in 'userland' is a much better option, because I recognize that it was my own need the one I was trying to solve.
Thanks!


Do not generate "*.pstore" files

The IEV cache should not store pstore files, just like the Relaton gems no longer store them.

@ronaldtse where do you see *.pstore files? I can't find such files.

If you clone https://github.com/CalConnect/csd-admin-documents , run make all, you can see the iev.pstore directories created.

@ronaldtse did it but still don't see any *.pstore files or directories. Maybe they remain form the old version? Did you try do delete them?

Not showing anymore, closing.


Clean up tags

Fixes pulumi/docs#2532
Cleans up our tags to a smaller, easier-to-use set. I optimized for:

Removing individual technologies/providers unless we have substantial content for that technology (e.g. Kubernetes)
Removing references to individual cloud resources
Removing "obvious" tags like pulumi and infrastructure
Removing customer names (and consolidating around the guest-post tag for guest posts)

I'd love feedback. Did I cut a tag you found useful? Are there even more tags we should cut?

The following blog post:    Refactoring Pulumi Code with aliases
has the tag: testing.  Can we switch this to a new tag called:  aliases?  We have a resource option called aliases that it will match.
Reason:
This issue shows up on many customer calls and slack questions and this shows people how to resolve renaming.  No one thinks to look for the word testing.

are we doing redirects for the tags ur removing?

are we doing redirects for the tags ur removing?

I personally wouldn't mind just letting these 404 and fall out of Google, since by removing them we're effectively saying we don't want them to be discovered anymore. If there are any we consider especially important and that we have good redirects for, we could redirect them, but in general I'd be okay just letting them go. (And FWIW, there isn't really a Hugo-based way to do these kinds of redirects, since tag pages aren't authored in the way regular pages are; we'd need to write custom redirect rules to do this.)

@spara thanks!

cloud-native (everything we do is pretty much cloud native)

Agreed, I almost cut this and wanted to see if anyone else suggested - will cut now.

features

I'm torn here - I like the general banner for "blog posts about new features, which may or may not ever be blogged about enough to get their own tags" - like enums. I'd like to keep this unless you feel strongly.

guest-post

I don't feel strongly here, but I did think it'd be useful to keep this so we could see the history of guest posts over time.

pulumi-interns

I liked this as a hiring mechanism that we can point future interns to. I'd like to keep this unless you feel strongly.

@cnunciato when you have a minute, can you help me correctly noindex the tags pages? I found tag.html and also the block_external_search_index variable, but not sure if I can make them work together since the latter seems to only be applicable to Markdown frontmatter, not HTML.

@infin8x We could probably add a robots.txt exclusion for them; I can push a commit that does that, either here or as a separate PR.
But given we're trimming these back to be more relevant, do we want/need to exclude them from indexing? Just want to be sure I understand what the goal is.

Hmm, good point. If we had landing pages in the docs for most tags (esp. the features ones), I would be tempted to say yes, since I'd rather land someone on the primary docs page. But, since we don't, and now that tags are consolidated and a bit more useful, perhaps we can wait for now.
OK - I've convinced myself. Will merge this in, we can discuss noindexing stuff later.

TBH, I would have just ranked the tags, and presented only the "Top N." That way we're not finding ourselves in a situation down the road where someone added a tag here and there -- for instance, why keep "digitalocean" but not "mysql"? I guess because we expect more DigitalOcean content -- and furthermore, why not encourage as much metadata tagging as someone wants to add? It also doesn't encounter the redirects issue, would be less churn (smaller changelist), etc ....

@joeduffy as the tags are currently designed, having "as much metadata as someone wants to add" led to the massive (and, I would say, hard-to-use) list of tags we had. This tactical change gets us to an easy-to-scan set of tags that's Top N-ish. The top file changed in the PR is BLOGGING.md, which discusses when to add tags, what tags we have, etc.

Correct, it led to this situation because we didn’t have the very simple thing I suggested: limit the index to the top N tags. We solved the busy index problem by losing valuable metadata about the posts. If I want to run a campaign in the future for microservice related content, I now don’t have those tags. This change feels like a step backwards and a mistake.

This is an example of what I had in mind, fwiw:
diff --git a/themes/default/layouts/partials/blog/sidebar.html b/themes/default/layouts/partials/blog/sidebar.html
index e0893e40..fb1bfbcb 100644
--- a/themes/default/layouts/partials/blog/sidebar.html
+++ b/themes/default/layouts/partials/blog/sidebar.html
@@ -45,12 +45,12 @@

-            {{ range $tag, $_ := .Site.Taxonomies.tags }}
-                {{ if $tag }}
+            {{ range first 20 .Site.Taxonomies.tags.ByCount }}
+                {{ if .Name }}

-
-                            {{ $tag }}
-
+
+                            {{ .Name }}
+                         ({{ .Count }})

                 {{ end }}
             {{ end }}


RFC: Ability to aggregate percentiles

Problem
Aggregating percentiles is a hard problem. They show the point at which a certain percentage of observed values occur. For example, the 99th percentile is the value which is greater than 99% of the values. An aggregation of percentiles that manipulates the data (e.g. average) would be mathematically incorrect since we’d only be handling the percentile and not the complete set of values for the different time series.
You can however, aggregate “group by, min/max” to group the information and simplify graphs and alerts. For instance you could look at p75 latency, using aggregation max grouping by site. This would give you the maximum value for p75 latency in in the given site across all instances, though it doesn’t tell you how broad the situation is, since it could apply to just one or many instances, you got the maximum value across all the percentiles in each group.
Suggestion
Having first-class support for an aggregatable data structures other than doubles would go a long way to solving this problem. Some type of native histogram support would be an ideal choice, assuming we can ingest them in a sane way.

@dmichel1 experimented with t-digests recently:

The t-digests are really nice since you don't have to worry about setting bucket boundaries yourself. This would require heroic to support storing and aggregations the t-digests.
The bucketed histograms, would be easier to ingest into heroic since the data format is the same as it is today and would only require adding a new aggregation to heroic.

The metrics in OpenCensus and OpenTelemetry all use the bucketed approach. From an integration perspective it would be easiest to adopt these in Heroic as opposed to a new format like t-digests where we would have to add the features to clients.

To compute accurate latency percentiles at a reasonable speed in a distributed environment, we need  a data structure that preserve the actual data distribution. Wether we choose bucketing, data clustering or sketch. We will have to send data to heroic in a format that is currently not  supported.
We are currently evaluating t-digest and data sketch implementations such as HdrHistogram, Circllhist and DDsketch.
We are not considering Opencensus because it uses bucketing with bucket boundaries. To get accurate percentile, you need to setup good bucket boundaries.That is possible if you have prior knowledge of the data distribution.
Furthermore, Opencensus is merging with OpenTelemetry and OpenTelemetry  and Opencensus don’t have the same distribution interface. So It is not clear what is the long term support model for Opencensus distribution.


Add ability to derive sort inputs for related data.

Describe the solution you'd like
In the example that Author has many Post I might want to write a query something like this:
query {
  authors {
    name
    results {
      posts(sort: [field: "NAME", order: "ASC"]) {
        name
      }
    }
  }
}

ie I'd like to be able to sort relationships as they're loaded.
Describe alternatives you've considered
We're currently using an action argument that is passed to Ash.Query.sort_input.

This is already supported :)


Numerical instability issues with bezier curves

I've included a failure test case in the demo, here:
https://github.com/velipso/polybool/blob/3ee7c440ebee9431fd84506679384b055d456d2a/demo/demo.html#L117
This will fail due to numerical instability.
I'm not sure exactly how to fix it... the solveCubic function is returning a T value very close to 1, but not exactly 1 (and not within striking distance of the epsilon value), so it ends up thinking there is an intersection between a curve and a line, but in reality they just share an endpoint.
I think the solveCubic function needs to take the epsilon value into account in a smarter way, or perhaps rewritten entirely.

I've changed things, but now there is some silliness here:
https://github.com/velipso/polybool/blob/80bfb194efef5a332e9528607ebf06b03f86dcee/src/Segment.ts#L464-L476

I fixed some of this, but now the only issue I notice is when two curves intersection point just barely touches


Accumulation store reset on lockup migration

changes made other than the main migration script:

remove Lock() as an exported method for locking tokens
use LockTokens() for single entry point for locking tokens
add createLock() that creates a lock with next lock ID
createLock() and deleteLock() now add/deletes lockrefs in it

Description

For contributor use:

[ ] Targeted PR against correct branch (see CONTRIBUTING.md)
[ ] Updated relevant documentation (docs/) or specification (x/<module>/spec/)
[ ] Added a relevant changelog entry to the Unreleased section in CHANGELOG.md
[ ] Re-reviewed Files changed in the Github PR explorer

Codecov Report

Merging #941 (0923c3d) into main (61aecb5) will increase coverage by 0.02%.
The diff coverage is 75.00%.

@@            Coverage Diff             @@
##             main     #941      +/-   ##
==========================================
+ Coverage   20.60%   20.63%   +0.02%
==========================================
  Files         193      193
  Lines       25038    25025      -13
==========================================
+ Hits         5160     5164       +4
+ Misses      18938    18921      -17
  Partials      940      940

Impacted Files
Coverage Δ

x/lockup/keeper/admin_keeper.go
25.92% <0.00%> (+4.71%)
:arrow_up:

x/lockup/keeper/msg_server.go
24.67% <0.00%> (ø)

x/lockup/keeper/migration.go
66.66% <69.44%> (+6.20%)
:arrow_up:

x/lockup/keeper/lock.go
58.74% <94.44%> (+3.72%)
:arrow_up:

x/lockup/keeper/store.go
85.96% <100.00%> (-1.43%)
:arrow_down:

x/lockup/keeper/lock_refs.go
30.76% <0.00%> (-7.70%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 61aecb5...0923c3d. Read the comment docs.

These are great API changes, that I also thought were problematic as I was reviewing the lockref code yesterday

RIP the merge conflicts. I do think the API changes this PR did  + a function to reset the accumulation store during a migration are both really useful things we still want to get in.
I think the move to proceed here may be to re-PR the API changes as one PR, and re-PR the migration logic, due to the merge conflicts.

I think the move to proceed here may be to re-PR the API changes as one PR, and re-PR the migration logic, due to the merge conflicts.

I totally agree. @mconcat do you want help here?

Yeah I will, re-PR seems more reasonable


Test fails with no matches for kind "Route" although routev1  is added to schema

Bug Report

What did you do?
I am following the code skeleton for writing test for controllers
Since my controller also uses openshift routes I added the route to the schema
//controllers/suite_test.go
import (
       ...
	routev1 "github.com/openshift/api/route/v1"
	"k8s.io/client-go/kubernetes/scheme"
	...
)

func TestAPIs(t *testing.T) {
	RegisterFailHandler(Fail)
	RunSpecsWithDefaultAndCustomReporters(t,
		"Controller Suite",
		[]Reporter{printer.NewlineReporter{}})
}

var _ = BeforeSuite(func(done Done) {
	logf.SetLogger(zap.LoggerTo(GinkgoWriter, true))

	By("bootstrapping test environment")
	testEnv = &envtest.Environment{
		CRDDirectoryPaths: []string{filepath.Join("..", "config", "crd", "bases")},
	}

	var err error
	cfg, err = testEnv.Start()
	Expect(err).ToNot(HaveOccurred())
	Expect(cfg).ToNot(BeNil())

	err = elmv1.AddToScheme(scheme.Scheme)
	Expect(err).NotTo(HaveOccurred())
	//ADD ROUTE-V1
	err = routev1.AddToScheme(scheme.Scheme)
	Expect(err).NotTo(HaveOccurred())

	k8sClient, err = client.New(cfg, client.Options{Scheme: scheme.Scheme})
	Expect(err).ToNot(HaveOccurred())

	k8sManager, err := ctrl.NewManager(cfg, ctrl.Options{
		Scheme: scheme.Scheme,
	})
	Expect(err).ToNot(HaveOccurred())

        //try to add route v1 again - does not help
	err = routev1.AddToScheme(k8sManager.GetScheme())
	Expect(err).NotTo(HaveOccurred())

	err = (&ELMReconciler{
		Client: k8sManager.GetClient(),
		Log:    ctrl.Log.WithName("controllers").WithName("ELM"),
	}).SetupWithManager(k8sManager)
	Expect(err).ToNot(HaveOccurred())

	go func() {
                //this call will fail with no matches for kind Route
		err = k8sManager.Start(ctrl.SetupSignalHandler())
		Expect(err).ToNot(HaveOccurred())
	}()

	k8sClient = k8sManager.GetClient()
	Expect(k8sClient).ToNot(BeNil())

	close(done)
}, 60)

What did you expect to see?
I expected the manager to start in the test suite without errors as it does in runtime
What did you see instead? Under which circumstances?
After manager.start is called the test fails with no matches for kind "Route" error
2021-01-26T13:55:05.853+0200    ERROR   controller-runtime.source       if kind is a CRD, it should be installed before calling Start   {"kind": "Route.route.openshift.io", "error": "no matches for kind \"Route\" in version \"route.openshift.io/v1\""}
github.com/go-logr/zapr.(*zapLogger).Error
        /home/avrahams/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128
sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start
        /home/avrahams/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/source/source.go:117
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
        /home/avrahams/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:143
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start
        /home/avrahams/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:184
sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1
        /home/avrahams/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/manager/internal.go:661
panic:
Your test failed.
Ginkgo panics to prevent subsequent assertions from running.
Normally Ginkgo rescues this panic so you shouldn't see it.

Environment
Operator type:

/language go

Kubernetes cluster type:
OpenShift

$ operator-sdk version
operator-sdk version: "v1.2.0", commit: "215fc50b2d4acc7d92b36828f42d7d1ae212015c", kubernetes version: "v1.18.8", go version: "go1.15.3", GOOS: "linux", GOARCH: "amd64"

$ go version (if language is Go)
go version go1.15.6 linux/amd64
$ kubectl version

Possible Solution

Additional context

HI @varshaprasad96 @rashmigottipati,
It is related to our chat. some references:

https://github.com/kubernetes-sigs/kubebuilder/issues/1772#issuecomment-723213345
blog post with the legacy layout https://developers.redhat.com/blog/2020/02/04/how-to-use-third-party-apis-in-operator-sdk-projects/
sdk docs: https://sdk.operatorframework.io/docs/building-operators/golang/advanced-topics/#adding-3rd-party-resources-to-your-operator

It is something that we could indeed doc in upstream/kb and link here. Upstream is also missing a docs related.

How to scaffold external/third-party apis? How config the test for external/third-party apis?

HI @varshaprasad96 @rashmigottipati,
It is related to our chat. some references:

https://github.com/kubernetes-sigs/kubebuilder/issues/1772#issuecomment-723213345
blog post with the legacy layout https://developers.redhat.com/blog/2020/02/04/how-to-use-third-party-apis-in-operator-sdk-projects/
sdk docs: https://sdk.operatorframework.io/docs/building-operators/golang/advanced-topics/#adding-3rd-party-resources-to-your-operator

It is something that we could indeed doc in upstream/kb and link here. Upstream is also missing a docs related.

How to scaffold external/third-party apis? How config the test for external/third-party apis?

@camilamacedo86 I understand that the code I expected to work as in kubernetes-sigs/kubebuilder#1772 (comment) is not working.
Is there a workaround that I can use until it's fixed ?

@camilamacedo86 I understand that the code I expected to work as in kubernetes-sigs/kubebuilder#1772 (comment) is not working.
Is there a workaround that I can use until it's fixed ?

Hi @avrahams,
It might be missing some detail. To solve this issue is required to do a POC and a few checks. Then, properly doc that as well.
@varshaprasad96 @rashmigottipati see that be solving this one we might be able to close  https://github.com/kubernetes-sigs/controller-runtime/issues/1191

Hi @avrahams,
It might be missing some detail. To solve this issue is required to do a POC and a few checks. Then, properly doc that as well.
@varshaprasad96 @rashmigottipati see that be solving this one we might be able to close  https://github.com/kubernetes-sigs/controller-runtime/issues/1191

@varshaprasad96 will try to reproduce this. While https://github.com/kubernetes-sigs/controller-runtime/issues/1191 might be the root cause, we are not sure.
/assign @varshaprasad96
/triage needs-information

Shows that for ENV TEST we need also update the CRDDirectoryPath to pass the right info. In this way, to solve this way we just need to confirm that by doing this test with for example mechached sample in the testdata and doc it.

While adding the CRDs seems to work, in case of OpenShift API resources it is a bit cumbersome. Since the api is not published via tags I have to use pseudo-versions. Keeping the CRD paths in sync with the go.mod is doable, but maybe streamlining that experience is possible? Especially since operator-sdk targets OCP, I kinda expect to not have to do these steps manually.
Storing the CRDs as fixtures in the project repository would also be possible, but that still would require to manually put them there each time I update the go.mod for the OCP api.
Another question:
While a lot of CRDs for OCP APIs can be found in the corresponding api repository, I am not able to find CRDs for stuff like routes, DeploymentConfigs, etc. and it seems they cannot be generated from the api repository either. Am I missing something here? Right now I do not find a way to add these resources to envtest.

Hi @mpreu,
External types are not officially supported by the tool yet. I mean the tool is not prepared for dealing with this scenario and users need to change the files manually.  However, see that we have an issue to starts to support in upstream. Then, it will also be applied in SDK. See:  https://github.com/kubernetes-sigs/kubebuilder/issues/1999
And then, I also understand that we need to check here how is the best approach to set up the CRD paths for external-types for Envtest in order to solve this scenaro. I mean, we need to confirm if the solution described by you still the best/recommended way to do it.

I am facing the same issue with envtest. Getting no matches for kind...
Seems like the CRDs are not being installed correctly.
Any workarounds?

While adding the CRDs seems to work, in case of OpenShift API resources it is a bit cumbersome. Since the api is not published via tags I have to use pseudo-versions. Keeping the CRD paths in this format in sync with the go.mod is doable, but maybe streamlining that experience is possible? Especially since operator-sdk targets OCP, I kinda expect to not have to do these steps manually.

@mpreu I don't see how this is a generally solvable problem unfortunately, since CRDs can be hosted anywhere for any project your operator depends on (doesn't have to be a k8s distro's APIs either, they could be another operator's). operator-sdk is meant to be general, i.e. doesn't specifically target OCP but does support it, but can't know where all distro's/operator's CRDs are hosted and appropriately download/version them. The best operator-sdk could do is wrap curl https://my.domain/crd.yaml, which isn't that useful.

While a lot of CRDs for OCP APIs can be found in the corresponding api repository, I am not able to find CRDs for stuff like Routes, DeploymentConfigs, Projects (project.openshift.io/v1), ... and it seems they cannot be generated from the api repository either

I would bring this issue up in the openshift/api repo.

Any workarounds?

@ahalim-intel you'll have to download the CRD manifests and reference them during envtest setup.

I've confirmed that this issue is a duplicate of https://github.com/kubernetes-sigs/controller-runtime/issues/1191, so all conversation should be moved to that issue. I'm going to close this with an FAQ update.
/remove-triage needs-information
/triage duplicate

Have submitted a PR upstream (https://github.com/kubernetes-sigs/controller-runtime/pull/1393) to document this.

Hi All,
As I mentioned. my problem was slightly different then the OP's one(no external CRD) but with same error.
What I was doing differently is that I added multi-group API in my project. The kubebuilder documentation doesn't mention anywhere that  suite_test.go  should be updated accordingly as adding multi-group api adds additional directory to the apis/ and controller/ tree.
So basically with generated suite_test.go we get:
	testEnv = &envtest.Environment{
		CRDDirectoryPaths: []string{filepath.Join("..", "config", "crd", "bases")},
	}

Since adding multi-group api puts everything one directory down we needed to add additional ".." to the filepath.
So, adding that solved my particular problem.
	testEnv = &envtest.Environment{
		CRDDirectoryPaths: []string{filepath.Join("..", "..", "config", "crd", "bases")},
	}

Took to me a good while to figure this simple error. I think this should added in somewhere like here?
I hope someone facing similar issue find this helpful.

Hi @ahalim-intel,

The kubebuilder documentation doesn't mention anywhere that suite_test.go should be updated accordingly as adding multi-group api adds additional directory to the apis/ and controller/ tree.

When you enable multi-group then the new scaffolds will be generating with the path changed as you described. However, it should be described in the doc linked by you too. WDYT about help us and push a PR to update it? Here is the place: https://github.com/kubernetes-sigs/kubebuilder/blob/master/docs/book/src/migration/multi-group.md

Hi @camilamacedo86,

WDYT about help us and push a PR to update it? Here is the place: https://github.com/kubernetes-sigs/kubebuilder/blob/master/docs/book/src/migration/multi-group.md

I've submitted a PR as suggested. PTAL  https://github.com/kubernetes-sigs/kubebuilder/pull/2054

Update: for OCP and in general externally-defined APIs that use extra-CRD validation, like webhooks and custom apiserver imperative validation, there is no good solution for this problem with envtest; positive test outcomes can be misleading if CRDs alone are used because some other validation could fail server-side in prod. Therefore OCP CRDs alone cannot solve this issue.
These facts should be documented in the envtest library, and an FAQ should be added for like-kind errors with this explanation.
/remove-triage duplicate
/kind documentation
/priority important-longterm

This is similar to the upstream issue (https://github.com/kubernetes-sigs/controller-runtime/issues/1191). This needs to be handled upstream  since envtest library needs more documentation.

Hi All, As I mentioned. my problem was slightly different then the OP's one(no external CRD) but with same error. What I was doing differently is that I added multi-group API in my project. The kubebuilder documentation doesn't mention anywhere that  suite_test.go should be updated accordingly as adding multi-group api adds additional directory to the apis/ and controller/ tree.
So basically with generated suite_test.go we get:
	testEnv = &envtest.Environment{
		CRDDirectoryPaths: []string{filepath.Join("..", "config", "crd", "bases")},
	}

Since adding multi-group api puts everything one directory down we needed to add additional ".." to the filepath.
So, adding that solved my particular problem.
	testEnv = &envtest.Environment{
		CRDDirectoryPaths: []string{filepath.Join("..", "..", "config", "crd", "bases")},
	}

Took to me a good while to figure this simple error. I think this should added in somewhere like here?
I hope someone facing similar issue find this helpful.

Thanks a lot!
It perfectly solves the problem of third-party crds.


Upgrade tensorflow to 1.4.0

As a developer, I'd like to upgrade the tensorflow-version from 1.1.0 to 1.4.0.

@sabbyanandan in #5 i upgraded to tenser-flow 1.3.0. Apparently 1.4.0 they have made some breaking API changes and it deserves more thoughtful work to upgrade to it.
Shall I reopen this? or create a new one to track this requirement?

Oh, oops! I assumed it was already applied based on the conversation we have had in slack. I'll reopen it.


WIP: First commit of a tool that reconfigures tests in response to test re…

…sults.
It's not done and extremely hacky, but if someone want to take a look I'm happy to hear any feedback.
cc @wojtek-t @eparis @kubernetes/sig-testing

@eparis @wojtek-t PTAL

All code is there, it just needs to be uncommented and wrapped in appropriate pod. I already tested it on K8s and at least read path works from container. I tested write path locally.


Slack coder/documentation tests

Aid new users by documenting radicle usage via a workflow.  Order the examples to help user navigation and remove redundant example tests already covered by the workflow.

This is really cool. I'd prefer if we didn't touch the existing tests though, as more may be added to them as more functionality is added, and we may not want that in the workflow examples. I'd also keep the workflow tests in their own folder, like: examples/workflow/1-rad-clone.md, examples/workflow/2-open-issue.md etc.

I'd prefer if we didn't touch the existing tests though, as more may be added to them in the future as more functionality is added, and we may not want that in the workflow examples.

They can be added as needed.  In the mean time, its duplicate code which needs to be maintained and adds noise.

Squashed an rebased because it was gettting unwiedly

Can we address further problems via PRs?  There can always be improvement, and we're beginning to run into merge conflicts.

@cloudhead should be good now


Education description field is missing

@aurelien-g I don't think this is a bug - JSON Resume schema does not have a description property for education - that is a LinkedIn field  - nor does this appear to be an existing JSON Resume schema field that it would make sense to map this free-text field to. Let me know if you have a different take on this or any other context to provide.


Add instanceGroupManagers.update

It appears not all of the API end-points appear in the generated libraries. In particular when I run grep on the file below it appears that instanceGroupManagers.update is absent.
https://raw.githubusercontent.com/google/google-api-go-client/master/compute/v1/compute-gen.go
grep -i compute.instanceGroupManagers compute-gen.go | grep "method id"
// method id "compute.instanceGroupManagers.abandonInstances":
// method id "compute.instanceGroupManagers.aggregatedList":
// method id "compute.instanceGroupManagers.delete":
// method id "compute.instanceGroupManagers.deleteInstances":
// method id "compute.instanceGroupManagers.get":
// method id "compute.instanceGroupManagers.insert":
// method id "compute.instanceGroupManagers.list":
// method id "compute.instanceGroupManagers.listManagedInstances":
// method id "compute.instanceGroupManagers.recreateInstances":
// method id "compute.instanceGroupManagers.resize":
// method id "compute.instanceGroupManagers.setInstanceTemplate":
// method id "compute.instanceGroupManagers.setTargetPools":

https://cloud.google.com/compute/docs/reference/rest/beta/instanceGroupManagers/update
Understand the API is largely generated what's required if anything to add that end-point?

Closing posted to the wrong Google Cloud library. :/


Fixed typo in ref to Github.

Just fixing a minor typo in the documentation.

@arkottke Awesome! Thanks a lot!


Sagemaker hosting instance runs out of diskspace

Hi Team,
We used this docker file to build and train model using Amazon Sagemaker.
After hosting the model and endpoint using the docker image, We get issues as the hosting instance disk is full. Not sure about the files it generates to fill the disk space.
AWS support team said that they cant do anything for a custom image.
Any pointers on this issue would be appreciated
Thank you

@bchalamayya, typically you can customize the amount of disk space you have on the instance. Which image are you using? can you share some more information?
Typically speaking, everything on the H2O-3 platform is performed in memory, so there should not be anything being written to disk unless it is specifically told to do so.

if you are talking about the algorithms hosted on sagemaker marketplace, it is likely that aws is copying the datasets provided into the instance which is where this issue is arising. We read in the data from disk to memory and then do all our work in memory


Can O' Spam: Watch vitalmobility9.wordpress.com

Can O' Spam requests the watch of the watch_keyword vitalmobility9\.wordpress\.com. See the MS search here and the Stack Exchange search in text, in URLs, and in code.

vitalmobility9\.wordpress\.com has been seen in 1 true positive, 0 false positives, and 0 NAAs.

Approved by VLAZ in Charcoal HQ


Issue Running Examples

Hello Jake,
I am attempting to run easyQuake and I have ran into a few preliminary challenges. One is though I have conda forge as an accessible channel, I cannot seem to download tensorflow-gpu==2.1.
Secondly, I do not have a machine with a GPU. Your write up does not explicitly state that a gpu is necessary. Can easyQuake be ran with a CPU?
Third, I am attempting to work through the idaho_example.py and I receive an error for the project_folder variable in the download_mseed(), detection_continous(), and association_sontinous() functions. When I try to run these this is what is returned: OSError: [Errno 30] Read-only file system: '/data'. Tho pointers "data/id" where pre-populated in the file, is this something I should change to a directory that exists or is this used to create a new directory to populate the downloaded files?
Bryan

You need to make sure that the conda environment is python 3.6, higher python versions will not have tensorflow-gpu==2.1 available. Future work is intended to bring everything up to compatibility with python 3.9 and the latest tensorflow.

The example codes assume that you are running a system with NVIDIA CUDA drivers and tensorflow-gpu installed. CPU is possible but I admittedly have not spent much time on thoroughly testing.

Define the full path on your system - that is the one that is tested I am not sure whether it handles relative paths very well.


Improve relations initialization time (#924)

Reduce computation required to generate relations on the first request.
Sample test case
PostgREST v<IP_ADDRESS>
$ time curl 'http://localhost:3000/x?select=x(*)'

real	0m27.601s
user	0m0.001s
sys	0m0.001s

PostgREST with combinations improvement
$ time curl 'http://localhost:3000/x?select=x(*)'

real	0m0.012s
user	0m0.001s
sys	0m0.001s

The performance improvement is great, though I can't really say I understand the combinations function easily, how about adding the reference implementation in a comment like:
-- From https://wiki.haskell.org/99_questions/Solutions/26
combinations :: ...

I must agree on this, I will add the reference.

Thanks @9too, good work tracing it back to this function. What a difference a good algorithm can make...


Allow fixed Y axis min and max values

It's pretty annoying when the graphs change the min/max and the graph is auto rescaled.

Done


STRING is not defined

In the example in the file app-global.js a global variable STRING is used in a function Number.prototype.format but it's not defined anywhere which causes error:
Uncaught ReferenceError: STRING is not defined

Hi @molda,
I have fixed it, sorry. BTW: www.componentator.com is free, you can download all jComponent components.
Many thanks!


Cannot install Airflow 2.7.3 with Python 3.11

Solution to issue cannot be found in the documentation.

[X] I checked the documentation.

Issue
Getting the following errors trying to install airflow 2.7.3 with python 3.11:
 > [server base 15/21] RUN micromamba install -y -n base -f /tmp/env.yml && micromamba clean --all --yes:
7.019 error    libmamba Could not solve for environment specs
7.019     The following packages are incompatible
7.019     ├─ airflow 2.7.3**  is installable with the potential options
7.019     │  ├─ airflow 2.7.3 would require
7.019     │  │  └─ python_abi 3.10.* *_cp310, which can be installed;
7.019     │  ├─ airflow 2.7.3 would require
7.019     │  │  └─ python_abi 3.8.* *_cp38, which can be installed;
7.019     │  └─ airflow 2.7.3 would require
7.019     │     └─ python_abi 3.9.* *_cp39, which can be installed;
7.019     └─ python >=3.11,<3.12  is not installable because there are no viable options
7.019        ├─ python [3.11.0|3.11.1|...|3.11.6] would require
7.019        │  └─ python_abi 3.11.* *_cp311, which conflicts with any installable versions previously reported;
7.019        └─ python 3.12.0rc3 would require
7.019           └─ _python_rc, which does not exist (perhaps a missing channel).
7.031 critical libmamba Could not solve for environment specs

Installed packages
N/A

Environment info
libmamba version : 1.5.1
     micromamba version : 1.5.1
           curl version : libcurl/7.88.1 OpenSSL/3.1.2 zlib/1.2.13 zstd/1.5.5 libssh2/1.11.0 nghttp2/1.52.0
     libarchive version : libarchive 3.6.2 zlib/1.2.13 bz2lib/1.0.8 libzstd/1.5.2
       envs directories : /opt/conda/envs
          package cache : /opt/conda/pkgs
                          /home/airflow/.mamba/pkgs
            environment : base (active)
           env location : /opt/conda
      user config files : /home/airflow/.mambarc
 populated config files :
       virtual packages : __unix=0=0
                          __linux=6.5.0=0
                          __glibc=2.35=0
                          __archspec=1=aarch64
               channels :
       base environment : /opt/conda
               platform : linux-aarch64

@adrianrego, can you provide more info on what environment you're starting from and what environment is in /tmp/env.yml?  The error message you're getting suggests some sort of a conflict and I can't be of much help without significantly more info about what you're trying to do.

Ah, sorry, my mistake.  The package just didn't build successfully.  I'll try again!

@adrianrego, the file should be there now.  Could you test again?

@xylar can confirm that it builds now.  thanks for the quick turnaround.


Refactor scope of all commands to avoid globals and keep CLI API internal

Fixes #33
Closes #66

Deferring deeper review to #128


Added A2 Online Judge

A2 Online Judge - Categories - is a treamendously rich list of problems over cross online judges, and the questions categoried according to topic and their difficulty levels.
It is worth adding A2OJ to this list for coding practice.

Sorry, I'm not adding this, its interface is not great and it seems to be an OJ link aggregator. I will add Code Chef though.


use better model for api result(error object should be visible when there is an error)

Update ApiResult and ApiResult classes to conditionally serialize the 'Errors' property.

Done.


Updated Jython Evaluator Script

Simply trying out the tutorial won't work and lead to an error mentioning "err" and "out" is not defined without really specifying the location which makes it pretty hard for someone just starting out to find the problem and fix it.
#
#  output.write(record): writes a record to processor output
#
#  error.write(record, message): sends a record to error
#

Thanks for the contribution!