internal/zstd: Bugs and crashes

What version of Go are you using (go version)?
Go tip, self compiled.
Does this issue reproduce with the latest release?
Yes.
What did you do?
Run some fuzz tests against the internal zstd package.
What did you expect to see?
No output.
What did you see instead?
A) Crash
With the following input, the internal/zstd decompressor can fail. To reproduce:
Add src/internal/zstd/testdata/fuzz/FuzzReader/e4f4dab13d374c2d with the following content:
go test fuzz v1
[]byte("(\xb5/\xfd00\xec\x00\x00&@\x05\x05A7002\x02\x00\x02\x00\x02\x0000000000000000")
This produces the following crash:
--- FAIL: FuzzReader (10.70s)
    --- FAIL: FuzzReader (0.00s)
        testing.go:1504: panic: runtime error: index out of range [2] with length 2
            goroutine 145795 [running]:
            runtime/debug.Stack()
                C:/go/src/runtime/debug/stack.go:24 +0x9b
            testing.tRunner.func1()
                C:/go/src/testing/testing.go:1504 +0x1ee
            panic({0x6952c0?, 0xc000130618?})
                C:/go/src/runtime/panic.go:914 +0x21f
            [github.com/klauspost/compress/z2.(*Reader).readLiteralsFourStreams(0xc001cb1e00](http://github.com/klauspost/compress/z2.(*Reader).readLiteralsFourStreams(0xc001cb1e00), {0xc000502d20, 0x1d, 0x20}, 0x9, 0xf, 0x2, {0x0, 0x0, 0x0})
                e:/gopath/src/[github.com/klauspost/compress/z2/literals.go:324](http://github.com/klauspost/compress/z2/literals.go:324) +0x1079
            [github.com/klauspost/compress/z2.(*Reader).readHuffLiterals(0xc001cb1e00](http://github.com/klauspost/compress/z2.(*Reader).readHuffLiterals(0xc001cb1e00), {0xc000502d20, 0x1d, 0x20}, 0x1, 0x26, {0x0, 0x0, 0x0})
                e:/gopath/src/[github.com/klauspost/compress/z2/literals.go:168](http://github.com/klauspost/compress/z2/literals.go:168) +0x11e7
            [github.com/klauspost/compress/z2.(*Reader).readLiterals(0xc001cb1e00](http://github.com/klauspost/compress/z2.(*Reader).readLiterals(0xc001cb1e00), {0xc000502d20, 0x1d, 0x20}, 0x0, {0x0, 0x0, 0x0})
                e:/gopath/src/[github.com/klauspost/compress/z2/literals.go:26](http://github.com/klauspost/compress/z2/literals.go:26) +0x30d
            [github.com/klauspost/compress/z2.(*Reader).compressedBlock(0xc001cb1e00](http://github.com/klauspost/compress/z2.(*Reader).compressedBlock(0xc001cb1e00), 0x1d)
                e:/gopath/src/[github.com/klauspost/compress/z2/block.go:35](http://github.com/klauspost/compress/z2/block.go:35) +0x44d
            [github.com/klauspost/compress/z2.(*Reader).readBlock(0xc001cb1e00)](http://github.com/klauspost/compress/z2.(*Reader).readBlock(0xc001cb1e00))
                e:/gopath/src/[github.com/klauspost/compress/z2/zstd.go:397](http://github.com/klauspost/compress/z2/zstd.go:397) +0xd10
            [github.com/klauspost/compress/z2.(*Reader).refill(0xc001cb1e00](http://github.com/klauspost/compress/z2.(*Reader).refill(0xc001cb1e00)?)
                e:/gopath/src/[github.com/klauspost/compress/z2/zstd.go:164](http://github.com/klauspost/compress/z2/zstd.go:164) +0xd4
            [github.com/klauspost/compress/z2.(*Reader).refillIfNeeded(0xc001cb1e00)](http://github.com/klauspost/compress/z2.(*Reader).refillIfNeeded(0xc001cb1e00))
                e:/gopath/src/[github.com/klauspost/compress/z2/zstd.go:149](http://github.com/klauspost/compress/z2/zstd.go:149) +0xa5
            [github.com/klauspost/compress/z2.(*Reader).Read(0xc001cb1e00](http://github.com/klauspost/compress/z2.(*Reader).Read(0xc001cb1e00), {0xc001d2a000, 0x2000, 0x0?})
                e:/gopath/src/[github.com/klauspost/compress/z2/zstd.go:128](http://github.com/klauspost/compress/z2/zstd.go:128) +0x45
            io.discard.ReadFrom({}, {0x6f0200, 0xc001cb1e00})
                C:/go/src/io/io.go:658 +0xad
            io.copyBuffer({0x6f02a0, 0x876a20}, {0x6f0200, 0xc001cb1e00}, {0x0, 0x0, 0x0})
                C:/go/src/io/io.go:416 +0x14f
            io.Copy(...)
                C:/go/src/io/io.go:389
            [github.com/klauspost/compress/z2.FuzzReader.func1(0x0](http://github.com/klauspost/compress/z2.FuzzReader.func1(0x0)?, {0xc00013a150, 0x26, 0x30})
                e:/gopath/src/[github.com/klauspost/compress/z2/fuzz_test.go:42](http://github.com/klauspost/compress/z2/fuzz_test.go:42) +0x10e
            reflect.Value.call({0x66dca0?, 0x6b9990?, 0x6a68e0?}, {0x6a7418, 0x4}, {0xc001c7b1a0, 0x2, 0x18?})
                C:/go/src/reflect/value.go:596 +0xce7
            reflect.Value.Call({0x66dca0?, 0x6b9990?, 0x8061a0?}, {0xc001c7b1a0?, 0x6a68e0?, 0x57c58d?})
                C:/go/src/reflect/value.go:380 +0xb9
            testing.(*F).Fuzz.func1.1(0x436800?)
                C:/go/src/testing/fuzz.go:335 +0x347
            testing.tRunner(0xc001ccb040, 0xc001c97170)
                C:/go/src/testing/testing.go:1595 +0xff
            created by testing.(*F).Fuzz.func1 in goroutine 19
                C:/go/src/testing/fuzz.go:322 +0x597

Checking that out1+out2+out3 are in bounds fixes the issue, but it seems a bit expensive.
B) Missing error reporting
The following inputs does not report expected problems:
zstd-internal-divergences.zip
Input and various outputs are reported as:
go: <nil>, klaus: output bigger than max block size (1024), zstd.exe: exit status 1: /*stdin*\ : Decoding error (36) : Data corruption detected

go: Error reported by internal/zstd.
klaus: Error reported by https://github.com/klauspost/compress/zstd
zstd.exe: Error reported from Zstandard CLI (64-bit) v1.5.4
Sorry for not having the time to dig into these right now, but I thought it would be the best to get them reported.

I attempted to make a fix for this issue that: https://github.com/golang/go/pull/63959

I attempted to make a fix for this issue that: https://github.com/golang/go/pull/63959

@aimuz Will you be picking up the un-reported corruption as well?

@aimuz Will you be picking up the un-reported corruption as well?

@klauspost If you suspect there's an underlying problem that hasn't been documented yet, please let me know, and I'll investigate it.

@aimuz I was thinking of the divergence in output in the "B) Missing error reporting" section.

@klauspost Can you give me more details?

@aimuz See section B) of the issue above. There is a zip file with reproducers.

I'm sorry I left the above information out, thank you, I'll look into it 😀

@klauspost All issues have been fixed

@aimuz If you are up for it, here is another batch: batch2.zip
Probably only as few root issues. Seeing unreported: output bigger than max block size, unexpected EOF errors.
Seeing zstd decompression error at XX: invalid symbol compression mode on seemingly valid content.

Seeing zstd decompression error at XX: invalid symbol compression mode on seemingly valid content.

It should be fine as described in the documentation here
https://datatracker.ietf.org/doc/html/rfc8878#name-symbol_compression_modes


EKU in User Authentication template

Is it required for the Always On VPN to have the Encrypting File System (<IP_ADDRESS>.4.1.3<IP_ADDRESS>) and Secure Email (<IP_ADDRESS>.<IP_ADDRESS>.4) EKU in it?
As documented to duplicate the User Template, this would be the case.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 0e886cf2-b2cc-1942-2e93-b83ba3f5c81a
Version Independent ID: 14185f76-c920-66a5-9d3b-51ddc4845ee8
Content: Configure the Server Infrastructure
Content Source: WindowsServerDocs/remote/remote-access/vpn/always-on-vpn/deploy/vpn-deploy-server-infrastructure.md
Product: windows-server-threshold
GitHub Login: @shortpatti
Microsoft Alias: pashort

@shortpatti - can you take a look at this question? Thanks!

Hi @makauf,  I sent an email to the remote access team this morning to get an answer to your question.

@makauf, thank you for your patience.  I've been in contact with engineering and support to update several areas in the content, including your question.

Create the User Authentication template - Step 5d ambiguity: After adding the VPN Users group, Read permission defaults to Allow.  The instruction is: "select the Enroll and Autoenroll check boxes in the Allow column".  Do I leave the Read permission in the Allow column unchanged (checked is the default) or should I clear so that ONLY Allow Enroll and Allow Autoenroll are checked?  I suspect that I am supposed to just make the changes specifically listed and leave Allow Read checked, but more clarification would be appreciated.

Create the User Authentication template - Step 4 warning: When making a duplicate of the User Template in Step 4, the remaining steps detail several additional configuration changes before saving the copied template in Step 10.   If "OK" or "Apply" buttons are clicked before ALL of the parameters are entered, many of the choices become fixed and no longer editable.  For example Cryptography tab ->  Provider Category field may show "Legacy Cryptographic Storage Provider" and become disabled, preventing any further change.  The only alternative is to delete the template and recreate, following the instructions verbatim.  Do not click "Apply" or "OK" at any time prior to Step 10.

Create the VPN Server Authentication template - Step 13 clarification:  In Step 4, a "Template Display Name" was chosen, probably from the examples given: "VPN Server Authentication or RADIUS Server".  In Step 13 a "Certificate Template to Issue" is chosen and instructions indicate to choose "RADIUS Server".  For Step 13, select the name you chose in Step 4 in place of "RADIUS Server".

Thank you for all the feedback.  It looks like I have some updating to do on this particular topic.

I've made the changes and they will be published later today.

after your update the EKUs on the VPN Template still contains EFS and Secure Email (initial question)

@MihaiSP, can you take a look at this issue?


Move into gem structure and use rspec

Fixes: #4, #8, #9

Merging first before fixing all Rubocop issues.


Function constructor end pf block issue

Not sure this has been addressed already, at least not in 2.1.1. Consider this code:
f = new Function("//");
Current Function constructor appends "}" to the code block, which then is ignored by the compiler, since the line is a comment.
Adding a '\n' to duk_bi_function_constructor() in duk_bi_function.c should fix this problem:
duk_push_string(ctx, "\n}");
duk_concat(ctx, 5);

@kakaraky Good point, there was a similar issue in the module loaders which was fixed. I'll fix this soon.

Fixed in master, backports to maintenance releases will be done later. Thanks @kakaraky !


Show if the push notifications are configured for the server

@borisyankov Updated!

Thanks @kunall17 for taking this on!
I've written down at https://github.com/zulip/zulip-mobile/issues/1507#issuecomment-405006721 the plan we settled on today. Closing this version.


[BUG] Should use "-" instead of "_" to connect dialogue act.

Describe the bug
When example code PPO.py is run, ValueError occurs.
File [MY CURRENT DIRECTORY]/ConvLab-3/convlab/util/multiwoz/lexicalize.py:37, in deflat_da(meta)
     35 dialog_act = {}
     36 for da in meta:
---> 37     d, i, s, v = da.split('_')
     38     k = '_'.join((d, i))
     39     if k not in dialog_act:

ValueError: not enough values to unpack (expected 4, got 1)

In VectorBase class, the dialogue act is joined and split by _, but ontology is connected by -.
Use - to join and split. Or replace - with _ of vec2act and act2vec in VectorBase class initialization.
To Reproduce
Run examples/agent_examples/test_PPO.py.
The original code gets the PPO arguments wrong, so fix line 22 that:
- sys_policy = PPO(False, seed=seed, use_action_mask=True, shrink=False)
+ sys_policy = PPO(False, seed=seed)

I use seed=0 and n_dialogues=10 but it does not matter.
Expected behavior
PPO.py runs without error.
Actual behavior
as stated above
Additional context
I have only tried PPO.py but this bug may affect some other code as the VectorBase class is used by many modules.

Hi! Thanks a lot for reporting this bug. I changed it from "-" to "_" because it created issues in the crosswoz dataset. But i will work on a general way the next days and let you know here once i fixed it :)

Hi @trtd56 , i pushed a new version that works without any splitting but uses tuples instead. Could you check whether the new code works for you? Thanks!

Sorry for the late reply.
Just checked and it works fine!
Thank you!

Awesome, thanks for checking!


Comments for changes

Putting comments in line with changes to reflect better units and cleaner example.

Makes sense to me. I'll make these changes and update this PR.

All requested changes were put in place. Clear to begin review process.


Improvements/#32 use docker

resolve #32

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


bug(#21) No longer cuts URLs with paths ✂

Coverage remained the same at 83.072% when pulling d3d93eb7e12352d094f4975de8be9144113ef1a8 on bug/url-cutting into 6458393e812c66d7d03611dd129fdc55a118c3d2 on master.


Complete email and password validation with password strength bar

Related Issue

Closes: #229
Description of Changes

This PR is regarding the #229 issue, previously there was password validation but that only checks the lenght of the password, what I've added is general validation and added a password strength bar which tells us the Strength of the password.
Checklist:

[x] My code adheres to the established style guidelines of this project.
[x] I have conducted a self-review of my code.
[x] I have included comments in areas that may be difficult to understand.
[x] I have made corresponding updates to the project documentation.
[x] My changes have not introduced any new warnings.

Screenshots

Original
Updated

![

Please provide any necessary screenshots to illustrate the changes made.

Failed to compile.
13:09:54.638 |
13:09:54.638 | Module not found: Error: Can't resolve 'react-password-strength-bar' in '/vercel/path0/src/components'
13:09:54.638 |
13:09:54.638 |
13:09:54.665 | error Command failed with exit code 1.
13:09:54.665 | info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
13:09:54.686 | Error: Command "yarn run build" exited with 1
13:09:55.235 | BUILD_UTILS_SPAWN_1: Command "yarn run build" exited with 1

Any reason for deleting the Banner.jsx file?
@Bhuvanchand26

@sourabhsikarwar , I deleted that accidently and that wasn't part of issue. I dont know how , is there any problem or should I consider to do that again?


✨ Disallow post swap entry point and adapter contract calls

Closes #7

@sushiwushi Could you review this PR / approve when you get a chance. I think once this is approved I can begin merging all the PRs in for the audit findings. Please inform me if I'm mistaken, thanks!

Hi @NotJeremyLiu, the fix LGTM. Sorry for the delay in checking the fixes!


fix: add image at the cursor location and not at the end

Fixes #11
PR Changes:

added all features for the editor
fix for adding image at the cursor location and not at the end
added divider for specific command groups

@kwinyyyc  can you please look at this PR at the earliest. Am planning to integrate this into our application.

Thanks, it looks good to me.

Okay ! Thanks @kwinyyyc


QRDA Export Not Handling Relevant DateTime/Period Accurately

QRDA Export Not Handling Relevant DateTime/Period Accurately

if relevantPeriod is null or both low and high are null, then check relevantDateTime
If both relevantPeriod & relevantDateTime are null, <effectiveTime nullFlavor='UNK'/

Pull requests into cqm-parsers require the following. Submitter and reviewer should :white_check_mark: when done. For items that are not-applicable, note it's not-applicable ("N/A") and :white_check_mark:.
Submitter:

[ ] This pull request describes why these changes were made.
[ ] Internal ticket for this PR:
[ ] Internal ticket links to this PR
[ ] Code diff has been done and been reviewed
[ ] Tests are included and test edge cases
[ ] Tests have been run locally and pass

Reviewer 1:
Name:

[ ] Code is maintainable and reusable, reuses existing code and infrastructure where appropriate, and accomplishes the task’s purpose
[ ] The tests appropriately test the new code, including edge cases
[ ] You have tried to break the code

Reviewer 2:
Name:

[ ] Code is maintainable and reusable, reuses existing code and infrastructure where appropriate, and accomplishes the task’s purpose
[ ] The tests appropriately test the new code, including edge cases
[ ] You have tried to break the code

@dczulada don't think we can run ruby 2.3, 2.4 builds anymore. cqm-validators expects ruby >= 2.5
Should I remove 2.3, 2.4 builds?

@adongare I'm OK with removing the older ruby versions.
I'm also onboard with this approach for addressing exporting a scenario with a RelevantPeriod with a null Low and High value.  Without out this workaround, there is no way to export a relevant Datetime from Bonnie.
However, I believe this is actually more of an issue with Bonnie allowing for the entry of both RelevantPeriod and RelevantDateTime.
-Dave

@dczulada right, this is an issue in bonnie. it allows both relevatDateTime and relevatPeriod. I had a discussion with SMEs regarding making it mutually exclusive however they want to allow both

Sorry, I missed your edits:

Technically, an interval can have a low and high of null. However, I do not believe this is what Measure Developers intend on testing with the uncheck both low and high values in the UI.
https://cql.hl7.org/04-logicalspecification.html#interval

That is happening because of the way interval widget implemented. Interval is created even if there is no low and high
https://github.com/MeasureAuthoringTool/bonnie/blob/d9e9a9925c99fc22a2789b8f9a1a8cbeb3ea5a4d/app/assets/javascripts/views/patient_builder/inputs/interval_datetime.js.coffee#L81
This can be fixed by updating this widget as well. If low and high both null, set whole interval object to null instead of creating interval with low=null and high=null  but I'm not sure whether this is right way to create interval(because like you mentioned low and high both can be null)
Please let me know your thoughts.

@adongare
I think an ideal approach would be this.
Step 1) Update cqm-reports in the manner that you've proposed here.
Step 2) Propose updating Bonnie UI to allow a user to select either 'relevantPeriod' or 'relevantDatetime'.  This way a user can intentionally test a scenario with a null start and end point.
Step 3) If users accept approach (in step 3), revert cqm-reports to previous behavior.

I'll propose this plan to SMEs and measure devs. Thanks Dev!


Change change-warning on all services to warning, and update health checks with name, description, and severity keys

Describe the bug
I can't get change warnings to display on the FE
To Reproduce
Steps to reproduce the behavior:

Install bitcoind
hit "config"
change any of the pruning settings. all of them have change warnings defined in the spec

Expected behavior
A change warning should pop up when toggling the variant between disabled, automatic, and manual variants. In addition, the change warnings on altering the pruning sizes
Actual behavior
No warning appears
Desktop (please complete the following information):

OS and version: Ubuntu 20.04
Browser and version: FF 94.0

Additional context
here's the relevant portion of the bitcoind config spec:
    pruning:
      type: union
      name: "Pruning Settings"
      description: |
        Blockchain Pruning Options
        Reduce the blockchain size on disk
      change-warning: |
        If you set pruning to Manual and your disk is smaller than the total size of the blockchain, you MUST have something running that prunes these blocks or you may overfill your disk!
        Disabling pruning will convert your node into a full archival node. This requires a resync of the entire blockchain, a process that may take several days. Make sure you have enough free disk space or you may fill up your disk.
      tag:
        id: mode
        name: Pruning Mode
        description: |
          - Disabled: Disable pruning
          - Automatic: Limit blockchain size on disk to a certain number of megabytes
          - Manual: Prune blockchain with the "pruneblockchain" RPC
        variant-names:
          disabled: Disabled
          automatic: Automatic
          manual: Manual
      variants:
        disabled: {}
        automatic:
          size:
            type: number
            nullable: false
            name: "Max Chain Size"
            description: "Limit of blockchain size on disk."
            change-warning: "Increasing this value will require re-syncing your node."
            default: 550
            range: "[550,1000000)"
            integral: true
            units: MiB
        manual:
          size:
            type: number
            nullable: false
            name: "Failsafe Chain Size"
            description: "Prune blockchain if size expands beyond this."
            default: 65536
            range: "[550,1000000)"
            integral: true
            units: MiB
      default: disabled

git commit is 613b05788ba929679f62a03d34b15802ee4a5c37, kind of old at this point

@chrisguida please confirm that the change warnings are coming back from the network tab. If so assign to @DrewAns otherwise assign to me

Here's what's coming back under the "pruning" key. Not seeing "change-warning" anywhere. Assigning @dr-bonez
{
    "default": "disabled",
    "description": "Blockchain Pruning Options\nReduce the blockchain size on disk\n",
    "display-as": null,
    "name": "Pruning Settings",
    "tag": {
        "description": "- Disabled: Disable pruning\n- Automatic: Limit blockchain size on disk to a certain number of megabytes\n- Manual: Prune blockchain with the \"pruneblockchain\" RPC\n",
        "id": "mode",
        "name": "Pruning Mode",
        "variant-names": {
            "automatic": "Automatic",
            "disabled": "Disabled",
            "manual": "Manual"
        }
    },
    "type": "union",
    "unique-by": null,
    "variants": {
        "automatic": {
            "size": {
                "default": 550,
                "description": "Limit of blockchain size on disk.",
                "integral": true,
                "name": "Max Chain Size",
                "nullable": false,
                "range": "[550,1000000)",
                "type": "number",
                "units": "MiB"
            }
        },
        "disabled": {},
        "manual": {
            "size": {
                "default": 65536,
                "description": "Prune blockchain if size expands beyond this.",
                "integral": true,
                "name": "Failsafe Chain Size",
                "nullable": false,
                "range": "[550,1000000)",
                "type": "number",
                "units": "MiB"
            }
        }
    }
}

Also found another issue, the description under the tag key isn't rendering anywhere either:
"description": "- Disabled: Disable pruning\n- Automatic: Limit blockchain size on disk to a certain number of megabytes\n- Manual: Prune blockchain with the \"pruneblockchain\" RPC\n"

Also found another issue, the description under the tag key isn't rendering anywhere either:
"description": "- Disabled: Disable pruning\n- Automatic: Limit blockchain size on disk to a certain number of megabytes\n- Manual: Prune blockchain with the \"pruneblockchain\" RPC\n"
This one is a FE issue, unless this is no longer a valid way to display a description

Actually this one isn't showing on the FE anywhere on 02x either. Perhaps we should just remove it from the spec?

Ok I have reviewed the specification. The field is no longer called change-warning, it is called warning. This is because the warning is shown always, not just when the field is changed. @chrisguida please change the field in your spec.

Cool, this needs to be updated on all packages then @elvece

Closing as won't fix

@ProofOfKeags we still need to track the services work somewhere, reopening until that exists

Tracking here @elvece :

[ ] synapse
[ ] mastodon
[ ] bitcoind
[ ] ride-the-lightning
[ ] thunderhub
[ ] c-lightning
[ ] vaultwarden
[ ] filebrowser
[ ] pages
[ ] lnd
[ ] btcpay
[ ] photoview
[ ] proxy
[ ] spark
[ ] bar
[ ] cups
[ ] sphinx

@dr-bonez do you have any thoughts on this, or should i create a new ticket?
https://github.com/Start9Labs/embassy-os/issues/832#issuecomment-978021827

The above checked services have this complete:
- change change-warning to warning
- change critical: true to severity: critical, and critical: false to severity: warning
- all health checks will now have a name: <name> field and a description: <desc> field

Renaming ticket to reflect all changes that need to happen to manifest.yaml and config_spec.yaml, since it's convenient to update both files at once.

Waiting for spec changes to be reflected in embassy-sdk pack before rebuilding s9pks
as of 4a143ea, you should be able to repack

@dr-bonez do you have any thoughts on this, or should i create a new ticket?
https://github.com/Start9Labs/embassy-os/issues/832#issuecomment-978021827

We should fix this on the FE

@dr-bonez do you have any thoughts on this, or should i create a new ticket?
#832 (comment)

We should fix this on the FE. Create a ticket for it.

Done

These services are now all updated and published to the beta 0.3 reg https://github.com/Start9Labs/embassy-os/issues/832#issuecomment-980141784


Can't find the control in the selector

I have imported the managed solution and published it.
When trying to set the control on a single text field I can't find the control.
I suppose it's called URL Image Control but the closest I can find is this:

And it doesn't contain the variables to set like the field to get the url from.
Maybe am I missing a step? So far importing the solution has been enough for a control to appear.
Regards.

Found out it's called Control_Name_Key probably because some issue with language settings


Feature request: support options long and short to replace height and width

For documents with mixed vertical and horizontal pages.

I don't quite get it. Can you explain even further to make it more understandable?

I don't quite get it. Can you explain even further to make it more understandable?

Sorry for my poor english.
vertical A4：
height: 297mm
width: 210mm
horizontal A4:
height: 210mm
width: 297mm
If I don't set options, the output images will adaptive the pages' direction, wertical or horizontal.
However, the default number is too small.
What should I do to set them?

Sorry for my poor english.
vertical A4：
height: 297mm
width: 210mm
horizontal A4:
height: 210mm
width: 297mm
If I don't set options, the output images will adaptive the pages' direction, wertical or horizontal.
However, the default number is too small.
What should I do to set them?

xpdf works well.


Issues adding a second region

Running FROM flyio/litefs:sha-f40cf36 AS litefs
At around timestamp 1:38, you'll see I try to add a second region to litefs-indie: https://www.youtube.com/watch?v=MwIisTAkgF4
This should be all set up with litefs and it runs just fine in a single region, however then I try to add another region:
fly vol create data --region maa --size 1
fly scale count 2

And when that region starts up, I get this:
2022-11-10T23:30:57Z runner[d6dfbaae] maa [info]Starting instance
2022-11-10T23:30:59Z runner[d6dfbaae] maa [info]Configuring virtual machine
2022-11-10T23:30:59Z runner[d6dfbaae] maa [info]Pulling container image
2022-11-10T23:31:08Z runner[d6dfbaae] maa [info]Unpacking image
2022-11-10T23:31:11Z runner[d6dfbaae] maa [info]Preparing kernel init
2022-11-10T23:31:11Z runner[d6dfbaae] maa [info]Setting up volume 'data'
2022-11-10T23:31:11Z runner[d6dfbaae] maa [info]Uninitialized volume 'data', initializing...
2022-11-10T23:31:11Z runner[d6dfbaae] maa [info]Encrypting volume
2022-11-10T23:31:18Z runner[d6dfbaae] maa [info]Opening encrypted volume
2022-11-10T23:31:20Z runner[d6dfbaae] maa [info]Formatting volume
2022-11-10T23:31:20Z runner[d6dfbaae] maa [info]Configuring firecracker
2022-11-10T23:31:20Z runner[d6dfbaae] maa [info]Starting virtual machine
2022-11-10T23:31:21Z app[d6dfbaae] maa [info]Starting init (commit: 81d5330)...
2022-11-10T23:31:21Z app[d6dfbaae] maa [info]Mounting /dev/vdc at /data w/ uid: 0, gid: 0 and chmod 0755
2022-11-10T23:31:21Z app[d6dfbaae] maa [info]Preparing to run: `docker-entrypoint.sh litefs -- node ./other/start.js` as root
2022-11-10T23:31:21Z app[d6dfbaae] maa [info]2022/11/10 23:31:21 listening on [fdaa:0:23df:a7b:1449:2:d64e:2]:22 (DNS: [fdaa::3]:53)
2022-11-10T23:31:21Z app[d6dfbaae] maa [info]LiteFS v0.3.0-beta1, commit=f40cf36930e1d0b259bbbdb72c3fd3a508ff1936
2022-11-10T23:31:21Z app[d6dfbaae] maa [info]config file read from /etc/litefs.yml
2022-11-10T23:31:21Z app[d6dfbaae] maa [info]Using Consul to determine primary
2022-11-10T23:31:24Z app[d6dfbaae] maa [info]initializing consul: key=<EMAIL_ADDRESS>hostname=d6dfbaae advertise-url=http://d6dfbaae.vm.litefs-indie.internal:20202
2022-11-10T23:31:24Z app[d6dfbaae] maa [info]LiteFS mounted to: /litefs/data
2022-11-10T23:31:24Z app[d6dfbaae] maa [info]http server listening on: http://localhost:20202
2022-11-10T23:31:24Z app[d6dfbaae] maa [info]waiting to connect to cluster
2022-11-10T23:31:24Z app[d6dfbaae] maa [info]existing primary found (126b5057), connecting as replica
2022-11-10T23:31:25Z app[d6dfbaae] maa [info]replica disconnected, retrying: process ltx stream frame: write ltx file: unexpected EOF
2022-11-10T23:31:26Z app[d6dfbaae] maa [info]existing primary found (126b5057), connecting as replica
2022-11-10T23:31:27Z app[d6dfbaae] maa [info]replica disconnected, retrying: process ltx stream frame: write ltx file: unexpected EOF
2022-11-10T23:31:29Z app[d6dfbaae] maa [info]existing primary found (126b5057), connecting as replica
2022-11-10T23:31:29Z app[d6dfbaae] maa [info]replica disconnected, retrying: process ltx stream frame: write ltx file: unexpected EOF
2022-11-10T23:31:31Z app[d6dfbaae] maa [info]existing primary found (126b5057), connecting as replica
2022-11-10T23:31:31Z app[d6dfbaae] maa [info]replica disconnected, retrying: process ltx stream frame: write ltx file: unexpected EOF
2022-11-10T23:31:33Z app[d6dfbaae] maa [info]existing primary found (126b5057), connecting as replica

... over and over...

No idea what's going on, but one thing I noticed was that the first region (den) does not have a /litefs/data/.primary file in it and neither does the new region (maa).
So then I tried to restart the server. It's worth noting that before restarting the server, I got logs like this in den:
2022-11-10T23:37:51Z app[126b5057] den [info]GET /healthcheck 200 - - 6.255 ms
2022-11-10T23:37:52Z app[126b5057] den [info]stream connected
2022-11-10T23:37:52Z app[126b5057] den [info]transaction file for txid<PHONE_NUMBER>000001 no longer available, resetting to snapshot
2022-11-10T23:37:52Z app[126b5057] den [info]http: error: stream error: db="sqlite.db" err=stream ltx (tx 0): write ltx snapshot file: snapshot checksum mismatch at tx<PHONE_NUMBER>000008: f6b0461071c271bd <> eedd3e4ae711584f
2022-11-10T23:37:52Z app[126b5057] den [info]stream disconnected
2022-11-10T23:37:54Z app[126b5057] den [info]stream connected
2022-11-10T23:37:54Z app[126b5057] den [info]transaction file for txid<PHONE_NUMBER>000001 no longer available, resetting to snapshot
2022-11-10T23:37:54Z app[126b5057] den [info]http: error: stream error: db="sqlite.db" err=stream ltx (tx 0): write ltx snapshot file: snapshot checksum mismatch at tx<PHONE_NUMBER>000008: f6b0461071c271bd <> eedd3e4ae711584f
2022-11-10T23:37:54Z app[126b5057] den [info]stream disconnected
2022-11-10T23:37:57Z app[126b5057] den [info]stream connected
2022-11-10T23:37:57Z app[126b5057] den [info]transaction file for txid<PHONE_NUMBER>000001 no longer available, resetting to snapshot
2022-11-10T23:37:57Z app[126b5057] den [info]http: error: stream error: db="sqlite.db" err=stream ltx (tx 0): write ltx snapshot file: snapshot checksum mismatch at tx<PHONE_NUMBER>000008: f6b0461071c271bd <> eedd3e4ae711584f
2022-11-10T23:37:57Z app[126b5057] den [info]stream disconnected
2022-11-10T23:37:59Z app[126b5057] den [info]stream connected
2022-11-10T23:37:59Z app[126b5057] den [info]transaction file for txid<PHONE_NUMBER>000001 no longer available, resetting to snapshot
2022-11-10T23:37:59Z app[126b5057] den [info]http: error: stream error: db="sqlite.db" err=stream ltx (tx 0): write ltx snapshot file: snapshot checksum mismatch at tx<PHONE_NUMBER>000008: f6b0461071c271bd <> eedd3e4ae711584f
2022-11-10T23:37:59Z app[126b5057] den [info]stream disconnected
2022-11-10T23:38:01Z app[126b5057] den [info]stream connected
2022-11-10T23:38:01Z app[126b5057] den [info]transaction file for txid<PHONE_NUMBER>000001 no longer available, resetting to snapshot
2022-11-10T23:38:01Z app[126b5057] den [info]http: error: stream error: db="sqlite.db" err=stream ltx (tx 0): write ltx snapshot file: snapshot checksum mismatch at tx<PHONE_NUMBER>000008: f6b0461071c271bd <> eedd3e4ae711584f
2022-11-10T23:38:01Z app[126b5057] den [info]stream disconnected
2022-11-10T23:38:03Z app[126b5057] den [info]stream connected
2022-11-10T23:38:03Z app[126b5057] den [info]transaction file for txid<PHONE_NUMBER>000001 no longer available, resetting to snapshot
2022-11-10T23:38:03Z app[126b5057] den [info]http: error: stream error: db="sqlite.db" err=stream ltx (tx 0): write ltx snapshot file: snapshot checksum mismatch at tx<PHONE_NUMBER>000008: f6b0461071c271bd <> eedd3e4ae711584f
2022-11-10T23:38:03Z app[126b5057] den [info]stream disconnected
2022-11-10T23:38:05Z app[126b5057] den [info]stream connected
2022-11-10T23:38:05Z app[126b5057] den [info]transaction file for txid<PHONE_NUMBER>000001 no longer available, resetting to snapshot
2022-11-10T23:38:05Z app[126b5057] den [info]http: error: stream error: db="sqlite.db" err=stream ltx (tx 0): write ltx snapshot file: snapshot checksum mismatch at tx<PHONE_NUMBER>000008: f6b0461071c271bd <> eedd3e4ae711584f
2022-11-10T23:38:05Z app[126b5057] den [info]stream disconnected
2022-11-10T23:38:07Z runner[126b5057] den [info]Shutting down virtual machine
2022-11-10T23:38:07Z app[126b5057] den [info]stream connected
2022-11-10T23:38:07Z app[126b5057] den [info]transaction file for txid<PHONE_NUMBER>000001 no longer available, resetting to snapshot
2022-11-10T23:38:07Z app[126b5057] den [info]http: error: stream error: db="sqlite.db" err=stream ltx (tx 0): write ltx snapshot file: snapshot checksum mismatch at tx<PHONE_NUMBER>000008: f6b0461071c271bd <> eedd3e4ae711584f
2022-11-10T23:38:07Z app[126b5057] den [info]stream disconnected
2022-11-10T23:38:07Z app[126b5057] den [info]Sending signal SIGINT to main child process w/ PID 528
2022-11-10T23:38:07Z app[126b5057] den [info]sending signal to exec process
2022-11-10T23:38:07Z app[126b5057] den [info]waiting for exec process to close
2022-11-10T23:38:07Z app[126b5057] den [info]signal received, litefs shutting down
2022-11-10T23:38:09Z app[126b5057] den [info]http: error: context canceled
2022-11-10T23:38:11Z app[126b5057] den [info]http: error: context canceled

So I restarted:
fly restart litefs-indie

And when they came back up, maa took primary (there was no .primary file) and den gave me this:
2022-11-10T23:38:16Z runner[ab1d8d25] den [info]Starting instance
2022-11-10T23:38:17Z runner[ab1d8d25] den [info]Configuring virtual machine
2022-11-10T23:38:17Z runner[ab1d8d25] den [info]Pulling container image
2022-11-10T23:38:17Z runner[ab1d8d25] den [info]Unpacking image
2022-11-10T23:38:17Z runner[ab1d8d25] den [info]Preparing kernel init
2022-11-10T23:38:17Z runner[ab1d8d25] den [info]Setting up volume 'data'
2022-11-10T23:38:17Z runner[ab1d8d25] den [info]Opening encrypted volume
2022-11-10T23:38:17Z runner[ab1d8d25] den [info]Configuring firecracker
2022-11-10T23:38:17Z runner[ab1d8d25] den [info]Starting virtual machine
2022-11-10T23:38:17Z app[ab1d8d25] den [info]Starting init (commit: 81d5330)...
2022-11-10T23:38:17Z app[ab1d8d25] den [info]Mounting /dev/vdc at /data w/ uid: 0, gid: 0 and chmod 0755
2022-11-10T23:38:17Z app[ab1d8d25] den [info]Preparing to run: `docker-entrypoint.sh litefs -- node ./other/start.js` as root
2022-11-10T23:38:17Z app[ab1d8d25] den [info]2022/11/10 23:38:17 listening on [fdaa:0:23df:a7b:d826:2:d5f1:2]:22 (DNS: [fdaa::3]:53)
2022-11-10T23:38:17Z app[ab1d8d25] den [info]config file read from /etc/litefs.yml
2022-11-10T23:38:17Z app[ab1d8d25] den [info]LiteFS v0.3.0-beta1, commit=f40cf36930e1d0b259bbbdb72c3fd3a508ff1936
2022-11-10T23:38:17Z app[ab1d8d25] den [info]Using Consul to determine primary
2022-11-10T23:38:20Z app[ab1d8d25] den [info]initializing consul: key=<EMAIL_ADDRESS>hostname=ab1d8d25 advertise-url=http://ab1d8d25.vm.litefs-indie.internal:20202
2022-11-10T23:38:20Z app[ab1d8d25] den [info]ERROR: cannot open store: open databases: open database("sqlite.db"): verify database file: database checksum (f6b0461071c271bd) does not match latest LTX checksum (eedd3e4ae711584f)
2022-11-10T23:38:20Z app[ab1d8d25] den [info]Starting clean up.
2022-11-10T23:38:20Z app[ab1d8d25] den [info]Umounting /dev/vdc from /data

... over and over ...

There's that nasty checksum error again 😭
Also, I lost all my data that was in the SQLite db in den before restarting.
So at this point, I have no idea what to try to get litefs working with multiple regions. Please advise.
P.S. At one point, I did get a .primary file and the contents were a couple random characters. I was expecting a region name because I believe that's what I need for the fly-replay header. I thought that I was supposed to determine whether I'm in the primary region based on whether $FLY_REGION is equal to the contents of that file. Please help correct my understanding. Thanks!
I realized that we're working with instances, not regions and I got that figured out 😅

I tried again (deleted and re-created volumes) at ~2:18 in the stream and here's what happened:
2022-11-11T00:09:04.276 runner[1f97497a] maa [info] Starting instance
2022-11-11T00:09:06.268 runner[1f97497a] maa [info] Configuring virtual machine
2022-11-11T00:09:06.279 runner[1f97497a] maa [info] Pulling container image
2022-11-11T00:09:14.939 runner[1f97497a] maa [info] Unpacking image
2022-11-11T00:09:17.637 runner[1f97497a] maa [info] Preparing kernel init
2022-11-11T00:09:17.878 runner[1f97497a] maa [info] Setting up volume 'data'
2022-11-11T00:09:17.883 runner[1f97497a] maa [info] Uninitialized volume 'data', initializing...
2022-11-11T00:09:19.220 runner[1f97497a] maa [info] Encrypting volume
2022-11-11T00:09:25.141 runner[1f97497a] maa [info] Opening encrypted volume
2022-11-11T00:09:27.136 runner[1f97497a] maa [info] Formatting volume
2022-11-11T00:09:27.346 runner[1f97497a] maa [info] Configuring firecracker
2022-11-11T00:09:28.041 runner[1f97497a] maa [info] Starting virtual machine
2022-11-11T00:09:28.339 app[1f97497a] maa [info] Starting init (commit: 81d5330)...
2022-11-11T00:09:28.367 app[1f97497a] maa [info] Mounting /dev/vdc at /data w/ uid: 0, gid: 0 and chmod 0755
2022-11-11T00:09:28.370 app[1f97497a] maa [info] Preparing to run: `docker-entrypoint.sh litefs -- node ./other/start.js` as root
2022-11-11T00:09:28.396 app[1f97497a] maa [info] 2022/11/11 00:09:28 listening on [fdaa:0:23df:a7b:1449:2:d67a:2]:22 (DNS: [fdaa::3]:53)
2022-11-11T00:09:28.443 app[1f97497a] maa [info] LiteFS v0.3.0-beta1, commit=f40cf36930e1d0b259bbbdb72c3fd3a508ff1936
2022-11-11T00:09:28.443 app[1f97497a] maa [info] config file read from /etc/litefs.yml
2022-11-11T00:09:28.443 app[1f97497a] maa [info] Using Consul to determine primary
2022-11-11T00:09:31.302 app[1f97497a] maa [info] initializing consul: key=<EMAIL_ADDRESS>hostname=1f97497a advertise-url=http://1f97497a.vm.litefs-indie.internal:20202
2022-11-11T00:09:31.312 app[1f97497a] maa [info] LiteFS mounted to: /litefs/data
2022-11-11T00:09:31.312 app[1f97497a] maa [info] http server listening on: http://localhost:20202
2022-11-11T00:09:31.312 app[1f97497a] maa [info] waiting to connect to cluster
2022-11-11T00:09:31.530 app[1f97497a] maa [info] existing primary found (81024706), connecting as replica
2022-11-11T00:09:32.558 app[1f97497a] maa [info] snapshot received for "sqlite.db", removing other ltx files:<PHONE_NUMBER>000001-0000000000000001.ltx
2022-11-11T00:09:35.858 app[1f97497a] maa [info] connected to cluster, ready
2022-11-11T00:09:35.858 app[1f97497a] maa [info] starting subprocess: node [./other/start.js]
2022-11-11T00:09:35.866 app[1f97497a] maa [info] waiting for signal or subprocess to exit
2022-11-11T00:09:36.048 app[1f97497a] maa [info] Found primary instance in .primary file: 81024706
2022-11-11T00:09:36.049 app[1f97497a] maa [info] Instance in (1f97497a) maa is not primary (the primary instance is 81024706).
I don't think I changed anything meaningful, but maybe... Watch the stream closely I guess 😅

Then at around 2:35 of the live stream, I pushed a new deploy and maa got up and going fine, but den had some issues:
2022-11-11T00:28:35.505 runner[451b551f] den [info] Starting instance
2022-11-11T00:28:35.595 runner[451b551f] den [info] Configuring virtual machine
2022-11-11T00:28:35.597 runner[451b551f] den [info] Pulling container image
2022-11-11T00:28:35.875 runner[451b551f] den [info] Unpacking image
2022-11-11T00:28:35.884 runner[451b551f] den [info] Preparing kernel init
2022-11-11T00:28:36.025 runner[451b551f] den [info] Setting up volume 'data'
2022-11-11T00:28:36.028 runner[451b551f] den [info] Opening encrypted volume
2022-11-11T00:28:36.178 runner[451b551f] den [info] Configuring firecracker
2022-11-11T00:28:36.233 runner[451b551f] den [info] Starting virtual machine
2022-11-11T00:28:36.414 app[451b551f] den [info] Starting init (commit: 81d5330)...
2022-11-11T00:28:36.436 app[451b551f] den [info] Mounting /dev/vdc at /data w/ uid: 0, gid: 0 and chmod 0755
2022-11-11T00:28:36.439 app[451b551f] den [info] Preparing to run: `docker-entrypoint.sh litefs -- node ./other/start.js` as root
2022-11-11T00:28:36.465 app[451b551f] den [info] 2022/11/11 00:28:36 listening on [fdaa:0:23df:a7b:d826:2:d679:2]:22 (DNS: [fdaa::3]:53)
2022-11-11T00:28:36.510 app[451b551f] den [info] config file read from /etc/litefs.yml
2022-11-11T00:28:36.510 app[451b551f] den [info] LiteFS v0.3.0-beta1, commit=f40cf36930e1d0b259bbbdb72c3fd3a508ff1936
2022-11-11T00:28:36.511 app[451b551f] den [info] Using Consul to determine primary
2022-11-11T00:28:38.737 app[451b551f] den [info] initializing consul: key=<EMAIL_ADDRESS>hostname=451b551f advertise-url=http://451b551f.vm.litefs-indie.internal:20202
2022-11-11T00:28:38.740 app[451b551f] den [info] ERROR: cannot open store: open databases: open database("sqlite.db"): verify database file: database checksum (cdf639f125d4ff14) does not match latest LTX checksum (d59b41abb307d6e6)
2022-11-11T00:28:39.454 app[451b551f] den [info] Starting clean up.
2022-11-11T00:28:39.455 app[451b551f] den [info] Umounting /dev/vdc from /data

I don't know what I'm doing wrong, but hopefully the litefs-indie app will be easier for you to debug: https://github.com/kentcdodds/litefs-indie

Ultimately what I'm trying to make work is kcd-staging (in preparation for kcd). In case it's helpful in this issue, I added this same config in that repo and got it running in one region just fine, but when I tried to spin up a second region (with a brand new volume) I ended up with this:
2022-11-11T02:47:38Z runner[037cde23] maa [info]Unpacking image
2022-11-11T02:47:42Z runner[037cde23] maa [info]Preparing kernel init
2022-11-11T02:47:42Z runner[037cde23] maa [info]Setting up volume 'data'
2022-11-11T02:47:42Z runner[037cde23] maa [info]Uninitialized volume 'data', initializing...
2022-11-11T02:47:43Z runner[037cde23] maa [info]Encrypting volume
2022-11-11T02:47:45Z app[58e0a562] den [info]HEAD / 200 88096 - 139.327 ms
2022-11-11T02:47:45Z app[58e0a562] den [info]GET /healthcheck 200 - - 163.434 ms
2022-11-11T02:47:48Z runner[037cde23] maa [info]Opening encrypted volume
2022-11-11T02:47:50Z runner[037cde23] maa [info]Formatting volume
2022-11-11T02:47:50Z runner[037cde23] maa [info]Configuring firecracker
2022-11-11T02:47:50Z runner[037cde23] maa [info]Starting virtual machine
2022-11-11T02:47:51Z app[037cde23] maa [info]Starting init (commit: 81d5330)...
2022-11-11T02:47:51Z app[037cde23] maa [info]Mounting /dev/vdc at /data w/ uid: 0, gid: 0 and chmod 0755
2022-11-11T02:47:51Z app[037cde23] maa [info]Preparing to run: `docker-entrypoint.sh litefs -- node ./other/start.js` as root
2022-11-11T02:47:51Z app[037cde23] maa [info]2022/11/11 02:47:51 listening on [fdaa:0:23df:a7b:1449:2:d707:2]:22 (DNS: [fdaa::3]:53)
2022-11-11T02:47:51Z app[037cde23] maa [info]LiteFS v0.3.0-beta1, commit=f40cf36930e1d0b259bbbdb72c3fd3a508ff1936
2022-11-11T02:47:51Z app[037cde23] maa [info]config file read from /etc/litefs.yml
2022-11-11T02:47:51Z app[037cde23] maa [info]Using Consul to determine primary
2022-11-11T02:47:53Z app[037cde23] maa [info]initializing consul: key=<EMAIL_ADDRESS>hostname=037cde23 advertise-url=http://037cde23.vm.kcd-staging.internal:20202
2022-11-11T02:47:53Z app[037cde23] maa [info]LiteFS mounted to: /litefs/data
2022-11-11T02:47:53Z app[037cde23] maa [info]http server listening on: http://localhost:20202
2022-11-11T02:47:53Z app[037cde23] maa [info]waiting to connect to cluster
2022-11-11T02:47:54Z app[037cde23] maa [info]existing primary found (58e0a562), connecting as replica
2022-11-11T02:47:54Z app[58e0a562] den [info]stream connected
2022-11-11T02:47:54Z app[58e0a562] den [info]transaction file for txid<PHONE_NUMBER>000001 no longer available, resetting to snapshot
2022-11-11T02:47:55Z app[58e0a562] den [info]HEAD / 200 89885 - 32.243 ms
2022-11-11T02:47:55Z app[58e0a562] den [info]GET /healthcheck 200 - - 40.063 ms
2022-11-11T02:48:05Z app[58e0a562] den [info]HEAD / 200 87886 - 27.752 ms
2022-11-11T02:48:05Z app[58e0a562] den [info]GET /healthcheck 200 - - 39.101 ms
2022-11-11T02:48:16Z app[58e0a562] den [info]HEAD / 200 92886 - 24.717 ms
2022-11-11T02:48:16Z app[58e0a562] den [info]GET /healthcheck 200 - - 32.817 ms
2022-11-11T02:48:26Z app[58e0a562] den [info]HEAD / 200 92584 - 133.178 ms
2022-11-11T02:48:26Z app[58e0a562] den [info]GET /healthcheck 200 - - 154.464 ms
2022-11-11T02:48:36Z app[58e0a562] den [info]HEAD / 200 90042 - 34.224 ms
2022-11-11T02:48:36Z app[58e0a562] den [info]GET /healthcheck 200 - - 43.565 ms
2022-11-11T02:48:46Z app[58e0a562] den [info]HEAD / 200 90352 - 58.980 ms
2022-11-11T02:48:46Z app[58e0a562] den [info]GET /healthcheck 200 - - 67.247 ms
2022-11-11T02:48:48Z app[58e0a562] den [info]http: error: stream error: db="sqlite.db" err=stream ltx (tx 0): write ltx snapshot file: snapshot checksum mismatch at tx<PHONE_NUMBER>000289: 80dc39cb091d1c47 <> ab7a0e009f85e969
2022-11-11T02:48:48Z app[58e0a562] den [info]stream disconnected
2022-11-11T02:48:50Z app[037cde23] maa [info]replica disconnected, retrying: process ltx stream frame: write ltx file: unexpected EOF
2022-11-11T02:48:52Z app[037cde23] maa [info]existing primary found (58e0a562), connecting as replica
2022-11-11T02:48:52Z app[58e0a562] den [info]stream connected
2022-11-11T02:48:52Z app[58e0a562] den [info]transaction file for txid<PHONE_NUMBER>000001 no longer available, resetting to snapshot
2022-11-11T02:48:56Z app[58e0a562] den [info]HEAD / 200 90478 - 26.509 ms
2022-11-11T02:48:56Z app[58e0a562] den [info]GET /healthcheck 200 - - 34.554 ms
2022-11-11T02:49:06Z app[58e0a562] den [info]HEAD / 200 89473 - 122.479 ms
2022-11-11T02:49:06Z app[58e0a562] den [info]GET /healthcheck 200 - - 147.410 ms
2022-11-11T02:49:16Z app[58e0a562] den [info]HEAD / 200 87706 - 28.204 ms
2022-11-11T02:49:16Z app[58e0a562] den [info]GET /healthcheck 200 - - 34.810 ms
2022-11-11T02:49:17Z runner[037cde23] maa [info]Shutting down virtual machine
2022-11-11T02:49:22Z app[037cde23] maa [info]Sending signal SIGINT to main child process w/ PID 530
2022-11-11T02:49:26Z app[58e0a562] den [info]HEAD / 200 89948 - 37.714 ms
2022-11-11T02:49:26Z app[58e0a562] den [info]GET /healthcheck 200 - - 46.174 ms
2022-11-11T02:49:36Z app[58e0a562] den [info]HEAD / 200 88882 - 47.604 ms
2022-11-11T02:49:36Z app[58e0a562] den [info]GET /healthcheck 200 - - 56.698 ms
2022-11-11T02:49:38Z runner[037cde23] maa [info]Starting instance
2022-11-11T02:49:40Z runner[037cde23] maa [info]Configuring virtual machine
2022-11-11T02:49:40Z runner[037cde23] maa [info]Pulling container image
2022-11-11T02:49:42Z runner[037cde23] maa [info]Unpacking image
2022-11-11T02:49:42Z runner[037cde23] maa [info]Preparing kernel init
2022-11-11T02:49:42Z runner[037cde23] maa [info]Setting up volume 'data'
2022-11-11T02:49:42Z runner[037cde23] maa [info]Opening encrypted volume
2022-11-11T02:49:42Z runner[037cde23] maa [info]Configuring firecracker
2022-11-11T02:49:43Z runner[037cde23] maa [info]Starting virtual machine
2022-11-11T02:49:43Z app[037cde23] maa [info]Starting init (commit: 81d5330)...
2022-11-11T02:49:43Z app[037cde23] maa [info]Mounting /dev/vdc at /data w/ uid: 0, gid: 0 and chmod 0755
2022-11-11T02:49:43Z app[037cde23] maa [info]Preparing to run: `docker-entrypoint.sh litefs -- node ./other/start.js` as root
2022-11-11T02:49:43Z app[037cde23] maa [info]2022/11/11 02:49:43 listening on [fdaa:0:23df:a7b:1449:2:d707:2]:22 (DNS: [fdaa::3]:53)
2022-11-11T02:49:44Z app[037cde23] maa [info]LiteFS v0.3.0-beta1, commit=f40cf36930e1d0b259bbbdb72c3fd3a508ff1936
2022-11-11T02:49:44Z app[037cde23] maa [info]config file read from /etc/litefs.yml
2022-11-11T02:49:44Z app[037cde23] maa [info]Using Consul to determine primary
2022-11-11T02:49:46Z app[037cde23] maa [info]initializing consul: key=<EMAIL_ADDRESS>hostname=037cde23 advertise-url=http://037cde23.vm.kcd-staging.internal:20202
2022-11-11T02:49:46Z app[037cde23] maa [info]LiteFS mounted to: /litefs/data
2022-11-11T02:49:46Z app[037cde23] maa [info]http server listening on: http://localhost:20202
2022-11-11T02:49:46Z app[037cde23] maa [info]waiting to connect to cluster
2022-11-11T02:49:46Z app[037cde23] maa [info]existing primary found (58e0a562), connecting as replica
2022-11-11T02:49:46Z app[58e0a562] den [info]HEAD / 200 89453 - 113.630 ms
2022-11-11T02:49:46Z app[58e0a562] den [info]GET /healthcheck 200 - - 126.227 ms
2022-11-11T02:49:47Z app[58e0a562] den [info]stream connected
2022-11-11T02:49:47Z app[58e0a562] den [info]transaction file for txid<PHONE_NUMBER>000001 no longer available, resetting to snapshot
2022-11-11T02:49:56Z app[58e0a562] den [info]HEAD / 200 88405 - 28.275 ms
2022-11-11T02:49:56Z app[58e0a562] den [info]GET /healthcheck 200 - - 35.323 ms
2022-11-11T02:49:57Z app[58e0a562] den [info]http: error: stream error: db="sqlite.db" err=stream ltx (tx 0): write ltx snapshot file: encode page frame: write page data: write tcp [fdaa:0:23df:a7b:d826:2:d698:2]:20202->[fdaa:0:23df:a7b:1449:2:d707:2]:39988: write: connection reset by peer
2022-11-11T02:49:57Z app[58e0a562] den [info]stream disconnected
2022-11-11T02:50:06Z app[58e0a562] den [info]HEAD / 200 90959 - 30.608 ms
2022-11-11T02:50:06Z app[58e0a562] den [info]GET /healthcheck 200 - - 38.972 ms
2022-11-11T02:50:16Z app[58e0a562] den [info]HEAD / 200 103576 - 30.451 ms
2022-11-11T02:50:16Z app[58e0a562] den [info]GET /healthcheck 200 - - 40.096 ms
2022-11-11T02:50:27Z app[58e0a562] den [info]HEAD / 200 119570 - 133.237 ms
2022-11-11T02:50:27Z app[58e0a562] den [info]GET /healthcheck 200 - - 143.371 ms
2022-11-11T02:50:37Z app[58e0a562] den [info]HEAD / 200 103991 - 29.256 ms
2022-11-11T02:50:37Z app[58e0a562] den [info]GET /healthcheck 200 - - 40.859 ms
2022-11-11T02:50:46Z runner[037cde23] maa [info]Shutting down virtual machine
2022-11-11T02:50:46Z app[037cde23] maa [info]Sending signal SIGINT to main child process w/ PID 530

So I deleted the volume and scaled back down.
Looks like it found the primary but just couldn't connect as a replica for some reason 🤷♂

From the sound of this, once #13 is supported then all my problems should be over 🤞 🙏

I've got matching errors with my Remix project using LiteFS too. :sweat_smile:

Logs
2022-11-11T17:26:02.733 app[50616fcd] fra [info] Starting init (commit: 81d5330)...

2022-11-11T17:26:02.761 app[50616fcd] fra [info] Preparing to run: `docker-entrypoint.sh litefs -- node ./start.js` as root

2022-11-11T17:26:02.793 app[50616fcd] fra [info] 2022/11/11 17:26:02 listening on [fdaa:0:9732:a7b:86:5061:6fcd:2]:22 (DNS: [fdaa::3]:53)

2022-11-11T17:26:02.841 app[50616fcd] fra [info] LiteFS v0.3.0-beta1, commit=f40cf36930e1d0b259bbbdb72c3fd3a508ff1936

2022-11-11T17:26:02.841 app[50616fcd] fra [info] config file read from /app/litefs.yml

2022-11-11T17:26:02.841 app[50616fcd] fra [info] Using Consul to determine primary

2022-11-11T17:26:03.963 app[50616fcd] fra [info] initializing consul: key=<EMAIL_ADDRESS>hostname=50616fcd advertise-url=http://50616fcd.vm.fontsource.internal:20202

2022-11-11T17:26:03.970 app[50616fcd] fra [info] LiteFS mounted to: /litefs/data

2022-11-11T17:26:03.970 app[50616fcd] fra [info] http server listening on: http://localhost:20202

2022-11-11T17:26:03.970 app[50616fcd] fra [info] waiting to connect to cluster

2022-11-11T17:26:03.972 app[50616fcd] fra [info] existing primary found (6414fe27), connecting as replica

2022-11-11T17:26:04.205 app[6414fe27] sjc [info] stream connected

2022-11-11T17:26:04.205 app[6414fe27] sjc [info] transaction file for txid<PHONE_NUMBER>000001 no longer available, resetting to snapshot

2022-11-11T17:26:04.205 app[6414fe27] sjc [info] http: error: stream error: db="sqlite.db" err=stream ltx (tx 0): write ltx snapshot file: encode ltx header: commit record required

2022-11-11T17:26:04.205 app[6414fe27] sjc [info] stream disconnected

2022-11-11T17:26:04.279 app[50616fcd] fra [info] replica disconnected, retrying: process ltx stream frame: peek ltx header: unmarshal header: invalid LTX file

2022-11-11T17:26:05.398 app[50616fcd] fra [info] existing primary found (6414fe27), connecting as replica

2022-11-11T17:26:05.473 app[6414fe27] sjc [info] stream connected

2022-11-11T17:26:05.473 app[6414fe27] sjc [info] transaction file for txid<PHONE_NUMBER>000001 no longer available, resetting to snapshot

2022-11-11T17:26:05.473 app[6414fe27] sjc [info] http: error: stream error: db="sqlite.db" err=stream ltx (tx 0): write ltx snapshot file: encode ltx header: commit record required

2022-11-11T17:26:05.473 app[6414fe27] sjc [info] stream disconnected

2022-11-11T17:26:05.546 app[50616fcd] fra [info] replica disconnected, retrying: process ltx stream frame: peek ltx header: unmarshal header: invalid LTX file

2022-11-11T17:26:07.073 app[50616fcd] fra [info] existing primary found (6414fe27), connecting as replica

2022-11-11T17:26:07.148 app[6414fe27] sjc [info] stream connected

2022-11-11T17:26:07.148 app[6414fe27] sjc [info] transaction file for txid<PHONE_NUMBER>000001 no longer available, resetting to snapshot

2022-11-11T17:26:07.148 app[6414fe27] sjc [info] http: error: stream error: db="sqlite.db" err=stream ltx (tx 0): write ltx snapshot file: encode ltx header: commit record required

2022-11-11T17:26:07.148 app[6414fe27] sjc [info] stream disconnected

2022-11-11T17:26:07.221 app[50616fcd] fra [info] replica disconnected, retrying: process ltx stream frame: peek ltx header: unmarshal header: invalid LTX file

2022-11-11T17:26:08.235 app[50616fcd] fra [info] existing primary found (6414fe27), connecting as replica

2022-11-11T17:26:08.310 app[6414fe27] sjc [info] stream connected

2022-11-11T17:26:08.310 app[6414fe27] sjc [info] transaction file for txid<PHONE_NUMBER>000001 no longer available, resetting to snapshot

2022-11-11T17:26:08.310 app[6414fe27] sjc [info] http: error: stream error: db="sqlite.db" err=stream ltx (tx 0): write ltx snapshot file: encode ltx header: commit record required

2022-11-11T17:26:08.310 app[6414fe27] sjc [info] stream disconnected

2022-11-11T17:26:08.383 app[50616fcd] fra [info] replica disconnected, retrying: process ltx stream frame: peek ltx header: unmarshal header: invalid LTX file

My setup is very similar to Kent's stuff (tysm @kentcdodds for going through all the troubleshooting!). Although my stack is different as I use Knex and better-sqlite3 instead of Prisma. Hopefully, this helps in giving more general context.
GH Actions
Repository

Thanks for submitting that, @ayuhito. The PRs are working locally for me. I'm going to grab some lunch and then test against the litefs-indie deployment on Fly when i get back. 👍

@benbjohnson, I just realised while the circumstances are similar, I have different errors from Kent's errors. Not sure if you missed that. Lmk if we should move this to a separate issue if the scope is different.
Kent
2022-11-10T23:38:03Z app[126b5057] den [info]stream connected
2022-11-10T23:38:03Z app[126b5057] den [info]transaction file for txid<PHONE_NUMBER>000001 no longer available, resetting to snapshot
2022-11-10T23:38:03Z app[126b5057] den [info]http: error: stream error: db="sqlite.db" err=stream ltx (tx 0): write ltx snapshot file: snapshot checksum mismatch at tx<PHONE_NUMBER>000008: f6b0461071c271bd <> eedd3e4ae711584f
2022-11-10T23:38:03Z app[126b5057] den [info]stream disconnected

Mine
2022-11-11T18:50:47.527 app[6414fe27] sjc [info] stream connected
2022-11-11T18:50:47.528 app[6414fe27] sjc [info] transaction file for txid<PHONE_NUMBER>000001 no longer available, resetting to snapshot
2022-11-11T18:50:47.528 app[6414fe27] sjc [info] http: error: stream error: db="sqlite.db" err=stream ltx (tx 0): write ltx snapshot file: encode ltx header: commit record required
2022-11-11T18:50:47.528 app[6414fe27] sjc [info] stream disconnected

Mine has a commit record required error instead of checksum mismatch. Is this perhaps related to my lack of volumes to persist the DB? This error is also still present after trying out the latest PRs too.

Well, I had to blow up the volumes one last time, but it seems to be working ok now. One thing I ran into was that the primary can change while an instance is running, so you need to make sure that you don't just set the primary instance in a life-long environment variable.
One thing I'm a bit concerned about. Does fly ensure 0 downtime on restarts and redeploys? How does it manage this? If so, I'm guessing it just keeps the existing primary up until a new version of the app is ready to become primary at which point, that new version becomes primary and all the other nodes are informed? Something like that?

Also, in case it's interesting, here are the logs from when I restarted (it looks like both instances may have restarted multiple times):
2022-11-11T19:01:22.793 runner[934eeb93] den [info] Shutting down virtual machine
2022-11-11T19:01:22.867 app[934eeb93] den [info] Sending signal SIGINT to main child process w/ PID 529
2022-11-11T19:01:22.868 app[934eeb93] den [info] sending signal to exec process
2022-11-11T19:01:22.868 app[934eeb93] den [info] waiting for exec process to close
2022-11-11T19:01:22.869 app[934eeb93] den [info] signal received, litefs shutting down
2022-11-11T19:01:22.869 app[934eeb93] den [info] stream disconnected
2022-11-11T19:01:24.476 app[96c01dd5] maa [info] existing primary found (934eeb93), connecting as replica
2022-11-11T19:01:24.596 app[934eeb93] den [info] http: error: context canceled
2022-11-11T19:01:24.733 app[96c01dd5] maa [info] replica disconnected, retrying: connect to primary: invalid response: code=503 ('http://934eeb93.vm.litefs-indie.internal:20202')
2022-11-11T19:01:25.972 app[96c01dd5] maa [info] existing primary found (934eeb93), connecting as replica
2022-11-11T19:01:26.092 app[934eeb93] den [info] http: error: context canceled
2022-11-11T19:01:26.226 app[96c01dd5] maa [info] replica disconnected, retrying: connect to primary: invalid response: code=503 ('http://934eeb93.vm.litefs-indie.internal:20202')
2022-11-11T19:01:27.833 app[96c01dd5] maa [info] existing primary found (934eeb93), connecting as replica
2022-11-11T19:01:30.877 app[96c01dd5] maa [info] HEAD / 200 - - 4.501 ms
2022-11-11T19:01:30.878 app[96c01dd5] maa [info] GET /healthcheck 200 - - 10.841 ms
2022-11-11T19:01:34.205 runner[c318b330] den [info] Starting instance
2022-11-11T19:01:34.318 runner[c318b330] den [info] Configuring virtual machine
2022-11-11T19:01:34.320 runner[c318b330] den [info] Pulling container image
2022-11-11T19:01:34.568 runner[c318b330] den [info] Unpacking image
2022-11-11T19:01:34.580 runner[c318b330] den [info] Preparing kernel init
2022-11-11T19:01:34.705 runner[c318b330] den [info] Setting up volume 'data'
2022-11-11T19:01:34.708 runner[c318b330] den [info] Opening encrypted volume
2022-11-11T19:01:34.834 runner[c318b330] den [info] Configuring firecracker
2022-11-11T19:01:34.904 runner[c318b330] den [info] Starting virtual machine
2022-11-11T19:01:35.076 app[c318b330] den [info] Starting init (commit: 81d5330)...
2022-11-11T19:01:35.102 app[c318b330] den [info] Mounting /dev/vdc at /data w/ uid: 0, gid: 0 and chmod 0755
2022-11-11T19:01:35.116 app[c318b330] den [info] Preparing to run: `docker-entrypoint.sh litefs -- node ./other/start.js` as root
2022-11-11T19:01:35.138 app[c318b330] den [info] 2022/11/11 19:01:35 listening on [fdaa:0:23df:a7b:d826:2:da30:2]:22 (DNS: [fdaa::3]:53)
2022-11-11T19:01:35.186 app[c318b330] den [info] config file read from /etc/litefs.yml
2022-11-11T19:01:35.186 app[c318b330] den [info] LiteFS commit=fabf62dacbc590ed94d79493a3691deb19b153bd
2022-11-11T19:01:35.187 app[c318b330] den [info] Using Consul to determine primary
2022-11-11T19:01:37.360 app[c318b330] den [info] initializing consul: key=<EMAIL_ADDRESS>hostname=c318b330 advertise-url=http://c318b330.vm.litefs-indie.internal:20202
2022-11-11T19:01:37.370 app[c318b330] den [info] LiteFS mounted to: /litefs/data
2022-11-11T19:01:37.370 app[c318b330] den [info] http server listening on: http://localhost:20202
2022-11-11T19:01:37.370 app[c318b330] den [info] waiting to connect to cluster
2022-11-11T19:01:37.420 app[c318b330] den [info] existing primary found (934eeb93), connecting as replica
2022-11-11T19:01:37.426 app[c318b330] den [info] replica disconnected, retrying: connect to primary: Post "http://934eeb93.vm.litefs-indie.internal:20202/stream": dial tcp: lookup 934eeb93.vm.litefs-indie.internal on [fdaa::3]:53: no such host ('http://934eeb93.vm.litefs-indie.internal:20202')
2022-11-11T19:01:37.370 app[96c01dd5] maa [info] replica disconnected, retrying: connect to primary: Post "http://934eeb93.vm.litefs-indie.internal:20202/stream": read tcp [fdaa:0:23df:a7b:14bf:2:da39:2]:54568->[fdaa:0:23df:a7b:d826:2:da30:2]:20202: read: connection reset by peer ('http://934eeb93.vm.litefs-indie.internal:20202')
2022-11-11T19:01:38.490 app[c318b330] den [info] existing primary found (934eeb93), connecting as replica
2022-11-11T19:01:38.499 app[c318b330] den [info] replica disconnected, retrying: connect to primary: Post "http://934eeb93.vm.litefs-indie.internal:20202/stream": dial tcp: lookup 934eeb93.vm.litefs-indie.internal on [fdaa::3]:53: no such host ('http://934eeb93.vm.litefs-indie.internal:20202')
2022-11-11T19:01:38.895 app[96c01dd5] maa [info] existing primary found (934eeb93), connecting as replica
2022-11-11T19:01:38.910 app[96c01dd5] maa [info] replica disconnected, retrying: connect to primary: Post "http://934eeb93.vm.litefs-indie.internal:20202/stream": dial tcp: lookup 934eeb93.vm.litefs-indie.internal on [fdaa::3]:53: no such host ('http://934eeb93.vm.litefs-indie.internal:20202')
2022-11-11T19:01:39.585 app[c318b330] den [info] existing primary found (934eeb93), connecting as replica
2022-11-11T19:01:39.592 app[c318b330] den [info] replica disconnected, retrying: connect to primary: Post "http://934eeb93.vm.litefs-indie.internal:20202/stream": dial tcp: lookup 934eeb93.vm.litefs-indie.internal on [fdaa::3]:53: no such host ('http://934eeb93.vm.litefs-indie.internal:20202')
2022-11-11T19:01:40.434 app[96c01dd5] maa [info] existing primary found (934eeb93), connecting as replica
2022-11-11T19:01:40.441 app[96c01dd5] maa [info] replica disconnected, retrying: connect to primary: Post "http://934eeb93.vm.litefs-indie.internal:20202/stream": dial tcp: lookup 934eeb93.vm.litefs-indie.internal on [fdaa::3]:53: no such host ('http://934eeb93.vm.litefs-indie.internal:20202')
2022-11-11T19:01:40.651 app[c318b330] den [info] existing primary found (934eeb93), connecting as replica
2022-11-11T19:01:40.656 app[c318b330] den [info] replica disconnected, retrying: connect to primary: Post "http://934eeb93.vm.litefs-indie.internal:20202/stream": dial tcp: lookup 934eeb93.vm.litefs-indie.internal on [fdaa::3]:53: no such host ('http://934eeb93.vm.litefs-indie.internal:20202')
2022-11-11T19:01:40.893 app[96c01dd5] maa [info] HEAD / 200 - - 7.789 ms
2022-11-11T19:01:40.895 app[96c01dd5] maa [info] GET /healthcheck 200 - - 13.669 ms
2022-11-11T19:01:41.713 app[c318b330] den [info] existing primary found (934eeb93), connecting as replica
2022-11-11T19:01:41.721 app[c318b330] den [info] replica disconnected, retrying: connect to primary: Post "http://934eeb93.vm.litefs-indie.internal:20202/stream": dial tcp: lookup 934eeb93.vm.litefs-indie.internal on [fdaa::3]:53: no such host ('http://934eeb93.vm.litefs-indie.internal:20202')
2022-11-11T19:01:41.721 app[96c01dd5] maa [info] existing primary found (934eeb93), connecting as replica
2022-11-11T19:01:41.732 app[96c01dd5] maa [info] replica disconnected, retrying: connect to primary: Post "http://934eeb93.vm.litefs-indie.internal:20202/stream": dial tcp: lookup 934eeb93.vm.litefs-indie.internal on [fdaa::3]:53: no such host ('http://934eeb93.vm.litefs-indie.internal:20202')
2022-11-11T19:01:42.778 app[c318b330] den [info] existing primary found (934eeb93), connecting as replica
2022-11-11T19:01:42.787 app[c318b330] den [info] replica disconnected, retrying: connect to primary: Post "http://934eeb93.vm.litefs-indie.internal:20202/stream": dial tcp: lookup 934eeb93.vm.litefs-indie.internal on [fdaa::3]:53: no such host ('http://934eeb93.vm.litefs-indie.internal:20202')
2022-11-11T19:01:42.969 app[96c01dd5] maa [info] existing primary found (934eeb93), connecting as replica
2022-11-11T19:01:42.975 app[96c01dd5] maa [info] replica disconnected, retrying: connect to primary: Post "http://934eeb93.vm.litefs-indie.internal:20202/stream": dial tcp: lookup 934eeb93.vm.litefs-indie.internal on [fdaa::3]:53: no such host ('http://934eeb93.vm.litefs-indie.internal:20202')
2022-11-11T19:01:43.853 app[c318b330] den [info] existing primary found (934eeb93), connecting as replica
2022-11-11T19:01:43.860 app[c318b330] den [info] replica disconnected, retrying: connect to primary: Post "http://934eeb93.vm.litefs-indie.internal:20202/stream": dial tcp: lookup 934eeb93.vm.litefs-indie.internal on [fdaa::3]:53: no such host ('http://934eeb93.vm.litefs-indie.internal:20202')
2022-11-11T19:01:44.208 app[96c01dd5] maa [info] existing primary found (934eeb93), connecting as replica
2022-11-11T19:01:44.215 app[96c01dd5] maa [info] replica disconnected, retrying: connect to primary: Post "http://934eeb93.vm.litefs-indie.internal:20202/stream": dial tcp: lookup 934eeb93.vm.litefs-indie.internal on [fdaa::3]:53: no such host ('http://934eeb93.vm.litefs-indie.internal:20202')
2022-11-11T19:01:44.946 app[c318b330] den [info] existing primary found (934eeb93), connecting as replica
2022-11-11T19:01:44.951 app[c318b330] den [info] replica disconnected, retrying: connect to primary: Post "http://934eeb93.vm.litefs-indie.internal:20202/stream": dial tcp: lookup 934eeb93.vm.litefs-indie.internal on [fdaa::3]:53: no such host ('http://934eeb93.vm.litefs-indie.internal:20202')
2022-11-11T19:01:45.460 app[96c01dd5] maa [info] existing primary found (934eeb93), connecting as replica
2022-11-11T19:01:45.465 app[96c01dd5] maa [info] replica disconnected, retrying: connect to primary: Post "http://934eeb93.vm.litefs-indie.internal:20202/stream": dial tcp: lookup 934eeb93.vm.litefs-indie.internal on [fdaa::3]:53: no such host ('http://934eeb93.vm.litefs-indie.internal:20202')
2022-11-11T19:01:45.995 app[c318b330] den [info] existing primary found (934eeb93), connecting as replica
2022-11-11T19:01:46.004 app[c318b330] den [info] replica disconnected, retrying: connect to primary: Post "http://934eeb93.vm.litefs-indie.internal:20202/stream": dial tcp: lookup 934eeb93.vm.litefs-indie.internal on [fdaa::3]:53: no such host ('http://934eeb93.vm.litefs-indie.internal:20202')
2022-11-11T19:01:46.711 app[96c01dd5] maa [info] existing primary found (934eeb93), connecting as replica
2022-11-11T19:01:46.717 app[96c01dd5] maa [info] replica disconnected, retrying: connect to primary: Post "http://934eeb93.vm.litefs-indie.internal:20202/stream": dial tcp: lookup 934eeb93.vm.litefs-indie.internal on [fdaa::3]:53: no such host ('http://934eeb93.vm.litefs-indie.internal:20202')
2022-11-11T19:01:47.144 app[c318b330] den [info] existing primary found (934eeb93), connecting as replica
2022-11-11T19:01:47.150 app[c318b330] den [info] replica disconnected, retrying: connect to primary: Post "http://934eeb93.vm.litefs-indie.internal:20202/stream": dial tcp: lookup 934eeb93.vm.litefs-indie.internal on [fdaa::3]:53: no such host ('http://934eeb93.vm.litefs-indie.internal:20202')
2022-11-11T19:01:48.648 app[c318b330] den [info] cannot acquire lease or find primary, retrying: no primary
2022-11-11T19:01:49.193 app[96c01dd5] maa [info] cannot acquire lease or find primary, retrying: no primary
2022-11-11T19:01:49.948 app[c318b330] den [info] cannot acquire lease or find primary, retrying: no primary
2022-11-11T19:01:50.906 app[96c01dd5] maa [info] HEAD / 200 - - 4.368 ms
2022-11-11T19:01:50.908 app[96c01dd5] maa [info] GET /healthcheck 200 - - 9.471 ms
2022-11-11T19:01:51.375 app[c318b330] den [info] cannot acquire lease or find primary, retrying: no primary
2022-11-11T19:01:51.657 app[96c01dd5] maa [info] cannot acquire lease or find primary, retrying: no primary
2022-11-11T19:01:52.704 app[c318b330] den [info] cannot acquire lease or find primary, retrying: no primary
2022-11-11T19:01:53.551 app[96c01dd5] maa [info] primary lease acquired, advertising as http://96c01dd5.vm.litefs-indie.internal:20202
2022-11-11T19:01:53.810 app[c318b330] den [info] existing primary found (96c01dd5), connecting as replica
2022-11-11T19:01:54.327 app[c318b330] den [info] connected to cluster, ready
2022-11-11T19:01:54.207 app[96c01dd5] maa [info] stream connected
2022-11-11T19:01:54.327 app[c318b330] den [info] starting subprocess: node [./other/start.js]
2022-11-11T19:01:54.333 app[c318b330] den [info] waiting for signal or subprocess to exit
2022-11-11T19:01:54.537 app[c318b330] den [info] Found primary instance in .primary file: 96c01dd5
2022-11-11T19:01:54.537 app[c318b330] den [info] Instance (c318b330) in den is not primary (the primary instance is 96c01dd5).  Deploying migrations.
2022-11-11T19:02:58.483 app[4d263b11] maa [info] stream connected
2022-11-11T19:03:00.760 app[4d263b11] maa [info] Prisma schema loaded from prisma/schema.prisma
2022-11-11T19:03:00.765 app[4d263b11] maa [info] Datasource "db": SQLite database "sqlite.db" at "file:/litefs/data/sqlite.db"
2022-11-11T19:03:01.016 app[4d263b11] maa [info] 1 migration found in prisma/migrations
2022-11-11T19:03:01.030 app[4d263b11] maa [info] No pending migrations to apply.
2022-11-11T19:03:01.672 app[c318b330] den [info] HEAD / 200 - - 4.135 ms
2022-11-11T19:03:01.674 app[c318b330] den [info] GET /healthcheck 200 - - 9.181 ms
2022-11-11T19:03:02.028 app[4d263b11] maa [info] npm notice
2022-11-11T19:03:02.028 app[4d263b11] maa [info] npm notice New major version of npm available! 8.19.2 -> 9.1.1
2022-11-11T19:03:02.028 app[4d263b11] maa [info] npm notice Changelog: <https://github.com/npm/cli/releases/tag/v9.1.1>
2022-11-11T19:03:02.028 app[4d263b11] maa [info] npm notice Run `npm install -g<EMAIL_ADDRESS>to update!
2022-11-11T19:03:02.029 app[4d263b11] maa [info] npm notice
2022-11-11T19:03:02.037 app[4d263b11] maa [info] Starting app...
2022-11-11T19:03:03.086 app[4d263b11] maa [info] Remix App Server started at http://localhost:8080 (http://<IP_ADDRESS>:8080)
2022-11-11T19:03:05.908 app[4d263b11] maa [info] HEAD / 200 - - 29.344 ms
2022-11-11T19:03:05.921 app[4d263b11] maa [info] GET /healthcheck 200 - - 83.095 ms
2022-11-11T19:03:11.685 app[c318b330] den [info] HEAD / 200 - - 4.277 ms
2022-11-11T19:03:11.688 app[c318b330] den [info] GET /healthcheck 200 - - 10.422 ms
2022-11-11T19:03:15.940 app[4d263b11] maa [info] HEAD / 200 - - 8.112 ms
2022-11-11T19:03:15.943 app[4d263b11] maa [info] GET /healthcheck 200 - - 16.400 ms

And here's when I deployed a new version that accounts for primary changing on the fly (no pun intended):
2022-11-11T19:29:39.054 runner[b7ace82d] den [info] Starting instance
2022-11-11T19:29:39.150 runner[b7ace82d] den [info] Configuring virtual machine
2022-11-11T19:29:39.151 runner[b7ace82d] den [info] Pulling container image
2022-11-11T19:29:41.701 runner[b7ace82d] den [info] Unpacking image
2022-11-11T19:29:43.373 runner[b7ace82d] den [info] Preparing kernel init
2022-11-11T19:29:43.505 runner[b7ace82d] den [info] Setting up volume 'data'
2022-11-11T19:29:43.508 runner[b7ace82d] den [info] Opening encrypted volume
2022-11-11T19:29:43.633 runner[b7ace82d] den [info] Configuring firecracker
2022-11-11T19:29:43.691 runner[b7ace82d] den [info] Starting virtual machine
2022-11-11T19:29:43.885 app[b7ace82d] den [info] Starting init (commit: 81d5330)...
2022-11-11T19:29:43.909 app[b7ace82d] den [info] Mounting /dev/vdc at /data w/ uid: 0, gid: 0 and chmod 0755
2022-11-11T19:29:43.916 app[b7ace82d] den [info] Preparing to run: `docker-entrypoint.sh litefs -- node ./other/start.js` as root
2022-11-11T19:29:43.939 app[b7ace82d] den [info] 2022/11/11 19:29:43 listening on [fdaa:0:23df:a7b:d826:2:da30:2]:22 (DNS: [fdaa::3]:53)
2022-11-11T19:29:43.985 app[b7ace82d] den [info] LiteFS commit=fabf62dacbc590ed94d79493a3691deb19b153bd
2022-11-11T19:29:43.985 app[b7ace82d] den [info] config file read from /etc/litefs.yml
2022-11-11T19:29:43.985 app[b7ace82d] den [info] Using Consul to determine primary
2022-11-11T19:29:46.190 app[b7ace82d] den [info] initializing consul: key=<EMAIL_ADDRESS>hostname=b7ace82d advertise-url=http://b7ace82d.vm.litefs-indie.internal:20202
2022-11-11T19:29:46.215 app[b7ace82d] den [info] LiteFS mounted to: /litefs/data
2022-11-11T19:29:46.215 app[b7ace82d] den [info] http server listening on: http://localhost:20202
2022-11-11T19:29:46.215 app[b7ace82d] den [info] waiting to connect to cluster
2022-11-11T19:29:46.237 app[b7ace82d] den [info] existing primary found (4d263b11), connecting as replica
2022-11-11T19:29:46.630 app[4d263b11] maa [info] stream connected
2022-11-11T19:29:46.750 app[b7ace82d] den [info] connected to cluster, ready
2022-11-11T19:29:46.750 app[b7ace82d] den [info] starting subprocess: node [./other/start.js]
2022-11-11T19:29:46.757 app[b7ace82d] den [info] waiting for signal or subprocess to exit
2022-11-11T19:29:46.954 app[b7ace82d] den [info] Found primary instance in .primary file: 4d263b11
2022-11-11T19:29:46.955 app[b7ace82d] den [info] Instance (b7ace82d) in den is not primary (the primary instance is 4d263b11).  Deploying migrations.
2022-11-11T19:31:08.444 app[8e37cfa4] maa [info] Prisma schema loaded from prisma/schema.prisma
2022-11-11T19:31:08.454 app[8e37cfa4] maa [info] Datasource "db": SQLite database "sqlite.db" at "file:/litefs/data/sqlite.db"
2022-11-11T19:31:08.689 app[8e37cfa4] maa [info] 1 migration found in prisma/migrations
2022-11-11T19:31:08.706 app[8e37cfa4] maa [info] No pending migrations to apply.
2022-11-11T19:31:09.816 app[b7ace82d] den [info] replica disconnected, retrying: connect to primary: Post "http://4d263b11.vm.litefs-indie.internal:20202/stream": read tcp [fdaa:0:23df:a7b:d826:2:da30:2]:52974->[fdaa:0:23df:a7b:14bf:2:da39:2]:20202: read: connection reset by peer ('http://4d263b11.vm.litefs-indie.internal:20202')
2022-11-11T19:31:09.744 app[8e37cfa4] maa [info] npm notice
2022-11-11T19:31:09.744 app[8e37cfa4] maa [info] npm notice New major version of npm available! 8.19.2 -> 9.1.1
2022-11-11T19:31:09.744 app[8e37cfa4] maa [info] npm notice Changelog: <https://github.com/npm/cli/releases/tag/v9.1.1>
2022-11-11T19:31:09.745 app[8e37cfa4] maa [info] npm notice Run `npm install -g<EMAIL_ADDRESS>to update!
2022-11-11T19:31:09.745 app[8e37cfa4] maa [info] npm notice
2022-11-11T19:31:09.753 app[8e37cfa4] maa [info] Starting app...
2022-11-11T19:31:10.190 app[8e37cfa4] maa [info] > start
2022-11-11T19:31:10.190 app[8e37cfa4] maa [info] > remix-serve build
2022-11-11T19:31:10.678 app[8e37cfa4] maa [info] Remix App Server started at http://localhost:8080 (http://<IP_ADDRESS>:8080)
2022-11-11T19:31:10.963 app[b7ace82d] den [info] existing primary found (8e37cfa4), connecting as replica
2022-11-11T19:31:11.362 app[8e37cfa4] maa [info] stream connected
2022-11-11T19:31:15.967 app[b7ace82d] den [info] HEAD / 200 - - 6.604 ms
2022-11-11T19:31:15.969 app[b7ace82d] den [info] GET /healthcheck 200 - - 14.494 ms
2022-11-11T19:31:20.349 app[8e37cfa4] maa [info] HEAD / 200 - - 28.750 ms
2022-11-11T19:31:20.358 app[8e37cfa4] maa [info] GET /healthcheck 200 - - 80.603 ms

(Also, you can watch the live stream here: https://www.youtube.com/watch?v=AA-1sweDAtA)
Based on the redeploy logs, it looks like a new instance is started up with the primary pointing to an old instance. Then the old primary instance is shut down causing the new replica instance to lose its head! While that's happening, a new instance to replace the old primary instance is started up and it decides it's the primary after which the first of the new instances finds it.
While all that's happening I'm concerned there will be downtime.
Also, what if the new code requires a migration? If a new instance with new code decides that an old instance is the primary, that seems problematic... 😬

I just realised while the circumstances are similar, I have different errors from Kent's errors. Not sure if you missed that. Lmk if we should move this to a separate issue if the scope is different.

@ayuhito Can you try with the latest Docker version (flyio/litefs:sha-fabf62d)? If you still have an issue then go ahead and open a separate issue. I suspect it's the same root cause though.

Does fly ensure 0 downtime on restarts and redeploys? How does it manage this? If so, I'm guessing it just keeps the existing primary up until a new version of the app is ready to become primary at which point, that new version becomes primary and all the other nodes are informed? Something like that?

@kentcdodds On the Fly.io side, you can choose your deployment strategy (rolling, immediate, etc). https://fly.io/docs/reference/configuration/#picking-a-deployment-strategy
As for LiteFS, it'll keep the same primary until that primary can no longer communicate with Consul. On a graceful shutdown, the primary will release the lease it has from Consul and another node will detect that and attempt to become the new primary.

Also, in case it's interesting, here are the logs from when I restarted (it looks like both instances may have restarted multiple times) [...] While all that's happening I'm concerned there will be downtime.

It looks like the primary on den is not destroying its lease even though it's receiving a SIGINT. If it doesn't explicitly destroy the lease such as during an unexpected shutdown, it relies on the TTL on Consul. Another node can't obtain the lease until the TTL + LockDelay is passed. The min TTL on Consul is 10s. The LockDelay is currently set to 5s (although that seems kinda high). I might drop that to 1s since LiteFS has a way to resolve divergent nodes.
Regardless, it should be releasing the lease on SIGINT so that's a bug. I added an issue for it (https://github.com/superfly/litefs/issues/171). I'll take a look at that.
2022-11-11T19:01:22.793 runner[934eeb93] den [info] Shutting down virtual machine
2022-11-11T19:01:22.867 app[934eeb93] den [info] Sending signal SIGINT to main child process w/ PID 529
2022-11-11T19:01:22.868 app[934eeb93] den [info] sending signal to exec process
2022-11-11T19:01:22.868 app[934eeb93] den [info] waiting for exec process to close
2022-11-11T19:01:22.869 app[934eeb93] den [info] signal received, litefs shutting down
2022-11-11T19:01:22.869 app[934eeb93] den [info] stream disconnected
2022-11-11T19:01:24.476 app[96c01dd5] maa [info] existing primary found (934eeb93), connecting as replica
2022-11-11T19:01:24.596 app[934eeb93] den [info] http: error: context canceled
2022-11-11T19:01:24.733 app[96c01dd5] maa [info] replica disconnected, retrying: connect to primary: invalid response: code=503 ('http://934eeb93.vm.litefs-indie.internal:20202')
2022-11-11T19:01:25.972 app[96c01dd5] maa [info] existing primary found (934eeb93), connecting as replica
2022-11-11T19:01:26.092 app[934eeb93] den [info] http: error: context canceled
2022-11-11T19:01:26.226 app[96c01dd5] maa [info] replica disconnected, retrying: connect to primary: invalid response: code=503 ('http://934eeb93.vm.litefs-indie.internal:20202')
2022-11-11T19:01:27.833 app[96c01dd5] maa [info] existing primary found (934eeb93), connecting as replica
2022-11-11T19:01:30.877 app[96c01dd5] maa [info] HEAD / 200 - - 4.501 ms
2022-11-11T19:01:30.878 app[96c01dd5] maa [info] GET /healthcheck 200 - - 10.841 ms

There's also another related issue for Primary Handoff (https://github.com/superfly/litefs/issues/11). Currently, when a node gracefully shuts down, it destroys its lease and it disconnects replica connections to it. Those replicas then check Consul and try to become leader themselves. This can cause a little downtime (typically subsecond) so the handoff would just be a mechanism where the primary actually sends its lease ID to another candidate node so that there is zero downtime. That issue hasn't been as high of a priority because subsecond downtime isn't a huge issue for most folks.

I'm going to close this one and I'll continue the graceful shutdown bug on https://github.com/superfly/litefs/issues/171. If the checksum mismatch issue reappears, we can reopen this.


create new scraper: cleveland museum of art

http://www.clevelandart.org/art/collection/search

https://github.com/ClevelandMuseumArt/openaccess


Subscription Level / Program

All listed are marked Not available which ones are available?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 60b2872e-438a-b86e-9ff5-6dc0021f3fe8
Version Independent ID: 8b9e935e-07ad-4992-edb9-e8361d867617
Content: EMS Benefit | Microsoft Docs
Content Source: subscriptions/vs-ems.md
Service: unspecified
GitHub Login: @ghogen

@evanwindom Can you take a peek at this question?

The eligibility information has been corrected.  This issue can be closed.

Fixed


subscribe to partitioned topic(s) error while non-partitioned topics ok through re.Pattern topics

Describe the bug
I tested on subscribing to multiple topics ( partitioned topics error, but non-partitioned topics ok)
To Reproduce
Steps to reproduce the behavior:

Using pulsar-admin to create one partitioned topic with one partition
Using one python consumer to subscribe.
e.g. TOPICS = re.compile(f"persistent://{MQ_TENANT}/{MQ_NAMESPACE}/.*")
Error with occur
In step 2, if you set your TOPICS to subscribe multiple non-partitioned topics seems OK.

Expected behavior
By using re.Pattern topics to subscribe should also be OK, just like the non-partitioned topics behavior.
Screenshots
│ 2021-09-09 23:24:59.043 INFO  [140028596160320] ConnectionPool:84 | Created connection for pulsar+ssl://pulsar-proxy.mq:6651
│ 2021-09-09 23:24:59.046 INFO  [140028147660544] ClientConnection:372 | [<IP_ADDRESS>:47898 -> <IP_ADDRESS>:6651] Connected to broker
│ 2021-09-09 23:24:59.105 INFO  [140028147660544] HandlerBase:55 | [persistent://dev/spapi/product_listing_offers-partition-0, spapi.consumer.to_datalake, 0] Getting connection from pool
│r2021-09-09 23:24:59.107 INFO  [140028147660544] ConnectionPool:84 | Created connection for pulsar+ssl://pulsar-broker-0.pulsar-broker.mq.svc.cluster.local:6651
│ 2021-09-09 23:24:59.109 INFO  [140028147660544] ClientConnection:374 | [<IP_ADDRESS>:47902 -> <IP_ADDRESS>:6651] Connected to broker through proxy. Logical broker: pulsar+ssl://pulsar-broker-0.pulsar-broker.mq.svc.cluster.local:6651
│ 2021-09-09 23:24:59.161 WARN  [140028147660544] ClientConnection:980 | [<IP_ADDRESS>:47902 -> <IP_ADDRESS>:6651] Received error response from server: BrokerPersistenceError (org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available) -- req_id: 0
│ 2021-09-09 23:24:59.161 ERROR [140028147660544] ConsumerImpl:263 | [persistent://dev/spapi/product_listing_offers-partition-0, spapi.consumer.to_datalake, 0] Failed to create consumer: BrokerPersistenceError
│ 2021-09-09 23:24:59.161 ERROR [140028147660544] MultiTopicsConsumerImpl:91 | Failed when subscribed to topic persistent://dev/spapi/product_listing_offers in TopicsConsumer. Error - BrokerPersistenceError
│ 2021-09-09 23:24:59.161 ERROR [140028147660544] MultiTopicsConsumerImpl:101 | Unable to create Consumer - [Muti Topics Consumer: TopicName - persistent://dev/spapi/.* - Subscription - spapi.consumer.to_datalake] Error - BrokerPersistenceError
 2021-09-09 23:24:59.161 ERROR [140028147660544] MultiTopicsConsumerImpl:412 | Closing the consumer failed for partition - persistent://dev/spapi/product_listing_offers-partition-0 with error - AlreadyClosed

Desktop (please complete the following information):

Base Docker Image: python:3.8.12-slim-buster (Alpine 9.2.0)

Additional context
Add any other context about the problem here.

reader-fd3917723f, 0] Failed to create consumer: BrokerPersistenceError

I have also been getting this error consistently and can reproduce it by:

Running the Golang client reader once to connect to pulsar.
Cancelling the client
Try rerun the client

Error:
root@pulsar-toolset-0:/pulsar/development# ./pulse | tee read.log

2022/04/06 06:12:51.392 c_client.go:68: [info] INFO  | ConnectionPool:84 | Created connection for pulsar://pulsar-broker.pulsar.svc.cluster.local:6650
2022/04/06 06:12:51.398 c_client.go:68: [info] INFO  | ClientConnection:364 | [<IP_ADDRESS>:57668 -> <IP_ADDRESS>:6650] Connected to broker
2022/04/06 06:12:51.402 c_client.go:68: [info] INFO  | HandlerBase:54 | [persistent://ragnarok/transactions/requests, reader-fd3917723f, 0] Getting connection from pool
2022/04/06 06:12:51.404 c_client.go:68: [info] INFO  | ConnectionPool:84 | Created connection for pulsar://pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local:6650
2022/04/06 06:12:51.405 c_client.go:68: [info] INFO  | ClientConnection:364 | [<IP_ADDRESS>:57670 -> <IP_ADDRESS>:6650] Connected to broker
2022/04/06 06:12:51.409 c_client.go:68: [info] WARN  | ClientConnection:971 | [<IP_ADDRESS>:57670 -> <IP_ADDRESS>:6650] Received error response from server: BrokerPersistenceError -- req_id: 0
2022/04/06 06:12:51.409 c_client.go:68: [info] ERROR | ConsumerImpl:263 | [persistent://ragnarok/transactions/requests,
reader-fd3917723f, 0] Failed to create consumer: BrokerPersistenceError
2022/04/06 06:12:51 Could not create reader: Failed to create Reader: BrokerPersistenceError

Here is my client (run on  the pulsar tool container)
package main

import (
        "context"
        "fmt"
        "log"

        "github.com/apache/pulsar/pulsar-client-go/pulsar"
)

func main() {
        // Instantiate a Pulsar client
        client, err := pulsar.NewClient(pulsar.ClientOptions{
                URL: "pulsar://pulsar-broker.pulsar.svc.cluster.local:6650",
        })

        if err != nil {
                log.Fatalf("Could not create client: %v", err)
        }

        // Use the client to instantiate a reader
        reader, err := client.CreateReader(pulsar.ReaderOptions{
                Topic:          "ragnarok/transactions/requests",
                StartMessageID: pulsar.EarliestMessage,
        })

        if err != nil {
                log.Fatalf("Could not create reader: %v", err)
        }

        defer reader.Close()

        ctx := context.Background()

        // Listen on the topic for incoming messages
        for {

                msg, err := reader.Next(ctx)

                if err != nil {
                        log.Fatalf("Error reading from topic: %v", err)
                } else {
                        fmt.Println(msg)
                }
        }
}


Create the ssdp or mdns infrastructure for 0.3

This issue is primarily to get basic discovery working on local wi-fi

10/29 - Just barely started on this, tomorrow should be full day on this.

10/30 - Started implementation. Lots of issues with the library. It has some fixable but fundamental issues about how they did things. The API they offer is synchronous but they are doing asynchronous things so it's a mess.

Working on it at https://github.com/thaliproject/Thali_CordovaPlugin/tree/story_03_vjrantal.

11/2 - He has pushed his current work to a branch. He has some issues with the SSDP library but those seem to be under control, so there is some work.

Please see https://github.com/thaliproject/Thali_CordovaPlugin/blob/story_001/thali/NextGeneration/thaliWifiInfrastructure.js

Also please check the version of the spec that is now in vNext - https://github.com/thaliproject/Thali_CordovaPlugin/blob/vNext/thali/NextGeneration/thaliWifiInfrastructure.js


Emulated EVM contract address differs from stated in startup log

Problem
When running emulator using v1.8.0-evm-preview-1 flow emulator --evm-enabled, the EVM address displayed in the startup log differs from the actual deployment address.
Steps to Reproduce
Running flow emulator --evm-enabled, see EVM=0x01cf0e2f2f715450. However, when retrieving that address's details, we see:
❯ flow accounts get 0x01cf0e2f2f715450
...
Address     0x01cf0e2f2f715450
Balance     0.00000000
Keys     0
Contracts Deployed: 0

Contracts (hidden, use --include contracts)

And retrieving the emulator-account address, we see the EVM contract is deployed to 0xf8d6e0586b0a20c7
❯ flow accounts get 0xf8d6e0586b0a20c7
...
Contracts Deployed: 16
Contract: 'FlowStakingCollection'
Contract: 'NodeVersionBeacon'
Contract: 'EVM'

Acceptance Criteria
Consistent EVM contract address in logs and deployment
Context
EVM bridge

I'm running latest release & pre-release and looks to be solved! Thanks