python311Packages.m2crypto: 0.40.1 -> 0.41.0

Automatic update generated by nixpkgs-update tools. This update was made based on information from passthru.updateScript.
meta.description for python311Packages.m2crypto is: A Python crypto and SSL toolkit
meta.homepage for python311Packages.m2crypto is: https://gitlab.com/m2crypto/m2crypto
meta.changelog for python311Packages.m2crypto is: https://gitlab.com/m2crypto/m2crypto/-/blob/0.41.0/CHANGES
Updates performed

Ran passthru.UpdateScript

To inspect upstream changes
Impact
Checks done

built on NixOS
The tests defined in passthru.tests, if any, passed
found 0.41.0 with grep in /nix/store/9vx955ghxy1hhk229nn4r53ra6q9b6r7-python3.11-m2crypto-0.41.0
found 0.41.0 in filename of file in /nix/store/9vx955ghxy1hhk229nn4r53ra6q9b6r7-python3.11-m2crypto-0.41.0

Rebuild report (if merged into master) (click to expand)

8 total rebuild path(s)

8 package rebuild(s)

First fifty rebuilds by attrpath
dnsviz
hash-slinger
python311Packages.debianbts
python311Packages.m2crypto
python311Packages.pysimplesoap
python312Packages.debianbts
python312Packages.m2crypto
python312Packages.pysimplesoap

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/9vx955ghxy1hhk229nn4r53ra6q9b6r7-python3.11-m2crypto-0.41.0 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A python311Packages.m2crypto https://github.com/r-ryantm/nixpkgs/archive/c663862e35cc0e701a82b9b88f95191fe8d59e3e.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/c663862e35cc0e701a82b9b88f95191fe8d59e3e#python311Packages.m2crypto

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/9vx955ghxy1hhk229nn4r53ra6q9b6r7-python3.11-m2crypto-0.41.0
ls -la /nix/store/9vx955ghxy1hhk229nn4r53ra6q9b6r7-python3.11-m2crypto-0.41.0/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

3 packages failed to build:

python312Packages.debianbts
python312Packages.m2crypto
python312Packages.pysimplesoap

5 packages built:

dnsviz
hash-slinger
python311Packages.debianbts
python311Packages.m2crypto
python311Packages.pysimplesoap

Maintainer pings
cc @andrew-d for testing.

@ofborg build python312Packages.m2crypto

Already on master.


Add Home Button in the Appbar  to Navigate to Home Page

yo! I can take it.
Will it have to be approved by @hkirat or someone else?

@shimiltharayil is this available to work on??

small change in the appbar  <Button
variant={"outline"}
onClick={() => {
router.push("/");
}}
>
Home

we should wait till someone confirms, as although the change is very small, but is the feature even required?

Hey, I'll take the issue.
Can this issue be assigned to me?

Click on daily code logo it will redirect to home page

#270 Issue resolved here

Just do it on the dailycode logo click


Adding vignette to package

Closes #39.
This PR adds a vignette to the package. We should think more about the organization of sections and the presentation of current methods.

Note to self: R CMD Rd2pdf . in the local repo folder creates the reference manual


Bad interaction between additionalProperties and $ref

Description
Compiling the following spec into a TypeScript-Fetch client results in invalid code.
Specifically, the generated code contains
export interface GenericMap extends null<String, number> {
}

which the TypeScript compiler rejects.
Swagger-codegen version
I tested it on swagger-codegen 2.2.1 (latest stable), 2.3.0 (7aebcfa) and master (d11d0f8).
Swagger declaration file content or url
...
definitions:
  GenericMap:
    additionalProperties:
      type: integer
  Container:
    type: object
    properties:
      works:
        type: object
        additionalProperties:
          type: string
      breaks:
        $ref: "#/definitions/GenericMap"

Command line used for generation
swagger-codegen generate -i test.yaml -l typescript-fetch
Steps to reproduce
Compile a swagger API containing the definitions mentioned above into a typescript-fetch client.
Further data
This was originally reported as https://github.com/swagger-api/swagger-codegen/issues/4839
@wing328 suggested to report the issue here as it is likely related to some problems with the resolution of the reference.

Can you try changing your definition to the following and see if it helps?
...
definitions:
  GenericMap:
    type: object
    additionalProperties:
      type: integer
  Container:
    type: object
    properties:
      works:
        type: object
        additionalProperties:
          type: string
      breaks:
        $ref: "#/definitions/GenericMap"

On each of the three versions of swagger-codegen I had tested, it generates the very same output as the original spec.

Thanks for checking.

Has anyone found a solution for this? I'm having the same issue with the typescript-axios vs.

Still broken on Codegen v3.0.31.

Tried it on v3.0.34 and the problem still occurs

Same issue for the version v3.0.36

So is anybody managing this project or what?. People been complaining about this since 2019 and still no answer?.

Any news about this ?

Same issue for the version v3.0.50

Still broken for the version v3.0.58


Mesh plotting

It would be nice to have a mesh plot method. We can follow this post to construct the mesh.
However, it might be a little bit tricky with AMR meshes. A straight-forward way is to plot cell-by-cell.

There is an unexpected side effects from plotmesh:

As we can see, the coarse cell centers are also displayed. I haven't seen this in 2D AMR, but only in 3D AMR.

The reason is that this projection is not a true 2D plot, but an overlap of all the cell-centered points onto this plane cut:

Improved in Commit ced0ea20ca51cb72184


Use multivalue records instead of simple records

Use multivalue records instead of simple records for the AWS Route53 ACME DNS challenge solver.
When using simple records, it would be impossible to create a second record in a Route53 hosted zone with the same name/type.
This way when either using the same domain name on multiple clusters or using the same CNAME record for multiple domains would create a new challenge instead of overwriting the other challenges.

Pull Request Motivation
Fix https://github.com/jetstack/cert-manager/issues/3460

Kind
bug

Release Note

NONE

Adding a link to related Slack chat https://kubernetes.slack.com/archives/C4NV3DWUC/p1643211118262600

/cc munnerz
/cc euank

I tested this on a personal cluster and I was able to get new certificates there :) (so that's good).
Also on my cluster, I manually added a txt record and then I saw that cert-manager added another one:
_acme-challenge.thrash1.fvlaicucm-thrash-va6.ethos.adobe.net.cm.sandbox.cloud.adobe.io.	10 IN TXT "test"
_acme-challenge.thrash1.fvlaicucm-thrash-va6.ethos.adobe.net.cm.sandbox.cloud.adobe.io.	10 IN TXT "Bz8K9u2TrHWT-XrlilAXDmmnPQyfa-UOQwi3SL7AdA0"

Given this also fixes a bug/issue, it'd be great if you can fill in the release notes so other users know what's changed 😄
Will this require changes to user's IAM permissions? These are the current roles that users are expected to create: https://cert-manager.io/docs/configuration/acme/dns01/route53/#set-up-an-iam-role

Given this also fixes a bug/issue, it'd be great if you can fill in the release notes so other users know what's changed 😄
Will do
Will this require changes to user's IAM permissions? These are the current roles that users are expected to create: https://cert-manager.io/docs/configuration/acme/dns01/route53/#set-up-an-iam-role
I don't expect and I didn't change anything on my end during my testing.

Thanks for clarifying all these points 😄 and thanks for identifying the bug & fixing it!
/lgtm
/approve

The way we use cert-manager is to generate only wildcard certificates. Our k8s clusters have generic ingress classes that tenants can use. We generate a wildcard for each generic ingress class, however, the generic ingress class domains are all unique (they also contain the cluster name in their URL).
We do however have a shared domain for all clusters, that's the domain we have encountered this issue with.
let's say the URL for the cluster is cluster1-stage-ue1.domain.com.
For each tier (sandbox, dev, stage, prod) we have a cert-manager validation domain hosted in AWS Route53 -
cm.{sandbox, dev, stage, prod}.testdomain.com
When we create each cluster, we'll create the following CNAME:
_acme-challenge.cluster1-stage-ue1.domain.com -> _acme-challenge.cluster1-stage-ue1.domain.com.cm.stage.testdomain.com

Doing this will ensure that each domain has its own unique challenge txt record. But for the common domain, used across all clusters there's no way to do something like this.
_acme-challenge.stage.commondomain.com -> _acme-challenge.stage.commondomain.com.cm.stage.testdomain.com

All clusters will have to compete to write in this record: _acme-challenge.stage.commondomain.com.cm.stage.testdomain.com

/cherry-pick release-1.7
/cherry-pick release-1.8

I don't think this qualifies for cherry picking as it's a long-standing bug that's been present for a long time. Cherrypicking is reserved for critical bugs, as defined here: https://cert-manager.io/docs/installation/supported-releases/#critical-bugs


Unexpected error when doing conditional query

Describe the bug
SafeQL throws an unsupported conditional expression error when doing a conditional query on Postgres. This is a feature in postgres.js but it seems like SafeQL doesn't allow for these kinds of queries. The queries below would throw this error Invalid Query: Unsupported conditional expression flags (true = 524288, false = 524288).
type A = { a: number | null; b: string | null; c: string | null };
export async function query(a: number) {
  return await sql<A[]>`
    SELECT
      *
    FROM
      try_safe_ql ${a ? sql`WHERE a = ${a}` : sql``}
`;
}

export async function query3(a: number) {
  return await sql<A[]>`
  SELECT
   *
  FROM
    try_safe_ql
  WHERE
    c is not null ${a ? sql`AND a = ${a}` : sql``}
`;
}

According to postgres.js%0A%60-,Building%20queries,-Postgres.js%20features), it is normal to do these kinds of queries
To Reproduce
Steps to reproduce the behavior:

Setup SafeQL
Use this code in .ts file

type A = { a: number | null; b: string | null; c: string | null };
export async function query(a: number) {
  return await sql<A[]>`
    SELECT
      *
    FROM
      try_safe_ql ${a ? sql`WHERE a = ${a}` : sql``}
`;
}

Expected behavior
I expected no error for these kinds of queries
Screenshots

Desktop (please complete the following information):

OS: MAC OS
PostgreSQL version 14
Version [e.g. 22]

Additional context
Add any other context about the problem here.

I'm not sure how I feel about nested template expressions. Doesn't it overcomplicate the query? I think CASE WHEN is the correct approach for conditions

I agree that support for non-conditional fragments might be beneficial:
const cols = sql("id", "name", "body");
const q = sql`select ${cols} from tbl order by ${cols}`;

I feel like conditions can go out of hand very quickly.
Each condition in the query will include multiple the number of "forms" the query can be.
For instance, a query with three conditions will have eight forms (2^3). Now, if you have a query with ten conditions, you'll have 1024 forms.
This means, the linter will have to check 1024 different forms of the query, which is not feasible:
const q = (a, b, c, d) => sql`
  select from tbl
  where true
    ${a ? sql`and a = ${a}` : sql``}
    ${b ? sql`and b = ${b}` : sql``}
    ${c > 1 ? sql`and d > 0` : sql`and d < 0`}
`;

1. select from tbl where true and a = $1 and b = $2 and d > 0
2. select from tbl where true and a = $1 and b = $2 and d < 0
3. select from tbl where true and a = $1 and d > 0
4. select from tbl where true and a = $1 and d < 0
5. select from tbl where true and b = $1 and d > 0
6. select from tbl where true and b = $1 and d < 0
7. select from tbl where true and d > 0
8. select from tbl where true and d < 0

Let's go to the next level - What if erros in multiple "forms"?
const q = (x) => sql`
  select
    id
  from
    tbl ${x
      ? sql`join tbl2 on tbl.id = tbl2.id`
      : sql`join tbl3 on tbl.id = tbl3.id`
    }
`;

1. select id from tbl join tbl2 on tbl.id = tbl2.id # Error! duplicate column "id" (tbl, tbl2)
2. select id from tbl join tbl3 on tbl.id = tbl3.id # Error! duplicate column "id" (tbl, tbl3)

Personally, I believe that rewriting the query above in the "PostgreSQL way" will make it more readable and maintainable, but that's just my opinion:
const q = (a, b, c, d) = > sql`
  select from tbl
  where true
    and case when ${a} then a = ${a} end
    and case when ${b} then b = ${b} end
    and case when ${c > 1} then d > 0 else d < 0 end
`;

To summarize:

non-conditional fragments should be supported.
conditional fragments should be noted as not supported due to the reasons written above.
add a recommendation of how to rewrite the query conditions in a standard way.

If it weren't due to the complexity (performance and product-wise), I would've added support for it (I'm open to suggestions).

Hm, but even if there are a lot of queries to run, aren't they all pretty fast? I mean, bailing out after a certain complexity seems ok (kind of like a max recursion depth that TS also has), but maybe up to 100 possible generated queries per query could be supported?

They are. Running a query that has 5 conditions will cost 2^5 * 5ms~ = 160ms~ each time you type, this is pretty noticeable. I could probably let the user decide how deep SafeQL should go, but that's not the only issue I wrote.

Ok, understood. 160ms on every keystroke (just for a single query) is indeed a long time, in case a query really takes 5ms to run. If there were 5 or 10 queries each with conditions in a single file, this would be a lot of waiting. I guess TypeScript is also kind of slow sometimes - so maybe the DX would still be somewhat acceptable, but I understand not wanting to go down this path.
The other concern I assume you're referring to was was what to do with the multiple error messages:

What if erros in multiple "forms"?

I guess either just the first error would need to be returned (fail fast) or they would all need to be returned in a combined error message.

function updateUser(id: number, { firstName: string, lastName?: string })
  const [user] = sql<User[]>`
    UPDATE
      users
    SET
      first_name = ${firstName},
      last_name = CASE WHEN ${lastName} THEN ${lastName} ELSE last_name END
    RETURNING *
  `;
  return user;
}

Ok, good to hear that I'm wrong!
Maybe for many (most?) use cases, the CASE WHEN syntax would work...? I guess it would be fine, if there's a way of doing this.
And for those cases that are impossible, I guess using multiple tagged template literals with a full separate query would still be possible, even though it means a bit of duplication:
const [user] = someCondition ?
  sql`SELECT ...` :
  sql`SELECT ...`;

I can't think of a case where you won't be able to achieve your goal using just Postgres. But yeah, on general, if you are across a situation where your query gets too complex, it might be a good idea to split it into smaller queries. Who knows, maybe you'll do a favor to your query planner.

@Newbie012 , Thanks for the detailed explanations, I am going to look into having the conditional queries rewritten using the suggested CASE...WHEN
👍

Happy to hear that! I'm closing this issue for now.


Consider sticking to semantic versioning

While it is not my business managing this project, I think it would be nice to consider sticking to semantic versioning scheme.
Today I got a bit surprised when I did mix deps.update --all :

And found out that a minor bandit version 1.4.1 introduced breaking changes:

Thanks for the note. I appreciate the perspective; these changes were ones that I thought long and hard about, and that we had a number of internal discussions about as well. The answer basically amounts to 'maintenance burden'. Allow me to explain:
If we were to have honoured semver for this change, we'd have had to cut a 2.0.0 release, which would have meant that anyone pinning to ~> 1.0 would have had to take explicit action to pull in this new version. At the same time, the vast majority of our usage is within Phoenix projects. The way Phoenix generates applications means that essentially all of our users are in fact pinned to ~> 1.0 in this exact way.
Experience seems to suggest that the majority of users don't do anything with most of their dependencies. If Phoenix generated a mix.exs file that was pinned as above, the likelihood is that it would stay that way forever. And so, we'd be in a position where we'd have two options:

Let the 1.0 release tree rot, and deny the vast majority of our user base any future improvements, or;
Carry out new work against both 1.0 and 2.0 trees for the rest of time

On balance, there is no scenario in which either of these is a reasonable tradeoff for something as small as a config option & a few telemetry changes (not to mention, a config option that was only introduced in a minor release a few months prior).
Moreover, we maintain a ruthlessly detailed CHANGELOG that is updated on every release, and is literally the first thing I point to when I publicize releases (if you're not already a member, you should join the #bandit Elixir slack channel where we announce every release, alongside such a link). Breaking changes (which, to be clear, we go to great lengths to avoid) are always called out prominently.
It's also worth noting that the project has never claimed to conform to semver; there has never been a mention of it in the README or elsewhere, and even personally I've never felt a need to bound myself to such a restrictive yoke. To be clear, I cannot and will not promise to not violate semver in the future, though I will continue to do my best to adhere to its spirit.
But ultimately, and with the utmost respect, I think you're expecting too much of maintainers. None of us work for you, and there isn't a project in the world that owes you conformance to semver. The reality of this or any endeavour (technological or otherwise) is compromise. Semver is, in many real and tangible ways, a bit of a prison for maintainers in that sense. It presumes a world where users (and other package maintainers with transitive dependencies) move fast, are hyper aware of their dependencies, and know the pedigree and development direction of everything in their deps folder. This, unfortunately, does not map to reality in any meaningful way. If we as a community are going to make this grand experiment work in a sustainable way, there needs to be some concession and adaptability on the part of users.
Turning back to the prescriptive aspects of your situation, you may want to look at the changes that just came out in 1.5.3 with the log_protocol_errors option. It now provides (and defaults to) a value of :short, which will cause one-line summaries of protocol errors to be logged. Moreover, I'll note that this is actually one of the more actively developed corners of the project at this point (a lot of people have a lot of opinions on how logging should work), so I can't promise it won't change again in the future (though I have no open plans for it to do so).
Thanks again for the discussion.

RomVer best versioning
https://github.com/romversioning/romver

Thank you for such an insightful and detailed answer.
And thank you for such a great project.


Buffer output to consumers

Is your feature request related to a problem? Please describe.
n/a
Describe the solution you'd like
TS should buffer output to consumers.
Currently TS losses data if it is unable to upload to upstream consumer.
Describe alternatives you've considered
n/a
Additional context
n/a

Thank you for submitting feedback. However this request is very vague and leaves a lot of questions about requirements.

How long do you expect TS to buffer the data? If it buffers all events indefinitely, the buffer could easily get large enough to cause problems for the BIG-IP.
Do you expect this data buffering to happen on all consumer data? Just forwarded event data? Just system poller data?
How much control do you expect to have over the buffering?

@dstokesf5 I would suggest something like a maximum size configuration for the buffer, where once the maximum size is reached old data is purged from the buffer.  Definitely open to options, as the current state (no buffering capabilities at all) is rough.

For buffering, I suggest setting up an OpenTelemetry collector with a fast and reliable network connection to your BIG-IP. Then you can let it handle the data buffering by using the new OpenTelemetry consumer in Telemetry Streaming: https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/setting-up-consumer.html#opent
Does this work for your use case?


Add serverside post-transforms to query nodes

https://user-images.githubusercontent.com/19550456/170859342-5eb8ca89-220f-4df1-b7d5-571fb1d92f82.mov

Closes #366

Based on #400

[x] I have followed (at least) the PR section of the contributing guide.

Perhaps a follow up PR can remove the concept of the apiNode

nice!


LFS-619: Use a QueryManager object for improved filter performance

When filters are added by LFS users, the usequerymanager=true URL parameter is added to the Forms.paginate request and the back-end code uses a QueryManager to execute the query in the same way that this process happens in Composum.
To test:

Start LFS and load 10,000+ Forms
Apply various filters (eg. Date of Birth = 2000-01-01) and note the improved performance. You may also wish to compare performance with the same queries using the same Forms on a build of the dev branch.

@sdumitriu I have addressed the issue that you have raised by removing the IteratorWrapper and the (Node) cast.
This PR, however, cannot be merged yet as the Previous/Next buttons on the Form cards are always disabled. I am currently working on that issue.

Huge improvement on filters defined as Source1 <> 123. It is almost instant on an instance build off this PR. On dev branch it is way too slow.


Paths containing .. bug

I can understand not supporting extraction of paths that are relative, but I think this is an edge case that isn't relative and also isn't supported:
Extracting paths containing `...` is not supported (nim-1.0.4/c_code/7_1/@m..@slib@spackages<EMAIL_ADDRESS>
This is from from nim-1.0.4-linux_x64.tar.xz.
I have similar checks in my library which it seems I was also not sure about (https://github.com/dom96/untar/blob/master/src/untar.nim#L149), but I think checking for just ".." is too broad.

Ah thanks for making an issue. I definitely would like to allow that path. The .. check was just "simple" in the moment, not really ever meant to be perfect. Let's improve it.

Changes released in zippy 0.7.1


Initial design proposal of simultaneous flow editing by multiple clients

This PR adds initial design note on simultaneous flow editing by multiple clients for discussion of implementation direction.

Hi ,
This is a feature which interests me as a user. Hence sharing my thoughts.
From an end user view - who knows very limited internals/backend of NR - locking option is something i would vote for (Option 2)
Already apologizing in case i have caught wrong end of some NR internals.

If locking feature enabled (via. settings.js) Every client needs to lock a flow else its read only. Only a user with write permissions (based on today) can lock.  (The design talks about client - not clear if it means user login is mandatory - thereby being able to capture user info for locked state - i am assuming so. )

Only the user who locked a flow or admin user can unlock it back.
I am aware that today  user role permissions are limited  to read/write ALL - so till the absence of more granularity- any user with write permission can unlock any flow. (Working in a team - people do know when someone goes on vacation). Once unlocked - its unlocked. And any user (including the one who locked it originally) needs to relock.

Its mentioned in above scenario when user wanders off.... defining time and then unlocking........ and then again comparing and  giving the lock back. when user comes back.  Maybe not a  day 1 feature . Once unlocked - user needs to relock since the unlocking was intentional by his peer team member.

(It does mean - if the original user did some changes on his client which are not yet "deployed"  - they would be lost . But being a team collaboration -  i would assume only when the user goes on long breaks - other team members would want to  unlock post discussing with him). Hence i mean not a day 1 feature.

I am not clear on "The problem with method 2 is that conflicts may occur when locks are not acquired and a flow is edited by multiple clients.  So, there still exist the same issues with current Node-RED implementation when conflicts occur.".

If working in lock feature enabled mode - i would assume a lock is mandatory i.e. for a user to be able to edit.
Does it refer to error scenario when trying to acquire a lock ?The only reason i would presume not to get a lock would be when either that flow is locked by others or for some reason NR went down and user still has editor open on his browser.
As i see if not locked - then no edit.

On notifying other users when a change happens - This may not be a day 1 improvement. But down the line. For Day 1 the current mechanism of Merge / Review continues when some flows change.
Down the line -
The enhancements can be - indicating to other users if a flow has been updated - some symbol on flow & allowing an option to refresh that flow/all flows.
Or refresh happening automatically when the user opens a flow in his editor instance which was updated by someone else & deployed already.
Or refresh happening only on locking (it means user sees a "stale" version of the flow till he locks or refreshes - not ideal way as user keeps seeing incorrect information on screen)

My above write up is based on the way we work across other systems - which allows us to track which user did what - there by maintaining some level of concurrency without overwriting & backtracking any potential issues.
In case above is completely flawed - please feel free to correct me.
Thanks.

Hi, Thank you for your comments.

If locking feature enabled (via. settings.js) Every client needs to lock a flow else its read only. Only a user with write permissions (based on today) can lock.  (The design talks about client - not clear if it means user login is mandatory - thereby being able to capture user info for locked state - i am assuming so. )

The original design assumed that the flow was always editable.  However, I think it is useful to be able to make the flow read only as you suggested, so I reflected it in the design.

Only the user who locked a flow or admin user can unlock it back.

The problem here is that Node-RED has no way of distinguishing users.　But I have added your suggestion to discussion part.

Are you thinking differently?

Here, I was assuming the default setting (adminAuth disabled) because many of our users are on the situation.
However, I think it will be possible to make that a prerequisite.

The problem here is that Node-RED has no way of distinguishing users.　But I have added your suggestion to discussion part.

Here i did mean "adminAuth" enabled. As without that tracking who did what is impossible (well almost).


Docs maintenance: fix broken links, update cloud manager articles

This PR contains the following:

Pins a separate Generic Findings Import article to the sidebar so that the parser is easy to find
Adds additional resources for OS users
fixes broken / irrelevant links

[sc-9103]

I inadvertently broke this PR - will fix shortly

This was fixed with a new PR #11376


Adjust to library API change: no more spdy-specific config.

(Upcoming on mod_pagespeed end)

LGTM
(Sorry I missed this earlier!)


Fix style of type comparisons in some unit tests

While I was setting up the dev environment, I ran the tests as described in the readme, and got some error messages from flake8:
tests/validators/allow_empty_string_test.py:33:16: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
tests/validators/allow_empty_string_test.py:49:16: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
tests/validators/none_to_unset_value_test.py:34:16: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
tests/validators/noneable_test.py:34:16: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
tests/validators/noneable_test.py:50:16: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`

I fixed the style according to these messages, now the tests run without flake8 complaining again :)

Usually isinstance would be the way to go, however, since these are all assertions in the unit tests to check the type of the validated result, I think a strict test with type(result) is type(expected_result) would be better. If the validator results with an object of a subclass of the expected type, something is wrong.


Fix style for consistency

Fix style of  " Services " to " Services " for consistency and beauty.

This change is

Your keen eye for beauty is appreciated :)


axis control

i have an axis camera that has pan till zoom controls view their web interfaces.
they also give access to pan till zoom via their "AXIS Media Control" SDK:
http://www.axis.com/techsup/cam_servers/dev/activex.htm
  AMC.MediaType:='mjpeg';
  AMC.MediaURL:='http://<IP_ADDRESS>/axis-cgi/mjpg/video.cgi';
  AMC.MediaUserName:='root';
  AMC.MediaPassword:='myPassw';

  AMC.PTZControlURL:='http://<IP_ADDRESS>/axis-cgi/com/ptz.cgi';

and i assume after one is logged on http://<IP_ADDRESS>/axis-cgi/com/ptz.cgi?zoom=2 would give it the control i want.
i do get a video feed with your addon.
do you see any chance of doing this without their SDK.
I am not on windows.

Hey there, yes, I've done this before -- although it was in Java.  But it could be updated to use ofxHTTP (or another http client) to issue the various calls.  A good starting point might be:
https://github.com/themaw/mawLib/blob/master/src/mxj/trunk/mawLib-mxj/src/maw/jit/ipcam/PTZCameraControl.java
which I wrote for another project.  All of it is simply constructing HTTP commands ...
The video stream and the PTZ control are separate.  You might even be able to use the standard ofLoadURL to do it, but ofxHTTP has a lot more advanced features for logging in w/ username pass, etc.  Keep me posted!

Also, by the way, when I wrote that I just reverse-engineered the javascript file that the Axis cam served up.  It may or may not be the exact commands used by your camera, but it's probably pretty similar.

thanks. looks great.
i did manage to get control using ofHttpUtils
also found this doc with lots of commands:
http://www.axis.com/techsup/cam_servers/dev/cam_http_api_2.php#api_blocks_ptz

Closing this as it's not a bug.  Currently there aren't plans to add camera control signals.  If I were to do this, I would use ofxHTTP which can make pretty extensible / configurable HTTP clients.


Missing "How to use" instructions

I am trying to run pymterm but I didn't find any instructions on how to install/use it. I am on Linux and setup.py complains about the platform (It seems to want to build a mac's "app"). Running pymterm.py  by itself yields the error:

$ python pymterm.py
ERROR:root:load configuration failed
Traceback (most recent call last):
File "pymterm/pymterm.py", line 43, in pymterm_main cfg = session_config.SessionConfig(args)
File "/home/hydrocat/Downloads/git/pymterm/pymterm/session_config.py", line 39, in init self.load_config()
File "/home/hydrocat/Downloads/git/pymterm/pymterm/session_config.py", line 185, in load_config self.config = json.load(f)
File "/usr/lib/python2.7/json/init.py", line 290, in load **kw)
File "/usr/lib/python2.7/json/init.py", line 338, in loads return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 366, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python2.7/json/decoder.py", line 382, in raw_decode obj, end = self.scan_once(s, idx)
ValueError: Expecting , delimiter: line 18 column 6 (char 301)
usage: (usage omitted)

Update the documents, please try again follow the documents to install/setup/run
Install & Setup & Run

How to install pymterm on Linux/Mac/Windows
How to setup pymterm
How to use pymterm


multi: add sqlite backend option

In this PR, the changes in https://github.com/lightningnetwork/lnd/pull/7251 are made use of & a new sqlite db backend option is given.
Depends on https://github.com/lightningnetwork/lnd/pull/7251
Fixes https://github.com/lightningnetwork/lnd/issues/6176
Replaces https://github.com/lightningnetwork/lnd/pull/6570

I used the async_payments_benchmark itest to benchmark some of the different flavours of the sqlite backend. Here are the results (note, I ran these on a mac M1 Pro). I ran the benchmark 4 times per flavour.
wallet.db and sphinx.db separate from main db:
All transactions starting with "BEGIN DEFERRED" (the default directive):

this causes SQLITE_BUSY errors.

All transactions starting with "BEGIN IMMEDIATE":

Benchmark info: Elapsed time:  10.920178333s
Benchmark info: TPS:  44.230046916029615
Benchmark info: Elapsed time:  10.195665s
Benchmark info: TPS:  47.373074733232215
Benchmark info: Elapsed time:  9.145208208s
Benchmark info: TPS:  52.81454385887942
Benchmark info: Elapsed time:  10.644479083s
Benchmark info: TPS:  45.37563522214871

Write txs use the "BEGIN IMMEDIATE" and reads use the "BEGIN DEFERRED":

Benchmark info: Elapsed time:  12.37037875s
Benchmark info: TPS:  39.04488373082352
Benchmark info: Elapsed time:  9.54607925s
Benchmark info: TPS:  50.596688687662
Benchmark info: Elapsed time:  6.8426615s
Benchmark info: TPS:  70.58656927571238
Benchmark info: Elapsed time:  11.509999s
Benchmark info: TPS:  41.963513637142796

wallet.db and sphinx.db in main db:
All transactions starting with "BEGIN DEFERRED" (the default directive):

this causes SQLITE_BUSY errors.

All transactions starting with "BEGIN IMMEDIATE":

Benchmark info: Elapsed time:  9.512753542s
Benchmark info: TPS:  50.77394235722542
Benchmark info: Elapsed time:  8.107619333s
Benchmark info: TPS:  59.57359123091429
Benchmark info: Elapsed time:  11.372332292s
Benchmark info: TPS:  42.47149903804446
Benchmark info: Elapsed time:  10.099899334s
Benchmark info: TPS:  47.82225881935706

Write txs use the "BEGIN IMMEDIATE" and reads use the "BEGIN DEFERRED"

Benchmark info: Elapsed time:  5.733995958s
Benchmark info: TPS:  84.23445072822635
Benchmark info: Elapsed time:  8.6996415s
Benchmark info: TPS:  55.51952916680532
Benchmark info: Elapsed time:  9.790121833s
Benchmark info: TPS:  49.33544323952439
Benchmark info: Elapsed time:  10.441266958s
Benchmark info: TPS:  46.258754032711515

The current PR state is equivalent to this last configuration: reads are deferred, writes are immediate and the macaroon & sphinx dbs are in the main db. Only the wallet.db is kept separate due to the deadlocks it would cause if kept in the same db.

@roasbeef: review reminder
@ellemouton, remember to re-request review from reviewers when ready

Thanks for all the performance tests @Roasbeef 🚀
For now, I have hard coded the synchronous=normal one and also added the config option so that users can add others. Let me know if you think we should also hard code any of the other pragma options.

Linter failures:
Error: config.go:1634:19: mnd: Magic number: 2, in <condition> detected (gomnd)
level=info msg="[runner] linters took 1m49.149737857s with stages: goanalysis_metalinter: 1m47.6[33](https://github.com/lightningnetwork/lnd/actions/runs/3912461448/jobs/6692053709#step:5:34)594715s"
level=info msg="File cache stats: 1115 entries of total size 15.2MiB"
level=info msg="Memory: 2462 samples, avg is 728.2MB, max is 2585.4MB"
level=info msg="Execution took 4m6.232140669s"
		if len(pair) != 2 {
		                ^
Error: cmd/lncli/cmd_payments.go:1659:7: mnd: Magic number: 2, in <case> detected (gomnd)
	case 2:
	     ^
Error: cmd/lncli/cmd_payments.go:1661:7: mnd: Magic number: 3, in <case> detected (gomnd)
	case 3:

I am asking with the whole issue around fsync on m1 in the back of my mind: https://bonsaidb.io/blog/acid-on-apple/

With this PR, by default we use the WAL mode, and then an option of "normal" synchronous = 1. On my M1 machine, this PR is actually around 10x faster than boltdb when running the async payment benchmark. This is the same if it's running with synchronous set to normal (the currnet PR), and even full (fsync the WAL after each commit). With the way we set up the config, a user can modify any of these default values. With more exp/testing, we might even start to set the values depending on the detected platform/arch.
Also IIUC, the post refers to the dynamically linked sqlite library that ships with mac. Here we use a statically compiled Go transliteration, so it doesn't appear that Apple's hack to use the less durable mode in order to speed up I/O applies. We can ofc run an strace test like in the post to confirm.
There's also the fullsync option which defaults to F_FULLFSYNC on mac.

On my M1 machine, this PR is actually around 10x faster than boltdb when running the async payment benchmark.

This sounds suspicious on its own and I think that it may be indicative of the fake fsync being used?

There's also the fullsync option which defaults to F_FULLFSYNC on mac.

Looks like this option is off by default. For LND on Mac, it seems to be that it should be set to true to make sure all has been done to make sure that channel updates are stored on disk physically.

Looks like this option is off by default. For LND on Mac, it seems to be that it should be set to true to make sure all has been done to make sure that channel updates are stored on disk physically.

IIUC, the default is actually synchronous=full:

Furthermore, transaction commit is much faster in WAL mode using synchronous=NORMAL than with the default synchronous=FULL. For these reasons, it is recommended that the synchronous setting be changed from FULL to NORMAL when switching to WAL mode. This compile-time option will accomplish that.

https://www.sqlite.org/compile.html

I ran the async payment itest on my machine locally (commenting out the current section, and also setting it to full) and didn't run into any timeouts fwiw (M1 laptop).

I don't think that synchronous=full is the same thing as fullsync=true. It seems to me that synchronous configures when sqlite syncs to disk and fullsync tells it how to sync.
Did you try the benchmark with fullsync=true? That's where I ran into problems.

Did you try the benchmark with fullsync=true? That's where I ran into problems.

I don't see that in the set of pragma options: https://www.sqlite.org/pragma.html. Looks like it's actually fullfsync (has an "f" before the sync).
On Mac, if I run w/ that I get less TPS (have a lot of other stuff running at the same time ofc), but the timeout still seems to be fine (this is with synchronous=normal. If I set that to full, then I hit a busy timeout. At this point, even if I increase the timeout to 10 minutes. It still hits the busy timeout, each time just after about 20 seconds or so of running time. This doesn't match my mental model of: "the ejected transaction will try for up to BUSY_TIMEOUT milliseconds until it gives up and returns the error".
However, if I modify the defaultSqliteMaxConnections value to be 1 instead of 50, then I have no trouble. The current 5 second busy wait timeout can then be left unmodified.

Continuing with the comment above, if I keep the max connections to a very small value (2 or so), keep the default synchronous open, and then remove fullfsync, I get about 3-4x the TPS (312 TPS in one eaxmple) compared to the default max connection value.
I think we're missing something here w.r.t what the database/sql library actually does with this value for sqlite. From what I gather, opening all the DB connections (up to 50 each time some goroutine needs to interact w/ the db) ends up slowing things down a lot. IIRC, the value was created initially as it matters more for postgres, where there exists a DBMS level param that controls the total number of active client connections (so seemingly less overhead).

But following the advice in the link above, max connections should be fixed to just 1 because sqlite already has its own way of dealing with concurrency?

Looking thought the database/sql code, the issue is that the library has its own connection pooling. So it's actually calling sqlite.Open many times in the background, even while processing normal queries over the logical database pointer. The main library takes a driver like sqlite, then adds this extra pooling in front of it. If we keep with the read/write connection design, then I think we want to set this to 2.

In any case, I think it is required to use fullfsync on mac (only) if you want to be sure that all data has been written to disk before returning to the caller.

Yeah agreed, I/O on Mac is already pretty slow due to similar issues in the Go runtime that we discovered in the past.

And then make synchronous=full mandatory? (https://github.com/lightningnetwork/lnd/pull/7251#issuecomment-1386356877) It does seem it doesn't https://github.com/lightningnetwork/lnd/pull/7252#discussion_r1073738763

Yeah, then we'd swap that out with the full setting we have today. People can still modify it if they wish, but we'll ship with the safer default.
Are you able to run the benchmark again w/ the connection pool fix I mentioned above? Without changing, that it didn't actually matter how high I set the busy timeout. I think some of the busy timeout errors were actually from database/sql and not the internal library, since even setting a 10 minute timeout, the test would error out after 20 seconds or so.

I ran the benchmark again, results in the other thread https://github.com/lightningnetwork/lnd/pull/7252#discussion_r1074022579.

If we keep with the read/write connection design, then I think we want to set this to 2. AFAICT, if we set this to `1``, then we only have a single logical connection

From what I see in the database/sql code, the pool is kept per DB instance. So I would think there are separate pools for read and write.
I briefly thought that our own "pooler" dbConnSet may have something to do with it, but because the dsn for the read conn differs from the write dsn, it should make two database/sql connections.

uncommitted writes would be seen in read transactions

I don't think this ever happens regardless of number of connections unless you set `PRAGMA read_uncommitted?

I don't think this ever happens regardless of number of connections
unless you set `PRAGMA read_uncommitted
https://www.sqlite.org/pragma.html#pragma_read_uncommitted?

Referring back to those isolation docs, this is what led me down that line
of reasoning ("...prior to the commit):

SQLite provides isolation between operations in separate database
connections. However, there is no isolation between operations that occur
within the same database connection.
In other words, if X begins a write transaction using BEGIN IMMEDIATE
https://www.sqlite.org/lang_transaction.html#immediate then issues one or
more UPDATE https://www.sqlite.org/lang_update.html, DELETE
https://www.sqlite.org/lang_delete.html, and/or INSERT
https://www.sqlite.org/lang_insert.html statements, then those changes
are visible to subsequent SELECT
https://www.sqlite.org/lang_select.html statements
that are evaluated in database connection X. SELECT
https://www.sqlite.org/lang_select.html statements on a different
database connection Y will show no changes until the X transaction commits.
But SELECT https://www.sqlite.org/lang_select.html statements in X will
show the changes prior to the commit.

I think in practice, most applications either have a central mutex around a
single connection, or have multiple processes/threads each with their own
connection. This would then support our approach of two connections: one
for read transactions, and one for writes (using immediate, so there's only
ever a single active write).
On Wed, Jan 18, 2023 at 12:12 PM 'Joost Jager' via sys-deploy
@.***> wrote:

I ran the benchmark again, results in the other thread #7252 (comment)
https://github.com/lightningnetwork/lnd/pull/7252#discussion_r1074022579
.
If we keep with the read/write connection design, then I think we want to
set this to 2. AFAICT, if we set this to `1``, then we only have a single
logical connection
From what I see in the database/sql code, the pool is kept per DB
instance. So I would think there are separate pools for read and write.
I briefly thought that our own "pooler" dbConnSet
https://github.com/lightningnetwork/lnd/blob/c38be044d362f3cb3d9ca602de15115f70c5bad4/kvdb/postgres/db_conn_set.go#L42
may have something to do with it, but because the dsn for the read conn
differs from the write dsn, it should make two database/sql connections.
uncommitted writes would be seen in read transactions
I don't think this ever happens regardless of number of connections unless
you set `PRAGMA read_uncommitted
https://www.sqlite.org/pragma.html#pragma_read_uncommitted?
—
Reply to this email directly, view it on GitHub
https://github.com/lightningnetwork/lnd/pull/7252#issuecomment-1387714907,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AN37PGL2FZCCLGYWWRUA5I3WTBFBLANCNFSM6AAAAAAS6ZIL2Y
.
You are receiving this because you commented.Message ID:
@.***>

However, there is no isolation between operations that occur within the same database connection.

If true, this is not good. In LND, we are using one or a few connections from many different goroutines. Read uncommitted is really not what we want.
I've tried to reproduce dirty reads on a single connection, but didn't succeed so far. This is my code: https://gist.github.com/joostjager/843d693a10a66e5c5f38b03efa3dc108
Maybe I am just lucky that the undefined behavior is 'no dirty reads' with my version of sqlite. Strictly following the docs though, it seems unsafe to use a connection concurrently. But this is what we do?
Or perhaps the only thing that saves us is that split between read and write connection in combination with IMMEDIATE. The read connection is separate and can never read dirty. The write connection is only one tx at a time, so also no dirty reads. But without IMMEDIATE, I think that in theory we may get dirty reads when there are concurrent txes that start by reading some data and are then upgraded.
PS: Github comments via email are bad 😞

PS: Github comments via email are bad 😞

Heh, realized I replied using my other email...
Looking at this section:

A query sees all changes that are completed on the same database connection prior to the start of the query, regardless of whether or not those changes have been committed.

So then the changes are visible when they are completed and before they have been committed. Which seems like writing to the WAL is a complete change, but it's only committed after the checkpoint process? We could also be reading a bit too deeply into everything. sqlite has a ton of docs which is great, but at times there's a bit of inconsistency between them.
Either way, it seems the main thing to avoid is read_uncommitted.
Yeah I think SQLITE_OPEN_FULLMUTEX projects us in this case. So then this serializes the operations on the connection object. Another connection object uses a distinct thread so it can access the DB concurrently.

If we use two or more connections, could shared cache be beneficial?

Hmm, idk...the docs (section 1.1) say that it's discouraged shortly after the intro paragraph. Looks like hte feature was added 15 or so years ago to work better on the now defunkt Symbian mobile phones.

Would there be something to learn from c-lightning as they use sqlite too?

Potentially. One big architecture difference is the CL is mainly process driven. It's possible they have a single process that handles all the DB access, with the others passing requests over file descriptors.
In terms of the main connection, it looks like they don't pass any of the thread safety related calls: https://github.com/ElementsProject/lightning/blob/82c94330a55d5925d802df86d43bf44e779d2738/db/db_sqlite3.c#L123-L143.
It looks like the only pragma option they have set is foreign_keys. One of their tests also looks to use the same busy timeout: https://github.com/ElementsProject/lightning/blob/82c94330a55d5925d802df86d43bf44e779d2738/tests/db.py#L31-L33.

latest push has:

max db conns = 2
synchronous=full
fullfsync=true

But on my Mac, the async payments benchmark test hangs with these options. Only if I remove the fullfsync=true do things work...

My tests above showed the opposite. The default value of 50 meant that just about every operation would lead to a timeout, which actually emanated from the database/sql library.

I just created a simple test program with a single connection and two (slow) goroutines that do reads. It showed that they are blocking each other out. Maybe 50 connections does allow a lot more concurrency on read, but also triggered another effect that caused the timeout.

with a single connection and two (slow) goroutines that do reads.

curious what pragma options you set on the connection? Do you set query_only at all? and is _txlock set as the "deferred" default?

I didn't add _txlock, left it to default deferred. But for reads that shouldn't matter? I tried with query_only, but no difference, still readers blocking each other.
This is my test code: https://gist.github.com/joostjager/9d44df6d2a7a4fba27c5f5308df89d0c

due to the two-conn work around being removed in the first PR, I have changed the "max-conns" here to be 1.
Also noticed that on my mac: async benchmark doesnt work with both synchronous=full and fullfsync=true set but does work with synchronous=normal + fullfsync=true or synchronous=full + fullfsync=false

With a single connection, I think that there will only ever be a single db transaction running (regardless of whether it is read or write) because of SQLITE_OPEN_FULLMUTEX.
But also with multiple connections that can all do read and write txes, maybe reads get blocked by writes and don't benefit from reading the WAL to not get blocked by writers?
I still don't fully understand how to best configure sqlite.

But also with multiple connections that can all do read and write txes, maybe reads get blocked by writes and don't benefit from reading the WAL to not get blocked by writers?

My understanding is that for the WAL log, readers can just read:

WAL provides more concurrency as readers do not block writers and a writer does not block readers. Reading and writing can proceed concurrently.

Without the WAL, they'd need to block as the writer would directly modify the db file/journal. With the WAL log, they can just consult the index, then read from the base db and apply w/e changes are staged in the WAL. I think this blog post explains the interaction pretty well: https://fly.io/blog/sqlite-internals-wal/.
In terms of the effect of SQLITE_OPEN_FULLMUTEX, I think its mainly guards certain critical blocks and doesn't block the process of creating new read/write transactions (fine grained, not some coarse mega lock).

Also noticed that on my mac: async benchmark doesnt work with both synchronous=full and fullfsync=true set but does work with synchronous=normal + fullfsync=true or synchronous=full + fullfsync=false

Attempting to reproduce the same thing on my Mac Book:
bolt on master:
    lnd_payment_test.go:348: 	Benchmark info: Elapsed time:  41.977280583s
    lnd_payment_test.go:349: 	Benchmark info: TPS:  11.506224159637577

the branch as is:
    lnd_payment_test.go:348: 	Benchmark info: Elapsed time:  26.357677917s
    lnd_payment_test.go:349: 	Benchmark info: TPS:  18.324831251104936

synchronous=normal + fullfsync=true:
    lnd_payment_test.go:348: 	Benchmark info: Elapsed time:  1.956410833s
    lnd_payment_test.go:349: 	Benchmark info: TPS:<PHONE_NUMBER>847257

synchronous=full + fullfsync=false:
    lnd_payment_test.go:348: 	Benchmark info: Elapsed time:  1.668359334s
    lnd_payment_test.go:349: 	Benchmark info: TPS:<PHONE_NUMBER>133335

So I'm not able to reproduce your instances of things "not working" @ellemouton with any of the params. This is an M1 macbook (the very first that came out).

Pushed a new tag after the merge of the base of this PR: kvdb/v1.4.0.

ok - ran it again with the latest kvdb tag and got  (Macbook pro M1-Pro chip):
The PR as is: max conn = 1, synchronous=full, fullfsync=true:
lnd_payment_test.go:348:    Benchmark info: Elapsed time:  1m41.150692875s
lnd_payment_test.go:349:    Benchmark info: TPS:  4.775053796189827

max conn = 1, synchronous=normal, fullfsync=true
 lnd_payment_test.go:348:    Benchmark info: Elapsed time:  2.898909541s
 lnd_payment_test.go:349:    Benchmark info: TPS:<PHONE_NUMBER>3249596

max conn = 1, synchronous=full, fullfsync=false
 lnd_payment_test.go:348:    Benchmark info: Elapsed time:  1.661073291s
 lnd_payment_test.go:349:    Benchmark info: TPS:<PHONE_NUMBER>4213963

erg - I accidentally had the etcd test fixes in this PR instead of the base kvdb one 🙈
so we will need to merge this fix PR first & tag a kvdb patch: https://github.com/lightningnetwork/lnd/pull/7352
Sorry about this 🙈 rookie mistake

Re-ran the lightning benchmark on cloud machine with the branch as is (20559d720bfde7b012bd841a208ecd0a9e339163), and got 50 tps. Looks in line with other results.

looks like the resolution_handoff itest is consistently failing for the sqlite backend 🙈

works if I chance maxDbConns to 2 (up from 1)

Still the question remains then whether a low number of max connections is ideal for performance given that readers do seem to block each other.

See my tests on the linux machine above, if you have enough cores, then things can scale well. Otherwise with a "normal" amount of cores, then performance suffers as you increase the number of max connections.

And also how to avoid deadlocks caused by multiple readers using a small set of connnections?

IIUC, the prior approach of distinct read/write connections allows this to succeed, as with the WAL mode, then readers don't block writers and the other way around. I think it's cool we were able to update the sqlite library so we can rely on the tx options, but it does seem like the two connection route allows for a better level of parallelism.

Dug up some more documentation/discussion re multi-threading and concurrency: https://sqlite.org/forum/forumpost/2a53b75034770302?raw

However, only one thread may be executing in a connection at a time. That means that if one thread executes a command on a connection, then for the duration of that thread executing "inside the library", any other thread attempting to enter the library "on a different thread" will have to wait. In other words the SQLite3 library is serially entrant per connection, not multiple-entrant per connection. This is a consequence of how it is built and cannot be changed. Note that a "statement" is a child of a "connection" and as such "statements" are not isolated from each other -- it is at the parent connection level that isolation occurs.
You do not have to keep opening/closing connections, but you probably want each thread to open and use its own connection so that the processing on each thread is not visible to (or have impact on) other threads/connections until that thread/connection commits its changes. If you do this SQLite3 will arbitrate each threads/connections access and view of the database individually, which is I think what you mean.

Which argues for the prior approach where we have a read connection that can parallelize reads using the WAL, and another write connection that serializes the writes with the exclusive lock.

Which argues for the prior approach where we have a read connection that can parallelize reads using the WAL, and another write connection that serializes the writes with the exclusive lock.

Also on a dedicated read connection with SQLITE_OPEN_FULLMUTEX, I don't see parallelization of reads happening in my tests. I believe that the only way to get this parallel reading from the WAL is to use a separate connection for each thread.

Also on a dedicated read connection with SQLITE_OPEN_FULLMUTEX, I don't see parallelization of reads happening in my tests. I believe that the only way to get this parallel reading from the WAL is to use a separate connection for each thread.

As was shown above, there seems to be a sweet spot: too many active connections (managed by the database/sql connection pool) serves to slow things down at a certain point. If I pull things down on mac, and run w/ 10 conns, then the async test doesn't even pass. On my linux machine (has a ton of cores), increasing that value beyond 2 sees a degrade in perf (~299 TPS (2 conns), to (50 conns) ~ 50 TPS). If I remove our call to SetMaxOpenConns, then I see a similar drop off (~90 TPS).
The purpose of the pooled connections in the stdlib package is to get around that libraries like sqlite only allow one "operation" on a connection at a single time: https://go.dev/doc/database/manage-connections
At this point, all we have are conjectures w/o proper instrumentation, so we're just in bikeshedding territory here. From all my tests above, sqlite is faster than bbolt, if it's actually doing that without giving us access to concurrent reads, then that's pretty amazing. The maxconns (increase number of conns in the pool, for greater concurrency at tradeoff of more over head) setting is available on the CLI, so we can further experiment w/ tweaking that to enable more concurrency once this lands. Once this lands, we can do further benchmarks for things like read-heavy workloads (a site that continually hits channelbalance every 2 seconds or w/e).

The only question that is still on my mind is why exactly more connections make it so much slower. I also thought of removing SetMaxOpenConns for no maximum so that every goroutine can have its own non-blocked reads, but apparently that doesn't work.
A limit of 1 leading to deadlocks and a limit of 10 making it incredibly slow feels like there is not too much leeway in setting the right number of connections.

I also thought of removing SetMaxOpenConns for no maximum so that every goroutine can have its own non-blocked reads, but apparently that doesn't work.

Yeah I tried that above, so leaving the library to its defaults, and that has similar degradation. I think the next route here once we land this PR is to run the same benchmarks connected to profiling tools, so we can get things like flame graphs to see why perf degrades so much.
It might be something in databases/sql (like the MaxIdleConnections config), or something else deeper in the sqlite library. It also might just be the fact that we make a ton of db transactions (as you've documented in other issues) in the payment and path finding critical path that slow things down.
FWIW as mentioned above, if you have a lot of cores and a fast SSD then things don't drop off so much. With my runs above, it at least appears as though the current defaults are as fast, or even faster than default bbolt.

I think the next route here once we land this PR is to run the same benchmarks connected to profiling tools, so we can get things like flame graphs to see why perf degrades so much.

I tried to reproduce the slow down with a simple (non lnd/kvdb) read benchmark, but I see nothing bad happening at 500 simultaneous connections.
One thing that I remember seeing in one of my tests is that the scheduling of sqlite transactions may not be optimal. In some cases I got one connection that got starved because it was never its turn to run. Maybe that has something to do with it too.
To me it seems that executing the benchmark that you propose isn't that much work, and this could hopefully complete the picture.
I tried to run in though, but got:
~/lightninglabs/lnd (sqlite-pt2 ✔) make itest icase=async_payments_benchmark dbbackend=sqlite temptest=true
 Building itest btcd and lnd.
CGO_ENABLED=0 go build -v -tags="rpctest" -o lntest/itest/btcd-itest -ldflags " -X github.com/lightningnetwork/lnd/build.Commit=kvdb/v1.4.1-13-gfe5254510" github.com/btcsuite/btcd
CGO_ENABLED=0 go build -v -tags="dev kvdb_sqlite autopilotrpc chainrpc invoicesrpc neutrinorpc peersrpc routerrpc signrpc verrpc walletrpc watchtowerrpc wtclientrpc rpctest btcd" -o lntest/itest/lnd-itest -ldflags " -X github.com/lightningnetwork/lnd/build.Commit=kvdb/v1.4.1-13-gfe5254510" github.com/lightningnetwork/lnd/cmd/lnd
github.com/lightningnetwork/lnd/kvdb/sqlbase
# github.com/lightningnetwork/lnd/kvdb/sqlbase
../../go/pkg/mod/github.com/lightningnetwork/lnd/kvdb@v1.4.1/sqlbase/db.go:12:2: could not import sync/atomic (open : no such file or directory)
make: *** [build-itest] Error 2

I tried, but ran into:

That's odd, which version of Go are you running?
Also re CLN: confirmed that all db access for them is single threaded (one connection for the entire process).
My point above was simply that further benchmarking shouldn't block forward progress towards landing this PR so users can start testing against it in master.

That's odd, which version of Go are you running? That looks like a GOPATH issue perhaps?

Not sure what it was, but go clean -modcache fixed it.


Update expressroute-routing.md 02

From the internal email thread, we know the China cloud support the service tag for BGP community value for Microsoft peering.
Add the below part info:
| China |  |
| Exchange Online |12076:5210 |
| SharePoint Online |12076:5220 |
| Skype For Business Online |12076:5230 |
| CRM Online |12076:5240 |
| Other Office 365 Online services |12076:5300 |
Remove:
Office 365 communities are not supported over Microsoft Peering for Azure China region.
Detailed info from PG team:
ExRBgpMooncakeCommunitiesList_2022-11-01.txt
"value": [
{
"name": "Exchange",
"id": "/subscriptions//resourceGroups//providers/Microsoft.Network/bgpServiceCommunities/Exchange",
"type": "Microsoft.Network/bgpServiceCommunities",
"properties": {
"serviceName": "Exchange",
"bgpCommunities": [
{
"serviceSupportedRegion": "Global",
"communityName": "Exchange",
"communityValue": "12076:5210",
"communityPrefixes": [],
"isAuthorizedToUse": false,
"serviceGroup": "O365"
}
]
}
},
{
"name": "OtherOffice365Services",
"id": "/subscriptions//resourceGroups//providers/Microsoft.Network/bgpServiceCommunities/OtherOffice365Services",
"type": "Microsoft.Network/bgpServiceCommunities",
"properties": {
"serviceName": "OtherOffice365Services",
"bgpCommunities": [
{
"serviceSupportedRegion": "Global",
"communityName": "Other Office 365 Services",
"communityValue": "12076:5300",
"communityPrefixes": [],
"isAuthorizedToUse": false,
"serviceGroup": "O365"
}
]
}
},
{
"name": "SharePointOnline",
"id": "/subscriptions//resourceGroups//providers/Microsoft.Network/bgpServiceCommunities/SharePointOnline",
"type": "Microsoft.Network/bgpServiceCommunities",
"properties": {
"serviceName": "SharePointOnline",
"bgpCommunities": [
{
"serviceSupportedRegion": "Global",
"communityName": "SharePoint Online",
"communityValue": "12076:5220",
"communityPrefixes": [],
"isAuthorizedToUse": false,
"serviceGroup": "O365"
}
]
}
},
{
"name": "",
"id": "/subscriptions//resourceGroups//providers/Microsoft.Network/bgpServiceCommunities/",
"type": "Microsoft.Network/bgpServiceCommunities",
"properties": {
"serviceName": "",
"bgpCommunities": [
{
"serviceSupportedRegion": "Global",
"communityName": "Skype For Business",
"communityValue": "12076:5230",
"communityPrefixes": [],
"isAuthorizedToUse": false,
"serviceGroup": "O365"
},
{
"serviceSupportedRegion": "Global",
"communityName": "CRM Online",
"communityValue": "12076:5240",
"communityPrefixes": [],
"isAuthorizedToUse": true,
"serviceGroup": "CRM"
}
]
}
},

@duongau - Can you review this old PR and determine whether it needs to be closed or merged?
IMPORTANT: When the changes are ready for publication, add a #sign-off comment to signal that the PR is ready for the review team to merge.

Thank you for you dedication to our documentation.
Unfortunately, we have been unable to review this issue in a timely manner. We sincerely apologize for the delayed response. We will create an internal tracking work item to address this issue. For now, we are closing this issue. If you have any additional information you would like to provide, please respond to this issue with any additional details.
Please continue to provide feedback about the documentation. We appreciate your contributions to our community.
#please-close