Strange crash after some time (mprotect WTFCrash())

Which version of PhantomJS are you using? Tip: run phantomjs --version.
2.1.1

What steps will reproduce the problem?

run my script https://github.com/k3a/phantomfb

it works for a while but then when used after couple of days again

it does SIGSEGV

Which operating system are you using?
GNU/Linux on Intel x86_84

Did you use binary PhantomJS or did you compile it from source?
First I used official Arch Linux binary but then compiled from the source to reproduce the bug with debug symbols

Please provide any additional information below.

#0  0x000055555721459c in WTFCrash () at wtf/Assertions.cpp:345
#1  0x0000555557261b07 in WTF::OSAllocator::commit (address=0x7ffe37cc0000, bytes=65536, writable=<optimized out>, executable=<optimized out>) at wtf/OSAllocatorPosix.cpp:189
#2  0x0000555557245364 in WTF::PageAllocationAligned::allocate (size=size@entry=65536, alignment=alignment@entry=65536, usage=usage@entry=WTF::OSAllocator::UnknownUsage,
    writable=writable@entry=true) at wtf/PageAllocationAligned.cpp:61
#3  0x0000555556f1822a in JSC::ExcessRegion::create (blockSize=65536) at heap/Region.h:179
#4  JSC::Region::create (superRegion=0x7ffff7eae058, blockSize=65536) at heap/Region.h:232
#5  JSC::BlockAllocator::allocate<JSC::MarkedBlock> (this=0x7ffff7eae058) at heap/BlockAllocator.h:152
#6  JSC::MarkedAllocator::allocateBlock (this=this@entry=0x7ffff7eb3df8, bytes=bytes@entry=32) at heap/MarkedAllocator.cpp:115
#7  0x0000555556f18c13 in JSC::MarkedAllocator::allocateSlowCase (this=0x7ffff7eb3df8, bytes=32) at heap/MarkedAllocator.cpp:97
#8  0x0000555556f752a1 in JSC::MarkedAllocator::allocate (bytes=<optimized out>, this=<optimized out>) at heap/MarkedAllocator.h:82
#9  JSC::MarkedSpace::allocateWithoutDestructor (bytes=<optimized out>, this=<optimized out>) at heap/MarkedSpace.h:205
#10 JSC::Heap::allocateWithoutDestructor (bytes=<optimized out>, this=<optimized out>) at heap/Heap.h:386
#11 JSC::allocateCell<JSC::JSFinalObject> (size=<optimized out>, heap=...) at runtime/JSCellInlines.h:97
#12 JSC::JSFinalObject::create (structure=0x7fff8beb9168, exec=<optimized out>) at runtime/JSObject.h:1052
#13 JSC::constructEmptyObject (structure=0x7fff8beb9168, exec=<optimized out>) at runtime/ObjectConstructor.h:65
#14 JSC::DFG::operationNewObject (exec=<optimized out>, structure=0x7fff8beb9168) at dfg/DFGOperations.cpp:417

From the source it seems that mprotect() failed. But I am not sure if it is a good idea to just crash the app without any explanation. In my case it could have been due to a temporary low memory situation. During that low memory situation, I haven't been actively interacting with the app, but maybe it does some allocations even while idling...

Hmm, closing as it is a WebKit part of the code and unlikely to be changed.


Backup nodes + httpchk

This patch allows to set stub backends as backup and adds the ability to use the Haproxy httpchk option with custom check ports for those nodes.

Can we please have this merged?
It has been quite a while, is the project still maintained?
Thanks

Sorry @Jaykah, the project maintainer @igor47 is really busy with other projects for Airbnb these days. We're doing our best to find time to respond to pull requests, and we're sorry contributors are waiting so long. Please bear with us in the meantime.

Hi,
Any updates on the potential merge?
Thanks!

@clizzin @igor47 Hi guys, when do you think it will be possible to have this reviewed?

@clizzin @igor47 Just wanted to bump this, since it is a one-line change that, if merged, would make our lives much easier :) There's not much to review either...

ah sorry about the delay; this could use a rebase.
also, we handle this problem for non-default servers with a server_options argument, which can cover all of the use cases you introduce plus any other arbitrary ones. maybe this PR should also just include a server_options argument for the backup servers section instead?

Rebase done.
The reason it's done without server_options is to assign "backup" labels to individual backup nodes instead of the whole group (if I understand your question correctly).

We're also adding something like this for the weight change as well in #131. Perhaps we should add something like server_options to the backend registrations themselves to allow this kind of metadata to be propagated from nerve? If we're going to do that it might make senseto keep all this frontend specific metadata in one top level key?

@jolynch Sure, that's a good approach for ZK-listed servers. However, this particular scenario deals with static default servers, so I don't see an overlap there.

@Jaykah The static registrations from the default servers should (afaik) be identical to what nerve is registering as far as Synapse can tell. In BaseWatcher's set_backends method they are just merged all together.
I guess what I'm saying is that we can solve both use cases with a generic "server_options" top level key in backend registration hashes.

@jolynch
What I am referring to is the "stub" watcher, rather than any backend
discovered via Zookeeper/Docker/etc. My use case is a server in Region A
that would use backends from Region B only if all local nodes fail. Those
backends are marked as "backup" in the Synapse config. At the same time,
region B's backends are primary for their own region (so they won't be
marked as "backup" in Nerve), and are discovered through a different path
in Zookeeper.
A better option would be to discover backup nodes from another Zookeeper
path, but that's a different discussion (we would still need default
servers if ZK fails).
On Thursday, August 27, 2015, Joseph Lynch<EMAIL_ADDRESS>wrote:

@Jaykah https://github.com/Jaykah The static registrations from the
default servers should (afaik) be identical to what nerve is registering as
far as Synapse can tell. In BaseWatcher's set_backends method they are just
merged all together.
I guess what I'm saying is that we can solve both use cases with a generic
"server_options" top level key in backend registration hashes.
—
Reply to this email directly or view it on GitHub
https://github.com/airbnb/synapse/pull/100#issuecomment-135567460.

--
Best Regards,
Jay Jideliov

Ah, that's good context. We're experimenting with zone failover by using multiple backends and a frontend acl. We've been relying on the implicit backend name for a bit but I opened #135 to make it explicit so you can reference backends in acls. That being said the HAProxy guys recommend combining multiple backends + acl with backups so we should probably support it. Definitely interested in coming up with a good way to do failover.
I think I'm starting to see your use case, I'm just trying to find the way to address a few similar feature requests. I feel like adding a bunch of HAProxy specific keys (like httpchk) is the wrong way going forward, especially if we ever want to be able to add more outputs than HAProxy/file output. I'm not sure the right abstraction though, what do you think?
Also just FYI you can use the state cache as an additional guard (it's the only guard we use) against zookeeper flakiness.

We've been relying on the implicit backend name for a bit but I opened #135 to make it explicit so you can reference backends in acls.

So if I understand correctly, in that case we would define two (or more) backends (main and backup) in frontend's Synapse and failover to that backup using ACL? If so, that seems to solve a part of what I am trying to achieve, since your example shows ZK discovery for the backup node, which can be done from a different ZK path.
Are there any limitations to such approach?

That being said the HAProxy guys recommend combining multiple backends + acl with backups so we should probably support it.

That's what I meant by
path, but that's a different discussion (we would still need default
servers if ZK fails).```

Having the backup option available on a per-node basis for stub watcher is complementary to #135

> I feel like adding a bunch of HAProxy specific keys (like httpchk) is the wrong way going forward, especially if we ever want to be able to add more outputs than HAProxy/file output. I'm not sure the right abstraction though, what do you think?

Sure, having more flexibility is always great, but we should not forget that even per Synapse's readme, "The heart of synapse is actually HAProxy, a stable and proven routing component."

> Also just FYI you can use the state cache as an additional guard (it's the only guard we use) against zookeeper flakiness.

What cache are you referring to? We are not using Curator, if you are talking about the patch cache.

@Jaykah
I think that we need a general way to inform config generators (HAProxy, fileoutput etc ...) of relevant metadata for backends. @bobtfish is working on adding weights in #131 and I think this change and that change need to be generalized.

Are there any limitations to such approach?

From my conversations with the HAProxy folks acls are preferred because they give you more control for when failover occurs, but the blog article I linked to seems to indicate they can be used together. In my opinion I think it strictly dominates backups because I can reason about failover really easily (does the backup backend have any traffic).

Having the backup option available on a per-node basis for stub watcher is complementary to #135

I agree, I'm just trying to figure out a way that we can unify #100, #131 and other similar features. I really do think that the right way to do this is to attach metadata to backends and we can have well specified behavior per output (HAProxy, files, etc ...)

Sure, having more flexibility is always great, but we should not forget that even per Synapse's readme, "The heart of synapse is actually HAProxy, a stable and proven routing component."

I can change the readme, but I believe it is in the best interest of the project to not make additional choices that tie us directly to HAProxy. We've already added file output which has been instrumental to getting things like Cassandra bootstrapping off Smartstack. We had to do some crazy stuff to get HAProxy to not drop traffic during reloads. HAProxy might not be the only solution we offer going forwards.

What cache are you referring to? We are not using Curator, if you are talking about the path cache.

I'm referring to the state_file_path and state_file_ttl options which allow you to cache backends between restarts. That and the restart_interval restart_jitter HAProxy options have helped out a ton for us.

I'm also potentially ok with using a bunch of top level keys as long as we namespace them or somehow make it obvious that httpchk doesn't do anything unless you're using HAProxy (as opposed to file or nginx or w.e. we build in future)

#83 also is relevant

@jolynch

I agree, I'm just trying to figure out a way that we can unify #100, #131 and other similar features. I really do think that the right way to do this is to attach metadata to backends and we can have well specified behavior per output (HAProxy, files, etc ...)

Sure, if we can select main and backup paths per Synapse (e.g. main path for the same region as Synapse, and then a separate "backup" block with a different ZK path for another region) that would solve the need for the backup option.

I'm also potentially ok with using a bunch of top level keys as long as we namespace them or somehow make it obvious that httpchk doesn't do anything unless you're using HAProxy (as opposed to file or nginx or w.e. we build in future)

Maybe prefix them with "haproxy" ?
haproxy-httpchk : boolean, use haproxy httpchk for this server, requires "option httpchk /" to be set under haproxy options
haproxy-chkport : custom port for httpchk

@Jaykah I think keeping it flat with well named tags can work going forward. Trying to come up with some kind of well defined nesting structure in the service registrations seems like a losing battle to me, but maybe I'm not creative enough.
Let's do this, but let's add some documentation in both nerve and synapse for what backend registrations can actually contain (e.g. right now it's host, port, name, weight?) and what they are. Let's try to keep metadata that could only ever be used for one config generator prefixed with that config generator.

@jolynch The issue with server_options (or haproxy_server_options) for that matter is that those are blanket rules that apply to all backends instead of individually (unless I am missing something).
So in my case it would be:
 services:
  database:
   default_servers:
    -
     name: "def-db1"
     host: "<IP_ADDRESS>"
     port: 3306
     haproxy-httpchk: true
     haproxy-chkport: 7201
    -
     name: "def-db2"
     host: "<IP_ADDRESS>"
     port: 3307
     haproxy-httpchk: true
     haproxy-chkport: 7202

which would allow different check ports per instance (useful for flat infrastructure with the possibility of having two nodes coexist on the same machine).
However if we have haproxy_server_options per backend, that would be fine as well.

@Jaykah, yes I mean in addition to the service backend's server options. So you can either set options per backend or per server.
Sweet, let's do that and I'll merge it.

#140


[3.x] [Net/ENet] Better handle truncated socket messages

Update ENet to latest upstream master branch (ea4607a90dbfbcf4da2669ea998585253d8e70b1 at the time of the PR), plus https://github.com/lsalzman/enet/commit/2a85cd64459f6ba038d233a634d9440490dbba12 , which fixes a DoS vulnerability in ENet library code originally discovered by @Facundo15 and reported to the Godot Security Team.
The fix is included in 4.0.4, 3.5.3, and 4.1.2, and all 4.2 beta releases.
See #79699 for the 4.x version.

Thanks!

Cherry-picked for 3.5.3.

I review the changelog before we merge upstream. I request that a description is always written for a pull request like this. If it's written in the master branch version, then that should be linked. I put a lot of effort into my pull requests so that they are understandable by the community, core members need to lead by example.

You are free to ask that, but this is not a polite, or constructive, way of providing feedback or criticism, please read the code of conduct.

@jasonwinterpixel This PR's description was intentionally vague because it fixes a security vulnerability, and we want to make it look uninteresting to potential malicious actors. I'm only mentioning that this is a security fix now because a release containing this fix (3.5.3) has been out for a month.

Serious question, how would you write it to be more polite?

First of all I first saw your very first message, before you edited it, which I think we can all agree was neither productive nor polite
But as I said, and though I understand your position, the way the message reads do me is demanding, so I'd suggest to approach it with a tone like "I'd like it if" or "I'd appreciate it if", as opposed to putting demands on the work of others, but language is complicated and perhaps the original intent does not always carry through in text, which I understand.
Have a nice day

@jasonwinterpixel the two PRs were originally under embargo, waiting to be applied in an official release, I have correctly updated the description now.

Thank you for updating the description. I will commence investigating to see if we are affected by any relevant security issues.


Import and export ActionType from dependency fusion-rpc-redux ^1.4.0

Looks good to me! Thanks @hugoe29 !

Oh, the export also needed to be changed. I went ahead and made the adjustment

Thanks @lhorie
It seems it is just waiting for the reviewer approval.


Tracking Issue: Validate Transactions

Flesh out zebra-chain and dependencies as needed to be able to validate a single transaction. This is enough extensive work that feeds into our Sync and validate the chain milestone.
Remaining chunks of work in approximate priority order:

[ ] implementing tree state management for Sapling (assuming checkpointing on Sapling activation)

[ ] nullifier set https://github.com/ZcashFoundation/zebra/issues/289
[ ] sapling note commitment (merkle) tree https://github.com/ZcashFoundation/zebra/issues/286 https://github.com/ZcashFoundation/zebra/issues/273

[ ] type out notes https://github.com/ZcashFoundation/zebra/issues/275
[ ] pull in merkle-tree hash functions and pedersen commitments https://github.com/ZcashFoundation/zebra/issues/334

[ ] the remaining pieces of validation

[ ] Spend, Output, and JoinSplit proof statements

[ ] Depends on Groth16 validation service - #305, maybe #406, #830

[ ] Which calls to bellman

[ ] SIGHASH?

[ ] consensus rules stuff - https://github.com/ZcashFoundation/zebra/issues?q=is%3Aissue++label%3Aconsensus+
[ ] script impl

[x] wrap zcash's zcashconsensus library and re-export it as a rust crate - https://github.com/ZcashFoundation/zebra/issues/622

[ ] integrate zcashconsensus with zebra-consensus https://github.com/ZcashFoundation/zebra/pull/708

[ ] publish the zcashconsensus crate so we don't have to use it as a git dependency
[ ] implement UTXO set tracking for inputs into verify_script
[x] add prerequisite consensus parameters https://github.com/ZcashFoundation/zebra/issues/716

[ ] reimplement zcashconsensus in rust - https://github.com/ZcashFoundation/zebra/issues/623

Duplicate of #196, merging.


OHRI-148 Today hts client tile and list

Hey @pirupius, why does this PR have all the commits? Are you sure your not having changes from working?

Be sure that UCSF-IGHS/ohri-148-today-hts-client-tile-and-list branch is rebased against dev

@samuelmale i also noticed it. Will rebase the feature branch to dev to see if it resolves it.

@samuelmale my branch PR to working had conflicts and during the rebase to resolve those conflicts, that's when these commits came in from the working branch.
Should i just create another branch and cherry-pick my commits only?

@samuelmale my branch PR to working had conflicts and during the rebase to resolve those conflicts, that's when these commits came in from the working branch.

We don't rebase our feature branches against working, you do that from your working folk.

Should i just create another branch and cherry-pick my commits only?

If that will help get rid of the other commits, yes.


Fix limit not being applied to query

Broken by https://github.com/spatie/laravel-queued-db-cleanup/pull/17
The above change broke the following test in our application which picked up that the deleteChunkSize is no longer being applied to the delete query:
1) Tests\Unit\Models\AddressTest::testCleanFactory
Failed asserting that two strings are equal.
--- Expected
+++ Actual
@@ @@
-'delete from `addresses` where `updated_at` <= ? and `addresses`.`type` = ? limit 1000'
+'delete from `addresses` where `updated_at` <= ? and `addresses`.`type` = ?'

This PR ensures that the new variable defined in #17 is used everywhere else in the function which fixes the above issue.

@freekmurze This is a major bug caused by the last release. Please can you take a look..

Thanks!


Release 1.1.2

This PR is to release 1.1.2 to the master branch. It should also bring the master branch up-to-date.

Wrong PR


LowRelayerFee coming up

Describe the bug
Seeing some transfers not processing due to LowRelayerFee. This should be a very rare occurence.
To Reproduce
Steps to reproduce the behavior:

Example: https://connextscan.io/tx/0x0c24bd10199b89da24956575cd793a57386471a775ba9c2605b8751835fa624c
Logs link: https://app.datadoghq.com/logs?query=@requestContext.transferId:0x0c24bd10199b89da24956575cd793a57386471a775ba9c2605b8751835fa624c&cols=service&index=mainnet-production&messageDisplay=inline&saved-view-id=1285114&stream_sort=desc&viz=stream&from_ts=1683698410356&to_ts=1683712810356&live=true

Additional context
Add any other context about the problem here.

More issues having trouble, @LayneHaber please check if related or file a new bug?


Password field is shown clear text on plan and always triggers change

For uptime_check_http resource (and maybe others) the password field isn't marked as sensitive so is always show clear text in the plan.
Also it triggers every plan as if it's been changed but it hasn't. I've used a lifecycle ignore_changes rule for now to get around this.

It was partially addressed by #14. Password field is now treated as sensitive.

The part where password makes resource constantly outdated will be addressed separately.

Password triggering resource update on every run is fixed with #16


Use docker latest tag for releases

staging images were tagged as latest
Only releases will be tagged as latest

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 87.716%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
6034

Relevant Lines:
6879

💛  - Coveralls


update the version of jmeter

Description

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):
Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist:

[ ] My code follows the code style of this project.
[ ] I have updated the documentation accordingly.

update jmeter

Codecov Report

:exclamation: No coverage uploaded for pull request base (trunk@e1d024a). Click here to learn what that means.
The diff coverage is 33.15%.

@@           Coverage Diff            @@
##             trunk       #1   +/-   ##
========================================
  Coverage         ?   61.11%
  Complexity       ?    11429
========================================
  Files            ?     1218
  Lines            ?    77733
  Branches         ?     7517
========================================
  Hits             ?    47503
  Misses           ?    27531
  Partials         ?     2699

Impacted Files
Coverage Δ
Complexity Δ

src/core/org/apache/jmeter/util/JMeterVersion.java
73.68% <ø> (ø)
2 <0> (?)

.../apache/jmeter/save/converters/ConversionHelp.java
77.1% <0%> (ø)
26 <0> (?)

src/core/org/apache/jmeter/JMeter.java
37.73% <0%> (ø)
45 <0> (?)

src/core/org/apache/jmeter/gui/HtmlReportUI.java
0% <0%> (ø)
0 <0> (?)

...ackend/graphite/AbstractGraphiteMetricsSender.java
0% <0%> (ø)
0 <0> (?)

...src/org/apache/jmeter/gui/HtmlReportGUISpec.groovy
6.25% <0%> (ø)
0 <0> (?)

.../core/org/apache/jmeter/samplers/SampleResult.java
75.51% <100%> (ø)
135 <0> (?)

...s/org/apache/jmeter/functions/ThreadGroupName.java
81.81% <100%> (ø)
5 <0> (?)

...g/apache/jmeter/functions/TestSimpleFunctions.java
100% <100%> (ø)
20 <1> (?)

...rg/apache/jmeter/util/keystore/JmeterKeyStore.java
24.44% <15.15%> (ø)
5 <2> (?)

... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e1d024a...0facf15. Read the comment docs.


Clarify the relationship between stem growth and woody productivity

@teixeirak I'm having trouble relating stem growth to woody productivity. Here's what i'm thinking:
Three cambial processes drive the relationship between stem growth and woody productivity: cell differentiation, cell enlargement, and cell wall thickening. Irreversible stem expansion occurs as a result of differentiation (20% of stem growth) and expansion (80% of stem growth), and accounts for ~10% of annual woody productivity[@cuny_woody_2015]. The remaining 90% of woody productivity is caused by cell wall thickening. It would stand to reason that years with identical stem growth have a similar number of cells which are similarly sized (increased latewood can be an issue here, as  you acknowledge). If we assume that cell wall thickness is controlled by processes within the tree (ie. hormonal responses instead of external climate drivers), then we can predict that cell wall thickness is relatively uniform within each cell type (early wood and late wood). Therefore, years with a similar amount of stem growth would sequester a similar amount of carbon during cell wall thickening, and we can say that stem growth equates to woody productivity.
But do we know what controls cell wall thickness? (Also, all this info is based on conifers, so it could just be wrong for our trees)

There's another issue here:
(3) woody productivity is the sum growth of all the trees in the forest. We don't actually calculate this, but rather infer that woody productivity will mirror the growth of the dominant trees. This is a reasonable assumption, and really we care more about how the trees are responding than how these 2 particular forests, with their particular species compositions, are responding. However, technically, we're not calculating woody productivity.

I'm more concerned about (3) than (2).

woody productivity is the sum growth of all the trees in the forest

So we need to make the case that our samples are indicative of the forests they come from? From a not-so-educated point of view, that seems like an easy logical jump for the chronologies. The dendrobands at SCBI also represent a pretty big sample size of the major species, and were spread throughout the plot, so I think that would be fine too. The HF dendrobands might be a little less certain but still seem representative enough for me...
So we could say something like, "Woody productivity is often calculated using a combination of stem growth measurements and forest censuses. By sampling key species across our study plots, we build a dendrometer band dataset that is representative of each species within their respective forests (ie. the response of the overall sample is likely the response of each individual). Because of this, we infer that woody productivity in each forest will mirror the growth of these dominant species. Similarly, tree core chronologies are built using a representative sample of tree cores from each site, so we draw the same conclusions about the correlation between ring width increment and woody productivity of each forest here."

Let me look at / work oon the wording in the context of the paper. I think we can be pretty brief.
I personally don't feel that it's necessary to prove that our sampled species are exactly representative of these forests, simply because the result we're finding applies to broadleaf deciduous trees in general and should not be dependent upon the particular composition of a patch of forest. If we were trying to quantify the C budget of a particular patch of forest, that would be a different issue.

Here's a relevant paragraph from Delpierre et al. 2016:

The bottom line is that:

stem volume growth is usually assumed to equate to woody productivity
however, this assumption is not fully valid
existing evidence (including in Delpierre, which I need to cite in this part of the discussion) does not indicate that the extra C in years with warm springs would go into formation of denser wood.

I think we can close this.


fix: throw out error when creating container with image non-existence

Signed-off-by: Allen Sun<EMAIL_ADDRESS>1.Describe what this PR did
This PR makes image check before creating a container, if the image does not exist, pouchd returns an error directly.
2.Does this pull request fix one issue?
fix #290
3.Describe how you did it
Return error in daemon's create code.
4.Describe how to verify it
root@ubuntu:~/go/src/github.com/alibaba/pouch# pouch create sdfghjkl
Error: failed to create container: {"message":"failed to create container: found 0 images, not only one: sdfghjkl"}

But I think the error message is not so readable. Maybe we should fix error message in following days. "message":"failed to create container: found 0 images, not only one: sdfghjkl"
5.Special notes for reviews
/cc @skoo87

LGTM


US 01.07.01

As a doer, I want to see what habits I will need to do today.

We would need to know today's date, and just query firebase for a date-done field to be put in a different category?


Include copy to another path

include copy to another path

@speduardo Thanks for the pull request. Can you give me the usage example? Like if the db is in the www directory what will be the path or if the db is at some other location then what will be the path?

por exemplo: eu coloco o db fora do projeto, um dos motivos é pra diminuir o tamanho do apk e outro flexibilidade para o usuário.
outro ponto é copiar o db interno da aplicacao para uma pasta externa

sory
for example : I put the db out of the project , one reason is to reduce the apk size and other flexibility for the user .
another point is to copy the internal db of application to an external folder

Can you show me the code, also I think the Pull request is incomplete, I do not see the usage of path variable in copyDBFrom function.

I`m commit all

i finish the code, but i don't get commit the code

tell me your email to sent the code

Yes, this is my email<EMAIL_ADDRESS>

It would be great, if this pull request be merged ASAP.
Thanks for both of you.

I haven't yet test the new code, but I have added the code in dev branch. You can download it from here and let me know how it goes when you test it.


Add support for SameSite cookie property

https://tools.ietf.org/html/draft-west-first-party-cookies-07
Basically add a new property SameSite with values of None, Lax and Strict.
The value should then be appended to cookies when the cookie is sent;
Set-Cookie: CookieName=CookieValue; SameSite=Lax;
Set-Cookie: CookieName=CookieValue; SameSite=Strict;

This has come up before (though I can't find the issue). We were going to wait for the RFC to finish.

Yea I couldn't find it either, I thought I had logged it before. I figured better safe than sorry.

This sounds like something that should be added to the CookiePolicy middleware.

Not so sure, after all we can set HTTP_Only and Secure on individual cookies, via CookieOptions. I feel it ought to go in there, along with the ability to override everything in the CookiePolicy middleware.

Yes, I meant in addition to implementing it here.

Ah :)

Backlogging until spec is more stable.  Ping me if you think otherwise.

@blowdart has requested a re-triage


Support for PDM's .pdm-python file

PDM places a .pdm-python (text) file in the root of the project with path to the python binary in the venv currently used.
I would like for it to be a suggestion when selection the interpreter - maybe even have the popup with a new venv found when it changes.
# Example
cat .pdm-python
/home/vscode/.local/share/venv/proxy/bin/python

See more https://pdm-project.org/latest/usage/venv/
Edit:
Multiple people have asked for use cases, so here is a list of some of them

You use multiple venv's due to either python version or packages installed
You keep your venv's out of the project folder to not "polute" the path (and save it from git clean)
You might have a shared venv between 2 or more projects (this is less common though)
I've tried to add the path with all the venv, but finding the correct between 30+ venvs is a hassle - especially is you have to change between two or more during development

Created a very not well tested prototype https://marketplace.visualstudio.com/items?itemName=KalleDK.pdminterpreter
When installed it wil change interpreter based on the .pdm-python automatic, and you can use "Select Interpreter based on PDM" to change between multiple pdm venvs

Also here for the code https://github.com/KalleDK/vscode-pdminterpreter @karthiknadig


NFQLB Update to change hash mode

Description
Set hash mode to 0 to load balance on tuple-5 using SCTP
Issue link
https://github.com/Nordix/nfqueue-loadbalancer/pull/21
Checklist

Purpose

[ ] Bug fix
[x] New functionality
[ ] Documentation
[ ] Refactoring
[ ] CI

Test

[ ] Unit test
[x] E2E Test
[x] Tested manually

Introduce a breaking change

[ ] Yes (description required)
[x] No

I tried with the e2e deployment modified with these:
---
apiVersion: meridio.nordix.org/v1
kind: Attractor
metadata:
  name: attractor-a-1
  namespace: red
  labels:
    trench: trench-a
spec:
  replicas: 2
  composites:
  - conduit-a-1
  gateways:
  - gateway-v4-a-1
  - gateway-v6-a-1
  vips:
  - vip-a-1-v4
  - vip-a-1-v6
  - vip-a-5-v4
  interface:
    name: ext-vlan0
    ipv4-prefix: <IP_ADDRESS>/24
    ipv6-prefix: 100:100::/64
    type: nsm-vlan
    nsm-vlan:
      vlan-id: 100
      base-interface: eth0
---
apiVersion: meridio.nordix.org/v1
kind: Vip
metadata:
  name: vip-a-5-v4
  namespace: red
  labels:
    trench: trench-a
spec:
  address: "<IP_ADDRESS>/32"
---
apiVersion: meridio.nordix.org/v1
kind: Flow
metadata:
  name: flow-a-w-sctp
  namespace: red
  labels:
    trench: trench-a
spec:
  stream: stream-a-i
  priority: 1
  vips:
  - vip-a-1-v4
  - vip-a-5-v4
  source-subnets:
  - <IP_ADDRESS>/0
  - 0:0:0:0:0:0:0:0/0
  source-ports:
  - any
  destination-ports:
  - "3500"
  protocols:
  - sctp

than ran these commands:
ncat --sctp <IP_ADDRESS> 3500 -p 58217
ncat --sctp <IP_ADDRESS> 3500 -p 58217

The traffic was load-balanced with the expected behavior (reached different targets)


Need a list of problematic citations

This could be compiled using the fin admin model validations, as well as google analytics reporting potentially once GA4 is configured. Discuss further to confirm.

Actually after chatting with @Vensberg, using fin admin should be fine, but we should also pull in the OA Policy ids.


Test simple migrate

Test a simple migration from version 11 to version 13

Could we rename __tests__/fixtures/test-prototype to __tests__/fixtures/test-v11-prototype to make it clear it is an old prototype?


json support

Hi,
how can I export the different collected metrics in json format ?
I saw that the java project has json module, but it was not clear to me how to use it.
would be great if you can share some example.
Thanks,

This question really belongs with the Java core project. The maintainers of this project have no experience with the json module. Please direct your question to the maintainers of the json module.


az k8s-extension create | Attempt to register a Resource Provider that is already registered

Fix issue #8293
This checklist is used to make sure that common guidelines for a pull request are followed.
Related command

General Guidelines

[ ] Have you run azdev style <YOUR_EXT> locally? (pip install azdev required)
[ ] Have you run python scripts/ci/test_index.py -q locally? (pip install wheel==0.30.0 required)
[ ] My extension version conforms to the Extension version schema

For new extensions:

[ ] My extension description/summary conforms to the Extension Summary Guidelines.

About Extension Publish
There is a pipeline to automatically build, upload and publish extension wheels.
Once your pull request is merged into main branch, a new pull request will be created to update src/index.json automatically.
You only need to update the version information in file setup.py and historical information in file HISTORY.rst in your PR but do not modify src/index.json.

Thank you for your contribution! We will review the pull request and get back to you soon.


[<IP_ADDRESS>9] Dark Steel Pickaxe right click on vacuum chest -> server crash

Not much more to say... Fully charged Dark Steel Pickaxe, right click on vacuum chest. Client was fine, server shut down.
Forge <IP_ADDRESS>1
Full crash log: http://pastebin.com/qvhAtung
Have a good day! :)

Already been fixed in #1895 for the upcoming 2.2.9 release.
Thanks for the report.


Synchronization between multiple editors

As discussed in #4850

@rlaverde, please work on this one.

I don't understand clearly what's the idea of this change, Sync all editorstacks, and move tabs in all editors is not the expected behavior, even you says that https://github.com/spyder-ide/spyder/pull/4850#issuecomment-318915842
I'm 👎 to this issue, It'll be confusing and somehow error prone, other editors doesn't sync tabs between multiple editors, neither in split view.

From a comment of @rlaverde :

Personally I'll prefer the both windows to be totally independent, not only the position of the tabs not to be synced, but also different files opened, and when closing a tab to only be closed in that window

Lets say I'm working on PR X with file 1, file 2, and file 3 opened in an Editor plugin. Now I need to work on PR Y that requires files 4, file 5, and file 6, but I don't want to close file 1, file 2 and file 3 because I will need them later. Then I need to work on another PR and so on. I usually end up with a very crowded tab bar in my editor.
One of the thing I would like is the ability to add more than one dockable editor plugin to Spyder. Each editor plugin would be completely independent from each other. The editorstacks within each editor plugin would be synced however.

@dalthviz, what was exactly this issue about? Sorry, I don't remember ;-)

@ccordoba12 I think this is related with the synchronization of the tabs to preserve the same order when opening a new window. Although I'm not quite sure either.


jenkins: Try local build for rpm-packaging first

Only use the "slow" way if a local build fails.

@dirkmueller any idea how I can execute the second shell builder only if the first failed?

We increased the build service project prio so builds should be faster now and we don't need this anymore.


Adds a newline between notes

What this PR does / why we need it:
Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #
Special notes for your reviewer:

Thanks for the contribution, before continue, you need to sign the CLA, also you should increment the chart version in the Chart.yml according to semver:

Given a version number MAJOR.MINOR.PATCH, increment the:

MAJOR version when you make incompatible API changes,
MINOR version when you add functionality in a backwards-compatible manner, and
PATCH version when you make backwards-compatible bug fixes.
Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format.

It seems that a new version of Redis has appeared, could you resolve the conflict?

/ok-to-test
/lgtm

@caarlos0 resolved - ty.

/lgtm


Domain activation is Failed - [BRIEF DESCRIPTION]

App Id: d2tz5th771zmyg
Region: ap-south-1
Step: Domain activation
Status: Failed

Note: Do not include information that is sensitive in nature such as your domain name, company etc.

Issue/question
A clear and concise description of what the issue/question is.
Error message
If there is an error message, please include it here.
Note: Be sure to check the message for sensitive information.
Additional information
Please add any other relevant information. Please feel free to include screenshots.

Hello @Abdulmoeidkhan Can you please share the error message that you see on the domain management screen in the console?


Err value: InvalidTickLen("xai\u{307}")'

2023-08-06T08:14:18.951224922+00:00 INFO ord::index::updater - Block 800453 at 2023-07-27 06:52:27 UTC with 3366 transactions…
thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: InvalidTickLen("xai\u{307}")', src/brc20/types/tick.rs:40:59
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
2023-08-06T08:15:14.653583764+00:00 INFO ord::index::updater - Block receiver disconnected: sending on a closed channel

hang on this error, how to fix it?

2023-08-06T08:14:18.951224922+00:00 INFO ord::index::updater - Block 800453 at 2023-07-27 06:52:27 UTC with 3366 transactions…
thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: InvalidTickLen("xai\u{307}")', src/brc20/types/tick.rs:40:59
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
2023-08-06T08:15:14.653583764+00:00 INFO ord::index::updater - Block receiver disconnected: sending on a closed channel

hang on this error, how to fix it?

We are aware of the issue and are currently working on fixing it. After conducting regression testing, we will merge #88.

The issue has been fixed by #88.


Blog app: add devise

In this PR we:

[x] Removed current_user method in ApplicationController
[x] Installed and set up devise
[x] Created a sign-up form that new users can register
[x] User can log in with a combination of email and password
[x] Stored password in the database
[x] Ask for a confirmation email
[x] Users can reset their password

Thanks @rloterh @Lizdev-05 😊


Migrate WML Client SDK

Investigate WML client sdk to v4.

This was delivered in downstream, we now use v4 for both ml and openscale.


Minor fixes

Going to put all my bug fixes in this PR. I'll re-add you as an approver when it's ready for merge @fixanoid


Clefable RCL 75 - Correct ability text in log

I over-removed when fixing the ability. Without powerUsed the "used
ability" text doesn't seem to appear in the game log, which is certainly
problematic.
Whoops.

Also, I suppose that any ability that doesn't display text when it is used is technically a bug?


HQQ FSDP

Store quantized weights same as compute dtype for FSDP training.
Added tests.
Tested 4-bit FSDP llama-7b training and compared it against BNB, shows similar loss plot. HQQ requires 50% less memory compared to BNB! Used alpaca dataset with context length: 512.

Related issue: https://github.com/mobiusml/hqq/issues/14
Note: 1-bit and 8-bit HQQLinear is failing in the test_hqq_linear test. Otherwise they work with Quantizer.quantize and Quantizer.dequantize

@mobicham - by way of background, this PR is need to allow HQQ to be used with FSDP for multi-GPU qlora training. We'll be releasing a blog post from Answer.AI soon showing folks how to use this functionality (we've also done it for bitsandbytes BTW). Let us know if you have any questions.

Thank you very much for your contribution @KeremTurgutlu  @jph00 ! Using multi-gpu to train quantized models is super valuable!
The PR looks mostly fine, I just have a few things if you don't mind. Fortunately, these are easy fixes. In fact, I already included them here (via gist since the PR is coming from your repo): https://gist.github.com/mobicham/af0b7676c587ff36c0607affc00795eb
Bugs
(B1): Error when loading quantized models via from_quantized - FIXED
The code breaks when trying to load a quantized model via model = HQQModelForCausalLM.from_quantized(model_id). That's because when loading a quantized model, the HQQLinear layer is initialized with linear_layer=None. So I moved the bias copy to the initialize(self) function.
https://gist.github.com/mobicham/af0b7676c587ff36c0607affc00795eb#file-quantize_fsdp_fix-py-L282
(B2): Error when predicting with models loaded via from_quantized  - FIXED
The code breaks when trying to do prediction with a saved quantized model because unpack_view_dtype is not defined in the meta-data. I put a check in load_state_dict(), if  unpack_view_dtype is not defined in meta, then add it.
https://gist.github.com/mobicham/af0b7676c587ff36c0607affc00795eb#file-quantize_fsdp_fix-py-L340
(B3): view_as_float as an argument of dequantize creates some issues  - FIXED
I moved  view_as_float to meta because this might create a couple of issues with some settings.
I also added default values for scale/zero when loading an (older) quantized model so it doesn't break.
(B4): torch.compile issues - OPEN
torch.compile fails with  view_as_float=True. This means that both model=torch.compile(model) and the HQQLinear.set_backend(HQQBackend.PYTORCH_BACKPROP_COMPILE) backend break.
Luckily, this doesn't happen when it's set to False.
I am not sure why that happens. I tried moving the view call inside @torch.jit.ignore and it still breaks.
Using torch.compile could also make training faster. I will try to take a look at it again.
Suggestions
(S1): Set default as  view_as_float=False
Setting view_as_float=False by default in the code and adding it as a quant param in hqq_base_quant_config()
This way, it is using the original int bitpacking by default and it's consistent with the format used by other packages like oobabooga as well as the already published models on Hugging Face. Also, there's the torch.compile issue.
So now you can simply pass it as an argument like this:
quant_config = BaseQuantizeConfig(nbits=4, group_size=64, offload_meta=True, view_as_float=True);
model.quantize_model(quant_config=quant_config, compute_dtype=torch.float16)

(S2): Slower forward pass with view_as_float
There's a slight overhead for the forward pass with the float view (~0.10 sec for Llama2-7B and ~0.25 sec for Llama2-70B on a Titan RTX) + there's the issue of torch.compile breaking. It is possible to revert back to int bitpacking after training for faster inference. This can be done via the patching functions:
def patch_linear_int_bitpacking(layer, patch_params):
	#or if you are using HQQ's peft, it should use layer.linear_layer instead
	if(hasattr(layer, 'W_q')):
		layer.W_q = layer.W_q.view(layer.meta['unpack_view_dtype'])
	return layer

model.base_class.patch_linearlayers(model, patch_linear_int_bitpacking, dict([(linear_tag, None) for linear_tag in model.base_class.linear_tags]))

I am not sure where we could put this, maybe in hqq/models/base.py. But no worries, this is not a blocker, we can take a look at it later.
(S3): Scale/Zero still treated as int
I see that view_as_float is set to False for the scale/zero point:
-Does the training work fine when the scale/zero point are quantized?
-Does the training work fine with meta_offloading=True?
If the scale/zero work fine with FSDP even-though they are int-packed when they are quantized, it means that, if we make W_q as a standard tensor instead of nn.Parameter it should work the same way and no need to do the float view. However, I think this would break a few things with the current implementation. I am just curious if it's done that way, maybe it will make training faster and use even less memory?
(S4): Tests
Regarding the tests, I am not sure why test_hqq_linear() would fail for 8-bit but not the rest. Luckily, there are other ways to check that the float view is not causing any issues:
This test checks the view dtype and compares with the original bitpacking logic. This one passes for all settings, which means everything is fine:
def test_floatview_bitpacking(self):
    shapes = [[32, 32], [256, 256], [512, 512], [1024, 1024], [2048, 2048], [4096, 4096], [8192, 8192], [8192, 4096], [8192, 128], [32, 4096]]
    for compute_dtype in [torch.float32, torch.float16, torch.bfloat16]:
        for nbits in [8, 4, 3, 2, 1]:
            for shape in shapes:
                bit_pack      = Quantizer.bit_to_packing[nbits]
                view_dtype    = Quantizer.unpack_view_dtype[bit_pack]
                W             = torch.randint(0, 2**nbits, shape, device='cuda').to(view_dtype)
                W_packed_orig = Quantizer.pack[bit_pack](W)
                W_packed_view = W_packed_orig.clone().view(compute_dtype)
                assert W_packed_view.dtype == compute_dtype
                assert torch.abs(W_packed_orig - W_packed_view.view(view_dtype)).max() < 1e-5

The second test I would like to suggest makes sure the forward pass produces the same results as the original bitpacking logic:
def test_hqq_linear_forward(self):
    HQQLinear.set_backend(HQQBackend.ATEN_BACKPROP)
    context_size = 4096
    batch_size   = 1
    for compute_dtype in [torch.float32, torch.float16, torch.bfloat16]:
        for nbits in [4,3,2]:
            for quant_zero in [False, True]:
                for quant_scale in [False, True]:
                    for offload_meta in [True, False]:

                    	#Maybe loop over group_sizes as well [8, 16, 32, 64, 128, 256]
                        quant_config_int   = BaseQuantizeConfig(nbits=nbits, group_size=64, quant_zero=quant_zero, quant_scale=quant_scale, offload_meta=offload_meta, view_as_float=False)
                        quant_config_float = BaseQuantizeConfig(nbits=nbits, group_size=64, quant_zero=quant_zero, quant_scale=quant_scale, offload_meta=offload_meta, view_as_float=True)

                        hqq_linear_int     = HQQLinear(self.m, quant_config_int, compute_dtype=compute_dtype,   del_orig=False)
                        hqq_linear_float   = HQQLinear(self.m, quant_config_float, compute_dtype=compute_dtype, del_orig=False)

                        x = torch.randn([batch_size, context_size, self.m.weight.shape[1]], device='cuda').to(compute_dtype)
                        with torch.no_grad():
                            y_int   = hqq_linear_int.forward(x)
                            y_float = hqq_linear_float.forward(x)

                        assert torch.allclose(y_int, y_float, rtol=1e-5)

I tried with a couple of combinations and it worked fine.
I also run some lm-evaluation-harness benchmarks and the numbers look ok.
(S5): Please provide an example
Can you please provide a full script example in the examples/fsdp folder so it's easy for people to get started? Something like this:
https://github.com/mobiusml/hqq/blob/master/examples/lora/train_hqq_lora_example.py

If you can review the gist I shared and check it works fine with your training code, you can simply do another commit with the changes and we can merge. We can take a look at the torch.compile issue + repacking to int later.
Looking forward to your answers and excited to see the blogpost on Answer.AI!

@mobicham thanks a lot for the detailed response and suggestions!
I incorporated the changes and also added new tests. FSDP training runs fine as before. Fixing torch.compile would definitely be great, I agree.
(S3): Scale/Zero still treated as int
FSDP only requires parameters and buffers to be a float type, so tensors in meta can be kept in torch.uint8. If we make W_q a non-parameter then it won't get sharded by FSDP when using FULL_SHARD strategy, which means losing all the memory savings of large model training - the main reason why we use FSDP. So the current approach looks good to me.
(S5): Please provide an example
Training script can be found here: https://github.com/AnswerDotAI/fsdp_qlora/blob/scaling_experiments/train.py. We are still actively working on it but should be finalized by the time we have the blog post ready.
FSDP model saving issue with HQQLinear
HQQLinear custom state_dict() method is causing issues when saving model with FSDP. So I added a custom model saving logic which will only save the updated lora weights: https://github.com/AnswerDotAI/fsdp_qlora/blob/6b2e126de531c369392e28916e432dc064159842/train.py#L1021-L1040

Thanks for the update @KeremTurgutlu ! I run more tests and it looks fine to me, so already merged!
I also created a new release https://github.com/mobiusml/hqq/releases/tag/0.1.5, updated the doc and made it available via pip.
Regarding saving FSDP models, LoRa weights should be saved separately anyway, that's how we do it as well https://github.com/mobiusml/hqq/blob/master/hqq/core/peft.py#L316 .
Feel free to close this if there aren't other questions. We can create a separate issue for torch.compile

SGTM!

@mobicham FYI there's a draft post here that we'll be announcing tomorrow, that includes a discussion of the great results we're seeing with HQQ. Let us know if you see any issues with it: https://www.answer.ai/posts/2024-03-06-fsdp-qlora.html

@jph00 Thank you very much for sharing the draft! It is a very nice read!
The only suggestion on our side would be using the official website for Mobius Labs https://www.mobiuslabs.com/ , the one mentioned in the article (https://mobiusml.github.io/) is very old!
Looking forward to see and share the final version!

@jph00 This reads really nice. I hope that future significant models will be trained using less powerful GPUs in remote parts of the world, potentially helping to level the playing field in AI. Upon this being live, would it be acceptable to link to and feature this on our blog, https://mobiusml.github.io/blog/, with a note expressing our delight at having contributed to this work?

Of course we'd be delighted!

Message ID: @.***>

@appoose FYI this is launched now. https://twitter.com/jeremyphoward/status/1765868543235805232

@jph00 It is now linked to our blog at https://mobiusml.github.io/blog/ . Looking forward for future collaborations!


fix(tokens): Tokens appear when they aren't really there

Related to the issue #93

:tada: This PR is included in version 1.15.4 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

217.4 PSF tokens awarded. TXID:
c324d59e8c463fcbf9711de4729a615d8f4f4a58b9c4a9c847f298088fc9bfba


URLs under /guide path do not have page titles changed

Expected behavior: page titles change to cover the current article top heading, that is visible in history and the browser panel.
Actual behavior: page title stays: "Developer Guides"
Apparent Fix Path: make this route use the logic of tech docs
File to be edited: this

https://github.com/DiamondLightSource/developer-portal/issues/681#issuecomment-2385688017


switchMap not unsubscribing depending on import of Observable

RxJS version: 5.5.6
In our app, we today encountered an issue with code which is essentially the same as the following. Please note that this exact code does also cause the issue in the app, using rxjs 5.5.6:
const pauser$ = new BehaviorSubject(false);
const timer$ = pauser$
  .switchMap(paused => paused ? Observable.never() : Observable.interval(1000))
  .startWith(0);

const id$ = Observable.interval(50).take(3)
id$
  .switchMap(() => timer$
    .switchMap(() => Observable.of(42).do(console.log))
  )
  .subscribe();

The behavior we saw was that instead of 0 0 1 2 … 2 … 2 …, all the (temporarily) started timers kept executing in parallel, as if switchMap didn't unsubscribe from it when id$ emitted.
After a lot of debugging, I eventually figured out that the bug only arises when Observable is imported as
import { Observable } from "rxjs";

but not when imported as
import { Observable } from "rxjs/Observable";

Unfortunately, I cannot reproduce the issue on Stackblitz independently of how I import. My feeling here is that there is some dependency which patches Observable, perhaps with an older, faulty version or something.
However, the problem I am having with this theory: I would think that in this case it breaks when importing from rxjs/Observable, but not when importing rxjs. But what we see is exactly the opposite.
I realize that with rxjs 5.5.0, patching Observable isn't the choice of things anymore and we do intend to move away from this. In the mean time, I would really like to understand how importing from rxjs breaks behavior so drastically and would appreciate any ideas on how to figure out what is messing with us here.

Possibly related: https://stackoverflow.com/questions/48815054/rxjs-switchmap-does-not-cancel-inner-merged-observable

This is getting stranger. Importing as
import { Observable } from "rxjs/Rx";
also makes the problem go away. So on goes through package.json and the other doesn't but I don't see why that should matter. I'm suspecting it has something to do with tree shaking. Will upload reproduce repo this weekend.

I'd look at the package-lock instead of only the direct dependencies. I do remember seeing "rx" in there which to my understanding is rxjs in an older version. But as explained in my first post I still can't make sense of it.

Reproduce steps:
https://github.com/kayjtea/rxjs3306
The problem arises from mixing the full RxJS import with the patch mechanism.

Interestingly enough, running the example on Stackblitz (https://stackblitz.com/github/kayjtea/rxjs3306) doesn't seem to reproduce the problem again.

I ran into the same issue just now.
My code is something like the following:
import {Observable} from 'rxjs';
// ....

Observable.combineLatest(paginationConfig$, filters$)
  .switchMap(([config, filters]) => this.api.getList(config, filters))
  .subscribe();

When I would change filters, the API call to getList is not cancelled (i.e. the observable is not being unsubscribed).
Changing to import to "rxjs/Observable" as described above also fixes the issue for me.
Additional info:
$ npm ls rxjs
<EMAIL_ADDRESS>C:\Development\kbart\kbui\source
+--<EMAIL_ADDRESS>| +--<EMAIL_ADDRESS>| | `--<EMAIL_ADDRESS>deduped
| `--<EMAIL_ADDRESS>deduped
`--<EMAIL_ADDRESS>

Seems like an issue with the build/bundling adding two versions of Observable. You may want to file an issue against angular-cli.
cc @IgorMinar @hansl

resolved using
import { Observable } from "rxjs/Observable";
instead of
import { Observable } from "rxjs"

The correct answer for RxJS 5.X is to just import from the proper location: rxjs/Observable
In 6.0.0 (beta) we've fixed this by only having one import location for Observable which is rxjs.


Log cluster App status on failure to standup cluster

What this PR does
Fixes https://github.com/giantswarm/roadmap/issues/3641
If the BeforeSuite fails we will attempt to grab the status of the cluster App to see if there has been any deployment errors.
Example:
{"level":"info","ts":"2024-08-22T13:55:13+01:00","msg":"Cluster App status: AppVersion='', Version='2.0.0-d7a8852053bca7655bf783c6e820cc85cdc4cc32', ReleaseStatus='deployed', ReleaseReason='', LastDeployed='2024-08-22 13:39:13 +0100 BST'"}

Checklist

[x] Update changelog in CHANGELOG.md.

Trigger e2e tests

/run cluster-test-suites

Failures are flakes and unrelated to this change.
This change can actually be seen in action in one of the failures:
{"level":"info","ts":"2024-08-23T08:10:14Z","msg":"connection to api-server not yet available - failed to get API group resources: unable to retrieve the complete list of server APIs: v1: Get \"[https://api.t-m6ubf43b4h3ng3qo3v.gaws.gigantic.io:6443/api/v1\](https://api.t-m6ubf43b4h3ng3qo3v.gaws.gigantic.io:6443/api/v1/)": EOF"}
  {"level":"info","ts":"2024-08-23T08:10:24Z","msg":"Attempting to get debug info for Cluster App"}
  {"level":"info","ts":"2024-08-23T08:10:24Z","msg":"Cluster App status: AppVersion='', Version='0.60.0', ReleaseStatus='deployed', ReleaseReason='', LastDeployed='2024-08-23 07:55:10 +0000 UTC'"}
  {"level":"info","ts":"2024-08-23T08:10:24Z","msg":"Getting events for the Cluster App"}
  {"level":"info","ts":"2024-08-23T08:10:24Z","msg":"No events found for Cluster App"}
  [FAILED] in [BeforeSuite] - /go/pkg/mod/github.com/giantswarm/cluster-standup-teardown@v1.21.0/pkg/standup/standup.go:75 @ 08/23/24 08:10:24.829
[BeforeSuite] [FAILED] [1204.110 seconds]

Adding skip label


Handler does not handle retry with same HttpRequestMessage instance

When using the standard resilience handler in Microsoft.Extensions.Http.Resilience, the retry logic reuses the same HttpRequestMessage instance for multiple requests. This causes an issue where the signature header is added twice.

var services = new ServiceCollection();

services.AddHttpClient("client name")
  .ConfigurePrimaryHttpMessageHandler(sp =>
  {
    var signatureAlgorithm = SignatureAlgorithm.Create(rsaOrECDsaAlgorithm);

    var config = new HttpMessageSigningConfiguration("key-id", signatureAlgorithm);

    return new SigningHttpMessageHandler(config);
  })
  .AddStandardResilienceHandler();

var serviceProvider => services.BuildServiceProvider();

var client = serviceProvider.GetRequiredService<IHttpClientFactory>().CreateClient("client name");

// Make requests using client.
// If a retry is triggered the "Signature" header will have one additional value for each retry attempt.

The cause for this is this line: https://github.com/IdentityStream/HttpMessageSigning/blob/e375fcae665184b7aeac99d366555d31faa1ab93/src/IdentityStream.HttpMessageSigning/Extensions/HttpRequestMessageExtensions.cs#L32-L33
The SetHeader method uses TryAddWithoutValidation(name, value), which adds the new value to the list of header values already present for that header.
It looks like the WcfHttpRequestMessage implementation is not affected, as the WebHeaderCollection.Set(name, value) method used replaces any existing value.
This change would change the behaviour to replace the existing value if there is one:
-            public void SetHeader(string name, string value) =>
-                Request.Headers.TryAddWithoutValidation(name, value);
+            public void SetHeader(string name, string value)
+            {
+                if (Request.Headers.Contains(name))
+                {
+                    Request.Headers.Remove(name);
+                }
+
+                Request.Headers.TryAddWithoutValidation(name, value);
+            }

I've also asked a question for whether delegating handlers are expected to handle processing the same HttpRequestMessage instance more than once: https://github.com/dotnet/extensions/discussions/4965https://github.com/dotnet/extensions/discussions/4965


Enhanced future date handling

Adding function to date_utils to fix future dates that follow a certain typo format, based on discussions with @mike.  This function is now called as part of the isanity check on the AbstractSite base class. Also added new unit test to test the function.  Bumped release version as well.

you can sqash this.  I am going to post a whole new PR that doesn't conflict


PIP module still thycotic

PIP module still thycotic
After installing through pip with version python-tss-sdk 1.1.0, the import "from delinea.secrets.server import PasswordGrantAuthorizer" fails. It is still thycotic "from thycotic.secrets.server import PasswordGrantAuthorizer"
Expected behavior
"from delinea.secrets.server import PasswordGrantAuthorizer" should be working
Actual behavior
"from thycotic.secrets.server import PasswordGrantAuthorizer" is needed for import
Your environment
Windows 10, Python 3.10.4, python-tss-sdk 1.1.0
Steps to reproduce
Install python -m pip install python-tss-sdk and try to import
from delinea.secrets.server import PasswordGrantAuthorizer

@sami1979 can you please verify that the issue is resolved in the Release v1.2.0?

Closing until we get more feedback.


APEX-101 #resolve Do not use int when calculating memory allocated because it will overflow

@gauravgopi123 please merge my PR. thanks.

@vrozov I can't get a hold of gaurav. Vlad, please merge my PR when you get a chance. Thanks.


azcopy :Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.

Getting build failure issues in one of the pipeline, which is related to azcopy.
azcopy task is throwing an exception as seen below.
403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
Repo: https://msdyneng.visualstudio.com/FinOps/_git/Application?path=%2F&version=GBmain&_a=contents
Pipeline: - https://msdyneng.visualstudio.com/FinOps/_build/results?buildId=974223&view=logs&j=ce039bdd-38e8-5807-41bf-d2c8f82d8dba&t=8edbf0ca-7ec3-5b5b-4460-9cb8b08a821b
Exception:
2021-07-30T10:13:38.9568002Z    User-Agent: [AzCopy/10.11.0 Azure-Storage/0.13 (go1.15; Windows_NT)]
2021-07-30T10:13:38.9568557Z    X-Ms-Client-Request-Id: [3f32ca2b-5a7c-44ab-424c-7846309cff48]
2021-07-30T10:13:38.9568981Z    X-Ms-Version: [2019-12-12]
2021-07-30T10:13:38.9569510Z    --------------------------------------------------------------------------------
2021-07-30T10:13:38.9570321Z    RESPONSE Status: 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
2021-07-30T10:13:38.9570926Z    Content-Length: [447]
2021-07-30T10:13:38.9571296Z    Content-Type: [application/xml]
2021-07-30T10:13:38.9571781Z    Date: [Fri, 30 Jul 2021 10:13:36 GMT]
2021-07-30T10:13:38.9572284Z    Server: [Microsoft-HTTPAPI/2.0]
2021-07-30T10:13:38.9572669Z    X-Ms-Error-Code: [AuthenticationFailed]
2021-07-30T10:13:38.9573162Z    X-Ms-Request-Id: [3185de5e-501e-013a-072b-855037000000]
2021-07-30T10:13:38.9574470Z Response Details: AuthenticationFailedServer failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. The specified signed resource is not allowed for the this resource level
2021-07-30T10:13:38.9575307Z
2021-07-30T10:13:38.9575806Z 2021/07/30 10:13:36 ==> REQUEST/RESPONSE (Try=1/52.0024ms, OpTime=52.0024ms) -- RESPONSE STATUS CODE ERROR
2021-07-30T10:13:38.9576562Z    PUT https://usweintdpl1catalog.blob.core.windows.net/product-ax?restype=container&se=2021-07-31t10%3A12%3A52z&sig=-REDACTED-&sp=rw&sr=b&sv=2015-12-11&timeout=180
2021-07-30T10:13:38.9577375Z    User-Agent: [AzCopy/10.11.0 Azure-Storage/0.13 (go1.15; Windows_NT)]
2021-07-30T10:13:38.9577940Z    X-Ms-Client-Request-Id: [abd3ca44-f05a-4d49-4df3-380db01696d3]
2021-07-30T10:13:38.9578459Z    X-Ms-Version: [2019-12-12]
2021-07-30T10:13:38.9578891Z    --------------------------------------------------------------------------------
2021-07-30T10:13:38.9579579Z    RESPONSE Status: 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
2021-07-30T10:13:38.9580328Z    Content-Length: [447]
2021-07-30T10:13:38.9580694Z    Content-Type: [application/xml]
2021-07-30T10:13:38.9581205Z    Date: [Fri, 30 Jul 2021 10:13:36 GMT]
2021-07-30T10:13:38.9581682Z    Server: [Microsoft-HTTPAPI/2.0]
2021-07-30T10:13:38.9582063Z    X-Ms-Error-Code: [AuthenticationFailed]
2021-07-30T10:13:38.9582592Z    X-Ms-Request-Id: [3185de68-501e-013a-102b-855037000000]
2021-07-30T10:13:38.9583750Z Response Details: AuthenticationFailedServer failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. The specified signed resource is not allowed for the this resource level

Hi ,
Is any update on this issue?


Flavor validation on package create

Is your feature request related to a problem? Please describe.
If you attempt to create a package with a flavor that doesn't exist, as long as there is a component you don't get an error or warning about passing in a bad flavor.
Example:  (I know this one is silly) but if you do zarf package create <dir> --flavor patchVersion9 it doesn't let you know you specified a bad flavor since you still have the podinfo component, but neither of the podinfo-flavor components get added to the created package.
zarf.yaml
kind: ZarfPackageConfig
metadata:
  name: podinfo
  version: 0.0.1

components:
  - name: podinfo
    required: true
    charts:
      - name: podinfo
        version: 6.4.0
        namespace: podinfo
        url: https://github.com/stefanprodan/podinfo.git
        gitPath: charts/podinfo
    images:
      - ghcr.io/stefanprodan/podinfo:6.4.0
    actions:
      onDeploy:
        after:
          - wait:
              cluster:
                kind: deployment
                name: podinfo
                namespace: podinfo
                condition: available

  - name: podinfo-flavor
    required: true
    only:
      flavor: patchVersion2
    charts:
      - name: podinfo
        version: 6.6.2
        namespace: podinfo-flavor
        url: https://github.com/stefanprodan/podinfo.git
        gitPath: charts/podinfo
    images:
      - ghcr.io/stefanprodan/podinfo:6.6.2
    actions:
      onDeploy:
        after:
          - wait:
              cluster:
                kind: deployment
                name: podinfo
                namespace: podinfo-flavor
                condition: available

  - name: podinfo-flavor
    required: true
    only:
      flavor: patchVersion3
    charts:
      - name: podinfo
        version: 6.6.3
        namespace: podinfo-flavor
        url: https://github.com/stefanprodan/podinfo.git
        gitPath: charts/podinfo
    images:
      - ghcr.io/stefanprodan/podinfo:6.6.3
    actions:
      onDeploy:
        after:
          - wait:
              cluster:
                kind: deployment
                name: podinfo
                namespace: podinfo-flavor
                condition: available

Describe the solution you'd like

Given a zarf.yaml with components using flavors
When running zarf package create  <dir> --flavor with an invalid flavor
Then I get a messaging saying something like "Learn how to spell buddy!" or "invalid flavor specified"

Describe alternatives you've considered
(optional) A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.

Great issue, We are adding on create validation in #2585 and will make sure this gets in as a validation error.


(GH-2403) Fix apply block on Windows

On Windows, bolt plans fail to run apply blocks, because of a bug that makes bolt_catalog start its outputs with unwanted warnings about encoding issues.
This PR fixes https://github.com/puppetlabs/bolt/issues/2403

I don't think we want to ignore non-JSON output on stdout since it generally means something has gone wrong (in this case it seems like an error was raised). Closing.


Revert "[CI] work with hpp-fcl v1, which is not yet in the binary repository"

This reverts commit eb73311af732c6efbbbeec4eade59f37cdc03ad9, because hpp-fcl v1 is in the binary respository now.

https://gepgitlab.laas.fr/gsaurel/pinocchio/pipelines/3690/builds :)


Set Swift version to 4.2 and update examples

CC @pusher/mobile

Codecov Report

Merging #110 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #110   +/-   ##
=======================================
  Coverage   78.73%   78.73%
=======================================
  Files          38       38
  Lines        1486     1486
=======================================
  Hits         1170     1170
  Misses        316      316

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7b2a116...130a184. Read the comment docs.


Suppress parking landuse labels until zoom 17

Previously in #126 we suppress parking POIs until zoom 17. Turns out some of these are landuse labels (from relation AOIs) and we need to do this for the other data type, too.

The current landuse filter is completely based on area, and not the "types" of landuse features. The shortest step to take here now is to have these features be filtered out at the transform layer based on zoom, but longer term it would be better to transition the landuse filter function to be based on zoom level too, like the one for pois. In other words, instead of returning a boolean that represents whether to include the feature or not, the function can return back an integer that corresponds with the zoom level to start including it in.

Follow up question, do we want to just filter it out from the labels layer and leave it in the landuse one? I'm thinking that it would be filtered out of both to keep them consistent.

The layer the feature is represented in (POIs or Landuse Labels) doesn't matter and shouldn't be changed at this time. But the zoom clamp we applied to parking areas in POIs also needs to be applied to Landuse Labels (so the isolated P parking icons we see on the map now at zoomed out city zooms are also suppressed to zoom 17).
Yeah, long term we want to support the kind (type) based default min_zoom ranges in addition to the current area test in the Landuse layers.
We should leave the features in the Landuse layer (only the area test), adding the kind test only for the Landuse-Labels layer.

Yea, I wasn't suggesting moving the layer for certain features. I was just trying to point out that for the pois layer we have the notion of different features appearing at different zoom levels. For landuse we don't at the moment. Sorry for any confusion!
If we want to keep those features in the landuse layer and only trim it from the landuse_labels layer, I think it's better to leave things the way they are for now, and just update the query used for landuse_labels to also filter out parking, ie amenity != 'parking' for z < 17.

Agree, your change works.


[WIP] Update puma to latest.  Be less conservative.

Puma does a pretty good job about following semantic versioning so it should be safe to update minor versions and will probably help us tackle the problems sooner rather than later.
https://github.com/puma/puma/blob/v4.2.1/History.md

[ ] Depends on some UI team members 👍.  It works for me but I didn't test puma:

[ ] with a debugger
[ ] consoles with websockets
[ ]  action cable with notifications

Just to clarify, consoles are NOT using action cable :wink: only notifications. However, consoles are also running through websockets.

@miq-bot add_reviewer @skateman

Do we have any sort of performance comparisons we can do? Just curious, I generally favor updating to the latest, especially since it seems like there have been a lot of fixes and updates since 3.7.

with a debugger

No issues with pry byebug, not noticing any issues with recompilation, ... LGTM IMO :)
(I didn't test actioncable or consoles.)

Both the notifications (AC) and the consoles work well.

@Fryguy this ready to go... thanks to @himdel and @skateman for testing the debugger, consoles, and notifications


Proteins in different samples types with the exact intensity values.

Processing data from three co-IP different sample types, using MetaMorpheus v0.0.320, and searched against the Mouse proteome database (Mouse_SP_Jan2022), we noticed that some of the identified proteins (in the proteinGroups output) had the exact normalized value. Most of those proteins were identified and quantified by at least 4 unique peptides. Each peptide had different intensity value in the different samples, but performing the calculation manually (by using the peptides output, and normalizing to the TIC), I found that the values are similar but not identical as in the proteinGroups output.
It looks very strange to me as it is hard to believe that in about 10 cases (from ~360 protein identifications), the proteins had the exact same value between the different sample types.
Please check the attached proteins and peptides file, and as an example, take the protein with the accession ID of Q60597. This protein had 6 unique peptides
Peptides-report.xlsx
Proteins-Report.xlsx

thanks for the comment. we have seen this. I was thinking that it was fixed. but perhaps there haven't been some updates passed through to this release of metamorpheus

I think this bug is fixed here: https://github.com/smith-chem-wisc/mzLib/pull/624
that is in a nuget package that metamorpheus uses. So, i think we need to update the nuget in metamorpheus and make a new release. That is challenging at the moment because of issues out of our control. There is currently no support for our installer wix in vs2022 and no support for .net 6 in vs2019. we are in never never land. I don't want to update metamorpheus and cause problems for all of our users. I may be able to get you a pre-release to try. please watch this space over the next couple days.

fixed in the next relase


[FEATURE REQUEST] Change printout of '/lp listgroups'.

Moving from PEX to this plugin, everything in LuckPerms is far more in-depth and superior...until I used /lp listgroups.
Sure it's printout in-game does what it is intended to do (list the groups), but could defiantly have room for improvement. PEX has a printout that save's a lot of time when trouble shooting, or to quickly check a parent, group weight, ladder etc. I believe this would really benefit LuckPerms.
This would also be handy/apply to /lp listtracks.
See: http://i.imgur.com/qqbZbhE.png
Cheers for reading, and hopefully considering this change.

So, what information should it show?
Group name - Weight - ????

Cheers for the quick reply :)
In order of the lowest group (default) to the highest:
Group Name - Group Weight - Track Order/Name - Parent
Example:
Default - #10 - 1 | Ranks - []
Guest - #20 - 2 | Ranks - [Default]
Member - #30 - 3 | Ranks - [Guest]
Pro - #40 - 4 | Ranks - [Member]
Staff - #50 - 1 | Staff - [Pro]
I know it can look messy in text, but once you have it applied to an actual permissions system - it looks a lot cleaner and a great tool for a quick go-to to get basic information on your servers groups.


refactor: Import the Tiles & Tile components from the starter

What's the purpose of this pull request?
Extract the Tiles and Tile component from the starters and move it to the @faststore/ui package.
How to test it?
Run yarn storybook locally and check the Tiles component page.
Its Storybook page:

The same component being used on the starter:

Starters Deploy Preview

https://github.com/vtex-sites/nextjs.store/pull/185 and the deploy preview

@eduardoformiga, merging this, but still feel free to review it (when you're back) and I'll get back to you later!

@eduardoformiga, merging this, but still feel free to review it (when you're back) and I'll get back to you later!

Looks Good To Me! 👏👏


TEP-0111 - Propagating workspaces in pipelinerun

The verbosity of writing specifications in Tekton Pipelines is a common pain point that creates difficulties in getting-started scenarios. In addition, the verbosity leads to long specifications that are error-prone, harder to maintain, and reach the etcd size limits for CRDs. Automatically propagating Workspaces will lead to better user experience. This addresses features proposed in TEP-0111. This is a follow up PR that addresses propagating workspaces in PipelineRuns. Propagating workspaces in Taskruns is being addressed by https://github.com/tektoncd/pipeline/pull/5081 and should be merged first.

Changes

Submitter Checklist
As the author of this PR, please check off the items in this checklist:

[x] Has Docs included if any changes are user facing
[x] Has Tests included if any functionality added or changed
[x] Follows the commit message standard
[x] Meets the Tekton contributor standards (including
functionality, content, code)
[x] Has a kind label. You can add one by adding a comment on this PR that contains /kind <type>. Valid types are bug, cleanup, design, documentation, feature, flake, misc, question, tep
[x] Release notes block below has been updated with any user facing changes (API changes, bug fixes, changes requiring upgrade notices or deprecation warnings)
[ ] Release notes contains the string "action required" if the change requires additional action from users switching to the new release

Release Notes
Workspaces are propagated in embedded specifications of pipelinerun without mutations.

/hold until https://github.com/tektoncd/pipeline/pull/5081 is merged.

/hold cancel

/check-pr-has-kind-label

/kind feature

/check-pr-has-kind-label

/retest


Add UP5K support

Thanks! Nice to have UP5K support, and also thanks for the couple code fixes.
Merged, and I added the fomu_rgb example to the example menu and fixed a couple indentations.
Cool that there's still interest in ice40_viewer, it was a fun little project to do.
- Kristian.

Thanks for doing all the hard work! They viewer has already proven to be very useful.
One thing we didn't know what (if anything) to do about was:
https://github.com/knielsen/ice40_viewer/blob/f7857940addf6817ffd3a4b32026ede1ccdd967c/ice40.js#L643
@jamesdiacono also considered changing the way databases are loaded, basing it on the .asc file loaded, per your code comment. However that was a much larger change. Perhaps he will create an issue to track that as possible future work.

Ah, this active_value is used to detect if a block ram tile is in use in a design and mark it in a different colour if so.
The bit to mark a ram tile active is opposite polarity for 1k and 8k devices. It appears from your 5k example that the bit for 5k is the same as 8k, I updated the code appropriately.
Yes, feel free to create an issue to track the change for database loading.
- Kristian.


Consider using async fs calls

Currently the code is using sync fs calls to do all filesystem activity.  This works, but isn't optimal, since it blocks the main thread, and means that the program is going to be unresponsive when processing large amounts of data.
I would suggest you consider using the async versions of fs calls, and potentially the Promise based versions, so you can keep your style of synchronous coding, but benefit from node's ability to schedule async tasks.

@humphd, I created a pull request for this issue, would you mind having a look at it? I'd love to learn how it should be done.


TBI - Enable telemetry when running in SAP Business Application Studio

Description
Currently telemetry data to improve extension quality is not enabled when running in SAP Business Application Studio (BAS). To have consistent data, we should also include telemetry data when running in BAS.
Value
Improve product quality through measurements.
Architecture Elaboration
Central telemetry setting in BAS is switched of and can't be enabled. Therefore, we need an own telemetry setting fro Guided Answers extension.

@Klaus-Keller  Can this be closed?


CNDB-9698: fix LCS partialCompactionsAcceptable when removing L0 sstable from L0/L1 compaction task for limited space

issue: https://github.com/riptano/cndb/issues/9698

@jasonstack could you share where and why do we use LCS and not UCS?

@jtgrabowski that's system.paxos table

Should we use UCS for it ?

Should we use UCS for it ?

we can test it after CC migration..


Please add link to the extension

Hi, thanks for maintaining a fork. I think it would be great to have a direct link to the extension https://addons.mozilla.org/en-US/firefox/addon/reimagined-twitter/

Oh. Thank you very much. I recently redid the whole repo, and I must have forgotten to re-add it.

This is fixed in 2777f57e9675873e9640958c052444ede2c6ad2b


Require console for UI_Console

Fixes scenarios where session isn't started immediately.
require 'debug/session'
DEBUGGER__.console # raises NameError

Thanks!


Add a "ZFS Intro for Ansible-NAS users" page

Ansible-NAS currently assumes that the user will know how to setup and configure ZFS, including snapshot management. It would be nice to at least point out that a program such as Sanoid (https://github.com/jimsalterjrs/sanoid/) can be used to automate the snapshot part of the NAS.
(Possibly this could be included in a "default ZFS scenario" for people not so familiar with these things as a reference where to start: Install root on "normal" ext4 file system, create pool assuming two mirrored hard drives, set up datasets for TV, Samba shares, movies, etc, set up snapshots of said datasets.)

Never heard of it but this looks awesome, will check it out!
On Fri, 29 Mar 2019, 23:46 Scot W. Stevenson<EMAIL_ADDRESS>wrote:

Ansible-NAS currently assumes that the user will know how to setup and
configure ZFS, including snapshot management. It would be nice to at least
point out that a program such as Sanoid (
https://github.com/jimsalterjrs/sanoid/) can be used to automate the
snapshot part of the NAS.
(Possibly this could be included in a "default ZFS scenario" for people
not so familiar with these things as a reference where to start: Install
root on "normal" ext4 file system, create pool assuming two mirrored hard
drives, set up datasets for TV, Samba shares, movies, etc, set up snapshots
of said datasets.)
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/davestephens/ansible-nas/issues/73, or mute the
thread
https://github.com/notifications/unsubscribe-auth/AAozMP1CKe-QwdIWoYllLsEqFbDDD-gBks5vbqXZgaJpZM4cTaO7
.

As in discussion for #69 , include the Sanoid stuff in an intro page documenting the basics of ZFS, including suggestions on what parameters to use. Might be worth waiting for ZoL 0.8 to be released for this because it includes cool stuff like TRIM support and native at-rest encryption. Should include example for snapshots etc. for most common use case.

Unless somebody has a better idea, I would suggest splitting this into two documents: One a general introduction to ZFS as related to the use in NASes, including the ability to move pools from FreeNAS to Ansible-NAS (and, of course, the reverse way); the other specific configuration suggestions and examples for the apps included.

I think we need to consider the target audience too. A lot of people likely
won't care about most of the ZFS options, and just want to know that their
data is mirrored on a pair of disks and will be recoverable. I for one fall
into that camp.
Me aside, there are always power users that want to go "full enterprise" on
their home stack, and that's cool - but I've always been pretty open about
Ansible-NAS not managing disks/partitions because a- it's hideously
complicated and b- it's near-on impossible to automate for every use case.
I also don't believe there should be differing ZFS partition configs for
different applications - for the same reasons above, and also that not
everyone uses ZFS.
Any ZFS stuff should be set system-wide and used by everything - let's not
forget this is a home NAS, not an enterprise grade storage system where
disk latency of utmost importance :-)
Guiding new users through creating a sensible ZFS setup, and giving power
users ideas of options to tweak is absolutely the right thing to do, but I
think the documentation needs to call those differing use cases out
clearly.
Tl;Dr; I think we're agreeing, I just want to be clear that none of this
stuff is ever going to be automated, and I don't want to be recommending
loads of manual tweaking every time a new container is spun up.
On Wed, 3 Apr 2019, 11:22 Scot W. Stevenson<EMAIL_ADDRESS>wrote:

Unless somebody has a better idea, I would suggest splitting this into two
documents: One a general introduction to ZFS as related to the use in
NASes, including the ability to move pools from FreeNAS to Ansible-NAS
(and, of course, the reverse way); the other specific configuration
suggestions and examples for the apps included.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/davestephens/ansible-nas/issues/73#issuecomment-479430381,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAozMN0KvLosqNBqexP_zeKwj0CQER7Kks5vdIDfgaJpZM4cTaO7
.

Right, I see it exactly the same way - automating ZFS is asking for trouble, and a complete guide would be in book length and require constant updating. My idea would be:
A "ZFS overview" for total newbies: Move the stuff about ZFS out of the 'overview.md' and stick it in this document; explain advantages of ZFS (self-healing, don't care about ransomware with snapshots, can move pools to and from FreeNAS, Linux, FreeBSD at will, etc) and disadvantages, references to further help. -- Write this once and never touch it again until some change to the core ZFS structure (in other words, probably never).
A "ZFS specific suggestions" text with example command lines. My current take would be: We don't do your ZFS setup for you, but we'll give you some pointers here because we're nice that way. Specifically: Root on ZFS is still too complicated with Ubuntu, use ext4 (mirrored with mdadm if you want to); this means Docker stuff stays there as well. Example of two hard drives in mirrored configuration for ZFS in one single pool for all data; set scrub cron job for this pool. Set up datasets for movie and TV files with larger bock sizes; set up datasets for shares with "normal" block sizes; example snapshot configuration for these shares. Never ever use deduplication, and you'll probably not benefit at all from L2ARC or SLOG in our case, so just don't. -- We walk the user through a simple default sample configuration, and update this for relevant major new ZFS features (at rest encryption, easy root on ZFS in Ubuntu setup menu), but that's it.
I mean, if this is too long, I won't get to watch the stuff I'm trying to store in the first place 😄.