Adding support for Python 2-3

This PR adds Python 2 and 3 support for OptiType using the six and future packages.
Also, Travis CI is configured to build and test for v3.*. The change starts out with the futurize results and then hand-fixing the remaining problems.

Could you please rebase to master again to get the most recent changes as of today.

I rebased this as asked for. The failure comes through some Docker-related addition independent after my changes.

ping :bell:

You can rebase again. It should work now.


Add FacebookStrategy

https://developers.facebook.com/docs/facebook-login/manually-build-a-login-flow/

Hi @sergiodxa, I would like to try and tackle this issue.

@chadelofson I recommend you to check how the GoogleStrategy works, it will probably be similar, just change the URLs and how to get the user profile

@sergiodxa ok, I have been looking at both Google and GitHub strategies.
For the google strategy, how did you determine what to use for GoogleStrategyOptions and GoogleExtraParams?

@chadelofson the StrategyOptions is anything you may need from the developer or the developer may be able to set, example, when using Google you need a client id and secret and you need a callback URL (usually called redirect_uri in the docs of the services), but the developer can also set more things like the scope, access type, etc., those are all used in the authorizationParams to pass them in the URL.
For the ExtraParams, this is whatever the provider returned after getting the token, if you see here https://github.com/sergiodxa/remix-auth/blob/main/src/strategies/oauth2.ts#L245-L256, that getAccessToken method is called from fetchAccessToken, by default (GoogleStrategy uses the default, GitHubStrategy overwrote it) it parses the response of fetchAccessToken as JSON, get the access_token and refresh_token and anything else returned in that response is called ExtraParam, so you need to see what the provider return here and add it to the interface.

I'm closing this since strategies are not going to be part of the Remix Auth package and they should be published independently.

ok, sounds good.   So what should the package names be called?

I recommend remix-auth-facebook

OK, thank you.  Same folder structure as what you had?  Do you have other strategies moved to their own project?

@chadelofson you can use https://github.com/sergiodxa/remix-auth-strategy-template as a base.

@pbteja1998 thanks!


Recurrent layer implementation

Thanks for this repository !
Have you looked at implementing it for recurrent layers as well ?

Hi Thibault,
Currently it doesn't support recurrent layers yet, but there is no reason why this functionality could not be added. I can't make any promises whether I will find time to do it soon though..


Implement GitHub PR Issue Checker Workflow

Closes #108
Usage:

When a pull request is created or edited, the workflow automatically triggers.

The workflow checks for:

A non-empty PR description.
A valid issue reference format (Fixes , Close, Closes, Closed , Fix , Fixed , Resolve , Resolves #Issue_number).

If either check fails, the PR will be marked as failing, and the author will receive clear error messages in the PR comments.
If all checks pass, a confirmation message will indicate that the PR is valid.

Note: Kindly add all the labels added to the issue, to this PR!
@ombhojane

do you really need 're' package, remove if unnecessary
actions/setup-python@v2, it supposed to be v3 right?

does'nt matter! It works! @ombhojane

do you really need 're' package, remove if unnecessary
actions/setup-python@v2, it supposed to be v3 right?

There are both v3 and v2!!


Accessibility

http://jsfiddle.net/1x8w9c3n/31/

Had issues with rebase. Just redone and created a new pull request.

Hi, Did I miss something?

why is that commented out

Since ActivateADA and initADA methods takes care of adding required tabindex properties, the commented code is redundant.

Please let me know if there is any update I need to make.

http://jsfiddle.net/equinoxs/1x8w9c3n/39/  created a pull request

Looks like the PR to review/follow is #1467.


Eslint

Killed jshint.
Applied eslint. Config was taken from Airbnb popular repo. Modified for stampit needs.
Tests were converted to ES6. (That was hard!)

Pushed one more commit for code coverage.
As I expected it's 100% :)
Usage:
$ npm tun cov
and maybe
$ open coverage/index.html

:+1:


[Key Vault] Support 7.5 API version in Administration

Description
Regenerates on https://github.com/Azure/azure-rest-api-specs/commit/7452e1cc7db72fbc6cd9539b390d8b8e5c2a1864 and adds support for service API version 7.5.
This PR uses DPG to generate for the first time, which changes the structure of generated code a decent bit. One generated model -- RoleDefinition -- now has some properties nested under a .properties attribute, but there are otherwise no significant changes.
KV's custom challenge authentication policy did get a slight re-work here, per the issue described in https://github.com/Azure/azure-sdk-for-python/issues/33751. Using DPG means that we're using new HTTP request types, and our old method of request manipulation no longer works. Johan pointed out that we shouldn't be modifying requests anyway; creating a bodiless request copy to prompt a challenge makes more sense, so this PR makes that change.
All SDK Contribution checklist:

[x] The pull request does not introduce [breaking changes]
[x] CHANGELOG is updated for new features, bug fixes or other significant changes.
[x] I have read the contribution guidelines.

General Guidelines and Best Practices

[x] Title of the pull request is clear and informative.
[x] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

[x] Pull request includes test coverage for the included changes.

API change check
APIView has identified API level changes in this PR and created following API reviews.
azure-keyvault-administration


update semantic release and other dev deps

Updated dev-deps
changed release workflow to node 16

:tada: This PR is included in version 3.12.1 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Why the results of concordance_index are different ？

We used the lifelines.utils.concordance_index and R package survcomp method concordance.index to calculate the concordance_index .
But, we found the results of concordance_index  is different .
Why  different ?
reference: https://www.rdocumentation.org/packages/survcomp/versions/1.22.0/topics/concordance.index

Hi @WenxiongLiao, I'm not sure. Do you have an example dataset and code that demonstrates the difference?

Hi @CamDavidsonPilon , There is an example
#for the lifelines.utils.concordance_index
import numpy as np
from lifelines.utils import concordance_index
risk = np.array([1,4,4,1,4,1,2,1,3,2,1,4,4,2,1,1,1,1,1,4,1,4,1,2,1,1,2,4,1,1,1,3,4,2,3,1,4,1,4,3])
time = np.array([80.73333,83.36667,82.80000,80.03333,20.03333,82.83333,82.16667,78.60000,79.70000,83.30000,70.63333,74.20000,71.10000,68.03333,71.30000,68.93333,71.53333,67.36667,71.90000,55.50000,71.66667,66.66667,71.23333,67.00000,42.06667,73.06667,72.33333,69.80000,67.06667,68.96667,48.80000,71.26667,55.13333,71.86667,71.00000,71.46667,71.63333,68.70000,68.43333,71.00000])
event = np.array([0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0])
concordance_index(event_times = time, predicted_scores = -risk, event_observed = event) # 0.5592105263157895

#for the R package survcomp method concordance.index
risk <- c(1,4,4,1,4,1,2,1,3,2,1,4,4,2,1,1,1,1,1,4,1,4,1,2,1,1,2,4,1,1,1,3,4,2,3,1,4,1,4,3)
time <- c(80.73333,83.36667,82.80000,80.03333,20.03333,82.83333,82.16667,78.60000,79.70000,83.30000,70.63333,74.20000,71.10000,68.03333,71.30000,68.93333,71.53333,67.36667,71.90000,55.50000,71.66667,66.66667,71.23333,67.00000,42.06667,73.06667,72.33333,69.80000,67.06667,68.96667,48.80000,71.26667,55.13333,71.86667,71.00000,71.46667,71.63333,68.70000,68.43333,71.00000)
event <- c(0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0)
cindex <- concordance.index(x=risk,surv.time = time,  surv.event = event,method = "noether")
print(cindex$c.index) #0.592
The c-index of  lifelines.utils.concordance_index is  0.5592, but the result of concordance.index  is 0.592
So, What caused the result to be different ?

Try concordance.index(..., outx=FALSE)


Generate missing exercise READMEs

The following exercises were missing READMEs:

collatz-conjecture
crypto-square
flatten-array
transpose

In preparation for v2 we're updating Configlet—our Exercism track linter—to require that we generate the exercise README. This is because in v2 we will not be generating READMEs on the fly, but deliver the README as defined in the directory for each implementation.
I'm going to go ahead and merge this as soon as the build passes, as this is janitorial work rather than a language-specific change.
See https://github.com/exercism/configlet/issues/116 for more details.

We have a version conflict on Travis—the phptest.phar that is getting downloaded is too new for the version of PHP.

I'm actually going to merge this even though the build is passing, since the build is a separate problem. I'm submitting a PR with a suggested fix for the build.


[Bug] Report 7390 "Whse. Calculate Inventory" - Items Not on Inventory not working from 18.1

Hello,
currently we are upgrading our product to 18.2. Now we found a bug in Report 7390 "Whse. Calculate Inventory".
Luckily we had enough Testcodeunits and we noticed there was something changed in the default behaviour.
On Report 7390 "Whse. Calculate Inventory", there is a option to include items without inventory in the journal.

But this boolean isn't working anymore due to the following update. The remove of a ";" makes a big difference and will break the normal behaviour.

Maybe this is a fix.

Kind regards,
Gunter Peeters

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. Please do not reply to this, as we do not monitor closed issues. If you have follow-up questions or requests, please create a new issue where you reference this one.


Refactor Pipeline

Relates to https://github.com/caos/zitadel/pull/30

[ ] Create debug-image
[ ] Abstract Short-SHA
[ ] Test NPM Build
[ ] Configure SEMREL to upload release artifacts
[ ] Maybe Generate Pre-Release Uploads from feature branches

@fforootd is it still open? IMO the new dockerfile does most of the bulletpoints

thats true
we can close this for the moment


Support a mempoolfullrbf=1 configuration

Can't.

I suspected that, can it be implemented?

Nope, AFAIK.

@rapidlab309 is correct.

With the last releases of bitcoin core there is support for full rbf so why it can't be implemented?

It's a fair point. It may be worth adding a configuration variable for this, to match the one in Bitcoin Core (default disabled).

Looks good for me, issue reopened?

In c57a4450 I've added config variable mempoolFullRbf to Sparrow's config file. It is set to false by default, but if set to true will enable RBF functionality on mempool transactions without checking the sequence flags on the transaction's inputs.


[SPARK-32659][SQL][FOLLOWUP] Broadcast Array instead of Set in InSubqueryExec

What changes were proposed in this pull request?

This is a followup of https://github.com/apache/spark/pull/29475.
This PR updates the code to broadcast the Array instead of Set, which was the behavior before #29475
Why are the changes needed?

The size of Set can be much bigger than Array. It's safer to keep the behavior the same as before and build the set at the executor side.
Does this PR introduce any user-facing change?

No
How was this patch tested?

existing tests

cc @wangyum

Test build #128978 has started for PR 29838 at commit 4e12c42.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33600/
Test PASSed.

Merged to master.
@cloud-fan . Could you make a backporting PR to branch-3.0? There was a conflict on branch-3.0.

Test build #128978 has finished for PR 29838 at commit 4e12c42.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128978/
Test PASSed.


Repo sync for protected CLA branch

The pull request is created from main638049096012058738sync_temp to main to fix git push error for protected CLA branch

Learn Build status updates of commit 8ebfce7:
:white_check_mark: Validation status: passed
For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the learn.microsoft.com contributor guidesPost your question in the Learn support channel


FEATURE: adds support for unreliable network

Allows to resend staged messages
Displays an icon when network is unreliable
Doesn't make the sed button unusable


Analytics > "Messages by Story" issue - Story rule

I have a chatbot dedicated to 2 targets : clients and advisors. Most of the knowledge are similar but for some of them, different answers have been set up.
To do so, here we are what we have done :
1/ Create 2 different connectors

2/ Create 2 different intents to get 2 stories. However, all sentence are qualified only on 1 of the 2 intents:
parler_humain : story for clients
parler_humain_cons : story for advisors

parler_humain : intent on which all sentences are qualified

3/ Story rule has been set up in order to redirect all "parler_humain" intents to the "parler_humain_cons" story for the advisor connector

Everything is working fine on the front interface : clients get their client answer and advisors get their advisor answer
However, in the statistics, this is not what we have as "parler_humain_cons" stories are found in clients connector which is not possible :

I don't know where the error can come from.
Could you please help me ?

Hi @francoisno ,
Yes, the graph shows only results filtered on client connector.
Hope so to :)


How to input the arguments of ConsoleConvertToIFCJson?

https://github.com/jmirtsch/GeometryGymIFCExamples/blob/master/ConsoleConvertToIFCJson/Program.cs#L14
just like this?but it doesn't work.
I try to input "myfile.ifc myfile.json",but nothing happened.
So,Can you tell me how to input ifc and output json?
thanks in advance.

Did you include the path to the file or set the working directory?
This particular example uses pipes, so make sure you also have the < and  > directions.

Try changing the code to:
DatabaseIfc db = new DatabaseIfc(args[0]);
db.WriteFile(args[1]);
It should work with the .json extension (else try .ifcjson).


⚠ Jenkins Plugins Artifactory has degraded performance

In 1ee3ebf, Jenkins Plugins Artifactory (https://get.jenkins.io/plugins/artifactory/) experienced degraded performance:

HTTP code: 200
Response time: 2369 ms

Resolved: Jenkins Plugins Artifactory performance has improved in 13577d5 after 49 minutes.


Add mmlu

Added MMLU task to catwalk

All 57 tasks are added but have to be specified individually. Ideally we would have a Catwalk "GroupedTask" such that by only specifying --task mmlu, all the 57 tasks could be run. This would also allow us to aggregate the statistics across the subtasks. This seems beyond my ability but let me know if there is a simple fix.
I am using the top-k dev examples as few-shot. The easiest way was to define a special EleutherMMLUTask for this
I am using the acc_raw metric but I am not sure if that is the ideal one for MMLU.

This will presumably not keep the initial instruction in the usual MMLU prompt (including the updated Eleuther one: https://github.com/EleutherAI/lm-evaluation-harness/blob/master/lm_eval/tasks/hendrycks_test.py#L134). There's a few way to incorporate that, each being a bit clunky (e.g., add prefix option to instance_as_rank_classification which is then filled by some appropriate method in the task). There's also a question whether that instruction should be there in the 0-shot case (the Eleuther code doesn't include it I think, which feels like a bug).
Doing the aggregation over subjects within catwalk would be nice, will require some thought as to how to do this (and how to format the output, as we'd want to also keep the per-subject and per-instance data).
One minor thing I suggest about the prompt, changing it slightly from the Eleuther one, is to add a space before each answer choice:
            choices = "".join(
                [f" {key}. {choice}\n" for key, choice in zip(keys, doc["choices"])]
            )

for consistent tokenization in the prompt and in the answer (e.g., " A" both places, vs "A" and " A")

I was thinking whether we should make that change (to add spacing before choices) in https://github.com/allenai/catwalk/blob/olmo-eval/catwalk/tasks/eleuther.py#L159 for all tasks?

At first I thought that we should make that change for all ranked classification tasks by overwriting the _process_doc function?

Struggling to find where the context creation is happening for RankedClassification tasks. Any pointers?

The call to instance_as_rank_classification happens here (through slight indirection) for the models we care about: https://github.com/allenai/catwalk/blob/olmo-eval/catwalk/models/language_model.py#L126.
Rather than messing with that, possibly the instance_as_rank_classification method can check if the inner task has some sort of "description" set and initialize prefix with it (might need to be added as a separate field to the task class, as pulling it out from, say, https://github.com/EleutherAI/lm-evaluation-harness/blob/master/lm_eval/tasks/hendrycks_test.py#L134 isn't very easy)
Btw, the space I'm talking about is in the middle of the instance prompt, so no central way to change that.

I did discover that call but I dont know how the context is being created for each instance. I dug a bit more and here is my current understanding.

The context is created in run_loglikeihood using the first entry of the tuple constructed here:
https://github.com/allenai/catwalk/blob/olmo-eval/catwalk/models/language_model.py#L140C5-L140C5
These instance_requests are being created inside the choices field of RankedClassificationInstance here (where the prefix field is used)
https://github.com/allenai/catwalk/blob/olmo-eval/catwalk/tasks/eleuther.py#L161-L165
The requests are being constructed here using the inner task function
https://github.com/allenai/catwalk/blob/olmo-eval/catwalk/tasks/eleuther.py#L111-L113

So it should include the initial instruction in the MMLU prompt. And I think it does. This is the first field in model_input tuple:
The following are multiple choice questions (with answers) about abstract algebra.

The cyclic subgroup of Z_24 generated by 18 has order
A. 4
B. 8
C. 12
D. 6
Answer: A

The following are multiple choice questions (with answers) about abstract algebra.

Find the order of the factor group Z_6/<3>.
A. 2
B. 3
C. 6
D. 12
Answer: B

The following are multiple choice questions (with answers) about abstract algebra.

Statement 1 | A permutation that is a product of m even permutations and n odd permutations is an even permutation if and only if n is even. Statement 2 | Every group is isomorphic to a group of permutations.
A. True, True
B. False, False
C. True, False
D. False, True
Answer: A

The following are multiple choice questions (with answers) about abstract algebra.

The cyclic subgroup of Z_24 generated by 18 has order
A. 4
B. 8
C. 12
D. 6
Answer:  A

Maybe it should not repeat for every instance?

Right, we don't want that instruction repeated, so might need to do some surgery on the inner task to avoid this which is a bit annoying.

As per our discussion, Changed the top-level function for generating ranked classification instances. Now we get correctly formatted instances such as:
The following are multiple choice questions (with answers) about abstract algebra.

Find all c in Z_3 such that Z_3[x]/(x^2 + c) is a field.
 A. 0
 B. 1
 C. 2
 D. 3
Answer: B

Statement 1 | If aH is an element of a factor group, then |aH| divides |a|. Statement 2 | If H and K are subgroups of G then HK is a subgroup of G.
 A. True, True
 B. False, False
 C. True, False
 D. False, True
Answer: B

Statement 1 | Every element of a group generates a cyclic subgroup of the group. Statement 2 | The symmetric group S_10 has 10 elements.
 A. True, True
 B. False, False
 C. True, False
 D. False, True
Answer: C

The cyclic subgroup of Z_24 generated by 18 has order
 A. 4
 B. 8
 C. 12
 D. 6
Answer:

Also added the space before choices

Good pt. Done!


TL;DR is now rendered as title

URL: https://yearn.github.io/yearn-devdocs/v2/process-and-procedures/emergency
TL;DR is now rendered as title :/

This change should go to the versioned docs


Adding pseudo-elements in available types and CSS selector grouping?

Hi kuya @jofftiquez ! I want to contribute in this project cause I find this really interesting :>.  What do you think about adding commonly used pseudo elements like ::after and ::before as available types??
I also want to try and implement css selector grouping similar to this:
h1, h2, h3 {
// some styles
}

If you're up for it, I can work on this <3

Hello @chaaals is this related to the previously accepted PR? Thanks <3

Hello @chaaals is this related to the previously accepted PR? Thanks <3

yup!!

Cool! Closing this now, thank you 💜


Add support for storing log files in specified path in bucket

I'm missing possibility to store log files in specified path not only on root folder.
With this tweak you can pass "path" attribute as "my/container/log.txt" and log files will be saved in  "my/container/*" instead bucket root folder.

Just one question (I haven't tried this yet): When you use the sink like this:
var logger = new LoggerConfiguration().WriteTo
    .AmazonS3(
        "awsSubPath/log.txt",
        "mytestbucket-aws",
        Amazon.RegionEndpoint.EUWest2,
        "ABCDEFGHIJKLMNOP",
        "c3fghsrgwegfn://asdfsdfsdgfsdg"
    )
    .CreateLogger();

, your changes will make it possible to log to e.g. mytestbucket-aws/awsSubPath/log.txt on S3 (as if S3 was a regular file system), correct?

Exactly as you described. After some testing I'm missing one more thing. When you set path with subfolder, then log file created on local machine is stored in same subfolder in application working directory and stores log files there. For example for path awsSubPath/log.txt and application run in C:\MyApplication files are stored locally in C:\MyApplication\awsSubPath\log.txt and on S3 mytestbucket-aws/awsSubPath/log.txt
I think it would be awesome to have possibility to configure separate path for files stored locally and separate for S3 bucket path.

@qrczak0 Yep, that was exactly what I was thinking about. Your new changes look better for this scenario. I'm not sure yet when I will be able to merge it and make a new version. I hope tomorrow.

I have updated my PR and decided to add new parameter "bucketPath" and there it is possible to define where files will be stored on Amazon S3 bucket. Path for local files will stay as it was before.

I will have to change this to have a default value of string.Empty because null crashes the line:
Key = Path.Combine(this.pathRoller.LogFileBucketPath,
    Path.GetFileName(this.currentFileName)).Replace("\\", "/")

This is now added in the version 1.0.5.

Great, thanks :)

Something is wrong with dependant AWS packages
https://www.nuget.org/packages/HaemmerElectronics.SeppPenner.Serilog.Sinks.AmazonS3/
AWSSDK.Core (>= 3.3.109)
where for AWSSDK.Core newest version is 3.3.104 (https://www.nuget.org/packages/AWSSDK.Core)

Yeah, I have seen that... I will provide a new package soon...

I have created a seperate issue to avoid duplicates (Others might find this issue as well).

Is fixed with https://github.com/SeppPenner/Serilog.Sinks.AmazonS3/commit/554de96d7308bea2277e36b9692c34df1955fb51 in version 1.0.6.


Ensure all services contribute to distributed tracing with proper name

Enforce consistent naming schema for open telemetry tracers

[x] OrdersService
[x] ProductsService
[x] NotificationService
[x] ShippingService
[x] PriceWatcherService
[x] PriceDropNotifierService
[x] Gateway

don't know yet if we should go with or without Service suffix... maybe this relates to the question if we can customize the service name used by daprd sidecars.

All our services instrument using the <NAME>Service schema. darpd sidecars use the appId as ServiceName.


Tab key does not insert correct number of spaces

This is a re-filing of https://github.com/microsoft/vscode/issues/80129 because they insist this is an extension-specific problem even though it happens without the Python extension and generally applies to most programming languages.
When pressing the Tab key and using spaces for indentation, VS Code inserts the number of spaces to reach the next tab stop from the current insertion point.
Most of the time when indenting code, you are inserting indentation on the left of the line, and you don't care about the current insertion point, but where the first non-whitespace character is, i.e., the start of the code.

Consider I have a line of code in a document with 4 spaces per tab. I click somewhere in the 100px of whitespace to the left of the first code character. Then there is a 75% chance that hitting Tab messes up the formatting of your document, possibly causing a syntax error in Python.
The algorithm for indenting should be:

If the cursor is in the block of whitespace to the left of the first non-whitespace character, insert n spaces.
If the cursor is not, insert enough spaces to reach the next tab stop.

Environment data

VS Code version: 1.48.0-insider (and all versions)
Extension version (available under the Extensions sidebar): v2020.7.96456
OS and version: macOS 10.15.6

@rgov, thanks for letting us know about this.  We agree that this does not appear to be related to the Python extension.  We'll look into the matter further.

Hmm, I expect what the VS Code team meant is that the behavior you desire must be provided by the Python extension and currently is not.  FWIW, this seems like something VS Code should take care of since every extension would have to do exactly the same thing.  Regardless, we'll proceed with this as an enhancement request.  Thanks!

Unfortunately there's not much we can do to address this issue, as it's not necessarily Python specific. Perhaps if other language users report the same issue to VS Code this could be solved upstream.
FWIW, hopefully #481 will address the underlying reason behind this issue.


Add hierarchicalDocumentSymbolSupport to Text Document Client Capabilities

This is already returned by the VS Code insiders build and in https://github.com/Microsoft/vscode-languageserver-node/blob/451eee78181d529cfa98ffc7198e2955cdfdc451/protocol/src/protocol.ts#L425.
It seems necessary in order to maintain backwards compatibility with clients that do not yet support the new DocumentSymbol type.

Thanks @Bubba !


Finding  iOS bundle ID (Bundle Identifier)  using intellij for Firebase Configuration

Hi, I am trying to configure iOS and Android App to Firebase, but I can't find (iOS bundle ID); I already found the Android Bundle ID. I am using IntelliJ on a Linux operating system (Ubuntu Linux distribution).
:information_source:

[x] When I search on YouTube, I found that people are using Xcode then, it's easy to find the iOS bundle ID.

:1st_place_medal:  I actually found it after browsing through all my files:
If someone else has the same problem, check below :information_source:
Android
In Android the package name is in the AndroidManifest:
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
...
package="com.example.appname">
iOS
In iOS the package name is the bundle identifier in Info.plist:
CFBundleIdentifier
$(PRODUCT_BUNDLE_IDENTIFIER)
which is found in Runner.xcodeproj/project.pbxproj:
PRODUCT_BUNDLE_IDENTIFIER = com.example.appname;


WIP: Modularize FC script (with no conflicts)

[Redo of #388]
Simple modularization of the FC script into one main script and two modules for reports and data loading/saving
I now wish to implement better visual reports with some QA/QC checks - but I was thinking that maybe we could have a fmri processing toolbox called in code/fmri with fMRI denoising, FC computation, QA/QC of FC, etc
What do you think ?

I discussed with @acionca, and what makes this PR WIP is the improvement of the visual report. I propose taking over the implementation of the visual report in another PR, as I would like to develop the after-denoising and after-FC estimation QC checkpoint. We agreed that apart from the visualization, this PR can be reviewed and merged.


Add wasm build instructions to example/wasm

Description:
The existing wasm example does not explain how to recompile an wasm filter. The PR describe the steps.
Risk Level:
Testing:
Docs Changes:
examples/wasm/README.md
Release Notes:
[Optional Fixes #Issue]
[Optional Deprecated:]

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approvers:
If they are not already assigned, you can assign the PR to them by writing /assign  in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

What is NOT in this PR:
How to utilize the newly built envoy_filter_http_wasm_example.wasm

@mandarjog: changing LGTM is restricted to assignees, and assigning you to the PR failed.

In response to this:

/lgtm

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.


Set ID manually

Our app does some periodic polling, which is started by a setInterval and not an external call to our app. This call is missing a trace id. Would it be possible for us to manually set an ID before kicking of the async calls?
Or code is essentially this
export async function setupBroadcastPolling(client: SomeClient) {
  async function loadMessages() {
    try {
      const data = await loadBroadcastMessages(client);

      if (data) {
        broadcastMessages = data.messages;
      }
    } catch (error) {
      logger.error({ error }, 'Unable to poll broadcast messages');
    }
  }

  // reload every minute
  setInterval(loadMessages, 60 * 1000).unref();

  await loadMessages();
}

It would be great if we could have an ID in our logs in this case as well. In this case I'd love to somehow set an ID if it's missing within the loadMessages function. Our client does a bunch of logging as it works, which are all missing the request ID

Would it be possible for us to manually set an ID before kicking of the async calls?

@SimenB
This sound a bit orthogonal with the main use case of cls-rtracer, as these async calls seem to happen outside of requests (and web framework, in general).
Do you have multiple of such polling tasks? If it's only a single one per node.js app, why do you need to have an ID assigned to it? I'd like to understand the use case better before suggesting anything.

Our use case for these IDs in general is to use them as trace IDs in logs. We pass them on to the backend for every call as well, and every log statement in both the node app and the backend logs it out. That way we can trace a call through the whole stack. The client also passes it (and is picked up by the middleware provided by this module), so we can associate client request with db queries all the way through.
But since we cannot set the the id for non-request bound async work, we cannot trace that "request" through the system

So, something like the following API would work for you?
rTracer.runWithId(() => {
  const id = rTracer.id() // a new id will be available here
})

And the function will be runWithId(fn[, id]), where id is an optional override for the id available in the function.

@SimenB I'm glad to hear that the feature works fine. Thanks for your feedback!

@puzpuzpuz I've found one issue with this. For some reason, the very first request a client makes doesn't get an ID. I haven't dug into it (tried reproducing in a bare bones app without success), but I just noticed that those requests have been getting a trace id recently, but that ID is the same one as the one provided during boot. We have essentially this code
import rTracer from 'cls-rtracer';
import start from '@private/express-starter';
import setupApp from './server';

async function main() {
  await rTracer.runWithId(async () => {
    const app = await setupApp();
    await start(app);
  });
}

This gives a trace ID for all logging during startup, which is very nice. However, this ID is kept even after the function completes, and used in later requests which do not have an ID.
Any idea about what's going on? While the issue would probably go away if I figured out why the first request is missing an ID, shouldn't the old one be "garbage collected" when the function provided to runWithId completes?

@SimenB hmm, sounds weird. I'll take a look at this issue later today.

Thanks! I'm having troubles reproducing this locally, it seems to only happen when deployed, for some reason... Will keep on trying to get a reproduction locally

@SimenB do you use the same node.js version locally and in production? Which one is that?

12.18.0 in prod, 12.18.1 locally

Any idea about what's going on? While the issue would probably go away if I figured out why the first request is missing an ID, shouldn't the old one be "garbage collected" when the function provided to runWithId completes?

@SimenB the ID will be available only within the function passed into runWithId() method. But if somehow the request is being handled within async call chain that corresponds to the function (which is weird), then the ID may be available there. But it sounds really weird to me and I'd appreciate if you had a reproducer.


Provide options for storing secrets outside the Nix store

Introduction
Dear module authors and maintainers,
We currently have many modules that force users to store their secrets in the world-readble Nix store. This is bad for security. We should give users the option of specifying their secrets in individual files which can be stored outside the Nix store with suitable ownership and permissions. Users could then also use nixops to manage their secret files.
There's still the convenient but unsafe option of storing the secret file in the Nix store using pkgs.writeTextFile. If https://github.com/NixOS/nix/issues/8 gets resolved these files can be encrypted / made private. Also see: https://github.com/NixOS/rfcs/pull/5.
Proposal
The list below contains all the options that force a secret being stored in the Nix store. I propose the following:

Each option should get a warning in the documentation of the form: "Warning: this secret is stored in the world-readable Nix store!"

Each option should get an alternative passwordFile option.

For backwards compatibility the passwordFile option should get a default based on the password option:

{ config = {
    passwordFile = mkDefault (toString (pkgs.writeTextFile {
      name = "password-file";
      text = cfg.password;
    }));
  };
}

Some upstream programs don't support setting a password using a file. In that case an issue should be created in the upstream issue-tracker asking for that feature. (See https://github.com/namecoin/namecoin-core/issues/148 for example). A URL to the issue should be placed in the list below and in the documentation of the password option so that it's easier to track when it gets resolved.

If after some time (lets use September 2017 for now) the upstream developers have not provided the feature to specify the password by file, the NixOS module should be changed such that the config file that contains the password is written to /run before the service starts up. So something like the following:

{
  systemd.services.my-service = {
    preStart = ''
      cat > /run/my-service/config << EOF
      ...
      password = $(cat "${cfg.passwordFile}")
      ...
      EOF
    '';
    script = "${pkgs.myService}/bin/my-service --config=/run/my-service/config";
  };
}

Lets use this issue for planning and to track progress. Please mention in the comments if you have provided a passwordFile option for one of the options below. Then I check the box to indicate it has been resolved. See PR https://github.com/NixOS/nixpkgs/pull/24146 for reference.

If we make sure the new options are backwards compatible we could consider cherry-picking them onto release-17.03 making sure users get these security fixes ASAP.

Secret options

[ ] basicAuth
nixos/modules/services/web-servers/nginx/vhost-options.nix#L118
@globin

[x] networking.defaultMailServer.authPass
nixos/modules/programs/ssmtp.nix#L92
PR: https://github.com/NixOS/nixpkgs/pull/24331

[x] networking.wireless.networks.*.psk
nixos/modules/services/networking/wpa_supplicant.nix#L49
@edolstra

[x] security.duosec.skey
nixos/modules/security/duosec.nix#L59
@thoughtpolice

[x] services.aiccu.password
nixos/modules/services/networking/aiccu.nix#L48
@edwtjo mentions: SixXS is closing down its IPv6 tunnel in June so it doesn't seem worth the effort to create a patch for aiccu to support password files. Lets just remove the service in 0606.

[ ] services.almir.director_password
nixos/modules/services/backup/almir.nix#L129
@domenkozar

[ ] services.bacula-dir.password
nixos/modules/services/backup/bacula.nix#L313
@domenkozar
Feature request for a PasswordFile parameter

[ ] services.bacula-[fd|sd].director.*.password
nixos/modules/services/backup/bacula.nix#L114
@domenkozar
See the feature request above.

[x] services.bepasty.servers.*.secretKey
nixos/modules/services/misc/bepasty.nix#L72
@makefu
PR: https://github.com/NixOS/nixpkgs/pull/24755

[ ] services.btsync.httpPass
nixos/modules/services/networking/btsync.nix#L175
@thoughtpolice

[ ] services.buildbot-worker.workerPass
nixos/modules/services/continuous-integration/buildbot/worker.nix#L56
@nand0p

[x] services.cadvisor.storageDriverPassword
nixos/modules/services/monitoring/cadvisor.nix#L54
@offlinehacker
PR: https://github.com/NixOS/nixpkgs/pull/24341

[ ] services.cassandra.keyStorePassword
nixos/modules/services/databases/cassandra.nix#L236
@cransom
See: https://issues.apache.org/jira/browse/CASSANDRA-13428

[ ] services.cassandra.trustStorePassword
nixos/modules/services/databases/cassandra.nix#L241
@cransom
See: https://issues.apache.org/jira/browse/CASSANDRA-13428

[ ] services.cgminer.pools.*.password
nixos/modules/services/misc/cgminer.nix#L60
@offlinehacker

[ ] services.cjdns.authorizedPasswords
nixos/modules/services/networking/cjdns.nix#L103
@ehmry

[ ] services.cfdyndns.apikey
nixos/modules/services/misc/cfdyndns.nix#L20
@colemickens

[ ] services.coturn.cli-password
nixos/modules/services/networking/coturn.nix#L249
@Ralith

[ ] services.coturn.static-auth-secret
nixos/modules/services/networking/coturn.nix#L174
@Ralith

[ ] services.cpuminer-cryptonight.pass
nixos/modules/services/misc/cpuminer-cryptonight.nix#L38
@ehmry

[ ] services.crowd.openidPassword
nixos/modules/services/web-apps/atlassian/crowd.nix#L53
@fpletz @globin

[ ] services.dd-agent.api_key
nixos/modules/services/monitoring/dd-agent.nix#L112
@shlevy

[ ] services.ddclient.password
nixos/modules/services/networking/ddclient.nix#L47
@rbvermaa

[ ] services.factorio.game-password
nixos/modules/services/games/factorio.nix#L144
@elitak

[ ] services.factorio.password
nixos/modules/services/games/factorio.nix#L130
@elitak

[ ] services.frab.secretKeyBas
nixos/modules/services/web-apps/frab.nix#L118
@fpletz

[ ] services.gammu-smsd.backend.sql.password
nixos/modules/services/misc/gammu-smsd.nix#L192
@zohl

[x] services.gitlab.databasePassword
nixos/modules/services/misc/gitlab.nix#L203
@fpletz  @offlinehacker
PR: #31358

[x] services.gitlab.secrets.secret
nixos/modules/services/misc/gitlab.nix#L326
@fpletz @offlinehacker
PR: #31358

[x] services.gitlab.smtp.password
nixos/modules/services/misc/gitlab.nix#L295
@fpletz @offlinehacker

[x] services.gogs.database.password
nixos/modules/services/misc/gogs.nix#L102
@schneefux
PR: https://github.com/NixOS/nixpkgs/pull/25116

[ ] services.grafana.database.password
nixos/modules/services/monitoring/grafana.nix#L137
@offlinehacker

[ ] services.grafana.security.adminPassword
nixos/modules/services/monitoring/grafana.nix#L157
@offlinehacker

[ ] services.grafana.security.secretKey
nixos/modules/services/monitoring/grafana.nix#L163
@offlinehacker

[ ] services.graylog.passwordSecret
nixos/modules/services/logging/graylog.nix#L68
@fadenb

[ ] services.graylog.rootPasswordSha2
nixos/modules/services/logging/graylog.nix#L82
@fadenb

[ ] services.hologram-server.ldapBindPassword
nixos/modules/services/security/hologram-server.nix#L68
@nand0p

[ ] services.hostapd.wpaPassphrase
nixos/modules/services/networking/hostapd.nix#L124

[x] services.httpd.extraSubservices..."limesurvey"...adminPassword
nixos/modules/services/web-servers/apache-httpd/limesurvey.nix#L143
@offlinehacker

[x] services.httpd.extraSubservices..."limesurvey"...dbPassword
nixos/modules/services/web-servers/apache-httpd/limesurvey.nix#L131
@offlinehacker

[x] services.httpd.extraSubservices..."mediawiki"...dbPassword
nixos/modules/services/web-servers/apache-httpd/mediawiki.nix#L207
@shlevy @ip1981

[x] services.httpd.extraSubservices..."owncloud"...adminPassword
nixos/modules/services/web-servers/apache-httpd/owncloud.nix#L403
@matejc

[x] services.httpd.extraSubservices..."owncloud"...dbPassword
nixos/modules/services/web-servers/apache-httpd/owncloud.nix#L429
@matejc

[x] services.httpd.extraSubservices..."owncloud"...SMTPPass
nixos/modules/services/web-servers/apache-httpd/owncloud.nix#L527
@matejc

[x] services.httpd.extraSubservices..."wordpress"...dbPassword
nixos/modules/services/web-servers/apache-httpd/wordpress.nix#L138
@qknight
PR: https://github.com/NixOS/nixpkgs/pull/24146

[ ] services.i2pd.proto.http.pass
nixos/modules/services/networking/i2pd.nix#L351
@edwtjo

[ ] services.icecast.admin.password
nixos/modules/services/audio/icecast.nix#L62
@k0ral

[ ] services.kibana.elasticsearch.password
nixos/modules/services/search/kibana.nix#L92
PR: https://github.com/NixOS/nixpkgs/pull/30662

[x] services.longview.mysqlPassword
nixos/modules/services/monitoring/longview.nix#L78
@rvl
PR: https://github.com/NixOS/nixpkgs/pull/24366

[x] services.matrix-synapse.macaroon_secret_key
nixos/modules/services/misc/matrix-synapse.nix#L545
@roblabla

[x] services.matrix-synapse.registration_shared_secret
nixos/modules/services/misc/matrix-synapse.nix#L453
@roblabla

[x] services.matrix-synapse.turn_shared_secret
nixos/modules/services/misc/matrix-synapse.nix#L434
@roblabla

[x] services.matrix-synapse.recaptcha_private_key
nixos/modules/services/misc/matrix-synapse.nix#L404
@roblabla

[ ] services.mattermost.localDatabasePassword
nixos/modules/services/web-apps/mattermost.nix#L108
@fpletz

[x] services.murmur.password
nixos/modules/services/networking/murmur.nix#L105
@thoughtpolice

[ ] services.mysql.replication.masterPassword
nixos/modules/services/databases/mysql.nix#L149
@edolstra

[ ] services.namecoind.rpc.password
nixos/modules/services/networking/namecoind.nix#L90
@rnhmjoj
See: https://github.com/namecoin/namecoin-core/issues/148

[ ] services.nntp-proxy.upstreamPassword
nixos/modules/services/networking/nntp-proxy.nix#L99
@fadenb

[ ] services.oauth2_proxy.cookie.secret
nixos/modules/services/security/oauth2_proxy.nix#L371
@jml

[ ] services.panamax.secretKey
nixos/modules/services/cluster/panamax.nix#L63
@matejc

[ ] services.prometheus.*.consul_sd_config.password
nixos/modules/services/monitoring/prometheus/default.nix#L243
@fpletz @doshitan

[ ] services.prometheus.*.scrape_config.basic_auth.password
nixos/modules/services/monitoring/prometheus/default.nix#L128
@fpletz @doshitan

[ ] services.prometheus.unifiExporter.unifiPassword
nixos/modules/services/monitoring/prometheus/unifi-exporter.nix#L45
@fpletz @doshitan

[ ] services.redis.requirePass
nixos/modules/services/databases/redis.nix#L160
@offlinehacker

[x] services.redmine.databasePassword
nixos/modules/services/misc/redmine.nix#L103
@domenkozar

[ ] services.redsocks.redsocks.password
nixos/modules/services/networking/redsocks.nix#L109
@Ekleog

[ ] services.rippleDataApi.couchdb.pass
nixos/modules/services/misc/ripple-data-api.nix#L109
@offlinehacker

[ ] services.rippled.ports.*.password
nixos/modules/services/misc/rippled.nix#L114
@ehmry

[ ] services.selfoss.database.password
nixos/modules/services/web-apps/selfoss.nix#L89
@regnat

[ ] services.terraria..password
nixos/modules/services/games/terraria.nix#L50
@pshendry @garbas

[ ] services.tor.torsocks.socks5Password
nixos/modules/services/security/torsocks.nix#L89
@thoughtpolice

[ ] services.tt-rss.database.password
nixos/modules/services/web-apps/tt-rss.nix#L163
@zohl

[ ] services.tt-rss.email.password
nixos/modules/services/web-apps/tt-rss.nix#L291
@zohl

[ ] services.wakeonlan.interfaces.*.password
nixos/modules/services/networking/wakeonlan.nix#L32

[ ] services.yandex-disk.password
nixos/modules/services/network-filesystems/yandex-disk.nix#L38
@grwlf @7c6f434c

[x] services.zabbixServer.dbPassword
nixos/modules/services/monitoring/zabbix-server.nix#L66
@robberer

This list was compiled by running the following in <nixpkgs> and manually inspecting and processing the result:
find . -type f -exec grep --color -nH -i -E -e '(secret|pass|key)' {} +`

FYI, check out how I handle secrets throughout all the apps here: https://github.com/ip1981/nixsap/tree/master/modules/apps
Especially, with Jenkins :)

I suspect @fpletz is now the maintainer for gitlab (sorry, I see you'r name is in this list a lot!)

Wow, I didn't know the situation was this bad. All of these options should be removed really. It's almost as if people don't realize that the Nix store is world readable...

@basvandijk — any plan for preventing future changes adding secrets to the Nix store, beyond eternal vigilance?

Hmm, I didn't re-read https://github.com/NixOS/nix/issues/8 completely, but it seemed to me last time that @edolstra 's solution using encryption in the store and decryption at startup time was working and there was mostly bikeshedding about encryption vs ACLs?
This would be much easier than trying to patch every single upstream program that does not accept password files, especially given that some may not be willing to do it as it adds quite a bit of complexity. Wouldn't it?

Besides being quite inconvenient, storing passwords/keys in a file with restricted access outside the store, may not solve the problem: they could end up in a systemd environment file or a unit file if you need to pass those as an command line argument.
Also expecting everyone upstream to comply seems incredibly too optimistic for me.

any plan for preventing future changes adding secrets to the Nix store, beyond eternal vigilance?

@jml we could add something to the PULL_REQUEST_TEMPLATE.md instructing contributors to use passwordFile instead of password options.

All of these options should be removed really...

@edolstra we could do that eventually but to ease the transition we should first provide a backwards compatible passwordFile option, then in a next release we could start throwing warnings when users use the password option and finally in a third release we can remove the password options.

@Ekleog regarding https://github.com/NixOS/nix/issues/8, even if we have the ability to encrypt files in the Nix store I think it would be best to only encrypt files that should be encrypted. Currently we have big config files that somewhere contain a password. It would nicer if the config file remains unencrypted because then it can be shared and it makes debugging easier. Only the password needs to be encrypted. So having passwords in individual files would still be desirable.

@basvandijk The encryption stuff allows you to encrypt only the "secret" parts of a configuration file. See https://github.com/edolstra/nixpkgs/commit/4c8212069429bf9fb959e00ce8d9345ac7cb7ff0#diff-6c3fcb531890fdce200531b9ac69e4f8R14 for an example.

@basvandijk Sometimes even the password stored in the configuration file needs to be readable. dnschain, for example, parses namecoin.conf to connect to the rpc server.

@rnhmjoj lets see how upstream reacts to a request for a rpcpasswordFile parameter...

The encryption stuff allows you to encrypt only the "secret" parts of a configuration file.

@edolstra that's great! What's needs to be done to get this merged into Nix?

Well, it's not clear whether this is the way to go. @kevincox listed the issues here: https://github.com/NixOS/nix/issues/8#issuecomment-265256236

@rnhmjoj regarding namecoind, would cookie based authentication work?

Besides being quite inconvenient, storing passwords/keys in a file with restricted access outside the store, may not solve the problem: they could end up in a systemd environment file or a unit file if you need to pass those as an command line argument.

@rnhmjoj regarding secrets in systemd unit files, we can always create a wrapper script that cats the password file and passes that on to the original script. I do something similar here.

@basvandijk Regarding upstream changes for password file options: I think some cat/sed trickery, maybe creating a config file in /run from a template should also do the job for most cases where upstream doesn't have or doesn't want to add a password file, shouldn't it?

@mbrgm sure and we should do that in case upstream doesn't provide a password file option.

@basvandijk I would not tick the package until the PR is merged.

@basvandijk That seems a valid alternative for authenticating to namecoind however dnschain does not support it, so it would break the service. Anyway, thank you for opening the issue.

I would not tick the package until the PR is merged.

@rasendubi makes sense. I've unticked the wordpress checkbox.

Well for aiccu; SixXS is closing down its IPv6 tunnel in June so it doesn't seem worth the effort to create a patch for aiccu to support password files. Lets just remove the service in 0606.

Nix encryption is going to take a while to get there given nix's release history speed. I don't think it should be a blocker for trying alternative implementations.
On the nixpkgs side, the protection would be based on unix file ACL. How about treating secrets like any other state? We could introduce a "mkState" interface that defines any kind of state reference on the filesystem.
let
  postgresState = mkState {
    type = "directory";
    name = "postgres";
    owner = "postgres";
    group = "postgres";
    mode = "0700";
    mustExist = true;
    # run a command if it's missing
    onMissing = "pg_init";
  };
  assert (toString postgresState) == "/var/lib/postgres";
  # mkSecret is a specialization of mkState with a default dir to /run/keys/${name}, mode = 0700 and mustExist = true
  nginxSecret = mkSecret {
    owner = "nginx";
  };

This would translate to (1) some activation script actions like creating the state dir (2) systemd service to initialize and/or check the secret, which can then be used as a dependency for other services.
I know it's still pretty vague but hopefully enough to convey the idea.

@zimbatm that would be quite interesting. Especially how that could work together with NixOps.
We'd need:

Generic mechanism to facilitate generating secrets .
A way to expose how a secret must be created from a service.
Make it possible for NixOps to manage those secrets (generating them on the deploying machine, then storing them in the statefile and on the necessary machines).

Your mkSecret function could be extended to take a nixpkgs provided method or generic command.
{
  s1 = mkSecret {  owner = "nginx"; method = "sha256"; };
  secretGeneratorProgram = stdenv.mkDerivation { /* ... */ };
  s2 = mkSecret { owner = "nginx"; command = "${secretGeneratorProgram}/bin/gen > $out"; };
}

Relevant nixops thread: NixOS/nixops#627

Would anyone like to review my module PR #24366.
If anyone has a module listed here and no time to fix it, I could do a couple.

Would anyone like to review my module PR #24366.

LGTM (but I haven't tested it).

If anyone has a module listed here and no time to fix it, I could do a couple.

That would be much appreciated. Thanks @rvl!

If anyone has a module listed here and no time to fix it, I could do a couple.

Can you take a look at gogs' database configuration?  Thanks 😃

I'd like to think of keys/passwords as of the data, not the code (which is in Nix store).
I'm handling secrets on top of existing facilities with https://github.com/ip1981/nixsap/blob/master/modules/deployment/keyrings.nix

@basvandijk in your 5 you have:
$(cat "${cfg.passwordFile}")

if cfg.passwordFile type is set to types.path and someone specifies a path instead of string, it would still get pulled into nix store.
I propose we create a type called types.nonStorePath that can be set to path or string, but the resolved value would be a string that's not in nix store.

I propose we create a type called types.nonStorePath that can be set to path or string, but the resolved value would be a string that's not in nix store.

I like that idea. Sounds pragmatic and easy to understand (don't know about the implementation part though). Anyway, I think any solution to that problem would be welcome to most people. Imho secret management is one of NixOS' major weaknesses in practice.

Groundwork is in https://github.com/NixOS/nixpkgs/pull/31715

Implementation shouldn't be too difficult. I think we should have three types:

path: any kind of absolute path
nonStorePath: path that must NOT be in nix store
 storePath: path that must be in nix store

Suppose there are secrets that are only needed during the evaluation of configuration.nix (such as secrets required to download a particular package or install a particular package), then this can be done by passing in variables during invocation. Then there are secrets that are needed all the time by the services (and these have to be the plain text secret), then this would mean that these secrets exist as plain text on the filesystem. Is there a solution that allows the secrets to still be encrypted on disk, while the nixos system can still access them only when needed (possibly decrypting on the fly with an in-memory key).

@domenkozar I like having those types.
However, setting the type of passwordFile to nonStorePath does mean we can't have backwards compatibility as described in point 3.

However, setting the type of passwordFile to nonStorePath does mean we can't have backwards compatibility as described in point 3.

How about just breaking it? As long as it’s not causing stuff to fail silently, that should be very much appropriate for a bad security issue like passwords in the store.
For stable users this will hit in the next release with a big notice. Unstable users will have to adjust, but that’s what they’re in for.

@basvandijk I think for sake of security we can change it to path and it will report an error. It will force the user to rethink how to set the passwords, which is a good thing.

What about keeping secrets in world-readable nix-store - it simplifies build and deploy - but encrypt them with something like cryfs so the secrets to be encrypted on all the intermediate stages ?

Software already sucks too much :) IOW, let's keep it simple.

@volth
"We can solve any problem by introducing an extra level of indirection." -- Andrew Koenig

I am reminding people discussing this issue to reflect and meditate on the root issue that is causing this bug. Encryption should not be used for fixing architectural flaws.
What are the assumptions of the nix store? What additional assumption should there be?
Please read my original rant on this topic.

My problem with secrets in nix store is a bit different - I am worrying about secrets of many machines are visible on the builders and also exposed to the public on my alternative to cache.nixos.org.
From another side - it is much convenient to deploy the secrets using the usual deployment machinery (nix-copy-closure) than to copy them into /run afterwards.

How is storing secrets in the nix store different from hardcoding them into the binary? Or a library used by the binary? No difference to me.

How is storing secrets in the nix store different from hardcoding them into the binary? Or a library used by the binary? No difference to me.

Exactly. If the binary stored in the nix store :)
There is the second part - the private key which:

is not in nix store, but is /etc/sshd/ (or on a hardware token)
is not part of any deployment, not the current one nor any deployment in the past.
will be needed to decrypt the secret stored in the nix store (embedded in the binary or as a separate file, this does not matter)

@volth Your problem is just a different manifestation of an architectural flaw.
I solution I came up with:
The /nix/store of an OS using Nix (e.g. NixOS) could check if the file / path requested is allowed for the caller environment, container or user. The store would be mounted using e.g. fuse or a custom kernel module which communicates with the nix daemon.
This would fix the information leak once an for all. Of course, a binary cache would still serve your plaintext files.
To supply secrets to a machine, I think that the method nixops send-keys demonstrates is perfect for online machines (VPS, Server etc.). For offsite, embedded, "install and forget" machines this could also be made permanent by doing scp -R secrets_of_machine machine:/etc/secrets and then on the machine creating a bootup systemd unit with cp -r /etc/secrets/ /run

@volth Regarding encryption, see https://github.com/NixOS/rfcs/pull/5.

@volth @mguentner Another possible solution is to write configs with “holes”, for example by writing a template with interpolated variables where the passwords should be, e.g. in https://dhall-lang.org. Then you have a principled way to “fill in the holes” before giving the config to a process.

Secrets are traditionally very static and it's easy to think that they are configuration. I would like to argue that they should be treated as state instead.
Ideally we want to be in a place where they can be rotated dynamically with for example HashiCorp Vault. When there is a breach we want to be able to quickly rotate the keys. And during the investigation is helps if each machine has a unique key to figure out where the attack was coming from. It also discourages having the secrets stored in git since they are comming from another place. The nixops user should only have one secret, how to access HashiCorp Vault (and maybe a SSH key).
In that regard I think this initiative makes sense. Encourage upstream to provide paths to secrets instead of embedding them in configuration. And when not possible, generate the config at runtime with a template. At some point we'll figure out how to source those secrets from other places than the filesystem.

@Profpatsch @zimbatm not all secrets are configuration and state, however. You may not want to have your system-wide config be effectively traceable by everyone -- security through obscurity does not do any good on its own, but that doesn't mean it isn't useful. You may very well want to keep source files or actual package derivations hidden from most users in a mixed public/private environment, if they are proprietary in nature.
Nix is based on the idea of 'deployment as memory management', but it is memory management as seen purely from a language or application perspective, with nothing from an OS perspective, where isolation and protection are the norm. This is perhaps an oversight.

@Shados maybe your case is better handled by running different things in different namespaces with only the allowed paths bind-mounted from the global store. In the OS RAM management terms, let the main store be like physical RAM and give the programs page tables according to their needs/access level.

@Shados these approaches are not mutually exclusive, even if paths are configurable it's still possible to point into the /nix/store. The difference is that path extractions is something we can do today without any changes to the nix language or runtime.
In Nix 2.0 and Linux it's also possible to maintain your own private nix store by passing the --store ~/your-store.

@zimbatm About your last comment about nix 2.0, can you provide an example of how you use that to manage secrets from a private nix store?

So if someone puts a password in plain text in a .nix file, that password gets uploaded to somewhere online that could be accessed by anyone?

@ob7 "World readable" in this sense means that any user on your system can read the file. It does not mean that files included in the nix store are somehow uploaded elsewhere.

@basvandijk You can mark redmine as complete as of 18.09 as a passwordFile option has been added.

@basvandijk the owncloud module was removed so you can tick that off.

This is a related topic I started on Discourse: In configuration.nix can I read a value from a file?

Also see https://github.com/NixOS/rfcs/pull/59

services.mattermost.localDatabasePassword needs adjustment too.

Related is my PR which adds types.secretPath: https://github.com/NixOS/nixpkgs/pull/78640

List Updates
These options now exist:

services.grafana.database.passwordFile
services.grafana.security.secretKeyFile
services.grafana.security.adminPasswordFile

These options need an associated passwordFile option:

services.grafana.provision.datasources.*.password
services.grafana.provision.datasources.*.basicAuthPassword

This issue has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/any-default-config-for-reproducible-configurations/5268/17

services.mattermost.localDatabasePassword is not possible via file as well.

I think we should add to this list the mpd module, which now gets the ability to store it's password "<pwd>@permissions" entry outside the store, at https://github.com/NixOS/nixpkgs/pull/95599 .

I just noticed that virtualisation.oci-containers.containers.<name>.environment doesn't allow scripting:
https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/virtualisation/oci-containers.nix#L242
Container environment variables are frequently used for secrets, too, thus I think we should provide an easy way to include them.
I see some different ways to solve this:

stop escaping the value of the environment variables and thus allow for shell scripting (potentially breaking current configs)
add an environmentFiles option that reads variables from files, similar to the passwordFile options
add an option or separate config to opt-in to shell scripting in environment variables

We have to link here https://github.com/Mic92/sops-nix as an elegant solution for secrets in Nix

For hashicorp Vault here's a PR that lets you upload the config via NixOps or use any path.
https://github.com/NixOS/nixpkgs/pull/107323
I don't think I've seen this approach before.
Example config:
{ config, ... }:
{
  services.vault.storageConfig = ''connection_url = "${somethingSensitive}"'';
  services.vault.configPath = "/run/keys/vault.hcl";
  users.users.vault.extraGroups = ["keys"];
  deployment.keys."vault.hcl" = {
    text = config.services.vault.configText;
    user = "vault";
  };
}

Note, vault storageConfig isn't super sensitive because a leak isn't supposed to affect confidentiality or integrity, but it will affect availability.

For hashicorp Vault here's a PR that lets you upload the config via NixOps or use any path.
https://github.com/NixOS/nixpkgs/pull/107323
I don't think I've seen this approach before.
Example config:
{ config, ... }:
{
  services.vault.storageConfig = ''connection_url = "${somethingSensitive}"'';
  services.vault.configPath = "/run/keys/vault.hcl";
  users.users.vault.extraGroups = ["keys"];
  deployment.keys."vault.hcl" = {
    text = config.services.vault.configText;
    user = "vault";
  };
}

Note, vault storageConfig isn't super sensitive because a leak isn't supposed to affect confidentiality or integrity, but it will affect availability.

Add services.bitwarden_rs.config.yubicoSecretKey to the list...

I'm trying to fix wpa_supplicant (networking.wireless) in #134618. Could anyone help by reviewing it?

There are now docs for how to handle this for new modules at https://github.com/NixOS/nixpkgs/pull/142282

These docs handle the easy part of the problem. They are about how use use secrets outside of the store, not how to manage the secrets themselves. Ideally we wouldn't need to push this complexity onto the user.

We can probably tick the networking.wireless.networks.*.psk box now that https://github.com/NixOS/nixpkgs/pull/134618 was merged.

This issue has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/agenix-encrypted-plaintext-passwords-and-builtins-readfile/18425/8

Some boxes can be checked (✅ done, 🙅 module removed, ❓ option doesn't seem to exist anymore):

basicAuth ✅
services.almir.director_password 🙅
services.btsync.httpPass 🙅
services.buildbot-worker.workerPass ✅
services.cassandra.keyStorePassword ❓
services.cassandra.trustStorePassword ❓
services.cfdyndns.apikey ✅
services.coturn.cli-password ✅ (can be stored as a hash)
services.coturn.static-auth-secret ✅
services.crowd.openidPassword ✅
services.dd-agent.api_key ✅ (now services.datadog-agent.apiKeyFile)
services.ddclient.password ✅
services.frab.secretKeyBas 🙅
services.grafana.database.password ✅ (grafana-specific alternative pointed in option's doc)
services.grafana.security.adminPassword ✅ (grafana-specific alternative pointed in option's doc)
services.grafana.security.secretKey ✅ (grafana-specific alternative pointed in option's doc)
services.httpd.extraSubservices.* ❓ (maybe I'm missing something)
services.panamax.secretKey 🙅
services.prometheus.*.consul_sd_config.password ✅ (now services.prometheus.scrapeConfigs.*.consul_sd_configs.*.basic_auth.password_file)
services.prometheus.*.scrape_config.basic_auth.password ✅ (now services.prometheus.scrapeConfigs.*.consul_sd_configs.*.basic_auth.password_file)
services.redis.requirePass ✅ (now services.redis.servers.<name>.requirePassFile)
services.tor.torsocks.socks5Password ✅ (although it should maybe first be better documented that using systemd.services.tor.serviceConfig.EnvironmentFile is possible)
services.tt-rss.database.password ✅
services.wakeonlan.interfaces.*.password 🙅

Maybe some new unsecure module options appeared since then!

Is there a conclusion to this issue that specifies exactly what behaviour is?

Can someone edit the issue? @basvandijk maybe :)


Flush doesn't clear parser buffer.

SerialPort version: 4.0.1
NodeJS Version: 4.2.1
Operating System and Hardware: RPi 2
Are you using Electron/Atom/NwJS?: No
Summary of Problem
Data remains in buffer of parser after flush is called.
Expected Behavior
No data in parser buffer after flush.
Actual Behavior
Data in parser buffer after flush.
Steps and Code to Reproduce the Issue
Use newline parser. Receive some data without newline. Call flush. Receive data + newline. Data event contains the data from before the flush as well as the data from after it.

Surely that is important functionality. If you know there is garbage in the parser, it seems the only way to empty it is to close and reopen the serial port.


Issue with TimberImage caption and WPML

Hello,
I'm facing an issue with the caption retrieved from timber using TimberImage and the new way WPML handles the translation of image's metadata.
Expected behavior
When in secondary language, when I use {{ Image( image ).caption }}, I'd expect to see the translated caption.
Actual behavior
The caption returned is the one in the main language.
Steps to reproduce behavior

In single.twig print the featured image caption of the post using {{ Image( image ).caption }}.
Upload an image, write down a caption, and set it as featured image of a post
Go to WPML -> Media Translation and translate the caption of the image you just uploaded.
Create a translation of the previously created post
The caption used is the one in the original language, not the one translated

What version of WordPress, PHP and Timber are you using?
Wordpress 4.9.8
Timber 1.8.3
WPML 4.1.3
WPML media translation 2.4.1
How did you install Timber? (for example, from GitHub, Composer/Packagist, WP.org?)
wp.org

Hello :-)
Anybody experiencing this issue as well?

This issue might still need our attention.

I took a deeper dive @langhenet  and found out that this will be fixed in 2.0 of Timber. See the following commit: https://github.com/timber/timber/commit/5a2771d5d1bf97dc8e086a606ba539129fb1f621
WPML hooks into wp_get_attachment_caption as well to update the translation. Since we are not going to patch this in the v1 version anymore, I am going to close this here.


Librarian's backpack doesn't change after picking a type

Description

No matter what type of a backpack you pick, character sprite isn't changing. I don't know if it changes in-game or not, but so far I've observed this issue only in a character creator.
Screenshots

Additional context

Can confirm after playing a round that in-game backpack does not change for librarian.

Lawyer's backpack also doesn't change

After much of a long time, I realized it's now wrking correctly.


Return middleware chain promise from callback()

Without this, there's no way of hooking into after Koa is done with the response. This can be extremely important if used with ctx.respond = false, as in #846 (see there for use case). While this is technically a breaking change, I don't expect it to break anything.

i.m.o. it's not really a breaking change :)

Yeah, the one situation I could see this breaking anything is if (handler(req, res)), but that shouldn't happen (hopefully). http://xkcd.com/1172/

Current coverage is 100% (diff: 100%)

Merging #847 into master will not change coverage

@@           master   #847   diff @@
====================================
  Files           4      4
  Lines         417    417
  Methods        81     81
  Messages        0      0
  Branches      102    102
====================================
  Hits          417    417
  Misses          0      0
  Partials        0      0

Powered by Codecov. Last update c62aa90...8836cd3

Current coverage is 100% (diff: 100%)
Merging #847 into master will not change coverage

I don't think that's how diffs work... :smile:

can we add a test for the use-case if we add this? not opposed to it.

This is great! One thing to be aware of is that even if you do ctx.respond = false and then use the promise-returning API from this PR to do further processing after koa finishes with a request, there is still the fact that koa has set res.statusCode = 404. It's not difficult to reset that value, but you risk overwriting something in the koa app explicitly setting status code to 404. Is there a way for koa to internally mark a request as "maybe send 404 if respond is reached" without actually changing the state of the res object -- seems important especially in this context, because once control is transferred back to the enclosing app, the enclosing app sees the 404 attached by koa.

Is there a way

yes :) suggestions welcomed. i haven't thought about this yet, so feel free to make suggestions.

Maybe app.usedAsMiddleware? It would remove the default statusCode setting and return early here (don't res.end() if there's nothing to send).

Okay, I've just added in app.usedAsMiddleware. Let me know how it looks and I can port it over to the v2 PR.

im ok with it. anyone else? @koajs/owners
though maybe we could use a name with only 1-2 words :P

Yeah a better name for the app variable would be appreciated.

I can rebase and pull the changes into the v2 branch if you're ready to merge.

Any updates on this?

If a better config name is needed, maybe disableDefaultResponse?

why not àpp.middlewareMode or app.mode = 'middleware'

Because that describes a common use case, not its function.

So as I look at this again, it occurs to me that instead perhaps we can be more generic. usedAsMiddleware is still a pretty specific name and purpose. I propose instead app.defaultStatus with an initial value of 404. Thus we can end up with something like the following:
const app = new Koa();
console.log(app.defaultStatus); // logs "404"
app.defaultStatus = null; // override it

// main app logic and routes here
app.use(async ctx => {
  if (ctx.url !== "/user") return;
  const user = await db.getUser();
  ctx.body = user;
  ctx.status = 200; // <-- set status here
});

// final middleware, responsible for just checking if something upstream
// set a status. if so, we want to let Koa respond. otherwise, we will assume
// the outer framework will handle the request
app.use(async ctx => {
  if (ctx.status == null) {
    ctx.respond = false;
  }
});

This also lets us drop checking of app.usedAsMiddleware in the respond function, and we can let ctx.respond do it's original job.
(This all assumes we are returning the promise as already implemented in this middleware)
Thoughts?

@PlasmaPower @jonathanong any thoughts on the comment above? would love to get support for embedding Koa into existing applications merged ASAP. Happy with the current implementation if people feel it's superior to what I proposed above.

@nickb1080 yes, that looks feasible. It's been a while since I looked at this issue. if all we want to merge is return fn.call(ctx).then(...), then I'm okay with that

@jonathanong How do you want to proceed? Do you want me to open a PR against v2.x with the single small change of adding return?

@nickb1080 why not master?

Why does the branch v2.x still exist? It seems to be just a previous point on master.

Ah! I hadn't noticed the switchover had already happened.

Yeah I agree. Closing this in favor of #848.


The WASM code cannot run on MacOS system

Describe the bug
On MacOS system running WebAssembly code in Easegress report following error whether using easegress-assemblyscript-sdk or easegress-go-sdk
thread '<unnamed>' panicked at 'called `Option::unwrap()` on a `None` value', crates/c-api/src/linker.rs:105:80
2022-07-12T10:23:37.039+08:00   ERROR   wasmhost/wasmhost.go:331        recovered from wasm error: runtime error: invalid memory address or nil pointer dereference
2022-07-12T10:23:37.039+08:00   ERROR   httpserver/mux.go:437   http-server: response is nil

To Reproduce
Just following README.md doc instruction
OS Version
Darwin 20.4.0 x86_64

@suchen-sci   This error log line ERROR   httpserver/mux.go:437   http-server: response is nil can be easily reproduced：using Easegress v2.0.0  don't configure proxy filter，just handle logic and return response in Wasm code.
The bug may be caused by that the response object is nil and not created by default in wasm code, like this issue

@jthann , the behavior has been changed because v2.0 does not have a default response. We can now use a ResponseBuilder before the WasmHost to create the response as a workaround. We will enhance the WasmHost filter to make it able to access requests/responses in any namespace later.

It work out okay after adding ResponseBuilder filter
name: wasm-demo-pipeline
kind: Pipeline
flow:
- filter: respBuilder
- filter: wasm
filters:
- name: respBuilder
  kind: ResponseBuilder
  template: |
    statusCode: 200
- name: wasm
  kind: WasmHost
  code: wasm-demo.wasm


Get images and merged column details from workbook to show in html table?

I want to load whole workbook in HTML with cell style, merged column and row detail and all the images.
How can I get all the things. i am unable to to achieve images, styles and merged details.

Extracting images and styles is something we offer in our Pro builds


tvos support

Well, you needs add process in script: https://github.com/axys1/buildware/blob/main/1k/build1.sh#L95, not only yml

@halx99 added all scripts...what more need be changed?

Let me merge firstly for github action checks

The ci build finished, you can download tv prebuilt from: https://github.com/axys1/buildware/actions/runs/2983205738 and test them

It build some libs for iOS instead of tvOS.
Can you add me on whatsapp or hangout for fast talk?
Email, chat, google, hangout<EMAIL_ADDRESS>WhatsApp: +5521993860010

OK


RegEx re-implemented as a module

Re-wrote nrex as a module using godot-specific parts and new features:
Removed use of mutable variables in RegEx. RegExMatch is returned instead. Example:
var re = RegEx.new()
re.compile("\\w+ (\\w+)")
var match = re.search("Hello world!")
print(match.get_string())
print(match.get_string(1))

Output:
Hello world
world

Added string substitutions. Example:
var re = RegEx.new()
re.compile("(\\w+) (\\w+)")
print(re.sub("John Doe", "\\2, \\1"))

Output:
Doe, John

Named groups are now supported. Example:
var re = RegEx.new()
re.compile("(?P<greet>\\w+) .+")
print(re.sub("Goodbye World", "You said \\g<greet>"))
var match = re.search("Hello world!")
print(match.get_names())
print(match.get_string("greet"))
print(match.expand("And I said \\g<greet>"))

Output:
You said Goodbye
[greet]
Hello
And I said Hello

Things left on the TODO list: Documentation and making RegEx an exportable resource.
I should also add the warning:
THIS WILL BREAK EXISTING CODE.

Issue thread discussing this was at #6853

You should likely also git rm -r drivers/nrex no?

@akien-mga Done. Pushed an ammended commit.
Also, did the documentation, added support for global substitution and added bounds checking (realised it was a major security flaw :P).
All that is left is exposing RegEx fields to the editor.

Right, so modifying it to inherit Resource instead seems to be working for the most part, except that using Ctrl-S doesn't actually update the file. You'd have to manually save it from the inspector panel. Could someone have a look and see why that's happening?

@leezh Merge conflicts, seems like too many people touched the same tests file ^^

Yay for compatibility breakage! :D
The new API looks quite good.


fix: 🐛 export default expr codegen should keep expr leading comments

Summary

export default expr codegen align to webpack:

Test Plan

Require Documentation?

[ ] No
[ ] Yes, the corresponding rspack-website PR is __

!bench


Remove path metadata

Removed path metadata as this should be handled by another plugin.
Currently this conflicts with both metalsmith-paths and metalsmith-path. It should be recommended to use one of these two plugins instead.

Recommended for a version 1.0, as this will cause some breaking changes

Reverts changed made by: https://github.com/segmentio/metalsmith-collections/pull/51

Bump

bump

bump


Changed context_processors in the default settings.py

https://groups.google.com/d/topic/django-developers/WlK_7OoThaQ/discussion

Ok, I think I addressed all of the comments. I changed around the docs to prefer the template tags over the context_processor.

Please update the project template itself! (django/conf/project_template). Also, this seems to fix ticket #24124.

ohh, wouldn't that be funny :)


pl_PL.lang

I polished your changes. Thanks for correcting my mistake.

It wasn't your mistake ^^ I forgot to change "Wooden Musket" and so on to "Musket with wooden Bayonet"

@CorruptedSlime 35d5fe707c5db7806ff71ee07670ae57ff873b42 Is that alright? We just wanted another name for the dart (noone knows, what this should be XD )


Code Question

In the trainning process introduced in the paper, you use span filter,span classification, while in your source code, In the inference process,  the span filtering can be found. Whether there any detail  haven't been introduced.

We essentially filter spans during sampling by using only ground truth entity mentions to train the relation classifier. This is actually also stated in the paper: "To train the relation classifier, we use ground truth relations as positive samples, and draw N_r negative samples from those entity pairs S_gt × S_gt that are not labeled with any relation"

I assume this answered your question, please write a comment otherwise.

Wow, Sorry for that, thanks for your help

@markus-eberts When sampling negative relations, will further limit the head/tail types could help performance? i.e. Don't sample [LOC] spans/entities for relation [PEOPLE]-(WORK FOR)-[ORG].

Hi @liebkne,
I also have this idea in mind and it is certainly worth a try. I would not omit these samples totally (since the model still encounters these samples during inference) but reducing them might help performance. Usually samples with entity types not matching a relation are easily to tell apart by the model, so there is probably no need to sample those often. It would be great to hear from you in case you tried it :).

@markus-eberts Thanks for your reply. If we can apply this 'relation type-entity types' mapping in training, I think we can also apply it during inference? I might try that next week.
I have one more question about the NER part. I noticed that in my cases the NER part almost always have very high recall compared to precision (The correct entity is found and so are some FPs. Also usually boundary errors are more than type errors) which I think due to the span prediction approach. I think in real application we need to find a way to keep the correct one.  Do you have any suggestion on this (during training/application)? Thank you!

Hi @liebkne ,
sorry for my late reply. Of course you could also apply the mapping during inference to filter pairs.
Regarding the NER part: Yes, we also found boundary errors to happen more frequently than type errors (depending on the dataset). You can try to oversample spans that are predicted incorrectly by the model. For example, in JEREX (-> https://github.com/lavis-nlp/jerex) we oversample negative spans that overlap with ground truth mentions, since these are usually the hardest to distinguish.


chore: switch to TypeScript tooling for declarations and API docs

Resolves #293. ~Draft as this is based on #314 (last conversions to TypeScript).~
Switch over to the TypeScript tooling — for generating type declarations and docs.
Convert all jsdoc comments to tsdoc, and use typedoc to generate the API docs. Attached typedoc-api.zip zip file to compare the (generated by jsdoc) old API docs with the new typedoc version.
Also tried running through api-extractor and api-documenter, as these have additional checks. These don't create a static site for the API directly, but support markdown or in a yaml format, could be used to generate an asciidoc-based API that could be included directly in the SDK docs. There are still a few warnings from api-extractor currently. But the typedoc generated site looks fine and a good replacement for what we were generating with jsdoc.
Using the SDK from TypeScript or JavaScript is basically unchanged. The API tools don't support legacy-style or synthetic imports, or all possible re-export declarations, so affected imports/exports and options have been updated.
Some things noticed for following up on, beyond just switching tools, have been created as separate issues:

#320
#321
#322

Rebased and ready for review now.


Fails to build with Ubuntu 22.04 LTS

When installing this gem into a Ubuntu server, I get an ERROR: Failed to build gem native extension.
Full Error
current directory: /home/dev/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/or-tools-0.10.1/ext/or-tools

/home/dev/.asdf/installs/ruby/3.2.2/bin/ruby extconf.rb --with-or-tools-dir\=/home/dev/or-tools
checking for rice/rice.hpp in /home/dev/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/rice-4.1.0/include... yes
checking for -lstdc++... yes
checking for -lortools... yes
creating Makefile

current directory: /home/dev/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/or-tools-0.10.1/ext/or-tools
make DESTDIR\= sitearchdir\=./.gem.20231101-63192-jsji3m sitelibdir\=./.gem.20231101-63192-jsji3m clean

current directory: /home/dev/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/or-tools-0.10.1/ext/or-tools
make DESTDIR\= sitearchdir\=./.gem.20231101-63192-jsji3m sitelibdir\=./.gem.20231101-63192-jsji3m
compiling assignment.cpp
compiling bin_packing.cpp
compiling constraint.cpp
compiling ext.cpp
compiling linear.cpp
compiling network_flows.cpp
compiling routing.cpp
routing.cpp: In function ‘void init_routing(Rice::Module&)’:
routing.cpp:356:57: error: ‘GetPickupIndexPairs’ is not a member of ‘operations_research::RoutingModel’
  356 |     .define_method("pickup_index_pairs", &RoutingModel::GetPickupIndexPairs)
      |                                                         ^~~~~~~~~~~~~~~~~~~
routing.cpp:357:59: error: ‘GetDeliveryIndexPairs’ is not a member of ‘operations_research::RoutingModel’
  357 |     .define_method("delivery_index_pairs", &RoutingModel::GetDeliveryIndexPairs)
      |                                                           ^~~~~~~~~~~~~~~~~~~~~
At global scope:
cc1plus: note: unrecognized command-line option ‘-Wno-shorten-64-to-32’ may have been intended to silence earlier diagnostics
make: *** [Makefile:240: routing.o] Error 1

make failed, exit code 2

Specs

Command: gem install or-tools -- --with-or-tools-dir=/home/dev/or-tools
Ruby: ruby 3.2.2 (2023-03-30 revision e51014f9c0) [aarch64-linux]
OS: Linux 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:23:16 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux
OR Tools: v9.5, installed from their repo

Steps I took to install OR Tools locally:
git clone --branch v9.5 https://github.com/google/or-tools/ --depth 1 \
  cd or-tools \
  make third-party && make cc

Then, inside the Rails project's folder:
bundle config build.or-tools --with-or-tools-dir=/home/dev/or-tools

Another variant of the --with-or-tools-dir produced a slightly different result:
gem install or-tools -- --with-or-tools-dir=/home/dev/or-tools/install_make
current directory: /home/dev/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/or-tools-0.10.1/ext/or-tools
/home/dev/.asdf/installs/ruby/3.2.2/bin/ruby extconf.rb --with-or-tools-dir\=/home/dev/or-tools/install_make
checking for rice/rice.hpp in /home/dev/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/rice-4.1.0/include... yes
checking for -lstdc++... yes
checking for -lortools... yes
creating Makefile

current directory: /home/dev/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/or-tools-0.10.1/ext/or-tools
make DESTDIR\= sitearchdir\=./.gem.20231101-63917-5gwyq7 sitelibdir\=./.gem.20231101-63917-5gwyq7 clean

current directory: /home/dev/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/or-tools-0.10.1/ext/or-tools
make DESTDIR\= sitearchdir\=./.gem.20231101-63917-5gwyq7 sitelibdir\=./.gem.20231101-63917-5gwyq7
compiling assignment.cpp
compiling bin_packing.cpp
compiling constraint.cpp
compiling ext.cpp
ext.cpp: In function ‘void Init_ext()’:
ext.cpp:33:9: error: ‘struct operations_research::CppFlags’ has no member named ‘stderrthreshold’
   33 |   flags.stderrthreshold = 2;
      |         ^~~~~~~~~~~~~~~
At global scope:
cc1plus: note: unrecognized command-line option ‘-Wno-shorten-64-to-32’ may have been intended to silence earlier diagnostics
make: *** [Makefile:240: ext.o] Error 1

make failed, exit code 2

Hi @theomelo, 0.10.1 only works with OR-Tools 9.6.


Add File Deduplication

Tip: You can reduce that size quite a bit by not downloading duplicates. A significant portion of the size is from the larger multi-chapter doujins and a lot of them have individual chapters as well as combination of chapters in addition to the full doujin. When I implemented my offliner I added a duplicate check that groups doujins by the hash of their cover image and only downloads the content of those with the most pages, utilizing redirects for the duplicates. This managed to identify 12.6K duplicates among the 119K I've crawled, reducing the raw size to 1.31TiB of CBZs.

https://www.reddit.com/r/DataHoarder/comments/1fg5yzy/comment/ln2efs3/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

I was told to copy this here:
Hey, I have been using your program since you posted it on reddit. I hope it's ok to request a feature here. The only issue I have had once you helped me figure out how to use nHA is the sheer amount of duplicates. I don't program, so I have no idea how hard it would be to have the program figure out which images are dupes before downloading them, but I have a feeling that would be a huge headache to implement.
I have the idea for what I think would be an easier workaround on your end. If you could make it, so there is a flag(I don't know the correct term) in the .env file so we could choose not to have the images put in a cbz file. Instead have it put each comic into its own subfolder along with the cbz xml file. What this would do is make it easy just to use a 3rd party duplicate file finder to scan the subfolders and delete the dupes.
Sure, you would still have to download the dupes, but it then becomes trivial to find and delete them. Sure it won't delete the xml files for the dupe folders, but once the pics are gone it is as simpler as sorting the folders by size and deleting the tiny ones. Then it would just be figuring out a way to batch making cbz files out of those subfolders after the de-duping.
This is just a thought. I know nothing about programming, so this might be something that isn't easy to do for your program or hell you might not feel like adding this kind of feature. Either way, I want to thank you for releasing it for free for everyone else. You didn't have to, but you made a lot of people's lives a lot easier in doing so.
I also thought of another solution, but I'm not sure if it is possible, do to not knowing how thumbnails are handled.  But if there is a program that can use the cbz's thumbnails and compare them against each other, then allow you to choose the cbz with the largest file size to keep and deleting the rest seems possible in my ignorant, uneducated opinion.

Currently, all images are temporarily downloaded to {LIBRARY_PATH}/{id}, copied to the CBZ, and then cleaned up. Implementing something like CLEANUP_TEMPORARY_IMAGES = false would be trivial.
Speaking for that setting would be that someone else had already requested it for compatibility reasons with another program and ease of implementation.
Speaking against that would be adding more complexity into the ./config/.env for edge cases with not many users and I also don't know yet if I want my users to have to rely on third party software for deduplication.
The problems I see with the hash generation approach is that I don't think that it is reliable enough. What if 2 unrelated hentai have the same cover because they're from the same magazine or maybe they screwed up and both happen to have a blank first page? And how do I implement a redirect from the deleted works to the kept work? I probably would need to implement changing the kept work after it has already been stored in the library and that sounds like a headache.
These are just some thoughts I am having here. Let me know what you think.

When creating the cbz, do you zip with compression or without? If you zip without, you could hash the entire cbz and discard it if it matches another. This would slow things down a bit I think. But it would keep duplication down if someone was just downloading the entire site.

This would reliably work for complete duplicates, but not for series like Sweet Guy which is uploaded countless times with a varying number of chapters. And then there's still the redirection issue from deleted work to kept work to solve. I would be hesitant to change any library entries after they have been moved into the library, that's why if I decide to do this, the process should be well thought out.
I think I am doing bzip2 compression at the moment. Can anybody confirm?
That said, I really appreciate your ideas and active participation @billsargent. Thank you. :)

I was looking at the specs for CBZ and here's the breakdown... CBZ should always be zip only, cb7 is 7zip, cba is ace, cbr is rar, cbt is tar. Your code ls creating zip files using deflate. If you could disable deflate and just use no compression, then theoretically, they should have the same hash.
Yeah as you said, some have different numbers of chapters but overall I think it would save space. The ones with varying numbers of chapters could then be left up  to the user to fix themselves.
See this for how to create zips without compression. I think you are using zipwriter...

oh I can't post links...?
https://docs.rs/zip/latest/zip/write/struct.ZipWriter.html

I think I am repeating myself, but I still don't see a way how to properly redirect someone from a deleted hentai to the corresponding kept hentai. I think this whole deduplication topic is a can of worms I don't want to touch myself, because I don't have the feeling I could offer a solution that lives up to my reliability standards.
I'm currently seriously thinking about adding CLEANUP_TEMPORARY_IMAGES = false setting that would make deduplication for third party software easier.

I think I am repeating myself, but I still don't see a way how to properly redirect someone from a deleted hentai to the corresponding kept hentai. I think this whole deduplication topic is a can of worms I don't want to touch myself, because I don't have the feeling I could offer a solution that lives up to my reliability standards.
I'm currently seriously thinking about adding CLEANUP_TEMPORARY_IMAGES = false setting that would make deduplication for third party software easier.

If you could place comicinfo in there as well, it would also help with creating the cbzs locally as well. That way the metadata is preserved. This would take the burden off you and others who have shell scripting or python scripting capabilities can do the deduplication.

Good point. Then how about calling it CLEANUP_TEMPORARY_FILES instead?

I agree. That sounds perfect.

Deduplication has been decided to be out of scope of this project for now, setting CLEANUP_TEMPORARY_FILES has been added to faciliate easier deduplication via third party tools.


Consumer API: add acknowledge in batch

Is your feature request related to a problem? Please describe.
I want to be able to process messages in batches and in parallel. One of the selling features of Pulsar for me, is being able to acknowledge messages separately which allows me to avoid re-processing messages if one message in a batch failed to be processed. However, I don't want to send acknowledgements for each. I'd like to wait on all tasks, see which ones succeeded and then acknowledge those in one commit.
Describe the solution you'd like
Currently the Consumer Api has: acknowledgeCumulative which works the same way that Kafka's offsets work. but that doesn't work here. I'm suggesting the follwoing method:
CompletableFuture<Void> acknowledgeInBatchAsync(List<MessageId> messageIds)
and the same for nacks.
Describe alternatives you've considered
I think what I want can still be achieved by the following: Let's say I have messages with Ids (1,2,3,4,5) and (2,3,4,5) are processed successfully while (1) isn't. Please confirm if I can achieve what I want with the following:
acknowledgeCumulative(5)
negativeAcknowledge(1)

for some reason, I can't find the method negativeAcknowledge on the consumer api.

@ShahOdin  regarding the api link, it has some issue and @tuteng is fixing this in Issue #5037
you could add the release version manually. e.g. https://pulsar.apache.org/api/client/2.4.2/org/apache/pulsar/client/api/Consumer.html

And you are right, the workaround above is good.
Regarding this feature,  it seems that without pulsar code change, we could also achieve it by application  side code changes?  e.g.  call several acknowledgeAsync, and wait for all the future complete?

we could also achieve it by application side code changes? e.g. call several acknowledgeAsync, and wait for all the future complete?

Well that was the obvious solution which I ruled out as I wanted to avoid making all those calls.

Closed as stale. Please create a new issue if it's still relevant to the maintained versions.
IIRC we have cumulative ack already: https://pulsar.apache.org/docs/2.10.x/concepts-messaging/#acknowledgement


implement presets

include concat and js-bundle plugins

partially implemented (the core plugins are shipped and working)


Fix dependabot merge conflict

This PR fixes the merge conflict of Bump sigs.k8s.io/controller-runtime from 0.18.4 to 0.19.0](https://github.com/ocp-power-automation/rsct-operator/pull/41)

my original question here was what else we need update with k8s dependencies in mod/code because we will be bumping to a newer version, I kicked in the rebase and seems like it create a new PR for that here - https://github.com/ocp-power-automation/rsct-operator/pull/56, if this is enough then we can merge that itself.

Hi @mkumatag, yes these changes are sufficient for new controller-runtime

Hi @mkumatag, yes these changes are sufficient for new controller-runtime

thanks.


how to query ?

downloading the respective CVE and CPE data is interesting, so is parsing them into an sqlite3 database
how can we actually query them in a sensible way ? i assumed through the server but this is just an API frontend

It's been a long time since there have been any updates so it's closed.


clear channel and connection on disconnect

as the title says - if we clear these, we can reconnect the model, which is useful. it will lose all it's subscriptions, but I think that's okay. this is mostly for testing purposes anyway for other projects when they want to use the rabbitmq model
fixes #39

[x] @anandkumarpatel