Load credentials from .NET SDK when running IIS locally

Expected Behavior
When configuring a local development machine with full IIS, and running the AppPool under my user account and "Load User Profile" set to true, I expect the SDK to pick up credentials from the .NET credential store in C:\Users<username>\AppData\Local\AWSToolkit .
Current Behavior
Without a workaround, I get the stack trace in https://stackoverflow.com/questions/29558015/amazon-runtime-amazonserviceexception-unable-to-find-credentials
Possible Solution
Try .NET credential store when only a profile name is supplied in config
Steps to Reproduce (for bugs)

Create an AWS profile in Visual Studio
Create simple web application that references the profile via AWSProfileName in Web.Config or appsettings.json
Test access to AWS resource(s) such as S3 buckets
Install full IIS
Configure web site in IIS
Set app pool up with your own credentials, set the "Load User Profile" option true
Open test website, get stack trace referenced above in current behavior

Context
This would be nice to have so that developers don't need to have unencrypted credentials lying around on their machines and risk disclosure or checking them into source control accidentally.

Could you provide more information?  Which platform are you targeting 4.5 or .net core?  Also, could you share what your web.config looks like?

What version of the AWSSDK.Core nuget package are you using?  I can't see how that stack trace would be produced with the latest version of AWSSDK.Core.  Are you using an older version?  Could you upgrade to the latest?

I'm using .NET 4.5 in a simple MVC project just to prove out the concept of using the VS credential store, just trying to list contents of an S3 bucket.
I confirmed that the IIS app pool is using my identity by removing all other permissions from the .aws/credentials file and hard-coding its path in the Web.config temporarily via the AWSProfilesLocation key.
I also confirmed the IIS Express hosting option does work as expected with only an AWSProfileName key set. In the VS debugger, the credentials associated are type "Amazon.Runtime.AssumeRoleAWSCredentialsOptions".
Relevant portion of Web.config:

Upgraded to <IP_ADDRESS> for AWSSDK.Core, might have been behind, but the current stack trace is only mildly changed:
Exception Details: Amazon.Runtime.AmazonServiceException: Unable to find credentials
Exception 1 of 4:
System.InvalidOperationException: The app.config/web.config files for the application did not contain credential information
at Amazon.Runtime.AppConfigAWSCredentials..ctor() in E:\JenkinsWorkspaces\v3-trebuchet-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Credentials_bcl\AppConfigAWSCredentials.cs:line 69
at Amazon.Runtime.FallbackCredentialsFactory.<>c.b__8_0() in E:\JenkinsWorkspaces\v3-trebuchet-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Credentials\FallbackCredentialsFactory.cs:line 48
at Amazon.Runtime.FallbackCredentialsFactory.GetCredentials(Boolean fallbackToAnonymous) in E:\JenkinsWorkspaces\v3-trebuchet-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Credentials\FallbackCredentialsFactory.cs:line 113
Exception 2 of 4:
Amazon.Runtime.AmazonClientException: Unable to find a default profile in CredentialProfileStoreChain.
at Amazon.Runtime.FallbackCredentialsFactory.GetAWSCredentials(ICredentialProfileSource source, String defaultProfileName) in E:\JenkinsWorkspaces\v3-trebuchet-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Credentials\FallbackCredentialsFactory.cs:line 69
at Amazon.Runtime.FallbackCredentialsFactory.GetCredentials(Boolean fallbackToAnonymous) in E:\JenkinsWorkspaces\v3-trebuchet-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Credentials\FallbackCredentialsFactory.cs:line 113
Exception 3 of 4:
System.InvalidOperationException: The environment variables AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY/AWS_SESSION_TOKEN were not set with AWS credentials.
at Amazon.Runtime.EnvironmentVariablesAWSCredentials.FetchCredentials() in E:\JenkinsWorkspaces\v3-trebuchet-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Credentials_bcl+coreclr\EnvironmentVariablesAWSCredentials.cs:line 80
at Amazon.Runtime.FallbackCredentialsFactory.<>c.b__8_2() in E:\JenkinsWorkspaces\v3-trebuchet-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Credentials\FallbackCredentialsFactory.cs:line 53
at Amazon.Runtime.FallbackCredentialsFactory.GetCredentials(Boolean fallbackToAnonymous) in E:\JenkinsWorkspaces\v3-trebuchet-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Credentials\FallbackCredentialsFactory.cs:line 113
Exception 4 of 4:
Amazon.Runtime.AmazonServiceException: Unable to reach credentials server
at Amazon.Runtime.URIBasedRefreshingCredentialHelper.GetContents(Uri uri) in E:\JenkinsWorkspaces\v3-trebuchet-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Credentials\URIBasedRefreshingCredentialHelper.cs:line 36
at Amazon.Runtime.InstanceProfileAWSCredentials.d__10.MoveNext() in E:\JenkinsWorkspaces\v3-trebuchet-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Credentials\InstanceProfileAWSCredentials.cs:line 0
at Amazon.Runtime.InstanceProfileAWSCredentials.GetFirstRole() in E:\JenkinsWorkspaces\v3-trebuchet-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Credentials\InstanceProfileAWSCredentials.cs:line 204
at Amazon.Runtime.FallbackCredentialsFactory.ECSEC2CredentialsWrapper() in E:\JenkinsWorkspaces\v3-trebuchet-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Credentials\FallbackCredentialsFactory.cs:line 92
at Amazon.Runtime.FallbackCredentialsFactory.GetCredentials(Boolean fallbackToAnonymous) in E:\JenkinsWorkspaces\v3-trebuchet-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Credentials\FallbackCredentialsFactory.cs:line 113

It's possible that there's an error in the credentials file or, of course, maybe there's some other bug.
-Did you write the credentials file programmatically, or with Visual Studio?
-Are you intentionally using assume role credentials?
-If you're creating assume role credentials please send the code that you used to do that.
Try to run this code first thing in your application.  This should give us more information.
CredentialProfile profile;
var netSdkCredsFile = new NetSDKCredentialsFile();
if (netSdkCredsFile.TryGetProfile("NAMED_PROFILE_IN_.NET_Credential_Store", out profile))
    AWSCredentialsFactory.GetAWSCredentials(profile, netSdkCredsFile, true)
else
    throw new Exception("Not finding profile");

In the meantime I'll improve the logging in the FallbackCredentialsFactory to make these things easier to diagnose in the future.

It's not a credentials file per se, it's the .NET SDK credential store that I'm trying to use.
Yes, intentionally using an assume role role profile, but it works fine outside of IIS context.
Running in IIS locally, your code throws the exception. Running in IIS Express, it successfully retrieves the profile. The IIS AppPool is set up under my account, and is configured to load the user profile (in case that matters at all). Anonymous auth to the site uses the app pool identity.
The code below is a workaround for VS 2017 AWS toolkit not yet being able to create assumed role profiles directly as previous versions were.
            var netSDKFile = new NetSDKCredentialsFile();

            // If we haven't already created an assume role profile, do so:
            if (!netSDKFile.TryGetProfile(assumeRoleProfileName, out assumeRoleProfile)) {
                CredentialProfile basicProfile;
                // Update the profile with the roleARN (limitation of current VS toolkit, in AWS we'd use instance role)
                if (netSDKFile.TryGetProfile(profileName, out basicProfile)) {
                    CredentialProfileOptions opts = new CredentialProfileOptions
                    {
                        RoleArn = roleARN,
                        SourceProfile = profileName
                    };
                    assumeRoleProfile = new CredentialProfile(assumeRoleProfileName, opts);
                    assumeRoleProfile.Region = RegionEndpoint.USWest2;
                    netSDKFile.RegisterProfile(assumeRoleProfile);
                    AWSCredentials creds;
                    AWSCredentialsFactory.TryGetAWSCredentials(basicProfile, netSDKFile, out creds);
                }
            }

What's the exception thrown by the code I sent when you run it in IIS locally?
The FallbackCredentialsFactory may mask the real problem.  So it will help to see the exception, if any, from the snippet I sent.

Your code does not throw an exception, the TryGetProfile() call just returns false & doesn't populate the profile out param.
The exception I included earlier happens on S3 client instance creation:
IAmazonS3 s3Client = new AmazonS3Client();

You wouldn't use the UserCrypto class directly under normal circumstances, but try running
Amazon.Runtime.Internal.Settings.UserCrypto.Encrypt("foo");
and see if that throws an exception.

var x = Amazon.Runtime.Internal.Settings.UserCrypto.Encrypt("foo");
That succeeds, or at least doesn’t throw an exception and x has a nice long hexadecimal string afterward.
The encrypted text is different when running in debug mode against local IIS than IIS Express, but both strings still decrypt to the original “foo” under either hosting approach.

Hm.  That's a good thing.  But not what I expected.
Here are a couple things to try:

Try to read in a basic profile and create AWSCredentials from it.  That may point to an issue with assume role profiles.
Back up your .NET Credentials file, delete it.  In your code write the basic and assume role profiles.  Then read the assume role profile and try to create AWSCredentials from it immediately after.  Try this in local IIS first; make sure the file is both written and read with local IIS, and not mixed with code running in IIS Express.
It looks like you're testing out the basic profile with this:

AWSCredentials creds;
AWSCredentialsFactory.TryGetAWSCredentials(basicProfile, netSDKFile, out creds);

But try running that for assume role profile too.  Also verify it returns true, and creds is populated.

Closing due to inactivity.

hi there, @mr-olson could you please reach me out on<EMAIL_ADDRESS>in case you were able to get it resolved for you

Here is the answer
https://github.com/aws/aws-sdk-net/issues/1139


Make properties on S3AsyncClient configurable

IngestRecordsUsingPropertiesSpecifiedMethod calls S3AsyncClient.create(). We should allow properties to be set on this client, specifically to control the chunk size used for uploads. It probably defaults to 8MB, which is too small. Suggest default value of 128MB instead.

There seem to be a bunch of places the client can be created. In order to name and document a property for this, I'd want to understand the semantics of why we'd care about this specific call to S3AsyncClient.create.
IngestRecordsUsingPropertiesSpecifiedMethod contains two functions, both of which can have an S3AsyncClient passed into them, and only one of which will create it if it's not set. I could spend some time looking into why the client would or would not be set when calling these functions.
As it is, I'm not sure if there's a clear use case where the client won't be set when calling that one function, or if we'd rather set properties on the client in a wider range of cases.

It's a lot cleared how the S3AsyncClient is used now, but I'm not sure there's a chunk size. For the CRT-based version there is, but we're not using that right now:
https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/crt-based-s3-client.html
That does get used by default by the transfer manager if you don't specify a client and CRT is on the classpath, so after issue #437 that will be used in any ingests that don't specify a client and don't use the ingest jobs queue.


Feature/new datasets

These are some helpful changes and/or additions that were necessary as I tested out the anemoi-datasets module on a few different datasets. Please let me know if you have any questions! Thanks!
Bug Fix

src/anemoi/datasets/create/functions/sources/xarray/variable.py
Bug fix - the arg in the class Variable is "variable" not "var". This caused a failure for me when working with levels until fixed.

Additions I found helpful

src/anemoi/datasets/create/init.py
Add the ability to work with cftime when needed.

src/anemoi/datasets/create/functions/sources/xarray/fieldlist.py
This is an update that was helpful when working with a curvilinear grid (but will not always be necessary - it depends on the dataset). It came after I noticed that some dimensions are not always set as coordinates/variables in the dataset. For example. a dataset with a curvilinear grid could have lat/lon as coordinates, but the relevant x and y index dimensions are not set as coordinates or variables. Therefore, this is a check to make sure that if dimension in a dataset was identified in a "flavour" that it is also set as a coordinate within the dataset.

Ability to read from Microsoft's Planetary Computer
There are a number of super useful datasets on planetary computer, and these updates make it possible to read most zarr stores from there. The URL to use in a yaml file is the STAC collection URL associated with the dataset. I chose to set it up this way because this url appeared to be the most consistent way to reference zarr datasets within planetary computer.

src/anemoi/datasets/create/functions/sources/xarray/init.py
Elif statement specifically for planetary computer. There are a few extra hoops to jump through for open_zarr to work.

src/anemoi/datasets/data/stores.py
Add class PlanetaryComputerStore() that mimics the already existing S3 store and HTTPS store.

Unit test addition

tests/xarray/test_zarr.py
I added two unit tests for additional datasets. The updates within this PR were able to make anemoi-datasets work with both of these datasets.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Thanks for the great contribution!


Wrongly indented object argument of method after add chained method

Description
As example below.
Input
The code looked like this before beautification:
a({
  a: 1
});

a({
  a: 1
})
  .then();

Expected Output
The code should have looked like this after beautification:
a({
  a: 1
});

a({
  a: 1
})
  .then();

Actual Output
The  code actually looked like this after beautification:
a({
  a: 1
});

a({
    a: 1 // should be 2 spaces
  }) // should be 0 space
  .then();

Steps to Reproduce

A function call;
A function call with an object argument;
Add a chained function call;

Environment
https://beautifier.io/ on Chrome 74 of macOS 10.14.4
Settings
{
  "indent_size": "2",
  "indent_char": " ",
  "max_preserve_newlines": "5",
  "preserve_newlines": true,
  "keep_array_indentation": true,
  "break_chained_methods": false,
  "indent_scripts": "keep",
  "brace_style": "collapse",
  "space_before_conditional": true,
  "unescape_strings": false,
  "jslint_happy": false,
  "end_with_newline": false,
  "wrap_line_length": "0",
  "indent_inner_html": false,
  "comma_first": false,
  "e4x": false,
  "indent_empty_lines": false,
  "unindent_chained_methods": false
}

Duplicate of #1651.
Please feel free to weigh in or help out there.


Remove children as a function

Hey Andrey, I love this component!
I was wondering if there is any specific reason why children is a function. IMHO the state could be lifted up since anyway we get to register onFocus and onBlur.  A minimal API like this https://codesandbox.io/s/vq5vzmxxrl would work as a baseline to build more complex ones.
FWIW it is probably better to avoid preventDefault/stopPropagation since other components (ancestors) might rely on those events.

I mean it doesn't hurt anybody and it is a built-in commodity so this is more like a generic question: do you tend to include both options when you build components?

Hey @giuseppeg! Thanks for asking and looking at the component :)

There are more to children being a function. First of all, ref handling. Since I want to make usages with CSS-in-JS to be as simple as possible, I aim exposing getRef function via arguments of the children function. See "Using with CSS-in-JS libs" here https://react-simple-focus-within.netlify.com
I have found both usages with React children and functions useful. I think that elevating the state in the user land is a bit cumbersome. To me, this component is the right abstraction for that that's why I think it should encapsulate the state. That being said, I noticed that sometimes it's still useful to be able to react on those state changes in a different way. That's why I have onFocus and onBlur callbacks.
I'll look into preventDefault/stopPropagation. Thanks!

oh yep forwarding refs is a good reason to do so already :) Thanks for taking the time to answer ;)


Loading Page

Adds:

a loading page later to be shown before the app renders
a test that checks if loading page renders

Notes:
The tests won't pass after adding background image to the page. This requires configuration for jest to understand svg-files since it doesn't go through webpack when testing. I couldn't make it work yet so someone else has to do it later.

There is a new pull request https://github.com/sos-lapsikyla/ylitse-web/pull/28 which replaces this one.


[Bug]Enhancement : Add an Edit icon in the header next to the application name indicating its editable

Steps to reproduce:

Navigate to home page
Click on the edit option of the application on hover
Observe the header for application name

Observation :
It is observed that application name does not seem to be editable only a known user can understand that the application name can be edited at the header.
Expectation
Adding an edit icon close to the application name makes user understand that the application name can be edited inside the application

@prapullac thank you for the enhancement request. We've intentionally taken a call not to add this edit icon today. If users still find it difficult, we'll consider bringing it back


How to run controller and worker outside of example "scope"

HI ,
We are doing a spike on uReplicator and I am trying to run controllers and workers. I have some questions on those and I did not see any other medium other than this to post questions.

What are the minimum set of arguments to get the controller and worker running?
I see arguments like srcKafkaZKPath and destKafkaZK path etc. Do we need to pas these explicitly or does it read the configurations from /config/consumer.properties and /config/producer.properties?
I see there are some Graphana related configs. Does it make use of Graphite metrics?
Could you give me some insight on actual production runtime infrastructure of Controller and Workers , their corresponding configurations etc?

Have you find any way of achieving this??

Yes. Thanks a lot for reaching out. There was issues in the class path settings in the script files generated by the build..
We are trying to use uReplicator in production for our Kafka replication, could you please share some thoughts on how the deployment infrastructure should look like. In terms of number of Controllers and workers?
Thanks & Regards
Sankaran
From: ravi45722 <notifications<EMAIL_ADDRESS>Reply-To: uber/uReplicator <reply<EMAIL_ADDRESS>Date: Wednesday, December 28, 2016 at 1:21 AM
To: uber/uReplicator <uReplicator<EMAIL_ADDRESS>Cc: Thomson Reuters <Sankaran.Bhattathiripad<EMAIL_ADDRESS>Author <author<EMAIL_ADDRESS>Subject: Re: [uber/uReplicator] How to run controller and worker outside of example "scope" (#8)
Have you find any way of achieving this??
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_uber_uReplicator_issues_8-23issuecomment-2D269437417&d=CwMCaQ&c=4ZIZThykDLcoWk-GVjSLm9hvvvzvGv0FLoWSRuCSs5Q&r=4WDAN6LSxgcos0UkbR6ng77O7Ac9Q1Cui0ZdrgsTwz3kl-idQg_F3fogaBeqwzQX&m=f4nmY3X2yug4A2DtLtYMC5jAni4ftj6AdcluH9barlI&s=5lhtDpadt9eibBw5yPV0W9Md-alZlhTNWZc2dMSZALw&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AVMG6In9XLFx8m-5FE9gjgc5G3TYEvHabSks5rMg4IgaJpZM4KjvEH&d=CwMCaQ&c=4ZIZThykDLcoWk-GVjSLm9hvvvzvGv0FLoWSRuCSs5Q&r=4WDAN6LSxgcos0UkbR6ng77O7Ac9Q1Cui0ZdrgsTwz3kl-idQg_F3fogaBeqwzQX&m=f4nmY3X2yug4A2DtLtYMC5jAni4ftj6AdcluH9barlI&s=5Rcf6QNgAW6tpq5Qsf_Jv41TVqmCNJTPZ4OD1hM9ois&e=.

Thank you for the reply.
We just completed our POC with uReplicator and beginning to think about the production deployment infrastructure. We are replicating the Kafka messages across DC’s . We had thought about both Local read- Remote write  (which is what basically you mentioned below ;- being closer to source DC) and Remote read- Local write.
Is there any reason why the worker nodes to be closer to the Source Data center?
Also wondering, do you have any replication audit/monitoring tools in place to make sure data is always in sync between data centers?
Thanks for your help.
Thanks & Regards
Sankaran
From: icefury71 <notifications<EMAIL_ADDRESS>Reply-To: uber/uReplicator <reply<EMAIL_ADDRESS>Date: Wednesday, December 28, 2016 at 2:07 PM
To: uber/uReplicator <uReplicator<EMAIL_ADDRESS>Cc: Thomson Reuters <Sankaran.Bhattathiripad<EMAIL_ADDRESS>Author <author<EMAIL_ADDRESS>Subject: Re: [uber/uReplicator] How to run controller and worker outside of example "scope" (#8)
We usually run 3 Controller nodes and 10-15 worker nodes. The worker nodes should ideally be closer to the source cluster DC (if you're replicating across DCs).
Is there something specific that you're looking for ?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_uber_uReplicator_issues_8-23issuecomment-2D269533090&d=CwMCaQ&c=4ZIZThykDLcoWk-GVjSLm9hvvvzvGv0FLoWSRuCSs5Q&r=4WDAN6LSxgcos0UkbR6ng77O7Ac9Q1Cui0ZdrgsTwz3kl-idQg_F3fogaBeqwzQX&m=tK1z00T44gxHSsKZqm3lxknb2lEUCX2bRnieIfXBWmI&s=al7e9vjjuIRpwJgOnL5QaDJ2zeRBRV_ANvCRG2iO1bI&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AVMG6Pwnl7lzRE0ZnlKEIy0gLnDysjzpks5rMsF0gaJpZM4KjvEH&d=CwMCaQ&c=4ZIZThykDLcoWk-GVjSLm9hvvvzvGv0FLoWSRuCSs5Q&r=4WDAN6LSxgcos0UkbR6ng77O7Ac9Q1Cui0ZdrgsTwz3kl-idQg_F3fogaBeqwzQX&m=tK1z00T44gxHSsKZqm3lxknb2lEUCX2bRnieIfXBWmI&s=XMjzGjgOLnnK2JRIdUDdLAHCjb93kA8vmHwBQ_plVX4&e=.

For auditing uber implemented chaperone for end to end auditing

@sankaranbpv : We are facing issues with UReplicator.
We are trying to get our POC Setup up. Our Source Cluster is in one server, and the target cluster along with UReplicator in another server. We are unable to replicate the messages across the cluster due to the below errors.  Have faced similar issues ?? Let us know about your configurations so we could try to complete our POC with this. Thanks !!
[2016-12-29 14:22:44,523] WARN [CompactConsumerFetcherThread-kloak-mirrormaker-test_kloakmms01-sjc1-0-3], Error in fetch Name: FetchRequest; Version: 1; CorrelationId: 240; ClientId: kloak-mirrormaker-test; ReplicaId: -1; MaxWait: 100 ms; MinBytes: 1 bytes; RequestInfo: [test2,0] -> PartitionFetchInfo(0,8388608). Possible cause: # java.nio.BufferUnderflowException (kafka.mirrormaker.CompactConsumerFetcherThread:83)
In the Controller Part: we are getting:
[2017-01-02 12:50:19,612] ERROR For topic:test2, number of partitions for IdealState: test2, {IDEAL_STATE_MODE=CUSTOMIZED, MAX_PARTITIONS_PER_INSTANCE=1, NUM_PARTITIONS=1, REBALANCE_MODE=CUSTOMIZED, REPLICAS=1, STATE_MODEL_DEF_REF=OnlineOffline, STATE_MODEL_FACTORY_NAME=DEFAULT}{0={testHelixMirrorMaker01=ONLINE}}{} # doesn't match ExternalView: test2, {BUCKET_SIZE=0, IDEAL_STATE_MODE=CUSTOMIZED, MAX_PARTITIONS_PER_INSTANCE=1, NUM_PARTITIONS=1, REBALANCE_MODE=CUSTOMIZED, REPLICAS=1, STATE_MODEL_DEF_REF=OnlineOffline, STATE_MODEL_FACTORY_NAME=DEFAULT}{}{} (com.uber.stream.kafka.mirrormaker.controller.validation.ValidationManager:152)

Very poor documentation.
@sankaranbpv : Could you please explain how to use our own consumer and producer properties ?

When you installed this uReplicator you will get kafka version 0.9 under /rooot/... (Search with . directories). In that you can change your kafka consumer producer properties.

Seriously Pathetic documentation, still not able to figure out what parameters need to be passed to start-controller.sh .

@ltnilaysahu Sorry about that. We will improve the documentation.
Here is a start command sample:
RUN_COMMAND="java $JAVA_OPTS -cp $CONTROLLER_CLASSPATH com.uber.ureplicator2.controller.ControllerStarter
com.uber.stream.kafka.mirrormaker.starter.MirrorMakerStarter
-helixClusterName $HELIX_CLUSTER_NAME #e.g. uReplicator
-mode $CONTROLLER_MODE #e.g. customized
-zookeeper $ZOOKEEPER_CONNECT #your helix zookeeper path
-port $CONTROLLER_PORT #your rest entry port
-env $ENVIRONMENT #your deployment/env or anything, it's for metrics
-instanceId $CLIENT_ID #e.g. your controller host name
-graphiteHost $METRICS_REPORTER_GRAPHITE_HOST #e.g. <IP_ADDRESS>
-graphitePort $METRICS_REPORTER_GRAPHITE_PORT #e.g. 4756
-enableAutoWhitelist $ENABLE_AUTO_WHITELIST #e.g. false
-enableAutoTopicExpansion $ENABLE_AUTO_TOPIC_EXPANSION #e.g. true
-patternToExcludeTopics $PATTERN_TO_EXCLUDE_TOPICS #e.g. ^__.*
-enableSrcKafkaValidation $ENABLE_SRC_KAFKA_VALIDATION #e.g. true
-srcKafkaZkPath $SRC_KAFKA_ZK #your src kafka cluster zk
-destKafkaZkPath $DEST_KAFKA_ZK #your dst kafka cluster zk
-autoRebalanceDelayInSeconds $AUTO_REBALANCE_DELAY_IN_SECONDS #e.g. 120
-refreshTimeInSeconds $REFRESH_INTERVAL_IN_SEC #e.g. 300
-backUpToGit $BACKUP_TO_GIT #e.g. false
-localBackupFilePath $LOCAL_FILE_BACKUP_PATH #e.g. /var/log/ureplicator-controller
-remoteBackupRepo $REMOTE_BACKUP_REPO #e.g. your remote repo
-localGitRepoClonePath ${LOGS_DIR}/$LOCAL_GIT_REPO_CLONE_PATH #e.g. /ureplicator-controller-bkp"

Hiya,
I'm also evaluating use of uReplicator over at the Wikimedia Foundation.  The docs are pretty lacking, and the source isn't particularly clear either.  Should I be using properties files or CLI opts?  Should the opts look like properties (e.g. with '.' in them like controller.helix.cluster.name) or like cli opts (e.g. --helixClusterName or maybe -helixClusterName).  It looks like MirrorMakerWorkerConf and ControllerConf use different CLI/properties parsers (joptsimple vs commons CLI/PropertiesConfiguration).
I've just tried working through all this a little bit, but don't fully understand some of the configuration options.

Does uReplicator support ZK chroots?  e.g. -srcKafkaZkPath zk1-1.analytics.eqiad.wmflabs/kafka/main-analytics.  I get NoNode for /brokers/topics when I start the controller.

Why does the controller need to explicitly configured with source and dest zookeeper paths, if that information is available in consumer.properties and producer.properties via Kafka?

I set -helixClusterName in a controller and -zookeeper zk.host.name/uReplicator0, and then matched helixClusterName in helix.properties for the worker.  I get Cluster structure is not set up for cluster: uReplicator-main-jumbo0 when I start my worker.

Thanks for this software though!  I hope we can use this! MirrorMaker is being especially flaky for us recently, and I would much rather use an open source option than look into Confluent's closed source Replicator.

@ottomata sorry for the bad documentation. We are improving the documentation.
Meanwhile, to solve your question:

The controller use cli opts, please refer to my answer above.
uReplicator supports ZK chroots, what is your node structure under zk1-1.analytics.eqiad.wmflabs/kafka/main-analytics?
Do you mean consumer.properties and producer.properties in worker configs?
Have you manually create zk.host.name/uReplicator0 before you start controller?

Ah ok, thanks.  Doing 4. helped.  Might be nice if uReplicator could auto create the helix zk path. :)  I created the znode with zkCli.sh like create /uReplicator-main-jumbo0 "".
My controller and worker are running now, but so far haven't had success getting anything to replicate.  I started the controller with -enableAutoWhitelist true, but I still kept getting kafka.consumer.ConsumerTimeoutException in the worker.  (I have created the topics with the same # of partitions in both by source and dest clusters.)
I then tried curl -X POST -d '{"topic":"date0", "numPartitions":"4"}' http://localhost:9000/topics to the controller, and saw Added fetcher for partitions ArrayBuffer([[date0,2] ... in the worker, but then got
INFO Reconnect due to error: (kafka.consumer.SimpleConsumer:78)
java.nio.channels.ClosedChannelException
	at kafka.network.BlockingChannel.send(BlockingChannel.scala:110)
	at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:85)
	at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:83)
...

Not much else came out.  I can see the worker being reassigned some partitions periodically, but it immediately dies.
My source Kafka cluster is <IP_ADDRESS>, and my dest Kafka cluster is 1.0.0.
Controller command:
./uReplicator-Distribution/target/uReplicator-Distribution-pkg/bin/start-controller.sh \
-port 9000 \
-helixClusterName uReplicator-main-jumbo0 \
-zookeeper zk1-1.analytics.eqiad.wmflabs/uReplicator-main-jumbo0 \
-enableAutoWhitelist true \
-srcKafkaZkPath zk1-1.analytics.eqiad.wmflabs/kafka/main-analytics \
-destKafkaZkPath zk1-1.analytics.eqiad.wmflabs/kafka/jumbo-analytics

Worker command:
./uReplicator-Distribution/target/uReplicator-Distribution-pkg/bin/start-worker.sh \
--consumer.config ./config/consumer.properties \
--producer.config ./config/producer.properties \
--helix.config ./config/helix.properties

Oh, and one more Q: :)

The controller use cli opts, please refer to my answer above.
Why do MirrorMakerWorkerConf and ControllerConf classes define properties with dots in the name then?  e.g. controller.helix.cluster.name?  It'd be super nice if we could eventually configure these processes with a single or few .properties config files, rather than having to write a custom wrapper script with configured CLI opts.  Just sayin!

(BTW, thanks for the quick response before :D )

uReplicator consumer is using <IP_ADDRESS>, which is not able to consume from <IP_ADDRESS> broker (https://kafka.apache.org/0102/documentation.html#upgrade_10_2_0). You might want to upgrade your source broker version first.
The difference between controller/consumer configuration format is due to historical reason. While we are developing Federated uReplicator, we are considering to support both format in controller to avoid the confusion. However, there is no timeline for this feature for now :(

Ooof, ok.  Right.  Yeah, this is why we are looking into uReplicator; we can't use 1.0 MirrorMaker with our 0.9 cluster, and the 0.9 MirrorMaker isn't doing so well when producing to your 1.0 cluster either.  We plan to upgrade the 0.9 cluster too, but it might be a while.  Eef.
Thanks!

Another solution is to use <IP_ADDRESS> kafka-client in your fork before broker upgrade is done, which is relatively easy. You can refer to #47

Sorry to thread-hijack, but would uReplicator work well with <IP_ADDRESS> on both clusters?

It should be. <IP_ADDRESS> kafka-client should work well with <IP_ADDRESS> brokers.

What are the dependencies on src/dest zk? Is there a way this can be avoided since 0.11+ Kafka do support admin client api's as well as consumers without requiring them to directly connect to zk. The reason we are looking at to avoid zk connection is security and limit access as much as possible. In addition, since this is multi-dc replication, exposing zk over plaintext will not be a good idea.

The src zk is for finding all brokers and committing offsets. For the first one, it is easy to change to use broker list directly. For the second one, we need to change the commit part to use broker instead of zookeeper. If it is urgent for you, you can welcome to submit pull request and we will review it for you.

@xhl1988 Thanks! I was under the impression that the offsets are managed externally in the same zookeeper that Helix relies on. Additionally, for the producer (destination) why should there be a need to commit the offset?

@paphillon Currently offset is still managed in zookeeper, not in the helix zookeeper, but the source kafka cluster zookeeper, under the consumers node.
Regarding producer, it doesn't commit offset, the worker commits produced messages in source cluster. So if a worker crashed, the new worker will start replicating from last committed offset so there will be no data loss (duplication through).

@xhl1988 - Thank you! Yes, that makes sense and going through the code this is what it's doing since it's using the old consumer. Here is my plan that I am working on, create the consumer / producer for replication using Spring Kafka framework using the new consumer pattern and use groups to commit the offsets. However, the partitions and topics will still be manually assigned. Each topic, partition corresponds to a KafkaMessageListenerContainer bean with a batch listener and maintains a registry of these containers with topic-partition as the key, so really don't need to worry much about threads. These containers will push messages to one producer instance per node and also allow for grouping of topics under a business unit and support sync/async in a mix mode. So far the POC looks promising :).

@paphillon you plan sounds great!! We also investigated the new consumer earlier since it natively provided security support. How do you handle worker failure or the framework handles that for you? Also if one KafkaMessageListenerContainer is assigned a large number of partitions (like 200+/300+), will there be any performance impact like long registry time?

@xhl1988 Each KafkaMessageListenerContainer will handle only one Topic one Partition, this is a single threaded container, so with the help of Helix, all partitions will be spread across a number of Nodes (Participants). In our case, there are at max 5-10 partitions, but I don't think scaling should be a problem. In addition, the Helix controller and workers will be deployed on Kubernetes, so if a worker crashes, Kubernetes will bring the worker back up on some another node. Kuberenetes will manage the nodes/workers and Helix will maintain the tasks within the worker pool.

I have cloned the master repo from the controller and worker .if we are using kafka version 2.11. My source cluster is on-prem and target cluster is AWS server. I want to use uReplicator instead confluent replication . Can I use below commands
./uReplicator-Distribution/target/uReplicator-Distribution-pkg/bin/start-controller.sh
-port 9000
-helixClusterName xxxxxxxxxxxxxxxxxxxxx
-zookeeper xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
-enableAutoWhitelist true
-srcKafkaZkPath xxxxxxxxxxxx
-destKafkaZkPath xxxxxxxxxxxxxxxxxxxxxxxxxxxx
./uReplicator-Distribution/target/uReplicator-Distribution-pkg/bin/start-worker.sh
--consumer.config ./config/consumer.properties
--producer.config ./config/producer.properties
--helix.config ./config/helix.properties

Can anyone help ? please let me know how to figure out the src kafka path and destkafka path?

@xhl1988 and @paphillon or @ottomata --anyone let me know if it is possible with Cdoe for the POC?
I really appreciate your time and efforts!!

@Gk8143197049  I haven't actually used uReplicator; we were looking into it as a MirrorMaker replacement but decided not to use it.  Good luck!

@ottomata I wanted to know how to configure my environment and complete my POC. can you let me details on configuring the parameters, if possible? What are you looking for replication in the open source? anything else available ? I only heard about the Mirus..

We are using the built in Kafka MirrorMaker
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330.
We one day hope to switch to MirrorMaker 2.0 once it is accepted and
available.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0
On Tue, Mar 12, 2019 at 4:22 PM Girish Kumar Pinumalla <
<EMAIL_ADDRESS>wrote:

@ottomata https://github.com/ottomata I wanted to know how to configure
my environment and complete my POC. can you let me details on configuring
the parameters, if possible? What are you looking for replication in the
open source? anything else available ? I only heard about the Mirus..
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/uber/uReplicator/issues/8#issuecomment-472165673, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABjPxi4CpqPiehpWDa5asU7q0Qi9WltYks5vWAyZgaJpZM4KjvEH
.

Are there any lags in mirror maker??

Not in normal operations, it scales just like any Kafka client.
On Tue, Mar 12, 2019 at 8:34 PM Girish Kumar Pinumalla <
<EMAIL_ADDRESS>wrote:

Are there any lags in mirror maker??
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/uber/uReplicator/issues/8#issuecomment-472233751, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABjPxqnkwDsYsWIZruEK5TtOBFAGBTo1ks5vWEd-gaJpZM4KjvEH
.

@ottomata . is the code available for kafka mirror maker 2.0 open source and if yes, please share me the link. I will clone and try doing POC.
Thanks

It is in development.  You can track the progress here:
https://issues.apache.org/jira/browse/KAFKA-7500.  I assume you could log
in there and ask, or join the kafka-dev mailing list and ask there.
On Wed, Mar 27, 2019 at 11:20 AM Girish Kumar Pinumalla <
<EMAIL_ADDRESS>wrote:

@ottomata https://github.com/ottomata . is the code available for kafka
mirror maker 2.0 open source and if yes, please share me the link. I will
clone and try doing POC.
Thanks
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/uber/uReplicator/issues/8#issuecomment-477204996, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABjPxipqlfDFxssQuW-G-1ulK0J36eYbks5va4wugaJpZM4KjvEH
.


dokku_config: allow list type

TASK [Configure the environment] *************************************************************************************************************************************************************************
fatal: [gitea]: FAILED! => {"changed": false, "meta": {"error": "All values must be keys, found invalid types for WHITELISTED_URIS", "present": false}, "msg": "All values must be keys, found invalid types for WHITELISTED_URIS"}

https://docs.gitea.io/en-us/config-cheat-sheet/

WHITELISTED_URIS: : If non-empty, list of POSIX regex patterns matching OpenID URI’s to permit.

So, I need to add the following config:
    - name: config env
      dokku_config:
        app: gitea
        config:
          WHITELISTED_URIS: ["foo.com"]

Shall we widen up the types that are accepted?

Actually, also "Foo: true" fails as well, which is pretty typical in an Ansible play. Looks like https://github.com/dokku/ansible-dokku/blob/master/library/dokku_config.py#L93 is the issue here. I think a bool should be easy to allow also.

The error should say All values must be strings.

Description here as to my reasoning.

Hmmm, Ansible does the following regarding trying to convert bools:
def to_bool(a):
    ''' return a bool for the arg '''
    if a is None or isinstance(a, bool):
        return a
    if isinstance(a, string_types):
        a = a.lower()
    if a in ('yes', 'on', '1', 'true', 1):
        return True
    return False

So, we could do that to address you concern in https://github.com/dokku/ansible-dokku/issues/6#issuecomment-484308360. We can extend the "in" check to cover other values that you might use / expect for truthiness.
I understand why it works the way it works now but it unfortunately breaks a very common expectation when writing Ansible plays that you can use integers, bools and not quote booleans. That is confusing coming from a strictly Ansible side of things.

The problem here is that folks won't be able to specify the difference between True (string) and True (bool) values, so trying to set those as values may result in issues at the application level. For instance, you might set true expecting that as the literal string, but then it gets set as the literal True string when we call config:set. The underlying Dokku command does not distinguish between value types, so attempting to support custom value types doesn't make sense insofar as I don't want to publish rules on how we serialize things.
The rule that you can use different types within Ansible only applies to when the input expects that. We do not, and the module expects a string value. Trying to coerce the value here doesn't make sense at all since you're attempting to set a literal value in your app's environment.

Makes sense! I'll get a docs patch in to avoid this being raised up again :sun_behind_large_cloud:


errors aren't Error objects

For most (all?) operations that potentially pass errors to callbacks or reject promises, the error objects are plain objects instead of actual Error objects. This is inconsistent with the node ecosystem and creates some issues when trying to integrate with services like Sentry. It would be nice if these were actual Error objects, or a subclass of Error specific to GetStream.

Thanks for pointing this out!
All the API operation functions involve the wrapPromise() function which deconstructs the response (or error) of the HTTP request.
The fix I have in mind is to change this function to reject / callback with an error that is new custom Error class. The custom error will still expose the 'error' and 'response' properties in order to minimise the chance of breaking existing implementations.
Here's what I've got so far, and feedback appreciated.
https://github.com/dwightgunning/stream-js/commit/3ffbb4d02456c95cad4a08bfe3e253dd7a3ebe66
If that looks like a reasonable approach we can look to get this out shortly in a new release.

Hey @dwightgunning, thanks for the reply! I left a comment in the PR of one small issue. This feels like the right fix to me - although I'm biased because it's eerily similar to my workaround.

Thanks for the review and picking that up! Since this will be a potential break for some people, we'll do a major release. That may make it a good time to include some other changes we've had in the wings for a while. I will come back to you shortly with guidance on when we're likely to get this packaged and released.


Same blank timepicker-bs4 and datepicker-bs4 issue with multiple rows

Hi Lesilent,
First, thanks for fixing the minScreenWidth in both datepicker-bs4 and timepicker-bs4. Both work great now.
But I've noticed a strange behavior with the timepicker-bs4 when there are multiple rows present. For example: Repeatedly clicking on the time pickers on the same row (between start and end time) works fine. But when I click 1) the 'March Hare' start time picker (which displays OK), 2) then the 'Crush' end time picker (which displays OK), 3) and finally the 'March Hare' end time picker, the picker comes up blank (see attached images).
This ONLY happens when I cycle through the sequence the second time. The first time I click the sequence described above, all time pickers display correctly. And I can just repeat this same sequence over and over again, where the first sequence displays OK but repeating it again displays the blank. Also, if I click the icon for the blank picker, the picker will redisplay correctly. So really kind of puzzling.
Do you have any idea, off hand, why this might be happening?
NOTE: I just tried this with the datepicker-bs4 and the same thing happens with the datepicker-bs4 also.

I can't replicate that issue. Can you create an example in jsfiddle or codepen?

Just a quick update, and sorry for the delay. I've been trying to finish up some more urgent app/db issues before going live so this one will have to wait a bit longer as it appears sporadic. BTW: I tried to reproduce the issue in jsfiddle per your suggestion but was not successful, so I'll get to it eventually, or maybe you can have a peak at it when the app is live (if that will help). Tks... :)

Do you see any errors in the browser console? Offhand, the issue that you're seeing may occur if event to populate the contents of the popover doesn't get triggered. However, I can't reproduce the conditions to make that happen.

No... no errors in the console, but I did some debugging and figured out what the problem was: it was the 'duplicate' id's in the input fields. When I removed them, the timepicker (and datepicker) work fine now.
I am calling both the timepicker and datepicker using a class so I didn't think twice about the id's being an issue. Maybe it was because the id's were all identical (I know... I know... bad HTML) but those id's are copied from a hidden template and are ONLY used to identify the fields in the enclosing div to extract the values on save. I ended up using the 'name' property instead of the 'id' and that seems to have fixed the issue.
That page is quite complicated as it allows managers to insert n+1 records with the start/end date/time for each, and since I was calling the pickers using a class, I didn't think it mattered if there were id's or duplicate id's. Not sure if there is anything you need to do on your end, but I thought I'd share what I found just in case.
Anyway, all is good now and thanks for your help and the awesome pickers.
Cheers... 😊


Should -1 Fractures Be Included in the Detection Task?

Hello,
I have a question regarding the fracture detection task. In the training dataset, there are fractures labeled as -1. Should these -1 labeled fractures be included in the detection task, or should they be ignored?
Thank you for your assistance.

Hi @hj001125 ,
Yes, in the detection task, all the mask instances can be used for detection training / eval. "-1" is only for classification task.
Jiancheng


[QUESTION] How can I show options and help when an arguments is missing?

First check

[x] I used the GitHub search to find a similar issue and didn't find it.
[x] I searched the Typer documentation, with the integrated search.
[x] I already searched in Google "How to X in Typer" and didn't find any information.
[x] I already searched in Google "How to X in Click" and didn't find any information.

Description
How can I show options and help when an arguments is missing?
Additional context
for example without a file_name in
def sequence(
    chain_code: str = typer.Option('A', '--chain', help='chain code', metavar='<CHAIN-CODE>'),
    no_chain_start: bool = typer.Option(False, '--no-chain-start/',help="don't include a start of chain link type for the first residue"),
    no_chain_end: bool = typer.Option(False, '--no-chain-end/', help="don't include a end of chain link type for the last residue"),
    entry_name: str = typer.Option('nmrview', help='a name for the entry'),
    pipe: Path = typer.Option(None, metavar='|PIPE|',
                              help='pipe to read NEF data from, for testing [overrides stdin !use stdin instead!]'),
    file_names: List[Path] = typer.Argument(..., help='input files of type nmrview.seq', metavar='<SEQ-FILE>')
):

I get
Usage: nmrview_seq.py [OPTIONS] <SEQ-FILE>
Try 'nmrview_seq.py --help' for help.

Error: Missing argument '<SEQ-FILE>'.

but I would like to see
Usage: nmrview_seq.py [OPTIONS] <SEQ-FILE>

  convert NMRVIEW .seq file to NEF

Arguments:
  <SEQ-FILE>  input files of type nmrview.seq  [required]

Options:
  --chain <CHAIN-CODE>            chain code  [default: A]
  --no-chain-start                don't include a start of chain link type for
                                  the first residue  [default: False]

  --no-chain-end                  don't include a end of chain link type for
                                  the last residue  [default: False]

  --entry-name TEXT               a name for the entry  [default: nmrview]
  --pipe |PIPE|                   pipe to read NEF data from, for testing
                                  [overrides stdin !use stdin instead!]

  --install-completion [bash|zsh|fish|powershell|pwsh]
                                  Install completion for the specified shell.
  --show-completion [bash|zsh|fish|powershell|pwsh]
                                  Show completion for the specified shell, to
                                  copy it or customize the installation.

  --help                          Show this message and exit.

Error: Missing argument '<SEQ-FILE>'.

You can use no_arg_is_help:
import typer

app = typer.Typer()

@app.command(no_args_is_help=True)
def main(some_arg, opt1=1, opt2=5):
    typer.echo(some_arg)

if __name__ == "__main__":
    app()

Thanks!

You can use no_arg_is_help:
import typer

app = typer.Typer()

@app.command(no_args_is_help=True)
def main(some_arg, opt1=1, opt2=5):
    typer.echo(some_arg)

if __name__ == "__main__":
    app()

Thanks, @captainCapitalism the no_args_is_help=True works fine.
I could not find it in the docs; where does this information come from?


"Could not find Template::Mojo"

When installing Task::Star with Panda, there is a warning during installation of Template::Mojo, and an error (pacage installation failure related to Mojo) during installing Bailador:
[user1<PHONE_NUMBER>_1 ]$ panda install Template::Mojo
==> Fetching Template::Mojo
==> Building Template::Mojo
Compiling lib/Template/Mojo.pm to mbc
==> Testing Template::Mojo
t/00-basic.t ..... ok
t/01-template.t .. ok
t/02-complex.t ... ok
All tests successful.
Files=3, Tests=21, 15 wallclock secs ( 0.12 usr  0.01 sys + 12.84 cusr  0.50 csys = 13.47 CPU)
Result: PASS
==> Installing Template::Mojo
===WARNING!===
The distribution Template::Mojo does not seem to have a "provides" section in its META.info file,
and so the packages will not be installed in the correct location.
Please ask the author to add a "provides" section, mapping every exposed namespace to a
file location in the distribution.
See http://design.perl6.org/S22.html#provides for more information.
[user1<PHONE_NUMBER>_1 ]$ panda install Bailador
==> Fetching Bailador
==> Building Bailador
Compiling lib/Bailador/Request.pm to mbc
Compiling lib/Bailador/Response.pm to mbc
Compiling lib/Bailador/Context.pm to mbc
Compiling lib/Bailador/Template.pm to mbc
Compiling lib/Bailador/Template/Mojo.pm to mbc
===SORRY!===
Could not find Template::Mojo in any of: /home/user1/.panda-work/1427816697_1/.panda-work/1427817064_1/blib/lib, /home/user1/.panda-work/1427816697_1/.panda-work/1427817064_1/lib, /home/user1/.perl6/2015.03-112-g01e1744/lib, /home/user1/.perl6/2015.03-112-g01e1744, /mnt/lib//perl6/rakudo/install/share/perl6/lib, /mnt/lib/perl6/rakudo/install/share/perl6/vendor/lib, /mnt/lib/perl6/rakudo/install/share/perl6/site/lib, /mnt/lib/perl6/rakudo/install/share/perl6, /mnt/lib/perl6/rakudo/install/share/perl6/vendor, /mnt/lib/perl6/rakudo/install/share/perl6/site
build stage failed for Bailador: Failed building lib/Bailador/Template/Mojo.pm
in method install at lib/Panda.pm:122
in method resolve at lib/Panda.pm:197
in sub MAIN at /mnt/lib/perl6/panda/bin/panda:20
in sub MAIN at /mnt/lib/perl6/panda/bin/panda:18
in block  at /mnt/lib/perl6/panda/bin/panda:82
Failure Summary
Bailador
*build stage failed for Bailador: Failed building lib/Bailador/Template/Mojo.pm

Same error:
==> Fetching Bailador
==> Building Bailador
Compiling lib/Bailador/Request.pm to mbc
Compiling lib/Bailador/Response.pm to mbc
Compiling lib/Bailador/Context.pm to mbc
Compiling lib/Bailador/Template.pm to mbc
Compiling lib/Bailador/Template/Mojo.pm to mbc
===SORRY!===
Could not find Template::Mojo in any of: /var/www/tmp/P6/.panda-work/1427818791_15/blib/lib, /var/www/tmp/P6/.panda-work/1427818791_15/lib, /home/zoffix/.perl6/2015.03-126-g5668f29/lib, /home/zoffix/.perl6/2015.03-126-g5668f29, /home/zoffix/.rakudobrew/moar-nom/install/share/perl6/lib, /home/zoffix/.rakudobrew/moar-nom/install/share/perl6/vendor/lib, /home/zoffix/.rakudobrew/moar-nom/install/share/perl6/site/lib, /home/zoffix/.rakudobrew/moar-nom/install/share/perl6, /home/zoffix/.rakudobrew/moar-nom/install/share/perl6/vendor, /home/zoffix/.rakudobrew/moar-nom/install/share/perl6/site
build stage failed for Bailador: Failed building lib/Bailador/Template/Mojo.pm
  in method install at lib/Panda.pm:122
  in block  at lib/Panda.pm:1
  in method resolve at lib/Panda.pm:187
  in sub MAIN at /home/zoffix/.rakudobrew/moar-nom/install/share/perl6/site/38:20
  in sub MAIN at /home/zoffix/.rakudobrew/moar-nom/install/share/perl6/site/38:18
  in block <unit> at /home/zoffix/.rakudobrew/moar-nom/install/share/perl6/site/38:82

Failure Summary
----------------
Task::Star
	*build stage failed for Bailador: Failed building lib/Bailador/Template/Mojo.pm

Followup:
Installing with panda install Template::Mojo installs just fine. The issue seems to exist when using panda install Task::Star

@zoffixznet It installs, but there is a warning:
===WARNING!===
The distribution Template::Mojo does not seem to have a "provides" section in its META.info file,
( This warning is present in the output copied in the first post )

@nbdsp that's fine. It's due to new feature in panda: (from http://pl6anet.org/)

FROGGS has merged a panda branch called “eleven”, which makes use of the CompUnitRepo functionality of rakudo. The effect is that installed distributions will not be stored as files on disk, but go into a database like thing. This is done to be able to install (and locate) several same-named versions of a distributions, or distributions made by different authors. These can now coexist, though as a consequence every distribution must have a “provides” section [^1] in its META.info.
Such a “provides” section (or an entire META.info) can be generated by running “panda gen-meta”. Modules without a proper “provides” have a panda icon with a strike-through “S11″ flag next to them on the modules list

This module simply hasn't been updated to have that section.

Sent PR to address the missing provides: https://github.com/tadzik/Template-Mojo/pull/15

Fixed now, thanks a lot :)


add support for deterministic entropy

In order to use RGB in LN we need to remove any type of entropy in transfers, so that channel counterparties can reconstruct RGB anchors in a deterministic way.
Right now in our rgb-lightning-sample we just duplicated methods with entropy, renamed them in <original_method_name>_static and changed the entropy fields to a fixed number. But ideally, for better privacy, this fixed number should be configurable via the method, so that the channel counterparties can derive this number via some secret channel data.
Here you can see the methods we had to duplicate (dbc_conclude -> dbc_conclude_static and add_raw_state -> add_raw_state_static).

Related: https://github.com/RGB-WG/rgb-std/issues/111#issuecomment-1872514299

This has been completed, so closing it


Item Level wrong

Some pieces of armor and weapons are showing item level 375. A couple weapons are showing 256.

Which means it's apparently a single byte, and there's an unknown field after ilvl.
@z16: ResourcesExtractor probably has the same bug.

I think it might be not even the full byte, possibly 7 bits is item level and the rest of the bits go to something else.
But at a minimum this is def 1 byte and no larger.

I think I or Wren will have a PR for this shortly


Docs/msgs

Maybe you also want to write some words about the audio messages @yunlong-wang-cn ? :)

yes


Defer creation of trust key file until needed

Currently push is the only operation making uses of the key, this change defers creation until the push operation is called.
Fixes #10442

I think we can write test for this.

@dmcgowan Integration tests failed to compile

LGTM

LGTM

LGTM


Composer updates and or composer dump-autoload scrubbed the top level db clean of all records

Bug description
May or may not be an issue but I've no idea how to find out either way,  I ran composer update and composer dump-autoload and the records for my top level database seemed to have vanished, luckily I had uat records and production records to fall back on.
Steps to reproduce
Run composer update && composer dump-autoload
Expected behavior
Loss of all records in in the top level database
Laravel version
8.83.16
stancl/tenancy version
3.5.7

Nevermind


any plans for other sonoff devices?

are there any plans for implementing other similar sonoff devices?
detailed instructions how to start off from a bare linux machine would be also very helpful

Very interested question! I'd love to see support for the TH10/TH16.

Guys, the original library is deprecated, see announcement here:- https://github.com/HomeACcessoryKid/ESP8266-HomeKit/issues/41
the new library is here:-
https://github.com/maximkulkin/esp-homekit
with examples here:-
https://github.com/maximkulkin/esp-homekit-demo

Here there are more Sonoff devices implementations, and it's based on esp-open-rtos and esp-homekit:
https://github.com/RavenSystem/esp-homekit-devices


Best way for forward a response from one stream to another?

Hello,
I'm looking for a way to "forward" an HTTP response from one stream to another while also being able to read the header and intercept the response based on some fields in that header. Basically something along these lines, but without the need to store the entire response body in the memory before it's forwarded:
void forward(tcp::socket source, tcp::socket sink, asio::yield_context yield) {
    beast::flat_buffer buffer;
    http::response<http::dynamic_body> rs;
    beast::http::async_read(source, buffer, rs, yield);
    if (!my_is_good_response(rs.base()) {
        my_send_bad_response(sink, yield);
        return;
    }
    beast::http::async_write(sink, rs, yield);
}

Howdy, inetic
I've written something similar for an HTTP proxy server.
To accomplish your goal, it's advised to follow the example here:
https://www.boost.org/doc/libs/1_70_0/libs/beast/doc/html/beast/more_examples/http_relay.html
The idea here is that you use Beast to read in the headers first, you can process them however you wish and then you can start writing back however you want.
The example in Beast doesn't complete the entire picture for HTTP as HTTP proxies are required by the RFC to remove/add certain header fields.
For a somewhat more complete example, here's my relay and its tests:
Impl: https://github.com/LeonineKing1199/foxy/blob/develop/include/foxy/detail/relay.hpp
Test: https://github.com/LeonineKing1199/foxy/blob/develop/test/relay_test.cpp
Let me know if you have any more questions or need clarification. The Beast buffer_body is a very fun type to play around with!

Those are all very nice examples. Thank you


Moves smaps and smaps_rollup into an async task since these can be expensive

What does this PR do?
The sample function of the Observer is now async and the smaps and smaps_rollup parsing has been moved into a spawned tokio task, which is awaited before the function returns.
Motivation
I've heard that the /proc/smaps file can take up to 250ms to populate in some circumstances, attempt to offload this off the main sample thread for now.
Related issues
Additional Notes

   FAIL [   7.536s] lading::bin/lading tests::inner_main_capture_has_data

failure is known flake, going to override and merge here.


Dynamic Payload Validation with Pydantic models

Discussed in https://github.com/samuelcolvin/pydantic/discussions/3342

Originally posted by uniyalnitin October 21, 2021
Example Code
Example payload 1
{
      "data": [
         {
            "customer_name": "Nitin",
            "customer_phone_number": "222222222",
            "customer_slug": "Nit22",
            "message": "Hi, Thanks for choosing us."
         }
      ],
      "identifier_slug": "notify_customer",
      "channel_type": ["sms"]
   }
}

Example payload 2
{
      "data": [
         {
            "name": "Nitin",
            "contact_number": "22222222222222"
            "email"<EMAIL_ADDRESS>            "subject": "50% Discount for you",
            "text": "Please use coupon flat200 for getting 50% discount"
         }
      ],
      "identifier_slug": "promo_email",
      "channel_type": ["email", "sms"]
   }
}

Description
I want to parse a dynamic request body (payload) with pydantic models. I am unable to find the solution to this problem.
The payload contains three parts:

identifier_slug: this is event identifier
channel_type: List of Enum ("email", "sms")
data: List of Dict

The data is dependent on identifier_slug and channel. data is unique for identifer_slug and channel. How can I build a dynamic pydantic model that serializers whats inside data based on identifer_slug and channel from same payload.
Operating System
Linux
Operating System Details
No response
Pydantic Version
1.8.2
Python Version
Python 3.8.10

I have a similar use case on a project.
I think the following works for your case, but I wonder if it is the best way to achieve this.
Any feedback from a pydantic veteran is welcome !
from enum import Enum
from typing import List, Literal, Union

from pydantic import BaseModel, Field

class ChannelType(str, Enum):
    SMS = "sms"
    EMAIL = "email"

class ModelIdentifier(str, Enum):
    NOTIFY_CUSTOMER = "notify_customer"
    PROMO_EMAIL = "promo_email"

class NotificationData(BaseModel):
    customer_name: str
    customer_phone_number: str
    customer_slug: str
    message: str

class NotificationModel(BaseModel):
    identifier_slug: Literal[ModelIdentifier.NOTIFY_CUSTOMER]
    channel_type: List[ChannelType] = Field(
        const=True,
        default=[
            ChannelType.SMS,
        ],
    )
    data: List[NotificationData]

class PromoData(BaseModel):
    name: str
    contact_number: str
    email: str
    subject: str
    text: str

class PromoModel(BaseModel):
    identifier_slug: Literal[ModelIdentifier.PROMO_EMAIL]
    channel_type: List[ChannelType] = Field(
        const=True, default=[ChannelType.EMAIL, ChannelType.SMS]
    )
    data: List[PromoData]

class Model(BaseModel):
    __root__: Union[NotificationModel, PromoModel]


Missing Functions

The following functions have been identified as missing;
sq_setnativeclosurename(..);
sq_getnativeclosurename(..);   <-  Reference ObjectCreation.cpp Line 196

About sq_getnativeclosurename in ObjectCreation.cpp, i actually mistyped that. It should have been sq_setnativeclosurename. I implemented that one, but i can't find sq_getnativeclosurename anywhere in the source.

Excellent news. Thank you!


Accessing members of structs from maps broken.

See the snippet below:
import "core:fmt.odin";

Foo :: struct {
	a: int,
}

main :: proc() {
	foos: map[string]Foo;

	foos["hey"] = Foo{1};
	foos["ho"] = Foo{2};
	foos["let's"] = Foo{3};
	foos["go"] = Foo{4};

	for v, s in foos {
		fmt.println(s, v, v.a);
	}

	fmt.println(foos["hey"],   foos["hey"].a);
	fmt.println(foos["ho"],    foos["ho"].a);
	fmt.println(foos["let's"], foos["let's"].a);
	fmt.println(foos["go"],    foos["go"].a);
}

Output:
hey Foo{a = 1} 1
ho Foo{a = 2} 2
let's Foo{a = 3} 3
go Foo{a = 4} 4
Foo{a = 1} 4
Foo{a = 2} 4
Foo{a = 3} 4
Foo{a = 4} 4

With only looking at the code, it seems that selector expressions (x.y) are not being handled correctly for values with addressing modes such as those for maps.
Lines 5214,  5229, and any line that does something of this nature, ir_build_addr(proc, expr).addr, should be replaced with a wrapping function which takes into account that expression's addressing mode.


10 Day IC looks like downsampled 1 day IC in rolling plots

In the rolling IC plots the longer timeframes look like downsampled versions of the shorter ones. Can we get a code correctness check on those plots and make sure that they're behaving as expected?

CC @luca-s

@twiecki Out of curiosity I had a look but I didn't see any obvious reason that explains a downsampling. @CaptainKanuk would it be possible to have a screenshot of the plots that raised the doubts?

These are the plots that invited suspicion. It could just be a coincidence with the factor, if the code looks good I won't worry about it.


Maintenance of IntroToPandasGuide class that provides guide for basics of Pandas library

The scope of the issue includes maintenance of the following classes:

[x] #651

[x] #652

[x] #653

[x] #654

Maintenance has been processed.

Call code is missing. Please add the following comment:

Class was checked by the following call:
#```python
#call_code
#```

call code:
from dynpy.utilities.documents.document import IntroToPandasGuide
IntroToPandasGuide(None);

Call code
from dynpy.utilities.documents.document import IntroToPandasGuide
IntroToPandasGuide(None);

The code was checked with the following call:
from dynpy.utilities.documents.document import IntroToPandasGuide
IntroToPandasGuide(None);


Unfiltered

Hey Topher, I think we have modified unfiltered Http headers. I tested it using curl and it was working fine. I had one doubt though. At the moment I have declared the scala Objects ValueTxClock and ConditionTxClock inside package.scala in unfiltered itself, since they are not defined in unfiltered source code. I hope this is fine.

Can one of the admins verify this patch?

Add to whitelist.

Yes indeed, you need to add those objects, and package.scala is a reasonable place to do so. Just a couple things:

Also update the tests. They need to check Read-TXClock and Value-TxClock, and supply Condition-TxClock.
Clean up the PR. There's some files (CrashChecks, build-scripts/*, etc) that don't belong in this PR.


opening > 10 files simultaneously results in only 10 files (tabs) opening

Issue:  Opening > 10 files simultaneously via R-click "open" (or via File > Open...) results in only 10 files (tabs) opening and only being able to select 10 tabs at a time

Environment:

Vim version:  8.0.1376

VIM - Vi IMproved 8.0 (2016 Sep 12, compiled Dec  7 2017 11:42:48)
macOS version
Included patches: 1-1376
Compiled by Homebrew
Huge version with MacVim GUI.  Features included (+) or not (-):
+acl               +clipboard         +digraphs          -footer            +libcall           -mouse_gpm         +odbeditor         +rightleft         +tcl               +virtualedit       -xpm
+arabic            +cmdline_compl     +dnd               +fork()            +linebreak         -mouse_jsbterm     +packages          +ruby              +termguicolors     +visual            -xsmp
+autocmd           +cmdline_hist      -ebcdic            +fullscreen        +lispindent        +mouse_netterm     +path_extra        +scrollbind        +terminal          +visualextra       -xterm_clipboard
-autoservername    +cmdline_info      +emacs_tags        -gettext           +listcmds          +mouse_sgr         +perl              +signs             +terminfo          +viminfo           -xterm_save
+balloon_eval      +comments          +eval              -hangul_input      +localmap          -mouse_sysmouse    +persistent_undo   +smartindent       +termresponse      +vreplace
+balloon_eval_term +conceal           +ex_extra          +iconv             -lua               +mouse_urxvt       +postscript        +startuptime       +textobjects       +wildignore
+browse            +cryptv            +extra_search      +insert_expand     +menu              +mouse_xterm       +printer           +statusline        +timers            +wildmenu
++builtin_terms    +cscope            +farsi             +job               +mksession         +multi_byte        +profile           -sun_workshop      +title             +windows
+byte_offset       +cursorbind        +file_in_path      +jumplist          +modify_fname      +multi_lang        +python            +syntax            +toolbar           +writebackup
+channel           +cursorshape       +find_in_path      +keymap            +mouse             -mzscheme          -python3           +tag_binary        +transparency      -X11
+cindent           +dialog_con_gui    +float             +lambda            +mouseshape        +netbeans_intg     +quickfix          +tag_old_static    +user_commands     -xfontset
+clientserver      +diff              +folding           +langmap           +mouse_dec         +num64             +reltime           -tag_any_white     +vertsplit         +xim
   system vimrc file: "$VIM/vimrc"
     user vimrc file: "$HOME/.vimrc"
 2nd user vimrc file: "~/.vim/vimrc"
      user exrc file: "$HOME/.exrc"
  system gvimrc file: "$VIM/gvimrc"
    user gvimrc file: "$HOME/.gvimrc"
2nd user gvimrc file: "~/.vim/gvimrc"
       defaults file: "$VIMRUNTIME/defaults.vim"
    system menu file: "$VIMRUNTIME/menu.vim"
  fall-back for $VIM: "/Applications/MacVim.app/Contents/Resources/vim"
Compilation: clang -c -I. -Iproto -DHAVE_CONFIG_H -DFEAT_GUI_MACVIM -Wall -Wno-unknown-pragmas -pipe  -DMACOS_X -DMACOS_X_DARWIN  -F/usr/local/opt/python/Frameworks -I/usr/local/Cellar/python/2.7.14/Frameworks/Python.framewo
rk/Versions/2.7/include/python2.7 -I/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/include/python2.7 -fno-strict-aliasing -fno-common -dynamic -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototype
s -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1
Linking: clang   -L.             -L /BuildRoot/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.Internal.sdk/usr/local/libressl/lib -L/BuildRoot/Applications/Xcode.app/Contents/D
eveloper/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.Internal.sdk/usr/local/lib -L.             -L /BuildRoot/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.Internal.s
dk/usr/local/libressl/lib -L/BuildRoot/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.Internal.sdk/usr/local/lib -F/usr/local/opt/python/Frameworks -L/usr/local/opt/python/Fram
eworks/Python.framework/Versions/2.7/lib/python2.7/config -lpython2.7 -framework CoreFoundation -L/usr/local/lib -o Vim -framework Cocoa -framework Carbon       -lm  -lncurses -liconv -framework AppKit   -fstack-protector  -
L/System/Library/Perl/5.18/darwin-thread-multi-2level/CORE -lperl -F/usr/local/opt/python/Frameworks -framework Python  -F/System/Library/Frameworks -framework Tcl -framework CoreFoundation -framework Ruby

OS: OS X 10.13.1
Terminal: GUI

Repro Steps:

Ensure MacVim prefs are:  Open Files from applications:  (•) in a new window, (v) with a tab for each file
Open n > 10 files via R-click "open" (or via File > Open)

Expected behavior:

Ideally:  All n files open as tabs in a given OS WindowManager (NS) window owned by MacVim (and are easily accessible, perhaps via a scrollable/compressible list of tabs or Expose- see OS X's Terminal.app)
Easier but not as-ideal:  First 10 files open as tabs in a given OS WindowManager (NS) window owned by MacVim and remaining files are accessible by clicking the >> that normally appear at the top right of the tab list when incrementally exceeding the 10 tab limit via File > Tab Open...

Actual behavior:

MacVim seems to process the files but only (the first?) 10 files (tabs) open and one needs to access the other n-10 files via the Buffers menu (which oddly lists all initially selected files, including those that weren't opened)

Easier but not as-ideal: First 10 files open as tabs in a given OS WindowManager (NS) window owned by MacVim and remaining files are accessible by clicking the >> that normally appear at the top right of the tab list when incrementally exceeding the 10 tab limit via File > Tab Open...

All files should already show up in the ">>" list. You are using a really old version of MacVim though. Can you update to latest first?
Otherwise there is an open PR that will make tabs better (#1120), but you wouldn't get that anyway if you don't update.

Closing issue for now unless this is reproduced on a recent build of MacVim.


About the preparation of the DIS dataset and the next research topic.

@xuebinqin Hii, do you have any chance to tell a bit about how you prepared and annotated the data V1 and V2?
For the last 10 days, I have been annotating some high res data for my validation dataset. I got 66 images annotated carefully. I worked on pre-predicted masks and fixed them. Thus, the process was easier for me. Although I did a little work, the burden was huge.
So, how did you manage to generate or annotate the data? It doesn't look like any artificial (or rendered) data to me.
Also, do you have any more ongoing research on image segmentation topic? What's next?
Thank you!

Hi, Akif,
Thanks for your interest.  That is one of the reasons why it takes us more
than a year for this paper. We do summarize some of the skills for manual
annotation. But the workload is still huge. For DIS V1.0, the average
labeling time for each image is 0.5 hours and some images take us up to 10
hours. In DIS V2.0 (unreleased one), the images' complexities are more
diversified and some images take up to 70 hours (the peacock in our github)
for labeling. We are trying to develop some semi-automatic ways and
self-supervised ways for segmenting highly accurate results. We probably
prepare a tutorial for annotating these highly accurate masks and share it
with the whole community later.
On Tue, Aug 2, 2022 at 3:58 AM Akif Faruk Nane @.***>
wrote:

@xuebinqin https://github.com/xuebinqin Hii, do you have any chance to
tell a bit about how you prepared and annotated the data V1 and V2?
For the last 10 days, I have been annotating some high res data for my
validation dataset. I got 66 images annotated carefully. I worked on
pre-predicted masks and fixed them. Thus, the process was easier for me.
Although I did a little work, the burden was huge.
So, how did you manage to generate or annotate the data? It doesn't look
like any artificial (or rendered) data to me.
Also, do you have any more ongoing research on image segmentation topic?
What's next?
Thank you!
—
Reply to this email directly, view it on GitHub
https://github.com/xuebinqin/DIS/issues/23, or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADSGORKVF4LQV2R7N42L7XTVXD5MLANCNFSM55KWOV2Q
.
You are receiving this because you were mentioned.Message ID:
@.***>

--
Xuebin Qin
PhD
Department of Computing Science
University of Alberta, Edmonton, AB, Canada
Homepage: https://xuebinqin.github.io/

@xuebinqin honestly, great work both in developing a new type of AI model (IS-NET) and also annotate all te data (DIS5K)! I wonder your data annotation methods. We appreciate you offering these to the community. Thank you much for your answer!


Python: Loops- while loop

Description 📜
I had fixed the issue for while loop in python
Fixes #557

Type of change 📝

[x] Documentation (Content Creation in the form of codes or tutorials)

Domain of Contribution 📊

[x] Python

Checklist ✅

[x] I follow Contributing Guidelines & Code of conduct of this project.
[x] I have performed a self-review of my own code or work.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] My changes generates no new warnings.
[x] I'm GWOC'21 contributor

I would like to contribute through documentation.

@kumarankm can you validate my pr

@kumarankm can you validate my pr

@ankurg132 is a mentor for you

Plz assign it to me

Hey @SouravKr2001 , the issue related to this PR is open for video contributions, you can comment if you're interested for that at #557. Also @naveeramesh can you also resolve the merge conflicts so that the supervisors can merge this one.

Hey @SouravKr2001 , the issue related to this PR is open for video contributions, you can comment if you're interested for that at #557. Also @naveeramesh can you also resolve the merge conflicts so that the supervisors can merge this one.

resolved @ankurg132

Looks good, @adityabisoi you can merge this one!


Copy only modified files in static directory

Copy only static files with different modified time or size.
I created a function called copy_if_modified that copies the file only if the destination file has the different size or modification date of the source file.
I'm learning rust now, all suggestions are welcome!

@Keats Do I need to make this work on Windows? :worried:

@ecarrara Yes, it needs to work on Windows.
I posted a comment on a outdated diff of a better way to do it imo but I'm not sure Github sent the notification as it was outdated:

We get the file path that triggered the notify in the serve command so I think a simple method Site::copy_static_file<P: AsRef>(path: P) -> Result<()> would be the easiest. Have to see how notify would handle things like copying several files at once but I believe it would work and be more efficient than going through each file

@Keats sorry for replying too late!
I agree with you about copying only the changed file that triggered de change notification. But I think we should keep the logic "copy only modified files" in build command, like Hugo does[1], to speedup the build time when we have a large number of big files in static directory.
[1] - https://github.com/spf13/fsync/blob/master/fsync.go#L238

But I think we should keep the logic "copy only modified files" in build command, like Hugo does[1], to speedup the build time when we have a large number of big files in static directory.

I think that can be added later on if needed and should default to copying everything if it is added. Let's keep things simple to get started and since you can co-locate static files in the content directory, I expect the static folder to be quite small anyway compared to Hugo. Even then, I'm not sure it would make a big difference benchmark wise. Same for the copy_if_modified.
In short, if we can make that PR be only about the copy_static_file method and keep everything else for later if it becomes necessary that would be great.

lgtm after checking the comment!

Thanks a lot!


Curl errors when installing applications

Describe the bug
Curl errors when installing applications due to existing configs
To Reproduce
Steps to reproduce the behavior:

Go to application setup
Click on applications with config cloning
You will see error when already existing config is found

Expected behavior
To clone the config and work
Screenshots

Checklist

[x] I checked for duplicate issues.
[x] I checked already existing discussions.
[x] This issue is not included in the roadmap.
[x] This issue is present on both stable and development branches.

could not reproduce

ran multiple times, success each time.


Locks on UI thread or async dispatches, can't have neither

From @bkase on April 7, 2016 0:6
Due to the use of ReadWriteLocks in many places (notably Promises), it's impossible to both avoid touching locks on the MainThread (which is very important esp. if you're using a lot of Promises) and also avoid any async dispatches (which is important when loading UIImages from a MemoryCacheLevel for example to avoid blinks)
One potential solution could be creating two different Promise types both conforming to some protocol PromiseType (and have Future's constructor take PromiseType rather than Promise).
where

MainThreadPromise has precondition(NSThread.isMainThread()) everywhere and has no locks
AtomicPromise is current Promise with the locks (but acquires locks)

Any thoughts?
Copied from original issue: WeltN24/Carlos#148

Hi @bkase, thanks for your suggestion!
I'm aware of this issue, and you're right that we should adopt some solution.
My only concern is that even with this supposed PromiseType, every CacheLevel decides internally what to use, and you can not force it unless you create your own, replicating its functionality but using another Promise (that you can do even now).
How would you proceed in this case?

Hi @bkase, I'm sorry you're right about the dependency Future has on Promise, I overlooked that. You can't effectively use another Promise implementation because of this.
Nevertheless, Promise is not a final class, so in theory one could subclass it to make it "main thread-safe" and produce that in your own CacheLevel implementation.
Here comes the catch, though (that is also the reason why I think having two implementations of MemoryCacheLevel would not solve the issue):
In basically all the transformation and composition functions, we always create new Promises that differ in some way from the original ones. We would have to determine which Promise to create everywhere in Carlos. This would require probably a "global setting" to define what type of Promise to create.
I have to think about it a little bit more but I'm afraid the complication would be quite big, and adding a "setting" to determine which Promise to create everywhere in the framework doesn't sound like a nice design solution either.
If you have any other idea, just write it here!

From @bkase on April 8, 2016 1:51
I wonder if there exists an easy, clean way to configure the built-in caches to either be threadsafe (use the "AtomicPromise") or fast (use the "MainThreadPromise").
I can imagine there may be something to using protocols to mixin the appropriate promise behavior.
This would require a huge refactoring, but imagine something like this:
protocol Asynchronous {
  associatedtype PromiseKind

  func makePromise() -> Self.PromiseKind
}
extension Asynchronous where Self.PromiseType == MainThreadPromise {
  func makePromise() -> MainThreadPromise {
    return MainThreadPromise()
  }
}
extension Asynchronous where Self.PromiseType == AtomicPromise {
  func makePromise() -> MainThreadPromise {
    return AtomicPromise()
  }
}

protocol MemoryCacheLevel: CacheLevel, Asynchronous {
   /*...*/
}
extension MemoryCacheLevel where Self.KeyType == Hashable {
  /* basically all of the current memory cache, but use `makePromise` */
}

class UIMemoryCache<K: Hashable, V/*...*/>: MemoryCacheLevel {
   typealias PromiseKind = MainThreadPromise
}
class AtomicMemoryCache: MemoryCacheLevel {
  typealias PromiseKind = AtomicPromise
}

What do you think about something like this?
Also:

you can not force it unless you create your own, replicating its functionality but using another Promise (that you can do even now).

Can you do this now? Future has a dependency on Promise, so even though Promise isn't exposed in any public APIs, there isn't a way to use a different Promise implementation and still use the CacheLevel protocol.


modsec detection and install

determine if cpanel built-in OWASP3 rules are in use, and if compatible, enable/install on target and migrate the source server's whitelist.


Flutter pub get doesn't download large files from Git LFS

Im creating a dart private package. Inside of that package I have a unity project with a file bigger than 100Mb, which GitHub doesn't support. So I used Git LFS to store that large file.
I add this package im pubspec.yaml in this way:
 my_package:
    git:
      url: git@github.com:my_package

When a run flutter pub get, throw this error:
Updating files:  99% (1500/1515)

Updating files: 100% (1515/1515)

Updating files: 100% (1515/1515), done.
Downloading unity/scarf/Assets/Emgu.TF.Lite/Plugins/iOS/libtfliteextern.a (171 MB)
Error downloading object: unity/scarf/Assets/Emgu.TF.Lite/Plugins/iOS/libtfliteextern.a (2914925): Smudge error: Error downloading unity/scarf/Assets/Emgu.TF.Lite/Plugins/iOS/libtfliteextern.a (2914925e5f2416673ccfb66d5dccd390b6bb3dc853d0f83079dfc08854016be4): error transferring "2914925e5f2416673ccfb66d5dccd390b6bb3dc853d0f83079dfc08854016be4": [0] remote missing object 2914925e5f2416673ccfb66d5dccd390b6bb3dc853d0f83079dfc08854016be4

Errors logged to /Users/bipo/.pub-cache/git/scarf_viewer-c8f68723d706dd384528795ee85c1375f00f13e3/.git/lfs/logs/20220301T154010.47026.log
Use `git lfs logs last` to view the log.
error: external filter 'git-lfs filter-process' failed
fatal: unity/scarf/Assets/Emgu.TF.Lite/Plugins/iOS/libtfliteextern.a: smudge filter lfs failed
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry with 'git restore --source=HEAD :/'

exit code: 128
pub get failed (server unavailable) -- attempting retry 1 in 1 second...

pub get failed (server unavailable) -- attempting retry 1 in 1 second...

Since pub command is failing, you'll need to report this in dedicated repo: https://github.com/dart-lang/pub/issues.
Closing from here.


How to test multi-reference text?

For example, there are 4 references for each article in DUC2004 dataset . How to test the rouge score for this ?

This feature is not currently supported.
Contributions are welcome!


cmd/asm: assembly language debugging with gdb broken in master tip

Please answer these questions before submitting your issue. Thanks!
What version of Go are you using (go version)?
master tip
Does this issue reproduce with the latest release?
No -- I can debug assembly in 1.10.1 and earlier releases
What operating system and processor architecture are you using (go env)?
Linux (Ubuntu 16.04)
What did you do?
Tried to debug assembly on both AMD64 and s390x
If possible, provide a recipe for reproducing the error.
cd /src/golang.org/x/crypto/curve25519
go test -c
gdb curve25519.test
break ladderstep_amd64.s:18
What did you expect to see?
break ladderstep_amd64.s:18
Breakpoint 1 at 0x4e6eaa: file
/src/golang.org/x/crypto/curve25519/ladderstep_amd64.s, line 18.
What did you see instead?
No source file named ladderstep_amd64.s
NOTES:
version of gdb:
gdb -v
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
(on both s390x and AMD64)
I also couldn't step into an assembly function from a go language file. Again, this works fine on go1.10.1 and earlier releases.

@dr2chase

Will look, I wonder if this has something to do with inlining changes because I made a point of (in theory) not modifying the assembly language pass.

I have updated the change https://golang.org/cl/110816, but I still can't make a breakpoint in an assembly file and see the assembly code.
What I did:
$cd math/big
$go test -c
$gdb ./big.test
(gdb) b arith_arm64.s : 34
(gdb) r -test.run ^$ -test.bench markAddVV
What I expected to see:
The program paused at arith_arm64.s line 34.
What I see instead:
[Inferior 1 (process 23459) exited normally]
If I make the breakpoints like this: b math/big.addVV, then the program will paused at the entry of function big.addVV, however I can't see the code of addVV but the Go code of the caller of big.addVV.
Besides, I use gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1

The bug is in link/internal/ld/dwarf.go, fixed here https://go-review.googlesource.com/c/go/+/102435/25/src/cmd/link/internal/ld/dwarf.go as part of a much larger CL.  The code that thought it had to reset is_stmt to 1 after each function was wrong-headed, and thus would (I think, have not verified this exactly) disable the is_stmt flag for every second function in an asm file.

This seems to be working now, so closing.  Please comment if you disagree.


25.1-rc1 crashes

I've noticed the latest build crashes from time to time, but haven't found a reliable way to crash it yet. It has happened on a few occasions when I press C-g inside a helm buffer, so if I had to guess it has to do with "undo" functionality. I'm on macOS 10.12 beta. Will update when/if I find a reliable way to reproduce.

Obviously, this was the wrong place for this report. My bad, sorry.

@mariusk that's OK. If you do file an upstream report, you can also paste the link here so we can monitor the situation, too.

Yeah, there's been some talk upstream about issues on Sierra: https://lists.gnu.org/archive/html/emacs-devel/2016-07/msg01125.html

Oh, that's you.


SR-3771 and SR-3751: libdispatch should not depend on external libkqueue

Revert back to libkqueue version that has the fix for
SR-2941 but does not have the changes to remove the
autotools-based build system.

@swift-ci Please test

@MadCoder would you mind merging this to master?  I screwed up in #203 and moved libkqueue too far forward.  I didn't notice (and neither did the CI testing) that because the latest libkqueue removed the autotools based build system it meant that we stopped building libkqueue from source when building libdispatch.  On both my machine and the Swift CI machines, libkqueue is installed, so the build just defaults back to using the system libkqueue which is not what we want.
Thanks & sorry for the churn.


is this built can work on macOS Mojave 10.14.6?

I completely followed the guid downloaded Mojave 10.14.6 and make a install usb disk.
But when I finished installation and selected Boot macOS from ... ,after a long time waiting,it just stuck at kextd stall[n],(240s):'appleACPICPU'.
Is that mean this built only work on macOS Mojave 10.14.5?

Same problem, waited for nearly 20 mins, it rebooted, then I met the #19, try to press space in the clover and select Boot macOS with injected kexts.

I followed the guide meticulously, and was successful. The only thing I did slightly differently was the first thing I did when booting into the USB was to go to Terminal (before Disk Utility) and zero out the SSD for a few thousand blocks (dd if=/dev/zero of=/dev/diskx bs=1m), Ctrl-C out, quit terminal, and then back to Disk Utility for a fresh erase/partition.

Instalation with the guide was ok, but after clover install to hd, efi replace when booting ( sloow), finally the NUC reboot. (i7-8705G 16+16 Gb `RAM) Mojave 10.14.6
Bios defaul + changes done

booting with these EFI
full.default.efi.zip
Could someone provide a succesful EFI i could try?

After 10.14.6 suplemental upgrade get a new kernel panic.

any suggestion ?

Add this patch to disable the extra log entries otherwise I cannot see the error message. https://github.com/RehabMan/Intel-NUC-DSDT-Patch/blob/master/config_nuc8_bc.plist#L261

After 10.14.6 suplemental upgrade get a new kernel panic.

Similar issue...I got to the clover screen, but it didn’t count down and auto boot, and it fails to recognize a usb keyboard.
I’ll mess with it some more tonight.

Add this patch to disable the extra log entries otherwise I cannot see the error message. https://github.com/RehabMan/Intel-NUC-DSDT-Patch/blob/master/config_nuc8_bc.plist#L261

seems to be CPU related, checked at clover BIOS the TRUST to yes as someone suggested, without succes

Add this patch to disable the extra log entries otherwise I cannot see the error message. https://github.com/RehabMan/Intel-NUC-DSDT-Patch/blob/master/config_nuc8_bc.plist#L261

seems to be CPU related, checked at clover BIOS the TRUST to yes as someone suggested, without succes

forget about it.
done a clean install, upgrade with combo and all is up & running.
thanks for the great work!!


Defined fonts not being loaded on some browsers, React

Some browsers does not load the used font, Kumbh Sans. In addition to Kumbh Sans, we also use Poppins in case Kumbh Sans fails to load, still there are times where neither is loaded.
Find more info: https://github.com/Satriaana/Satriaana-Web-Site/blob/dev/README.md

Can I pick this? @nimeshkavinda

Can I pick this? @nimeshkavinda

You sure can! shall I assign you?

Fixed.


Doors Won't Trigger

It appears that after the recent update to 0.8.16, I can't seem to trigger any @Door I have in my Trigger journal.
Here is a screenshot of the relevant triggers. The 0.8.17 update didn't appear to fix this either.

Looking at the console, I get the warning for each door:

trigger-happy |  Can't manage the config with entity 'door' and key 'undefined' on '@Door[each door coords]

I have inserted console.log into the macro it triggers with Advanced Macros just to test if it appears, and it does not, so it appears the @Door trigger isn't working for me.

On my test door seem to work, i will investigate it anyway, but maybe you can try the new configuration
@Door[*]@Tag[human readable label]
where the label of the @Tag is the list of tag you put with the Tagger module on the placeable object.
so you can move the door without retouch the coordinates .

K should be fixed in 0.8.20


hplip: add missing dependency

Graphical programs (e.g. hp-setup) error with:
error: No module named enum

This adds the enum module which fixes the error.
Motivation for this change
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[x] Assured whether relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

@GrahamcOfBorg build hplip

@hhm0 Still an issue on current 19.03:
[userp@nix01:~]$ hp-toolbox

HP Linux Imaging and Printing System (ver. 3.19.1)
HP Device Manager ver. 15.0

Copyright (c) 2001-15 HP Development Company, LP
This software comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to distribute it
under certain conditions. See COPYING file for more details.

error: No module named enum

[userp@nix01:~]$ nixos-version
19.03.173144.973a2705351 (Koi)

Cherry picked in 77295b0bd26555c39a1ba9c1da72dbdb651fd280


Validation comparison

Solved issue with tests and mapplots due to lack of values (all NaN or NanInf) in Kendall tau metrics (hardcoded)
Updated notebook
Updated colorbars

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

18 of 18 (100.0%)  changed or added relevant lines in 4 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 81.776%

Totals

Change from base Build<PHONE_NUMBER>:
0%

Covered Lines:
1160

Relevant Lines:
1362

💛  - Coveralls


Added Chile ID (RUT) validator

resolve #79
It includes the validator itself and unit tests. The algorithm was based on wikipedia info.

You'll have to fix the conflicts. We're now namespacing by continent, so make sure to get those right!

Should I namespace as Latin America or Central/South America? Latin America seems easiest

You'll have to fix the conflicts. We're now namespacing by continent, so make sure to get those right!

Should I namespace as Latin America or Central/South America? Latin America seems easiest

Please check now

I think Geography makes more sense here, so SouthAmerica 👍

It's done, mate!

Looking good on my end :+1:
Just needs @JoaoFSCruz to take a look

I have fixed the branch name because it was erroneously targetting a tag.
Please change the merge target to release-1.3.0

Done. I already changed the target to release-1.3.0 and add the codeblocks for Brazil, Chile and others minor fixes. However I did not add the code block for other European countries that do not yet have that (I realized it is necessary to add it when there are comments)

Valeu @gheleri! 😁
@all-contributors please add @gheleri for code and documentation