Information when using primary / secondary certificate set-up

We have found that for clusters using a two certificate set-up (primary / secondary), these steps do not work. An attempt to upgrade the cluster will result in an error message stating that a two certificate cluster cannot be changed to a one certificate cluster. The easiest solution for us was to delete the cluster and recreate it, but that might not be a viable option for others.
What's also confusing is the information around node-to-node security and providing a valid custom certificate when running Service Fabric on a custom domain. We have a CNAME record set-up that points to our public IP. To secure this, we need a certificate for our own domain. It's not entirely clear whether switching to the common name based certificate will also impact node-to-node security or not. I was expecting node-to-node to be able to use different certs than the traffic going to the Service Fabric Explorer, but that seems to be one and the same (or I'm still not completely getting it). Any way, I was failing to find good information on these scenarios in the docs.
Another item (should maybe be split into separate topics); the article does not describe the procedure of upgrading a certificate when using the common name selector. I can only assume we update the vault URLs point to a cert which has the same CN, but this should be described as well.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: d34f6d36-58e7-ab10-e659-bdd10a64deca
Version Independent ID: 25d94e4b-8c95-7423-2665-7c09d438a16b
Content: Update an Azure Service Fabric cluster to use certificate common name
Content Source: articles/service-fabric/service-fabric-cluster-change-cert-thumbprint-to-cn.md
Service: service-fabric
GitHub Login: @rwike77
Microsoft Alias: ryanwi

Thanks for the feedback! We are currently investigating and will update you shortly.

@jsiegmund all valid feedback. I will assign to the author to review and see how we might be able to update/ add more information around your ask.

@jsiegmund
Will you confirm my understanding of this issue:
You receive an ARM deployment error, when you are upgrading Azure resource Type == 'Microsoft.ServiceFabric/clusters', by replacing the following cluster resource properties:
certificate.thumbprint
certificate.thumbprintSecondary
certificate.x509StoreName
Ref:
https://docs.microsoft.com/rest/api/servicefabric/sfrp-model-certificatedescription
with the following cluster resource properties:
certificateCommonNames.commonNames.certificateCommonName
certificateCommonNames.commonNames.certificateIssuerThumbprint
certificateCommonNames.x509StoreName
Ref:
https://docs.microsoft.com/rest/api/servicefabric/sfrp-model-servercertificatecommonnames
This single deployment is also upgrading Azure resource type == Microsoft.Compute/virtualMachineScaleSets, to declare the common name of the cluster certificate to be ACL'd to the SF runtime, by replacing the following scale set extension "Microsoft.Azure.ServiceFabirc" publisher property:
virtualMachineProfile.extensionProfile.extensions.properties.settings.certificate.thumbprint
with the following scale set extension "Microsoft.Azure.ServiceFabric" publisher property:
virtualMachineProfile.extensionProfile.extensions.properties.settings.certificate.commonNames
Its assumed that all of these certificates where deployed and installed on all of your Azure Service Fabric cluster hosts machines, by using the following property:
Microsoft.Compute/virtualMachineScaleSets.properties.osProfile.secrets.vaultCertificates.certificateUrl
Which installs the certificate on linux hosts at /var/lib/sfcerts, and in the appropriate certificate store on Windows; further its assumed all certificates are valid:

x509 Certificates
PEM (Linux) or PFX (Windows) formatted
Not Expired
Explicitly trusted by the user cause they declared it in their template
Issued by Certificate Authority (E.G. you can NOT securely identify a self signed certificate by its common name, and shouldn't be using self-signed certificates in production. However, it is a supported operation to switch from a self signed certificate declared by thumbprint, to a CA issued certificate declared by common name.)

Refs:
https://docs.microsoft.com/azure/service-fabric/service-fabric-configure-certificates-linux
https://docs.microsoft.com/azure/security/azure-service-fabric-security-best-practices
I also assume you understand that the SF runtime can only use one cluster certificate, and deterministically uses the validate certificate that expires further into the future.
E.G. 'thumbprint', and 'secondaryThumbprint' properties do not affect the SF runtime behavior, the algorithm used is a valid certificate that expires further into the future; thus if your 'secondaryThumbprint' expires further into the future, it will be used instead of 'thumbprint' declared certificate. So manually swapping them isn't a supported operation any more, as the runtime automates this operation.

Finally if everything I stated to this point is true, then you need to perform 2 deployments to resolve your issue.
First deployment:
Remove the ARM properties for the cluster certificate that isn't being used, which I assume to be the following, if your secondary was provisioned to expire further into the future:
Microsoft.ServiceFabric/clusters.properties.certificate.thumbprint
and leaving your cluster resource with the following cluster certificate properties:
Microsoft.ServiceFabric/clusters.properties.certificate.thumbprintSecondary
Microsoft.ServiceFabric/clusters.properties.certificate.x509StoreName
Note: If both are valid certificates, then it actually doesn't matter if you remove 'thumbprint' or 'thumbprintSecondary', again the algorithm will use a valid certificate declared.
Secondary Deployment:
Follow the guidance provided in this document:
https://docs.microsoft.com/azure/service-fabric/service-fabric-cluster-change-cert-thumbprint-to-cn
Again for Common name swapping isn't a thing any more; the runtime will only use one valid certificate, and its the one that expires further into the future.
My understanding is this behavior is intended to not imply the user's intentions.
E.G. The failure of not allowing 'thumbprint' and 'secondaryThumbprint' to be replaced by 'certificateCommonName' in a single deployment, is to mitigate the runtime from guessing the user's intention. As removal of either 'thumbprint' or 'secondaryThumbprint' but not both, has the behavior of using the remaining valid certificate declared by its thumbprint, which maybe different than the cert the runtime would select when 'certificateCommonName' is declared by the user. As the user could have deployed a 3rd CA issued certificate to their host, that may or may not expire further into the future, than either of the user's 'thumbprint' or 'secondaryThumbprint' certificates; which may result in the user expecting either one of the currently declared by thumbprint certificates to be used (pending they where issued by a CA), or another CA issued certificate that happens to expire further into the future to be used.
Once you confirm this context resolves your issue, @rwike77 please create a PR to add the following note to this document, and close this:
If you have 2 thumbprint's declared in your template, you need to perform two deployments, the first is to remove one of the two thumbprints.

@jsiegmund @aljo-microsoft   I added the note to the article.  The updates just merged and should publish live tomorrow.  #please-close


Export-DbaInstance - add "-AzureCredential" option

Summary of new feature

Clear/concise description of what the problem is that the new feature can solve.
Add the -AzureCredential option to Restore-DbaDatabase inside of Export-DbaInstance in the 'Databases' section to allow the user to choose wether restore from disk or url.

Proposed technical details (if applicable)

e.g. Blog post reference that shows example code or functionality
'Databases' section:

if ($Exclude -notcontains 'Databases') {

                $fileCounter++

                Write-Message -Level Verbose -Message "Exporting database restores"

                Write-ProgressHelper -StepNumber ($stepCounter++) -Message "Exporting database restores"

                Get-DbaDbBackupHistory -SqlInstance $server -Last | Restore-DbaDatabase -SqlInstance $server -NoRecovery:$NoRecovery -WithReplace -OutputScriptOnly -WarningAction SilentlyContinue | Out-File -FilePath "$exportPath\$fileCounter-databases.sql" -Append:$Append

                Get-ChildItem -ErrorAction Ignore -Path "$exportPath\$fileCounter-databases.sql"

                if (-not (Test-Path "$exportPath\$fileCounter-databases.sql")) {

                    $fileCounter--

                }

            }

Latest version of dbatools as of writing

Please provide the latest released version of dbatools module.

ModuleType Version    Name                                PSEdition
Script     1.0.93     dbatools                            Desk

ooo this looks fun! i'll take a look


[SPARK-29994][CORE] Add WILDCARD task location

What changes were proposed in this pull request?
This PR adds a new WILDCARD task location that can match any host. This WILDCARD location can be used together with other regular locations in the list of preferred locations to indicate that the task can be assigned to any host/executor if none of the preferred locations is available.
Why are the changes needed?
This is motivated by the requirement from LocalShuffledRowRDD. When the number of initial mappers of LocalShuffledRowRDD is smaller than the number of worker nodes, it can cause serious regressions if short-running tasks all wait on their preferred locations while they could have otherwise finished quickly on non-preferred locations too.
We have a "locality wait time" configuration that allows a task set to downgrade locality requirement after a certain time has passed. Yet, this configuration affects all task sets in the scheduler, and tasks all differ in penalty of locality miss. Thus, we need this finer-grained option for individual tasks to opt out of locality.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Added UT.

cc @cloud-fan @jiangxb1987 @gatorsmile

Test build #114270 has started for PR 26633 at commit bb91814.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19113/
Test PASSed.

Test build #114270 has finished for PR 26633 at commit bb91814.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114270/
Test PASSed.

Test build #114415 has started for PR 26633 at commit 72a946c.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19252/
Test PASSed.

Test build #114415 has finished for PR 26633 at commit 72a946c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114415/
Test PASSed.

From my POV this is a shortcut  to allow some tasks to get rid of the 3 seconds locality wait time limitation from delay scheduling. This change looks good to me because it avoids to change the global locality wait time, thus the influence can be restricted in a desired way.
One concern that I shall rise is, how shall we restrict the WILDCARD locality being used less properly? i.e, include WILDCARD into preferedLocations where the penalty of locality miss is nontrivial.
Also cc @squito @tgravescs

I don't think there is a need to restrict it. Every RDD should "know" their own locality preference as well as the penalty for a locality miss. If we ever needed to make sure the WILDCARD is being used properly, we would have to worry about whether other regular preferred locations are returned correctly and truly reflect their best possible locality choice.
That said, the 3 minute global locality wait time is not gonna work for all cases, and yet the WILDCARD location alone is not fine-grained enough either. Ideally we should have penalty, or say, the importance of locality, "encoded" with location itself, in the form of wait time, e.g., "wildcard, 2s", so that the task can still try to wait a minimum time before getting randomly assigned. However, this would involve big changes to the current Spark scheduler.

Could we please hold this PR for a couple more days to allow more eyes on it, since it's a critical improvement to Spark Core?

Sure, @jiangxb1987 !

I'm not seeing how this is different then any other task in Spark?  In my opinion the locality fall back in Spark is broken.  I generally recommend people to set the locality delay to 0 on the node side because you can get very weird results where tasks wait way to long to be scheduled. On most networks these days its better to just run the task somewhere then wait for locality.  I realize though there are other conditions this was added for and I've never spent the time to go look at a proper solution for it.
This just seems like a workaround to doing the right fix that can only be used by this specialized RDDs.  Maybe that is ok for now but I would like to make sure its very clear on that and somewhat hesitate because this is essentially a public API that would be hard to remove if it starts to get used elsewhere.
I'm assuming the intention is to just have LocalShuffledRowRDD always add it to the preferred locations?  I'm a bit surprised that change isn't in here as well as it seems relatively small and would show the use of it, maybe I'm wrong and its large though, which would make sense to split apart.
Are there other specific usecases you have for this?
I think this should be discussed more before going in.
@squito

I generally recommend people to set the locality delay to 0 on the node side because you can get very weird results where tasks wait way to long to be scheduled.

well, if we look at resource utilization, it may be better to wait for locality, and save resources for other jobs/tasks.
This is really a hard problem, and the default 3 seconds locality wait may not be optimal either. We can only know the optimal solution if we know what jobs/tasks will be submitted in the future.
For LocalShuffledRowRDD, we don't need an optimal solution. We only need to avoid regressions. Compared to the norma shuffle reader, which fetches shuffle blocks from different hosts, this new WILDCARD location won't make things worse. It tries to satisfy the locality, and maximums the resource utilization like the normal shuffle reader.
Is it possible to make this thing internal? e.g. do not document it publicly. This is not a perfect solution but I'm afraid there is no perf solution.

Thanks for the feedback, @tgravescs! This is a workaround. A complete solution would be bring the current locality fallback to task level instead instead of node level, as I said in the previous comment. An RDD knows the importance of locality based on its job and/or data size and sets a wait time for itself. Setting the cluster/node level wait time would definitely affect other workloads and is not a solution here.
I could add the usage by LocalShuffledRowRDD here in this PR but was thinking it was a Spark SQL change and better be put in a separate PR. I'm fine either way, it's one line of code change plus some code comments.
Back to its usage. Normally a LocalShuffledRowRDD can just read locally, using its map output locations. And we try to match its parallelism with the original ShuffledRowRDD, so we get the same parallel level and better locality. But the problem is when we have less mappers (from the shuffle map stage) than the number of worker nodes, e.g., 5 vs. 10, and if we stick to the preferred locations, the LocalShuffledRowRDD will suffer from locality wait and be even slower than the original ShuffledRowRDD. If we had a way to know we'd end up with this situation, we could back out of the LocalShuffledRowRDD optimization and fall back to the regular shuffle. But the number of nodes is dynamic and we can't decide at compile-time.
The WILDCARD location can make it totally adaptive in this sense. Given that Spark offers resources always from high locality level to low locality level, a node can never be "stolen" (for a specific task set) as long as it has unassigned tasks of higher locality match. So when the number of mappers is equal to or larger than the number of worker nodes, even with WILDCARD location specified, the read can be near completely local. Otherwise, if the number of mappers is smaller, you can get roughly "number_of_mappers/number_of_nodes" locality hit. In the worst case, if the number of mappers is significantly smaller, the LocalShuffledRowRDD will regress to a regular ShuffledRowReader but cannot be worse.

I am wondering whether you should return here with true for WildcardLocation without checking the host equality, shouldn't you?
https://github.com/apache/spark/blob/72a946cb5649a08a8bfc8de03924fd95349347e1/streaming/src/main/scala/org/apache/spark/streaming/scheduler/ReceiverTracker.scala#L287-L290

For a while I've been saying that we should set the locality wait to 0 in general (the biggest problem IMO is https://issues.apache.org/jira/browse/SPARK-18886).  You will notice even Kay comments in the discussion there:

But often for Spark, you have one job running alone, in which case delay scheduling should arguably be turned of altogether

Spark 3.0 might be a good chance to change that default.

Fixing the scheduler locality algorithm is definitely more changes. The locality delay to me should be a per task delay, if a task doesn't get scheduled in 3 seconds then fall to the next locality. Right now it waits for any tasks to not be scheduled for 3 seconds at that locality.  I know Kay has an argument for the FairScheduler use case but I don't know that I agree with that or that it isn't handled by the per task delay. If you really want your task to wait that long for locality you can simple set it higher. I'm not sure the code changes required to make that change though and if we really wanted to leave the old way in there with a config how ugly the code gets.
| But the problem is when we have less mappers (from the shuffle map stage) than the number of worker nodes, e.g., 5 vs. 10, and if we stick to the preferred locations, the LocalShuffledRowRDD will suffer from locality wait and be even slower than the original ShuffledRowRDD.
I'm not sure I follow this statement.  If you have less mappers -lets say you have 5 and you have 10 worker nodes (assuming this is standalone mode - or do you mean executors?) - the 5 maps will run one 5 of those nodes. Your LocalShuffledRowRDD uses the map output location as the preferred locations so why wouldn't the scheduler schedule on those nodes?  Are you saying the 10 workers nodes (not sure if you mean executors or workers?) are being used by others (either job or stage) and some might be busy and the delay on waiting is more then just reading over the network?  Is this case with dynamic allocation or not?  It sounds to me like the normal case of you ran on some executors, you may not have the same executors when your reduce phase runs so you are being delayed scheduling because you can't get node locality.  I think you could have the same thing with shuffledRowRDD with a small number maps/reducers.
The issue I want to understand is why are we special casing this one RDD for a performance improvement when in my opinion the majority of jobs would get a benefit from not having to wait for the locality (as implemented today). Changing the default like Imran mentioned might be a good first step and then fixing the algorithm would be the second in my opinion.
Do you think a default of 0 node locality would solve your problem? Obviously if a user does set it then it gets applied again though.

Changing the default locality wait time to 0 (or whatever it is) is based on the assumption that all workloads do not have serious penalty from a locality miss, coz we are looking at shuffles only. There can be exceptions where locality does matter a lot and it would be worth some wait time.
Back to the local shuffle reader. I'll explain from the very beginning what it does.
Background: In Adaptive Query Execution, we may decide to change a sort-merge-join to broadcast-hash-join in the middle of query execution. A sort-merge-join usually (not always) comes with both sides topped by a shuffle and a sort. Once the shuffle is materialized (the map stage is done), we come to a natural sync point where we can try re-planning based on the shuffle stats and data metrics we've got so far. So if one side is small enough to be broadcast, we can change the SMJ into a BHJ, but note that the shuffles underneath have either been completed or been started at least.
LocalShuffleReader:  We can't save the materialization (map stage) cost apparently, but what we can do is avoid the network cost of the reduce stage of a shuffle, because a BHJ does not care about the shuffle partitioning at all. The LocalShuffledRowRDD alters the behavior of a shuffle reduce stage, by just fetching data from a single mapper and setting the preferred location as where the mapper output is. Note that there can be multiple LocalShuffledRowRDD partitions for each mapper, which means a LocalShuffledRowRDD partition can fetch all the output or part of the output from a single mapper.
Problem: The number of shuffle mappers is decided by the number of partitions of the upstream operator. The number of mappers can sometimes be small, esp. when the upstream operator is a file scan. If it's smaller than the number of worker nodes (we are talking about "hosts" here and it doesn't matter much how many executors per host, coz the locality specifies hosts only), the tasks will all be scheduled on those hosts but not the others. For example, we have 10 worker nodes, each having 4 executors, and then we have one of the join tables has 5 partitions after scan and default shuffle partition number is 200. So after shuffle map stage, we'll get 5 mappers, each having 200 shuffle blocks. In the reduce stage, if using a regular ShuffledRowRDD, we'll get 200 shuffle reduce tasks, and assume there's no skew, the 200 tasks will have no location preference and be evenly scheduled on the 4 * 10 = 40 executors. And if otherwise, we use the LocalShuffledRowReader, in order to match parallelism, we'd like to have 200 output partitions too, which means there will be 200 / 5 = 40 tasks for fetching data from each mapper. Yet, because of the locality preference, all these 200 tasks will be scheduled on 5 hosts only, while the other 5 hosts will not share the workload.

Thanks for the explanation.

There can be exceptions where locality does matter a lot and it would be worth some wait time.

What are these cases? I'm sure there are but based on what myself and many others I've talked to, its the exception instead of the rule.  Doesn't matter whether its HDFS data or shuffle data.  People are setting this to zero now anyway so changing default makes sense to me.

we'd like to have 200 output partitions too, which means there will be 200 / 5 = 40 tasks for fetching data from each mapper.

I don't follow this logic how do you go from 200 output partitions to 40 tasks?  I would expect 200 output partitions to have 200 tasks. Doesn't matter to much as the main issue is your next sentence.

all these 200 tasks will be scheduled on 5 hosts only, while the other 5 hosts will not share the workload

Now this part I understand.  But goes back to what I said before, I don't see how this is any different  then any other RDD.   About a month ago, we specifically had a job that during shuffle was hitting this same thing. We had 20 nodes, but only 1 node was being scheduling on and it had significant impact on the job time, we set locality delay to 0 and worked around the issue. Another example, running on YARN. Let say I have 10 tasks reading hdfs data, I get 10 nodes, 5 of those nodes actually have HDFS blocks on them.  With locality turned on those 5 nodes will be loaded up and depending on how long they take could keep the other 5 tasks from running as quickly as they should.
So what happens if these 5 mappers have very large data or skewed data?  Is it better to skip locality?  That is going to add to the network usage - who is to say I want that vs waiting? It might be that the tasks take long enough that it actually does fall back to rack locality - it depends on the harmonics of when tasks finish and are scheduled.  I'm willing to bet that in general having locality delay 0 still is more performant, which is why we go back to locality delay 0 for a default. If they actually need locality then they can turn it back on.  Now that will also affect your RDD here as well - but this seems more of a very specific case.

what kind of performance impact do you see if you just don't set preferred locations at all in your RDD?

I don't follow this logic how do you go from 200 output partitions to 40 tasks? I would expect 200 output partitions to have 200 tasks. Doesn't matter to much as the main issue is your next sentence.

It's 200 tasks overall, but each mapper has 50. That simple, but doesn't really matter.

what kind of performance impact do you see if you just don't set preferred locations at all in your RDD?

It would be no different from ShuffledRowRDD, and why would we bother to do the LocalShuffledRowRDD in the first place.

But goes back to what I said before, I don't see how this is any different then any other RDD.

It is no different from any other RDDs (you mentioned). The only difference is that this RDD has a definitive "baseline" or "goal": it looks to perform no worse than a regular shuffle and better if possible. For other RDDs, I can't say what the target is, and what performance impact is considered acceptable.
Yes, setting the locality wait to 0 would solve the problem of LocalShuffledRowRDD perfectly, and the effect is equivalent to this PR's proposal on a single task set alone. This leaves us only one difference of opinion: the exceptions, which I choose not to disclose.

Please correct me if I'm wrong, it seems to me that the LocalShuffleRowRDD is a special case of ShuffleRowRDD, with the only difference that ShuffleRowRDD normally doesn't have preferred locations, while LocalShuffleRowRDD has a preferred locations list. On execution time, if the preferred locations list is short, it's highly possible that the tasks from LocalShuffleRowRDD would wait for preferred locations(executors/hosts) due to delay scheduling, which sometimes make the wait time even longer than the task duration.
Set the locality wait time to 0 should be an answer to this case (and possibly many other use cases, too). But on the other hand, it would cause regression to other jobs/stages, where task locality is critical(those exceptions  as Thomas mentioned), we just can't ignore those regressions, and I can image how many efforts it would take to fix the regressions on exception cases.
How about we accept the current PR as a temporary solution to workaround the delay scheduling issue, thus those RDDs that don't want to wait for perfect locality can just add WILDCARD to their perferredLocations? To me it's better than setting the locality wait time to 0 directly, as it won't affect other workloads.

Of course, it worth a separated JIRA/PR to discussion changing the default value of delay scheduling.

I'm on the other side, I would rather see default set to 0 which I think most people do anyway and I believe will help a lot of other cases then add extra one off maintenance code here.
But if others disagree I'm ok with this it just needs to he heavily documented as internal only developer api that should go away.
Do we know how much performance diff this makes and how often?

Users can always change change setting back if they really need locality and 3.0 is perfect time to change default if we arent going to dix

@tgravescs Changing the default value of spark.locality.wait is a very important topic. We need to collect more feedbacks from the community, instead of making the decision among us. 3 second is just a magic number. Anybody knows the history? Why we chose 3 seconds instead of 1 second or 0.5 second?
Also, the perf is related to the environment and the workload patterns [e.g., the cost of shuffling data, the current workload sizing, and the cluster's resource availability]. When running a short or streaming query in an idle local cluster, setting it to zero might not be a bad idea. When running it in a cloud environment, I do not know which value is the best. This really needs to do more performance testing using common workloads to find the next magic number.
Normally, we should be really careful when introducing any performance related change. The decisions we made will impact a lot of end users. Any larger than 5% perf regression for a single query [from the perf benchmark] is not acceptable when I worked for a commercial database.

I agree the default setting change needs to happen in a bigger conversation, but if that conversation is going to happen we shouldn't check this in until that is had in my opinion.
I have not seen a real argument why this RDD is different than any other. But if we fix the real issue with locality then it helps everything. The argument that its a special version of ShuffledRowRDD and that sometimes you hit this locality issue doesn't convince me. I can hit the locality issue with ShuffledRowRDD, I might not hit the issue with the LocalShuffleRowRDD. Why not change ShuffledRowRDD or HadoopRDD to use this as well because I can hit the same issue? The only argument I can see is limited scope, but at the same time does it only turn it on then when you hit the case described with mappers < reducers and I have more executors then mappers?  If it turns it on more than that, then one could argue you aren't following the semantics defined by Spark for locality wait.
I don't see any concrete numbers here on performance impact or how much this affects users or why we should special case this?  If it has a huge impact then I can see why we would special case it but I haven't seen any evidence of that.  Do we have any cases this is seen in production - is there negative impact of user just setting node locality wait = 0?
Again the main issue I have is that once it's introduced anyone can use it in an RDD - therefore I consider it a public interface.  You say its limited impact and only used by adaptive execution but once introduced nothing stopping others from using it.
Adding more people to get opinions.
@vanzin @dongjoon-hyun @srowen

@tgravescs

People are setting this to zero now anyway so changing default makes sense to me.

Not sure how representative these "people" are. So let's bring the whole thing to dev list discussion.

Again the main issue I have is that once it's introduced anyone can use it in an RDD - therefore I consider it a public interface. You say its limited impact and only used by adaptive execution but once introduced nothing stopping others from using it.

I actually see it the other way. If we do see that regular ShuffledRowRDD suffer from locality wait when it does happen to have a preferred location (because of satisfying REDUCER_PREF_LOCS_FRACTION and some other conditions), we might end up finding this new location handy.
I don't see why I need to make an argument about how LocalShuffledRowRDD is different from other RDDs. On the contrary, if other RDDs have the same requirement, they can opt to use this approach as well. But I have to point out we can't rule out the possibility that there are other other RDDs that don't enjoy the world of no locality wait at all.
As I pointed out in the very beginning https://github.com/apache/spark/pull/26633#issuecomment-558379888, each RDD should know their own locality preference as well as the importance of such locality. If we ever had to worry about this location being used improperly, we'd have to worry about if any other regular location is returned correctly by the RDD as well.
That said, this is a partial solution only. I'd like to see a complete fix as well, but I don't think we should go completely the other way, by changing the default wait time to 0.

Seems we all agree that the delay scheduling is problematic. And theoretically delay scheduling can be critical for some jobs (we can easily think of a custom RDD to prove this).
Setting locality wait to 0 by default is another topic. Even if we do it, people that run jobs that need delay scheduling still need to set the locality wait. For these users, we need this WILDCARD location feature to enable AQE.
That said, changing the default locality wait to 0 doesn't solve the problem. It doesn't mean locality wait is always 0. As long as it can be non-zero, we need to deal with it in AQE.

It would be good to go through all the places where TaskLocation or its host member is used.
Like this place which returns a Seq[String] from the locations host:
https://github.com/apache/spark/blob/72a946cb5649a08a8bfc8de03924fd95349347e1/core/src/main/scala/org/apache/spark/rdd/CoalescedRDD.scala#L179-L181
Which latter will be used to create only one PartitionGroup for all the partitions where WildcardLocation is the preferred location.

Nobody has answered my questions above as to why this RDD should be treated differently and the impact of this.  You just keep saying this is for adaptive scheduling.  As far as I can see, this is purely another instance of https://issues.apache.org/jira/browse/SPARK-18886 and I don't see why we aren't using the same workaround or really fixing the real issue.

As I pointed out in the very beginning #26633 (comment), each RDD should know their own locality preference as well as the importance of such locality. If we ever had to worry about this location being used improperly, we'd have to worry about if any other regular location is returned correctly by the RDD as well.

I don't agree. HadoopRDD for instance knows its locality, but how important the locality is very user/cluster specific. I don't see how the LocalShuffledRowRDD is any different.  You are saying the user never cares about the locality on this - please explain to me why and how it is different from HadoopRDD?  If we were to turn this on for HadoopRDD though then we would essentially be bypassing the locality settings.

Even if we do it, people that run jobs that need delay scheduling still need to set the locality wait. For these users, we need this WILDCARD location feature to enable AQE.

Again why is AQE different?  lets say I really want my HadoopRDD to use locality but then the shuffledRDD hits this issue.  As a user I can't just turn locality off for my shuffleRDD so what makes the LocalShuffledRowRDD any different?  From what has been described here, this is a very particular case. You have more nodes and reducers then maps, the maps finish very quickly (probably within 3 seconds), these are the same conditions other RDDs can hit the same issue

I don't see why I need to make an argument about how LocalShuffledRowRDD is different from other RDDs.

You do because you are doing a one off hack here that Spark has to maintain, adds a new public api, and we can't use for anything else, why not fix locality for all RDDs as they can hit the same issue.

why not fix locality for all RDDs as they can hit the same issue.
So what fix exactly are you talking about here?

@attilapiros Very good point. I'll go thru all references of host as well as TaskLocation.apply.

It is the same problem as https://issues.apache.org/jira/browse/SPARK-18886 . It would be great if we can solve that problem first, but seems there is no conclusion yet.
There is one difference in LocalShuffledRowRDD: it has a baseline. It's converted from the normal shuffle reader, and we shouldn't be slower than it. The normal shuffle reader mostly doesn't have locality (a reducer needs to read blocks from many hosts), so the WILDCARD location is a good solution. It kinds of turn off the locality wait for LocalShuffledRowRDD, to make it not slower than normal shuffle reader.
Do we have an ETA about when we can resolve https://issues.apache.org/jira/browse/SPARK-18886  ? We can't remove locality wait as nowadays we usually run many jobs on a Spark cluster. It's unclear to me what's the best solution to it.
BTW, this feature won't be documented and it's not that public to me. Users can only know it by reading the discussion here. We can still remove it later if https://issues.apache.org/jira/browse/SPARK-18886 is resolved. To me this is just a workaround to turn off delay scheduling for certain tasks instead of globally, which does have value.

any more comments? This workaround makes sense to me when there is no perfect solution for https://issues.apache.org/jira/browse/SPARK-18886 . This is also an important fix to turn on AQE by default in 3.0.

In my opinion the ideal thing is to fix SPARK-18886, its the perfect time, this is a new major release and this isn't something impacting production now so we don't really need a "quick fix".  I disagree with your comment there is no perfect solution, no one here has  tried and really no one here has give me any metrics as to why this is so important of a fix.
But I realize that is a lot more change so I'm ok with this going in as a temporary fix.  Please update based on the comments made - I want to make sure this is clearly documented in the code has a hack that will go away and no on else should use it.
Also can someone give me any performance metrics - how much of a different does the LocalShuffledRowRDD make?

Could someone help review my proposed solution for SPARK-18886 here:
https://github.com/apache/spark/pull/26696
The idea is to only reset scheduling delay timers if allocated slots, based on the scheduling policy (FIFO vs FAIR), are fully utilized.

Test build #114802 has started for PR 26633 at commit 78a123a.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19624/
Test PASSed.

@tgravescs Our benchmark comparing AQE w/ LSR (local shuffle reader) with AQE w/o LSR showed that before locality wait fix, there were 2 queries with over 10% regressions, and after the fix, there was no regression and one query had over 27% improvement.
@attilapiros Thank you again for the careful review! I changed a code in a way that WildcardLocation is only recognized and handled by DAGScheduler and TasksetManager internally and calling TaskLocation.apply will not instantiate a WildcardLocation, so the new location will have no effect or impact over all other usages of TaskLocation. I also refined addPendingTask so that the code change is minimized.
I also added javadoc stating the limited application and the experimental nature of the WildcardLocation class.

Test build #114802 has finished for PR 26633 at commit 78a123a.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114802/
Test FAILed.

Test build #114823 has started for PR 26633 at commit e821543.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19646/
Test PASSed.

Test build #114823 has finished for PR 26633 at commit e821543.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114823/
Test FAILed.

retest this please

Test build #114863 has started for PR 26633 at commit e821543.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19686/
Test PASSed.

Test build #114863 has finished for PR 26633 at commit e821543.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114863/
Test PASSed.

thanks for the performance stats, obviously its going to change some depending on job size and such when you say 10% and 27%, what are the total job run times?

are you sure that in your case the slowdown is even caused by SPARK-18886?  Even when that is solved, you could still get end up with one 3s wait for almost all tasks.
I really do see how this can help, I see why folks want this.  But my hesitance is that we're going to start putting in these random changes to delay scheduling, which will make the code even harder to understand; users will end up with even more knobs to tune; and we may be stuck with this even after SPARK-18886 because it would still be a performance regression against this change.
I agree with Tom's point -- I don't see how we know that ignoring locality waits is right for just this one RDD but not for others.  Though I want the default locality wait set to 0, I could see a cluster admin wanting to increase the locality wait because they know their cluster is very network constrained.  In fact this may be against the wishes of of one particular spark application, but still best for the cluster as a whole.   In that case, you really might want a 3s wait on LocalShuffledRowRDD

Thank you @squito , for the feedback!

In fact this may be against the wishes of of one particular spark application, but still best for the cluster as a whole. In that case, you really might want a 3s wait on LocalShuffledRowRDD

Actually, as I stated in previous comments, the only difference LocalShuffledRowRDD bears from other RDDs is the fact that it has been optimized from ShuffledRowRDD and there comes a worse case guarantee that this new RDD cannot perform any worse than the original ShuffledRowRDD, both to the application and to the cluster.

In fact this may be against the wishes of of one particular spark application, but still best for the cluster as a whole. In that case, you really might want a 3s wait on LocalShuffledRowRDD

Actually, as I stated in previous comments, the only difference LocalShuffledRowRDD bears from other RDDs is the fact that it has been optimized from ShuffledRowRDD and there comes a worse case guarantee that this new RDD cannot perform any worse than the original ShuffledRowRDD, both to the application and to the cluster.

I think you're comparing LocalShuffledRowRDD vs. ShuffledRowRDD;  I'm comparing LocalShuffledRowRDD with this change vs. LocalShuffledRowRDD without it.


[QUESTION] Does Shaka player have SDK ?

Hello,
I'd like know does Shaka Player have SDK for iOS/Android APP developing or not? Or Does Shaka player only for browser playback?

If you are asking about native app development, video playback on iOS is generally handled exclusively by Apple's AVPlayer using HTTP Live Streaming (HLS).
If you want to play video in a browser on iOS, Shaka does not support this yet, but you can follow progress on #997. You can play streaming video in Safari on iOS using The <video> element.
For video playback in native apps on Android, the Google standard is ExoPlayer, though there are other alternatives on Android. If you would like to play video in a browser on Android (or via Progressive Web App), Shaka supports this.
You can also use Shaka in TV/set-top-box HTML5 environments (Tizen 2017 TV's are officially supported).

Shaka Player only supports browser environments and environments where there is a JavaScript interpreter with the required types available (e.g. MSE).  This can include smart TVs like the Tizen TV.  We do not support native apps since they aren't written in JavaScript (some frameworks allow you to write JavaScript, but they don't have MSE or other browser types we need).

I see.
Thank you very much for your information.


Allow uploading from URL

I would expect this to be used like
requests.post(
    "http://pomf.website/upload.php",
    data="http://image.url/here"
)

Might add this as an option, it would allow for a lot of abuse tho.

I recommend making it optional so the server owner can decide.

it would allow for a lot of abuse tho.

Any more than normal uploading?

Since I've decided to not update Pomf anymore, seeing as Uguu has the same functionality, lets take this issue to Uguu instead. Do you mind opening the same issue?


Enable Autocomplete for Static Objects

When we add type to the static object, it stops showing an error when tried to access a key that doesn't exist.
To avoid those scenarios better option is to avoid providing types for those static objects.
Description
Fixes #6303
Type of change

Refactor

How Has This Been Tested?

Manually

Checklist:

[ ] My code follows the style guidelines of this project
[ ] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes

Test coverage results :test_tube:
:green_circle: Total coverage has increased
// Code coverage diff between base branch:release and head branch: refactor/enable-autocomplete-for-static-objects

Status
File
% Stmts
% Branch
% Funcs
% Lines

:green_circle:
total
51.51 (0.01)
33.47 (0)
29.31 (0.01)
52.14 (0.01)

:green_circle:
app/client/src/actions/orgActions.ts
68 (0.14)
100 (0)
0 (0)
52.94 (0.31)

:green_circle:
app/client/src/actions/pageActions.tsx
67.39 (0.36)
25 (0)
32 (0)
57.14 (0.69)

:green_circle:
app/client/src/reducers/entityReducers/datasourceReducer.ts
9.43 (0.17)
0 (0)
0 (0)
9.8 (0.18)

:green_circle:
app/client/src/reducers/uiReducers/onBoardingReducer.ts
30 (1.43)
100 (0)
0 (0)
30 (1.43)

:green_circle:
app/client/src/reducers/uiReducers/propertyPaneReducer.tsx
70.83 (7.87)
47.37 (4.51)
66.67 (16.67)
68.18 (8.18)

/ok-to-test sha=1c29c47

/ok-to-test sha=b4178ff


Problem Installing Parcels

Hi
I followed the instructions to install Parcels. Apparently everything worked out, but  when I try to import parcels I get:
"ImportError: No module named parcels"
I am using anaconda2, the directory py2_parcels was created inside ~/anaconda2/envs/py2_parcels
Do you know what can be missing?
Thank you
Ana

Did you activate the environment? Step 3 from http://oceanparcels.org/#installing is always necessary, before using parcels as installed following the guide.

I did...
$ source ~/anaconda2/bin/activate py2_parcels
I get no error message.
But still:
In [2]: from parcels import FieldSet
ImportError                               Traceback (most recent call last)
 in ()
----> 1 from parcels import FieldSet
ImportError: No module named parcels
Any ideas?

What does conda list -n py2_parcels say?

Another possibility: Do you happen to use ipython although it is not installed in the py2_parcels env?

packages in environment at /home/ana/anaconda2/envs/py2_parcels:

blas                      1.1                    openblas    conda-forge
bzip2                     1.0.6                h470a237_2    conda-forge
ca-certificates           2018.10.15           ha4d7672_0    conda-forge
cloog                     0.18.0                        0
dbus                      1.13.0               h3a4f0e9_0    conda-forge
expat                     2.2.5                hfc679d8_2    conda-forge
ffmpeg                    3.2.4                hf82bc7d_4    conda-forge
fontconfig                2.11.1                        6
freetype                  2.5.5                         2
gcc                       4.8.5                         7
geos                      3.6.2                hfc679d8_3    conda-forge
gettext                   <IP_ADDRESS>             h5e8e0c9_1    conda-forge
glib                      2.56.2               h464dc38_1    conda-forge
gmp                       6.1.2                         0    conda-forge
gst-plugins-base          1.12.5               hde13a9d_0    conda-forge
gstreamer                 1.12.5               h5856ed1_0    conda-forge
hdf4                      4.2.12                        0
hdf5                      1.10.1                        2    conda-forge
icu                       54.1                          0
isl                       0.12.2                        0
jbig                      2.1               h470a237_1001    conda-forge
jpeg                      8d                            2
krb5                      1.16.2               hbb41f41_0    conda-forge
libcurl                   7.62.0               hbdb9355_0    conda-forge
libedit                   3.1.20170329         haf1bffa_1    conda-forge
libffi                    3.2.1                hfc679d8_5    conda-forge
libgcc                    7.2.0                h69d50b8_2    conda-forge
libgcc-ng                 7.2.0                hdf63c60_3    conda-forge
libgfortran               3.0.0                         1    conda-forge
libiconv                  1.15                 h470a237_3    conda-forge
libpng                    1.6.35               ha92aebf_2    conda-forge
libsodium                 1.0.16               h470a237_1    conda-forge
libssh2                   1.8.0                         2    conda-forge
libstdcxx-ng              7.2.0                hdf63c60_3    conda-forge
libtiff                   4.0.6                         2
libxcb                    1.13                 h470a237_2    conda-forge
libxml2                   2.9.4                         0
libxslt                   1.1.29                        4    conda-forge
mpc                       1.1.0                         4    conda-forge
mpfr                      3.1.5                         0    conda-forge
ncurses                   6.1                  hfc679d8_1    conda-forge
openblas                  0.2.19                        2    conda-forge
openssl                   1.0.2p               h470a237_1    conda-forge
pandoc                    2.3.1                         0    conda-forge
pcre                      8.41                 hfc679d8_3    conda-forge
pixman                    0.32.6                        0    conda-forge
proj4                     4.9.3                h470a237_8    conda-forge
pthread-stubs             0.4                  h470a237_1    conda-forge
python                    2.7.15               h33da82c_4    conda-forge
readline                  7.0                  haf1bffa_1    conda-forge
sqlite                    3.25.3               hb1c47c0_0    conda-forge
tk                        8.6.8                         0    conda-forge
x264                      1!152.20180717       h470a237_1    conda-forge
xorg-libxau               1.0.8                h470a237_6    conda-forge
xorg-libxdmcp             1.1.2                h470a237_7    conda-forge
xz                        5.2.4                h470a237_1    conda-forge
zeromq                    4.2.5                hfc679d8_6    conda-forge
zlib                      1.2.11               h470a237_3    conda-forge

I am using ipython. But if I run "python example_peninsula.py --fieldset 100 100"
I have the same problem:
Traceback (most recent call last):
File "example_peninsula.py", line 1, in
from parcels import FieldSet, ParticleSet, ScipyParticle, JITParticle, Variable
ImportError: No module named parcels

Could you, within the conda prompt do the following:

source ~/anaconda2/bin/activate py2_parcels
python
(You're now in a python terminal)
import parcels

Does that work? If so, it means that the other program when you run parcels does not use the conda python.

Hardly any of the requirements seem to be there. Can you rerun this by first deleting the env
conda env remove -n py2_parcels
And then run the installation again?

Still get the same problem:
Python 2.7.15 | packaged by conda-forge | (default, Oct 12 2018, 14:10:50)
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import parcels
Traceback (most recent call last):
File "", line 1, in
ImportError: No module named parcels

conda env remove -n py2_parcels
I get:
CondaEnvironmentError: Environment error: cannot remove current environment. deactivate and run conda remove again

Then first
source activate base

And then remove and recreate.

PS: I've edited your above comment to list code as code. Check "Edit" to see how this is done.

Why reinstall? -- It looks like only part of the installation has actually been done. Ffmpeg is in the env. Jupyter, cartopy, and parcels itself aren't. Either because sth went wrong with conda or because you did not copy the whole installation line.
Let's see if this solves it.

thank you very much for your help. Its reinstalling ("Solving package specifications: ....."). I have to leave now, but will tell you later if it solved the problem.

Thank you for your help. This problem is solved, there was a problem with the installation due to the version of anaconda. I had to update it and now it seems parcels is working (with a lot of warnings).

That is good to know.
What kind of warnings do you get?  But it would be good to know if there's sth new.)

When I run python example_peninsula.py --fieldset 100 100
I get the warnings listed bellow, but it creates an output, so it seems to be working:
/home/ana/anaconda2/envs/py2_parcels/lib/python2.7/site-packages/pandas/_libs/init.py:4: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from .tslib import iNaT, NaT, Timestamp, Timedelta, OutOfBoundsDatetime
/home/ana/anaconda2/envs/py2_parcels/lib/python2.7/site-packages/pandas/init.py:26: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from pandas._libs import (hashtable as _hashtable,
/home/ana/anaconda2/envs/py2_parcels/lib/python2.7/site-packages/pandas/core/dtypes/common.py:6: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from pandas._libs import algos, lib
/home/ana/anaconda2/envs/py2_parcels/lib/python2.7/site-packages/pandas/core/util/hashing.py:7: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from pandas._libs import hashing, tslib
/home/ana/anaconda2/envs/py2_parcels/lib/python2.7/site-packages/pandas/core/indexes/base.py:7: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from pandas._libs import (lib, index as libindex, tslib as libts,
/home/ana/anaconda2/envs/py2_parcels/lib/python2.7/site-packages/pandas/tseries/offsets.py:21: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
import pandas._libs.tslibs.offsets as liboffsets
/home/ana/anaconda2/envs/py2_parcels/lib/python2.7/site-packages/pandas/core/ops.py:16: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from pandas._libs import algos as libalgos, ops as libops
/home/ana/anaconda2/envs/py2_parcels/lib/python2.7/site-packages/pandas/core/indexes/interval.py:32: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from pandas._libs.interval import (
/home/ana/anaconda2/envs/py2_parcels/lib/python2.7/site-packages/pandas/core/internals.py:14: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from pandas._libs import internals as libinternals
/home/ana/anaconda2/envs/py2_parcels/lib/python2.7/site-packages/pandas/core/sparse/array.py:33: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
import pandas._libs.sparse as splib
/home/ana/anaconda2/envs/py2_parcels/lib/python2.7/site-packages/pandas/core/window.py:36: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
import pandas._libs.window as _window
/home/ana/anaconda2/envs/py2_parcels/lib/python2.7/site-packages/pandas/core/groupby/groupby.py:68: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from pandas._libs import (lib, reduction,
/home/ana/anaconda2/envs/py2_parcels/lib/python2.7/site-packages/pandas/core/reshape/reshape.py:30: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from pandas._libs import algos as _algos, reshape as _reshape
/home/ana/anaconda2/envs/py2_parcels/lib/python2.7/site-packages/pandas/io/parsers.py:45: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
import pandas._libs.parsers as parsers
/home/ana/anaconda2/envs/py2_parcels/lib/python2.7/site-packages/pandas/io/pytables.py:50: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from pandas._libs import algos, lib, writers as libwriters
INFO: Generating NEMO FieldSet output with basename: peninsula
Peninsula: Advecting 20 particles for 1 day, 0:00:00

But now I was trying to follow the tutorials and wasn't finding the directories with the input data. I suppose I have to create them with create_testfields.py, right? but this doesn't seem to be working.
The output I have for  "python create_testfields.py" is below. Any ideas?
/home/ana/anaconda2/envs/py2_parcels/lib/python2.7/site-packages/pandas/_libs/init.py:4: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from .tslib import iNaT, NaT, Timestamp, Timedelta, OutOfBoundsDatetime
/home/ana/anaconda2/envs/py2_parcels/lib/python2.7/site-packages/pandas/init.py:26: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from pandas._libs import (hashtable as _hashtable,
/home/ana/anaconda2/envs/py2_parcels/lib/python2.7/site-packages/pandas/core/dtypes/common.py:6: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from pandas._libs import algos, lib
/home/ana/anaconda2/envs/py2_parcels/lib/python2.7/site-packages/pandas/core/util/hashing.py:7: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from pandas._libs import hashing, tslib
/home/ana/anaconda2/envs/py2_parcels/lib/python2.7/site-packages/pandas/core/indexes/base.py:7: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from pandas._libs import (lib, index as libindex, tslib as libts,
/home/ana/anaconda2/envs/py2_parcels/lib/python2.7/site-packages/pandas/tseries/offsets.py:21: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
import pandas._libs.tslibs.offsets as liboffsets
/home/ana/anaconda2/envs/py2_parcels/lib/python2.7/site-packages/pandas/core/ops.py:16: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from pandas._libs import algos as libalgos, ops as libops
/home/ana/anaconda2/envs/py2_parcels/lib/python2.7/site-packages/pandas/core/indexes/interval.py:32: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from pandas._libs.interval import (
/home/ana/anaconda2/envs/py2_parcels/lib/python2.7/site-packages/pandas/core/internals.py:14: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from pandas._libs import internals as libinternals
/home/ana/anaconda2/envs/py2_parcels/lib/python2.7/site-packages/pandas/core/sparse/array.py:33: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
import pandas._libs.sparse as splib
/home/ana/anaconda2/envs/py2_parcels/lib/python2.7/site-packages/pandas/core/window.py:36: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
import pandas._libs.window as _window
/home/ana/anaconda2/envs/py2_parcels/lib/python2.7/site-packages/pandas/core/groupby/groupby.py:68: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from pandas._libs import (lib, reduction,
/home/ana/anaconda2/envs/py2_parcels/lib/python2.7/site-packages/pandas/core/reshape/reshape.py:30: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from pandas._libs import algos as _algos, reshape as _reshape
/home/ana/anaconda2/envs/py2_parcels/lib/python2.7/site-packages/pandas/io/parsers.py:45: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
import pandas._libs.parsers as parsers
/home/ana/anaconda2/envs/py2_parcels/lib/python2.7/site-packages/pandas/io/pytables.py:50: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from pandas._libs import algos, lib, writers as libwriters
Traceback (most recent call last):
File "create_testfields.py", line 20, in
generate_testfieldset(xdim=5, ydim=3, zdim=2, tdim=15)
File "create_testfields.py", line 15, in generate_testfieldset
fieldset = FieldSet.from_data(data, dimensions)
File "/home/ana/anaconda2/envs/py2_parcels/lib/python2.7/site-packages/parcels/fieldset.py", line 84, in from_data
allow_time_extrapolation=allow_time_extrapolation, time_periodic=time_periodic, **kwargs)
File "/home/ana/anaconda2/envs/py2_parcels/lib/python2.7/site-packages/parcels/field.py", line 103, in init
self.data = self.reshape(self.data, transpose)
File "/home/ana/anaconda2/envs/py2_parcels/lib/python2.7/site-packages/parcels/field.py", line 288, in reshape
('Field %s expecting a data shape of a [ydim, xdim], [zdim, ydim, xdim], [tdim, ydim, xdim] or [tdim, zdim, ydim, xdim]. Flag transpose=True could help to reorder the data.')
AssertionError: Field %s expecting a data shape of a [ydim, xdim], [zdim, ydim, xdim], [tdim, ydim, xdim] or [tdim, zdim, ydim, xdim]. Flag transpose=True could help to reorder the data.

That's what Step 4 here http://oceanparcels.org/#installing is for:  Run `parcels_get_examples <path_to_the_desired_example_dir> to get all examples.

Ops, I had just copied the directory. Great, it seems to be working now. I've been able to complete the first tutorial "Running particles in an idealised field". I will continue to the next ones :)
Thank you very much for your help

I followed the steps in  http://oceanparcels.org/#installing
I kept receiving the same error 'No module named parcels'
until I separated it as follows, then it ran smoothly:
from parcels import FieldSet
from parcels import ParticleSet
from parcels import Variable
from parcels import ScipyParticle, ScipyInteractionParticle
from parcels import DiffusionUniformKh
from parcels import NearestNeighborWithinRange, MergeWithNearestNeighbor
Also you may need to install xarray if you did not install it already.


Open url with source code that is urlencoded

Currently the url can only contain the sequence id, https://neoneye.github.io/lodalab/?id=1113
It would be nice to have an url parameter ala code=mul%20$0,2 that opens the editor with that program loaded into the editor.
With urlencoding, it's possible to make a QR code containing an entire loda programs.

There is now a source url query parameter, for this purpose. Try this url:
https://neoneye.github.io/lodalab/?source=%3B hello world
mul %240%2C2


Failed to get namespace using GetFromDocker

Hello, when I try to get namespace from a docker container by its id function call fails with:
2018/07/18 13:13:08 Failed to get Linux namespace for pod default/pod-1 with error: Unable to find
container: d64735b6ad2c77732b62b293c7573a591b6fec4bb47df525b813b3ac4f04b732

But if I run docker inspect for this ID docker finds this container. Anything I am doing wrong?
[root@kube-3 ~]# docker inspect d64735b6ad2c77732b62b293c7573a591b6fec4bb47df525b813b3ac4f04b732
[
    {
        "Id": "d64735b6ad2c77732b62b293c7573a591b6fec4bb47df525b813b3ac4f04b732",
        "Created": "2018-07-18T15:00:15.785074265Z",
        "Path": "/bin/sh",
        "Args": [
            "-xec",
            "while true; do ( echo \"Doing nothing\"; sleep 30; ) done\n"
        ],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 29842,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2018-07-18T15:00:15.918309182Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },

You should be able to use GetFromPath instead of GetFromDocker. Have a look at https://github.com/vishvananda/netns/blob/master/netns_linux.go:
// ` gets a handle to a network namespace
// identified by the path
func GetFromPath(path string) (NsHandle, error) {
	fd, err := syscall.Open(path, syscall.O_RDONLY, 0)
	if err != nil {
		return -1, err
	}
	return NsHandle(fd), nil
}

You should be able to get the namespace from /var/run/docker/netns.  You can also do ln -s /var/run/docker/netns /var/run/netns and use GetFromName but that just calls GetFromPath in the background, but this is good only if you desire to also be able to control the network namespaces with ip netns (for example if you also run some bash commands on the netns)

Seems like the following is easier
CONTAINER="my_container"
PID=$(docker inspect ${CONTAINER} | jq .[0].State.Pid)
NETNS=/proc//${PID}/ns/net


Unit tests for "Type" under arrow/charp

A port of most of the unit tests under arrow/cpp/type-test.cc to csharp.

/cc @eerhardt @chutchinson

using System;

you'll need a copyright header here. Get one from another file in this same project.

Refers to: csharp/test/Apache.Arrow.Tests/TypeTests.cs:1 in 63ee7fc. [](commit_id = 63ee7fcb0eb4e6f2abc577b9271cdedfa33fe214, deletion_comment = False)

@pgovind thank you for your contribution. Can you open a JIRA issue for the changelog and add it to the PR title? see https://cwiki.apache.org/confluence/display/ARROW/Contributing+to+Apache+Arrow

Ok, so I see Stephen's point and I updated the PR to make it much narrower now. Like he suggested, I added the equality checks as static methods in the test case code. It's static because other unit tests will likely need those methods. I also didn't need to implement GetHashCode or IEquatable this way. If we really want IEquatable, we can implement it against https://issues.apache.org/jira/browse/ARROW-4666 when the need comes up.

Just waiting for the CI to finish now.

@xhochy @kszucs @wesm: This can be merged now


PEP PLU export - missing data fields for some lines: price, dept, etc.

Steve Dudek reported an error with PEP of GenPOS Rel 2.3.1 when exporting PLUs from the PLU Maintenance dialog of PEP using the Export button. The CSV text file generated contained some PLUs were missing fields such as price and dept.
For example in the following subset of an exported PLU file, see PLU numbers 3111, 3145, and 3274.
0000000003106,  , 1, NUTS MACADAMIA, 269, 6, 0, 57,0,1,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0, 2, 0, 0, 6
0000000003111,  , 1, PAPAYA -Solo SUNRISE,0,1,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0, 2, 0, 0, 6
0000000003112,  , 1, PAPAYA - MERIDOL, 129, 6, 0, 62,0,1,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0, 2, 0, 0, 6
0000000003125,  , 1, PEPPER HABANERO, 269, 6, 0, 68,0,1,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0, 2, 0, 0, 6
0000000003126,  , 1, PLUMCOT, 149, 6, 0, 72,0,1,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0, 2, 0, 0, 6
0000000003127,  , 1, POMEGANATE LARGE, 229, 6, 0, 74,0,1,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0, 2, 0, 0, 6
0000000003130,  , 1, PUMPKINS JUMBO, 59, 6, 0, 76,0,1,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0, 2, 0, 0, 6
0000000003133,  , 1, PUMPKINS WHITE, 59, 6, 0, 76,0,1,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0, 2, 0, 0, 6
0000000003137,  , 1, SAPOTE - WHITE, 129, 6, 0, 82,0,1,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0, 2, 0, 0, 6
0000000003138,  , 1, SAPOTE - BLACK, 129, 6, 0, 82,0,1,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0, 2, 0, 0, 6
0000000003139,  , 1, SAVORY, 299, 6, 0, 36,0,1,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0, 2, 0, 0, 6
0000000003144,  , 1, Tangerine Fall Glo, 229, 6, 0, 89,0,1,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0, 2, 0, 0, 6
0000000003145,  , 1, TOMATOES ITAL YELLOW,0,1,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0, 2, 0, 0, 6
0000000003156,  , 1, Midknight -Valencia, 169, 6, 0, 61,0,1,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0, 2, 0, 0, 6
0000000003165,  , 1, RADDICHIO TREVISO, 229, 6, 0, 78,0,1,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0, 2, 0, 0, 6
0000000003166,  , 1, CABBAGE- TUSCAN, 49, 6, 0, 12,0,1,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0, 2, 0, 0, 6
0000000003168,  , 1, RADDICHIO CASTLE FR, 229, 6, 0, 78,0,1,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0, 2, 0, 0, 6
0000000003169,  , 1, LETTUCE - CATALOGNA, 229, 6, 0, 43,0,1,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0, 2, 0, 0, 6
0000000003273,  , 1, BEETS - GOLDEN, 149, 6, 0, 8,0,1,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0, 2, 0, 0, 6
0000000003274,  , 1, POTATOES YAMS YELLOW,0,1,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0, 2, 0, 0, 6
0000000003276,  , 1, POTATOES YAMS WHITE, 149, 6, 0, 75,0,1,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0, 2, 0, 0, 6

Investigation using the PEP file supplied by Steve, it appears that the problem is due to the number of characters in the PLU mnemonic. If the PLU mnemonic has less than 20 characters, then the line of PLU data is correct. However if the PLU mnemonic is 20 characters, then the PLU data line is incorrect with missing price, dept, and other data elements.
Looking in the source code of the open source version of OpenGenPOS, it looks like the character array for the PLU mnemonic is not terminated with an end of string if the mnemonic is 20 characters long. Experimenting by adding an explicit end of line terminator has addressed the problem and using Steve's PEP data file, the modified version of OpenGenPOS PEP is now correctly exporting PLUs with a mnemonic of 20 characters.
0000000003106,  , 1, NUTS MACADAMIA, 269, 6, 0, 57,0,1,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0, 2, 0, 0, 6, 0, 0, 0, 0, 1
0000000003111,  , 1, PAPAYA -Solo SUNRISE, 129, 6, 0, 62,0,1,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0, 2, 0, 0, 6, 0, 0, 0, 0, 1
0000000003112,  , 1, PAPAYA - MERIDOL, 129, 6, 0, 62,0,1,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0, 2, 0, 0, 6, 0, 0, 0, 0, 1
0000000003125,  , 1, PEPPER HABANERO, 269, 6, 0, 68,0,1,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0, 2, 0, 0, 6, 0, 0, 0, 0, 1
0000000003126,  , 1, PLUMCOT, 149, 6, 0, 72,0,1,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0, 2, 0, 0, 6, 0, 0, 0, 0, 1
0000000003127,  , 1, POMEGANATE LARGE, 229, 6, 0, 74,0,1,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0, 2, 0, 0, 6, 0, 0, 0, 0, 1
0000000003130,  , 1, PUMPKINS JUMBO, 59, 6, 0, 76,0,1,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0, 2, 0, 0, 6, 0, 0, 0, 0, 1
0000000003133,  , 1, PUMPKINS WHITE, 59, 6, 0, 76,0,1,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0, 2, 0, 0, 6, 0, 0, 0, 0, 1
0000000003137,  , 1, SAPOTE - WHITE, 129, 6, 0, 82,0,1,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0, 2, 0, 0, 6, 0, 0, 0, 0, 1
0000000003138,  , 1, SAPOTE - BLACK, 129, 6, 0, 82,0,1,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0, 2, 0, 0, 6, 0, 0, 0, 0, 1
0000000003139,  , 1, SAVORY, 299, 6, 0, 36,0,1,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0, 2, 0, 0, 6, 0, 0, 0, 0, 1
0000000003144,  , 1, Tangerine Fall Glo, 229, 6, 0, 89,0,1,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0, 2, 0, 0, 6, 0, 0, 0, 0, 1
0000000003145,  , 1, TOMATOES ITAL YELLOW, 149, 6, 0, 86,0,1,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0, 2, 0, 0, 6, 0, 0, 0, 0, 1
0000000003156,  , 1, Midknight -Valencia, 169, 6, 0, 61,0,1,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0, 2, 0, 0, 6, 0, 0, 0, 0, 1
0000000003165,  , 1, RADDICHIO TREVISO, 229, 6, 0, 78,0,1,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0, 2, 0, 0, 6, 0, 0, 0, 0, 1
0000000003166,  , 1, CABBAGE- TUSCAN, 49, 6, 0, 12,0,1,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0, 2, 0, 0, 6, 0, 0, 0, 0, 1
0000000003168,  , 1, RADDICHIO CASTLE FR, 229, 6, 0, 78,0,1,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0, 2, 0, 0, 6, 0, 0, 0, 0, 1
0000000003169,  , 1, LETTUCE - CATALOGNA, 229, 6, 0, 43,0,1,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0, 2, 0, 0, 6, 0, 0, 0, 0, 1
0000000003273,  , 1, BEETS - GOLDEN, 149, 6, 0, 8,0,1,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0, 2, 0, 0, 6, 0, 0, 0, 0, 1
0000000003274,  , 1, POTATOES YAMS YELLOW, 149, 6, 0, 75,0,1,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0, 2, 0, 0, 6, 0, 0, 0, 0, 1
0000000003276,  , 1, POTATOES YAMS WHITE, 149, 6, 0, 75,0,1,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0, 2, 0, 0, 6, 0, 0, 0, 0, 1

fix added to github repository.


Scheduled Upgrades for AgriDigital Onfarm platform - 16th November,2022 during 5:30 P.M - 2:30 A.M AEDT

Please be advised of an important upgrade to the AgriDigital Onfarm & AgriDigital Insights. There will be scheduled downtime for the upgrades to occur on 16th November,2022 during 5:30 P.M - 2:30 A.M AEDT.
Updates will be provided to you via email. Thank you for your understanding.


SurfaceGo: Should mention disabling secure boot in the instructions

A user following the instructions in the dashboard, was getting a "secure boot failed" error.
Since we also don't have docs with instructions for the surface go, it would be great to have an extra step in the provisioning instructions.

Implemented by https://github.com/balena-os/balena-intel/pull/401
Will be available in 2.83.18+rev1 or higher


Scenario Start/End callbacks not fired in subcontexts

Something like the following does not work:
defmodule GlobalContext do
  use WhiteBread.Context

  scenario_starting_state fn state ->
    Application.ensure_all_started(:hound)
    Hound.start_session
    state
  end

  scenario_finalize fn _state ->
    Hound.end_session
  end

  # Global given/then/and/ect you want to share with all contexts
end

defmodule MyContext do
  use WhiteBread.Context

  subcontext GlobalContext
end

It looks like here that steps are respected:
https://github.com/meadsteve/white-bread/blob/master/lib/white_bread/context/setup.ex#L9
But below the scenario start/end events are not (or maybe I'm wrong).
Either way I can't get the code in those events to start hound without coping and pasting it into each file. Also since the contexts are loaded first it seems, I can't do a use HoundSetup either...

@mgwidmann this was initially deliberate as I thought it'd be confusing to have lots of callbacks potentially triggering (and the required rules around ordering).
How I would implement this would be using plain functions/modules:
defmodule GlobalContextHelper do

  def start(state) do
    Application.ensure_all_started(:hound)
    Hound.start_session
    state
  end

  def stop(_state)
    Hound.end_session
  end
end

defmodule MyContext do
  use WhiteBread.Context

  scenario_starting_state fn state ->
    GlobalContextHelper.start(state)
  end

  scenario_finalize fn state ->
    GlobalContextHelper.stop(state)
  end
end

It'll mean a little boiler plate in each executed context but it'll make the order of what's happening very explicit.
Do you think it would make the intended use clearer if I renamed subcontext to import_steps_from so you'd have something like:
defmodule MyContext do
  use WhiteBread.Context
  import_steps_from GlobalContext
end

Yeah, that would make it clearer for sure. I'd like though to not have to define a scenario_starting_state and scenario_finalize for each context and do something like use GlobalContextHelper, but it wont work because its not compiled yet. Would I have to put that into test/support (with the modification that phoenix adds to compile those in the test env)?
I'd prefer to not have to move code there since theres no integration testing in the test folder, seems odd to have that code compiled and available.

To be honest I never really anticipated there being that many contexts doing setup and finalise. The contexts are inspired by this cucumber implementation: http://behat.org/en/latest/user_guide/configuration/suites.html and when using this I've not normally seen more than a few top level contexts.  Or a few different contexts for different user types of the system.


Update admin-support.md

@jaimeo to review and schedule changes

@traya1 this is in the same situation as the other one I sent you. Different date: needs to go live 7/21.

Not a problem, I'll help guide the transition over to the internal repos.

@jaimeo and @traya1 Please resolve. Thanks

Hi @StuEwen,
Can you contact me on Teams so I can help you move the changes to the internal repos? Thanks!


Fixed possible memory leak

I just realised that this is for 1.0.2 which is long unsupported.  It doesn't look like it is applicable to 1.1.1 or master.

Yep. But we still using 1.0.2 and I decided to do this PR. Any chances to merge it to 1.0.2?

I think it very unlikely.


[Bugfix] Fix Qwen2-VL LoRA weight loading

FIX #11406
Upcoming PRs include:

Convert hf_to_vllm_mapper to static variable for all models
Support substr and subfix mapping

Thanks for fixing! Can you add a LoRA test for this model to avoid regressions?

Sure

@DarkLight1337 All lora tests have passed


Support for weatherapi.com

I noticed that from https://www.worldweatheronline.com/developer/api/ there is mention of a free API. This link takes you to another site, https://www.weatherapi.com/ which appears to be the same company. This second API provides a perpetual free tier for 20k requests per month which should be reasonable for a personal use self hosted wttr.in.
The API, however, seems to be a vastly different data format, with two different calls needed for current weather/forecast data. It also appears to be missing hourly data. If you're willing to forego hourly data it's possible to have a completely free self-hosted solution with https://www.weatherapi.com/, but it would require some changes in this project to manage the alternate data shape.

Yes, agree. There are several popular different sources (sometimes even free), that we could use.
We could gather a list of such services, and think about some modular structure, or better to say
data format for their support.
For the beginning we could transform all the data from different sources into the format that wttr.in
already uses, maybe though this format is not ideal, and should be changed into something new. But the core idea stays the same: for each new data source, a transformation file will be needed, that will become a hierarchical data structure and return a hierarchical data structure as output.
Maybe we need some generalized approach here: some data transformation tables, like jq, but for the whole data.
I think we should just start and see.
Your recent PR was a very big and useful step in this direction


No free tier available

This documentation is out of date. There is no free option available.
Is this going remain, or is there a bug?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 73d91616-a93f-84af-7900-5aa34b361ffd
Version Independent ID: f202a678-4853-d024-55c5-19f89558415c
Content: How to use the SendGrid email service (.NET)
Content Source: articles/sendgrid-dotnet-how-to-send-email.md
Service: multiple
GitHub Login: @georgewallace
Microsoft Alias: gwallace

@HamishBrowne Thank you for the feedback!  We are investigating this and will get back to you shortly.

@HamishBrowne Thank you for the feedback!  We are investigating this and will get back to you shortly.

@HamishBrowne You are correct, there is no longer a free option for SendGrid although there are free benefits for paid customers:

"Azure customers receive up to 25,000 emails per month for free with paid packages starting at only $9.95 per month."

@HamishBrowne You are correct, there is no longer a free option for SendGrid although there are free benefits for paid customers:

"Azure customers receive up to 25,000 emails per month for free with paid packages starting at only $9.95 per month."

@HamishBrowne The correct screenshots would be as shown below and the documentation will be updated shortly.

Thank you again for bringing this to our attention and for helping to improve our documentation!  We are going to close this thread as resolved but if there are any further questions regarding the documentation, please tag me in your reply and we will be happy to continue the conversation.

@HamishBrowne The correct screenshots would be as shown below and the documentation will be updated shortly.

Thank you again for bringing this to our attention and for helping to improve our documentation!  We are going to close this thread as resolved but if there are any further questions regarding the documentation, please tag me in your reply and we will be happy to continue the conversation.


The system cannot find the path specified.

Whenever I try to run : makestep 2 i get the following error
cd ./step2_spritesheet_to_generative_sheet; npm run generate
The system cannot find the path specified.
make: *** [Makefile:10: step2] Error 1
Im not sure what im supposed to be doing any help would be great!

Hi there! What folder are you running it in? What operating system are you running on? Can you try copy and paste the command in the terminal instead?

im running it in the following folder (  'C:/Users/hignacio/Documents/Generative_Gif_Engine-main' )
And im using Visual Studio Code and the
jalagar
/
Generative_Gif_Engine
engine

Are you on a Mac or PC? Also can you try run cd ./step2_spritesheet_to_generative_sheet individually before running npm run generate?

Im on PC , I tried running the following command cd ./step2_spritesheet_to_generative_sheet but once  I tried to run npm generate after it I got the following error

<EMAIL_ADDRESS>generate
node index.js

npm ERR! code ENOENT
npm ERR! syscall spawn bash
npm ERR! path C:\Users\hignacio\Documents\Generative_Gif_Engine-main\step2_spritesheet_to_generative_sheet
npm ERR! errno -4058
npm ERR! enoent spawn bash ENOENT
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent
npm ERR!      C:\Users\hignacio\AppData\Local\npm-cache_logs\2022-04-26T23_21_46_899Z-debug-0.log
PS C:\Users\hignacio\Documents\Generative_Gif_Engine-main\step2_spritesheet_to_generative_sheet>

Hmm that is very strange, try the answer here https://stackoverflow.com/questions/49415958/what-could-cause-an-error-related-to-npm-not-being-able-to-find-a-file-no-conte to reinstall the node modules. It might be you installed them globally or something and they can't be found.
You should see the output like this:
><EMAIL_ADDRESS>generate
> node index.js

generator started undefined
(node:48333) [DEP0147] DeprecationWarning: In future versions of Node.js, fs.rmdir(path, { recursive: true }) will be removed. Use fs.rm(path, { recursive: true }) instead

Hi @Nachi4265 going to close this issue, I will need some more information and feel free to create a discussion board item so others can see!


MusicXML Time Signature Parsing issues

We found several issues with the MusicXML parser while trying to parse this file: http://www.choeurlimours.com/private/partitions/P292_79_150421181329.mxl
The result of parsing this file is that there are many more time signatures in the resulting NoteSequence than in the MusicXML file, and many of them have a numerator of 0.
I did a little bit of investigation, and the issue is in _fix_time_signature. The two main problems were:

If there is a measure that contains nothing but <forward duration='16' /> without any notes, then this is interpreted as a duration of 0.
For part of this particular score, it has a time signature of 12/8, but in _fix_time_signature, that is interpreted using Fraction, which turns it in to 3/2. This means that the measure appears to have a new time signature since it's different from what was in the score.

@jsawruk would you have time to look into this?

I'll investigate it in a few days. Thanks for your initial analysis.

This is an unusual file. Finale and Sibelius imported the <forward duration='16' /> by simply omitting the measures, giving a fractured score layout. Dorico inserted a whole measure rest in those measures. My plan is to insert a rest when encountering a <forward> in some cases. It might be ok to do in all cases; I need to think about this more.

The file was created in MuseScore, and when you open the file with that program, it interprets it as an empty measure.
I think adding a rest to what would otherwise be an empty measure is a reasonable solution.

It definitely makes sense to add a whole measure rest if the <forward> is the only element in that measure. I am just wondering if it makes sense to add rest for any <forward> element. I don't think it would make sense to generalize, because the model might learn too many rests. I will just insert rests for the whole measure rest case.


Remove redundant evaluate plan in Sample Extraction code

In the sample extraction code, a plan is evaluated by the engine twice. This PR removes a redundant evaluation of that plan.

LGTM.


Standalone UMD for Code Playgrounds/CDN

A standalone UMD module would be nice when playing around with ReactDND in a code playground like jsFiddle or CodePen. You could just add a URL to the standalone script and access it via:
window.ReactDND
Creating a standalone bundle with webpack
node ./node_modules/webpack/bin/webpack.js --output-library ReactDND --output-library-target umd ./dist/ReactDND.min.js --output-file ReactDND-standalone.js

See standalone section in webpack for browserify users
Creating a standalone bundle with Browerify
npm install browserify derequire --save-dev

node ./node_modules/browserify/bin/cmd.js ./dist/ReactDND.min.js --standalone ReactDND | node ./node_modules/derequire/bin/cmd.js -f define -t _defi_ | node ./node_modules/derequire/bin/cmd.js -f require -t _dereq_ > ReactDND-standalone.js

It's in the dist folder.

@gaearon hmmm, I could of swore it didn't emit anything to the window object but it seems to be there now.
Looks like it was completely my oversight.


[4.0][Bug] SimpleMDE loads font-awesome and overrides line-awesome icons

A cosmetic issue, but one that leaves the user a bit confused.  The SimpleMDE fields loads the font-awesome css and changes the look and feel of the existing icons when line-awesome is used.
You can see an example at https://demo.backpackforlaravel.com/admin/monster/create#bigtexts
PHP VERSION:
PHP 7.3.1-1+ubuntu18.04.1+deb.sury.org+1 (cli) (built: Jan 13 2019 10:19:33) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.3.1, Copyright (c) 1998-2018 Zend Technologies
with Zend OPcache v7.3.1-1+ubuntu18.04.1+deb.sury.org+1, Copyright (c) 1999-2018, by Zend Technologies
with blackfire v1.24.3~linux-x64-non_zts73, https://blackfire.io, by Blackfire
LARAVEL VERSION:
laravel/framework                       v6.7.0
BACKPACK VERSION:
backpack/crud                           4.0.24
backpack/generators                     2.0.5
eduardoarandah/backpacklogviewer        1.0.6

Not an issue, but an option.  You can avoid that the FontAwesome is automagically downloaded by adding an attribute like this example
        $this->crud->addField([
            'name' => 'remarks',
            'label' => 'Remarks',
            'type' => 'simplemde',
            'simplemdeAttributes' => [
                'promptURLs' => true,
                'status' => false,
                'spellChecker' => false,
                'forceSync' => true,
                'autoDownloadFontAwesome' => false,
            ],
        ]);

Maybe it can be improved in the field itself by looking at the config

Hi @timbertens ,
Ugh. You're right. I noticed this a while back, and haven't since - I figured it was fixed. But I thought it was because of the icon_picker field type, not simplemde.
I agree this should be fixed - now that I see it, I can't unsee it 😄
@pxpm could you please take a look at this too? What do you think would be a good solution?
Cheers!

icon_picker did really cause issue... if any CRUD contains icon_picker  with iconset set to fontawesome, the following icon cannot display (for my case)...
fa fa-newspaper-o
fa fa-thumb-tack
fa fa-group
fa fa-files-o
fa fa-cloud-upload
Example: Icon Picker + File Picker in same CRUD

@tabacitu i don't create new issue... i think it is closely related. Thanks for the hard work 💪

@kiddtang indeed the iconpicker has a lineawesome-fontawesome conflict too. That's why in 4.1 (which will be launched soon enough) we're using la la-newspaper - see PR https://github.com/Laravel-Backpack/CRUD/pull/2554
Just tested everything on 4.1 - no more conflicts now between the two fonts. But you'll have to wait for 4.1 to come out to get the upgrade sorry. I don't think it's going to be more than 7-10 days though.


Diamond task.memory

Dear all,
I am trying to use geneshot on a workstation.
I installed nextflow & run the "nextflow run Golob-Minot/geneshot --help" successfully.
I then tried to run geneshot on a small dataset (2 samples)
nextflow \
    run \
    -with-docker ubuntu:20.04 \
    Golob-Minot/geneshot \
        --manifest manifest_test_1.csv \
        --output output/ \
        -w work/ \
        -resume

See below the steps that were successful:

executor >  local (15)
[-        ] process > Preprocess_wf:Barcodecop                               -
[45/e0f438] process > Preprocess_wf:TrimGalore (1)                           [100%] 2 of 2 ✔
[d3/2ded70] process > Preprocess_wf:Download_hg_index (Download human ref... [100%] 1 of 1 ✔
[5a/fb987c] process > Preprocess_wf:BWA_remove_human (Remove human reads)    [100%] 2 of 2 ✔
[-        ] process > Preprocess_wf:CombineReads:JoinFASTQ                   -
[3b/9cba79] process > countReads (Count the number of reads per sample)      [100%] 2 of 2 ✔
[81/632675] process > countReadsSummary (Summarize the number of reads pe... [100%] 1 of 1 ✔
[f3/e839ae] process > assembly_wf:assembly (De novo metagenomic assembly)    [100%] 2 of 2 ✔
[68/41c9aa] process > assembly_wf:prodigal (Identify protein-coding genes)   [100%] 2 of 2 ✔
[e7/dfc9e4] process > assembly_wf:parseGeneAnnotations (Summarize every a... [100%] 2 of 2 ✔
[2c/41afe2] process > assembly_wf:linclustRound1 (Cluster genes with simi... [100%] 1 of 1 ✔
[-        ] process > assembly_wf:dedupRound1                                -

And the error message:

Execution cancelled -- Finishing pending tasks before exit
Error executing process > 'assembly_wf:dedupRound1 (Deduplicate sequences by alignment with DIAMOND)'
Caused by:
Cannot invoke method toMega() on null object

I think I saw a related issue here: https://github.com/nextflow-io/nextflow/issues/244
But I'am unsure which file to modifiy:
.nextflow/assets/Golob-Minot/geneshot/main.nf
OR
.nextflow/assets/Golob-Minot/geneshot/modules/assembly.nf
Any help is welcomed

My apologies for not responding more promptly!
The first fix to try in this case is to use the specific version of the pipeline (v0.9) which is currently best supported. You can do this by modifying your code above to:
nextflow \
    run \
    -with-docker ubuntu:20.04 \
    Golob-Minot/geneshot \
       -r 0.9 \
        --manifest manifest_test_1.csv \
        --output output/ \
        -w work/ \
        -resume

Please do try this updated version of the pipeline and let me know if that helps to resolve your issue.

Feel free to open this if you haven't been able to resolve it with the suggestion above


Use resource_type for MiqReport#add_schedule

Instead of towhat, which is now deprecated.  This is causing warnings in the manageiq-api repo, most of which are fixed via https://github.com/ManageIQ/manageiq-api/pull/516 now.
Links

https://github.com/ManageIQ/manageiq/pull/17545
https://github.com/ManageIQ/manageiq-api/pull/516

@NickLaMuro Any reason this was closed?


MSSQL: fix tests

Q
A

Is bugfix?
✔

New feature?
❌

Breaks BC?
❌

Tests pass?
✔

Fixed issues

Merged. Thanks!


Microbit won't flash from Chromebook using bluetooth

Describe the bug
During flashing the chromebook repeatedly reports that the microbit is connected, then disconnected. The flashing then fails with either a "too many open connections" or "gatt error" error code, seemingly at random.  Frustratingly the app defaults to bluetooth, so I could not flash code to the microbit until I sent the code to makecode online and flashed from there via the USB.
To Reproduce
Steps to reproduce the behaviour. e.g.:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior

Screenshots/Recordings

micro:bit version

V2.0, brand new, out of the box today.

Device

Make: Asus
Model: chromebook
OS: version 130.0.6723.126
App Version [e.g. 2.8.8]
Additional context
Add any other context about the problem here.! Brand new microbit, 1st time using the app on this device. Never had problems with bluetooth stability before on this device.

Are you using the Android App on the Chromebook?
If that's the case, for Chromebooks it's recommended to use the MakeCode editor directly in the browser with WebUSB, as it generally provides a better experience.

I was using the app. I'll stick to the browser then, it's just frustrating
having the bluetooth not working because I've had this issue with iPads
before, but with V1. I was hoping to buy a set of Microbits to use with my
home education groups, but they'll be providing their own devices, so I'm
frustrated that I can't rely on the bluetooth flashing again with V2.
On Tue, 17 Dec 2024 at 10:02, Carlos Pereira Atencio <
@.***> wrote:

Are you using the Android App on the Chromebook?
If that's the case, for Chromebooks it's recommended to use the MakeCode
editor directly in the browser with WebUSB, as it generally provides a
better experience.
—
Reply to this email directly, view it on GitHub
https://github.com/microbit-foundation/microbit-android/issues/80#issuecomment-2548007360,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/BNM44NPJYOCTA43D5N4Z7PL2F7ZBXAVCNFSM6AAAAABS4VK3E6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNBYGAYDOMZWGA
.
You are receiving this because you authored the thread.Message ID:
@.***>

--
Visit our website http://www.stemventures.co.uk/, or follow us on
Instagram https://www.instagram.com/stem.ventures or Facebook
https://www.facebook.com/profile.php?id=61556115621075 for up-to-date
news and events.
By making a booking with us you agree to our terms of service
https://www.stemventures.co.uk/terms-of-service and terms and conditions
https://www.stemventures.co.uk/terms-and-conditions.