Run authz module tests

Work has been conducted on the authz module using the previous version of Atomkraft.
It could be fine in this repository:
https://github.com/rnbguy/Authz-Audit
In order to validate the prototype, we have to transpose the work in the new version.
It will also allow prioritise the next developments in terms of bug fixing, enhancements and new features.
First step: Tests execution
This part is crucial to get users feedback.

Configuration to set-up the testnet (that could be used by default as a replacement of the Docker part)
Write reactors (authz and bank) : We can start by the transactions we need and identify the missing parts, if any, regarding the modules APIs.
Import the traces and executing them
Result processing

Second step: Traces generation

Traces generation from existing model
Optimising model and traces format : The existing model wasn't written originally for end-to-end testing but unit testing. The model could be cleaned up to better understand the logic and generate traces with only the necessary information needed for testing.

First version of Mirel's authz reactor port is live here.

Potential new features after discussion between Rano, Tesnim and Hernán:

Predictable data structure in action steps for easy integration.
Probably implement Python data classes for direct deserialization.
Use the same class to implement a method that returns a summary - def summy(self): str.
Init action step should be processed in atomkraft library to setup the testnet.
Repeat codes at the beginning and end of each reactor should happen in Atomkraft library, hidden away from the reactor file.
These codes can include user-defined invariant checks, expected outcome comparisons, etc.


verify jsx new installation

verifying the jumpscale installation according to the following docs
https://github.com/threefoldtech/jumpscaleX/tree/development/docs/Installation

first  issue asking for password for local host
https://github.com/threefoldtech/jumpscaleX/issues/589

https://github.com/threefoldtech/jumpscaleX/issues/590
https://github.com/threefoldtech/jumpscaleX/issues/600
verified and solved but the other still exist

@Dinaamagdy could you have a look a PR #631 ? Issues #589 and #600 are solved if the installation prerequisites are followed correctly as described in the doc here

lets just restart installer and create new bugs if relevant
above referred things seem to be ok


Bug fixes

Is the pull request based off the lastest version?
Yes
What features have you added?
No
What bugs did you fix?
Uncomment function toAddress and in function getMessages change Message for getMessage on line 486
Is your code valid PSR-2?
Yes
Has anything in your pull request already been fixed?
Anything else?
Optional things below
[] Added your name to the credits
[] Add a update name below:

Looks good to me!


Credential Management API Article Appears Twice in /updates

I've verified that there's no duplicate article in the repo. Both cards link to the same URL.

One should be translated and the other english.

This should be fixed in the DevSite update, if not, please reopen.


Application metrics api

This should be ok to review commit-by-commit.
Note that the components are not set up in the model yet, so this is just for adding the code.

@olaaun Please review the last 8 commits after merge. The only significant changes are a bug fix and adding the components to metrics-proxy via the config model.

LGTM.
Perhaps we should cache metrics differently? The cache doesn't consider if two requests are for different consumers, so we could end up returning the wrong metric set

Good catch. I forgot the cache when I added consumer handling. Will create a new PR to fix.


[SPARK-12613] [SQL] Outer Join Elimination by Parent Join Condition

This PR is another enhancement to Optimizer. It does not conflict with the other PRs (https://github.com/apache/spark/pull/10567 and https://github.com/apache/spark/pull/10551).
Given an outer join (OJ) is involved in another join (called parent join PJ), when the join type of PJ  is inner, left-semi, left-outer and right-outer, checking if the join condition of the  PJ satisfies the following two conditions:

there exist null filtering predicates against the columns in the null-supplying side of PJ.
these columns are from OJ.

If having such join predicates, execute the elimination rules:

full outer -> inner if both sides of OJ have such predicates
left outer -> inner if the right side of OJ has such predicates
right outer -> inner if the left side of OJ has such predicates
full outer -> left outer if only the left side of OJ has such predicates
full outer -> right outer if only the right side of OJ has such predicates

If applicable, this can greatly improve the performance, since outer join is much slower than inner join, full outer join is much slower than left/right outer join.
BTW, since the rule is different from the rule in https://github.com/apache/spark/pull/10567, I did not merge them in the same one for simplifying the code review.

Test build #48631 has started for PR 10566 at commit d6a6e9c.

Test build #48631 has finished for PR 10566 at commit d6a6e9c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48631/
Test PASSed.

Merged build finished. Test PASSed.

btw i created this: https://issues.apache.org/jira/browse/SPARK-12616
seems like something you can do?

Sure, I can make a try! Thank you!

To you want to update this now?

Will do it tonight. Thanks!

Test build #51943 has started for PR 10566 at commit 1a9ebdf.

First, will add test cases to OuterJoinEliminationSuite tomorrow.
Second, the current fix does not cover all the possible cases. I need to get the inputs from you about the issues this PR is facing:
    val df = Seq((1, 2, "1"), (3, 4, "3")).toDF("int", "int2", "str").as("a")
    val df2 = Seq((1, 2, "1"), (5, 6, "5")).toDF("int", "int2", "str").as("b")
    val df3 = Seq((1, 3, "1"), (4, 6, "5")).toDF("int", "int2", "str").as("c")

    // Full -> Left
    val full2Left = df.join(df2, $"a.int" === $"b.int", "full")
      .join(df3, $"c.int" === $"a.int", "right").select($"a.*", $"b.*", $"c.*")

In the above case, the parent join condition $"c.int" === $"a.int" is not eligible for the current two ways we are using to decide if the predicates are null filtering.

The first way is based on the constraints. If the parent join is full outer, the parent join will not have any IsNotNull constraint. In the current constraint propagation, its constraints is Set.empty[Expression]. Thus, $"c.int" === $"a.int" is not eligible for using the first way.

The second way is based on the run-time evaluation, canFilterOutNull. This requires that all the attributes are from the same side. In the predicate $"c.int" === $"a.int", $"a.int" is from the left side, but $"c.int" is not from the left side. (Actually, $"c.int" is from the other side in the parent join.) Thus, it is not eligible for the second way too.

However, the parent join condition $"c.int" === $"a.int" is very common in the join condition. We definitely can use such predicates as null-filtering predicates. Maybe we can keep the original way as the third way, as shown in the following link:
https://github.com/gatorsmile/spark/blob/d6a6e9cc31b0f7547b35cf25884135ea65b03676/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala#L798-L799
Does that look good to you? Thanks! : )

Test build #51943 has finished for PR 10566 at commit 1a9ebdf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51943/
Test PASSed.

Merged build finished. Test PASSed.

The first way is based on the constraints. If the parent join is full outer, the parent join will not have any IsNotNull constraint. In the current constraint propagation, its constraints is Set.empty[Expression]. Thus, $"c.int" === $"a.int" is not eligible for using the first way.

Why isn't the constraint present?  We should fix that instead of inventing another unrelated way to reason about nullability.

The existing constraint propagation is bottom up. The join conditions of full-outer joins will not filter out NULL.
Here, it is top down. The join conditions of full-outer joins can filter out the NULL of the child outer joins.
Will open a separate PR for top-down constraint propagation. Thanks for your suggestions!

: ) Basically, top-down constraint propagation has been down in the optimizer rules:

  PushPredicateThroughJoin,

  PushPredicateThroughProject,

  PushPredicateThroughGenerate,

  PushPredicateThroughAggregate,

Plan to add a new rule in optimizer for constraints pushdown.

After more thinking, in my opinion, the best way is to add extra Filter between two Join and let the existing Filter-condition-based rule to do outer join elimination, but we need another rule to remove unnecessary Filter which only contains Null constraints.
Let me first create a PR to do Filter removal/cleaning.

The  will be based on

Thanks for the pull request. I'm going through a list of pull requests to cut them down since the sheer number is breaking some of the tooling we have. Due to lack of activity on this pull request, I'm going to push a commit to close it. Feel free to reopen it or create a new one. We can also continue the discussion on the JIRA ticket.


rewrite templates with JSX

I'm working on a proposal for tcomb-form, and I needed the bootstrap templates, so I quickly rewrote them. Maybe you can use this as a base. I tried reimplementing most of the logic that existed in uvdom-bootstrap (but haven't yet ported the templates from tcomb-form's boostrap directory)
https://github.com/Xananax/tcomb-bootstrap-templates-temp
There are also additional libraries, for modals and so on. They're written in my own horrendous style (comma first, etc), but I can run a quick regex search-and-replace if you want to use them. Only dependency is classnames (and React, obviously).

Alternatively, you can just tell me what matters to you in terms of coding style (apart from 2 spaces for indents and trailing commas) and I can submit a PR some weeks from now.

Thanks a lot @Xananax, at first sight seem very helpful. I'll take a closer look after the v0.7.7 release (not so far hopefully)


Fix example 2 in Get-CsTenantFederationConfiguration.md

Closes https://github.com/MicrosoftDocs/office-docs-powershell/issues/9062.

Docs Build status updates of commit 9cb8435:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

skype/skype-ps/skype/Get-CsTenantFederationConfiguration.md
:white_check_mark:Succeeded
View (skype-ps)

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

@dariomws

@tiburd Please merge this PR. Thanks

Docs Build status updates of commit c67ab94:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

skype/skype-ps/skype/Get-CsTenantFederationConfiguration.md
:white_check_mark:Succeeded
View (skype-ps)

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


RangeError on Chrome for Android only

The following snippet of code works well on all major browsers (desktop) and it works on Firefox for android as well as safari for IOS
const HijriT = Temporal.Now.zonedDateTime("islamic-umalqura", timeZone);
console.log(HijriT.day);

However on Chrome for Android it throws the following error:
RangeError: era must be ah, not bc

My chrome version is 142.0.6367.179
Tested no Android 14.
If it helps the error is coming from here

This looks like a bug in Intl.DateTimeFormat on this version of Chrome for Andriod. To confirm, run this code on the android device:
new Intl.DateTimeFormat(`en-US-u-ca-islamic-umalqura`, { era: 'short' })
  .formatToParts(new Date(2024, 04, 27))
  .find(part => part.type === 'era')
  .value

If it returns "AH" (as it does on Chrome for Mac 125.0.6422.77) then it's working properly. If it returns "BC" then there's a problem with the calendar data available on that version of Chrome.
It might be that this particular Android version was not compiled with non-Gregorian calendar data. You can check this by simply running this:
new Intl.DateTimeFormat(`en-US-u-ca-islamic-umalqura`, { era: 'short' })
  .formatToParts(new Date(2024, 04, 27))

Do the results look like Gergorian dates or Islamic calendar dates? This will be useful into to put in your bug report.
@sffc if this is indeed an Android-specific Chrome bug with Intl.DateTimeFormat, where should a bug report be filed?

Thank you for the swift reply. It seems that indeed the era is not being resolved properly. as:
console.log(
  "Era",
  new Intl.DateTimeFormat("en-US-u-ca-islamic-umalqura", { era: "short" })
    .formatToParts(new Date(2024, 4, 27))
    .find((part) => part.type === "era")?.value
);

Outputs "Era BC"
However the rest of the data seems to be working fine only the era seems problematic as:
console.log(
  "date",
  new Intl.DateTimeFormat(`en-US-u-ca-islamic-umalqura`, {
    era: "short",
  }).formatToParts(new Date(2024, 4, 27))
);

Prints
[
    {
        "type": "month",
        "value": "11"
    },
    {
        "type": "literal",
        "value": "/"
    },
    {
        "type": "day",
        "value": "19"
    },
    {
        "type": "literal",
        "value": "/"
    },
    {
        "type": "year",
        "value": "1445"
    },
    {
        "type": "literal",
        "value": " "
    },
    {
        "type": "era",
        "value": "BC"
    }
]

I would be happy to file a bug wherever it should be if I am directed.

Looks like this bug was already reported here: https://issues.chromium.org/issues/40856332
You'll want to watch that issue for status.
If you want to try making a PR to work around this issue in Temporal polyfills, we'd be happy to review it. Please file the PR in the proposal-temporal repo and the commit will eventually percolate down here to this repo.
The fix you'll probably want to try is to override the reviseIntlEra method in helperIslamic. Take a look at the implementation of this method in helperGregory for inspiration.

Thank you for the guidance @justingrant I appreciate it. I will raise a PR today or tomorrow.

@justingrant FYI. I have raised a PR for this here as advised.

@justingrant will the patch be published to npm? if yes, then I would appreciate it if I could know when

it will eventually, the production polyfill in the other repo needs to get caught up with upstream and that may take a while.
There's another polyfill from @arshaw that's probably closer to upstream at this point. FYI Adam, you may want to pick up this fix too.

@khawarizmus, I just released v0.2.5 of temporal-polyfill (aka the "fullcalendar" polyfill) and it contains an attempt at a bugfix for your issue. Would you mind testing?

Hey @arshaw. Thank you for the fix and apologies for taking some time to reply. I have already managed to get thinsg working by using pnpm-patch-i. I have a lot of code (including published libraries) that rely on the @js-temporal/polyfill hence switching to your polyfill might not be straight forward for me.
However I went ahead and tested your polyfill (using v0.2.5).
The following code
const HDate = FullTemporal.Now.zonedDateTime("islamic-umalqura", timeZone);
console.log(HDate.day);

console.log(
  new FullIntl.DateTimeFormat(locale.value, {
    era: "short",
    calendar: Calendars.UMM_AL_QURA,
  }).format(HDate.toInstant())
);

console.log(
  new FullIntl.DateTimeFormat(locale.value, {
    dateStyle: "full",
    calendar: Calendars.UMM_AL_QURA,
  }).format(HDate.toInstant())
);

outputs:
26
11/26/1445 BC, 10:18:22 AM
Monday, November 26, 1445 BC

The numbers are correct. the line console.log(HDate.day) doesn't throw an error. However the era is still wrong BC instead of AH. Note that the Months names i.e (November instead of Dull-Qidah) are wrong for both your polyfill and the patched version of @js-temporal/polyfill and it's something that should be fixed at Chrome level i suppose, unless you guys know of a way to fix it. currently i am just hard coding the months in an array and reaching to the right index when i want to display them.

Thanks for checking @khawarizmus. Glad my fix works for fixing the numbers, but sad the fix doesn't apply to Intl.DateTimeFormat. More extensive monkeypatching/polyfilling would need to happen on Intl.DateTimeFormat to get this to work as you guessed.


Disable encoding when nobody is connected

Is your feature request related to a problem? Please describe.
Running the container on my server, I realize it always uses about 85-90% of one CPU core, even if there is no one connected to the session.
Describe the solution you'd like
I guess the load is that high because video / audio encoding is permanently active. So as long as no one has logged in or after the last person has logged out, the encoding should automatically be paused.
Describe alternatives you've considered
–

Pretty sure this was implemented ages ago,
Sounds a lot more like firefox/chrome taking up cpu instead

Pretty sure this was implemented ages ago,
Sounds a lot more like firefox/chrome taking up cpu instead

Running top inside the freshly started container shows me that the neko process uses 80% CPU while everything else looks close to 0.

Running top inside the freshly started container shows me that the neko process uses 80% CPU while everything else looks close to 0.

Pretty sure you also noticed that that jumps significantly when you do connect to neko

Pretty sure you also noticed that that jumps significantly when you do connect to neko

Yeah, when connected it seems to go a bit over 100% depending on what is happening in the session, but not much more.
So is idling at 80% CPU expected behaviour? If yes, why is it that high?
I should probably have added that I am using nurdism/neko:firefox image with the compose file from the repo.

Yeah, when connected it seems to go a bit over 100% depending on what is happening in the session, but not much more.
So is idling at 80% CPU expected behaviour? If yes, why is it that high?
I should probably have added that I am using nurdism/neko:firefox image with the compose file from the repo.

Lemme see if I can reproduce with that image,
Since on my install it does drop down to 0% usage

Lemme see if I can reproduce with that image,
Since on my install it does drop down to 0% usage

I can indeed reproduce on nurdism/neko:firefox
Personally I've been using m1k1o/neko:latest

I can indeed reproduce on nurdism/neko:firefox
Personally I've been using m1k1o/neko:latest

I suppose that one is built from your dev branch?

I suppose that one is built from your dev branch?

You would have to ask @m1k1o yourself

You would have to ask @m1k1o yourself

Oops, how did I make that connection ;)

Oops, how did I make that connection ;)

@djmaze yes. I kept my master branch in sync with original repository and made my changes in dev branch.

@djmaze yes. I kept my master branch in sync with original repository and made my changes in dev branch.

So the question is if there is a tangible reason for this behaviour in the nurdism version or if the image build is somehow borked. I can e.g. see that your fork does not use a custom gstreamer build, so it does not seem to be needed anymore. Either way, I might as well switch to your version then, so thanks ;)

So the question is if there is a tangible reason for this behaviour in the nurdism version or if the image build is somehow borked. I can e.g. see that your fork does not use a custom gstreamer build, so it does not seem to be needed anymore. Either way, I might as well switch to your version then, so thanks ;)

Confirming that when using m1k1o/neko:latest, the neko process stays below 1% of CPU when idling.

Confirming that when using m1k1o/neko:latest, the neko process stays below 1% of CPU when idling.

I guess, the reason can be just old image on docker hub. nurdism/neko:firefox was last updated 9 months ago. At that time, that feature might have not even been implemented and pipelines were always running. nurdism/neko:latest was updated 2 months ago, that might work as expected.

I guess, the reason can be just old image on docker hub. nurdism/neko:firefox was last updated 9 months ago. At that time, that feature might have not even been implemented and pipelines were always running. nurdism/neko:latest was updated 2 months ago, that might work as expected.

Okay, and latest is always a firefox image? I can't discern that from the build script.

Okay, and latest is always a firefox image? I can't discern that from the build script.

@djmaze yes, both in my and nurdism's. There is also m1k1o/neko:chromium available. In nurdism's too, but that one has not been updated for a while. My fork uses ungoogled chromium with h265 support.

@djmaze yes, both in my and nurdism's. There is also m1k1o/neko:chromium available. In nurdism's too, but that one has not been updated for a while. My fork uses ungoogled chromium with h265 support.

Hi,
Thanks @djmaze for opening this issue and @GigaFyde and @m1k1o for spoting about old version of images.
Some graphs from m1k1o/neko:latest

Original images from nurdism are not working. Thanks @m1k1o I'll be following your work to see if I can help as I'm testing on serving multiple neko rooms


Deleting a run in the frontend does not remove the run from the list of runs.

If I delete a run in the frontend, the run is not removed from the list of runs.
Only after reloading the page, the run is deleted. The run should be deleted directly.

I double-checked it again, it does not work.
Maybe a good issue for @theelstner?

Had a look again, we still need to reload the page, but deleting itself works again: https://github.com/tira-io/tira/issues/362
(but the issue of this ticket remains: it would be better if we would not have to reload the page)

This works now in the vuetify UI.


Support file uploads within models

I'm thinking something like this:
import APIModel
import Realm

class User: Object, ApiTransformable {
     dynamic var name = ""
     dynamic var avatar = ""

     func JSONDictionary() -> [String:Anyobject] {
         var avatarPath = NSBundle.mainBundle().pathForResource("avatar", ofType:"jpg")

         return [
              "name": name,
              "avatar": NSData(contentsOfFile: avatarPath)
         ]
     }
}

I'll work on this next.

Closed by 0.10.0 release.


Revert from hard coding back to a page query

@theianjones I know you've probably tried something similar. I reverted back to just using a page query and used the same syntax I use on my blog. It worked on develop and then I ran gatsby build and it worked then too. If this doesn't work then my best guess is that it's something on netlify's side that isn't building correctly.

That's so weird it returns a completely empty array!

Ian, this should fix it. Make sure to clear the cache on Netlify. There's something with 1.0.13 of gatsby-theme-brain that seemed to have caused the issue.
All I know is when I explicitly used 1.0.12 in develop mode it worked but when I used 1.0.13 it returned an empty array for the pagequery.


Remove DEQMAP_RBCAON from SGID

Summary

Internal Dataset Name: SGID.ENVIRONMENT.DEQMAP_RBCAON
External Dataset Name: opensgid.environment.deqmap_rbcaon
ArcGIS Online Url: https://utah.maps.arcgis.com/home/item.html?id=0c4c3a7f24b84fae804e8f658f4e1b10

A short summary of the situation.
DEQ has determined that this table is no longer utilized or necessary, and has not been getting updated, and has requested removal from SGID.
Migration Guide
A short summary of how users affected by this deprecation can modify their workflows and projects to continue to get similar or improved functionality
It is either unused or, if it is being used, it is no longer correct data. They could get in contact with Rick Saathoff at DEQ/DERR if there are further questions.

This item will not be shelved. ~The replaced data is still accessible via our shelved policy in AGOL (a link to the shelved item).~
Action items

Assign a person who should complete the task by replacing name with their github @name.
Check [x] the box when the task is completed and add the date of completion.
~Strike~ out all items that do not apply.

Soft Delete
The purpose of the soft delete is to ensure that all of our users and applications have gracefully migrated off of the dataset. Soft deletes will remain in effect for 14 days. During this time, we will have the ability to restore the dataset to its original SGID offering(s). After these 14 days, the item is then ready for a hard delete.
Note: If this dataset is being replaced, then wait until the new data is publicly available before completing these steps:

[x] Prepend "(Mature Support)" to the end of the item title. (@nathankota , completed: 2022/03/03)
[x] Remove all tags other than "Deprecated". (@nathankota , completed: 2022/03/03)
[x] Add note to ArcGIS Online description and default popup noting when layer will be deleted and any replacement layer (@nathankota , completed: completed: 2022/03/03)
[x] Unshare item from Open Data (@nathankota , completed: 2022/03/03)
~[ ] Update the SGID Index row as deprecated by pasting the related Porter issue URL in the Deprecated field (name, completed: 2021/00/00)~ THERE IS NO STEWARDSHIP TABLE RECORD
[x] Mark ArcGIS Online item as deprecated in preparation for future deletion (@nathankota, completed: 2022/03/18)

[x] Change Authoritative field to d in SGID.META.AGOLItems to automatically set the Deprecated AGOL flag. Allow the d to persist through one run of Auditor - currently, Auditor runs daily at 5:00am (@jacobdadams , completed: 2022/03/03)

[x] After one successful run of Auditor, remove the row from the SGID.META.AGOLItems table.  It is recommended to complete the soft delete process before moving on to these steps. If you decide to skip the soft delete, note that you will need to incorporate some of those steps here.

[x] Manually remove data from the Internal SGID (@eneemann, completed: 2022/03/18)

~Deprecated database layers can be backup up on Google Drive > AGRC Share > Team Projects > SGID > [deprecated layers]~

[x] Remove ArcGIS Online item: (@nathankota , completed: 2022/03/21)

Manually delete the AGOL item
~-OR-~
~Unshare by changing the AGOL_ITEM_ID field in SGID.META.AGOLItems to something other than an Item ID and manually~ ~changing the sharing settings (remove the item from all SGID groups, remove Public sharing),~
~-OR-~
~Shelve the data by copying the row from SGID.META.AGOLItems to the AGOLItems_shelved table in AGOL and changing the~ ~AGOL_ITEM_ID field in SGID.META.AGOLItems to shelved or some other note.~
~Deprecated AGOL items can be backup on Google Drive > AGRC Share > Team Projects > SGID > deprecated layers.~

~[ ] Remove Farm from AGOL connection (name, completed: 2021/00/00)~
~[ ] Update relevant gis.utah.gov data pages (name, completed: 2021/00/00)~
~[ ] Add this porter url to the Deprecated field of the [Stewardship] (https://docs.google.com/spreadsheets/d/11ASS7LnxgpnD0jN4utzklREgMf1pcvYjcXcIcESHweQ/edit#gid=1) record (name, completed: 2021/00/00)~ THERE IS NO STEWARDSHIP TABLE RECORD
[x] Remove row from SGID.META.AGOLItems table (@nathankota , completed: 2022/03/18)
[x] Delete Google Drive data (@ZachBeck , completed: 2022/03/03)
[ ] Remove row from SGID.META.ChangeDetection (@stdavis , completed: 2021/00/00)
[ ] Remove data from forklift hashing and receiving (@stdavis , completed: 2021/00/00)
[ ] Remove row from data/hashed/changedetection.gdb/TableHashes (@stdavis , completed: 2021/00/00)

~### Shelve/Static~
~Choose one based on situation.~

~[ ] Upload to UtahAGRC/AGRC_Shelved folder in AGOL (New shelved item not already in AGOL) (name, completed: 2021/00/00)~
~[ ] Move existing AGOL item to AGRC_Shelved AGOL folder (shelving an item already in AGOL) (name, completed: 2021/00/00)~
~[ ] Upload to appropriate UtahAGRC/{SGID Category} folder in AGOL (for static datasets) (name, completed: 2021/00/00)~

~Add record to table.~

~[ ] Add record to AGOLItems_shelved table in ArcGIS Online  with shelved or static in the CATEGORY field (name, completed: 2021/00/00)~

:robot: Automation validation

Assign yourself or someone to check the item by replacing name with their github @name.
Check [x] the box and add the date of verification 2020/01/01 when the task is verified.
~Strike~ out all items that do not apply.

[ ] Remove data from Open SGID (@steveoh on 2021/00/00)
[ ] sgid-index (@steveoh on 2021/00/00)
[x] Auditor sets appropriate shelved/static/Deprecated information (@jacobdadams on 2022/03/04)

Are there service dependencies

Does an application depend on this data? No
[ ] Are clients querying this data with the WebAPI? (@steveoh on 2021/00/00)

Notification

[ ] Twitter (@steveoh)

Group Task Assignments

Check [x] the box when you have assigned all the tasks relevant to your group.

[x] Data Team (@nathankota)
[ ] Dev Team (@steveoh)
~[ ] Cadastre Team (@rkelson)~

conductor results for tasks - 176

check
status

nathankota has completed 9 out of 9 tasks
:+1:

jacobdadams has completed 2 out of 2 tasks
:+1:

eneemann has completed 1 out of 1 tasks
:+1:

ZachBeck has completed 1 out of 1 tasks
:+1:

@stdavis has completed 0 out of 3 tasks
:no_entry:

@steveoh has completed 0 out of 5 tasks
:no_entry:

conductor results for ENVIRONMENT.DEQMAP_RBCAON

check
status

internal sgid
:+1:

meta table
:+1:

stewardship
:no_entry: Did not find ENVIRONMENT.DEQMAP_RBCAON in the worksheet


Preliminary Julia docs

This PR adds our preliminary Julia documentation to readthedocs, mainly as links to the Julia readme

https://sphericart--84.org.readthedocs.build/en/84/


✨ feat(gentest): Allow custom headers and example config

🗒 Description
This PR also fixes a bug with the default config path which now points to ~/.eets/gentest/config.json instead of ~/.eets/gentest.
🔗 Related Issues
close #891
✅ Checklist

[ ] All: Set appropriate labels for the changes.
[ ] All: Considered squashing commits to improve commit history.
[ ] All: Added an entry to CHANGELOG.md.
[ ] All: Considered updating the online docs in the ./docs/ directory.
[ ] Tests: All converted JSON/YML tests from ethereum/tests have been added to converted-ethereum-tests.txt.
[ ] Tests: A PR with removal of converted JSON/YML tests from ethereum/tests have been opened.
[x] Tests: Included the type and version of evm t8n tool used to locally execute test cases:  e.g., ref with commit hash or geth 1.13.1-stable-3f40e65.
[x] Tests: Ran mkdocs serve locally and verified the auto-generated docs for new tests in the Test Case Reference are correctly formatted.

@danceratopz  let me know if this needs a changelog entry.

@danceratopz Let me know where should I document this so the team is aware.

@danceratopz Let me know where should I document this so the team is aware.

As this config will be limited to gentest (for starters), you could re-use gentest's docstrings in the HTML page for gentest under here:
https://ethereum.github.io/execution-spec-tests/main/library/cli/
which gets generated from:
https://github.com/ethereum/execution-spec-tests/blob/619e7d8340802f457f5f4e1ada0bff3826a1c27c/src/cli/evm_bytes.py
and here:
https://github.com/ethereum/execution-spec-tests/blob/619e7d8340802f457f5f4e1ada0bff3826a1c27c/docs/library/cli/evm_bytes.md
But feel free to leave this until to you start work on gentest proper, it's ok for me if we can skip it for now.
If you're very motivated, you could add a short explainer under the developer docs:
https://ethereum.github.io/execution-spec-tests/main/dev/

I have made the following updates

Default configs: Now using a template. Also Default config is itself a Config model so it will also be validated
https://github.com/ethereum/execution-spec-tests/blob/7a5df435d4063f2936c09d89454ec39aa116b7e9/src/config/env.py#L89-L97

Overriding exiting environment: Show a friendly message

❯ uv run env_init
🚧 The configuration file 'execution-spec-tests/env.yaml' already exists. Please update it manually if needed.

Nudge user if the env.yaml does not exist

FileNotFoundError: The configuration file 'execution-spec-tests/env.yaml' does not exist. Run `uv run env_int` to create it.

If this looks okay then I will start documenting it.


Configuration to disable root

Hi,
first of all many thanks for working on this project. I really appreciate your effort and dedication.
I was playing around with the config parameters in installer-config.txt. If understood the documentation correctly, it shall be possible to disable the root account by settings rootpw as well as root_ssh_pubkey empty. So this is what I set:
username=pi
userpw=test
userperms_admin=1
rootpw=
root_ssh_pubkey=
root_ssh_pwlogin=0
However, after completing the installation I was still able to login with root/raspbian. Please correct me if my understanding is wrong. Otherwise there may be a bug somewhere.
In the meantime I use a workaround with post-install.txt containing
sed -E "s/^([root:]+:)([^:]+)(..+)$/\1!\3/g" -i /rootfs/etc/shadow
Looking forward to your feedback.
Thanks again.
BR, Mike

Hi,
Take a look at scripts/opt/raspberrypi-ua-netinst/install.sh on line 155
variable_set "rootpw" "raspbian"
maybe this needs to be blank, so therefore you can't login

@busmei Do you mean local or SSH login?

@FooDeas local


[Bug report] Crash in ExtendedImage.memory() on phones web browser

Version
8.0.1
Platforms
Web
Device Model
All Phone devices on web
flutter info
[✓] Flutter (Channel stable, 3.10.2, on macOS 13.4 22F66 darwin-arm64, locale en-US)
    • Flutter version 3.10.2 on channel stable at /Users/milad/Documents/development/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 9cd3d0d9ff (5 days ago), 2023-05-23 20:57:28 -0700
    • Engine revision 90fa3ae28f
    • Dart version 3.0.2
    • DevTools version 2.23.1

[✓] Android toolchain - develop for Android devices (Android SDK version 32.0.0-rc1)
    • Android SDK at ...
    • Platform android-33, build-tools 32.0.0-rc1
    • Java binary at: /Applications/Android Studio.app/Contents/jbr/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b802.4-9586694)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 14.3)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Build 14E222b
    • CocoaPods version 1.11.3

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2022.2)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b802.4-9586694)

[✓] VS Code (version 1.78.0)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.62.0

[✓] Connected device (2 available)
    • macOS (desktop) • macos  • darwin-arm64   • macOS 13.4 22F66 darwin-arm64
    • Chrome (web)    • chrome • web-javascript • Google Chrome 113.0.5672.126

[✓] Network resources
    • All expected network resources are available.

• No issues found!

How to reproduce?
In the new version, a very strange bug has occurred, in the web version of the application, it works well in computer browsers, but in phone browsers (Android and iOS), when we select the photo in the format Uint8List, and use in ExtendedImage.memory(photo),  the app crash and stop.
I changed all parameters of ExtendedImage.memory() but not work
Logs
No response
Example code (optional)
Column(children: <Widget>[

            Expanded(
          child: ExtendedImage.memory(
            _photoWeb!, /// Uint8List type
            fit: BoxFit.contain,
            mode: ExtendedImageMode.editor,
            enableLoadState: true,
            extendedImageEditorKey: editorKey,
            initEditorConfigHandler: (ExtendedImageState? state) {
              return EditorConfig(
                maxScale: 8.0,
                cropRectPadding: const EdgeInsets.all(20.0),
                hitTestSize: 20.0,
                cropLayerPainter: _cropLayerPainter!,
                initCropRectType: InitCropRectType.imageRect,
                cropAspectRatio: _aspectRatio!.value,
              );
            },
            cacheRawData: true,
          ),
        ),
      ])

Contact
<EMAIL_ADDRESS>

what version of extended_image_library do you use ?

try to use 3.4.2 for extended_image_library, and any error log?

oh. i got it, you check this issue
https://github.com/fluttercandies/extended_image/issues/546#issue-1571393515

8.0.2 is ready for this.


OssFuzz: Do not read/use undefined values in UndoXYB

Refactor code to extract common part. This allows to deal with MSAN in one place (instead of 6).

@deymo PTAL. In the next PR I'll add lib/sanitizers.h and after that apply unpoison-process-poison scheme to this code.

@deymo PTALl nota-bene, now MSAN fails at later stage, but I think PR is big enough, to split more fixes in other PR


Add sample app that uses Gemini function calling to interact with BQ

This PR adds:

Source code for SQL Talk demo app
App uses function calling in Gemini to interact with BQ in natural language
README includes details on how to run the app, deploy the app, and links to learning resources

Sample app is deployed at https://sql-talk-r5gdynozbq-uc.a.run.app/

Thanks @holtskinner for the review!


Import requests for instructor training

We get ~10 requests/week for instructor training. They're currently stored in the Google Docs spreadsheet that the Google Form puts them in; we should build something similar to the 'import instructor data' tool to import that data, but then we need to figure out how to store it:

Add a flag to Person to indicate that they want instructor training?  (I don't like this.)
Create a dummy Event with unspecified dates and location and register wannabe instructors in it?  (I don't much like this either.)
Create a new table that's just a list of people who want instructor training?  (I could live with this.)

We will also then need something to track people who've asked for instructor training and have:

Been in a training event.
Completed training.

Thoughts on UI and implementation?

This one will become less important once we change the way we do instructor training, so it is no longer essential.

After some more thought, I think it would be useful to have this.

Will this one be ready for V1.2?

@gvwilson do you have some idea how you want this implemented?

It's a lower priority right now - we're not going to take more
instructor training applications until 2016.

I'd like to revive this one - a form that people can fill in to request instructor training for themselves or a group would be useful.

This has become a higher priority...

@gvwilson @pbanaszkiewicz  I would like to work on this one, this is something along the lines of my GSOC proposal, and would only help in the long run.

We will want something similar to the form used to request a workshop
[1] that creates an "I would like instructor training" record for an
individual, and another for requesting training for groups.  Please do
HTML-only mockups and post for us to review the appearance, and then
we'll talk workflow.
Thanks!
[1] https://amy.software-carpentry.org/workshops/swc/request/

@gvwilson
extending from the idea in https://github.com/numfocus/gsoc/issues/99 , would you like to have a form field for username and password , for them to login later.

Not initially - I can't see them needing to log in until they're accepted.

okay thankyou, will be posting mockups soon :)

when an individual is applying

when applying as a group, dynamic tabs each containing forms to be filled.
we will be using django formset for this, although handling multiple forms on the backend for a group registration could be a little tricky to handle.
@gvwilson

Groups could be larger than 3 people, so we'll need an "add another
person" option.
Cheers,
Greg
On 2016-03-12 1:01 PM, shubhsingh594 wrote:

individual1
https://cloud.githubusercontent.com/assets/15982349/13724250/33d70c90-e8a6-11e5-8846-c985df3f8424.png
when an individual is applying
group
https://cloud.githubusercontent.com/assets/15982349/13724340/3fd6dd20-e8a8-11e5-9b4a-d6b6e1d42f89.png
when applying as a group, dynamic tabs each containing forms to be filled.
we will be using django formset for this, although handling multiple
forms on the backend for a group registration could be a little tricky
to handle.
@gvwilson https://github.com/gvwilson
—
Reply to this email directly or view it on GitHub
https://github.com/swcarpentry/amy/issues/419#issuecomment-195781588.

--
Dr Greg Wilson
Director of Instructor Training
Software Carpentry Foundation

@gvwilson
yeah it is dynamic, when we click oh the add person button , another tab will open -> person4
like so:

after clicking on the add person button

We don't need to import old data: we'll ask people to re-apply.


Android client fixes

Update upcoming summary footer
Run SchedulerService as a foreground service
Make task description optional

Codecov Report

Merging #3 (95c4c0a) into master (72cbf4a) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master       #3   +/-   ##
=========================================
  Coverage     94.53%   94.53%
  Complexity       67       67
=========================================
  Files             6        6
  Lines           238      238
  Branches         28       28
=========================================
  Hits            225      225
  Misses            4        4
  Partials          9        9

Flag
Coverage Δ
Complexity Δ

kotlin
94.53% <ø> (ø)
0.00 <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 72cbf4a...95c4c0a. Read the comment docs.


HADOOP-16445. Allow separate custom signing algorithms for S3 and DDB

This is very similar to the original patch.
In terms of testing - have run tests against a bucket in us-east-1 (including with the patch posted on HADOOP-16477). Struggling a bit with failures though, which seem completely unrelated to the patch. Still trying to get my test configuration file to a state where tests pass.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
97
Docker mode activated.

_ Prechecks _

+1
dupname
0
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+1
mvninstall
1454
trunk passed

+1
compile
36
trunk passed

+1
checkstyle
25
trunk passed

+1
mvnsite
37
trunk passed

+1
shadedclient
820
branch has no errors when building and testing our client artifacts.

+1
javadoc
31
trunk passed

0
spotbugs
71
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
70
trunk passed

_ Patch Compile Tests _

+1
mvninstall
45
the patch passed

+1
compile
36
the patch passed

+1
javac
36
the patch passed

+1
checkstyle
21
hadoop-tools/hadoop-aws: The patch generated 0 new + 33 unchanged - 1 fixed = 33 total (was 34)

+1
mvnsite
36
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
851
patch has no errors when building and testing our client artifacts.

+1
javadoc
27
the patch passed

+1
findbugs
71
the patch passed

_ Other Tests _

+1
unit
86
hadoop-aws in the patch passed.

+1
asflicense
32
The patch does not generate ASF License warnings.

3865

Subsystem
Report/Notes

Docker
Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1332/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1332

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle

uname
Linux 0fb3ea60aedb 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 2ae7f44

Default Java
1.8.0_222

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1332/1/testReport/

Max. process+thread count
434 (vs. ulimit of 5500)

modules
C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1332/1/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
90
Docker mode activated.

_ Prechecks _

+1
dupname
0
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+1
mvninstall
1255
trunk passed

+1
compile
35
trunk passed

+1
checkstyle
25
trunk passed

+1
mvnsite
39
trunk passed

+1
shadedclient
795
branch has no errors when building and testing our client artifacts.

+1
javadoc
25
trunk passed

0
spotbugs
58
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
56
trunk passed

_ Patch Compile Tests _

+1
mvninstall
32
the patch passed

+1
compile
26
the patch passed

+1
javac
26
the patch passed

+1
checkstyle
20
hadoop-tools/hadoop-aws: The patch generated 0 new + 33 unchanged - 1 fixed = 33 total (was 34)

+1
mvnsite
34
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
864
patch has no errors when building and testing our client artifacts.

+1
javadoc
25
the patch passed

+1
findbugs
71
the patch passed

_ Other Tests _

+1
unit
89
hadoop-aws in the patch passed.

+1
asflicense
30
The patch does not generate ASF License warnings.

3590

Subsystem
Report/Notes

Docker
Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1332/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1332

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle

uname
Linux 306f19c4b5ff 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / ee7c261

Default Java
1.8.0_222

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1332/2/testReport/

Max. process+thread count
306 (vs. ulimit of 5500)

modules
C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1332/2/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
79
Docker mode activated.

_ Prechecks _

+1
dupname
1
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+1
mvninstall
1193
trunk passed

+1
compile
30
trunk passed

+1
checkstyle
23
trunk passed

+1
mvnsite
35
trunk passed

+1
shadedclient
794
branch has no errors when building and testing our client artifacts.

+1
javadoc
26
trunk passed

0
spotbugs
58
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
56
trunk passed

_ Patch Compile Tests _

+1
mvninstall
32
the patch passed

+1
compile
26
the patch passed

+1
javac
26
the patch passed

+1
checkstyle
18
hadoop-tools/hadoop-aws: The patch generated 0 new + 33 unchanged - 1 fixed = 33 total (was 34)

+1
mvnsite
30
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
839
patch has no errors when building and testing our client artifacts.

+1
javadoc
22
the patch passed

+1
findbugs
60
the patch passed

_ Other Tests _

+1
unit
67
hadoop-aws in the patch passed.

+1
asflicense
29
The patch does not generate ASF License warnings.

3441

Subsystem
Report/Notes

Docker
Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1332/3/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1332

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle

uname
Linux 83581ae0122f 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / b69ac57

Default Java
1.8.0_222

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1332/3/testReport/

Max. process+thread count
353 (vs. ulimit of 5500)

modules
C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1332/3/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
64
Docker mode activated.

_ Prechecks _

+1
dupname
0
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+1
mvninstall
1175
trunk passed

+1
compile
38
trunk passed

+1
checkstyle
25
trunk passed

+1
mvnsite
40
trunk passed

+1
shadedclient
730
branch has no errors when building and testing our client artifacts.

+1
javadoc
28
trunk passed

0
spotbugs
60
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
58
trunk passed

_ Patch Compile Tests _

+1
mvninstall
36
the patch passed

+1
compile
28
the patch passed

+1
javac
28
the patch passed

+1
checkstyle
23
hadoop-tools/hadoop-aws: The patch generated 0 new + 33 unchanged - 1 fixed = 33 total (was 34)

+1
mvnsite
32
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
842
patch has no errors when building and testing our client artifacts.

+1
javadoc
27
the patch passed

+1
findbugs
65
the patch passed

_ Other Tests _

+1
unit
80
hadoop-aws in the patch passed.

+1
asflicense
31
The patch does not generate ASF License warnings.

3415

Subsystem
Report/Notes

Docker
Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1332/4/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1332

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle

uname
Linux 29262084f437 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 915cbc9

Default Java
1.8.0_222

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1332/4/testReport/

Max. process+thread count
415 (vs. ulimit of 5500)

modules
C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1332/4/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
42
Docker mode activated.

_ Prechecks _

+1
dupname
0
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+1
mvninstall
1124
trunk passed

+1
compile
34
trunk passed

+1
checkstyle
26
trunk passed

+1
mvnsite
38
trunk passed

+1
shadedclient
720
branch has no errors when building and testing our client artifacts.

+1
javadoc
27
trunk passed

0
spotbugs
59
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
57
trunk passed

_ Patch Compile Tests _

+1
mvninstall
35
the patch passed

+1
compile
27
the patch passed

+1
javac
27
the patch passed

+1
checkstyle
19
hadoop-tools/hadoop-aws: The patch generated 0 new + 31 unchanged - 1 fixed = 31 total (was 32)

+1
mvnsite
32
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
749
patch has no errors when building and testing our client artifacts.

+1
javadoc
26
the patch passed

+1
findbugs
64
the patch passed

_ Other Tests _

+1
unit
75
hadoop-aws in the patch passed.

+1
asflicense
33
The patch does not generate ASF License warnings.

3227

Subsystem
Report/Notes

Docker
Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1332/5/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1332

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle

uname
Linux 714a7ba8d01b 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / e04b8a4

Default Java
1.8.0_222

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1332/5/testReport/

Max. process+thread count
447 (vs. ulimit of 5500)

modules
C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1332/5/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
46
Docker mode activated.

_ Prechecks _

+1
dupname
1
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+1
mvninstall
1116
trunk passed

+1
compile
32
trunk passed

+1
checkstyle
29
trunk passed

+1
mvnsite
42
trunk passed

+1
shadedclient
736
branch has no errors when building and testing our client artifacts.

+1
javadoc
28
trunk passed

0
spotbugs
62
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
60
trunk passed

_ Patch Compile Tests _

+1
mvninstall
34
the patch passed

+1
compile
32
the patch passed

+1
javac
32
the patch passed

+1
checkstyle
24
hadoop-tools/hadoop-aws: The patch generated 0 new + 31 unchanged - 1 fixed = 31 total (was 32)

+1
mvnsite
35
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
748
patch has no errors when building and testing our client artifacts.

+1
javadoc
28
the patch passed

+1
findbugs
61
the patch passed

_ Other Tests _

+1
unit
80
hadoop-aws in the patch passed.

+1
asflicense
31
The patch does not generate ASF License warnings.

3253

Subsystem
Report/Notes

Docker
Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1332/6/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1332

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle

uname
Linux 1c1df7add7e1 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / e04b8a4

Default Java
1.8.0_222

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1332/6/testReport/

Max. process+thread count
412 (vs. ulimit of 5500)

modules
C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1332/6/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

we also use the credentials for talking to STS (session credentials) and I have an ambition to talk to Amazon SQS to subscribe to changes in an S3 bucket for spark streaming. As a result, I'm thinking about how to make that possible after your changes.
I'm also slowly trying to stop S3AUtils getting any worse as the one stop "throw all our static static stuff in here.  S3AUtils is a merge troublespot and has grown too big. I'm reluctant to move existing stuff, because it makes backporting so hard, but I'd like to make the amount of new stuff we had there nearly non-existent. In particular #970 is adding a new class org.apache.hadoop.fs.s3a.impl.NetworkBinding where networking stuff can go; I'm not sure the best place for authentication stuff. Maybe a class alongside that org.apache.hadoop.fs.s3a.impl.AwsConfigurationFactory. What do you think?
Also, rather than a new method createAwsConfForDdb alongside the existing one, I'd rather than existing one was extended to take a string declaring what the configurations for, e.g : "s3", "ddb", "sts" ... I'm proposing a string over an enum to maintain binary compatibility for applications which add/use a new service. If the name is unknown, we could just warn and return the "default" configuration.
So how about adding the new operations into some into some AwsConfigurationFactory.createAwsConf(string purpose,...) method, with S3AUtils.createAwsConf() tagged as deprecated and invoking the new method. I'm reluctant to cut it, as I suspect some people (me) have been using it elsewhere.
With that, I'm now going to to some minor review of the patch at the line-by-line level. Those comments come secondary to what I've just suggested.

Removed the new method in S3AUtil, and introduced a SignerManager (more changes coming to this soon). Also incorporated one of the changes from the patch on HADOOP-16505 which uses the correct way to initialize signers.
On the AwsConfigurationFactory - that makes sense. However, I don't think that should be in this patch. It's unrelated, and will end up moving more code from S3AUtils (including some public static methods). That's better handled in a separate refactoring only patch.

Updated. Have left the timeouts on the test, since these tests don't need the default 10 minute timeout.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
39
Docker mode activated.

_ Prechecks _

+1
dupname
1
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 2 new or modified test files.

_ trunk Compile Tests _

+1
mvninstall
1024
trunk passed

+1
compile
35
trunk passed

+1
checkstyle
28
trunk passed

+1
mvnsite
39
trunk passed

+1
shadedclient
739
branch has no errors when building and testing our client artifacts.

+1
javadoc
30
trunk passed

0
spotbugs
59
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
57
trunk passed

_ Patch Compile Tests _

+1
mvninstall
33
the patch passed

+1
compile
27
the patch passed

-1
javac
27
hadoop-tools_hadoop-aws generated 2 new + 15 unchanged - 0 fixed = 17 total (was 15)

-0
checkstyle
20
hadoop-tools/hadoop-aws: The patch generated 16 new + 34 unchanged - 1 fixed = 50 total (was 35)

+1
mvnsite
32
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
791
patch has no errors when building and testing our client artifacts.

+1
javadoc
29
the patch passed

+1
findbugs
76
the patch passed

_ Other Tests _

+1
unit
96
hadoop-aws in the patch passed.

+1
asflicense
33
The patch does not generate ASF License warnings.

3240

Subsystem
Report/Notes

Docker
Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1332/7/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1332

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle

uname
Linux ca9b87211251 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 01d7924

Default Java
1.8.0_222

javac
https://builds.apache.org/job/hadoop-multibranch/job/PR-1332/7/artifact/out/diff-compile-javac-hadoop-tools_hadoop-aws.txt

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-1332/7/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1332/7/testReport/

Max. process+thread count
415 (vs. ulimit of 5500)

modules
C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1332/7/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

Had forgotten how difficult it is to get a patch into Hadoop. Are there any published coding standards or templates for IntelliJ / Eclipse that you're aware of?
There's multiple revisions of the patch just fixing checkstyles, which is rather annoying.

Had forgotten how difficult it is to get a patch into Hadoop. Are there any published coding standards or templates for IntelliJ / Eclipse that you're aware of?

not AFAIK. Spark has some automated checks for import ordering but as there's no IDE config which works 100% of the time there either, also a PITA. But as it is 100% consistent, it is probably better at avoiding a lot of backport-merge-hell, which imports always seem to generate

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
162
Docker mode activated.

_ Prechecks _

+1
dupname
0
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 2 new or modified test files.

_ trunk Compile Tests _

+1
mvninstall
1293
trunk passed

+1
compile
34
trunk passed

+1
checkstyle
29
trunk passed

+1
mvnsite
40
trunk passed

+1
shadedclient
809
branch has no errors when building and testing our client artifacts.

+1
javadoc
31
trunk passed

0
spotbugs
73
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
69
trunk passed

_ Patch Compile Tests _

+1
mvninstall
38
the patch passed

+1
compile
32
the patch passed

-1
javac
32
hadoop-tools_hadoop-aws generated 2 new + 15 unchanged - 0 fixed = 17 total (was 15)

+1
checkstyle
21
hadoop-tools/hadoop-aws: The patch generated 0 new + 34 unchanged - 1 fixed = 34 total (was 35)

+1
mvnsite
38
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
869
patch has no errors when building and testing our client artifacts.

+1
javadoc
27
the patch passed

+1
findbugs
68
the patch passed

_ Other Tests _

+1
unit
86
hadoop-aws in the patch passed.

+1
asflicense
32
The patch does not generate ASF License warnings.

3791

Subsystem
Report/Notes

Docker
Client=19.03.2 Server=19.03.2 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1332/8/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1332

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle

uname
Linux b6fdcb367432 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / e97f0f1

Default Java
1.8.0_222

javac
https://builds.apache.org/job/hadoop-multibranch/job/PR-1332/8/artifact/out/diff-compile-javac-hadoop-tools_hadoop-aws.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1332/8/testReport/

Max. process+thread count
411 (vs. ulimit of 5500)

modules
C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1332/8/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
100
Docker mode activated.

_ Prechecks _

+1
dupname
0
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 4 new or modified test files.

_ trunk Compile Tests _

+1
mvninstall
1208
trunk passed

+1
compile
32
trunk passed

+1
checkstyle
23
trunk passed

+1
mvnsite
37
trunk passed

+1
shadedclient
806
branch has no errors when building and testing our client artifacts.

+1
javadoc
24
trunk passed

0
spotbugs
59
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
57
trunk passed

_ Patch Compile Tests _

+1
mvninstall
34
the patch passed

+1
compile
29
the patch passed

+1
javac
29
the patch passed

+1
checkstyle
21
hadoop-tools/hadoop-aws: The patch generated 0 new + 42 unchanged - 1 fixed = 42 total (was 43)

+1
mvnsite
35
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
875
patch has no errors when building and testing our client artifacts.

+1
javadoc
23
the patch passed

+1
findbugs
64
the patch passed

_ Other Tests _

+1
unit
85
hadoop-aws in the patch passed.

+1
asflicense
31
The patch does not generate ASF License warnings.

3582

Subsystem
Report/Notes

Docker
Client=19.03.2 Server=19.03.2 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1332/9/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1332

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle

uname
Linux 712a7a1f798a 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / f16eb09

Default Java
1.8.0_212

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1332/9/testReport/

Max. process+thread count
310 (vs. ulimit of 5500)

modules
C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1332/9/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

Tested against us-east-2 and ap-south-1 (-Ds3guard -Ddynamo -Dauth). There's a standard set of test failures with and without the patch.
[ERROR]   ITestMagicCommitMRJob>AbstractITCommitMRJob.testMRJob:146->AbstractFSContractTestBase.assertIsDirectory:327 » FileNotFound
[ERROR]   ITestDirectoryCommitMRJob>AbstractITCommitMRJob.testMRJob:146->AbstractFSContractTestBase.assertIsDirectory:327 » FileNotFound
[ERROR]   ITestPartitionCommitMRJob>AbstractITCommitMRJob.testMRJob:146->AbstractFSContractTestBase.assertIsDirectory:327 » FileNotFound
[ERROR]   ITestStagingCommitMRJob>AbstractITCommitMRJob.testMRJob:146->AbstractFSContractTestBase.assertIsDirectory:327 » FileNotFound
[ERROR]   ITestDynamoDBMetadataStore.setUp:159 » TestTimedOut test timed out after 10000...
[ERROR]   ITestS3GuardConcurrentOps.testConcurrentTableCreations:178->deleteTable:87 » IllegalArgument

last two are intermittent and we should have a go at fixing (longer timeouts, more retries).
other ones are new. I will have to run locally to see if they occur for me, and if not, assume its some config quirk.
Do bear in mind those MR test jobs are spawning new processes. Anything you do for FS test setup in the JUnit process has no effect there

tests all went well for me

I think you've managed to get stuff into the latest commit which weren't there earlier (SignerManager). Was that your intent? Because that's a bigger change. I'm happy with the smaller patch and its details; lets get that in before the next step

The change the latest commit (SignerManager) removes the code from S3AUtils and moves it to a SignerManager class (non-static). The actual signer registration has picked up bits from the patch on HADOOP-16505 (duplicate of this jira), since that was doing the registration in a better way (once per unique signer per JVM, rather than each time a FS instances is created).
I do plan to make additional changes to this class in a follow up patch, but it's essentially the same as the old patch, with a non-static Closeable class, without adding an extra static method to S3AUtils. Please let me know if there's specific concerns around this (i.e. change back to what?)

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
98
Docker mode activated.

_ Prechecks _

+1
dupname
0
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 4 new or modified test files.

_ trunk Compile Tests _

+1
mvninstall
1192
trunk passed

+1
compile
32
trunk passed

+1
checkstyle
24
trunk passed

+1
mvnsite
36
trunk passed

+1
shadedclient
793
branch has no errors when building and testing our client artifacts.

+1
javadoc
25
trunk passed

0
spotbugs
56
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
55
trunk passed

_ Patch Compile Tests _

+1
mvninstall
34
the patch passed

+1
compile
27
the patch passed

+1
javac
27
the patch passed

+1
checkstyle
19
hadoop-tools/hadoop-aws: The patch generated 0 new + 42 unchanged - 1 fixed = 42 total (was 43)

+1
mvnsite
31
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
825
patch has no errors when building and testing our client artifacts.

+1
javadoc
22
the patch passed

+1
findbugs
60
the patch passed

_ Other Tests _

+1
unit
81
hadoop-aws in the patch passed.

+1
asflicense
29
The patch does not generate ASF License warnings.

3485

Subsystem
Report/Notes

Docker
Client=18.09.7 Server=18.09.7 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1332/10/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1332

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle

uname
Linux 15cf078856e3 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / f260b5a

Default Java
1.8.0_222

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1332/10/testReport/

Max. process+thread count
308 (vs. ulimit of 5500)

modules
C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1332/10/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

tests all went well for me
Do you have fs.s3a.metadatastore.authoritative, fs.s3a.metadatastore.impl set in core-site.xml or auth-keys.xml

The MR tests seem quite broken to me otherwise. They use cluster configuration, which is created from new JobConf() - this will not pick up the test properties. Can get at least 2 of them to pass (haven't tried the other 2) with these 2 config keys set in core-site.xml or auth-keys.xml.
That said, I'm seeing references to "part-m-00000", "part-m-00001", "_SUCCESS" in S3Guard - with a 0 size, and marked as deleted, and cannot explain where these come from. Maybe some of the committer code runs on the client.

Patch LGTM, only thing needed is to get the imports consistent. I know its a PITA but its to keep merge hell vaguely under control.
You are going to need to document the new signing options in the markdown, rather than just the source code -either here or in a followup JIRA. That would also be good time to list what the standard AWS signers are as I don't see any real docs of them at all.

Updated the imports again. They should be consistent now.
java.*
\n
org.apache.*
\n
Everything else
\n
import static *

is the formatting options I'm using.
I'll update the docs in the next jira, which will make some changes to the value set in some of the config options.
Does the LGTM qualify as a +1? I'll go ahead and squash-merge the patch if that's the case.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
39
Docker mode activated.

_ Prechecks _

+1
dupname
1
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 4 new or modified test files.

_ trunk Compile Tests _

+1
mvninstall
1041
trunk passed

+1
compile
36
trunk passed

+1
checkstyle
29
trunk passed

+1
mvnsite
40
trunk passed

-1
shadedclient
110
branch has errors when building and testing our client artifacts.

+1
javadoc
31
trunk passed

0
spotbugs
60
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
57
trunk passed

_ Patch Compile Tests _

+1
mvninstall
34
the patch passed

+1
compile
28
the patch passed

+1
javac
28
the patch passed

+1
checkstyle
21
hadoop-tools/hadoop-aws: The patch generated 0 new + 42 unchanged - 1 fixed = 42 total (was 43)

+1
mvnsite
32
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

-1
shadedclient
29
patch has errors when building and testing our client artifacts.

+1
javadoc
24
the patch passed

+1
findbugs
59
the patch passed

_ Other Tests _

+1
unit
82
hadoop-aws in the patch passed.

+1
asflicense
26
The patch does not generate ASF License warnings.

1815

Subsystem
Report/Notes

Docker
Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1332/11/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1332

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle

uname
Linux e64d223753e4 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / b3173e1

Default Java
1.8.0_222

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1332/11/testReport/

Max. process+thread count
342 (vs. ulimit of 5500)

modules
C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1332/11/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

Also, the *MRJob tests pass after applying HADOOP-16591 or setting configs in auth-keys.xml

/retest

+1

Thanks for the reviews @steveloughran . Merging.
The shadedclient failure isn't related. See: https://github.com/apache/hadoop/pull/1432#issuecomment-531755823


Get series data time range

Is your feature request related to a problem? Please describe.
From paulverheul on discord

Hi all! Just a quick question ... what I'm trying to do is load a chart with the data for a day, and then load more data when users zoom out. In order to do this, I need the date/time for the left most chart boundary. Is there a way to get this? I expected the VisibleTimeRangeChange event would be of help, but that only return the time range of the data in the chart, not the time range of the complete chart.

Describe the solution you'd like
Add method series.getDataRange() to get most left (past) and right (recent) time from series data,
and the return value should be similar to chart.timeScale().getVisibleRange()
Additional context

I guess it's #335, isn't it?

I guess it's #335, isn't it?

Nope, this one is to get the actual timerange of the data from certain series regardless of what the viewport see.
For example if I have 2 year of daily data and set it to series, the viewport initially maybe only show about a month of data, the series.getDataRange() basically return the range of the data like
var data = [];
data.push({ time: '2018-02-18', value: 1234 });
data.push({ time: '2018-02-19', value: 1234 });
// ...
// more data here~
// ...
data.push({ time: '2020-02-17', value: 1234 });
data.push({ time: '2020-02-18', value: 1234 });

series.setData(data);

console.log(series.getDataRange()); // output: { from: '2018-02-18', to: '2020-02-18' } the real series data range
console.log(chart.timeScale().getVisibleRange()); // output: { from: '2020-01-18', to: '2020-02-18' } only viewport see


Auto-selection of space fails when there is only 1 space

If the user has only access to one space and there are multiple spaces, the single space is not selected.

Hard to reproduce this issue

Duplicate of: https://github.com/cloudfoundry-incubator/stratos/issues/2469


Allow links to specific line number of any file

This PR fixes #125320
It removes the check on Markdown files when navigating to a file with a specific line number.

[xxx](topic.md#L5) was working, and still works
[xxx](test.cpp#L3) was not working (opening the file but without revealing any specific line) and is now fixed

I ended pushing a fix since I was looking into a few similar bugs in the area. Please test out the next insiders build to make sure the fix works as expected


Accordion's "at most 1 open item" invariant gets violated after using JS API

Hello!
When you try to collapse/expand accordions with JS, reopening the collapsed accordion by clicking causes expanding both accordions.
See http://getbootstrap.com/javascript/#collapse-example-accordion

Originally posted to: https://github.com/patternfly/patternfly/issues/162

Collapse's toggle, show, and hide methods are asynchronous. It's not valid to call them twice sequentially like that. You need to wait for the hidden.bs.collapse event before making subsequent calls.

This PR would be usefull https://github.com/twbs/bootstrap/pull/17823 :smile:

Indeed! I do intend to get around to that PR, I just have a lot of things on my TODO list.

Haha it's ok good luck for your TODO list :wink:

Same happens when I call it like this:
$("#collapseOne").one('hidden.bs.collapse', function () {
  $("#collapseTwo").collapse('show');
});
$("#collapseOne").collapse('hide');

But even if I wait between typing in the two collapse calls.

Perhaps this is related to #17335.

This also prompts the question of whether this can be fixed in v3 given that some folks might be relying on the "broken" behavior.

IMO we should add data-parent in the collapse element because we allow user to toggle collapse element without passing by the <a> which contain the data-parent attribut.
That's why the collapse behavior is strange because they are no parent

X-ref #18418


[Question] Tooling changes

As promissed in https://github.com/dr-prodigy/python-holidays/issues/338#issuecomment-662592391 I'm working on a PR to change the CI from travis to github actions and also add CD.
Since you liked the idea of using bump2version, for releas process.
I was wondering if would also be interested in some other changes:

Autoformat code with black, so formatting code to look nice is a no brainer
Add  pre-commit to run back and flake8 on commit, like that  there won't be commits like fix flake8 which I saw a lot on other projects
Style check the tests with flake8, IMHO tests should also comply with styling rules
Use pytest as testrunner, pytest is more prowerful (and gives prettier output) than good old unittest and would also allow to restructure the tests to mimic the projects structure, which again would make it easier to find corresponding tests
Use tox so contributors can tests multiple python version locally

Also, since the coveralls action doesn't support xml reports, I would use codecov instead.

This would be so nice!

We are on the way to release this soon.. Closing this question, thx! 👍


Lacking responsiveness for mobile phones

Unfortunately, I don't really have a clue on how to tackle this, but I thought I'd report it anyway: if you try to use the shoutbox from a mobile phone, you will most likely end up with the on-screen keyboard hiding the input box. Furthermore, you cannot scroll up, so you basically have to type your shout text blindly.
Tested this on an Android device, I don't know about iOS.

Hm, I wonder if this is still an issue now that we've moved to a sidebar on mobile?

Currently, this just looks a bit weird (tried with an almost empty shoutbox - here, the textarea floats at the bottom until you activate the on-screen keyboard...). I will report back as soon as I have some more tests.

Be sure to update to the latest; that should like a bug from a few commits ago.

Oh, all right. Will do after my importer finishes (i.e. tomorrow...)

Works all right now.


/ts(d) unexpected behaviour with repeated headings

Context: I have lots of longish notes with many tasks covering some broad areas like "Car & Bike", "Finance", separated in a number of H3 sections, e.g. "Tax Return 2022". I generally want to keep enough history of when I've done things, in case I hit a problem later I need to refer back to.  So I move completed items to the "## Done" section, but under a similar H3 "Finance" etc. section.
Describe the bug: When I run /tsd from the H3 section with a mixture of open and completed tasks, I find all the completed tasks in the H2 Done > H3 section moved up into the H3 section with open tasks in.
Expected behavior: All items under the H2 Done section not to be changed let alone moved.
Suggestion: Add a simple call to helpers/paragraph.js::findEndOfActivePartOfNote() to only process items up to the H2 Done section.  I'm not trying this as a PR as I don't know the rest of this plugin's commands well enough to fiddle.
((In time I want smart auto-archiving down to the H2 Done section, but that's an issue for another time.)

I will have a look this weekend. Thx for this and the other one. I will try to address them together.

Done. Pls test. Thanks @jgclark


Testable createTextMaskInputElement

This will let you pass in your own function to easily mock and test, should be merged after the react 15.5 one.

Are you going to continue your work in this PR?


move react, react-dom to peerDependencies

react and react-dom being a peerDependencie prevents to duplicate react packages when bundling (with webpack or browserify), see this article for more context.
also, this approach is taken by other popular project

Thanks!


Add keda operator

Description
Changes proposed in this pull request:

Introduces Keda manager to install KEDA in kyma runtime
Defines Keda CRD that allows configuring Keda behavior to the end user

Related issue(s)
Fixes https://github.com/kyma-project/kyma/issues/15232
https://github.com/kyma-project/kyma/issues/15330

@moelsayed many, many thx for the review

I think the CR name should be something more specific than Keda, maybe KedaManager ?
https://github.com/kyma-project/keda-manager/pull/1#discussion_r967024607

I think we should fix the apiGroup name as it has a redundant operator there.
https://github.com/kyma-project/keda-manager/pull/1#discussion_r967002423

Resources in the chart should have more specific names and unique names, since this is supposed to be deployed with other similar components.

In my opinion we should stick to properties defined chart's values, that would allow for better adoption and is consistent.

I think It makes sense to make namespace configurable for the operator chart as well.

I think it suits better (if needed) here

@m00g3n Thank you for the replies.

In my opinion we should stick to properties defined chart's values, that would allow for better adoption and is consistent.
I am sorry, I didn't clarify. I completely agree with you regarding the keda chart. My comment was about the operator chart. It's very generic and doesn't specify which component is this.

As discussed, the remaining comments will be addressed in separate issues to unblock the PoC.


Broken interface on Ubuntu

See screenshot:

Running Ubuntu 20 with OpenJDK 17 from openjdk-17-jdk, running latest McDeob

I forgot to reply to this.
Since I don't have Linux, I won't be able to test this, and try to fix it.
I will leave this here in hopes someone who uses Linux sees this, and could potentially do a PR to fix

fixed in McDeob 2.2.0


Mention lack of support for AVIF sequences, fix #5736

Hello. Quick edit, hope I got it right.
https://gif2avif.com/browser-test/ explains it well - both Firefox and Chrome currently only support still AVIF images. Animated AVIFs, a.k.a. AVIF Sequences, don't seem to be supported.
Also, fixes #5736 - Firefox 86 removes the flag.

I think since the primary use of AVIF will still be static images support should still be y. Though I'm fine including the note to those re: AVIF sequences.
Support behind a flag should still be n FYI. Thanks!

I think since the primary use of AVIF will still be static images support should still be y. Though I'm fine including the note to those re: AVIF sequences.
Support behind a flag should still be n FYI. Thanks!

Looks good, thanks!

Looks good, thanks!


Special case for empty comments in doc comment kind

Part of #4103
Fix ui/empty/empty-comment.rs macros

bors r+
I wonder though, if we can/need fix something on the macro side as well.. It seems weird that macro non-panicing depends on an impl details of a function elsewhere. Perhaps we can lift some invariants to types or something...

I wonder though, if we can/need fix something on the macro side as well.. It seems weird that macro non-panicing depends on an impl details of a function elsewhere. Perhaps we can lift some invariants to types or something...

The panic is actually raised by here:
https://github.com/rust-analyzer/rust-analyzer/blob/f654f4943552888c2b8f079cf687970268d81f18/crates/ra_mbe/src/syntax_bridge.rs#L164-L177
It bascially try to convert a comment text to #[doc(sth)] , I think we could move this logic to ast::make ?

bors retry

So, I think semantically what we do here is that we extract contents of the comment. We have DocCommentsOwner::doc_comment_text which does a similar job. I probably would expect an ast::Comment::contents() -> String method to be used to implement both getting docs (which is stiched form many line comments) and for lowering doc comemnts to attributes.


Allow disabling SpacingBetweenDeclarationsWithAnnotationsRule for enum entries

Expected Behavior
Enabling SpacingBetweenDeclarationsWithAnnotationsRule should allow omitting new line for the enum values. I.e. the following should be allowed:
enum class Foo {
    @Annotation BAR_1,
    @Annotation BAR_2
}

The code style doesn't seem to forbid that. Visually such placement makes enums easier to read for some people, cause enum in this simple case visually does not differ much from a multiline constructor with annotated arguments.
This behavior should be configurable via flag.
Current Behavior
Currently the following is enforced:
enum class Foo {
    @Annotation BAR_1,

    @Annotation BAR_2
}

Additional information

Current version of ktlint: 0.46.1

We do not want to add flags for all kinds of minor tweaks.
In case you really dislike the newline, you can always disable the entire rule. Or on a case by case base you can suppress a rule for a specific enumeration:
@Suppress("ktlint:spacing-between-declarations-with-annotations")
enum class Foo {
    @Annotation BAR_1,
    @Annotation BAR_2
}


document browser-only support

Is your feature request related to a problem? Please describe.
We use Karma in a browser to run our tests. The local server that is created uses Node.js dependencies that can't be provided ('http', 'net', 'path', etc.) so the tests fail to run. I thought I wanted a new feature of browser-only support that appears to already exist 🎉 https://github.com/zivl/sentry-testkit/blob/v5.0.5/src/browser.ts
Describe the solution you'd like
Document this feature! (Sorry if I overlooked it 🙈 ). Our code ended up looking something like:
import sentryTestkit from 'sentry-testkit/dist/browser';

import { ErrorLogger } from './index';

const { sentryTransport, testkit } = sentryTestkit();

// abridged for clarity
const errorLogger = new ErrorLogger({
  transport: sentryTransport,
});

errorLogger.captureException(new Error('testing'));

testkit.reports()[0].error.message == 'testing'

Describe alternatives you've considered
I considered writing my own stripped-down transport but thankfully won't have to 😅
Additional context
Thanks for building this!

hey @maxbeatty!
Actually you're right, it will be a good point for documentation and THIS IS the reason why we added sentry-testkit/dist/browser :)
I'll try to add it soon, you're more than welcome to PR of-course


mad_rpc_open_port: client_register for mgmt 1 failed

Hi，
I encountered an error message when I exec "ibnetdiscover" or perfquery in command. So, I can't profile metrics by this exporter. Can you help me ?
thanks.
error log:
[root@test-k8snode-32-43 1]# ibnetdiscover
ibwarn: [298726] mad_rpc_open_port: client_register for mgmt 1 failed
/var/tmp/OFED_topdir/BUILD/rdma-core-2307mlnx47/libibnetdisc/ibnetdisc.c:802; can't open MAD port ((null):0)
ibnetdiscover: iberror: failed: discover failed
[root@test-k8snode-32-43 1]# perfquery
ibwarn: [299010] mad_rpc_open_port: client_register for mgmt 1 failed
perfquery: iberror: failed: Failed to open '(null)' port '0'
ibstatus is ok:
[root@test-k8snode-32-43 1]# ibstatus
Infiniband device 'mlx5_0' port 1 status:
default gid:     fe80:0000:0000:0000:966d:aeff:feb3:832e
base lid:        0x0
sm lid:          0x0
state:           4: ACTIVE
phys state:      5: LinkUp
rate:            400 Gb/sec (4X NDR)
link_layer:      Ethernet
Infiniband device 'mlx5_1' port 1 status:
default gid:     fe80:0000:0000:0000:966d:aeff:feec:c6a0
base lid:        0x0
sm lid:          0x0
state:           4: ACTIVE
phys state:      5: LinkUp
rate:            400 Gb/sec (4X NDR)
link_layer:      Ethernet

The IB devices appear to be configured as Ethernet and not Infiniband.  I can replicate the same errors on some of my systems where I've configured the Mellanox cards to do only Ethernet and not Infiniband.  When a device is setup as Ethernet it won't be able to communicate to the IB fabric in a way that's required by things like ibnetdiscover and perfquery.

thanks


Error building for web

Target dart2js failed: ProcessException: Process exited abnormally with exit code 1:
../../../.pub-cache/hosted/pub.dev/pdfrx-0.4.10/lib/src/widgets/pdf_page_text_overlay.dart:95:7:
Error: The non-abstract class '_PdfTextRenderBox' is missing implementations for these members:

Selectable.boundingBoxes
class _PdfTextRenderBox extends RenderBox with Selectable, SelectionRegistrant {
^^^^^^^^^^^^^^^^^
../../../snap/flutter/common/flutter/packages/flutter/lib/src/rendering/selection.dart:147:18:
Info: 'Selectable.boundingBoxes' is defined here.
List get boundingBoxes;
^^^^^^^^^^^^^
Error: Compilation failed.
Command: /home/ind/snap/flutter/common/flutter/bin/cache/dart-sdk/bin/dart --disable-dart-dev
/home/ind/snap/flutter/common/flutter/bin/cache/dart-sdk/bin/snapshots/dart2js.dart.snapshot
--platform-binaries=/home/ind/snap/flutter/common/flutter/bin/cache/flutter_web_sdk/kernel --invoker=flutter_tool -Ddart.vm.product=true
-DFLUTTER_WEB_AUTO_DETECT=true -DFLUTTER_WEB_CANVASKIT_URL=https://www.gstatic.com/flutter-canvaskit/04817c99c9fd4956f27505204f7e344335810aed/
--native-null-assertions --no-source-maps -o /home/ind/Documents/Mobile/sc/.dart_tool/flutter_build/846120b4c5c0513ff1e25f1352e8304a/app.dill
--packages=.dart_tool/package_config.json --cfe-only /home/ind/Documents/Mobile/sc/.dart_tool/flutter_build/846120b4c5c0513ff1e25f1352e8304a/main.dart

Duplicate of #52
Are you using master or beta? Or recently released 3.19?

Recently released 3.19

OK, anyway, 0.4.29 or later has fixed the issue. I recommend you should use 1.0.0+.

thanks I see its got an [ANALYSIS ISSUE]

I don't know actually, but it seems to be an analyzer issue??

if you put  sdk: '>=3.2.5 <4.0.0' problem the message will go away

Yes, but then, pdfrx now uses API introduced on 3.3.0-somewhere... How do I determine that...

Oh snap so leave it the way it is the message will go away on its own in about 2 months

https://github.com/dart-lang/web/blob/main/pubspec.yaml#L7

Oh, web 0.5.0 depends on somewhat earlier version
https://github.com/dart-lang/web/blob/a54a1f0447979f9a3ea220199eca849ffb214e91/pubspec.yaml

OK, I'll change the version to 3.3.0-76.0.dev anyway.

1.0.2 get back to the 3.3 beta :(

Again, get another :(

1.0.3 changes flutter version...

I've created an issue on dart-lang/pub-dev.

1.0.4 did rollback everything to pre-Flutter 3.19/Dart 3.3 environment...

I've added the following note to README.md anyway:
NOTE: On pub.dev, 1.0.0+ versions gets [ANALYSIS ISSUE]. It does not affect your code consistency but API reference is not available until pub.dev upgrades their toolchains.

The version discussion is moved to #77.


minor grammar fix

playback (noun) -> play back (verb)
there's one more instance in the About section of the project too fyi.

Thanks!


Sample annotations with same value but different dtypes are unloadable

I'm not sure if this belongs here or in the scanpy repo, and this is a hybrid bug and feature request.  This is also somewhat related to #31, as both issues stem from the same function.
If two sample annotations have the same "value" but different dtype, a dataset saved as h5ad becomes unreadable.  This stems from the way that the categories for 'object' typed annotations are defined.  Minimal example:
import scanpy.api as sc

# load any dataset:
dataset = sc.read('/path/to/dataset')
test1 = dataset.copy()[:5, :5]
test2 = dataset.copy()[:5, :5]

# add annotations
test1.obs['sampleid'] = 1
test2.obs['sampleid'] = '1'

test_combined = test1.concatenate([test2])

test_combined.save('test.h5ad')
test_combined = sc.read('test.h5ad')

The last line fails for me with:
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-85-2cfbeaca8a55> in <module>()
----> 1 test_combined = sc.read_h5ad('test.h5ad')

/projects/flynnb/software/anaconda/envs/scc/lib/python3.6/site-packages/anndata/readwrite/read.py in read_h5ad(filename, backed)
    343         # load everything into memory
    344         d = _read_h5ad(filename=filename)
--> 345         return AnnData(d)
    346
    347

/projects/flynnb/software/anaconda/envs/scc/lib/python3.6/site-packages/anndata/base.py in __init__(self, X, obs, var, uns, obsm, varm, raw, dtype, shape, filename, filemode, asview, oidx, vidx)
    634                 obsm=obsm, varm=varm, raw=raw,
    635                 dtype=dtype, shape=shape,
--> 636                 filename=filename, filemode=filemode)
    637
    638     def _init_as_view(self, adata_ref, oidx, vidx):

/projects/flynnb/software/anaconda/envs/scc/lib/python3.6/site-packages/anndata/base.py in _init_as_actual(self, X, obs, var, uns, obsm, varm, raw, dtype, shape, filename, filemode)
    753                 raise ValueError(
    754                     'If `X` is a dict no further arguments must be provided.')
--> 755             X, obs, var, uns, obsm, varm, raw = self._from_dict(X)
    756
    757         # init from AnnData

/projects/flynnb/software/anaconda/envs/scc/lib/python3.6/site-packages/anndata/base.py in _from_dict(ddata)
   1872                     d_true_keys['obs'][k_stripped] = pd.Categorical.from_codes(
   1873                         codes=d_true_keys['obs'][k_stripped].values,
-> 1874                         categories=v)
   1875                 if k_stripped in d_true_keys['var']:
   1876                     d_true_keys['var'][k_stripped] = pd.Categorical.from_codes(

/projects/flynnb/software/anaconda/envs/scc/lib/python3.6/site-packages/pandas/core/categorical.py in from_codes(cls, codes, categories, ordered)
    616                 "codes need to be convertible to an arrays of integers")
    617
--> 618         categories = CategoricalDtype._validate_categories(categories)
    619
    620         if len(codes) and (codes.max() >= len(categories) or codes.min() < -1):

/projects/flynnb/software/anaconda/envs/scc/lib/python3.6/site-packages/pandas/core/dtypes/dtypes.py in _validate_categories(categories, fastpath)
    325
    326             if not categories.is_unique:
--> 327                 raise ValueError('Categorical categories must be unique')
    328
    329         if isinstance(categories, ABCCategoricalIndex):

ValueError: Categorical categories must be unique

Looking at the attributes of test_combined, I see this:
test_combined.obs.sampleid.astype('category')
AAACCTGAGAACAACT-1-0    1
AAACCTGAGCTAGTTC-1-0    1
AAACCTGAGGGAAACA-1-0    1
AAACCTGCAATCACAC-1-0    1
AAACCTGCAATCGAAA-1-0    1
AAACCTGAGAACAACT-1-1    1
AAACCTGAGCTAGTTC-1-1    1
AAACCTGAGGGAAACA-1-1    1
AAACCTGCAATCACAC-1-1    1
AAACCTGCAATCGAAA-1-1    1
Name: sampleid, dtype: category
Categories (2, object): [1, 1]

and inspecting h5file['uns']['sampleid_categories'] yields [b'1', b'1'].   Because some of the values are strings, the dtype of the column in the dataframe gets set as 'object' which causes is_string_dtype(data.obs.sampleid) to be True.
I think the logic in base. df_to_records_fixed_width should probably be changed to sanitize user-defined inputs like this or display a warning if mixed dtypes are detected.
I'm using anndata==0.6.4 and scanpy==1.2.1.

Thank you for pointing this out! I'll work on a solution as soon as possible.


WordPress Integration

The first task is to transcribe all the Jupyter notebooks already completed into HTML so that they can be worked on with WordPress. Since HTML is not a valid programming language this could be considered a non-technical piece of work :(

Not gonna fix this