network log

Hi,
i have some individuals pulling misc activities that are in violation to your code of conduct. Could you kindly help me understand why your github domain would be showing up several times in my network log? This has been an ongoing thing all while doing activities online that dont include gaming, app or website development, etc. Basically just normal stuff like google searching music, porn, shopping, and self help books and task. Also could you recommend a network tool that’s simple to understand and monitor dns hijacks phishing malware iphone mitm and or signs of pegasus useage/etc attempts?

Thank you for this Anon @appledevs1.
There isn't anything actionable in your question, so I am closing this now.


Bumping version to 2.0.1 for opensearch-transport

Signed-off-by: Vacha Shah<EMAIL_ADDRESS>Description
Bumping version to 2.0.1 for opensearch-transport.
Issues Resolved
#91
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@BrianHawley I am not able to add you for reviewer, so tagging you here.

This is also going to need a release of the opensearch gem in order to enable that dependency restriction change.


[DISCUSSION] Thought process on how to add elevation

Hello! GIS Specialist here! Great work on this project!!
I Just wanted to start a discussion on how to achieve elevation with the OpenStreetMap (OSM) data set in to Arnis, since I see it is on the todo list!
I believe the most suitable format would have to be Digital Elevation Model (DEM) data or some other raster input.
I'd be happy to help out with implementing this, but wanted to see what the current thought process for achieving this would be! I haven't coded anything Minecraft related before, but I work extensively with GIS data and enjoy coding in Rust!
Additional thoughts

I think implementing the scaling factor in the current Rust version would be helpful for this (which I saw from #49 that it was implemented in Python)
I think the process could be:

Grab OSM data from Overpass API from a Bounding Box
Grab DEM data for the same Bounding Box
Map the elevation in Minecraft
Add the OSM data to the elevation in Minecraft

Hi there! That's amazing, thank you for reaching out! Really cool to get feedback and input from people who actually know what they're doing. ;)
I had a pretty similar thought process. I already played around in the "old" Python version with implementing some sort of elevation, but I failed with getting a data source which is:

free
preferably doesn't need an API token
has a large coverage of mainly the US and Europe
is purely the ground level (filtered out tree and building heights)

In regards of the implementation, like you mentioned we'd have to map the elevation data to the exact same area as the elements using the bbox and the scaling factor. I'm pretty sure this is just some maybe annoying fine tuning work, but sounds doable. For the next step, my approach would have been to use the position as an index when placing blocks to retrieve the according elevation at that exact coordinate. But this assumes that we have a data source which fits our criterias. I can imagine some pretty weird structures if there's e.g. a sudden peak in elevation due to an unfiltered tree height, if you know what I mean.
I will use your Issue as a discussion thread for further updates on this topic. Thanks for the DEM hint, I will look into this further. Curious to hear your thoughts on this! :)

Thanks for the detailed and supportive reply!!
In terms of free APIs we could potentially utilize:

https://portal.opentopography.org/apidocs/

But I think users would need to sign up with an API key and we'd have to parameterize that as a command line argument.
Open Topography

Endpoint: https://portal.opentopography.org/API/globaldem

Query parameters:

parameter
description
example

demtype
Dataset type
"SRTMGL1", "SRTMGL3", "AW3D30", "NASADEM"

south, north, west, east
Bounding box coordinates
41.0, 42.0, -72.0, -71.0

outputFormat
Output format
"GTiff" for GeoTIFF

Request format:

    params = {
        "demtype": dem_type,
        "south": south,
        "north": north,
        "west": west,
        "east": east,
        "outputFormat": "GTiff"
    }

Not sure if there is another free solution without API keys, but I'll keep looking around!

Hi, another GISer here :) Very nice work and I'm thinking about this same topic when exploring the generated world.
This open-sourced elevation API project open-elevation could be a good candidate but it's not active any more unfortunately. Some of its forks, e.g. this one according to this ticket, are well maintained.
There is a free public API at https://open-elevation.com/ but the availability could not be guaranteed. The self-hosted service is encouraged and easy to setup. Since I guess the elevation support in Arnis will be optional, to have a simple script launching open-elevation locally when required could be a feasible solution.

Another issue is the world height limit. I could stand in the ocean and end up 960 meters above sea level with 8 hours of walking where I used to live.

Hey on the note of height limit, in the latest releases of mc in world data packs you can set the height limit to a max of 2000+ so this part shouldn’t be too much of an issue

Hi, another GISer here :) Very nice work and I'm thinking about this same topic when exploring the generated world.
This open-sourced elevation API project open-elevation could be a good candidate but it's not active any more unfortunately. Some of its forks, e.g. this one according to this ticket, are well maintained.
There is a free public API at https://open-elevation.com/ but the availability could not be guaranteed. The self-hosted service is encouraged and easy to setup. Since I guess the elevation support in Arnis will be optional, to have a simple script launching open-elevation locally when required could be a feasible solution.

Open Elevation is, afaik, just a wrapper around the SRTM dataset. We could use this dataset directly if we wanted, without requiring the wrapper, however it's captured from satellites so I'd be worried about that messing it up. I suppose it would be relatively easy to check the dataset to see if this is an issue or not.

Here's WIP elevation (:

Hi there! That's amazing, thank you for reaching out! Really cool to get feedback and input from people who actually know what they're doing. ;)
I had a pretty similar thought process. I already played around in the "old" Python version with implementing some sort of elevation, but I failed with getting a data source which is:

free
preferably doesn't need an API token
has a large coverage of mainly the US and Europe
is purely the ground level (filtered out tree and building heights)

In regards of the implementation, like you mentioned we'd have to map the elevation data to the exact same area as the elements using the bbox and the scaling factor. I'm pretty sure this is just some maybe annoying fine tuning work, but sounds doable. For the next step, my approach would have been to use the position as an index when placing blocks to retrieve the according elevation at that exact coordinate. But this assumes that we have a data source which fits our criterias. I can imagine some pretty weird structures if there's e.g. a sudden peak in elevation due to an unfiltered tree height, if you know what I mean.
I will use your Issue as a discussion thread for further updates on this topic. Thanks for the DEM hint, I will look into this further. Curious to hear your thoughts on this! :)

Some years ago, a modder named Gegy made an earth mod with elevation called Terrarium. It can be found here on github, perhaps it might give you some tips

Hi there! That's amazing, thank you for reaching out! Really cool to get feedback and input from people who actually know what they're doing. ;)
I had a pretty similar thought process. I already played around in the "old" Python version with implementing some sort of elevation, but I failed with getting a data source which is:

free
preferably doesn't need an API token
has a large coverage of mainly the US and Europe
is purely the ground level (filtered out tree and building heights)

In regards of the implementation, like you mentioned we'd have to map the elevation data to the exact same area as the elements using the bbox and the scaling factor. I'm pretty sure this is just some maybe annoying fine tuning work, but sounds doable. For the next step, my approach would have been to use the position as an index when placing blocks to retrieve the according elevation at that exact coordinate. But this assumes that we have a data source which fits our criterias. I can imagine some pretty weird structures if there's e.g. a sudden peak in elevation due to an unfiltered tree height, if you know what I mean.
I will use your Issue as a discussion thread for further updates on this topic. Thanks for the DEM hint, I will look into this further. Curious to hear your thoughts on this! :)

Some years ago, a modder named Gegy made an earth mod with elevation called Terrarium. It can be found here on github, perhaps it might give you some tips

https://www.gebco.net/data_and_products/gridded_bathymetry_data/
"GEBCO’s current gridded bathymetric data set, the GEBCO_2024 Grid, is a global terrain model for ocean and land, providing elevation data, in meters, on a 15 arc-second interval grid. It is accompanied by a Type Identifier (TID) Grid that gives information on the types of source data that the GEBCO_2024 Grid is based on. "
@louis-e

Another issue is the world height limit. I could stand in the ocean and end up 960 meters above sea level with 8 hours of walking where I used to live.

You can allways cap it, or restrict the world height by injecting a datapack:
{     "type": "minecraft:overworld",     "generator": {         "type": "minecraft:noise",         "settings": {             "min_y": -64,             "height": 512         }     } }


About the F30K dataset

I would like to ask you, is this the target feature data set obtained from the F30K data through the bottom-up attention model?

Yes. Image features extracted from faster r-cnn are included in the links provided by the authors


Remove FRT holder check from distributeFeeRebate

Removing this check should be okay since distributing the fee rebate happens as the last value
transfer before an end-state.
This also means that calling distributeFeeRebate needs to be done more carefully.
The reason for this change was to allow for the funding script to correctly return the fee rebate even when redeemerAddress == rebateTokenHolder.

🙌


THC TDA pbc mGW

Added functionality to complete TDA pbc calculations using imported THC integrals. Structurally very similar to the initial THC implementation for TDA for molecules. Includes the main run file (pbc/thc.py) along with some tests (tests/test_kthcgw.py). Changes are also made in other files to align the method for running the new method with the molecular code.
Additionally, added a feature to allow loading THC integrals into the CDERI methods so that easy comparisons can be made between methods.
No example was created as the functionality is the same as the original THC example, except using the functions with K as per example 06_solids

I've taken a quick look, and looks good to me - nice work - but I'll leave it to Ollie to review if he wants before merging.
One quick point though (I might have mentioned this before when you implemented thc for molecules if I remember) - it does seem a little weird specifying that you want to use thc via the 'polarizability' keyword, which is really separate to the form of the interaction kernel. Not super important, but might be worth thinking about splitting that to a separate specifier - it's independent to the choice of how the polarizability is built after all...

I've taken a quick look, and looks good to me - nice work - but I'll leave it to Ollie to review if he wants before merging. One quick point though (I might have mentioned this before when you implemented thc for molecules if I remember) - it does seem a little weird specifying that you want to use thc via the 'polarizability' keyword, which is really separate to the form of the interaction kernel. Not super important, but might be worth thinking about splitting that to a separate specifier - it's independent to the choice of how the polarizability is built after all...

yeah maybe in theory it could be a different keyword, but the reason it is specified as part of the polarizability is that implementationally, only the polarizability class needs to change between CD and THC implementations.

as per some of my comments I would rather keep file inputs and file paths to a bare minimum -- only where there is no other obvious solution (i.e. the THC in the current state). I think they make for difficult to maintain code and it makes more sense for us to just include examples showing how one would go about hacking/duck typing loaded integrals in on the PySCF side

as per some of my comments I would rather keep file inputs and file paths to a bare minimum -- only where there is no other obvious solution (i.e. the THC in the current state). I think they make for difficult to maintain code and it makes more sense for us to just include examples showing how one would go about hacking/duck typing loaded integrals in on the PySCF side

Based on this conversation, I changed the code so that there is now only a singular import, thc_opts["file_path"] which is used for  doing both the THC code and for building CDERIs from THC integrals.

A more general comment/query, how general/user friendly do we want the THC import functionality to be for the time being? For example the THC import files are design for a specific data structure given what we were sent. In the time between receiving different THC integrals, their data structure changes and so the import process needed to change/didn't work for the old format. So how much guidance should we give people if they have their own? or should it be more there just as an internal tool functionality for the time being?

A more general comment/query, how general/user friendly do we want the THC import functionality to be for the time being? For example the THC import files are design for a specific data structure given what we were sent. In the time between receiving different THC integrals, their data structure changes and so the import process needed to change/didn't work for the old format. So how much guidance should we give people if they have their own? or should it be more there just as an internal tool functionality for the time being?

I think if their interface is that unstable, then we don't worry about it for the time being -- it would maybe be better suited to a dev branch until we have a stable interface, but if the code is clean and implemented to production quality then it's fine to have it in master and just update the interface whenever they do.

What features, if any, are not supported by this PR? Does it support all self-consistencies, Fock loops etc.?

What features, if any, are not supported by this PR? Does it support all self-consistencies, Fock loops etc.?

So I have tested everything seen in an example; evGW (neither, g0, w0), scGW (g0, w0, fock_loop and all combinations) and qsGW (eta, srg and solver (with combinations)). For chempot, see above and there is no UHF implemented. Is there anything else?

What features, if any, are not supported by this PR? Does it support all self-consistencies, Fock loops etc.?

So I have tested everything seen in an example; evGW (neither, g0, w0), scGW (g0, w0, fock_loop and all combinations) and qsGW (eta, srg and solver (with combinations)). For chempot, see above and there is no UHF implemented. Is there anything else?

Sounds about everything - i guess we should try to make a nice FEATURES.md to list what combinations of features are actually available


Integration with WRAS

[x] Fall 2022: provide Acartia API access, documentation, and technical support to Ocean Wise and Lapis.
[x] Fall 2023: post Orca Network moderated sightings of key cetaceans in Washington to new Ocean Wise / WRAS API
[ ] 2024: post other cetacean sightings and Orcasound moderated hearings of key cetaceans to WRAS API

[ ] Invite WRAS to ingest all trusted observations within revised Acartia trust model?
[ ] Enhance attribution for WRAS end users by including comment field in modal, i.e. attribution back to individual observer, not just organization and sourceentity?

[ ] 2024+: possibly automate an (annual? monthly?) exchange of Whale Report data from within Washington to Acartia...

Latest API documentation:

WRAS API docs
Acartia: endpoint docs | contributing docs
Postman API query web app

@dylanse This would be an optimal place for you to interact with Acartia folks like me and @aalaydrus as we aspire to push some records through the new WRAS API.
The initial goal is to send trusted visual sightings in WA from Orca Network across the border. Acoustic detections (raw or moderated) from Orcasound and OrcaHello AI will be a future consideration.

@dylanse Got any updates for us?

@scottveirs The staging version of our API was just recently deployed to accept hydrophone detections. There is a meeting scheduled with our technical team next week to discuss the timeline for the readiness of the API to receive sightings data from Acartia. I anticipate your team should be able to start testing very soon.
For your reference, I can also share this API documentation with you to preview, https://oceanwise.stoplight.io/docs/sightings-network-api/80d0ea7ea0c45-create-a-sighting
This API and this documentation will serve both our autonomous and observed detections. Let me know if you have any questions, I will send you your API token soon as it's ready.

@dylanse Any updates on when tokens may be issued?

7/12 update from Dylan via email: "our vendor has been making progress as of lately with the remaining POST endpoint on our API and the final testing version is to be delivered no later than next week. The necessary work to receive to hydrophone detections has been finished and it’s just a matter of adjusting some remaining fields to accommodate our observed sightings to the same endpoint."
@aalaydrus -- Dylan indicated that Acartia devs should be set up to begin testing soon. "access key and newest documentation will be provided next week"

Hey @scottveirs and @aalaydrus  - I just wanted to follow-up on this as I'm taking a look at this issue for the first time in a while - thanks Scott for sharing the other day. Dylan followed up with documentation and the API key on the 21st of July first thing in the morning.

Our updated API documentation
The API key was included in the email, I won't share it here!

We are close to releasing a new production version of our applications, we'd love to include any changes we might need to make to accommodate Acartia sightings in this release if possible 😊.
Let either of us know if there's any questions! Thanks!

Thanks for following up during the busy summer season, @almitch . Looking forward to more progress this month...
Thanks for the API docs and key. Ali and I have discussed and he is working on test submissions of the requested real time sightings moderated by Orca Network.
@aalaydrus will seek answers here and/or via email. I'll provide other updates on scheduled calls tomorrow.

Thanks for following up during the busy summer season, @almitch . Looking forward to more progress this month...
Thanks for the API docs and key. Ali and I have discussed and he is working on test submissions of the requested real time sightings moderated by Orca Network.
@aalaydrus will seek answers here and/or via email. I'll provide other updates on scheduled calls tomorrow.

Great! Thanks so much Scott. Looking forward to our chats tomorrow.
Again, always here to help if you need anything.

@almitch and @dylanse -- Is the mock/test API server functional with the token you provided?
Or do you want us to test the production API with a single Orca Network moderated sighting?

@scottveirs, I know the developers were performing some migration changes today to prepare the production environment, I will report back once I confirm that the testing API is ready again.

Hi @dylanse,
It seems that the both the live server as well as the mock server doesn’t seem to work through postman (with token)
However, I’ve discovered that the docs that they sent on their API seems to have a section to test the API call through the site itself here:
https://oceanwise.stoplight.io/docs/sightings-network-api/1f68387dbeb27-get-sightings
Trying through the above site to do a GET request on the live server and the mock server also gave me the same set of errors
I tried the POST on the mock server and I seem to also get a similar error message which seems to be outputted from Prism

@aalaydrus The staging server appears to be operational from my end. However i just noticed that there was a {basepath} appended to the url that shouldn't be there. I have removed it from the documentation.
The mock server API is provided out of the box by the documentation provider but it's not something we use or rely on for any testing use.
Hope that helps, look forward to seeing some test data coming in soon!

Hey @dylanse and @almitch -- we'll keep testing this weekend and next week, but in the interim (and maybe to accelerate our progress) can you answer these two queries?

Have any other external entities (outside of Ocean Wise and Lapis) succeeded in pushing records via your new API?
If so, could you put us in touch with any of them so we could compare notes/scripts/approaches?

I'm facilitating a hackathon this Mon-Wed with Microsoft. One goal is to start pushing OrcaHello data (automated ML detections moderated by bioacoustic experts) from Azure to Acartia. So, I'm particularly curious to know if any Canadian hydrophone hosts have started to add acoustic detections to WRAS via the API.
Thanks,
Scott

Hi @scottveirs

Yes, we have had live hydrophone data streaming successfully to the API since June 2023. You can also see this data by performing a GET request for the type: autonomous (see API documentation)
That is a possibility but I'd like to provide support from our end first. I'm happy to get into a chat/call with @aalaydrus and help diagnose what the issue is as we don't see any anything precluding a successful connection

Hi @dylanse finally got both GET and POST to work from my end. Took me some time to figure out that the token field it was expecting was x-api-key as I assumed that the API url is sitting behind an API Gateway in AWS and it is what is expected to authorise requests.
I'll communicate with @scottveirs for entering the other data and other pending tasks.

Hey @dylanse -- can you provide answers to these two questions ASAP ^^^ this week?
With them we should be able to start POSTing any new observations of any of the target species every 5 minutes...

Hi @aalaydrus,

Thanks for spotting the duplicate Killer Whale value (14) in our speciesId field. I have made a note to correct this but will need to verify it's not there for backward compatibility first. Please proceed and use 1 as the value for Killer Whales.
Correct, a Southern Resident would be speciesId=1 and ecotype=3

Also, something new but not documented yet is that there is a /reference endpoint you can hit to access the possible values programmatically if needed.

Hey @dylanse and @almitch
A couple questions and comments as @aalaydrus and I deploy the transfer script to AWS this week:

Your API documentation could be improved by verbally describing/defining the dateReceived and id fields (beyond specifying their format as string and uuid respectively). Are we correct in assuming these fields will be populated upon successful post of an Acartia observation?
Your API documentation is confusing (to Ali initially, and now also to me as I study it more closely) because the landing page at https://oceanwise.stoplight.io/docs/sightings-network-api/80d0ea7ea0c45-create-a-sighting defaults to the Autonomous data scheme, when we were asked to post Acartia observations (Orca Network sightings, presumably via your Observed data scheme). Additionally, when the dropdown is used to change the API documentation to Observed the API examples do not update (Send API Request button code and curl example code stay the same; compare 2 screenshots below). You probably made this UI default/decision to accommodate Canadian providers of automated acoustic detection algorithms. Could confirm that we're interpreting the situation correctly, and if so could you reduce confusion for future visual observation providers, e.g. by adding explanatory text to the top of the WRAS API guidance?
Related to 2 (and in anticipation of some day adding Orcasound automated acoustic detections), in your default example API query -- for what appears to be an automated click detector (detectorName = SpectroDetector-v3.5.1...) -- shouldn't the sourceType on line 20 be equal to "Autonomous"? Instead it seems that the default was assumed -- as "sourceType": "Observed"... (see first screenshot below.)

Thanks Scott, this is really useful feedback, especially given that this API should be general enough for several types of data from different orgs so it's great to have some input into it. Thanks.
Dylan and I have a meeting tomorrow to discuss ongoing integration work so we can work on some ways to improve the clarity of the documentation.
I'll leave @dylanse to get back to you on the more specific questions/
Cheers.

You're welcome, @almitch . When you and @dylanse meet, it might be efficient to give also us a yes/no or comments/concerns regarding any fields and/or values that we're test-posting to the WRAS API (required fields in bold)
{
  **"longitude": -124.06493,
  "latitude": 49.39968,
  "speciesId": 1,** // 1, 2, 3, 4, or 5
  "sightedDate": "2023-02-02T06:39:32.000Z",
  "ecotype": null, // unless KW: then 3 if SRKW, and 0 otherwise
  "comments": "The full Acartia comment string.",
  "idConfidence": 0,
  "orgName": "Orca Network",
  "sourceType": "Observed",
  "sourceName": "other",
  "sourceEntity": "Acartia",
  "externalID": "a unique string equal to the record identifier in Acartia",
}

Note that we are not providing the following fields (intentionally leaving them as nulls in the posted record):
dateReceived
id
countMeasure
numberOfAnimals

The last we could provide in the future, but only with an ambiguous value for countMeasure...
We're eager to finalize these last details and automate posts via AWS Lambda/Eventbridge as soon as we have answers in hand!

@dylanse Here is are two more tiny bits of feedback about the WRAS API based on our successful test posts (via axios.post in Ali's Node.js script):
A) Upon getting one of our test records, we noticed that the travelDirection field had been populated with a value of 10 when we did not include that field in our POST. Is that a bug on your end, or an intentional feature?
B) In the example API Request there seems to be an errant final row (see screenshot below). Perhaps this is leftover from creating the example?

@almitch Alex -- any new updates from the OW IT team before we start posting?

Hi @aalaydrus @scottveirs, thank you for your feedback on our API documentation.
I've taken steps to revise and clarify the previously mentioned ambiguities in our documentation. The id is a UUID that can be generated and included with the sighting. If this field is missing, a new UUID will be generated and returned to you. The dateReceived field is automatically populated in both autonomous and observed responses, indicating when the sighting was logged on our end.
As part of the documentation we are currently using, it has a component that automatically auto-generates example requests and responses. Unfortunately, this doesn't work perfectly and should not be relied upon. It would be my preference to remove that component altogether but did not see a way to remove it last time I checked. Apologies for any confusion this has caused.
Your example request appears to be in good order, so feel free to proceed whenever you're satisfied with the explanations provided. Once we see your data coming in, we'll take a look and send any necessary feedback as soon as possible.
cc @almitch

Hi @dylanse I'm getting the following error from the live server
error: {         length: 85,         name: 'error',         severity: 'FATAL',         code: '53300',         file: 'proc.c',         line: '360',         routine: 'InitProcess'       }
I was hoping to ask if there's a maximum amount of requests at a specific period to interact with the server. I am also getting this from a simple GET request from Postman. Thanks!

@dylanse Any thoughts on this error we’re getting? Ali has continued trying since his query, but to no avail… we will have another call tomorrow at 8am to troubleshoot, so even preliminary thoughts from you and/or @almitch would be welcome.
Is the WRAS server refusing requests from our token, either because it’s expired or maybe due to too requests? (Nothing wrong with our plan to post new qualifying observations as frequently as once every 5 minutes, right?)

This looks like an error on the postgres side. I have no direct access to those logs so will have to get the devs to take a look. However, I do see several Acartia sightings when I perform a GET for observed sightings on the API (with your token). If it does not work for you, there could be some throttling going on if there a large amount of sightings. I will confirm tomorrow and report back soon as I can

@dylanse Thank you for the update. We are standing by.
In the mean time, can you please confirm that posting "whale sightings" as often as once every 5 minutes will not be a problem?
If it will be a problem, please let us know how often we should run the post script.

@dylanse Out of curiosity and to get a point of comparison, how often is JASCO posting acoustic detections to the autonomous API?

@dylanse Is there a most basic no-auth request we can make of the mock and/or live WRAS APIs that would allow us to assess if it is responsive, (prior to engaging in additional complexities of bearer tokens and authentication)?
I'm imagining you might have something like Acartia's no-auth endpoint --
https://acartia.io/api/v1/sightings/current
?

@dylanse Quick report after call this eve with Ali:
Glad to report that basic GETs with our bearer token again seem to be working as of an hour or so ago...
and BTW --
https://6671737ff1.execute-api.us-west-2.amazonaws.com/v1/reference
is helpful!

Hi @scottveirs and @aalaydrus, I contacted the development team yesterday regarding the issue you experienced. They are currently investigating it and believe that the problem may stem from their existing configuration being unable to accommodate the high volume of requests.
JASCO currently sends a relatively low volume; this is the first time we've encountered an error related to a high number of requests.
For this initial phase, our API requires a token for access for all endpoints.
In the future, I'm happy to jump on a call/chat if needed to expedite any troubleshooting, feel free to drop me an email!

@dylanse Thanks for the update. In an effort to reduce the volume of requests, as of 2023-10-12 18:25:50 (UTC-07:00) we are posting only new Orca Network observations every 10 minutes to https://6671737ff1.execute-api.us-west-2.amazonaws.com/v1/sightings
You should see some TEST points I generated. I've confirmed at least a couple were successfully posted this evening, by checking via a GET.

Can you confirm receipt of the TEST point with Sighted Date = 2023-10-13T01:11:00.000Z and okay the formatting?
BTW: We still don't understand why the travelDirection value is getting set to 10 when we only provide a null.
Under the assumption that you don't test for duplicates (and therefore do not want all qualifying points we GET via https://acartia.io/api/v1/sightings/current ), @aalaydrus wrote custom code to implement our initial low volume scheme: posting only recent observations (with age less than 10 minutes). We will likely need to revisit this age threshold in consultation with your devs (to avoid hitting any high volume caps) and the data providers (who may require more or less time to submit a new observation).
Assuming you also receive the suite of new points that will likely be submitted tomorrow during "live" data entry by Orca Network and/or us, we hope you can proceed with any QA/QC and pass points ASAP on to WRAS end-users.  (SRKWs, Bigg's, and humpback sightings were abundant today and will likely be so tomorrow as well).

Heya @dylanse We've encountered the following error message when the we tried posting GET and POST requests:
{
    "message": "Internal server error"
}

Not sure if this has to do with our token or if the server is currently set offline.

@dylanse Just tried a simple GET and we are still getting an Internal server error message.

Can you give us an ETA on when this might be fixed? I am eager to continue testing the performance of our deployed POST script this week!

@scottveirs @aalaydrus I can confirm that I am getting the same error. I have a meeting with the devs in 2 hours, will report back shortly!

@dylanse Perhaps it would be helpful to share with the Lapis folks that we are attempting a post every 10 minutes (for the last 9 days or so)?

@dylanse and @almitch I think I just noticed a major error in the WRAS API "dateReceived field's time formatting.
It appears your time stamp has a value that is accurate in local time, but the format with the Z suffix is indicating the wrong time zone. (Z implies Zulu aka GMT, whereas for the time value of the field to remain accurate the time zone should be Pacific or -800Z or equivalent.)
My strong evidence from recent tests for this claim:
Step 1) Orca Network observation occurs:
2023-10-13 11:53:00 and Alisa got it submitted to Acartia at only 1.5 minutes later (at 11:54:30)! Then Ali's script automatically posted it to the WRAS API at 18:55:48. The screenshot indicates that the WRAS API created a record of it at 11:56:49.
Step 2) Acartia record is created:
2023-10-13 11:54:30
Step 3) Ali's script posted it to the WRAS API using this GMT date-time stamp:
2023-10-13 18:55:48 GMT
or 11:55:48 local
Here is the log entry for that POST:
  comments: '[Orca Network] Js, Ks, and some Ls. Many have stalled off Maxwelton, others spread to the north (Rachel Haight, Jim Pasola) ',
  ecotype: 3,
  idConfidence: 0,
  longitude: -122.47444,
  orgName: 'Orca Network',
  latitude: 47.9359,
  speciesId: 1,
  sightedDate: '2023-10-13 18:53:00',
  sourceType: 'observed',
  sourceName: 'other',
  sourceEntity: 'Acartia',
  externalID: 'e5a7faf0-58c2-40c1-bf18-a00da2c1c921'
}

Step 4) WRAS record is created (based on a GET):
2023-10-13T11:56:49.000Z
The date-time stamp in Step 4 is based on and copy/pasted from a GET to https://6671737ff1.execute-api.us-west-2.amazonaws.com/v1/sightings which suggests that the returned time value is reasonable, but the time zone designated by .000Z is not.

Hi @scottveirs and @aalaydrus, just confirming for you that the API is operational again. The reason for the error was identified with a function that was not closing the db connections.
I will follow up again on the timezone issue you raised. Your detailed feedback here is greatly appreciated, thank you!

@dylanse -- Can you confirm/deny that successful posts are being transmitted to WRAS end-users at this point? Or will we need a new bearer token for a production version of your API?

Also @dylanse or @almitch -- based on feedback from Orca Network staff today, we will be posting every 15 minutes, rather than every 10 minutes. Related to this, it would be valuable to eventually get answers to these two question from you:

How old is "too old" for an observation to be useful (i.e. worth POSTing to the API) from the perspective of WRAS end-users?

How often is "too often" (aka redundant) for observations of the same individual or group? (In other words, if Orca Network is observing the same species/ecotype every 2 minutes, will each observation be shared via each of the WRAS notification methods? If not, what subset is shared via which notification method?)

Hey @scottveirs - apologies for the delay, I've got email notifications turned on now so I should be responding a bit quicker.
In answer to your questions...

Posts are not currently being transmitted to end users as we need to get the acknowledgements sorted out in the alerts. We are waiting to hear back from Lapis on a quote and estimated delivery time for that. It is unfortunate we can't do this ourselves as it is a very easy thing to do but we are relying on Lapis to do the work in their time frame. You will need a new token for the production API (@dylanse correct me if I'm wrong on the token).

Too old, 15mins is good for us, from conversations with mariners the more 'real-time' the better, extrapolating from that 20 mins and above it likely getting towards the "too old" spectrum, particularily for Killer Whales. The last thing mariners want is to recieve an alert and to be on top of the animals before they know it (worst case).

This is an interesting question. Does Orca Network often report that frequently? For our sightings network reports we do not have any aggregation functions so multiple sightings in the same area are sent as multiple alerts. However, for our Hydrophone alerts via JASCO these are grouped until there is 30min of silence (if you see an alert from the hydrophone you know they v.likely are in the area). If Orca Network do record same species/ecotypes that often perhaps we will have to look at grouping these sightings together over a certain period, or, only alerting every say 30mins if the prev alert was the same species/ecotype + group size for instance.

Thanks Scott.

Thanks for the updates, Alex @almitch !
It is still relatively rare for Orca Network and other dense US sighting networks to generate unique observations more than every half hour. However, in the past two years there been many high-visibility, pleasant-weather days when a popular pod or matriline was tracked with unique locations reported every 5-10 minutes. I suspect Ocean Wise could delay adding the type of intelligence you describe for visual sightings this year, but would suggest you plan for it.
On a related topic, raw detections from Orcasound hydrophones by humans or machines are grouped temporally into "candidates." They are currently added manually to Acartia, but once moderated candidates are added via the Acartia API, then a similar issue to the JASCO situation may arise for WRAS end-users.

Hi @aalaydrus , @scottveirs , just following up on the timezone issue. Would you be able to convert and send your timestamps in in ISO 8601 format and we can always assume they are sent in UTC?

Hey @dylanse -- to be clear the date-times we are submitting are sent in UTC. (All Acartia timestamps are in GMT aka UTC aka Zulu time zone.)
We will try (briefly) to write more custom javascript to meet your request, but we don't have the budget to buy full ISO specs, so could you please just confirm you want the format you currently provide as an example in the WRAS API docs under Observed data scheme's sightedDate field?
2023-02-02T06:39:32.000Z

Hey again @dylanse Dylan, I was just looking to see if we could again use the WRAS API to test for successful POST using GET, but I'm not seeing data more recent than 10/13/23.  Before we start hacking again, please answer both of these queries:

Can you confirm that we will be able to again use a GET as before to verify that new WRAS records have been created?

These are the most recent (still tripled, BTW) records we receive with our bearer token:

Can you check WRAS API logs to see if this most-recent POST made it through?

2023-10-26T19:50:17.596Z	a5653ac2-d186-454b-af72-e27fb07ee1f5	INFO	{
  comments: '[Orca Network] Jpod spread out milling with directional changes (Rachel Haight)',
  ecotype: 3,
  idConfidence: 0,
  longitude: -122.41083,
  orgName: 'Orca Network',
  latitude: 47.88697,
  speciesId: 1,
  sightedDate: '2023-10-26 19:45:00',
  sourceType: 'observed',
  sourceName: 'other',
  sourceEntity: 'Acartia',
  externalID: 'f2cfb92d-5fcc-4327-a73a-b24731af0122'
}

Hey @dylanse -- You can ignore the previous post. Ali and I just met and were able to confirm that his efforts yesterday to meet your ISO8601 request were successful. I'm not sure what was going on with my previous GET requests, but this morning I successfully retrieved this test point...

...and can confirm that the UTC date-time stamp in the WRAS record (2023-10-27T16:27:00.000Z)is consistent with the local time of the test observation (9:27 am Pacific time, currently -07:00Z).

Thanks for you efforts @scottveirs and @aalaydrus - this is great. Glad that this is working as expected! We queried Lapis on a similar thing we noticed with the API and turns out there was a bug in the API they have now fixed. Not sure on the details but this may have been the cause of the previous GET request weirdness.

Hey @dylanse and @almitch -- Ali and I just submitted a TEST point and are satisfied with the performance of the AWS-based script that is currently posting new [Orca Network] sightings to the test server every 15 minutes.
Here is evidence that the date-time stamps look consistent to us:

Test point was submitted to Acartia at 8:22

Adding 7 hours now (NOTE: daylight savings kicks in on 11/5...), we see the Acartia record has a consistent creation date-time: "created":"2023-11-02 15:22:00" (8:22+7:00=15:22)

AWS Cloudwatch says that the script found and POSTed the new observation 2-3 minutes later at 2023-11-02T15:24:41

Which matches the minute and even second values in the WRAS dateReceived field (despite the local/Zulu bug we documented earlier) when we do a GET request to the WRAS test API using the old bearer token yielding this response:

            "dateReceived": "2023-11-02T08:24:41.000Z",
            "id": "18b55107-5ef3-44b8-a295-0a85bf6236ef",
            "behaviour": null,
            "comments": "[Orca Network] TEST not a real sighting (Scott Veirs)",
            "countMeasure": null,
            "disstressedState": null,
            "ecotype": null,
            "experience": null,
            "howUHeard": null,
            "howUHeardSpecifics": null,
            "idConfidence": 0,
            "locationDesc": null,
            "longitude": -128.77306,
            "numberOfAnimals": null,
            "numberOfAnimalsMax": null,
            "numberOfAnimalsMin": null,
            "orgName": "Orca Network",
            "photos": null,
            "reporterEmail": null,
            "reporterLastName": null,
            "reporterName": null,
            "reporterPhone": null,
            "resiPostCode": null,
            "seaState": null,
            "sightingDistance": null,
            "sightingPlatform": null,
            "travelDirection": 10,
            "vesselName": null,
            "whenInWater": null,
            "whyInWater": null,
            "windSpeed": null,
            "latitude": 46.09225,
            "speciesId": 2,
            "sightedDate": "2023-11-02T15:22:00.000Z",
            "sourceType": "observed",
            "sourceName": "other",
            "sourceEntity": "Acartia",
            "externalID": "7dde6dbb-880b-4571-89b8-36f7dfd24645"

Most importantly, the "sightedDate": "2023-11-02T15:22:00.000Z", part of the return has the correct value in the ISO 8601 format that you specified.
I've double-checked that the latitude and longitude also match the initial record in Acartia.
Can you confirm this test POST looks good to you?
If so, we suggest continuing to let the automated script POST the test API today. If those points also look good to you we can try a brief switch to the production API tomorrow. (BTW -- I tested the new bearer token and a GET to the PROD endpoint and the request was successful.)

Hi @scottveirs  - apologies for the late reply, I have been looking into the API and database most of today to try to confirm what is going on as I noticed some strange behavior. (@dylanse has been super busy today with other projects - he's in high demand! - so this spiel is also for his information)
So. All in all your POST looks good to me.
Not sure why the travelDirection field is autoing to 10, but the front end is  ? which is the expected output. If we had some dev control I would love to just write a string detection logic to grab it from the comments, same with ecotype, but we'll work with what we've got for now.
Although your POST looks good, I have noticed some interesting bits in the visualization of the data sent through and in the database.
In a simple GET to review data in the test database I can see there are duplicates appearing from Acartia, but when I POST at my end there is no duplication.
An example of the duplication:
        {
            "dateReceived": "2023-11-01T17:13:13.000Z",
            "id": "04dca8d7-276b-441c-bbf6-c80ad0454ff2",
            "behaviour": null,
            "comments": "[Orca Network] Biggs T109A2s southbound 50 yards from Bush Point, moved offshore after passing point (Cindi Crowder Rausch)",
            "countMeasure": null,
            "disstressedState": null,
            "ecotype": 0,
            "experience": null,
            "howUHeard": null,
            "howUHeardSpecifics": null,
            "idConfidence": 0,
            "locationDesc": null,
            "longitude": -122.61157,
            "numberOfAnimals": null,
            "numberOfAnimalsMax": null,
            "numberOfAnimalsMin": null,
            "orgName": "Orca Network",
            "photos": null,
            "reporterEmail": null,
            "reporterLastName": null,
            "reporterName": null,
            "reporterPhone": null,
            "resiPostCode": null,
            "seaState": null,
            "sightingDistance": null,
            "sightingPlatform": null,
            "travelDirection": 10,
            "vesselName": null,
            "whenInWater": null,
            "whyInWater": null,
            "windSpeed": null,
            "latitude": 48.03067,
            "speciesId": 1,
            "sightedDate": "2023-11-02T00:05:00.000Z",
            "sourceType": "observed",
            "sourceName": "other",
            "sourceEntity": "Acartia",
            "externalID": "59bc1246-b78e-4137-959b-1f159a732b18"
        },
        {
            "dateReceived": "2023-11-01T17:16:28.000Z",
            "id": "c573c45b-d811-426e-8c98-f613551ccb25",
            "behaviour": null,
            "comments": "[Orca Network] Biggs T109A2s southbound 50 yards from Bush Point, moved offshore after passing point (Cindi Crowder Rausch)",
            "countMeasure": null,
            "disstressedState": null,
            "ecotype": 0,
            "experience": null,
            "howUHeard": null,
            "howUHeardSpecifics": null,
            "idConfidence": 0,
            "locationDesc": null,
            "longitude": -122.61157,
            "numberOfAnimals": null,
            "numberOfAnimalsMax": null,
            "numberOfAnimalsMin": null,
            "orgName": "Orca Network",
            "photos": null,
            "reporterEmail": null,
            "reporterLastName": null,
            "reporterName": null,
            "reporterPhone": null,
            "resiPostCode": null,
            "seaState": null,
            "sightingDistance": null,
            "sightingPlatform": null,
            "travelDirection": 10,
            "vesselName": null,
            "whenInWater": null,
            "whyInWater": null,
            "windSpeed": null,
            "latitude": 48.03067,
            "speciesId": 1,
            "sightedDate": "2023-11-02T00:05:00.000Z",
            "sourceType": "observed",
            "sourceName": "other",
            "sourceEntity": "Acartia",
            "externalID": "59bc1246-b78e-4137-959b-1f159a732b18"
        },
        {
            "dateReceived": "2023-11-01T17:14:15.000Z",
            "id": "bee36e1c-a65d-406e-8864-842470d0bc50",
            "behaviour": null,
            "comments": "[Orca Network] Biggs T109A2s southbound 50 yards from Bush Point, moved offshore after passing point (Cindi Crowder Rausch)",
            "countMeasure": null,
            "disstressedState": null,
            "ecotype": 0,
            "experience": null,
            "howUHeard": null,
            "howUHeardSpecifics": null,
            "idConfidence": 0,
            "locationDesc": null,
            "longitude": -122.61157,
            "numberOfAnimals": null,
            "numberOfAnimalsMax": null,
            "numberOfAnimalsMin": null,
            "orgName": "Orca Network",
            "photos": null,
            "reporterEmail": null,
            "reporterLastName": null,
            "reporterName": null,
            "reporterPhone": null,
            "resiPostCode": null,
            "seaState": null,
            "sightingDistance": null,
            "sightingPlatform": null,
            "travelDirection": 10,
            "vesselName": null,
            "whenInWater": null,
            "whyInWater": null,
            "windSpeed": null,
            "latitude": 48.03067,
            "speciesId": 1,
            "sightedDate": "2023-11-02T00:05:00.000Z",
            "sourceType": "observed",
            "sourceName": "other",
            "sourceEntity": "Acartia",
            "externalID": "59bc1246-b78e-4137-959b-1f159a732b18"
        },

Before I ask Lapis about this, do you think this could be on your end that duplicates are being sent? You can see that the sightedDate remains the same between the detections but dateRecieved varies. Any thoughts?
This is also shown in the front-end test env for the app from alerts occurring yesterday:

Secondly, the front end map is visualizing the alerts as if the sightedDate is PST, not UTC. Take this example...
I submit this "POST"...
{
    "longitude": -123.21438868917642,
    "latitude": 49.30010974606236,
    "speciesId": 1,
    "sourceType": "observed",
    "idConfidence": 0,
    "travelDirection": 2,
    "sourceName": "app",
    "sourceEntity": "string",
    "externalID": "TEST-alex-4",
    "sightedDate": "2023-11-02T17:10:00.000Z"
}

and straight away an alert is generated, therefore it is treating the sightedDate as PST, rather than the UTC format it is in.

If I submit a UTC time now (+7) I would expect the alert to appear, it does not. Meaning, alerts are currently being visualized 7hrs in the future (soon to be 8). Looks like there might be a discrepancy between how Lapis are treating dates that needs to be fixed asap. I will write up a bug report to Lapis now and see if they can deal with it ASAP.

Hey @almitch  -- Thanks for the updates.

We noticed the triple-record issue last week and Ali resolved it over the weekend. The latest test point we submitted was a successful verification that only a single record was posted by the de-bugged Lambda function. I'll be watching tomorrow to ensure we don't get more triplets, but we are 98% sure the issue is resolved.

I think we should table the direction question for now. We have plans to add it as an input field for Conserve.io apps and to eventually derive it for Orcasound acoustic detections. Either way, we elected to not parse it from the comments during this initial phase of API exchanges.

That is potentially a problematic bug for the WRAS front end! It seems similar (possibly related?) to the ambiguity previously mentioned in the dateReceived field. Maybe you should try an ISO 8601 format that indicates the time zone and see what the front end displays?

I'd suggest trying a format like 2023-11-02T19:44:40−07:00 ... and if it works for your test point, then maybe Ali and I send in the same format (after converting to PST of course) as an interim solution for [Orca Network] sightings?

Thanks @scottveirs -

Looks like there are no more tripled reports, thanks for checking in on this.

That's all fine, I was just thinking aloud on what we could potentially do our end in the future.

Both datetime fields are displaying UTC, but processing the times as PST.

So, what's the solution? The format you suggested is not accepted by the API, however, a work around for now could be to not include sightedDate on the POST. The database logic assumes if no sightedDate has been provided then the detection is live and presents the dateRecieved on the front end.
Although not accurate, I suggest we do this until we get the datetime fields functioning as they were intended to. What do you think? Is that easy enough your end?

With SRKWs coming into Puget Sound for the 4th time as I type, Alex @almitch , this sounds like a good hack to me. We will give it a try now.

Great, I'll be watching!

As an oceanographer makes me feel a little ill manually subtracting 7 hours from a proper Acartia UTC timestamp (total javascript hack), but this humpback test looks promising @almitch --
Without the subtraction:

With the subtraction (after forcing the Lambda function to run between scheduled 15 minute events):

Next will be a test of a gray whale submitted via the scheduled run.
If that works I'll try switching to PROD API and token...

Looks like both have worked great. Apologies for the ill provoking task! Hopefully Lapis can sort this out ASAP, I also highly dislike sticky taping anything, especially automated processes!

Hey @almitch -- It looks like the gray whale test POSTed with no errors from the Lambda function's perspective...

, but I don't see it with a GET request to the WRAS test API -- even after waiting for ~5-10 minutes.
Do you see any evidence of it?

Ok, I'll take that previous post as confirmation the gray whale test arrived.
Proceeding to switch to new endpoint and token, then await a real sighting for an end-to-end test!

Oh -- I see it now in the test API response to my GET, @almitch .
I expected it to be at the top of the response, but found it further down:

Proceeding with swap...

This is from my GET request...

Just a little further down as the GET orders by time.

Ok swapped over to live WRAS endpoint, and I see this new report that should come in the next 15 minute cycle:
"created":"2023-11-03 23:35:00", ....
"data_source_comments":"[Orca Network] J pod at least southbound fast, spread out north-south and from midchannel to Whidbey (Cindi Crowder Rausch)

And there we go! That has gone through on the correct time as well! Excellent!

Ok, great! @aalaydrus or I will switch the hack to -8 at midnight on Saturday.

And from the GET on the prod API we can see the data:
"sightings": [
        {
            "dateReceived": "2023-11-03T16:47:46.000Z",
            "id": "32dd2fc9-c05d-4040-afec-e32e08efd5f1",
            "behaviour": null,
            "comments": "[Orca Network] J pod at least southbound fast, spread out north-south and from midchannel to Whidbey (Cindi Crowder Rausch)",
            "countMeasure": null,
            "disstressedState": null,
            "ecotype": 3,
            "experience": null,
            "howUHeard": null,
            "howUHeardSpecifics": null,
            "idConfidence": 0,
            "locationDesc": null,
            "longitude": -122.61686,
            "numberOfAnimals": null,
            "numberOfAnimalsMax": null,
            "numberOfAnimalsMin": null,
            "orgName": "Orca Network",
            "photos": null,
            "reporterEmail": null,
            "reporterLastName": null,
            "reporterName": null,
            "reporterPhone": null,
            "resiPostCode": null,
            "seaState": null,
            "sightingDistance": null,
            "sightingPlatform": null,
            "travelDirection": 10,
            "vesselName": null,
            "whenInWater": null,
            "whyInWater": null,
            "windSpeed": null,
            "latitude": 48.02038,
            "speciesId": 1,
            "sightedDate": "2023-11-03T16:35:00.000Z",
            "sourceType": "observed",
            "sourceName": "other",
            "sourceEntity": "Acartia",
            "externalID": "10c3993d-39ce-4d59-ab43-33f26b3c4e1a"
        },

This seems to have worked great, thank you so much Scott and Ali for the work!
I wonder if, given the SKRW are in the Sound, that we leave the API as is for the weekend, and revisit on Monday to make sure all is well?  I'm happy to take your steer on this given the time change switch coming up!

Thanks for helping with an end-to-end test, Alex. I'm comfortable leaving it running as I'm confident we will be able to change the script that night (when there won't be any incoming sightings), so won't contaminate the production site with erroneous time stamps.

Fantastic, look forward to picking this up again on Monday! Thanks so much both, and have a brilliant weekend!

Hi @scottveirs, hope you're doing okay.
We've just had a meeting with our developers, we're clarifying the attributions in app and it should be done by the end of the week.
Sorry to add more work to your plate, but we're wondering if you could edit your POST slightly to ensure that it aligns with how we will also attribute hydrophone detections and Whale Report detections etc.
What this - could you please submit...
orgName: Acartia
sourceEntity: Orca Network via Whale Alert

(flipping the current submission of orgName and sourceEntity)
This way we do not have to hard code any values, and we're ensuring all organizations are following the same structure.
To confirm, following the data sharing agreement, does the above look okay in terms of attributing the correct orgs (I assume we would like to put Whale Alert on the info card to attribute them as well)?
Lastly, the website is updated with the current data providers, please do let me know what you think!
Thanks so much again!

@almitch,
I've just made the requested field flip. I've modified the sourceEntity value to Orca Network via Conserve.io apps.
This should clarify for WRAS end users that Orca Network and other Washington sightings providers are using not only Whale Alert mobile app for geo-referencing occurrence data, but also the Cascadia web app. Both apps were developed by Conserve.io. Eventually, another Conserve.io mobile app, Ocean Alert will also provide observations to the Acartia.io data cooperative...

Re attribution, @almitch -- from the perspective of Beam Reach, the current attribution on the WRAS web page looks good to me:

I'll start an email thread to get feedback from the other data cooperative members.

Thanks a lot @scottveirs - that's brill. Much appreciated.
The updates to the app should be ready for testing my OW Staff today, so hopefully they will be able to go live tomorrow or Wednesday.
I'll await your email, or look out for another GitHub post, RE the website attribution.
Cheers!

Hi @scottveirs - Lapis pushed an update to the API just now which fixes the timezone issue we were experiencing before.
When you get the chance could you please remove the "hack" you put on your POST and submit data in UTC (which will be displayed as PST on the front end). This will ensure that mariners are alerted at the correct time!
Cheers!

Hey @almitch - The hack has been removed. You should see next Orca Network posts in UTC now...

Happy summer, @almitch and @dylan-oceanwise -- I may have just found a bug and/or ambiguity in the WRAS API.
I was trying to check on an Orca Network Acartia data point that WSF said didn't show up via WRAS a few days ago. I tried using the date range parameters in the API call, replicating the date-time format that is returned by the WRAS API from a GET request with no parameters, which PostMan formats like this:
https://sightingsapi.ocean.org/sightings?sightedFrom=2024-07-14T16:00:00Z&sightedTo=2024-07-14T19:00:00Z
Here's a screenshot of the query and WRAS API response:

The response appears to be what I'd expect if the sightedFrom and sightedTo parameters were submitted in the local/Pacific time zone (GMT-7 or Z-7), but I tried to specify my desired time range in GMT-0 with the trailing Z -- consistent with the date-time format of the WRAS API responses. (You've previously assured me that all response times really are in GMT-0 or Z.)
So, I think this suggests either that your API documentation should make the input and output date-time formats and time zones unambiguous (and ideally provide some syntax-perfect examples)...
...or, there is a bug in the API backend code which is assuming the input From/To times aren't already GMT-0 times and is therefore adding 7 hours to them (despite the user trying to specify they are already in GMT-0 with the trailing Z).

@scottveirs  - thanks so much for the detailed description of this bug (which certaintly is a bug), I've been taking a look at this this morning.
We recently had to do a rollback on some changes to WRAS and updates to the API due to a deprecaited docker compose script that was found... As of the time of writing your comment it might have overlapped with our downtime due to this rollback. I'm also imagining this bug may have been introduced then.
You're totally right its erronously adding 7hrs. The database currently assumes all data is in GMT.  It does not impact performance of WRAS, data is being logged and reported correctly, but spat out by the API incorrectly when using the filters.
I've raised a ticket with our devs to address this. Thanks for pointing this out.
If i've mentioned this before, ignore, but we are moving away from our current developers to a new local group, so, I'm hoping, there will not be any issues like this in the new year.


Document how plugins should test for working gulp-sourcemaps support

Resolved in https://github.com/floridoo/gulp-sourcemaps/commit/90f2128c50791925c617323f0a44248c748b4d37


Upgrade semantic-release-gitmoji to fix security vulnerabilities

I created a PR, momocow/semantic-release-gitmoji#45, which fixes the security vulnerabilities in semantic-release-gitmoji. Once that is merged and a new version is released, we should upgrade the dependency to fix the security vulnerabilities.

:tada: This issue has been resolved in version 1.2.0-alpha.3 :tada:
The release is available on:

npm package (@alpha dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Add new component: show/hide truncated text

A pure css component with :target, row truncation via line-clamp and flex
in order to create together a fully functional show/hide toggle for
truncated text(in specific rows that we assign with the help of a mixin).
Demo: http://codepen.io/srekoble/pen/WGBzZa
Cheers!

Can you maybe add also Mozilla prefixes to support Firefox (if possible)?

Sure @PMK will add in the next hours

@PMK updated solution with a firefox fallback.
demo: https://codepen.io/srekoble/pen/WGBzZa?editors=1100

Super cool demo! I'm going to take a look and see if we could simplify it in any way for the website code

Thanks @una 😄 . Feel free to request any changes.

When will this be added?


I canot reopen issues you've closed

I have provided the needed information for an issue, but since I did not close it I have no chance to reopen it.
https://github.com/Microsoft/testfx/issues/554

@Slesa Apologies for that. Please ping on the original issue, one of us would re-open for such cases. Thanks for understanding.
I have re-opened the original issue.


[WIP] ⚡❓ AMP app integration

refs TryGhost/Ghost#7769
Adds Google AMP to apps in order to disable and enable it on request.

@AileenCGN can you rebase this when you get a chance?

@acburdine already on it, but it's going to be completely different... No checkbox in the settings, but an option in apps, just like Slack.

Right, carry on then :)

Coverage decreased (-0.9%) to 80.216% when pulling 6f347a12529b2e0e2b3294f7cbd398e530fb0779 on AileenCGN:amp-optional-alpha into<PHONE_NUMBER>712528a9e3093d1f4bea081a0eb1ee on TryGhost:master.

Coverage decreased (-0.9%) to 80.216% when pulling 6f347a12529b2e0e2b3294f7cbd398e530fb0779 on AileenCGN:amp-optional-alpha into<PHONE_NUMBER>712528a9e3093d1f4bea081a0eb1ee on TryGhost:master.

Coverage increased (+0.2%) to 81.284% when pulling da89cdc10b2d2c944f6846dd9555dfc4a8ceeb60 on AileenCGN:amp-optional-alpha into<PHONE_NUMBER>712528a9e3093d1f4bea081a0eb1ee on TryGhost:master.

@kevinansfield this is ready for review!
We'll fix the styles a bit later, when we (@JohnONolan)  decided, how it should look like.

Coverage increased (+0.2%) to 81.284% when pulling 9db4e71ea24d8ef55ce256d34310d75a59f249d7 on AileenCGN:amp-optional-alpha into<PHONE_NUMBER>712528a9e3093d1f4bea081a0eb1ee on TryGhost:master.

Coverage increased (+0.2%) to 81.279% when pulling 6c14fb11a8695b92df865ab496c03daa5025c658 on AileenCGN:amp-optional-alpha into 8b86b7eeb1a5ad80218bf7b0acb18230f7939e91 on TryGhost:master.


Support copy project_contents default language to new resource

Following up on a conversation with @camallen. Handy to have, so the frontend doesn't need to fill the fields manually.

Just fleshing this out, specifically being able to specify  a new language code that would copy the default language (en) for a *_contents resource to seed new language code translations.

The new API should support filling in the default lang strings. this has to happen.

Should this issue be reworded? It seems like it's overly concerned with implementation details, and I'm not even sure they're correct anymore since we now have translation resources. Perhaps:
Support creating and prefilling a new language for a project
Panoptes should have a function to create a new language for a project. This should prefill the strings with those from the default language. This should create translations for all associated resources in the project, so that everything is ready to be translated.

yep that's the what we want please reword

This look to be handled by the translations app now, it'll create a new translation for a new language key but copy the primary language strings over.


I hope this is okay under your code of conduct, I think it is. thanks for reading.


Set better default resources

The opentelemetry-collector only defines limits by default,  if you don’t specify requests, Kubernetes will automatically set requests to equal to Pod’s limits. This means that with the default Helm  values, opentelemtry will take up 2Gb and 1cpu, which is insane.
I think there should be default requests set so that K8s does not schedule these pods with such high resources.

This is an interesting topic since it really comes down to "what should the chart suggest as a starting point for resources". The current value came from the old standaloneCollector value, which was set high to accommodate the "both" mode where agent collectors forwarded traces to a standalone collector.
The collector is efficient, but in my experience it does take a some resources to run properly. I think we could lower this default value tho since the old "both" mode is no longer supported by a single instance of the chart.


Bulk Insert throwing error during test

On the notes box I'm getting in the develop branch
../src/MatrixOps.chpl:19: In function 'wFromPG':
> ../src/MatrixOps.chpl:31: error: A standalone or leader iterator is not found for the iterable expression in this forall loop
[Error matching compiler output for /home/buddha314/numsuch/test/bulk_insert-test]

@Tshimanga I know you warned me about this, I'm just putting it here to catalog the effort.

Hmm, odd. The tests just ran fine for me...  I also have the benchmarks tests running fine for batch/serial/parallel insertion, parallel/serial extraction, and random matrix generation. How did this error arise exactly?

Here is the complete error from running the batch insert test
macbuddha:test buddha$ start_test bulk_insert-test.chpl

[Removing log file with duplicate name /Users/buddha/github/chapel/test/Logs/buddha.darwin.log
[Starting Chapel regression tests -<PHONE_NUMBER>49]
[Warning: start_test not invoked from a subdirectory of $CHPL_HOME]
[starting directory: "/Users/buddha/github/buddha314/numsuch/test"]
[Logs directory: "/Users/buddha/github/chapel/test/Logs"]
[log_file: "/Users/buddha/github/chapel/test/Logs/buddha.darwin.log"]
[CHPL_HOME: /Users/buddha/github/chapel]
[host platform: darwin]
[target platform: darwin]
[valgrind: OFF]
[performance tests: OFF]
[compiler performance tests: OFF]
[number of trials: 1]
[performance graph generation: OFF]
[compiler: "/Users/buddha/github/chapel/bin/darwin/chpl"]
[compopts: "--cc-warnings"]
[execopts: ""]
[launchcmd: ""]
[comm: "none"]
[localeModel: "flat"]
[numlocales: "(default)"]
[max_locales: "(default)"]
[tests: "/Users/buddha/github/buddha314/numsuch/test/bulk_insert-test.chpl"]
[directories: ""]

### Chapel Environment ###
machine info: Darwin macbuddha.lan 16.7.0 Darwin Kernel Version 16.7.0: Mon Nov 13 21:56:25 PST 2017; root:xnu-3789.72.11~1/RELEASE_X86_64 x86_64
CHPL_HOME: /Users/buddha/github/chapel *
script location: /Users/buddha/github/chapel/util
CHPL_TARGET_PLATFORM: darwin
CHPL_TARGET_COMPILER: clang
CHPL_TARGET_ARCH: native
CHPL_LOCALE_MODEL: flat *
CHPL_COMM: none *
CHPL_TASKS: qthreads
CHPL_LAUNCHER: none *
CHPL_TIMERS: generic
CHPL_UNWIND: none
CHPL_MEM: jemalloc
CHPL_ATOMICS: intrinsics
CHPL_GMP: gmp
CHPL_HWLOC: hwloc
CHPL_REGEXP: re2
CHPL_AUX_FILESYS: none
##########################

[Cleaning file /Users/buddha/github/buddha314/numsuch/test/bulk_insert-test.chpl]
[Starting /Users/buddha/github/chapel/util/test/sub_clean bulk_insert-test.chpl Mon Feb 05 08:27:49 PST 2018]
[Starting sub_clean - Mon Feb 05 08:27:49 PST 2018]
[pwd: /Users/buddha/github/buddha314/numsuch/test]
Cleaning test: bulk_insert-test

[Working on file bulk_insert-test.chpl]
[Starting /Users/buddha/github/chapel/util/test/sub_test Mon Feb 05 08:27:49 PST 2018]
[Starting subtest - Mon Feb 05 08:27:50 PST 2018]
[test: /Users/buddha/github/buddha314/numsuch/test/bulk_insert-test.chpl]
[Executing compiler /Users/buddha/github/chapel/bin/darwin/chpl -o bulk_insert-test --cc-warnings -M../src -M/Users/buddha/github/cdo/src -L/usr/local/Cellar/openblas/0.2.20/lib -lblas -I/usr/local/Cellar/openblas/0.2.20/include bulk_insert-test.chpl < /dev/null]
[Elapsed compilation time for "/Users/buddha/github/buddha314/numsuch/test/bulk_insert-test (compopts: 1)" - 6.153 seconds]
[Executing diff bulk_insert-test.good bulk_insert-test.1.comp.out.tmp]
1,5c1,2
< 1.0 0.0 1.0 0.0
< 1.0 0.0 1.0 1.0
< Something is a foot
< {1..2, 1..4}
< 3
---
> ../src/MatrixOps.chpl:19: In function 'wFromPG':
> ../src/MatrixOps.chpl:31: error: A standalone or leader iterator is not found for the iterable expression in this forall loop
[Error matching compiler output for /Users/buddha/github/buddha314/numsuch/test/bulk_insert-test]
[Elapsed time to compile and execute all versions of "/Users/buddha/github/buddha314/numsuch/test/bulk_insert-test" - 6.159 seconds]
[Finished subtest "/Users/buddha/github/buddha314/numsuch/test/bulk_insert-test" - 6.401 seconds]

[Removing /var/folders/g5/4zwyfk513hd94tjs1fm5mdq00000gn/T/chplTestTmpDir.QCwoGS directory]
[Done with tests -<PHONE_NUMBER>56]
[Log file: /Users/buddha/github/chapel/test/Logs/buddha.darwin.log ]

[Test Summary -<PHONE_NUMBER>56]
[Error matching compiler output for /Users/buddha/github/buddha314/numsuch/test/bulk_insert-test]
[Summary: #Successes = 0 | #Failures = 1 | #Futures = 0 | #Warnings = 0 ]
[Summary: #Passing Suppressions = 0 | #Passing Futures = 0 ]
[END]```

I'll dig deeper this morning.

And the chapel evn
> printchplenv
machine info: Darwin macbuddha.lan 16.7.0 Darwin Kernel Version 16.7.0: Mon Nov 13 21:56:25 PST 2017; root:xnu-3789.72.11~1/RELEASE_X86_64 x86_64
CHPL_HOME: /Users/buddha/github/chapel *
script location: /Users/buddha/github/chapel/util
CHPL_TARGET_PLATFORM: darwin
CHPL_TARGET_COMPILER: clang
CHPL_TARGET_ARCH: native
CHPL_LOCALE_MODEL: flat
CHPL_COMM: none
CHPL_TASKS: qthreads
CHPL_LAUNCHER: none
CHPL_TIMERS: generic
CHPL_UNWIND: none
CHPL_MEM: jemalloc
CHPL_ATOMICS: intrinsics
CHPL_GMP: gmp
CHPL_HWLOC: hwloc
CHPL_REGEXP: re2
CHPL_AUX_FILESYS: none

Okay, this was the obvious error, but I was using the master branch of CDO, not develop.  Got past that error, now I need to clean up some files so it passes.  Looks like we can close this one.


When specific renewBefore, duration_until_renewal in log is alternately show two different value.

Describe the bug:
When I set renewBefore: 360h at Certificate, then check the cert-manager Pod log:
I0326 07:49:26.726970       1 util.go:177] cert-manager/controller/certificates "msg"="certificate scheduled for renewal" "duration_until_renewal"="1798h41m20.273130427s" "related_resource_kind"="Secret" "related_resource_name"="cert-wildcard" "related_resource_namespace"="cessna-test" "resource_kind"="Certificate" "resource_name"="certificate" "resource_namespace"="cessna-test"
I0326 07:49:26.727055       1 sync.go:311] cert-manager/controller/certificates "msg"="certificate does not require re-issuance. certificate renewal scheduled near expiry time." "related_resource_kind"="CertificateRequest" "related_resource_name"="certificate-1651197699" "related_resource_namespace"="cessna-test" "resource_kind"="Certificate" "resource_name"="certificate" "resource_namespace"="cessna-test"
I0326 07:49:26.728733       1 controller.go:144] cert-manager/controller/certificates "msg"="finished processing work item" "key"="cessna-test/certificate"
I0326 07:49:26.900336       1 controller.go:138] cert-manager/controller/certificates "msg"="syncing item" "key"="cessna-test/certificate"

I0326 07:49:26.903319       1 sync.go:304] cert-manager/controller/certificates "msg"="updated Secret resource metadata as it was out of date" "related_resource_kind"="Secret" "related_resource_name"="cert-wildcard" "related_resource_namespace"="cessna-test" "resource_kind"="Certificate" "resource_name"="cert-wildcard" "resource_namespace"="cessna-test"
I0326 07:49:26.904075       1 util.go:177] cert-manager/controller/certificates "msg"="certificate scheduled for renewal" "duration_until_renewal"="1438h41m20.095980428s" "related_resource_kind"="Secret" "related_resource_name"="cert-wildcard" "related_resource_namespace"="cessna-test" "resource_kind"="Certificate" "resource_name"="cert-wildcard" "resource_namespace"="cessna-test"
I0326 07:49:26.904136       1 sync.go:311] cert-manager/controller/certificates "msg"="certificate does not require re-issuance. certificate renewal scheduled near expiry time." "related_resource_kind"="Secret" "related_resource_name"="cert-wildcard" "related_resource_namespace"="cessna-test" "resource_kind"="Certificate" "resource_name"="cert-wildcard" "resource_namespace"="cessna-test"
I0326 07:49:26.905356       1 controller.go:144] cert-manager/controller/certificates "msg"="finished processing work item" "key"="cessna-test/cert-wildcard"
I0326 07:49:26.905411       1 controller.go:138] cert-manager/controller/certificates "msg"="syncing item" "key"="cessna-test/cert-wildcard"

I0326 07:49:26.906854       1 util.go:177] cert-manager/controller/certificates "msg"="certificate scheduled for renewal" "duration_until_renewal"="1438h41m20.093197017s" "related_resource_kind"="Secret" "related_resource_name"="cert-wildcard" "related_resource_namespace"="cessna-test" "resource_kind"="Certificate" "resource_name"="cert-wildcard" "resource_namespace"="cessna-test"
I0326 07:49:26.906896       1 sync.go:311] cert-manager/controller/certificates "msg"="certificate does not require re-issuance. certificate renewal scheduled near expiry time." "related_resource_kind"="Secret" "related_resource_name"="cert-wildcard" "related_resource_namespace"="cessna-test" "resource_kind"="Certificate" "resource_name"="cert-wildcard" "resource_namespace"="cessna-test"
I0326 07:49:26.907794       1 controller.go:144] cert-manager/controller/certificates "msg"="finished processing work item" "key"="cessna-test/cert-wildcard"
I0326 07:49:27.121253       1 controller.go:138] cert-manager/controller/certificates "msg"="syncing item" "key"="cessna-test/cert-wildcard"

I0326 07:49:27.130299       1 sync.go:304] cert-manager/controller/certificates "msg"="updated Secret resource metadata as it was out of date" "related_resource_kind"="CertificateRequest" "related_resource_name"="certificate-1651197699" "related_resource_namespace"="cessna-test" "resource_kind"="Certificate" "resource_name"="certificate" "resource_namespace"="cessna-test"
I0326 07:49:27.136073       1 util.go:177] cert-manager/controller/certificates "msg"="certificate scheduled for renewal" "duration_until_renewal"="1798h41m19.863999693s" "related_resource_kind"="Secret" "related_resource_name"="cert-wildcard" "related_resource_namespace"="cessna-test" "resource_kind"="Certificate" "resource_name"="certificate" "resource_namespace"="cessna-test"
I0326 07:49:27.136348       1 sync.go:311] cert-manager/controller/certificates "msg"="certificate does not require re-issuance. certificate renewal scheduled near expiry time." "related_resource_kind"="CertificateRequest" "related_resource_name"="certificate-1651197699" "related_resource_namespace"="cessna-test" "resource_kind"="Certificate" "resource_name"="certificate" "resource_namespace"="cessna-test"
I0326 07:49:27.138097       1 controller.go:144] cert-manager/controller/certificates "msg"="finished processing work item" "key"="cessna-test/certificate"
I0326 07:49:27.146331       1 controller.go:138] cert-manager/controller/certificates "msg"="syncing item" "key"="cessna-test/certificate"

The "duration_until_renewal" has alternately appear in "duration_until_renewal"="1798h..." and "duration_until_renewal"="1438h..."
Expected behaviour:
I set renewBefore: 360h at Certificate, and the valid duration of certificate is 2160h, so duration_until_renewal should be around 1800h, so the 1798h is correctly here.
Steps to reproduce the bug:

Create a new Certificate, and set renewBefore is 360h

apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: certificate
spec:
  secretName: cert-wildcard
  duration: 2160h
  renewBefore: 360h
  dnsNames:
  - “*.api.example.com”
  issuerRef:
    name: letsencrypt
    kind: Issuer

Check the cert-manager Pod log, it will appear two different duration_until_renewal in log

Environment details::

Kubernetes version: v1.15.3:
Cloud-provider/provisioner (e.g. GKE, kops AWS, etc):
cert-manager version : 0.14.0 and 0.14.1
Install method (e.g. helm or static manifests): static manifests

/kind bug

It seems you are having 2 certificates here: certificate and cert-wildcard which is why you will see 2 different renewal times being logged.


Allow links into closed tabs

JS update to allow a link into a closed tab.
Tested with
:mc-conf:`~browser.hsts_seconds`

which goes to a tab in this section
@rushenn Ravind and Andrea came up with this, need a JS intervention to ensure it's actually reasonable. 🙏🏻
Staged
http://<IP_ADDRESS>:9000/staging/link-into-tab/linux/reference/minio-server/settings/console.html#strict-transport-security

@feorlen Sorry for the delayed response, somehow I missed the notification for this conversation. Added a comment, please take a look.

@ravindk89 Got it, currently, it works only on the page load. I didn't think about the hash-change on click. Will send a fix soon.

@ravindk89 @feorlen Updated the code here https://github.com/minio/docs/pull/1126/commits/2659212edb0e1cd6203bbf0c7f51221792d6266f
It should work now on both hash change and clicks as well.

Restaged and working for me. 🎉
@rushenn @ravindk89

Great! Let's shipit

Tested after merge, works 🎉


Update docs with info about onError

Closes: https://github.com/facebook/react/issues/25961
onError for the picture element works differently in React to Vanilla HTML

From the linked issue it sounds like the issue is only about bubbling. But this talks about source not being supported.
Could you clarify what the take-away from the issue was? I also left a note in the issue since I had trouble reproducing.

If following vanilla specs while creating a React app, a user will not successfully trigger the onerror of a <picture>.
If the difference between vanilla HTML/JS and React is desired due to the way React bubbles these events, the "React way" to trigger onerror should be documented.


Wooden tree fixes (CSS and bugfix)

Some pages in ManageIQ caused the tree unexpected visual changes. This commit fixes it.
Before:  After:

The react-wooden-tree component had a bug with default value: https://github.com/brumik/react-wooden-tree/pull/35 and a checkable bug: https://github.com/brumik/react-wooden-tree/pull/38

react-wooden-tree: rendering HTML: https://github.com/brumik/react-wooden-tree/pull/41

@miq-bot add_reviewer @skateman
@miq-bot add_reviewer @himdel

Could we have some screenshots about the visualities?

Could we have some screenshots about the visualities?

There was no visual change here. The CSS change just enforced some rules to apply also in the manageiq-ui-classic context as well.

Some pages in ManageIQ caused the tree unexpected visual changes.

???

There was no visual change here.

???

@brumik @skateman : merge?

Yes, I think there won't be more things for now.

:tada: This PR is included in version 0.11.58 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Issues sending multiple messages with HighLevelProducer

We are having trouble sending multiple messages to the same topic & partition with the HighLevelProducer. It's only sending the first message in the partition.
Here's our setup:

Using HighLevelProducer with the keyed partitioner type
Sending multiple payloads in an array to the send function
Each payload has a topic, a key, a single KeyedMessage, and a compression attribute

Because we are using the keyed partitioner type, and the keys for each message are unique, we must create a single payload per key instead. The HighLevelProducer send function determined the partition based on the partitioner, so there could be multiple payloads with the same topic and partition. The HighLevelProducer uses Client to sendProduceRequests with an array of payloads, which seems reasonable.
The Client then splits the payloads by broker based on leader per partition and uses the protocol.encodeProduceRequest to encode the list of payloads per broker.
The protocol _encodeProduceRequest immediately calls groupByTopic to split the payloads into a map of topics -> partitions -> payloads. This is where the problem arises. The issue is that groupByTopic assumes at this time that there is only one payload per topic partition, which isn't the case in this instance.
Therefore, the issue is either in protocol.groupByTopic because it should create an array of payloads per partition OR the issue is in HighLevelProducer.buildPayloads because it should be grouping the individual payloads by partition.
Unfortunately, we are unable to group them ourselves because we don't know the partition beforehand as that logic is in the HighLevelProducer.
Hopefully this is clear enough to be able to get the issue fixed. Please let me know if there's anything that needs further explanation.

Could you create small test case to reproduce this?
Sounds like the issue could be fixed by updating the buildPayloads method to normalize the payload by topic partition and attributes.

'use strict';

const kafkaNode = require('kafka-node');

const partitionerTypes = kafkaNode.HighLevelProducer.PARTITIONER_TYPES;
const theTopic = 'test.topic';

const compressionTypes = {
  none: 0,
  gzip: 1,
  snappy: 2,
};

const kafkaClient = new kafkaNode.Client('localhost:2181', 'test-consumer-group');

const theMessages = [
  { id: 'A', data: 1 },
  { id: 'B', data: 2 },
  { id: 'C', data: 3 },
  { id: 'D', data: 4 },
  { id: 'E', data: 5 },
];

const updateMetadataForTopic = (topic, cb) => {
  kafkaClient.loadMetadataForTopics([topic], (err, metadata) => {
    if (err) {
      cb(err);
      return;
    }

    kafkaClient.updateMetadatas(metadata);
    cb();
  });
};

const kafkaProducer = new kafkaNode.HighLevelProducer(kafkaClient, { partitionerType: partitionerTypes.keyed });

const sendMessages = (topic, messages, cb) => {
  const payloads = messages.map(message => ({
    topic,
    key: message.id,
    messages: [new kafkaNode.KeyedMessage(message.id, JSON.stringify(message))],
    attributes: compressionTypes.snappy,
  }));

  kafkaProducer.send(payloads, cb);
};

kafkaClient.on('ready', () => {
  console.log('Kafka client ready.');
});

kafkaClient.on('error', err => {
  console.error(`Kafka client error: ${err.stack || err.message}`, err);
  kafkaClient.close();
});

kafkaProducer.on('ready', () => {
  console.log(`Kafka producer ready.`);

  // required when using keyed partitioner for some reason
  updateMetadataForTopic(theTopic, err => {
    if (err) {
      console.error(`Error updating metadata for topic: ${err.stack || err.message}`, err);
      kafkaClient.close();
      return;
    }

    sendMessages(theTopic, theMessages, err2 => {
      if (err2) {
        console.error(`Error sending messages: ${err2.stack || err2.message}`, err2);
        kafkaClient.close();
        return;
      }

      console.log(`Sent ${theMessages.length} messages`);
      kafkaClient.close();
    });
  });
});

Added the test above.
Also notice that the metadata for a topic needs to be updated on the client before you can use the keyed partitioner. this is something we get around by just doing the update ourselves, but it was kind of tough to find the issue there.

I think the partitioners need the total number of partitions in a topic in order to calculate the partition to assign and that's retrieved from a call to client.refreshMetadata().
So I took a stab at fixing this. Maybe you point your package.json to SOHU-Co/kafka-node#payload-normalize and see if the issue persists or if you find any new issues? Basically my change just tries to normalize the ProduceRequest by topic partition.

Thank you so much for the really quick turnaround. I've tested this in the code path that was breaking before and it appears that this is exhibiting the correct behavior. After looking at the diff, I think this solution makes a lot of sense.

Thank you for testing. I will work on getting this published soon.

published in 1.0.7


LUCENE-10657: CopyBytes now saves one memory copy on ByteBuffersDataOutput

This is derived from LUCENE-10627 at #987
Jira: https://issues.apache.org/jira/browse/LUCENE-10657
The abstract method copyBytes in DataOutput have to copy from input to a copyBuffer and then write into ByteBuffersDataOutput.blocks, i think there is unnecessary, we can override it, copy directly from input into output.
with override this method, we can:

Reduce memory copy in Lucene90CompressingStoredFieldsWriter#copyOneDoc -> bufferdDocs.copyBytes(DataInput input)
Reduce memory copy in Lucene90CompoundFormat.writeCompoundFile -> data.copyBytes when input is BufferedChecksumIndexinput and output is ByteBuffersDataOutput
Reduce memory IndexWriter#copySegmentAsIs ->CopyFrom -> copyBytes

This looks good to me, can you add a CHANGES entry that copyBytes now saves one memory copy on ByteBuffersDataOutput?

Thanks for your review and suggestions, I have been updated the CHANGES entry


[docker] Intermittent build failures running bootstrap-vcpkg.sh

Describe the bug
I'm trying to build vcpkg in a debian 10 environment in docker. Fairly simple Dockerfile:
FROM debian:10

RUN apt update
RUN apt install -y git make curl gcc g++ unzip tar autoconf automake libtool libssl-dev valgrind

# Install vcpkg
RUN git clone https://github.com/Microsoft/vcpkg.git
WORKDIR /vcpkg
RUN ./bootstrap-vcpkg.sh

Sometimes it will pass. Sometime it will fail with:
[31/77] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/binarycaching.cpp.o
FAILED: CMakeFiles/vcpkglib.dir/src/vcpkg/binarycaching.cpp.o
/usr/bin/g++-8  -DVCPKG_DISABLE_METRICS=0 -DVCPKG_USE_STD_FILESYSTEM=1 -I../include -O3 -DNDEBUG   -pthread -std=c++17 -MD -MT CMakeFiles/vcpkglib.dir/src/vc
pkg/binarycaching.cpp.o -MF CMakeFiles/vcpkglib.dir/src/vcpkg/binarycaching.cpp.o.d -o CMakeFiles/vcpkglib.dir/src/vcpkg/binarycaching.cpp.o -c ../src/vcpkg/
binarycaching.cpp
g++-8: fatal error: Killed signal terminated program cc1plus
compilation terminated.

or with:
[67/77] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/export.prefab.cpp.o
FAILED: CMakeFiles/vcpkglib.dir/src/vcpkg/export.prefab.cpp.o
/usr/bin/g++-8  -DVCPKG_DISABLE_METRICS=0 -DVCPKG_USE_STD_FILESYSTEM=1 -I../include -O3 -DNDEBUG   -pthread -std=c++17 -MD -MT CMakeFiles/vcpkglib.dir/src/vc
pkg/export.prefab.cpp.o -MF CMakeFiles/vcpkglib.dir/src/vcpkg/export.prefab.cpp.o.d -o CMakeFiles/vcpkglib.dir/src/vcpkg/export.prefab.cpp.o -c ../src/vcpkg/
export.prefab.cpp
g++-8: fatal error: Killed signal terminated program cc1plus
compilation terminated.

or with:
[31/77] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/commands.ci.cpp.o
FAILED: CMakeFiles/vcpkglib.dir/src/vcpkg/commands.ci.cpp.o
/usr/bin/g++-8  -DVCPKG_DISABLE_METRICS=0 -DVCPKG_USE_STD_FILESYSTEM=1 -I../include -O3 -DNDEBUG   -pthread -std=c++17 -MD -MT CMakeFiles/vcpkglib.dir/src/vcpkg/commands.ci.cpp.o -MF CMakeFiles/vcpkglib.dir/src/vcpkg/commands.ci.cpp.o.d -o CMakeFiles/vcpkglib.dir/src/vcpkg/commands.ci.cpp.o -c ../src/vcpkg/commands.ci.cppg++-8: fatal error: Killed signal terminated program cc1plus
compilation terminated.

Given the ambiguity of the error messages and my lack of knowledge with some of these tools, I'm having a hard time figuring out what's wrong. Or whether this is a vcpkg bug or something I'm doing. I've built this image several times consecutively; no environment changes took place in between.
Environment

OS:

Host: MacOS Catalina 10.15.5
Container: Debian 10

Compiler: g++ (Debian 8.3.0-6) 8.3.0

To Reproduce
Steps to reproduce the behavior:

Create Dockerfile as mentioned above
run docker build --no-cache -t vcpkg/example .
Whether it succeeds or fails, run step 2 again.

Would be curious to know if someone can actually reproduce this as I'm not finding anything close in the issues.
Expected behavior
Consistent build results.
Failure logs

failure logs were pasted above

Same behavior on ubuntu in docker
FROM ubuntu:18.04

RUN apt update
RUN apt install -y build-essential tar curl zip unzip g++ git

RUN git clone https://github.com/microsoft/vcpkg
WORKDIR /vcpkg
RUN ./bootstrap-vcpkg.sh

note: changed the dockerfile to further match the getting started page. Same intermittent issues.

I cannot reproduce this on my local machine with Ubuntu 18.04. It might be something related with docker.
@ras0219-msft
Could you please help take a look?

For anyone running into this error now, check your docker memory limits. I had 2GB RAM assigned in the docker settings (Settings > Resources > Advanced), updated that to 10GB and restarted docker and it worked.
I was getting this error on any flavour of Debian/Ubuntu I could try but got success on Debian:10
(OSX 11.2.3, 16gb 2019 MBP)

@jamesalbert ,Could you try @undefinedfunction's suggestion to check your docker memory limits and configure the RAM to be bigger and see if it solves your problem? I don't have a docker test environment to reproduce this problem

Now in non-ARM Linux, vcpkg will download binary directly, can you please check whether this issue still exist?

We hope your question was answered to your satisfaction; if it wasn't, you can reopen with more info.


Change custom permissions in Android export to an array

Don't limit the amount of permissions and makes it simpler to understand.

Looks good to me.
@Calinou Can you take note of the compat breakage in your changelog?

Thanks!

@akien-mga Done :slightly_smiling_face:


Second factor next to Master/RegularKey signature for authorizing transactions

After reading a bit through the specs of U2F, FIDO2 and WebAuthn, I'm almost certain that it might be possible to:

Attach a list of https://www.w3.org/TR/webauthn/#public-key-credential to an AccountRoot
Add a few fields and flags (e.g. "2FA can be disabled with a n second lockdown period using a non-2FA transaction")
Add a few transaction types related to this (add/remove/modify 2FA list)
Add an https://www.w3.org/TR/webauthn/#authentication-assertion to transactions next to the signature(s) of the signing keypair(s)
Have validators check the signature of the address AND the additional 2FA signature before applying a transaction

A usage scenario might be that it would then be possible to get my account in a state where:

Every transaction requires me to also attach a signature from a YubiKey device next to the one from the actual wallet
If I followed best practices (kept my MasterKey to myself and only handed my RegularKey to a wallet provider) I can deal with a loss/disclosure of the RegularKey without any impact on my side (the attacker can't sign Payment transactions with just the RegularKey, only try to disable 2FA after e.g. 60 days) giving me time to switch the RegularKey
If I lose my 2FA device, I'm locked out of my account for a while, but will be able to regain access, depending on the implementation. I also could attach multiple devices, to be safe(r) against accidental loss.

I'm not sure if this is better done through multisig instead of natively supported, but some nicer features like a lockdown period are hard (impossible?) to implement that way though without external trust.

Interesting!
I'll have to read more about this, but my initial reaction is along the lines of your trailing comment:

I'm not sure if this is better done through multisig instead of natively supported, but some nicer features like a lockdown period are hard (impossible?) to implement that way though without external trust.

I too think that you're better off setting up multisigning. But I'll need to read more carefully to see if using WebAuthn may enable some cool use cases.
Depending on what you mean by "lockdown period" I think you can implement it without needing an external trust; just small improvements to XRP Ledger.

Just some side notes and random thoughts on this:

Attach a list of https://www.w3.org/TR/webauthn/#public-key-credential to an AccountRoot
Add a few fields and flags (e.g. "2FA can be disabled with a n second lockdown period using a non-2FA transaction")
Add a few transaction types related to this (add/remove/modify 2FA list)

It should also be possible to extract an exact public key and generate a valid xrp address from the public key credential returned by an authenticator so no extra field would be needed. The challenge then remains where to store Credential ID which is needed to make a valid signature. Imo, It's probably best not to store it on chain.
Another problem with WebAuthn is that although the new underlying protocol CTAP2 supports EdDsa signature algorithm (it doesn't support secp256k1 yet) the old U2F now called CTAP1 which webauthn also wraps supports only EC P-256 (secp256r1). At this time, most of the webauthn ready devices are the U2F device which would not be compatible with xrp authentication algorithms.
Then again, maybe it'd make sense to implement P-256 into xrpl. If anything, it makes possible to use SecureEnclave on iphone to sign xrp transactions.

-Add an https://www.w3.org/TR/webauthn/#authentication-assertion to transactions next to the signature(s) of the signing keypair(s)
-Have validators check the signature of the address AND the additional 2FA signature before applying a transaction

Xrpl's signature verification procedure will have to be modified in a way to allow hash tx first and concatenate this hash with webauthn specific data (authenticatorData). Then use the resulting data to verify signature against. What data is signed by an authenticaticator device can be seen here, Figure 4.

A usage scenario might be that it would then be possible to get my account in a state where:

Every transaction requires me to also attach a signature from a YubiKey device next to the one from the actual wallet
If I followed best practices (kept my MasterKey to myself and only handed my RegularKey to a wallet provider) I can deal with a loss/disclosure of the RegularKey without any impact on my side (the attacker can't sign Payment transactions with just the RegularKey, only try to disable 2FA after e.g. 60 days) giving me time to switch the RegularKey
If I lose my 2FA device, I'm locked out of my account for a while, but will be able to regain access, depending on the implementation. I also could attach multiple devices, to be safe(r) against accidental loss.

I'd argue this use case is interesting not only for WebAuth based devices but for any signature providing device. Imo the implementation should be generalized rather than fix specified for one protocol. Therefore, the easiest way would probably be to modify SignerList to allow weights based on transaction time delays.
This will also require an implementation of delayed transaction which was discussed in #2945

Generalizing/offering more options in multisign would also be a way I guess.

Most of what you want can be achieved with a 2-of-3 multisig scheme. The wallet owner keeps two keys (one hot, one cold) and the system operator keeps one. This allows transactions to be executed with both the owner's hot key and the operator's key. If the operator fails, the owner can sign transactions with both the hot and cod keys. The operator, holding only one key, cannot execute a transaction without the owner.

I agree with @JoelKatz that in this case, probably multisig is the solution.
What it doesn't give you is necessary time delay before owner can recover its account. Of course, if such setup is even required in the first place. I can't think of any good case where you would benifit from such setup but custodian account recovery. In this case, your account can be recovered with custodian keys after some time passed and custodians can't act against you before transaction isn't  fully executed..

Closing this - I think this can be achieved via other mechanisms.