automation is not running

This is related to the blueprint: auto_update_scheduled.yaml
Hi, Thank you for creating the blueprint. I highly appreciate the efforts in creating it.
I created automation based on the blueprint. It is triggered, but it is not running.
It is stopped at the 3rd condition.

When checking the traces (see below), I believe the trigger status is swapped. It is from "on" to "off". Should it not be from "off" to ''on"?
How can this be changed? Is this something that needs to be updated in the blueprint?
Best, Sebastian
this:
entity_id: automation.auto_update_home_assistant_on_a_schedule_base
state: 'on'
attributes:
last_triggered: '2023-03-09T08:00:00.011443+00:00'
mode: single
current: 1
id: '1678241096997'
friendly_name: Update_Auto Home Assistant on a schedule base
last_changed: '2023-03-08T15:17:58.497737+00:00'
last_updated: '2023-03-09T08:00:00.011657+00:00'
context:
id: 01GV2P4M06X5FS3YBZZA2KV825
parent_id: 01GV2P4M010K7XBKQWAXQ86WB9
user_id: null
trigger:
id: HA Schedule based
idx: '0'
alias: null
platform: state
entity_id: schedule.auto_update_schedule
from_state:
entity_id: schedule.auto_update_schedule
state: 'on'
attributes:
editable: true
next_event: '2023-03-09T03:30:00-05:00'
friendly_name: Auto_Update_Schedule
last_changed: '2023-03-09T08:00:00.001905+00:00'
last_updated: '2023-03-09T08:00:00.001905+00:00'
context:
id: 01GV2P4M010K7XBKQWAXQ86WB9
parent_id: null
user_id: null
to_state:
entity_id: schedule.auto_update_schedule
state: 'off'
attributes:
editable: true
next_event: '2023-03-10T03:00:00-05:00'
friendly_name: Auto_Update_Schedule
last_changed: '2023-03-09T08:30:00.002221+00:00'
last_updated: '2023-03-09T08:30:00.002221+00:00'
context:
id: 01GV2QVHT2HKT4C7YD7NDAW0NH
parent_id: null
user_id: null
for: null
attribute: null
description: state of schedule.auto_update_schedule
input_schedule_monthday: 1
input_pause_entities: []
temp_input_update_exclusions: []
input_update_exclusions: []

If you go to Developer Tools > Status and look for your schedule (schedule.auto_update_schedule?), what is this status?
Could you please share your automation's yaml formatted as code?
I'm using this exactly same version and it is working fine with me, so it is probably something in your settings.

Hi, thanks for looking into it.
The status of the schedule is currently 'off'. (8:50pm)
The schedule is set up to become 'on' at 3am every day for 30min.
(Also, the automation is triggered when it returns to 'off' at 3.30am)
My automation's yaml is:
- id: '1678241096997'   alias: Update_Auto Home Assistant on a schedule base   description: ''   use_blueprint:     path: edwardtfn/auto_update_scheduled.yaml     input:       restart_bool: true       schedule_entity: schedule.auto_update_schedule       actions_pre_update:       - service: notify.mobile_app_iphoneseb         data:           message: Updates are installed for {{ updates_list }}       actions_pre_restart:       - service: notify.mobile_app_iphoneseb         data:           message: Updates done, restarting       actions_pos_update:       - service: notify.mobile_app_iphoneseb         data:           message: Updates completed for {{ updates_list }}
Thanks a lot for your help in advance.
Best, Sebastian

Your automation is set to backup before the updates, could y you please double check if you don't have a backup started around 03:00AM?
You can try increasing the time for the update. Now is only 30min, try increasing that to 3h, or so, and let me know the result.
Also, if you could take a look in your Logbook and filter for the past 24h for only the activity related to this automation, please share the results with me.

hi,
Thanks for looking into this. No, I don't have any other backup started at 3 am.
I increased the helper to 3h as recommended.
Here are the logs as requested:
March 9, 2023
Auto-update Done! triggered by automation Update_Auto Home Assistant on a schedule base triggered by state of Auto_Update_Schedule
4:00:23 AM - 3 days ago
Auto-update Running post-update actions... triggered by automation Update_Auto Home Assistant on a schedule base triggered by state of Auto_Update_Schedule
4:00:18 AM - 3 days ago
Auto-update 0 item(s) pending a restart: - triggered by automation Update_Auto Home Assistant on a schedule base triggered by state of Auto_Update_Schedule
4:00:18 AM - 3 days ago
Auto-update Remaining updates: - Home Assistant Core Update triggered by automation Update_Auto Home Assistant on a schedule base triggered by state of Auto_Update_Schedule
4:00:18 AM - 3 days ago
Auto-update Finishing update process. triggered by automation Update_Auto Home Assistant on a schedule base triggered by state of Auto_Update_Schedule
4:00:18 AM - 3 days ago
Auto-update Backup triggered. triggered by automation Update_Auto Home Assistant on a schedule base triggered by state of Auto_Update_Schedule
3:00:18 AM - 3 days ago
Auto-update Backing up Home Assistant. triggered by automation Update_Auto Home Assistant on a schedule base triggered by state of Auto_Update_Schedule
3:00:08 AM - 3 days ago
Auto-update Running pre-update actions... triggered by automation Update_Auto Home Assistant on a schedule base triggered by state of Auto_Update_Schedule
3:00:07 AM - 3 days ago
Auto-update List of updates: - Home Assistant Core Update triggered by automation Update_Auto Home Assistant on a schedule base triggered by state of Auto_Update_Schedule
3:00:07 AM - 3 days ago
Auto-update A new update is available for Home Assistant. triggered by automation Update_Auto Home Assistant on a schedule base triggered by state of Auto_Update_Schedule
3:00:07 AM - 3 days ago
Auto-update Variables: input_backup_bool: True, input_restart_bool: True, input_schedule_monthday: 1, input_update_exclusions: [], input_core_os_update_mode: all, input_firmware_update_mode: all, updates_list: ['update.home_assistant_core_update'] triggered by automation Update_Auto Home Assistant on a schedule base triggered by state of Auto_Update_Schedule
3:00:07 AM - 3 days ago
Update_Auto Home Assistant on a schedule base triggered by state of Auto_Update_Schedule
3:00:00 AM - 3 days ago
In the logs, it seems like that is run at 3am. But why did it re-trigger at 3.30am, see my first screenshot?

I've just uploaded an updated version. Could you please try that one?


[STORM-1596] Do not use single Kerberos TGT instance between multiple threads.

This needs to be ported to 1.x-branch as well.

The Travis CI test failure seems unrelated to this issue.

+1 we might want to look at porting this to 0.10.x also.

Here #1180  is pull request to port these changes to 0.10.x-branch


Send asset evictions notices to Flutter framework

[x] Send asset eviction notices to Flutter framework

Blocked by https://github.com/flutter/flutter/pull/5241

@Hixie


BRS-732: BCparks api maxing out and causing slow - unresponsiveness i…

…n alerts/advisories
Jira Ticket:
BRS-732
Jira Ticket URL:
https://bcparksdigital.atlassian.net/browse/BRS-732
Description:
This changes the way park-access-statues api call works.  It used to be a large and expensive operation.  Now, it is done in a cron once every 5 minutes, instead of every time a pageload of https://bcparks.ca/wildfire or the https://bcparks.ca/explore/parks page (or others).   The cron saves this cached payload into a new record type in strapi.  There is a new endpoint created to pull the cached data that the cron created.  The old endpoint is now force limited to _limit: 10 to avoid any future performance issues.  Consumers of the old endpoint should use the new cached endpoint.
New endpoint:
[2022-07-28T17:14:45.031Z] debug GET /park-access-statuses-cache (52 ms) 200
Old endpoint:
[2022-07-28T17:15:18.373Z] debug GET /park-access-statuses?_limit=1200 (1940 ms) 200

Changes looks good, just one comment on console logs, would it be safe on PROD to print values or these are just for testing purposes?

Changes looks good, just one comment on console logs, would it be safe on PROD to print values or these are just for testing purposes?

I'm wanting to continue to view the logs with detail in PROD just in case we have other cases to deal with.

Changes looks good, just one comment on console logs, would it be safe on PROD to print values or these are just for testing purposes?

I'm wanting to continue to view the logs with detail in PROD just in case we have other cases to deal with.

Sounds good as long as it’s basically for tracing the issue.


Remove deprecated finder methods

cleans up specs
removed plenty of deprecation warnings
new installs will get default from formtastic form builder
merges split tests (via shared contexts) and uses default class finder there

closes #1104
closes #1106
closes #1107
needs:

[ ] wiki page describing the input finders

@justinfrench it contains the diff between master and 4.0-dev.
Are you sure there will be some releases of 3.x ? Because otherwise we could have 3.1-stable for 3.1.x and master pointing to 4.0.

@mikz I don't think there'll be a 3.2 or 3.3, but I prefer we switch master to track our 4.x development after the RCs and/or Betas (4.0.0 has been released).

@justinfrench ok! Will adjust the plan in #1197

@mikz I actually don't care too much — if you already have a nice plan, roll with it :)

@justinfrench my only point for using master is that is is the default branch and people could open PRs that would be not mergeable afterwards. We could also change the default branch to 4.0-dev temporarily.

@mikz you're right, let's just go with master

@justinfrench I think this is good to go into 4.0-dev. The spec changes are just copying part of a file to another file.

:+1:


Outdated stages docs

Describe the feature
parts of https://github.com/paradigmxyz/reth/blob/main/docs/repo/crates/stages.md are no longer accurate, for example the Pipeline struct
probably a good idea to reduce/remove code here.
Additional context
No response

Hey I can work on this 🤓

@matias-gonz the ideal solution would be to start moving all these notes into the code comments! that way we don't need to maintain outdated code in the docs


Support array/dictionary indexing

#269 adds some new methods, which shows the need for some new language elements:
var env = Libc.environ();
print env.get_Item("PATH");

The above works, but is anywhere near the elegance we are striving for. Here's the suggested syntax that should be supported:
var env = Libc.environ();
print env["PATH"];

...in other words, the same as in C#, Ruby, Python and JavaScript (but interestingly enough, not in Java; or rather, Java only supports it for built-in arrays (String[], Object[] etc), not for other types like Map<TKey, TValue>.)
Initially, we can make it work just for Dictionary<TKey, TValue>, but in the longer run it would be great to support other indexable types as well (List<T> and arrays - the latter would need language support to be able to define arrays though, and the former cannot be instantiated from Perlang since we currently don't support neither the new keyword nor using generics. :see_no_evil: However, as noted above, we might still run into arrays or lists being returned from library-level functions, so perhaps we should support such indexing in the language right away. :thinking:).

This feels like one of the more critical things we are missing at the moment. Let's aim for including it in 0.3.0; we can always bump it a few more versions if needed.

Retained open despite #363 being merged because of the missing features: https://github.com/perlang-org/perlang/pull/363/files#r1104893169

What we have now is good enough for getting 0.3.0 out the door. Moving the remaining work to the 0.4.0 iteration.

This is not yet working for compiled mode (#406). We should fix that, hopefully in the next 0.6.0 iteration.


Add macOS arm builds to CI

Announcement: https://github.com/actions/runner-images/issues/8439
Blog Post: https://github.blog/2023-10-02-introducing-the-new-apple-silicon-powered-m1-macos-larger-runner-for-github-actions/

It's not actually running arm. Kotlin still builds for x86_64 and skips all arm tests
https://github.com/toxicity-io/sqlite-mc/actions/runs/6404721150/job/17386647745?pr=49

@05nelsonm You can try FlyCI's M1 and M2 runners. They are on average 2x faster and 2x cheaper than GitHub's AND we have a free tier for OSS projects (see below).
Install Instructions

Install FlyCI app and
Easily replace one line of code and start using FlyCI runners:

jobs:
 ci:
-    runs-on: macos-latest
+    runs-on: flyci-macos-large-latest-m1
   steps:
   - name: 👀 Checkout repo
     uses: actions/checkout@v4

500 mins/month Free for Public Repos
Since your repo is public, FlyCI offers 500 mins/month of free M1 runner usage with the flyci-macos-large-latest-m1 runner for public projects.
Don't hesitate to contact us in case the free tier doesn't suit your needs or you experience any issues with the runners. Our team is here to support you!
Best Regards,
Veselina Radeva
Product Manager at FlyCI


[SPARK-31359][SQL][3.0] Speed up timestamps rebasing

What changes were proposed in this pull request?
In the PR, I propose to optimise the DateTimeUtils.rebaseJulianToGregorianMicros() and rebaseGregorianToJulianMicros() functions, and make them faster by using pre-calculated rebasing tables. This approach allows to avoid expensive conversions via local timestamps. For example, the America/Los_Angeles time zone has just a few time points when difference between Proleptic Gregorian calendar and the hybrid calendar (Julian + Gregorian since 1582-10-15) is changed in the time interval 0001-01-01 .. 2100-01-01:

i
local  timestamp
Proleptic Greg. seconds
Hybrid (Julian+Greg) seconds
difference in minutes

0
0001-01-01 00:00
-62135568422
-62135740800
-2872

1
0100-03-01 00:00
-59006333222
-59006419200
-1432

...
...
...
...
...

13
1582-10-15 00:00
-12219264422
-12219264000
7

14
1883-11-18 12:00
-2717640000
-2717640000
0

The difference in microseconds between Proleptic and hybrid calendars for any local timestamp in time intervals [local timestamp(i), local timestamp(i+1)), and for any microseconds in the time interval [Gregorian micros(i), Gregorian micros(i+1)) is the same. In this way, we can rebase an input micros by following the steps:

Look at the table, and find the time interval where the micros falls to
Take the difference between 2 calendars for this time interval
Add the difference to the input micros. The result is rebased microseconds that has the same local timestamp representation.

Here are details of the implementation:

Pre-calculated tables are stored to JSON files gregorian-julian-rebase-micros.json and julian-gregorian-rebase-micros.json in the resource folder of sql/catalyst. The diffs and switch time points are stored as seconds, for example:

[
  {
    "tz" : "America/Los_Angeles",
    "switches" : [ -62135740800, -59006419200, ... , -2717640000 ],
    "diffs" : [ 172378, 85978, ..., 0 ]
  }
]

The JSON files are generated by 2 tests in RebaseDateTimeSuite - generate 'gregorian-julian-rebase-micros.json' and generate 'julian-gregorian-rebase-micros.json'. Both tests are disabled by default.
The switches time points are ordered from old to recent timestamps. This condition is checked by the test validate rebase records in JSON files in RebaseDateTimeSuite. Also sizes of the switches and diffs arrays are the same (this is checked by the same test).

The Asia/Tehran, Iran, Africa/Casablanca and Africa/El_Aaiun time zones weren't added to the JSON files, see SPARK-31385

The rebase info from the JSON files is placed to hash tables - gregJulianRebaseMap and julianGregRebaseMap. I use AnyRefMap because it is almost 2 times faster than Scala's immutable Map. Also I tried java.util.HashMap but it has worse lookup time than AnyRefMap in our case.
The hash maps store the switch time points and diffs in microseconds precision to avoid conversions from microseconds to seconds in the runtime.

I moved the code related to days and microseconds rebasing to the separate object RebaseDateTime to do not pollute DateTimeUtils. Tests related to date-time rebasing are moved to RebaseDateTimeSuite for the same reason.

I placed rebasing via local timestamp to separate methods that require zone id as the first parameter assuming that the caller has zone id already. This allows to void unnecessary retrieving the default time zone. The methods are marked as private[sql] because they are used in RebaseDateTimeSuite as reference implementation.

Modified the rebaseGregorianToJulianMicros() and rebaseJulianToGregorianMicros() methods in RebaseDateTime to look up the rebase tables first of all. If hash maps don't contain rebasing info for the given time zone id, the methods falls back to the implementation via local timestamps. This allows to support time zones specified as zone offsets like '-08:00'.

Why are the changes needed?
To make timestamps rebasing faster:

Saving timestamps to parquet files is ~ x3.8 faster
Loading timestamps from parquet files is ~x2.8 faster.
Loading timestamps by Vectorized reader ~x4.6 faster.

Does this PR introduce any user-facing change?
No
How was this patch tested?

Added the test validate rebase records in JSON files to RebaseDateTimeSuite. The test validates 2 json files from the resource folder - gregorian-julian-rebase-micros.json and julian-gregorian-rebase-micros.json, and it checks per each time zone records that

the number of switch points is equal to the number of diffs between calendars. If the numbers are different, this will violate the assumption made in RebaseDateTime.rebaseMicros.
swith points are ordered from old to recent timestamps. This pre-condition is required for linear search in the rebaseMicros function.

Added the test optimization of micros rebasing - Gregorian to Julian to RebaseDateTimeSuite which iterates over timestamps from 0001-01-01 to 2100-01-01 with the steps 1 ± 0.5 months, and checks that optimised function RebaseDateTime.rebaseGregorianToJulianMicros() returns the same result as non-optimised one. The check is performed for the UTC, PST, CET, Africa/Dakar, America/Los_Angeles, Antarctica/Vostok, Asia/Hong_Kong, Europe/Amsterdam time zones.
Added the test optimization of micros rebasing - Julian to Gregorian to RebaseDateTimeSuite which does similar checks as the test above but for rebasing from the hybrid calendar (Julian + Gregorian) to Proleptic Gregorian calendar.
The tests for days rebasing are moved from DateTimeUtilsSuite to RebaseDateTimeSuite because the rebasing related code is moved from DateTimeUtils to the separate object RebaseDateTime.
Re-run DateTimeRebaseBenchmark at the America/Los_Angeles time zone (it is set explicitly in the PR #28127):

Item
Description

Region
us-west-2 (Oregon)

Instance
r3.xlarge

AMI
ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20190722.1 (ami-06f2f779464715dc5)

Java
OpenJDK 64-Bit Server VM 1.8.0_242 and OpenJDK 64-Bit Server VM 11.0.6+10

Test build #121009 has started for PR 28163 at commit df3f9a6.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25701/
Test PASSed.

@cloud-fan A lot of conflicts occurred because 3.0 doesn't include the changes https://github.com/apache/spark/pull/27618

Test build #121009 has finished for PR 28163 at commit df3f9a6.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121009/
Test FAILed.

The fail https://github.com/apache/spark/pull/28163#issuecomment-611414563 is not related to changes:
org.apache.spark.sql.kafka010.KafkaMicroBatchV2SourceSuite

I will re-run the build

jenkins, retest this, please

Test build #121026 has started for PR 28163 at commit df3f9a6.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25717/
Test PASSed.

Test build #121026 has finished for PR 28163 at commit df3f9a6.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121026/
Test FAILed.

Again not related test failure https://github.com/apache/spark/pull/28163#issuecomment-611452946:
org.apache.spark.sql.hive.thriftserver.CliSuite.SPARK-26321 Should not split semicolon within quoted string literals

jenkins, retest this, please

Test build #121029 has started for PR 28163 at commit df3f9a6.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25720/
Test PASSed.

Test build #121029 has finished for PR 28163 at commit df3f9a6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121029/
Test PASSed.

thanks, merging to 3.0!

We need explicitly say the reason why we need it is for fixing the regression instead of making it faster. Is my understanding right?

Saving timestamps to parquet files is ~ x3.8 faster
Loading timestamps from parquet files is ~x2.8 faster.
Loading timestamps by Vectorized reader ~x4.6 faster.

I think we should use 2.4 as the baseline. E.g. before this PR, the rebase makes 3.0 x times slower than 2.4, after this PR, it's x percent. @MaxGekk can you update it?


Improve documentation

CONTRIBUTING and CODE_OF_CONDUCT

r? @yn-bot
(rustbot-yn has picked a reviewer for you, use r? to override)

Some changes occurred in README
cc @yn-bot


add spectrum mask to desi

Adds a spectrum mask to DESI indicating whether a pixel in a spectrum is bad. Follows the SDSS convention of mask True=bad pixel. This mask is just a boolean which is turned on if any bit in the DESI SPECMASK is on, or if the ivar is below 1e-6 (following SDSS).

tests fail because of this https://github.com/huggingface/datasets/pull/6976 (numpy 2.0 breaks some things in HF datasets, they fixed it a couple hours ago but the pip install in the GH action runner still uses the old version of HF datasets)


bug(storage): ObjectError: NoSuchKey: The specified key does not exist

Describe the bug
Running with S3 storage and encountered error: Hummock error: ObjectStore failed with IO error Internal error: read "hummock_001/241/9710.meta" in block None failed, error: NoSuchKey: The specified key does not exist.
2022-09-08T09:12:50.561605Z ERROR risingwave_storage::monitor::monitored_store: Failed in iter: Hummock error: ObjectStore failed with IO error Internal error: read "hummock_001/241/9710.meta" in block None failed, error: NoSuchKey: The specified key does not exist.
  backtrace of `ObjectError`:
   0: <risingwave_object_store::object::error::ObjectError as core::convert::From<risingwave_object_store::object::error::ObjectErrorInner>>::from
             at ./src/object_store/src/object/error.rs:38:10
   1: <T as core::convert::Into<U>>::into
             at /rustc/9067d5277d10f0f32a49ec9c125a33828e26a32b/library/core/src/convert/mod.rs:550:9
   2: <risingwave_object_store::object::error::ObjectError as core::convert::From<aws_smithy_http::result::SdkError<E>>>::from
             at ./src/object_store/src/object/error.rs:81:9
   3: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual
             at /rustc/9067d5277d10f0f32a49ec9c125a33828e26a32b/library/core/src/result.rs:2134:27
   4: <risingwave_object_store::object::s3::S3ObjectStore as risingwave_object_store::object::ObjectStore>::read::{{closure}}
             at ./src/object_store/src/object/s3.rs:352:20
   5: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/9067d5277d10f0f32a49ec9c125a33828e26a32b/library/core/src/future/mod.rs:91:19
   6: <async_stack_trace::StackTraced<F> as core::future::future::Future>::poll
   7: <futures_util::future::future::fuse::Fuse<Fut> as core::future::future::Future>::poll
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.23/src/future/future/fuse.rs:86:37
   8: risingwave_object_store::object::MonitoredObjectStore<OS>::read::{{closure}}
             at ./src/object_store/src/object/mod.rs:478:46
   9: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/9067d5277d10f0f32a49ec9c125a33828e26a32b/library/core/src/future/mod.rs:91:19
  10: risingwave_object_store::object::ObjectStoreImpl::read::{{closure}}
             at ./src/object_store/src/object/mod.rs:309:9
  11: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/9067d5277d10f0f32a49ec9c125a33828e26a32b/library/core/src/future/mod.rs:91:19
  12: risingwave_storage::hummock::sstable_store::SstableStore::sstable::{{closure}}::{{closure}}::{{closure}}
             at ./src/storage/src/hummock/sstable_store.rs:370:48
  13: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/9067d5277d10f0f32a49ec9c125a33828e26a32b/library/core/src/future/mod.rs:91:19
  14: risingwave_common::cache::LruCache<K,T>::lookup_with_request_dedup::{{closure}}::{{closure}}
             at ./src/common/src/cache.rs:816:38
... omitted ...

To Reproduce

Modify risedev.yml to use S3 storage.
Start the cluster with ./risedev d full
Create sources:

CREATE SOURCE person (
    "id" BIGINT,
    "name" VARCHAR,
    "email_address" VARCHAR,
    "credit_card" VARCHAR,
    "city" VARCHAR,
    "state" VARCHAR,
    "date_time" TIMESTAMP,
    "extra" VARCHAR
) with (
    connector = 'nexmark',
    nexmark.table.type = 'Person',
    nexmark.split.num = '8',
    nexmark.min.event.gap.in.ns = '0'
) ROW FORMAT JSON;

CREATE SOURCE auction (
    "id" BIGINT,
    "item_name" VARCHAR,
    "description" VARCHAR,
    "initial_bid" BIGINT,
    "reserve" BIGINT,
    "date_time" TIMESTAMP,
    "expires" TIMESTAMP,
    "seller" BIGINT,
    "category" BIGINT,
    "extra" VARCHAR
) with (
    connector = 'nexmark',
    nexmark.table.type = 'Auction',
    nexmark.split.num = '8',
    nexmark.min.event.gap.in.ns = '0'
) ROW FORMAT JSON;

CREATE SOURCE bid (
    "auction" BIGINT,
    "bidder" BIGINT,
    "price" BIGINT,
    "channel" VARCHAR,
    "url" VARCHAR,
    "date_time" TIMESTAMP,
    "extra" VARCHAR
) with (
    connector = 'nexmark',
    nexmark.table.type = 'Bid',
    nexmark.split.num = '8',
    nexmark.min.event.gap.in.ns = '0'
) ROW FORMAT JSON;

Create MView:

CREATE MATERIALIZED VIEW nexmark_q7
AS
SELECT
    B.auction,
    B.price,
    B.bidder,
    B.date_time
from
    bid B
    JOIN (
        SELECT
            MAX(price) AS maxprice,
            window_end as date_time
        FROM
            TUMBLE(bid, date_time, INTERVAL '10' SECOND)
        GROUP BY
            window_end
    ) B1 ON B.price = B1.maxprice
WHERE
    B.date_time BETWEEN B1.date_time - INTERVAL '10' SECOND
    AND B1.date_time;

Wait for 1 hour

Expected behavior
No response
Additional context
Full logs here: risingwave-log.tgz

@Little-Wallace Please take a look a feel free to hand over to others

We suspect the prefix is wrong originally but @StrikeW can reproduce this bug easily in his environment with and without the multi-prefix support.
I turned on S3 access log in his reproduction runs and saw the following log entries:
2b30ef01931902ad5eb4617f63d13a8ca2a976cf5f08049a941d01162522175e siyuanw-test-bucket [09/Sep/2022:05:50:41 +0000] <IP_ADDRESS> arn:aws:iam::639303875316:user/siyuan_wang T4PASD0TDD9B0NT0 REST.POST.UPLOADS hummock_001/3882.data "POST /siyuanw-test-bucket/hummock_001/3882.data?uploads&x-id=CreateMultipartUpload HTTP/1.1" 200 - 391 - 34 34 "-" "aws-sdk-rust/0.46.0 os/linux lang/rust/1.64.0-nightly" - xE1lcxxUYTTOfuFlCwjIbPw+iUUPHwgKHM+aRMleRQ5mSxJwP8pZtL/6MnacVYywojAzzJoTRgM= SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader s3.ap-southeast-1.amazonaws.com TLSv1.2 -
2b30ef01931902ad5eb4617f63d13a8ca2a976cf5f08049a941d01162522175e siyuanw-test-bucket [09/Sep/2022:05:50:41 +0000] <IP_ADDRESS> arn:aws:iam::639303875316:user/siyuan_wang T4PDZME7MAYB1FKX REST.PUT.OBJECT hummock_001/3882.data "PUT /siyuanw-test-bucket/hummock_001/3882.data?x-id=PutObject HTTP/1.1" 200 - - 128 47 36 "-" "aws-sdk-rust/0.46.0 os/linux lang/rust/1.64.0-nightly" - TLSd30dLE2QI7auidTgR3VjPR8un353CKHIOu87Q3rU0C4Pc2c3R8UKdf3dT/oLsUvKpF6WplMg= SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader s3.ap-southeast-1.amazonaws.com TLSv1.2 -
2b30ef01931902ad5eb4617f63d13a8ca2a976cf5f08049a941d01162522175e siyuanw-test-bucket [09/Sep/2022:05:50:41 +0000] <IP_ADDRESS> arn:aws:iam::639303875316:user/siyuan_wang T4PFFADZXY1Y02P4 REST.DELETE.UPLOAD hummock_001/3882.data "DELETE /siyuanw-test-bucket/hummock_001/3882.data?x-id=AbortMultipartUpload&uploadId=6HLmyZm1OHEiKOmM8eRSjTkj70i0lWP_zSDkegyH75tH4smAZcYoeXthYMGK4BaJwT_ZbDGqzFpkGFb0SUBgPPRwhRYi1ZAvJkxKU8lgvFLu.vJzkm0kP7G8wnU6hOPRWnqo58ufXUgcjqRA6WXp1A-- HTTP/1.1" 204 - - - 16 15 "-" "aws-sdk-rust/0.46.0 os/linux lang/rust/1.64.0-nightly" - ngQbvzc8ylzGqCkm1JDs3aJnhjA6WryL+1ArhKZqgSISlYD8IEMupADivIPBhagbGE1b9SgiTSQ= SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader s3.ap-southeast-1.amazonaws.com TLSv1.2 -

The sequences are:
t1: CreateMultipartUpload #3882
t2: PutObject #3882
t3: AbortMultipartUpload #3882
I guess AbortMultipartUpload in t3 will delete the object created in t2 :(

I added some logs and successfully reproduced this issue in my environment. Here are my findings:

UploadMultipart for a ~64MB SST failed with "error writing a body to connection: Broken pipe (os error 32)":

2022-09-09T14:02:33.919524Z  INFO risingwave_object_store::object::s3: multipart error: io error: error writing a body to connection: Broken pipe (os error 32)
  backtrace of `ObjectError`:
   0: <risingwave_object_store::object::error::ObjectError as core::convert::From<risingwave_object_store::object::error::ObjectErrorInner>>::from
             at ./src/object_store/src/object/error.rs:38:10
   1: <T as core::convert::Into<U>>::into
             at /rustc/9067d5277d10f0f32a49ec9c125a33828e26a32b/library/core/src/convert/mod.rs:550:9
   2: <risingwave_object_store::object::error::ObjectError as core::convert::From<aws_smithy_http::result::SdkError<E>>>::from
             at ./src/object_store/src/object/error.rs:81:9
   3: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual
             at /rustc/9067d5277d10f0f32a49ec9c125a33828e26a32b/library/core/src/result.rs:2134:27
   4: risingwave_object_store::object::s3::S3StreamingUploader::upload_next_part::{{closure}}
             at ./src/object_store/src/object/s3.rs:128:33
   5: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/9067d5277d10f0f32a49ec9c125a33828e26a32b/library/core/src/future/mod.rs:91:19
   6: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tracing-0.1.36/src/instrument.rs:272:9
   7: tokio::runtime::task::core::CoreStage<T>::poll::{{closure}}
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.20.1/src/runtime/task/core.rs:165:17

However, the compaction task of this SST still succeeded without error.

2022-09-09T14:02:52.748846Z  INFO risingwave_storage::hummock::compactor: Finished compaction task in 24396.777332999998ms:
Compaction task id: 14903, target level: 6
Compaction watermark: 0
Compaction target_file_size: 67108864
Compaction task status: "SUCCESS"
Compaction # splits: 1
Compaction task status: 1
Compaction Sstables structure:
Level 5 ["[id: 20333, 32768KB]"]
Level 6 ["[id: 18597, 64864KB]", "[id: 18674, 64932KB]", "[id: 18744, 37293KB]"]
Compaction task output:
SstableInfo: id=20517, KeyRange="[74000007d5da0080000000009bfc010082a5ec9bc9040354fff5684d91bdffff, 74000007d5df0080000000019ecb980082a5ebf7be840321fff56850218fffff]", size=66554342
SstableInfo: id=20581, KeyRange="[74000007d5df0080000000019ecb980082a5ed65a7000125fff5684a6a58ffff, 74000007d5e4008000000000069ea20082a5f2b72380025bfff5683523b2ffff]", size=66758300
SstableInfo: id=20642, KeyRange="[74000007d5e4008000000000069ea20082a5f3900bc4008bfff56831c092ffff, 74000007d5e8008000000000002f5e0082a5ec6068840197fff5684e7f58ffff]", size=51472712

A later read on this SST failed since the object is not successfully written to S3:

2022-09-09T14:03:59.890248Z ERROR risingwave_storage::monitor::monitored_store: Failed in iter: Hummock error: ObjectStore failed with IO error Internal error: read "hummock_001/122/20517.meta" in block None failed, error: NoSuchKey: The specified key does not exist.
  backtrace of `ObjectError`:
   0: <risingwave_object_store::object::error::ObjectError as core::convert::From<risingwave_object_store::object::error::ObjectErrorInner>>::from
             at ./src/object_store/src/object/error.rs:38:10
   1: <T as core::convert::Into<U>>::into
             at /rustc/9067d5277d10f0f32a49ec9c125a33828e26a32b/library/core/src/convert/mod.rs:550:9
   2: <risingwave_object_store::object::error::ObjectError as core::convert::From<aws_smithy_http::result::SdkError<E>>>::from
             at ./src/object_store/src/object/error.rs:81:9
   3: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual
             at /rustc/9067d5277d10f0f32a49ec9c125a33828e26a32b/library/core/src/result.rs:2134:27
   4: <risingwave_object_store::object::s3::S3ObjectStore as risingwave_object_store::object::ObjectStore>::read::{{closure}}
             at ./src/object_store/src/object/s3.rs:354:20

Additional notes:

Also saw "TransientError: connection closed before message completed" in the log so I guess there is a network blip.
Did not see big increase in compaction and object store throughput when the issue happened.
Still not sure why compaction task didn't catch and report an error

Full logs:
repro-logs.tar.gz

Devil in the details🥵

The result of writer output is not checked.
upload_join_handles.push(upload_join_handle.and_then(|_| async move {
    if let Some(tracker) = tracker_cloned {
        tracker.inc_ssts_uploaded();
    }
    Ok(())
}));

Fixed by https://github.com/risingwavelabs/risingwave/pull/5261


Teams link not working in Bot config App

Hi,
I encountered that the teams link ID that is provided does not work properly.
In the controler HomeControler.cs the link is parsed with an ending string of %40thread.skype or @thread.skype
https://github.com/OfficeDev/microsoft-teams-faqplusplus-app/blob/08a69787255e6fc53984b83a8ddc3e28f87f0e38/Source/Microsoft.Teams.Apps.FAQPlusPlus.Configuration/Controllers/HomeController.cs#L23
but in fact the url of the team link you get has chaned to @thread.tacv2. The code shoulkd be changed.
I could mitigate from it by just using the string that is trying to pars by using @thread.skype instead of @thread.tacv2.

PR #93 is fixing it


Evaluation code error

Hello. Thank you for your work!
After training using your code, I keep getting "multimodality" as 0 when evaluating. Can you tell me why?
Thank you!

Hi, this is not error, please refer to 4.2. Part-Coordinated Transformer:

If you want to evaluate the MultiModality (which takes a long time), just delete --skip-mmod.

Thank you so much!


make exe name shorter

too much typing with simplewebserver..
sws.exe ?
webs.exe ?
or maybe just leave it to user to rename..

i think leave it to user for now..


Reauth workflows

Feature or Bugfix

Feature

Detail

Add Configurable feature to have sensitive API Operations require re-auth before proceeding

For specific sensitive actions, such as destructive actions like "deletes" of AWS Resources, there should be additional steps for a user before they can proceed with performing said action

Enforce a user to re-authenticate themselves before they are able to successfully call the API request performing the sensitive action

Technical Implementation:
Backend Process

There is a SSM Parameter with a list of API Names defining which API actions require re-auth workflow

API Names are configurable (in cdk.json)

There is a configured time to live (TTL) for any user's latest authentication time until they would have to re-authentication again for any sensitive action

TTL is configurable (in cdk.json)

API Handler Lambda is updated to

Determine whether API Request is for a sensitive action
If so

Pull latest authentication time from auth_time header from API Request
Ensure current time is within the  TTL window from the latest authentication time (auth_time) and if not return a

Frontend Error Handling

If receiving custom 401 error --> Prompts user to either

Continue their current session without performing the sensitive action
OR Re-Authentication by

Logging Out and logging back in to the data.all UI
Logging back in to the UI and then navigating back to perform the sensitive action once again

force log out and log back in flow will give user new tokens and update latest auth_time in future API Requests

User Experience

User A logs in to data.all and immediately requests a sensitive action (e.g. DELETE_DATASET) on the data.all UI for DatasetA

Cognito Auth allows API Request and API Handler Lambda determines that this request requires re-auth but the user's latest auth_time is within the configured TTL time window (configured at 5 minutes) so the delete action is allowed

User A waits 10 minutes and requests another sensitive action (e.g. DELETE_DATASET) on the data.all UI for DatasetB

Cognito Auth allows API Request but API Handler Lambda determines that this request requires re-auth and the user's latest auth_time is NOT within the configured TTL time window anymore so a 401 is returned

UserA decides to re-authenticated is a logged out of data.all
UserA provides proper credentials to its Identity Provider and receives a new token with a new auth_time
Now UserA navigates back to the Datasets View and is able to perform DELETE_DATASET on the data.all UI for DatasetB

Relates

Issue #571

Security
Please answer the questions below briefly where applicable, or write N/A. Based on
OWASP 10.

Does this PR introduce or modify any input fields or queries - this includes
fetching data from storage outside the application (e.g. a database, an S3 bucket)?

Is the input sanitized?
What precautions are you taking before deserializing the data you consume?
Is injection prevented by parametrizing queries?
Have you ensured no eval or similar functions are used?

Does this PR introduce any functionality or component that requires authorization?

How have you ensured it respects the existing AuthN/AuthZ mechanisms?
Are you logging failed auth attempts?

Are you using or adding any cryptographic features?

Do you use a standard proven implementations?
Are the used keys controlled by the customer? Where are they stored?

Are you introducing any new policies/roles/users?

Have you used the least-privilege principle? How?

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Additional Considerations/Requirements

(Requirement) Ensure that the log out workflow forces full log out for both Cognito IdP and for federated data.all users using integrated 3rd party IdPs

Have documentation on how to implement - to add to data.all documentation pages

User Experience of force logout and log back in and manually navigating back to the data.all page to perform the sensitive operation is sub-optimal

Currently when a user opts to re-authentication, after another proper signin, they are navigated back to data.all’s homepage and not back to the previous page they were just at to perform the sensitive operation
We could look into

retrying to API operation immediately after login
redirect user back to the page they previously were on rather than manually navigating back in the data.all UI

In the latest commits I have added the ability to retry the API Request that required ReAuth upon a User re-logging in to the data.all UI. For retry logic, the reauth operation will be retried if:

There exists a re-auth sensitive action to retry
The user has re-logged in within the TTL (configured in the cdk.json)
The user logs in as the same user as the before when trying to perform the sensitive action

For queries, the user will also be redirected to the page they were formerly on. For mutations this redirection is not built in as it may not make sense for most mutations (i.e. take the example of a user deleting a dataset… upon re-auth and the retry of the DeleteDataset it does not make sense to navigate the user back to the particular dataset’s view as that view no longer exists)
Testing in AWS Deployment

[x] ReAuth on Sensitive Query --> Retry API Request + Navigate back to old page + Success Message
[x] ReAuth on Sensitive Query + wait > TTL before re-login --> No Retry API Request + Error Message
[x] ReAuth on Sensitive Mutation --> Retry API Request + Success Message
[x] ReAuth on Sensitive Mutation w/ Bad Inputs --> Retry API Request + Error Message
[x] Login as New User between ReAuth log in --> No Retry API Request
[x] Normal Logout and Login functional

@dbalintx and I tested in existing deployment with internet_facing=true and custom_domain.
Reauth workflow and logout/login successful.
The only remark is that it is relatively easy to make mistakes when defining the reauth_apis. To use this feature we need to make it very clear in the docs that the query is the one defined in the gql schema and how to obtain it.
But overall, incredible work @noah-paige


[idea] Folders in the notebook directory

It would be great to be able to create folders in the notebook directory (1 or 2 level(s) would be enough I think). It would allow to easily organize notebooks.
Would it be difficult to implement?

Hi, any progress on this :)?


Allow specification of a maximum return payload size from the database server

This PR adds a new configurable option for a maximum allowed return payload size. Either due to bad data or an incorrectly configured query the UniRPC server may return an extremely large payload which can saturate a network or cause MVIS to become unresponsive. If the payload size exceeds the maximum, a DbServerError will be thrown.
I chose a high default value of 1GB but made it a configurable connection and individual request option.

I think it makes sense to build the docs as part of the release instead of the merge to main.
In this case though I'll create a separate PR for the documentation changes since one of us will need to update the GitHub Action.

Also, how do you feel about the 1GB default? I set it high because I have no idea what other people do with MVIS but also seems like if it's that high the default value won't catch any issues. Maybe 100MB?

Also, how do you feel about the 1GB default? I set it high because I have no idea what other people do with MVIS but also seems like if it's that high the default value won't catch any issues. Maybe 100MB?

1GB does seem like a lot considering its solely a parameter for use with MVOM and not for MVIS in general.  I had sort of interpreted that as a "this is basically infinite" default value.  However, if we're going to go with an enormous default value, we might be better off just making it optional.  It's a likely footgun for people if they don't pass it in though, so a sane default is probably good.  I'd be comfortable with 100MB.

I updated the default size to 100MB and reverted all the documentation changes. I will do a separate documenation PR>


Package toml is not being recognized as a requirement for pylama

Error
The library is not reading the project's pyproject.toml file. Even running the command pylama -o pyproject.toml, the below error occurs: (thanks @alorence for the stacktrace).
File "main.py", line 100, in shell
    options = parse_options(args)
  File "main.py", line 203, in parse_options
    cfg = get_config(options.options, rootdir=rootdir)
  File "main.py", line 272, in get_config
    return get_config_toml(cfg_path)
  File "main.py", line 290, in get_config_toml
    config = config_toml.Namespace()
NameError: name 'config_toml' is not defined. Did you mean: 'get_config_toml'?

Cause
It happens because the module config_toml cannot be imported at config module, due to an ImportError raised when the module toml tried to be imported. For some reason, when installing the library pip install pylama, the toml library is not being recognized as a required package for the project.
Fix (work-arround)
We can install toml library manually to fix the problem using the command pip install toml.

Looking at the documentation, I saw that it is proposital to consider toml library as a optional library. To install the library, the documentation suggests to perform pip install pylama[toml]. The issue can be closed.


Iss632 after rebase on future

I might've accidentally messed something up while trying to rebase my old branch to the future branch instead of working off the main branch (😅) ! I apologize for this inconvenience, but here this backup branch should have all the previous debugging improvement changes before Konstantinos's comments and also the changes made from Konstantinos's suggestions.

Once you fix the commit signing and the DCO passes, we can squash and merge :)

I fixed the commit signing! Ready for the final steps for merging! Thank you so much!!

Thank you!


Mask preprocessor module cleanup and add tests

when work is done, it should close #226 (until refactorization with dask arrays)

guessing you are since they have nice unit tests - if you do, could you maybe do up a little their docstrings?

ok guys, cleaned up, added documentation and somewhat improved the test coverage (by 10% meh) but the other stuff will be hard to test (can probably add more unit tests but I think it should be OK for now) :beer:

cheers ever so much @bouweandela - I will have a stab at implementing your suggestions tomorrow :beer:

OK @bouweandela now we have numpy style documentation for public functions :beers:

ok monsieur le Bouwe @bouweandela I spick and spanned the documentaion and now there are no warnings (there may be typos but I reckon it's best to fix those once we know we are ready to produce the almost final version of the docs). Think this one's ready to fly :airplane:

arite @bouweandela fixed your last batch of suggestions - note that isort, despite running it on the two modules, failed to sort the imports correctly (needs an update maybe?) so the dask one I left it as is since I dunno what's the correct order, I fixed the other one manually. Also, only the public functions have numpy-stile documentation, didn't bother fixing it proper for the private ones, you want me to?

Also, only the public functions have numpy-stile documentation, didn't bother fixing it proper for the private ones, you want me to?

Hi V, either add it properly or don't add it. Adding something half done just makes a mess.

monsieur le @bouweandela le work is now le done :beer:

Replace iris.Cube.cube by iris.cube.Cube everywhere

Please fix the merge conflicts

I have just tested it with:
---
documentation:
  description: |
    Test recipe for OBS, no preprocessor or diagnostics are applied,
    just to check correct reading of the CMORized data.

  authors:
    - righ_ma

preprocessors:
  nopp:
    mask_window_threshold:
      threshold_fraction: 0.9
      min_value: 0
      time_window: 2

diagnostics:
  MPI-ESM-LR:
    description: MPI-ESM-LR
    variables:
      ta:
        preproc: nopp
        mip: Amon
    additional_datasets:
      - {dataset: MPI-ESM-LR,  project: CMIP5, exp: historical,  ensemble: r1i1p1,  start_year: 1990,  end_year: 2000}
    scripts: null


Update version to 2.2.2

Bumps AJS dependency to ^3.0.0 to add improved behavior to assumesPageView

Codecov Report

Merging #7 into master will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff          @@
##           master     #7   +/-   ##
=====================================
  Coverage     100%   100%
=====================================
  Files           1      1
  Lines          20     20
=====================================
  Hits           20     20

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e37c7a0...fdae6e8. Read the comment docs.


Add "-c CONTAINER_NAME" to "acorn logs"

Signed-off-by: Darren Shepherd<EMAIL_ADDRESS>Checklist

[ ] The title of this PR would make a good line in Acorn's Release Note's Changelog
[ ] The title of this PR ends with a link to the main issue being address in parentheses, like: This is a title (#1216). Here's an example
[ ] All relevant issues are referenced in the PR description. NOTE: don't use GitHub keywords that auto-close issues
[ ] Commits follow contributing guidance
[ ] Automated tests added to cover the changes. If tests couldn't be added, an explanation is provided in the Verification and Testing section
[ ] Changes to user-facing functionality, API, CLI, and upgrade impacts are clearly called out in PR description
[ ] PR has at least two approvals before merging (or a reasonable exception, like it's just a docs change)

Tested with acorn version v0.7.0-38-gdc95c02b+dc95c02b
Able to get logs for containers by names using  - acorn logs -c <container_nam> <app name>
acorn containers mytesti
NAME                             APP       IMAGE                                                                     STATE     RESTARTCOUNT   CREATED    MESSAGE
mytesti.app-847596cc77-m6bhz     mytesti   sha256:f25073198f5172e51f6cd0264d8a5998990d0c33ecad538e7040fa189ae2be98   running   0              105s ago
mytesti.cache-6fcf4dd69d-jcg7k   mytesti   redis:alpine                                                              running   0              107s ago
mytesti.db-57f8fcbd6b-p8xqd      mytesti   postgres:alpine                                                           running   0              107s ago
sangeethahariharan@Sangeethas-MacBook-Pro acorn-test-app % acorn logs -c app mytesti
app-847596cc77-m6bhz:  * Serving Flask app 'app.py'
app-847596cc77-m6bhz:  * Debug mode: off
app-847596cc77-m6bhz: WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
app-847596cc77-m6bhz:  * Running on all addresses (<IP_ADDRESS>)
app-847596cc77-m6bhz:  * Running on http://<IP_ADDRESS>:5000
app-847596cc77-m6bhz:  * Running on http://<IP_ADDRESS>:5000
app-847596cc77-m6bhz: Press CTRL+C to quit


Add support for assembly resolution from config file

The code is mostly copied from AssemblyFoldersEx and follows most of the
same logic. I didn't implement all the features and options from that
class as they don't seem to be used/needed.
In Dev15, Visual Studio setup will drop this file into MSBuild folder in
order for Add References dialog and Build to find reference assemblies.
This will allow for projects to reference, say Microsoft.Build.dll, without
providing a hint path. Previously this was done via the registry but Dev15
will no longer write those entries in all cases. When they are not in the
registry, MSBuild will use this file.

LGTM with minor edits.

:shipit:


clang++/c++11/libc++: GTest pack failed

I tried to compile GTest but I cannot build it with the latest hunter (currently 0.12.43). I got the following error:
[100%] Completed 'GTest-Debug'
[100%] Built target GTest-Debug
-- [hunter] Build step successful (dir: XXXXX/hunter/_Base/fdaeb5a/b8ad68d/d411be5/Build/GTest)
CMake Error: archive_read_disk_entry_from_file: Can't lstat --
CMake Error: Problem creating tar: XXXXX/hunter/_Base/Cache/raw/cache.tar.bz2

[hunter ** INTERNAL **] Packing failed
[hunter ** INTERNAL **] [Directory:XXXXX/hunter/_Base/Download/Hunter/0.12.43/fdaeb5a/Unpacked/cmake/projects/GTest]

------------------------------ WIKI -------------------------------
    https://github.com/ruslo/hunter/wiki/error.internal
-------------------------------------------------------------------

CMake Error at XXXXX/hunter/_Base/Download/Hunter/0.12.43/fdaeb5a/Unpacked/cmake/modules/hunter_wiki.cmake:12 (message):
Call Stack (most recent call first):
  XXXXX/hunter/_Base/Download/Hunter/0.12.43/fdaeb5a/Unpacked/cmake/modules/hunter_internal_error.cmake:13 (hunter_wiki)
  XXXXX/hunter/_Base/Download/Hunter/0.12.43/fdaeb5a/Unpacked/cmake/modules/hunter_pack_directory.cmake:59 (hunter_internal_error)
  XXXXX/hunter/_Base/Download/Hunter/0.12.43/fdaeb5a/Unpacked/cmake/modules/hunter_save_to_cache.cmake:78 (hunter_pack_directory)
  XXXXX/hunter/_Base/Download/Hunter/0.12.43/fdaeb5a/Unpacked/cmake/modules/hunter_download.cmake:422 (hunter_save_to_cache)
  XXXXX/hunter/_Base/Download/Hunter/0.12.43/fdaeb5a/Unpacked/cmake/projects/GTest/hunter.cmake:160 (hunter_download)
  XXXXX/hunter/_Base/Download/Hunter/0.12.43/fdaeb5a/Unpacked/cmake/modules/hunter_add_package.cmake:87 (include)
  CMakeLists.txt:16 (hunter_add_package)

-- Configuring incomplete, errors occurred!

I used polly 0.8.1 for builds. I got the same result with gcc as well. I expected that this worked with earlier releases, as I found v0.9.9 was the last working version.

---
Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/30648403-clang-c-11-libc-gtest-pack-failed?utm_campaign=plugin&utm_content=tracker%2F1032790&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F1032790&utm_medium=issues&utm_source=github).

CMake Error: archive_read_disk_entry_from_file: Can't lstat --
CMake Error: Problem creating tar: XXXXX/hunter/_Base/Cache/raw/cache.tar.bz2

This is not Hunter error message, this message comes from CMake itself. Since you've mentioned that it was working before I think it make sense to check:

Does CMake version changed (may be it's CMake bug?)
Do you have enough disk space? Probably CMake can't allocate space for creating file?

I used the same CMake in both cases (stock ubuntu 15.04/CMake 3.0.2)
I have 60 GB free disk space as well. The cache.tar.bz2 is created but it is empty.
I could reproduce it with hunter-simple project. I got the same error.
If I changed Hunter version to v0.9.9 then it works well. (But it doesn't work with v0.10.0, and both versions use GTest-1.7.0-hunter-10)

If I changed Hunter version to v0.9.9 then it works well

This version without cache feature. I.e. it doesn't run cmake -E tar to save archive with installed files for GTest.

CMake 3.0.2

I'm pretty sure it's CMake bug. See recommended CMake version: https://github.com/ruslo/hunter#notes-about-version-of-cmake Which lead to this comment: https://github.com/ruslo/hunter/pull/198#issuecomment-120630877

I upgraded CMake to 3.2.2 and now it works properly.
Thanks for the fast help!

@tatraian Glad it helps :)


Demo1 catalog updates

Updates in pipeline run-test-suite
Update in task getkubeconfig and runsuite
Added parameter description in those tasks
Catalog updates for demo2 to be added in another PR

Look like this branch isn't updated, please rebase with main branch

Still need to fetch changes from main

Look like this branch isn't updated, please rebase with main branch

Still need to fetch changes from main

Done.


Add possibility to use a private key plus some other features

I made various fix and added a small feature to accept and array of commands in pre/post deploy hooks. Please check the commits and merge if you're ok with it.

Ups ... already merged ..


change to using byte buffer references for more efficient buffer use

I'm not sure about the proper reference closing everywhere just yet.  This is a first draft

tests pass locally

tracking down some dangling references.  do not merge yet

found dangling references fixed.

I took a quick look through this and it looks fine, but I assume most of it is iterative on top of the code I've already reviewed. Let me know if there are things you want me to take a closer look at.

i think we should just go with it. i'll merge and release.


chore(samples): remove region tags.

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

[ ] Make sure to open an issue as a bug/issue before writing your code!  That way we can discuss the change, evaluate designs, and agree on the general idea
[ ] Ensure the tests and linter pass
[ ] Code coverage does not decrease (if any source code was changed)
[ ] Appropriate docs were updated (if necessary)

Fixes #<issue_number_goes_here> 🦕

@chenlei1216 Please could you confirm that you'd like to remove region tags?

@chenlei1216 Please could you confirm that you'd like to remove region tags?

Thanks for checking. I don't think they're used by our docs so I think it should be fine to remove.

@chenlei1216 Please could you confirm that you'd like to remove region tags?

Thanks for checking. I don't think they're used by our docs so I think it should be fine to remove.

But I don't know what's the purpose of this PR. I assume the region tags are automatically generated. So they could be added back automatically later?

@chenlei1216 Please could you confirm that you'd like to remove region tags?

Thanks for checking. I don't think they're used by our docs so I think it should be fine to remove.

But I don't know what's the purpose of this PR. I assume the region tags are automatically generated. So they could be added back automatically later?

Discussed offline, that we're not gonna use the interactive tutorial samples in cloud doc, so SGTM to remove the tags.


kubectl apply -f config/samples/cache_v1alpha1_memcached.yaml not work

reference "https://sdk.operatorframework.io/docs/building-operators/golang/tutorial/"
env:
1)operator-sdk version
operator-sdk version: "v1.26.1", commit: "4582a8414e65f40ebea65c74729f121e1f3e3b9a", kubernetes version: "1.25.0", go version: "go1.19.5", GOOS: "linux", GOARCH: "amd64"
2)go version go1.20.3 linux/amd64
3)Kustomize Version: v4.5.7
4)kind v0.17.0 go1.19.2 linux/amd64
local k8s env,
when execute as testdata's config files , memcached-operator-controller seems work,
memcached-operator-system   memcached-operator-controller-manager-6587fdd7f4-zgp2g   2/2     Running   0          16m

but "kubectl apply -f config/samples/cache_v1alpha1_memcached.yaml " not create any deployment service about memcached, nothing changed and no logs pop up from pod "memcached-operator-controller-manager-6587fdd7f4-zgp2g "

@t00350320  Could you help provide your operator's controllers/memcached_controller.go file and the logs of manager container please?
$ kubectl logs memcached-operator-controller-manager-6587fdd7f4-zgp2g -c manager


fix the problem of extension Broken on WSL2

Use Webview.asWebviewUri to ensure that resources are accessible in both the host environment and the wsl environment.
related issue: https://github.com/pomber/git-history/issues/196

it's work! I check this on wsl2


Name spacing for Modifying tags before they're sent

I'm struggling with the name spacing example in the readme file:
https://github.com/ralphjsmit/laravel-seo#modifying-tags-before-they-are-rendered
SEOManager::tagTransformer(function (TagCollection $tags): TagCollection {
    $tags = $tags->reject(fn(Tag $tag) => $tag instanceof OpenGraphTag);

    $tags->push(new MetaTag(name: 'custom-tag', content: 'My custom content'));
    // Will render: <meta name="custom-tag" content="My custom content">

    return $tags;
});

The error is:
TypeError
App\Http\Controllers\PostController::App\Http\Controllers\{closure}(): Argument #1 ($tag) must be of type App\Models\Tag, RalphJSmit\Laravel\SEO\Tags\RobotsTag given

The only one that sort of seems okay so far is this one:
use RalphJSmit\Laravel\SEO\Facades\SEOManager;

Any clues what other name spaces to use? I tried automatic importing in Visual Studio code but ended up with the above error.

Hey Eugene, with the namespaces, you should only import names that start with RalphJSmit\Laravel\SEO.
So I think you should have:
use RalphJSmit\Laravel\SEO\Facades\SEOManager;
use RalphJSmit\Laravel\SEO\TagCollection;
use RalphJSmit\Laravel\SEO\Support\Tag;
use RalphJSmit\Laravel\SEO\Support\OpenGraphTag;
use RalphJSmit\Laravel\SEO\Support\MetaTag;

The issue with the error is that you imported App\Models\Tag. That model is a class in your project directory. The package doesn't know that this model exists, because everyone is free to create it's own models. That's why every import should start with RalphJSmit\Laravel\SEO, because that are only the files that exist.
Let me know if that works!


Content available at tipping costs

I will send content wherever you want I also make solos or fetish videos for extra but they are only yours they are not leaked.  I have a lot and nothing I won’t help u with but yes it’s a price we can discuss between us and discreet $Ladyking0612 must have cash app!!I may take pay pal depending. @pornhub I will give you my model link to my website also 😘💜


Broken demo

Hi, this demo is broken:
https://codepen.io/dimsemenov/pen/jnohA
But I suspect that the patterns param has some issue itself.

Fixed, thanks. Just url to the video changed.


Login URL mangled on beta build

Home Assistant Android version:
1.3.1(71)
Android version:
10
Phone model:
Pixel XL4
Home Assistant version:
103.5
Last working Home Assistant release (if known):
Description of problem:
After new install of beta version to support testing, after logging in with OTP error occurs and URL is replaced with garbage.

Traceback (if applicable):

Additional information:

If I should report beta bugs elsewhere pls let me know.

I just validated with the latest, that url is your cloud access url.  This is to be expected if you have a cloud account setup.  If you have the Remote UI disabled, a fix in HA 0.105 (https://github.com/home-assistant/home-assistant/pull/30453) should keep it from being returned.


[Feature Request] Add ability to fill vertical white space with tiles

As different tile types often need to have a larger footprint to be useful outside of fullscreen view (e.g. Graph tiles), but others don't need such a large footprint (e.g. Parameter Select tiles), there are often cases like the one in the attached image in which I end up with a good amount of vertical white space. Since we can only move tiles left and right within a dashboard right now, there's no way to take a short tile and place it beneath another short tile, if they're in the same row as a tall tile.
The most straightforward solution would be drag-and-drop tile placement, but I know that's difficult for this tech stack. So perhaps an up/down arrow to complement the left/right arrow functionality?

Hi @emigre459 ,
Thanks for the suggestion. This is unfortunately an artifact of how react-material deals with grid layouts so there's not too much I can do about it at this point.
On the other hand - I've been doing some investigating into a resizable grid layout and it is looking awesome:
https://react-grid-layout.github.io/react-grid-layout/examples/0-showcase.html
issue there is that I'll have to change a bunch of things, including the format in which dashboards are stored. I'll probably put in in the 2.1 release for this reason. What do you think?

I just played around with the react-grid-layout you linked to and it seems to solve all the problems! I'm sure a lot would be needed to make sure errors don't pop up/text floats off into space/etc. in the tile contents as a result, but this would definitely solve the problem.
From an engineering perspective, totally understand the need to wait to put it into a minor version instead of a patch version. Do you have an estimate as to when you expect the next minor version to be put out?

Good news @emigre459 ! It's coming soon ⏲

Closed on<PHONE_NUMBER>2ada6cf75d433113632ad1a42cd573
Version 2.1.1
Enjoy!


Encoding bug with i18n of help

The resource bundles for i18n of help are processed with an invalid encoding:

az             Werkzeug Kommando f�r die Azure Kommandoschnittstelle.

Should be für instead of f�r.
IMHO the resource bundles are encoded with UTF-8 what is correct but for some reason our GraalVM build native image treats them as something like ISO-8859-1.

Retested and confirmed as fixed. Thanks @diiinesh


[Feature] Restricted nameservers without search domains

Use case
I use the restricted_nameservers option for my own implementation of App Connectors (the DNS server adds routes to the subnet router) so I can route services like 1Password over the tailnet. The issue with the current implementation is that 1password.com will also be added as a search domain, which I don't want.
Description
It would be nice if it was possible to add a restricted nameserver without the domain being added as a searchdomain.
Contribution

[ ] I can write the design doc for this feature
[ ] I can contribute this feature

How can it be implemented?
I've been digging in the Tailscale client sourcecode, and I think this can be implemented by adding the domain to DNSConfig.MatchDomains instead of DNSConfig.SearchDomains, but I'm not really sure about this.

I will dig into this a little bit further tomorrow and see if I can implement this myself.

I am facing the same issue, it's really frustrating especially because I get some dns resolution errors because of this.

My current fix is a slight modification of the v0.22.3 code, where I remove line 448 from the file hscontrol/config.go . This keeps headscale from appending the restricted servers to the search domains.

I think DNSConfig.MatchDomains is filled via https://github.com/tailscale/tailscale/blob/218110963dbb12002529a4a91ffc149b70ac21ac/net/dns/manager.go#L325 and https://github.com/tailscale/tailscale/blob/7aec8d4e6b4e72d5053d9ff7d819c28f9e035c2c/net/dns/config.go#L146
Especially because of https://github.com/tailscale/tailscale/blob/7aec8d4e6b4e72d5053d9ff7d819c28f9e035c2c/net/dns/config.go#L148 I think we only need to fill Routes and not Domains. https://github.com/tailscale/tailscale/blob/7aec8d4e6b4e72d5053d9ff7d819c28f9e035c2c/tailcfg/tailcfg.go#L1568

I think we have implemented this wrongly the whole time, looking into it now, it does not seem that tailscale saas, nor the client adds split dns to search domains, and I dont think there is a good reason to do it. I will address this as part of #2034.


Revert "Set priority class for chart-operator pod (#195)"

Removes the priority class. chart-operator runs in control planes but is unused. The priority classes are missing so this broke and triggered alerts.
The issue I was trying to fix is a problem reported by @pipo02mix with tenant clusters. Due to cluster-autoscaler the chart-operator pod can be evicted as it misses the priority class.
But we don't need a fix until cluster-autoscaler ships. I'll reintroduce once we have a fix that works for tenants and control planes.

FYI @giantswarm/team-spirit we need a fix for this before cluster-autoscaler ships.
Can we just create the priority classes for control planes? Seems like it would be useful there too.


Added Ubuntu 2020 (Focal) to Makefile.release

Had a little internal dialogue about whether this crosses the line to "feature" WRT https://github.com/segmentio/aws-okta/issues/278, and decided to just merge and release it.
I think I'm going to merge these sorts of things as long as they're trivial. Eventually I'm sure there will be something that's non-trivial, but we'll cross that bridge when we come to it.


Have a better logging system

I'm thinking of something like rust-analyzer where it outputs logs with times in an outputChannel. Would it be overkill

I would just make sure we console.log/console.warn wherever we think it could be useful to debug the extension in the future.

I guess I'll close this then. I had some implementation a few days ago, but not clean.


Nerd Font renders only half of an icon in some cases

Font used "Hack NF" (Hack font complete windows compatible)
https://github.com/ryanoasis/nerd-fonts/tree/master/patched-fonts/Hack/Regular/complete
Here's the comparison of the same font as it's displayed in the new Windows Terminal and Extraterm, as you can see Extraterm  cuts some icons in half. (oh-my-zsh with Powerlevel9k theme in WSL)

This is a duplicate of issue #77 .

This is a duplicate of issue #77 .


Refactor restGetAddressableByTopic to use operator

Fix #1492
Refactor the restGetAddressableByTopic REST function handler to use the
operator pattern in efforts to increase unit test coverage.
Update the restGetAddressableByTopic to use the TopicExecutor operator
rather than directly using the DBClient. This will allow for easy
mocking during testing.
Add tests for the newly added TopicExecutor operator and all of the
necessary layers needed by the operator.
Add tests for the REST handler restGetAddressableByTopic to cover the
different situations the function is expected to handle.
Signed-off-by: Anthony M. Bonafide<EMAIL_ADDRESS>

Codecov Report

Merging #1493 into master will increase coverage by 0.21%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #1493      +/-   ##
==========================================
+ Coverage   18.62%   18.84%   +0.21%
==========================================
  Files          89       89
  Lines        9093     9100       +7
==========================================
+ Hits         1694     1715      +21
+ Misses       7249     7235      -14
  Partials      150      150

Impacted Files
Coverage Δ

internal/core/metadata/rest_addressable.go
28.9% <100%> (+7%)
:arrow_up:

...nternal/core/metadata/operators/addressable/get.go
100% <100%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9421ecd...851b836. Read the comment docs.


Removed unnecessary public and small typo fix

Since Kotlin now has public as the default scope I removed cases of public. I also fixed a small typo.

Thanks


Feature: Progressive loading

Hi, @mkkellogg
Will the project support progressive loading in the future? Also, can the project support more user-friendly interactions such as pressing the forward/backward keys to move forward and backward scenes?

I would love to support progressive loading at some point, I think it's a nicer user experience and a pretty cool loading effect. It will take some work because of the way I currently use a SplatTree to  organize splats to boost performance -- basically I can't build the SplatTree until all the splats are loaded. I have some ideas to address that, I just haven't had time to implement them. Eventually I will :)
As for keyboard controls, that will be much easier to add; I should be able to get those into an upcoming release.

+1 for progressive loading, please!
It's a big blocker for using this library for anything in prod, since making the user wait for a double-digit MiB file to finish downloading before presenting anything is really tough.

Hi, has there been any progress regarding this ? Thank you.

Progress has been slow, but I have made some. I have a work-in-progress branch progressive_loading: https://github.com/mkkellogg/GaussianSplats3D/tree/progressive_loading that has everything I've done so far. Currently I'm only supporting progressive loading for my own .ksplat files for a couple of reasons:

I need to know how many splats are in the file ahead of time so I can allocate resources properly. .splat files don't have that info, but .ply files do so they could potentially be streamed as part of a future update. I know the Content-Length header could be used to determine the number of splats for a .splat, but that header is not guaranteed to be present for all web servers. That will also be a future update thing :)
I wanted to be able to control the layout of the splats in the file, because that has an impact on the kind of effects you can use to unveil the splats (right now I'm organizing them from the center of the scene going outward).

@gotoeasy I updated the progressive_loading  branch so that .splat files can be streamed now. It's still very much a work in progress, but you can try that branch out to preview it.

@gotoeasy I updated the progressive_loading branch so that .splat files can be streamed now. It's still very much a work in progress, but you can try that branch out to preview it.

Would you consider a transition effect where downloading and rendering happen simultaneously?
Like in https://github.com/antimatter15/splat

I'm not sure I understand, isn't that exactly what my progressive_loading branch is doing? You should be able to see the scene rendering long before the download is complete and you should see the splats updating in real-time.

I'm not sure I understand, isn't that exactly what my progressive_loading branch is doing? You should be able to see the scene rendering long before the download is complete and you should see the splats updating in real-time.

I saw the transition effect after pulling the new code. great!!!

Great! Let me know if you have any questions or other feedback. I hope to merge these updates into the main branch soon.

Great! Let me know if you have any questions or other feedback. I hope to merge these updates into the main branch soon.

I tried multiple models for viewing, and the results seemed quite good.
However, during the "Processing splats..." phase, it seems like the CPU usage is very high, causing short periods of lag. Nevertheless, this is not the same issue.

Apart from the default transition effect, it might be worth considering multiple built-in effect options for developers to choose from during development.

Definitely worth considering, although to some extent the kinds of transitions possible depend on how the splats are organized in the file and the order in which they are downloaded. I'll work on adding more effects in a later release :)

For what it's worth, I updated the .ksplat format and the exporter for it to organize the splats going from a scene's center outward. The result is kind of similar to Luma's approach for unveiling the scene from the center outwards, although I don't shrink the splats at point during the transition.

Amazing work! Are there any limits in this progressive_loading branch on the compressionLevel? Thx!

Great! Let me know if you have any questions or other feedback. I hope to merge these updates into the main branch soon.

When streamView is set to true, calling the viewer.dispose() method can easily lead to exceptions, especially when the model file has not fully loaded and displayed. After several repeated openings and closings, increased memory, CPU, or GPU usage may occur, potentially resulting in newly opened models failing to display.

Ah, that's a great point. I'll have think a bit about how to best handle that. Hopefully it's as simple as cancelling the download as part of the call to dispose().

I've made some updates to hopefully address the most recent issues with disposing while in the middle of loading. The exceptions you were seeing should now be prevented, and there was a memory leak that I fixed as well. Take a look when you have some time!

It's ok now. Great!

Can the Viewer.addSplatScenes method add this feature?

Eventually I would like to support streaming multiple scenes simultaneously, but for this upcoming release it will just be for single scenes.

During the transition effect, the parameter splatAlphaRemovalThreshold is not functioning.
Testing with splatAlphaRemovalThreshold = 250 clearly demonstrates this.

Yeah I noticed this issue too. I have a fix in the progressive_loading_updates branch, can you see if that works for you?

Going to close this for now, At some point in the future I hope to add support for progressively loading multiple scenes.


Any way to get complete history for pages visits?

Hi, I am new here and curious if it is possible to query complete view counts for a page?
The getDomains.statistics.pages query needs a range param which can't be null and only allows to query data up to 6 months.
So, 2 questions (I could probably figure this out from code but want to confirm)

are the views stored indefinitely in the DB? i.e. does Ackee delete 6 months+ old records?
is there any way to query that historic data from the API? IMO the simplest idea would be to make the range param nullable which will allow to not have any created filters in the aggregations.

Happy to make a PR for the latter. Thanks!

I did end up figuring this out! Re your questions @alexeyqu:

The MongoDB does keep all of the events, views, etc.
You do have to change the range filters in the code in multiple places, but I was able to add "Last 12 Months" as well as "Last 7 Years" (and I believe there would be no limit here)

Check it out here: https://github.com/UnlockNYC/Ackee/commit/3a70457ed2595fe288f4698914f5d595d35a6598

@mab253 This works fine. It would be great if at least the "Last 12 Months" option gets merged. Could be helpful for a yearly review. Did you consider creating a PR for this?


Add table aws_ssoadmin_permission_set

Integration test logs

Logs
% node tint.js aws_ssoadmin_permission_set
No env file present for the current environment:  staging
 Falling back to .env config
No env file present for the current environment:  staging
customEnv TURBOT_TEST_EXPECTED_TIMEOUT undefined

SETUP: tests/aws_ssoadmin_permission_set []

PRETEST: tests/aws_ssoadmin_permission_set

TEST: tests/aws_ssoadmin_permission_set
Running terraform
data.aws_region.alternate: Refreshing state...
data.aws_caller_identity.current: Refreshing state...
data.aws_partition.current: Refreshing state...
data.aws_ssoadmin_instances.main: Refreshing state...
data.aws_region.primary: Refreshing state...
data.null_data_source.resource: Refreshing state...
aws_ssoadmin_permission_set.main: Creating...
aws_ssoadmin_permission_set.main: Creation complete after 2s [id=arn:aws:sso:::permissionSet/ssoins-xxxxxxxxxxxxxxxx/ps-xxxxxxxxxxxxxxxx,arn:aws:sso:::instance/ssoins-xxxxxxxxxxxxxxxx]

Warning: Deprecated Resource

The null_data_source was historically used to construct intermediate values to
re-use elsewhere in configuration, the same can now be achieved using locals

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Outputs:

account_id = xxxxxxxxxxxx
aws_partition = aws
aws_region = us-east-1
resource_arn = arn:aws:sso:::permissionSet/ssoins-xxxxxxxxxxxxxxxx/ps-xxxxxxxxxxxxxxxx
resource_name = turbottest58627

Running SQL query: test-get-query.sql
Warning: executeQueries: query 1 of 1 failed: pq: rpc error: code = Unknown desc = InvalidParameter: 1 validation error(s) found.
- minimum field size of 10, DescribePermissionSetInput.PermissionSetArn.

{}
[
  {
    "arn": "arn:aws:sso:::permissionSet/ssoins-xxxxxxxxxxxxxxxx/ps-xxxxxxxxxxxxxxxx",
    "description": "steampipe-test",
    "name": "turbottest58627",
    "relay_state": "https://s3.console.aws.amazon.com/s3/home?region=us-east-1#",
    "session_duration": "PT1H"
  }
]

✘ FAILED

Running SQL query: test-list-query.sql
[
  {
    "arn": "arn:aws:sso:::permissionSet/ssoins-xxxxxxxxxxxxxxxx/ps-xxxxxxxxxxxxxxxx",
    "description": "steampipe-test",
    "name": "turbottest58627",
    "relay_state": "https://s3.console.aws.amazon.com/s3/home?region=us-east-1#",
    "session_duration": "PT1H"
  }
]
✔ PASSED

Running SQL query: test-notfound-query.sql
Warning: executeQueries: query 1 of 1 failed: pq: rpc error: code = Unknown desc = InvalidParameter: 1 validation error(s) found.
- minimum field size of 10, DescribePermissionSetInput.PermissionSetArn.

{}
null

✘ FAILED

Running SQL query: test-turbot-query.sql
[
  {
    "account_id": "xxxxxxxxxxxx",
    "akas": [
      "arn:aws:sso:::permissionSet/ssoins-xxxxxxxxxxxxxxxx/ps-xxxxxxxxxxxxxxxx"
    ],
    "partition": "aws",
    "region": "us-east-1",
    "tags": {
      "Name": "turbottest58627"
    },
    "title": "turbottest58627"
  }
]
✔ PASSED

TEARDOWN: tests/aws_ssoadmin_permission_set

SUMMARY:

✘ tests/aws_ssoadmin_permission_set failed.

0/1 passed.

Example query results

Results
> select
  name,
  ps.arn,
  created_date,
  description,
  relay_state,
  session_duration,
  tags
from
  aws.aws_ssoadmin_permission_set as ps
join
  aws.aws_ssoadmin_instance as i on ps.instance_arn = i.arn;
+---------+-------------------------------------------------------------------------+---------------------+-------------------+-------------------------+------------------+--------------------+
| name    | arn                                                                     | created_date        | description       | relay_state             | session_duration | tags               |
+---------+-------------------------------------------------------------------------+---------------------+-------------------+-------------------------+------------------+--------------------+
| Billing | arn:aws:sso:::permissionSet/ssoins-xxxxxxxxxxxxxxxx/ps-xxxxxxxxxxxxxxxx | 2021-09-13 07:15:16 | Access to billing | http://www.example.com/ | PT1H             | {"Type":"Billing"} |
+---------+-------------------------------------------------------------------------+---------------------+-------------------+-------------------------+------------------+--------------------+

This builds on #658 - the tests are currently not working due to an issue where the call on aws/table_aws_ssoadmin_permission_set.go:161 isn't getting the expected key back:
	var arn string
	if h.Item != nil {
		arn = *h.Item.(*PermissionSetItem).PermissionSetArn
		plugin.Logger(ctx).Trace("getSsoAdminPermissionSet", "arn_Item", arn)
	} else {
		arn = d.KeyColumnQuals["arn"].GetStringValue()
		plugin.Logger(ctx).Trace("getSsoAdminPermissionSet", "arn_KeyColumnQuals", arn)
		plugin.Logger(ctx).Trace("getSsoAdminPermissionSet", "d.KeyColumnQuals", d.KeyColumnQuals)
	}

I'm expecting d.KeyColumnQuals to contain an entry for arn, but I can only see instance_arn and region based on the trace logs. The table definition includes arn as a key column:
			KeyColumns:        plugin.AllColumns([]string{"instance_arn", "arn"}),

Is there anything else that needs to be done to be able to retrieve that value?

Hi @LalitTurbot 👋
I've addressed the comments, but the tests aren't currently passing (see the above comment).
The following query works fine:
select name, ps.arn
from aws.aws_ssoadmin_permission_set as ps
join aws.aws_ssoadmin_instance as i on ps.instance_arn = i.arn

The following does not:
select name, ps.arn
from aws.aws_ssoadmin_permission_set as ps
join aws.aws_ssoadmin_instance as i on ps.instance_arn = i.arn
where ps.arn = 'arn:aws:sso:::permissionSet/ssoins-1234567890abcef/ps-1234567890abcef';

The latter results in a call to getSsoAdminPermissionSet, but when that function tries to read d.KeyColumnQuals["arn"] there is no value - even though there's an appropriate value in the WHERE clause in that query. I'm not sure if I'm doing something wrong or if this is a potential bug? (I've declared the columns required in Table.Get)