[WIP][SPARK-31919][SQL] Push down more predicates through Join

What changes were proposed in this pull request?

Currently, in the rulePushPredicateThroughJoin, if the condition predicate of Or operator can't be entirely pushed down, it will be thrown away.
In fact, the predicates under Or operators can be partially pushed down.
For example, says a and b are able to be pushed into one of the joined tables, while c can't be pushed down, the predicate
a or (b and c)
can be converted as
(a or b) and (a or c)
We can still push down (a or b).
We can't push down disjunctive predicates only when one of its children is not partially convertible.
The common way is to convert the condition into conjunctive normal form(CNF), so that we can find all the predicates that can be pushed down by going over the CNF predicate.
However, CNF conversion result can be huge, the recursive implementation can cause stack overflow on complex predicates. There were PRs for it such as #10444, #15558, #28575.
There is also non-recursive implementation: #28733 . It should be workable but this PR proposes a simpler implementation.
Essentially, we just need to traverse predicate and extract the convertible sub-predicates like what we did in https://github.com/apache/spark/pull/24598 . There is no need to maintain the CNF result set.
Why are the changes needed?

Improve query performance. PostgreSQL, Impala and Hive support similiar feature.
Does this PR introduce any user-facing change?

No
How was this patch tested?

Unit test and benchmark test.

SQL
Before this PR
After this PR

TPCDS 5T Q13
84s
21s

TPCDS 5T q85
66s
34s

TPCH 1T q19
37s
32s

I will add more test cases.

Test build #123587 has started for PR 28741 at commit 2ca483e.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28211/
Test PASSed.

Test build #123587 has finished for PR 28741 at commit 2ca483e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123587/
Test PASSed.

Just to check; it seems this PR doesn't depend on CNF conversion, but we could get the totally same performance gains with https://github.com/apache/spark/pull/28733?

It seems this solution cannot be fully optimized.
PostgreSQL:
 Aggregate  (cost=77.33..77.34 rows=1 width=128)
   ->  Nested Loop  (cost=0.00..77.31 rows=1 width=32)
         Join Filter: (store_sales.ss_sold_date_sk = date_dim.d_date_sk)
         ->  Nested Loop  (cost=0.00..67.18 rows=1 width=36)
               Join Filter: ((store_sales.ss_addr_sk = customer_address.ca_address_sk) AND ((((customer_address.ca_state)::text = ANY ('{TX,OH,TX}'::text[])) AND (store_sales.ss_net_profit >=
'100'::numeric) AND (store_sales.ss_net_profit <= '200'::numeric)) OR (((customer_address.ca_state)::text = ANY ('{OR,NM,KY}'::text[])) AND (store_sales.ss_net_profit >= '150'::numeric) AND (s
tore_sales.ss_net_profit <= '300'::numeric)) OR (((customer_address.ca_state)::text = ANY ('{VA,TX,MS}'::text[])) AND (store_sales.ss_net_profit >= '50'::numeric) AND (store_sales.ss_net_profi
t <= '250'::numeric))))
               ->  Nested Loop  (cost=0.00..56.90 rows=1 width=54)
                     Join Filter: ((store_sales.ss_cdemo_sk = customer_demographics.cd_demo_sk) AND ((((customer_demographics.cd_marital_status)::text = 'M'::text) AND ((customer_demographics.
cd_education_status)::text = 'Advanced Degree'::text) AND (store_sales.ss_sales_price >= 100.00) AND (store_sales.ss_sales_price <= 150.00) AND (household_demographics.hd_dep_count = 3)) OR ((
(customer_demographics.cd_marital_status)::text = 'S'::text) AND ((customer_demographics.cd_education_status)::text = 'College'::text) AND (store_sales.ss_sales_price >= 50.00) AND (store_sale
s.ss_sales_price <= 100.00) AND (household_demographics.hd_dep_count = 1)) OR (((customer_demographics.cd_marital_status)::text = 'W'::text) AND ((customer_demographics.cd_education_status)::t
ext = '2 yr Degree'::text) AND (store_sales.ss_sales_price >= 150.00) AND (store_sales.ss_sales_price <= 200.00) AND (household_demographics.hd_dep_count = 1))))
                     ->  Nested Loop  (cost=0.00..46.10 rows=1 width=76)
                           Join Filter: (store_sales.ss_hdemo_sk = household_demographics.hd_demo_sk)
                           ->  Nested Loop  (cost=0.00..33.61 rows=1 width=76)
                                 Join Filter: (store_sales.ss_store_sk = store.s_store_sk)
                                 ->  Seq Scan on store_sales  (cost=0.00..23.60 rows=1 width=80)
                                       Filter: ((((ss_sales_price >= 100.00) AND (ss_sales_price <= 150.00)) OR ((ss_sales_price >= 50.00) AND (ss_sales_price <= 100.00)) OR ((ss_sales_price >
= 150.00) AND (ss_sales_price <= 200.00))) AND (((ss_net_profit >= '100'::numeric) AND (ss_net_profit <= '200'::numeric)) OR ((ss_net_profit >= '150'::numeric) AND (ss_net_profit <= '300'::num
eric)) OR ((ss_net_profit >= '50'::numeric) AND (ss_net_profit <= '250'::numeric))))
                                 ->  Seq Scan on store  (cost=0.00..10.00 rows=1 width=4)
                           ->  Seq Scan on household_demographics  (cost=0.00..12.45 rows=3 width=8)
                                 Filter: ((hd_dep_count = 3) OR (hd_dep_count = 1) OR (hd_dep_count = 1))
                     ->  Seq Scan on customer_demographics  (cost=0.00..10.75 rows=1 width=1036)
                           Filter: ((((cd_marital_status)::text = 'M'::text) AND ((cd_education_status)::text = 'Advanced Degree'::text)) OR (((cd_marital_status)::text = 'S'::text) AND ((cd_e
ducation_status)::text = 'College'::text)) OR (((cd_marital_status)::text = 'W'::text) AND ((cd_education_status)::text = '2 yr Degree'::text)))
               ->  Seq Scan on customer_address  (cost=0.00..10.24 rows=1 width=520)
                     Filter: (((ca_country)::text = 'United States'::text) AND (((ca_state)::text = ANY ('{TX,OH,TX}'::text[])) OR ((ca_state)::text = ANY ('{OR,NM,KY}'::text[])) OR ((ca_state
)::text = ANY ('{VA,TX,MS}'::text[]))))
         ->  Seq Scan on date_dim  (cost=0.00..10.12 rows=1 width=4)
               Filter: (d_year = 2001)
(22 rows)

After this PR(set spark.sql.constraintPropagation.enabled=false to ignore infer IsNotNull):
*(7) HashAggregate(keys=[], functions=[avg(cast(ss_quantity#10 as bigint)), avg(UnscaledValue(ss_ext_sales_price#15)), avg(UnscaledValue(ss_ext_wholesale_cost#16)), sum(UnscaledValue(ss_ext_wholesale_cost#16))])
+- Exchange SinglePartition, true, [id=#252]
   +- *(6) HashAggregate(keys=[], functions=[partial_avg(cast(ss_quantity#10 as bigint)), partial_avg(UnscaledValue(ss_ext_sales_price#15)), partial_avg(UnscaledValue(ss_ext_wholesale_cost#16)), partial_sum(UnscaledValue(ss_ext_wholesale_cost#16))])
      +- *(6) Project [ss_quantity#10, ss_ext_sales_price#15, ss_ext_wholesale_cost#16]
         +- *(6) BroadcastHashJoin [ss_hdemo_sk#5], [hd_demo_sk#61], Inner, BuildRight, (((((((cd_marital_status#54 = M) AND (cd_education_status#55 = Advanced Degree)) AND (ss_sales_price#13 >= 100.00)) AND (ss_sales_price#13 <= 150.00)) AND (hd_dep_count#64 = 3)) OR (((((cd_marital_status#54 = S) AND (cd_education_status#55 = College)) AND (ss_sales_price#13 >= 50.00)) AND (ss_sales_price#13 <= 100.00)) AND (hd_dep_count#64 = 1))) OR (((((cd_marital_status#54 = W) AND (cd_education_status#55 = 2 yr Degree)) AND (ss_sales_price#13 >= 150.00)) AND (ss_sales_price#13 <= 200.00)) AND (hd_dep_count#64 = 1)))
            :- *(6) Project [ss_hdemo_sk#5, ss_quantity#10, ss_sales_price#13, ss_ext_sales_price#15, ss_ext_wholesale_cost#16, cd_marital_status#54, cd_education_status#55]
            :  +- *(6) BroadcastHashJoin [ss_cdemo_sk#4], [cd_demo_sk#52], Inner, BuildRight, ((((((cd_marital_status#54 = M) AND (cd_education_status#55 = Advanced Degree)) AND (ss_sales_price#13 >= 100.00)) AND (ss_sales_price#13 <= 150.00)) OR ((((cd_marital_status#54 = S) AND (cd_education_status#55 = College)) AND (ss_sales_price#13 >= 50.00)) AND (ss_sales_price#13 <= 100.00))) OR ((((cd_marital_status#54 = W) AND (cd_education_status#55 = 2 yr Degree)) AND (ss_sales_price#13 >= 150.00)) AND (ss_sales_price#13 <= 200.00)))
            :     :- *(6) Project [ss_cdemo_sk#4, ss_hdemo_sk#5, ss_quantity#10, ss_sales_price#13, ss_ext_sales_price#15, ss_ext_wholesale_cost#16]
            :     :  +- *(6) BroadcastHashJoin [ss_sold_date_sk#0], [d_date_sk#79], Inner, BuildRight
            :     :     :- *(6) Project [ss_sold_date_sk#0, ss_cdemo_sk#4, ss_hdemo_sk#5, ss_quantity#10, ss_sales_price#13, ss_ext_sales_price#15, ss_ext_wholesale_cost#16]
            :     :     :  +- *(6) BroadcastHashJoin [ss_addr_sk#6], [ca_address_sk#66], Inner, BuildRight, ((((ca_state#74 IN (TX,OH) AND (ss_net_profit#22 >= 100.00)) AND (ss_net_profit#22 <= 200.00)) OR ((ca_state#74 IN (OR,NM,KY) AND (ss_net_profit#22 >= 150.00)) AND (ss_net_profit#22 <= 300.00))) OR ((ca_state#74 IN (VA,TX,MS) AND (ss_net_profit#22 >= 50.00)) AND (ss_net_profit#22 <= 250.00)))
            :     :     :     :- *(6) Project [ss_sold_date_sk#0, ss_cdemo_sk#4, ss_hdemo_sk#5, ss_addr_sk#6, ss_quantity#10, ss_sales_price#13, ss_ext_sales_price#15, ss_ext_wholesale_cost#16, ss_net_profit#22]
            :     :     :     :  +- *(6) BroadcastHashJoin [ss_store_sk#7], [s_store_sk#23], Inner, BuildRight
            :     :     :     :     :- *(6) Filter ((((ss_net_profit#22 >= 100.00) AND (ss_net_profit#22 <= 200.00)) OR ((ss_net_profit#22 >= 150.00) AND (ss_net_profit#22 <= 300.00))) OR ((ss_net_profit#22 >= 50.00) AND (ss_net_profit#22 <= 250.00)))
            :     :     :     :     :  +- *(6) ColumnarToRow
            :     :     :     :     :     +- FileScan parquet default.store_sales[ss_sold_date_sk#0,ss_cdemo_sk#4,ss_hdemo_sk#5,ss_addr_sk#6,ss_store_sk#7,ss_quantity#10,ss_sales_price#13,ss_ext_sales_price#15,ss_ext_wholesale_cost#16,ss_net_profit#22] Batched: true, DataFilters: [((((ss_net_profit#22 >= 100.00) AND (ss_net_profit#22 <= 200.00)) OR ((ss_net_profit#22 >= 150.0..., Format: Parquet, Location: InMemoryFileIndex[file:/Users/yumwang/opensource/spark/sql/core/spark-warehouse/org.apache.spark...., PartitionFilters: [], PushedFilters: [Or(Or(And(GreaterThanOrEqual(ss_net_profit,100.00),LessThanOrEqual(ss_net_profit,200.00)),And(Gr..., ReadSchema: struct<ss_sold_date_sk:int,ss_cdemo_sk:int,ss_hdemo_sk:int,ss_addr_sk:int,ss_store_sk:int,ss_quan...
            :     :     :     :     +- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int, true] as bigint))), [id=#213]
            :     :     :     :        +- *(1) ColumnarToRow
            :     :     :     :           +- FileScan parquet default.store[s_store_sk#23] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex[file:/Users/yumwang/opensource/spark/sql/core/spark-warehouse/org.apache.spark...., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<s_store_sk:int>
            :     :     :     +- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int, true] as bigint))), [id=#222]
            :     :     :        +- *(2) Project [ca_address_sk#66, ca_state#74]
            :     :     :           +- *(2) Filter ((ca_country#76 = United States) AND ((ca_state#74 IN (TX,OH) OR ca_state#74 IN (OR,NM,KY)) OR ca_state#74 IN (VA,TX,MS)))
            :     :     :              +- *(2) ColumnarToRow
            :     :     :                 +- FileScan parquet default.customer_address[ca_address_sk#66,ca_state#74,ca_country#76] Batched: true, DataFilters: [(ca_country#76 = United States), ((ca_state#74 IN (TX,OH) OR ca_state#74 IN (OR,NM,KY)) OR ca_st..., Format: Parquet, Location: InMemoryFileIndex[file:/Users/yumwang/opensource/spark/sql/core/spark-warehouse/org.apache.spark...., PartitionFilters: [], PushedFilters: [EqualTo(ca_country,United States), Or(Or(In(ca_state, [TX,OH]),In(ca_state, [OR,NM,KY])),In(ca_s..., ReadSchema: struct<ca_address_sk:int,ca_state:string,ca_country:string>
            :     :     +- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int, true] as bigint))), [id=#231]
            :     :        +- *(3) Project [d_date_sk#79]
            :     :           +- *(3) Filter (d_year#85 = 2001)
            :     :              +- *(3) ColumnarToRow
            :     :                 +- FileScan parquet default.date_dim[d_date_sk#79,d_year#85] Batched: true, DataFilters: [(d_year#85 = 2001)], Format: Parquet, Location: InMemoryFileIndex[file:/Users/yumwang/opensource/spark/sql/core/spark-warehouse/org.apache.spark...., PartitionFilters: [], PushedFilters: [EqualTo(d_year,2001)], ReadSchema: struct<d_date_sk:int,d_year:int>
            :     +- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int, true] as bigint))), [id=#238]
            :        +- *(4) ColumnarToRow
            :           +- FileScan parquet default.customer_demographics[cd_demo_sk#52,cd_marital_status#54,cd_education_status#55] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex[file:/Users/yumwang/opensource/spark/sql/core/spark-warehouse/org.apache.spark...., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<cd_demo_sk:int,cd_marital_status:string,cd_education_status:string>
            +- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int, true] as bigint))), [id=#246]
               +- *(5) Filter (((hd_dep_count#64 = 3) OR (hd_dep_count#64 = 1)) OR (hd_dep_count#64 = 1))
                  +- *(5) ColumnarToRow
                     +- FileScan parquet default.household_demographics[hd_demo_sk#61,hd_dep_count#64] Batched: true, DataFilters: [(((hd_dep_count#64 = 3) OR (hd_dep_count#64 = 1)) OR (hd_dep_count#64 = 1))], Format: Parquet, Location: InMemoryFileIndex[file:/Users/yumwang/opensource/spark/sql/core/spark-warehouse/org.apache.spark...., PartitionFilters: [], PushedFilters: [Or(Or(EqualTo(hd_dep_count,3),EqualTo(hd_dep_count,1)),EqualTo(hd_dep_count,1))], ReadSchema: struct<hd_demo_sk:int,hd_dep_count:int>

@wangyum Thanks for point it out.
I made a mistake. This solution is not as powerful as CNF conversion since it can't even push down such predicate
(p1(t1) or p2(t2)) and p3(t1)

to t1
with CNF we can push down
p1(t1) or p3(t1)

I am closing this one and reopening #28733


Add tags for stats extracted by configurable regexes.

Add configurable tags for stats objects.  As a part of this process, a few new concepts were added:
TagExtractor - an object that is used to extract a particular tag from stat names.
Metric - a general interface for metrics that provides identifying information.
Histograms - these existed in some form, but now there is an explicit class for them. Note: I didn't add a macro for the creation of these.  This is mainly because it would be unused, but I am open to adding one if there's demand.
As a part of this addition, there is a set of default regexes.  These have been tested, but I am especially interested in comments on the fragility of these.  I'm somewhat new to writing regex, so please let me know if you find any issues.
This is still missing tests for the configuration and reading via a test sink.  I will be adding these, but wanted to go ahead and get this out for review to get a head start.

Closing.  Will introduce as two separate PRs as @htuch and @mattklein123 have suggested.


Make a few BugsnagCrashReport properties public

Move a few BugsnagCrashReport property definitions to the header file
to make them public:

error
device
deviceState
app
appState

Perhaps some more of the properties should be made public as well?
We need this in order to access this data in a addBeforeSendBlock
block. While upgrading from 5.3.0 to 5.6.3, we changed our code to
use addBeforeSendBlock instead of addBeforeNotifyBlock, which has
been deprecated. However, addBeforeSendBlock provides
BugsnagCrashReport as an argument, rather than a raw NSDictionary
containing the error, device, deviceState, appState, etc.
fields. We're consuming this information in order to log analytics data
to our own servers. By exposing these properties on
BugsnagCrashReport, it allows us to keep logging this information in
the addBeforeSendBlock block.
to: @kattrali
cc: @lelandrichardson

Addresses #122

Looks good. Thanks, @spikebrehm!


Fix: some typos

Hi Klaus,
Thank you for your amazing tool and work. I've gone through the doc of sacred, and I noticed some typos that I fixed. If it can help 😄

Coverage remained the same at 83.851% when pulling 9d0671d768ad565a66cc35df8f26ecdd9593ba45 on antoinemertz:doc-typo into 370bcd000b0a31fed01da963e269d1c6a759ad7a on IDSIA:master.

Thanks! :-)


How to clean the caches

Before the last release we could reset the caches by passing an environment variable like:
      - name: Remove all caches and database of the trivy scanner
        uses: aquasecurity/trivy-action@d9cd5b1c23aaf8cb31bb09141028215828364bbb
        env:
          TRIVY_RESET: true
        with:
          scan-type: "image"

But now this --reset flag is not longer there and there is only now the utility command
trivy clean --all
But there is no option to pass this information as an environment variable.
How can I clean the caches using the github action? Thanks!

@VictorDelCampo thanks for the question, we will get back to you on this.

Looks like we will need to add this in as a feature for Trivy-action as currently there's no possible way from the action to clear the cache.


Support JSON / any Ruby Hash

Would be great to use to parse incoming JSON for our API.

basically a parse_json and parse_hash generalisation.
Makes sense and should be easy to implement.

Nope, I don't think yack should know about JSON/YAML, it should just work with arbitrary Hashes.
yack.process(YAML::load(whatever))
yack.process(JSON.load(whatever))
yack.process({this: :works, too: true})

Implemented and released in 0.0.2


[SPARK-46733][CORE] Simplify the BlockManager by the exit operation only depend on interrupt thread.

What changes were proposed in this pull request?
This PR propose to simplify the BlockManager.
Why are the changes needed?
Currently, close or destroy BlockManager depend on interrupt thread and the volatile variable stopped.
In fact, we can change the stopped to a local variable on stack and let the close operation of BlockManager only depend on interrupt thread.
For further optimization, this PR using running instead of stopped.
Does this PR introduce any user-facing change?
'No'.
How was this patch tested?
GA tests.
Was this patch authored or co-authored using generative AI tooling?
'No'.

Unlike ReloadingX509TrustManager , here the InterruptedException is caught and ignored in some of the downstream methods: so we cant rely on that pattern here.

ContextCleaner catches InterruptedException but ignore it. How can downstream capture it?

Unlike ReloadingX509TrustManager , here the InterruptedException is caught and ignored in some of the downstream methods: so we cant rely on that pattern here.

I reverted the ContextCleaner.

@dongjoon-hyun @srowen @LuciferYang

This change of this pr is fine to me.
However, I would like to raise a question: Would it be simpler and clearer to implement the logic for cleaning up the DownloadFile resource using the java.lang.ref.Cleaner mechanism instead of the current approach?

Java.lang.ref.Cleaner cannot respond to interrupts.

cc @mridulm
Could you take a review again?

Since you looked into this - do you have any other thoughts on this @LuciferYang ? Thanks

no more, the change is fine to me

Merged to master.
Thanks for working on this @beliefer !
Thanks for the review @LuciferYang :-)

@mridulm @LuciferYang Thank you!


Prefer SystemVerilog-2017 instead of 2012

I'd like to propose to prefer SystemVerilog-2017 instead of 2012.
Reasons are;

SystemVerilog-2017 is just a clarified version of SystemVerilog-2012.
Language Reference Manual (LRM) of SystemVerilog-2017 is available for download at no cost.

The current description;
L181

Prefer SystemVerilog-2012.
All RTL and tests should be developed in SystemVerilog, following the SystemVerilog-2012 standard, except for prohibited features.

As described in this blog, SystemVerilog 2017 has no new features.  The only functional change is removal of the operator overloading.  Other changes are clarifications.
We have a reference to SystemVerilog 2012 which costs $449.
On the other hand the SystemVerilog 2017 LRM is part of IEEE GET Program.  Readers can download it at no cost.
Preferring SystemVerilog-2017 encourage readers to access genuine clarified LRM and let them know SystemVerilog is a stable standard.
If you agree, I will propose a pull-request.
Thanks.

I'm supportive of this. Internally Google has migrated to recommending SystemVerilog-2017 as well. Anyone else have opinions here? @imphil @tjaychen @eunchan @msfschaffner @tomroberts-lowrisc @martin-lueker @sriyerg

SGTM

Yes, I think that's a reasonable move.

Thanks for the detailed description.  I think this is fine.

SGTM. I will check & confirm if our simulators are already relying on the 2017 spec. We may need to do the same for other ASIC tools as well.

SGTM


Symbolic links to folders can lead to the same file opened in two tabs

General Description
When a folder that is itself a symlink is used as a workspace, VS Code does not recognize that the files in that folder are the same as files in the linked-to folder. Saving new files or opening existing files will use the linked-to folder, the Explorer Tree will use the symlinked folder, and Search will allow both, allowing the same file to be opened concurrently in two different editor tabs in the same project.

Versions Used

VSCode Version: 1.46.1
From Code > About Visual Studio Code:
Version: 1.46.1
Commit: cd9ea6488829f560dc949a8b2fb789f3cdc05f5d
Date: 2020-06-17T21:17:14.222Z (23 hrs ago)
Electron: 7.3.1
Chrome: 78.0.3904.130
Node.js: 12.8.1
V8: 7.8.279.23-electron.0
OS: Darwin x64 19.4.0

OS Version
From Apple > About This Mac: OS X Catalina 10.15.4
From Help > Report Issue (copy-pasted):
CPUs | Intel(R) Core(TM) i7-5557U CPU @ 3.10GHz (4 x 3100)
-- | --
GPU Status | 2d_canvas: enabledflash_3d: enabledflash_stage3d: enabledflash_stage3d_baseline: enabledgpu_compositing: enabledmetal: disabled_offmultiple_raster_threads: enabled_onoop_rasterization: disabled_offprotected_video_decode: unavailable_offrasterization: enabledskia_renderer: disabled_off_okvideo_decode: enabledviz_display_compositor: enabled_onviz_hit_test_surface_layer: disabled_off_okwebgl: enabledwebgl2: enabled
Load (avg) | 8, 5, 6
Memory (System) | 16.00GB (2.35GB free)
Process Argv | --disable-extensions test_project_link
Screen Reader | no
VM | 0%

Steps to Reproduce

Setup

Create a new directory, e.g. mkdir test_project
Create a symbolic link to that directory, e.g. ln -s test_project test_project_link
Open the symlinked folder in VS Code, e.g. code  --disable-extensions test_project_link

Reproduce

Create a new file in the symlinked workspace, and save it, e.g. to example.txt

BUG 1: File is shown in the open editors in the non-symlinked directory, e.g example.txt - ~/test_project

Select the created file in the Explorer Tree.

BUG 2: It opens a second editor for the same file.

Use the command palette to search for your file, e.g. ⌘P and type example.txt.

BUG 3: Both the non-symlinked and symlinked file will appear, and can be open in separate editors.

Extra Notes

Does this issue occur when all extensions are disabled?: Yes
Does this issue occur in VSCode Insiders?: Yes

Version: 1.47.0-insider
Commit: 84f52fe1c0124307046ac0f5443493cda946e60f
Date: 2020-06-17T06:56:23.406Z
Electron: 7.3.1
Chrome: 78.0.3904.130
Node.js: 12.8.1
V8: 7.8.279.23-electron.0
OS: Darwin x64 19.4.0

I also experience this issue when debugging on Ubuntu 19.10.
I store my projects in ~/Code which is symlinked to a /media dir. When the debugger needs to display a file, such as when it hits a breakpoint, it opens a new tab with the /media/... path instead of the symlinked one.

There might already be issues like this. You might find them if you search for label feature-reqeust and some keyword like symlink.
However for now leaving this issue open and assigning to backlog. Thanks

Hi @isidorn.
I have searched as you suggested (is:issue symlink label:feature-request), and while there are many issues that match that search, none seem to cover this; most are about editing symbolic links as files, instead of following them.
A couple issues look lightly related:

#44635: mouseover in the Editor shows the "real path" and not symlink path

Seems related, and could be added to this list; a solution to this issue might solve #44635, but it seems less likely the other way around unless it were intentional.

#18837 (and dupes #32649, #30682): Do not resolve symbolic links when opening folders

The feature request there might provide a solution to this issue, but it is not the only possible solution, and that particular solution might not be a complete solution to this issue.

This is a great write-up of a pressing issue. I second this feature request. There are lots of projects where it makes sense to symlink directories.
It's especially useful when using C/C++ and CMake Tools in a multi-root workspace where each workspace folder has its own CMakeLists.txt, toolchain file for its particular target architecture, and build folder. In such a project, the intellisense provided by CMake Tools gets messed up when symlinks are followed between workspace folders instead of treated like regular files. If there was a way to treat symlinks as regular files (by not following the symlinks) within VS Code, then the intellisense provided by CMake Tools would probably work.
I think the settings request posed by https://github.com/microsoft/vscode/issues/97030 is a sensible one and would address this isssue.

@MBetters: from what I can tell, your issue sounds more similar to #18837. This issue (#100533) occurs specifically when the root workspace directory is a symlink (or one of its parents is a symlink); your issue sounds like it involves symlinks that exist inside the workspace.
As such, this issue does not appear solvable by #97030; the root directory or its parents are not necessarily even visible inside the explorer, much less followable, and so a setting concerned specifically with the explorer would not necessarily solve this issue.

I have the same issue on windows

mkdir dir1\a.txt
mklink \D dir1 dir2
code dir2\a.txt
it will show dir1

I'm seeing the same issue here on macOS, perhaps VS Code should realpath the workspace when it is opened? Doe that seem reasonable?
Below, foobar points to bar but VS Code treats them separately, leading to duplicate file issues seen above.
mkdir -p /tmp/foo/bar && touch /tmp/foo/bar/bar.txt
ln -s /tmp/foo/bar /tmp/foo/foobar
code /tmp/foo/foobar
code /tmp/foo/bar

We are unlikely to detect symbolic links pointing to the same file, thus closing as out of scope.

@isidorn When you open a file in the editor it should internally store the path to the original file. You can resolve a file's symlink to get the target file. When opening what looks to be a new file the editor could look through the open tabs and compare the internal path.
The tab's title should remain unchanged, the only difference is storing the canonical path.

We have the infrastructure in place for mapping paths to canonical ones:
https://github.com/microsoft/vscode/blob/e9565331e70e3f5145ae049a3f8d7e50846f4972/src/vs/workbench/services/uriIdentity/common/uriIdentityService.ts#L70-L70
However this is code that can execute sync and does not require any disk lookup. We use this to fix path-casing issues on file systems that are case-insensitive (Windows, macOS).
We are unlikely to ever make this async given how often this method is used, e.g. for problem markers that can potentially produce millions of entries.

I run into this issue with bazel. It builds by creating a workspace with symlinked source files and builds in that folder. This is all good, but then when a compile_commands.json is generated it will contain those symlinked files.
I can see how it is not trivial to integrate any solution to this issue into VS Code as a whole. Maybe a filename resolver hook for opening files and implement this in a plugin would be better. Not totally integrated, but still provide some relieve.

I also experience this issue when debugging on Ubuntu 19.10.
I store my projects in ~/Code which is symlinked to a /media dir. When the debugger needs to display a file, such as when it hits a breakpoint, it opens a new tab with the /media/... path instead of the symlinked one.

If you didn't solve your problem yet try to use below setting of "launch" -> "configurations":
"sourceFileMap": {
"/media/": "~/Code", (or maybe /home/YOUR_HOME if ~ doesn't work)
},

I had an issue where some files showed up multiples times (each with multiple possible paths, due to symlinks). I fixed it with the File: Clear Recently Opened command. Afterwards, closing and opening the same file with code symlink1/myfile and code hardlink1/myfile always resulted in the same path.

OneDrive compounds this symbolic link issue on MacOS (Ventura 13.2).
I believe I've experienced this whether starting up with "zsh" or the "Javascript Debug Terminal".
Closing VSCode at: /Users/myself/OneDrive/projects/foo
VSCode reopens at: /Users/myself/Library/CloudStorage/OneDrive-Personal/projects/foo
Changing settings to:
"debug.javascript.terminalOptions": {
    "cwd" : "${workspaceFolder}"
}

Yields a path with .../Library/CloudStroage/OneDrive-Personal/... instead of the symbolic link .../OneDrive/...
Even kluges (tried so many) like this fails as cwd will automatically expand the Symbolic link:
"debug.javascript.terminalOptions": {
    "cwd" : "${workspaceFolder}/../../../../../OneDrive"
}

One solution may be to add an option for the path to not have it's symbolic links expanded.
e.g,
"expandLinks" : false

Affected by this too. Is there a good workaround for this? I want to use symlinks to easily switch in between different git worktrees of a project behind a symlink. But this behavior makes the usage quite annoying.

Why is this labeled out-of-scope?
A Microsoft product "Visual Studio Code" is failing with a Microsoft product "OneDrive" on a platform they both support.
Further, this issue doesn't happen in the standalone Terminal, it only in Visual Studio Code's terminal.
If not Visual Studio Code's team, who should be addressing this issue?  The OneDrive team or ???

If vs-code doesn't supported multiple folders in the same workspace referencing a shared symlinked folder, what is the alternative for sharing source across projects?  Perhaps the vs-code team could address the issue direction as part of the vs-code documentation?
I've been using symlinked folders for years and as a result many of the "features" of vs-code are unreliable for me: global search/replace, code navegation, refactoring.  If there is a solution to shared source folders, I'd love to know what it is.

Meet with this problem as well on Linux. It should be noticed as softlink are used not-so-rarely.

Has this issue been resolved yet?

Quite disappointing it is considered "out-of-scope".
In the meantime, this workaround did the job for me:
# Function: command_realpath
# Description: This function resolves the real path of files or directories passed as arguments.
# Parameters:
#   - command_name: The name of the command to be executed.
#   - args: The list of arguments to be passed to the command.
# Returns: None
command_realpath() {
    local command_name="$1"
    shift
    local args=()
    for arg in "$@"; do
        if [[ -f "$arg" || -d "$arg" ]]; then
            args+=("$(readlink -f "$arg")")
        else
            args+=("$arg")
        fi
    done
    command "$command_name" "${args[@]}"
}

alias code='command_realpath code'

The worst part (that happened to me a few times) is making (unsaved) editing changes in a file and its symlinked incarnation and then having to resolve conflicts when saving the 2nd one!

We use symlinks quite a bit here when we need some things on local storage instead of NFS, but still want to maintain a convenient file structure. Very annoying that vscode gets so confused by such a basic concept as symlinks. :/
@synek317 provided a nice workaround for invoking vscode on the command line. Thanks for that. Has anyone found something to adjust things when opening workspaces directly from vscode?

@CendioOssman  : I am using this in the .vscode/settings.json file at the workspace root.
The symlinks still lead to files being opened "twice" if I deliberately open them, but at least, they don't appear in search results, which dramatically reduces the probability of having them edit through the link and the proper path.
I guess this will do for now (It's a WSL2 project in a Win 11 host, if it matters)
{
    // This is a workaround for a VSCode issue: symlinked folders are not properly handed and
    // the same file can appear multiple times in the search results (this is not a bug because
    // the following setting is precisely meant to allow this behavior), but also be opened several
    // times in the editor, with VSCode unable to understand that the file is the same, which can
    // lead to a different set of in-memory changes appear for the same underlying file, thus
    // leading to conflicts and potential data loss if not handled properly.
    "search.followSymlinks":false
}

Hi, I'm experiencing this issue as well.  I only ever access the files in question via the symlink, but repeatedly end up with two copies of the files open in VS Code.  At no point is this file ever referenced except through the symlink, so VS Code should never have a version of it open that uses the absolute non-symlinked path.

@isidorn, you said that:

we are unlikely to detect symbolic links pointing to the same file, thus closing as out of scope.

but in this case, the non-symlinked version is not referenced anywhere, so you wouldn't need to detect whether they're pointing at the same file or not.  You just need to find any code that converts the symlinked filenames into the non-symlinked versions and stop them from doing so.  Please would it be possible to re-open this?

Well this is pretty annoying, I encountered this while using debugger on a C project. As a temporary workaround I resolved symlink before opening project folder, that way opening files from built-in terminal doesn't create duplicate tabs.
There is no reason to not resolve symbolic links either when the project is opened or when opening a file. Either of these approaches would solve the issue.

The worst part (that happened to me a few times) is making (unsaved) editing changes in a file and its symlinked incarnation and then having to resolve conflicts when saving the 2nd one!

I'm almost certain that's what deleted a source file for me, I had to restore my changes one by one since last commit.

It seems to me that rather than saying it can't be done, it would be far better to actually just check if two things are the same file the way that you are supposed to on Unix-like systems. Check if they have the same inode number rather than comparing paths. No need to resolve the symlink. just call stat() which is probably being done anyway when opening a file.
Please re-open and actually fix this. It's VERY easy to loose work by editing the same file in two places without vscode noticing and saving over work. I've done it multiple times. It also seems to interfere with correct lookup of symbol locations with the clang extension.

It seems to me that rather than saying it can't be done, it would be far better to actually just check if two things are the same file the way that you are supposed to on Unix-like systems. Check if they have the same inode number rather than comparing paths. No need to resolve the symlink. just call stat() which is probably being done anyway when opening a file.
Please re-open and actually fix this. It's VERY easy to loose work by editing the same file in two places without vscode noticing and saving over work. I've done it multiple times. It also seems to interfere with correct lookup of symbol locations with the clang extension.

That's for hard links. Soft symlinks have their own inode. You have to resolve it first

When opening a file in an editor it has to follow the symlink. Also stat() follows symlinks by default, so you WILL get the inode number of the actual file that would be opened. Even if this is only partially fixed just for editor windows, which HAVE to call open and pay the cost of the symlink being resolved ANYWAY, it would be a significant improvement.
The inode is the unique identifier of a file, and is what should be used to answer the question of if two things are the same FILE, comparing paths only ever tells you if things are the same path witch is not the same thing.
There will still be problems, but it would prevent very confusing behaviour I see CONSTANTLY. There is a symlink in the path to the git repo for the code I work with freqently, it's not inside the repo, but every time I jump to a symbol location it is the RESOLVED path. This seems to also confuse vscode and leads to cases where it doesn't seem to fully be able to use all the refactoring tools since the locations clang gives for things seem to be the resolved path not the one that vscode sees. It makes working with this repo very frustrating.

Either having an option to follow symlinks for all opened files or making it the default behaviour would definitely solve the issue with duplicate tabs of the same file. Though I can see some cases where you wouldn't want to resolve symlink so imo it's better to be kept as a toggle.

I don't think it's correct to check inode. Like @Lassebq pointed out, softlink or symbolic links have different inodes from their targets.
Both Linux and MacOS have readlink. In particular, '-f' or '--canonicalize' mode could be very helpful here. Every time vscode open a new file, I think it should go through so-called canonicalization because the path may contains softlinks.

stat follows symlinks by default......???
$ stat .profile
  File: .profile -> /home/lassebq/dots/config/profile
  Size: 33        	Blocks: 0          IO Block: 4096   symbolic link
Device: 8,18	Inode: 1838658     Links: 1
Access: (0777/lrwxrwxrwx)  Uid: ( 1000/ lassebq)   Gid: (  100/   users)
Access: 2024-05-28 23:17:03.153715491 +0300
Modify: 2024-04-26 22:26:48.260790036 +0300
Change: 2024-04-26 22:36:12.874086004 +0300
 Birth: 2024-04-26 22:26:48.260790036 +0300
$ stat dots/config/profile
  File: dots/config/profile
  Size: 3042      	Blocks: 8          IO Block: 4096   regular file
Device: 8,18	Inode: 1835251     Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/ lassebq)   Gid: (  100/   users)
Access: 2024-05-29 14:06:49.270614111 +0300
Modify: 2024-05-19 15:19:40.797907278 +0300
Change: 2024-05-19 15:19:40.804573953 +0300
 Birth: 2024-04-05 01:39:22.804816521 +0300

1838658 does not equal 1835251. So unless you're referring to some other stat soft symlinks are a unique file with different inode.

I'm talking about the stat syscall not the command which does give you the information about the file pointed at by the symlink.
In either case what stat does with symlinks is irrelevant to what happens if one of the directories in the path to the file is a symlink, that will give you the info about the file with the command too.
From the man page:

 stat() and fstatat() retrieve information about the file pointed
 to by pathname; the differences for fstatat() are described
 below.

 lstat() is identical to stat(), except that if pathname is a
 symbolic link, then it returns information about the link itself,
 not the file that the link refers to.

 fstat() is identical to stat(), except that the file about which
 information is to be retrieved is specified by the file
 descriptor fd.

I don't think the core issue is "how" to solve the problem so much, but that Microsoft has chosen to ignore the issue.   Refactoring, search/replace, and symbol navigation all cause duplicate instances of files to be opened which then leads to corrupted files.
I wish they would at least provide a supported alternate workaround for sharing code across multiple projects

Not even just symlinks within the project. Symlinks in the path to one project that doesn't have symlinks anywhere inside the git repo or project directory cause issues too. I'm using both the ctags extension and the clangd extension. The build system produces a compile_commands.json with resolved paths which if there is a symlink ANYWHERE along the absolute path to the file causes this problem, and seems to just make refactoring and jump to symbol stop working entirely sometimes since the path of the file I'm working in doesn't match the path clangd knows.

I don't think the core issue is "how" to solve the problem so much, but that Microsoft has chosen to ignore the issue. Refactoring, search/replace, and symbol navigation all cause duplicate instances of files to be opened which then leads to corrupted files.
I wish they would at least provide a supported alternate workaround for sharing code across multiple projects

Microsoft folks don't own you. they should not work over time and address your problems.
It's an open-source project and you can create a PR anytime. that's why it's a "how" problem.

Is there any workaround for this? It's very irritating to to have a file that is being debugged open two times and get confused over the additional break points that are being created.

@Lassebq Thank you, but I open my project from within VScode and there I have multiple directories.
I read somewhere that you can circumvent the issue by setting pathMappings within the launch.json for the debugging configuration. Does anyone happen to know how the localroot/remoteroot should be set? Assuming the symlinked file is in my home directory ~/file and the debugger opens the resolved path beegfs/homes/xxx/file?

Unfortunately I don't know how to deal with in-workspace symlinks. My main problem was how it deals with symlinks outside of workspace. And due to the way my /home is laid out, every project I open will have duplicate tabs, one opened by debugger and one opened directly from explorer sidebar. This can be worked around by resolving all parent symlinks prior to opening a directory


Create Cloudformation_Template.yaml

Initial CF Template. Added Simple S3 bucket, Lambda, RestAPI, DDB Table.
Not test yet!!

Merging to dev branch


Setup precompiled widgets configuration

At the moment, the user still needs to do #include "xwidgets/xprecompiled.hpp" to get the precompiled widgets.
It cannot be done at a base widget level because it causes circular includes. Not quite sure what is the cleanest way to do this.

Maybe each block
extern template class xmaterialize<xtogglebutton>;
extern template xmaterialize<xtogglebutton>::xmaterialize();
extern template class xtransport<xmaterialize<xtogglebutton>>;
extern template class xgenerator<xtogglebutton>;
extern template xgenerator<xtogglebutton>::xgenerator();
extern template class xtransport<xgenerator<xtogglebutton>>;

should be in the corresponding header.
@gouarin thoughts?

Done in the next commit.


update dependencies ui-components and KUI

Update ui-components to current 0.179.0 version and pick up KUI packages at 10.3.25 with fixes for https://github.com/open-cluster-management/backlog/issues/13777, https://github.com/open-cluster-management/backlog/issues/13779 and https://github.com/open-cluster-management/backlog/issues/13803

Contains update to ui-components to v0.179.0 for https://github.com/open-cluster-management/backlog/issues/12617


The hosted instance is not working

9/10 times, the server responds with a 400 or 502 status code, even when using the form given on the homepage to grab a favicon

Experiencing the same issue here. Sometimes it works, sometimes there's a 502, and I can't really put my finger on how or why


Drop status column from content report tables

Description

See the following for implementation description and rationale: https://docs.openverse.org/projects/proposals/trust_and_safety/content_report_moderation/20231208-implementation_plan_django_admin_moderator_access.html#backfill-moderationdecision-and-drop-report-status
Blocked by #3641.

This is not directly related to this project but I wanted to suggest inheriting from OpenLedgerModel in the report models (currently they do not, and instead use a custom field for created_at) which would introduce a migration so it's possible to run both migrations simultaneously to reduce the downtime on that table.

Good call! Unifying the date columns is a great idea, especially given how small the tables are at the moment. Would it require following the zero downtime migration policy? If so, we should create the issues for each part of the process, maybe as follow-up tasks to the project rather than part of it.

This is still blocked by https://github.com/WordPress/openverse/pull/4530, but can be tackled after that is run!

The backfill was run between Tuesday and Wednesday so this is unblocked.


duplicate calisphere-ids - 99 records appear in 2 collections

While combining the existing individual collection indices into one large stage index, I noticed that 99 records are in 2 collections.
46 duplicates
26976 asset-library/UCM/Merced County Historical Society (200 items)
27358 /asset-library/UCM/Merced County Historical Society/Sheet Music Collection (46 items)
Collection 27358 is a subset of 26976
1 duplicate
26867 http://www.luna.blackgold.org/luna/servlet/oai blackgold~1~1
26866 http://www.luna.blackgold.org/luna/servlet/oai blackgold~2~2
2 records with the same ark: ark:/13030/c8445k8n
33 duplicates
27926 http://lib-metadata.ucsd.edu/solr/blacklight collections_tesim:bb2022061d
27323 http://lib-metadata.ucsd.edu/solr/blacklight/ collections_tesim:bb8605360n
These look like records with the same ark
1 duplicate
26176 http://lib-metadata.ucsd.edu/solr/blacklight/ collections_tesim:bb13322220
26900 http://lib-metadata.ucsd.edu/solr/blacklight/ collections_tesim:bb8228040z
13 duplicates
28121
28125
5 duplicates
28121
27768
We will need to resolve this problem so that calisphere-id is unique and reharvest these collections. The new combined index contains only one copy of each of the duplicate records (whichever it encountered 2nd).

Update on UCM's 46 duplicates
26976 asset-library/UCM/Merced County Historical Society (200 items) --> updated filepath to asset-library/UCM/Merced County Historical Society/Merced Falls
27358 /asset-library/UCM/Merced County Historical Society/Sheet Music Collection (46 items)
Collection 27358 is a subset of 26976
These ARK conflicts should be resolved with a reharvest to the new single index.

Update on BlackGold's 1 duplicate
2 records with the same ark: ark:/13030/c8445k8n
26867 http://www.luna.blackgold.org/luna/servlet/oai blackgold11

Not on Calisphere
on luna: http://www.luna.blackgold.org/luna/servlet/detail/blackgold~1~1~405~155

Remove the ARK from this record.

26866 http://www.luna.blackgold.org/luna/servlet/oai blackgold22

on production: https://calisphere.org/item/ark:/13030/c8445k8n/
on luna: http://www.luna.blackgold.org/luna/servlet/detail/blackgold~2~2~90~760

Okay to keep the ARK assigned to this record.

To do: Contact BlackGold and ask them to remove the ARK from 26867. Reharvest after the ARK conflict is resolved.

Update on UCLA's duplicates
13 duplicates
28121 (246 records) https://digital.library.ucla.edu/catalog/oai metadataPrefix=oai_dpla&set=member_of_collection_ids_ssim:q0v2f41z-89112

OAI: https://digital.library.ucla.edu/catalog/oai?verb=ListRecords&metadataPrefix=oai_dpla&set=member_of_collection_ids_ssim:q0v2f41z-89112
on production: https://calisphere.org/collections/28121/

28125 (13 records) https://digital.library.ucla.edu/catalog/oai metadataPrefix=oai_dpla&set=member_of_collection_ids_ssim:s7nf2000zz-89112

OAI: https://digital.library.ucla.edu/catalog/oai?verb=ListRecords&metadataPrefix=oai_dpla&set=member_of_collection_ids_ssim:s7nf2000zz-89112
on production: https://calisphere.org/collections/28125/

5 duplicates
28121 (246 records) https://digital.library.ucla.edu/catalog/oai metadataPrefix=oai_dpla&set=member_of_collection_ids_ssim:q0v2f41z-89112

OAI: https://digital.library.ucla.edu/catalog/oai?verb=ListRecords&metadataPrefix=oai_dpla&set=member_of_collection_ids_ssim:q0v2f41z-89112
on production: https://calisphere.org/collections/28121/

27768 (5 records) https://digital.library.ucla.edu/catalog/oai metadataPrefix=oai_dpla&set=member_of_collection_ids_ssim:k13t1n-89112

OAI: https://digital.library.ucla.edu/catalog/oai?verb=ListRecords&metadataPrefix=oai_dpla&set=member_of_collection_ids_ssim:k13t1n-89112
on production: https://calisphere.org/collections/27768/

To do:

[ ] I think this is an issue with records being part of multiple sets. Let's discuss.

Update on UCSD's duplicates
These look like records with the same ark
33 duplicates
27926 http://lib-metadata.ucsd.edu/solr/blacklight collections_tesim:bb2022061d

on production: https://calisphere.org/collections/27926/
on -stage: https://calisphere-stage.cdlib.org/collections/27926/
27323 http://lib-metadata.ucsd.edu/solr/blacklight/ collections_tesim:bb8605360n
on production: https://calisphere.org/collections/27323/
on -stage: https://calisphere-stage.cdlib.org/collections/27323/

Note: Example of a record that is part of both sets: https://library.ucsd.edu/dc/object/bb1716281n
1 duplicate
26176 http://lib-metadata.ucsd.edu/solr/blacklight/ collections_tesim:bb13322220

on production: https://calisphere.org/collections/26176/
on -stage: https://calisphere-stage.cdlib.org/collections/26176/
26900 http://lib-metadata.ucsd.edu/solr/blacklight/ collections_tesim:bb8228040z
on production: https://calisphere.org/collections/26900/
on -stage: https://calisphere-stage.cdlib.org/collections/26900/

To do:

[ ] I think this is an issue with records being part of multiple sets. Let's discuss.

@christinklez Can you try re-running the create_index_stage task for collection 28121?

@barbarahui Re-ran only the create_stage_index task, resulting in a red square with this message:
https://7a8067cb-3b99-477e-a883-7e311175a9b4.c3.us-west-2.airflow.amazonaws.com/log?dag_id=harvest_collection&task_id=create_stage_index&execution_date=2024-05-09T20%3A25%3A53%2B00%3A00
[2024-05-09, 22:02:44 UTC] {{logging_mixin.py:150}} INFO - deleted records with collection_id `28121` from index `rikolti-stg-combined-20240508151215`
[2024-05-09, 22:02:45 UTC] {{logging_mixin.py:150}} INFO - WARNING - document already exists; not creating.
 {'create': {'_index': 'rikolti-stg-combined-20240508151215', '_id': 'ark:/21198/zz000100hh', 'status': 409, 'error': {'type': 'version_conflict_engine_exception', 'reason': '[ark:/21198/zz000100hh]: version conflict, document already exists (current version [2])', 'index': 'rikolti-stg-combined-20240508151215', 'shard': '0', 'index_uuid': '11KZILPMsrT8eb863fOrk8-Q'}}}
[2024-05-09, 22:02:45 UTC] {{logging_mixin.py:150}} INFO - ('[ark:/21198/zz000100hh]: version conflict, document already exists (current '
 'version [2])')
[2024-05-09, 22:02:45 UTC] {{logging_mixin.py:150}} INFO - WARNING - document already exists; not creating.
 {'create': {'_index': 'rikolti-stg-combined-20240508151215', '_id': 'ark:/21198/zz000100j1', 'status': 409, 'error': {'type': 'version_conflict_engine_exception', 'reason': '[ark:/21198/zz000100j1]: version conflict, document already exists (current version [2])', 'index': 'rikolti-stg-combined-20240508151215', 'shard': '0', 'index_uuid': '11KZILPMsrT8eb863fOrk8-Q'}}}
[2024-05-09, 22:02:45 UTC] {{logging_mixin.py:150}} INFO - ('[ark:/21198/zz000100j1]: version conflict, document already exists (current '
 'version [2])')
[2024-05-09, 22:02:45 UTC] {{logging_mixin.py:150}} INFO - WARNING - document already exists; not creating.
 {'create': {'_index': 'rikolti-stg-combined-20240508151215', '_id': 'ark:/21198/zz0000ztdp', 'status': 409, 'error': {'type': 'version_conflict_engine_exception', 'reason': '[ark:/21198/zz0000ztdp]: version conflict, document already exists (current version [2])', 'index': 'rikolti-stg-combined-20240508151215', 'shard': '0', 'index_uuid': '11KZILPMsrT8eb863fOrk8-Q'}}}
[2024-05-09, 22:02:45 UTC] {{logging_mixin.py:150}} INFO - ('[ark:/21198/zz0000ztdp]: version conflict, document already exists (current '
 'version [2])')
[2024-05-09, 22:02:45 UTC] {{logging_mixin.py:150}} INFO - WARNING - document already exists; not creating.
 {'create': {'_index': 'rikolti-stg-combined-20240508151215', '_id': 'ark:/21198/zz0000zt6k', 'status': 409, 'error': {'type': 'version_conflict_engine_exception', 'reason': '[ark:/21198/zz0000zt6k]: version conflict, document already exists (current version [2])', 'index': 'rikolti-stg-combined-20240508151215', 'shard': '0', 'index_uuid': '11KZILPMsrT8eb863fOrk8-Q'}}}
[2024-05-09, 22:02:45 UTC] {{logging_mixin.py:150}} INFO - ('[ark:/21198/zz0000zt6k]: version conflict, document already exists (current '
 'version [2])')
[2024-05-09, 22:02:45 UTC] {{logging_mixin.py:150}} INFO - WARNING - document already exists; not creating.
 {'create': {'_index': 'rikolti-stg-combined-20240508151215', '_id': 'ark:/21198/zz0000zt73', 'status': 409, 'error': {'type': 'version_conflict_engine_exception', 'reason': '[ark:/21198/zz0000zt73]: version conflict, document already exists (current version [2])', 'index': 'rikolti-stg-combined-20240508151215', 'shard': '0', 'index_uuid': '11KZILPMsrT8eb863fOrk8-Q'}}}
[2024-05-09, 22:02:45 UTC] {{logging_mixin.py:150}} INFO - ('[ark:/21198/zz0000zt73]: version conflict, document already exists (current '
 'version [2])')
[2024-05-09, 22:02:45 UTC] {{logging_mixin.py:150}} INFO - WARNING - document already exists; not creating.
 {'create': {'_index': 'rikolti-stg-combined-20240508151215', '_id': 'ark:/21198/zz0000ztqb', 'status': 409, 'error': {'type': 'version_conflict_engine_exception', 'reason': '[ark:/21198/zz0000ztqb]: version conflict, document already exists (current version [2])', 'index': 'rikolti-stg-combined-20240508151215', 'shard': '0', 'index_uuid': '11KZILPMsrT8eb863fOrk8-Q'}}}
[2024-05-09, 22:02:45 UTC] {{logging_mixin.py:150}} INFO - ('[ark:/21198/zz0000ztqb]: version conflict, document already exists (current '
 'version [2])')
[2024-05-09, 22:02:45 UTC] {{logging_mixin.py:150}} INFO - WARNING - document already exists; not creating.
 {'create': {'_index': 'rikolti-stg-combined-20240508151215', '_id': 'ark:/21198/zz0000ztrv', 'status': 409, 'error': {'type': 'version_conflict_engine_exception', 'reason': '[ark:/21198/zz0000ztrv]: version conflict, document already exists (current version [2])', 'index': 'rikolti-stg-combined-20240508151215', 'shard': '0', 'index_uuid': '11KZILPMsrT8eb863fOrk8-Q'}}}
[2024-05-09, 22:02:45 UTC] {{logging_mixin.py:150}} INFO - ('[ark:/21198/zz0000ztrv]: version conflict, document already exists (current '
 'version [2])')
[2024-05-09, 22:02:45 UTC] {{logging_mixin.py:150}} INFO - WARNING - document already exists; not creating.
 {'create': {'_index': 'rikolti-stg-combined-20240508151215', '_id': 'ark:/21198/zz0000ztpt', 'status': 409, 'error': {'type': 'version_conflict_engine_exception', 'reason': '[ark:/21198/zz0000ztpt]: version conflict, document already exists (current version [2])', 'index': 'rikolti-stg-combined-20240508151215', 'shard': '0', 'index_uuid': '11KZILPMsrT8eb863fOrk8-Q'}}}
[2024-05-09, 22:02:45 UTC] {{logging_mixin.py:150}} INFO - ('[ark:/21198/zz0000ztpt]: version conflict, document already exists (current '
 'version [2])')
[2024-05-09, 22:02:45 UTC] {{logging_mixin.py:150}} INFO - WARNING - document already exists; not creating.
 {'create': {'_index': 'rikolti-stg-combined-20240508151215', '_id': 'ark:/21198/zz0000zt94', 'status': 409, 'error': {'type': 'version_conflict_engine_exception', 'reason': '[ark:/21198/zz0000zt94]: version conflict, document already exists (current version [2])', 'index': 'rikolti-stg-combined-20240508151215', 'shard': '0', 'index_uuid': '11KZILPMsrT8eb863fOrk8-Q'}}}
[2024-05-09, 22:02:45 UTC] {{logging_mixin.py:150}} INFO - ('[ark:/21198/zz0000zt94]: version conflict, document already exists (current '
 'version [2])')
[2024-05-09, 22:02:45 UTC] {{logging_mixin.py:150}} INFO - WARNING - document already exists; not creating.
 {'create': {'_index': 'rikolti-stg-combined-20240508151215', '_id': 'ark:/21198/zz0000zt8m', 'status': 409, 'error': {'type': 'version_conflict_engine_exception', 'reason': '[ark:/21198/zz0000zt8m]: version conflict, document already exists (current version [2])', 'index': 'rikolti-stg-combined-20240508151215', 'shard': '0', 'index_uuid': '11KZILPMsrT8eb863fOrk8-Q'}}}
[2024-05-09, 22:02:45 UTC] {{logging_mixin.py:150}} INFO - ('[ark:/21198/zz0000zt8m]: version conflict, document already exists (current '
 'version [2])')
[2024-05-09, 22:02:45 UTC] {{logging_mixin.py:150}} INFO - WARNING - document already exists; not creating.
 {'create': {'_index': 'rikolti-stg-combined-20240508151215', '_id': 'ark:/21198/zz0000zv1g', 'status': 409, 'error': {'type': 'version_conflict_engine_exception', 'reason': '[ark:/21198/zz0000zv1g]: version conflict, document already exists (current version [2])', 'index': 'rikolti-stg-combined-20240508151215', 'shard': '0', 'index_uuid': '11KZILPMsrT8eb863fOrk8-Q'}}}
[2024-05-09, 22:02:45 UTC] {{logging_mixin.py:150}} INFO - ('[ark:/21198/zz0000zv1g]: version conflict, document already exists (current '
 'version [2])')
[2024-05-09, 22:02:45 UTC] {{logging_mixin.py:150}} INFO - WARNING - document already exists; not creating.
 {'create': {'_index': 'rikolti-stg-combined-20240508151215', '_id': 'ark:/21198/zz0000ztsc', 'status': 409, 'error': {'type': 'version_conflict_engine_exception', 'reason': '[ark:/21198/zz0000ztsc]: version conflict, document already exists (current version [2])', 'index': 'rikolti-stg-combined-20240508151215', 'shard': '0', 'index_uuid': '11KZILPMsrT8eb863fOrk8-Q'}}}
[2024-05-09, 22:02:45 UTC] {{logging_mixin.py:150}} INFO - ('[ark:/21198/zz0000ztsc]: version conflict, document already exists (current '
 'version [2])')
[2024-05-09, 22:02:45 UTC] {{logging_mixin.py:150}} INFO - WARNING - document already exists; not creating.
 {'create': {'_index': 'rikolti-stg-combined-20240508151215', '_id': 'ark:/21198/zz0000zt52', 'status': 409, 'error': {'type': 'version_conflict_engine_exception', 'reason': '[ark:/21198/zz0000zt52]: version conflict, document already exists (current version [2])', 'index': 'rikolti-stg-combined-20240508151215', 'shard': '0', 'index_uuid': '11KZILPMsrT8eb863fOrk8-Q'}}}
[2024-05-09, 22:02:45 UTC] {{logging_mixin.py:150}} INFO - ('[ark:/21198/zz0000zt52]: version conflict, document already exists (current '
 'version [2])')
[2024-05-09, 22:02:45 UTC] {{taskinstance.py:1824}} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/decorators/base.py", line 220, in execute
    return_value = super().execute(context)
  File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/operators/python.py", line 181, in execute
    return_value = self.execute_callable()
  File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/operators/python.py", line 198, in execute_callable
    return self.python_callable(*self.op_args, **self.op_kwargs)
  File "/usr/local/airflow/dags/rikolti/dags/shared_tasks/indexing_tasks.py", line 61, in update_stage_index_for_collection_task
    raise e
  File "/usr/local/airflow/dags/rikolti/dags/shared_tasks/indexing_tasks.py", line 58, in update_stage_index_for_collection_task
    update_stage_index_for_collection(collection_id, version_pages)
  File "/usr/local/airflow/dags/rikolti/record_indexer/update_stage_index.py", line 17, in update_stage_index_for_collection
    add_page(version_page, index)
  File "/usr/local/airflow/dags/rikolti/record_indexer/add_page_to_index.py", line 105, in add_page
    bulk_add(records, index)
  File "/usr/local/airflow/dags/rikolti/record_indexer/add_page_to_index.py", line 43, in bulk_add
    raise(
Exception: 0 errors in bulk indexing 25 records: []
[2024-05-09, 22:02:45 UTC] {{taskinstance.py:1345}} INFO - Marking task as FAILED. dag_id=harvest_collection, task_id=create_stage_index, execution_date=20240509T202553, start_date=20240509T220243, end_date=20240509T220245
[2024-05-09, 22:02:45 UTC] {{logging_mixin.py:150}} INFO - Message sent to SNS with Message ID: fe9a5ca7-3486-51a0-9307-02107ec3ec29
[2024-05-09, 22:02:45 UTC] {{standard_task_runner.py:104}} ERROR - Failed to execute job 129040 for task create_stage_index (0 errors in bulk indexing 25 records: []; 187)
[2024-05-09, 22:02:45 UTC] {{local_task_job_runner.py:225}} INFO - Task exited with return code 1
[2024-05-09, 22:02:45 UTC] {{taskinstance.py:2653}} INFO - 0 downstream tasks scheduled from follow-on schedule check

@christinklez can you try one more time?

@christinklez I'm going ahead and trying it to verify the fixes are good :)

Done! Green squares, looks like it went through!
Here's what the log says, just as an fyi:
[2024-05-09, 22:30:27 UTC] {{logging_mixin.py:150}} INFO - WARNING - document already exists; not creating.
 {'create': {'_index': 'rikolti-stg-combined-20240508151215', '_id': 'ark:/21198/zz000100hh', 'status': 409, 'error': {'type': 'version_conflict_engine_exception', 'reason': '[ark:/21198/zz000100hh]: version conflict, document already exists (current version [2])', 'index': 'rikolti-stg-combined-20240508151215', 'shard': '0', 'index_uuid': '11KZILPMsrT8eb863fOrk8-Q'}}}
[2024-05-09, 22:30:27 UTC] {{logging_mixin.py:150}} INFO - WARNING - document already exists; not creating.
 {'create': {'_index': 'rikolti-stg-combined-20240508151215', '_id': 'ark:/21198/zz000100j1', 'status': 409, 'error': {'type': 'version_conflict_engine_exception', 'reason': '[ark:/21198/zz000100j1]: version conflict, document already exists (current version [2])', 'index': 'rikolti-stg-combined-20240508151215', 'shard': '0', 'index_uuid': '11KZILPMsrT8eb863fOrk8-Q'}}}
[2024-05-09, 22:30:27 UTC] {{logging_mixin.py:150}} INFO - WARNING - document already exists; not creating.
 {'create': {'_index': 'rikolti-stg-combined-20240508151215', '_id': 'ark:/21198/zz0000ztdp', 'status': 409, 'error': {'type': 'version_conflict_engine_exception', 'reason': '[ark:/21198/zz0000ztdp]: version conflict, document already exists (current version [2])', 'index': 'rikolti-stg-combined-20240508151215', 'shard': '0', 'index_uuid': '11KZILPMsrT8eb863fOrk8-Q'}}}
[2024-05-09, 22:30:27 UTC] {{logging_mixin.py:150}} INFO - WARNING - document already exists; not creating.
 {'create': {'_index': 'rikolti-stg-combined-20240508151215', '_id': 'ark:/21198/zz0000zt6k', 'status': 409, 'error': {'type': 'version_conflict_engine_exception', 'reason': '[ark:/21198/zz0000zt6k]: version conflict, document already exists (current version [2])', 'index': 'rikolti-stg-combined-20240508151215', 'shard': '0', 'index_uuid': '11KZILPMsrT8eb863fOrk8-Q'}}}
[2024-05-09, 22:30:27 UTC] {{logging_mixin.py:150}} INFO - WARNING - document already exists; not creating.
 {'create': {'_index': 'rikolti-stg-combined-20240508151215', '_id': 'ark:/21198/zz0000zt73', 'status': 409, 'error': {'type': 'version_conflict_engine_exception', 'reason': '[ark:/21198/zz0000zt73]: version conflict, document already exists (current version [2])', 'index': 'rikolti-stg-combined-20240508151215', 'shard': '0', 'index_uuid': '11KZILPMsrT8eb863fOrk8-Q'}}}
[2024-05-09, 22:30:27 UTC] {{logging_mixin.py:150}} INFO - WARNING - document already exists; not creating.
 {'create': {'_index': 'rikolti-stg-combined-20240508151215', '_id': 'ark:/21198/zz0000ztqb', 'status': 409, 'error': {'type': 'version_conflict_engine_exception', 'reason': '[ark:/21198/zz0000ztqb]: version conflict, document already exists (current version [2])', 'index': 'rikolti-stg-combined-20240508151215', 'shard': '0', 'index_uuid': '11KZILPMsrT8eb863fOrk8-Q'}}}
[2024-05-09, 22:30:27 UTC] {{logging_mixin.py:150}} INFO - WARNING - document already exists; not creating.
 {'create': {'_index': 'rikolti-stg-combined-20240508151215', '_id': 'ark:/21198/zz0000ztrv', 'status': 409, 'error': {'type': 'version_conflict_engine_exception', 'reason': '[ark:/21198/zz0000ztrv]: version conflict, document already exists (current version [2])', 'index': 'rikolti-stg-combined-20240508151215', 'shard': '0', 'index_uuid': '11KZILPMsrT8eb863fOrk8-Q'}}}
[2024-05-09, 22:30:27 UTC] {{logging_mixin.py:150}} INFO - WARNING - document already exists; not creating.
 {'create': {'_index': 'rikolti-stg-combined-20240508151215', '_id': 'ark:/21198/zz0000ztpt', 'status': 409, 'error': {'type': 'version_conflict_engine_exception', 'reason': '[ark:/21198/zz0000ztpt]: version conflict, document already exists (current version [2])', 'index': 'rikolti-stg-combined-20240508151215', 'shard': '0', 'index_uuid': '11KZILPMsrT8eb863fOrk8-Q'}}}
[2024-05-09, 22:30:27 UTC] {{logging_mixin.py:150}} INFO - WARNING - document already exists; not creating.
 {'create': {'_index': 'rikolti-stg-combined-20240508151215', '_id': 'ark:/21198/zz0000zt94', 'status': 409, 'error': {'type': 'version_conflict_engine_exception', 'reason': '[ark:/21198/zz0000zt94]: version conflict, document already exists (current version [2])', 'index': 'rikolti-stg-combined-20240508151215', 'shard': '0', 'index_uuid': '11KZILPMsrT8eb863fOrk8-Q'}}}
[2024-05-09, 22:30:27 UTC] {{logging_mixin.py:150}} INFO - WARNING - document already exists; not creating.
 {'create': {'_index': 'rikolti-stg-combined-20240508151215', '_id': 'ark:/21198/zz0000zt8m', 'status': 409, 'error': {'type': 'version_conflict_engine_exception', 'reason': '[ark:/21198/zz0000zt8m]: version conflict, document already exists (current version [2])', 'index': 'rikolti-stg-combined-20240508151215', 'shard': '0', 'index_uuid': '11KZILPMsrT8eb863fOrk8-Q'}}}
[2024-05-09, 22:30:27 UTC] {{logging_mixin.py:150}} INFO - WARNING - document already exists; not creating.
 {'create': {'_index': 'rikolti-stg-combined-20240508151215', '_id': 'ark:/21198/zz0000zv1g', 'status': 409, 'error': {'type': 'version_conflict_engine_exception', 'reason': '[ark:/21198/zz0000zv1g]: version conflict, document already exists (current version [2])', 'index': 'rikolti-stg-combined-20240508151215', 'shard': '0', 'index_uuid': '11KZILPMsrT8eb863fOrk8-Q'}}}
[2024-05-09, 22:30:27 UTC] {{logging_mixin.py:150}} INFO - WARNING - document already exists; not creating.
 {'create': {'_index': 'rikolti-stg-combined-20240508151215', '_id': 'ark:/21198/zz0000ztsc', 'status': 409, 'error': {'type': 'version_conflict_engine_exception', 'reason': '[ark:/21198/zz0000ztsc]: version conflict, document already exists (current version [2])', 'index': 'rikolti-stg-combined-20240508151215', 'shard': '0', 'index_uuid': '11KZILPMsrT8eb863fOrk8-Q'}}}
[2024-05-09, 22:30:27 UTC] {{logging_mixin.py:150}} INFO - WARNING - document already exists; not creating.
 {'create': {'_index': 'rikolti-stg-combined-20240508151215', '_id': 'ark:/21198/zz0000zt52', 'status': 409, 'error': {'type': 'version_conflict_engine_exception', 'reason': '[ark:/21198/zz0000zt52]: version conflict, document already exists (current version [2])', 'index': 'rikolti-stg-combined-20240508151215', 'shard': '0', 'index_uuid': '11KZILPMsrT8eb863fOrk8-Q'}}}
[2024-05-09, 22:30:27 UTC] {{logging_mixin.py:150}} INFO - added 25 records to index `rikolti-stg-combined-20240508151215` from page `28121/vernacular_metadata_2024-05-09T20:26:04/mapped_metadata_2024-05-09T20:26:19/with_content_urls_2024-05-09T20:26:36/data/0.jsonl`
[2024-05-09, 22:30:27 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had is_shown_at field removed
[2024-05-09, 22:30:27 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had is_shown_by field removed
[2024-05-09, 22:30:27 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had item_count field removed
[2024-05-09, 22:30:27 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had media_source field removed
[2024-05-09, 22:30:27 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had thumbnail_source field removed
[2024-05-09, 22:30:27 UTC] {{logging_mixin.py:150}} INFO - added 25 records to index `rikolti-stg-combined-20240508151215` from page `28121/vernacular_metadata_2024-05-09T20:26:04/mapped_metadata_2024-05-09T20:26:19/with_content_urls_2024-05-09T20:26:36/data/1.jsonl`
[2024-05-09, 22:30:27 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had is_shown_at field removed
[2024-05-09, 22:30:27 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had is_shown_by field removed
[2024-05-09, 22:30:27 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had item_count field removed
[2024-05-09, 22:30:27 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had media_source field removed
[2024-05-09, 22:30:27 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had thumbnail_source field removed
[2024-05-09, 22:30:28 UTC] {{logging_mixin.py:150}} INFO - WARNING - document already exists; not creating.
 {'create': {'_index': 'rikolti-stg-combined-20240508151215', '_id': 'ark:/21198/n1b317', 'status': 409, 'error': {'type': 'version_conflict_engine_exception', 'reason': '[ark:/21198/n1b317]: version conflict, document already exists (current version [2])', 'index': 'rikolti-stg-combined-20240508151215', 'shard': '0', 'index_uuid': '11KZILPMsrT8eb863fOrk8-Q'}}}
[2024-05-09, 22:30:28 UTC] {{logging_mixin.py:150}} INFO - WARNING - document already exists; not creating.
 {'create': {'_index': 'rikolti-stg-combined-20240508151215', '_id': 'ark:/21198/n1fs6z', 'status': 409, 'error': {'type': 'version_conflict_engine_exception', 'reason': '[ark:/21198/n1fs6z]: version conflict, document already exists (current version [2])', 'index': 'rikolti-stg-combined-20240508151215', 'shard': '0', 'index_uuid': '11KZILPMsrT8eb863fOrk8-Q'}}}
[2024-05-09, 22:30:28 UTC] {{logging_mixin.py:150}} INFO - WARNING - document already exists; not creating.
 {'create': {'_index': 'rikolti-stg-combined-20240508151215', '_id': 'ark:/21198/n16c9s', 'status': 409, 'error': {'type': 'version_conflict_engine_exception', 'reason': '[ark:/21198/n16c9s]: version conflict, document already exists (current version [2])', 'index': 'rikolti-stg-combined-20240508151215', 'shard': '0', 'index_uuid': '11KZILPMsrT8eb863fOrk8-Q'}}}
[2024-05-09, 22:30:28 UTC] {{logging_mixin.py:150}} INFO - WARNING - document already exists; not creating.
 {'create': {'_index': 'rikolti-stg-combined-20240508151215', '_id': 'ark:/21198/n1xw4k', 'status': 409, 'error': {'type': 'version_conflict_engine_exception', 'reason': '[ark:/21198/n1xw4k]: version conflict, document already exists (current version [2])', 'index': 'rikolti-stg-combined-20240508151215', 'shard': '0', 'index_uuid': '11KZILPMsrT8eb863fOrk8-Q'}}}
[2024-05-09, 22:30:28 UTC] {{logging_mixin.py:150}} INFO - WARNING - document already exists; not creating.
 {'create': {'_index': 'rikolti-stg-combined-20240508151215', '_id': 'ark:/21198/n12k6g', 'status': 409, 'error': {'type': 'version_conflict_engine_exception', 'reason': '[ark:/21198/n12k6g]: version conflict, document already exists (current version [2])', 'index': 'rikolti-stg-combined-20240508151215', 'shard': '0', 'index_uuid': '11KZILPMsrT8eb863fOrk8-Q'}}}
[2024-05-09, 22:30:28 UTC] {{logging_mixin.py:150}} INFO - added 9 records to index `rikolti-stg-combined-20240508151215` from page `28121/vernacular_metadata_2024-05-09T20:26:04/mapped_metadata_2024-05-09T20:26:19/with_content_urls_2024-05-09T20:26:36/data/10.jsonl`
[2024-05-09, 22:30:28 UTC] {{logging_mixin.py:150}} INFO -     all 9 records had is_shown_at field removed
[2024-05-09, 22:30:28 UTC] {{logging_mixin.py:150}} INFO -     all 9 records had is_shown_by field removed
[2024-05-09, 22:30:28 UTC] {{logging_mixin.py:150}} INFO -     all 9 records had item_count field removed
[2024-05-09, 22:30:28 UTC] {{logging_mixin.py:150}} INFO -     all 9 records had media_source field removed
[2024-05-09, 22:30:28 UTC] {{logging_mixin.py:150}} INFO -     all 9 records had thumbnail_source field removed
[2024-05-09, 22:30:29 UTC] {{logging_mixin.py:150}} INFO - added 25 records to index `rikolti-stg-combined-20240508151215` from page `28121/vernacular_metadata_2024-05-09T20:26:04/mapped_metadata_2024-05-09T20:26:19/with_content_urls_2024-05-09T20:26:36/data/2.jsonl`
[2024-05-09, 22:30:29 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had is_shown_at field removed
[2024-05-09, 22:30:29 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had is_shown_by field removed
[2024-05-09, 22:30:29 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had item_count field removed
[2024-05-09, 22:30:29 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had media_source field removed
[2024-05-09, 22:30:29 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had thumbnail_source field removed
[2024-05-09, 22:30:29 UTC] {{logging_mixin.py:150}} INFO - added 25 records to index `rikolti-stg-combined-20240508151215` from page `28121/vernacular_metadata_2024-05-09T20:26:04/mapped_metadata_2024-05-09T20:26:19/with_content_urls_2024-05-09T20:26:36/data/3.jsonl`
[2024-05-09, 22:30:29 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had is_shown_at field removed
[2024-05-09, 22:30:29 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had is_shown_by field removed
[2024-05-09, 22:30:29 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had item_count field removed
[2024-05-09, 22:30:29 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had media_source field removed
[2024-05-09, 22:30:29 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had thumbnail_source field removed
[2024-05-09, 22:30:30 UTC] {{logging_mixin.py:150}} INFO - added 25 records to index `rikolti-stg-combined-20240508151215` from page `28121/vernacular_metadata_2024-05-09T20:26:04/mapped_metadata_2024-05-09T20:26:19/with_content_urls_2024-05-09T20:26:36/data/4.jsonl`
[2024-05-09, 22:30:30 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had is_shown_at field removed
[2024-05-09, 22:30:30 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had is_shown_by field removed
[2024-05-09, 22:30:30 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had item_count field removed
[2024-05-09, 22:30:30 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had media_source field removed
[2024-05-09, 22:30:30 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had thumbnail_source field removed
[2024-05-09, 22:30:31 UTC] {{logging_mixin.py:150}} INFO - added 25 records to index `rikolti-stg-combined-20240508151215` from page `28121/vernacular_metadata_2024-05-09T20:26:04/mapped_metadata_2024-05-09T20:26:19/with_content_urls_2024-05-09T20:26:36/data/5.jsonl`
[2024-05-09, 22:30:31 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had is_shown_at field removed
[2024-05-09, 22:30:31 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had is_shown_by field removed
[2024-05-09, 22:30:31 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had item_count field removed
[2024-05-09, 22:30:31 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had media_source field removed
[2024-05-09, 22:30:31 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had thumbnail_source field removed
[2024-05-09, 22:30:31 UTC] {{logging_mixin.py:150}} INFO - added 25 records to index `rikolti-stg-combined-20240508151215` from page `28121/vernacular_metadata_2024-05-09T20:26:04/mapped_metadata_2024-05-09T20:26:19/with_content_urls_2024-05-09T20:26:36/data/6.jsonl`
[2024-05-09, 22:30:31 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had is_shown_at field removed
[2024-05-09, 22:30:31 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had is_shown_by field removed
[2024-05-09, 22:30:31 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had item_count field removed
[2024-05-09, 22:30:31 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had media_source field removed
[2024-05-09, 22:30:31 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had thumbnail_source field removed
[2024-05-09, 22:30:32 UTC] {{logging_mixin.py:150}} INFO - added 25 records to index `rikolti-stg-combined-20240508151215` from page `28121/vernacular_metadata_2024-05-09T20:26:04/mapped_metadata_2024-05-09T20:26:19/with_content_urls_2024-05-09T20:26:36/data/7.jsonl`
[2024-05-09, 22:30:32 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had is_shown_at field removed
[2024-05-09, 22:30:32 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had is_shown_by field removed
[2024-05-09, 22:30:32 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had item_count field removed
[2024-05-09, 22:30:32 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had media_source field removed
[2024-05-09, 22:30:32 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had thumbnail_source field removed
[2024-05-09, 22:30:32 UTC] {{logging_mixin.py:150}} INFO - added 25 records to index `rikolti-stg-combined-20240508151215` from page `28121/vernacular_metadata_2024-05-09T20:26:04/mapped_metadata_2024-05-09T20:26:19/with_content_urls_2024-05-09T20:26:36/data/8.jsonl`
[2024-05-09, 22:30:32 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had is_shown_at field removed
[2024-05-09, 22:30:32 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had is_shown_by field removed
[2024-05-09, 22:30:32 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had item_count field removed
[2024-05-09, 22:30:32 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had media_source field removed
[2024-05-09, 22:30:32 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had thumbnail_source field removed
[2024-05-09, 22:30:33 UTC] {{logging_mixin.py:150}} INFO - added 25 records to index `rikolti-stg-combined-20240508151215` from page `28121/vernacular_metadata_2024-05-09T20:26:04/mapped_metadata_2024-05-09T20:26:19/with_content_urls_2024-05-09T20:26:36/data/9.jsonl`
[2024-05-09, 22:30:33 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had is_shown_at field removed
[2024-05-09, 22:30:33 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had is_shown_by field removed
[2024-05-09, 22:30:33 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had item_count field removed
[2024-05-09, 22:30:33 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had media_source field removed
[2024-05-09, 22:30:33 UTC] {{logging_mixin.py:150}} INFO -     all 25 records had thumbnail_source field removed
[2024-05-09, 22:30:33 UTC] {{logging_mixin.py:150}} INFO -

Review indexed records at: https://rikolti-data.s3.us-west-2.amazonaws.com/index.html#28121/vernacular_metadata_2024-05-09T20:26:04/mapped_metadata_2024-05-09T20:26:19/with_content_urls_2024-05-09T20:26:36/data/

Or on opensearch at: https://search-rikolti-2-xxbcriyfw5iqysaj7p3fhhscae.us-west-2.es.amazonaws.com/_dashboards/app/dev_tools#/console with query:
{
  "query": {
    "bool": {
      "filter": {
        "terms": {
          "collection_url": [
            28121
          ]
        }
      }
    }
  }
}
[2024-05-09, 22:30:33 UTC] {{logging_mixin.py:150}} INFO - Message sent to SNS with Message ID: e0d91176-e06c-5930-ad14-4fc1b0a5ac3f
[2024-05-09, 22:30:33 UTC] {{python.py:183}} INFO - Done. Returned value was: None
[2024-05-09, 22:30:33 UTC] {{taskinstance.py:1345}} INFO - Marking task as SUCCESS. dag_id=harvest_collection, task_id=create_stage_index, execution_date=20240509T202553, start_date=20240509T223025, end_date=20240509T223033
[2024-05-09, 22:30:33 UTC] {{local_task_job_runner.py:225}} INFO - Task exited with return code 0
[2024-05-09, 22:30:33 UTC] {{taskinstance.py:2653}} INFO - 0 downstream tasks scheduled from follow-on schedule check


CDN v3.3.6 cross-origin request blocked

Hello
Getting above error during the last few days. Did work well before that.
This is the code, just in case:
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/js/bootstrap..js" integrity="sha384-0mSbJDEHialfmuBBQP6A4Qrprq5OVfW37PRR3j5ELqxss1yVqOtnepnHVP9aJ7xS" crossorigin="anonymous"></script>
If I, instead, place  the CDN code for version 3.3.5 it works perfectly fine.
See enclosure.
Regards
Bernard

Thanks for reporting. I'll take a look.

I did notice that you have two periods in the path, is that a reporting typo, or is that what you have in the actual code?
https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/js/bootstrap..js

Additionally, I noticed that you have "sha384", but we provide "256" and "512" on the CDN, mind trying:
<link href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/css/bootstrap.min.css" rel="stylesheet" integrity="sha256-7s5uDGW3AHqw6xtJmNNtr+OBRJUlgkNJEo78P4b0yRw= sha512-nNo+yCHEyn0smMxSswnf/OnX6/KwJuZTlNZBjauKhTK0c+zT+q5JOCx0UFhXQ6rJR9jg6Es8gPuD2uZcYDLqSw==" crossorigin="anonymous">

Hello
Thank you for your prompt reply.
The 2 periods were in the actual code, which I promise you I copied from
your download page a few months ago and it was actually working till
recently.
But then, of course, it stopped working.
I did correct the code, ie, removed the additional period at it is working
now.
Thanks again.
Bernard
On Sat, Feb 20, 2016 at 12:14 AM, Joshua Mervine<EMAIL_ADDRESS>wrote:

Additionally, I noticed that you have "sha384", but we provide "256" and
"512" on the CDN, mind trying:

—
Reply to this email directly or view it on GitHub
https://github.com/MaxCDN/bootstrap-cdn/issues/637#issuecomment-186463015
.

Awesome! Let us know if you have any more issues. :)


m.2 ssd is working?

I'm planning on getting a m.2 ssd for the t450, is it working (having a m.2ssd and sata ssd at the same time)?

sergio scar

Yes. I currently have an m.2 and sata ssd plugged in. m.2 has Big Sur, and sata ssd has Windows 10. Works fine.

Nice! Thanks!


[bug] mod-1-9 stopped working

I'm not sure when it appeared, but I cannot use mod-[digit] anymore to open a note.

Sorry, my mistake, it works fine.


[Android] Set the command line to webkit Preference

The command of "--allow-running-insecure-content" in assets directory do not effect the Webkit behavior because not set it to Webkit preference.

@huningxin @junmin-zhu @lincsoon @alexshalamov PTAL.

The compiled error for Android-X86 is not cause by this commit.

@halton PTAL, thanks.

@darktears PTAL, thanks.


Both Verbatim and Suggested Place of birth fields to be displayed in the Search Record Details

FreeCEN policy is ' as is’. Therefore we must display the verbatim fields. The work put in on the alternative fields demands that these are displayed where they are not null. I can see 2 options:
Display them in 2 new display fields. In FC1 we used the term ‘validators choice’. The term is not critical. It could be validator’s suggestion or ‘suggested POB’ or something like that.
Display them as ‘suggested POB’ in the notes field concatenated with any existing notes.

I know we agreed on the suggestion from @AlOneill however looking at the code will require a significant rewrite which is increasing the estimate to 13.
It is also important to note that the CSV entry has always included both fields.

Fond a better way. Deployed to test3

deployed


feat: Vulkan-Headers 1.1.130-beta1 (it is headers-only library)

Dear BCR team. My second pull request :)
Give ma a tip please. This is headers-only library and it is potentially platform-agnostic. Does not require build at all (only bundling and configuring includes).
Currently I specified in matrix.platforms huge list. As I understand this is needed to just check it on every possible platform. But what is best practice for such libraries? Maybe it is just too much to specify so many platforms and enough to have only one?

Hello @bazelbuild/bcr-maintainers, modules without existing maintainers (vulkan_headers) have been updated in this PR. Please review the changes.

And one more question. Vulkan-Headers requires to provide license its mixed MIT + Apache 2.0. How to deal with that?

Your contribution here is assumed to be under Apache 2.0 if not stated otherwise by you. If you add different license headers to your created Bazel build files you can change this situation (sometimes I do this when I copy Bazel build files from other projects, that have a different license). Since you are only referencing a plain URL (https://github.com/KhronosGroup/Vulkan-Headers/archive/refs/tags/v1.1.130.tar.gz) and not copying the content of the archive it does not matter under which license the files in the referenced archive are. Of course, if someone is using your module to build some code that makes use of Vulkan headers the corresponding software has to follow the MIT license - e.g. stating that it used Vulkan Header which are provided under MIT. Resolving license issues automatically via Bazel is an unsolved problem since Bazel exists (take a look at rules_license) - there is specialized software that can help you to clear copyright issues and I would also suggest a manual inspection.


Created context object and refactored App to be a Context Provider

 Create React context for current item on page


Add react dev tools and hook up redux dev tools

As per conversation on gitter I've added the react dev tools and tweaked the redux dev tools to pick up the store.
An issue I ran into relates to permissions whilst using electron-devtools-installer, It was reported as closed in another issue but I found that the issue continued to persist for me so I followed the advice in the link. Essentially the react devtools extension on my mac possibly on other platforms required me to chmod 777 the path of the extension, I don't know how much of an issue this maybe for others if it is then it might be worth looking into the standalone app, like react-devtools or simply add the advice re chmod to the dev wiki whilst electron-installer work on the issue hopefully

Awesome, thanks for getting this working again, @Akin909 ! This will make debugging so much easier 👍

I don't know how much of an issue this maybe for others if it is then it might be worth looking into the standalone app, like react-devtools or simply add the advice re chmod to the dev wiki whilst electron-installer work on the issue hopefully

I think, since it's one-time step, adding it to the wiki is probably simpler and sufficient. I'm cool with either approach though.

Cool @bryphe I personally would find it really frustrating to have to switch between a standalone app and electron. I can add a section in the wiki explaining what to do if it comes up for anyone else

Cool @bryphe I personally would find it really frustrating to have to switch between a standalone app and electron.

Ah, I think I skimmed the react-devtools repo too fast - I thought it was an embedded UI, just outside the devtools. I agree, having it be standalone would be painful.

Added the following to the docs

Note: some users may have issues with the react-devtools due to OS permissions.
to resolve this follow these steps:

Navigate to chrome://extensions in your browser, and find its extension ID, which is a hash string like fmkadmapgofadopljbjfkapdkoienihi.

Find out filesystem location used by Chrome for storing extensions:

on Windows it is %LOCALAPPDATA%\Google\Chrome\User Data\Default\Extensions;
on Linux it could be:
~/.config/google-chrome/Default/Extensions/
~/.config/google-chrome-beta/Default/Extensions/
~/.config/google-chrome-canary/Default/Extensions/
~/.config/chromium/Default/Extensions/
on macOS it is ~/Library/Application Support/Google/Chrome/Default/Extensions.

Once you have found the extension location you will need to grant the current user read and right permissions
using chmod 777 -R path/to/extension (N.B. the command recursively grants the current user access to the folder and its subfolders)

Sweet, thanks for taking this on @Akin909 ! Looks great!


zstd is a runtime dependency but is not documented in installation

After following doc/installation.md to install build and runtime dependencies before compiling enroot,I attempt to invoke enroot via the pyxis plugin for Slurm and get the following error. It appears that zstd is an enroot runtime dependency.
pyxis: importing docker image: alpine
slurmstepd: error: pyxis: child 11486 failed with error code: 1
slurmstepd: error: pyxis: failed to import docker image
slurmstepd: error: pyxis: printing enroot log file:
slurmstepd: error: pyxis:     [ERROR] Command not found: zstd
slurmstepd: error: pyxis: couldn't start container
slurmstepd: error: spank: required plugin spank_pyxis.so: task_init() failed with rc=-1
slurmstepd: error: Failed to invoke spank plugin stack
slurmstepd: error: pyxis: child 11499 failed with error code: 1

The problem goes away when I install the zstd package on Ubuntu 20.04. It appears to be the same package name in Debian and on RedHat derivatives. In fact, the list of runtime dependencies declared in the packaging files is considerably longer than in the documentation, although many are commonly installed by default:
Requires: bash >= 4.2, curl, gawk, jq >= 1.5, parallel, shadow-utils, squashfs-tools
Requires: coreutils, grep, findutils, gzip, glibc-common, sed, tar, util-linux, zstd

and
Depends: ${shlibs:Depends}, ${misc:Depends},
 bash (>= 4.2),
 curl,
 gawk | mawk,
 jq (>= 1.5),
 parallel,
 passwd,
 squashfs-tools,
 zstd,
 bsdmainutils,
``

Yeah I think it's just zstd missing from the doc


unable to install application: pueue

Sorry for the late reply.
This seems to be resolved by https://github.com/huangnauh/carrot/commit/2e20b0eac3f4f63e437bd8ccbeb61983bf202a7e


add more support for getThreadHistoryGraphQL

Because KevinSalmon doesn't merge my PR on his repo. I directly create this new PR.
As title of commits, I test whole my threads(8xx) and get many undiscover information. Finaly add more data from API response.

@bsansouci and I merged these changes in separately as part of our overhaul last week. Feel free to let us know if we missed something, but I think we got it all.
Thanks for the contribution :)


BugGuide name provider

Adds BugGuide as a potential name provider. @pleary, I made a slight change to the way ApiEndpointCache works (doesn't cache empty or failed responses). I think it's fine, but I wanted you to take a look before I merged it in.

this looks fine to me. ApiEndpointCache is really only useful when it successful caches stuff, and since you added the success? check, it should work OK

on second thought, maybe we could do something like:
if api_endpoint_cache
  api_endpoint_cache.update_attributes(
    request_completed_at: Time.now,
    success: !response.body.blank?,
    response: response.body
  )
end

which will set the request_completed_at, so we don't think there's an endless request going on. That will match what we're doing around https://github.com/inaturalist/inaturalist/blob/master/lib/meta_service.rb#L96


[DMR] v4.0.6 patch for slow ready handle diagnosis

What is changed and how it works?

Issue Number: Close #xxx
What's Changed:

Add more details to help the diagnosis of slow ready handle.

code changes
Related changes

PR to update pingcap/docs/pingcap/docs-cn:
Need to cherry-pick to the release branch

Check List
Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression

Consumes more CPU
Consumes more MEM

Breaking backward compatibility

Release note
None

@iosmanthus @cosven PTAL

@Connor1996 PTAL

LGTM

/test

/build

@Connor1996 help me merge this PR?

/merge

/run-build


LDAP:AUTH_USER_REGISTRATION NOT WORKING

i deployed Superser with Docker-Compose,and want to login in and authory with ldap.
Thus I install slapd on my Ubuntu Server,my superset_config.py like
#AUTHENTICATION FONFIG
104 AUTH_TYPE = AUTH_LDAP
105 AUHT_USER_REGISTRATION = True
106 AUTH_USER_REGISTRATION_ROLE = "Alpha"
107 AUTH_LDAP_UID_FIELD = "cn"
108 AUTH_LDAP_FIRSTNAME_FIELD = "givenName"
109 AHTU_LDAP_LASTNAME_FIELD= "sn"
110 AUTH_LDAP_EMAIL_FIELD = "mail"
112 AUTH_LDAP_SERVER = "ldap://xxxxxx:389"
114 AUTH_LDAP_SEARCH = "cn=%s,ou=groups,dc=xx,dc=xx"
With the above configuration,I can login wiht ldap user，but already in Superset.It is Seem that  `AUHT_USER_REGISTRATION = True' not working.Because when login with the user both in ldap and Superset, ldap can verify the password.
If the user not in Superset,it can not login.
Thank you from the bottom of my heart for your help or advices

User in Superset

User in ldap

i can login when use admin,test1 or test2, but test3 just failed.

ldap logs output like this when login with test1,but when login with test3,there is noting came out.

Closing this as stale since it's been silent for so long, and we're trying to steer toward a more actionable Issues backlog. If people are still encountering this in current versions (currently 3.x) please re-open this issue, open a new Issue with updated context, or raise a PR to address the problem. Thanks!


Find Pivot in The Array using Binary Search in C++

@nishant-ai Kindly assign me this under Hacktoberfest 2022.

Kindly assign me that issue
Thanks

Hello sir, Kindly assign it to me on that issue. I want to contribute to this issue.
Thank you.


FitMeUp Kanban Board setup

There are five people in the group @killy10o10, @muskan2532150, @John-Kibirige, @skyv26, @barakadanny. we split ourselves into the backend and front end.
Backend

@skyv26
@barakadanny
@John-Kibirige

Front end

@killy10o10
@muskan2532150

This is the link to the Kanban Board and the screenshot attached below:

Hi @killy10o10,
Wow, you did it 🎉

Congratulations! 🎉
To highlight

All tasks are pointed✔
All tasks have been assigned✔
Great job ✔

Optional suggestions
Every comment with the [OPTIONAL] prefix won't stop the approval of this PR. However, I strongly recommend you to take them into account as they can make your code better. Some of them were simply missed by the previous reviewer and addressing them will really improve your application.
Cheers and Happy coding!👏👏👏
Feel free to leave any questions or comments in the PR thread if something is not 100% clear.
Please, remember to tag me in your question so I can receive the notification.

As described in the Code reviews limits policy  you have a limited number of reviews per project (check the exact number in your Dashboard). If you think that the code review was not fair, you can request a second opinion using this form.


bxSlider with lazyloaded images aren't triggered on page load

Hi,
First of all, I'm sorry if it's not an issue directly regarding lazysizes as there are other scripts involved (mainly bxSlider as mentioned in the title).
I'd like to know if my problem can be solved at the lazysizes level or if I should go and knock on the door of the other libraries involved (any help is welcomed anyway :) )
I have a much larger case on a project, involving several slideshow and carousel that I have tried to sum up in  a test case on jsfiddle.
A vertical carousel of small images, that I need to convert to an horizontal carousel if the screen is smaller than 600px.
For this, I use bxSlider and enquire.js to reload my bxSlider with new parameters depending on the screen size.

Expected result : Vertical carousel for big screens
Expected result : Horizontal carousel for small screens
The result in Chrome : Vertical carousel for small screens, breaks everything

The problem here is that, on certain browsers (I'd say WebKit or Blink browsers), the bxSlider is not triggered on page load (probably because images aren't loaded at first) and as a consequence, if the screen size is below 600px, the carousel stays vertical because the bxSlider is never "reloaded" with the parameters contained in the enquire.js query.
The weird thing is that it works perfectly fine in Firefox.
And of course, if I don't lazyload images, it works fine everywhere.
So maybe it's more a problem with bxSlider and the way it handles preload ? I don't know, and I must say I don't know where to look anymore...
I would be grateful for any help solving this.
I hope my explainations are clear :/

@oilvier
Sorry, for taking so long to answer. This is clearly an issue with bxslider's reload method.
You initialize with vertical mode and without any information how many slides to show. This yields to the default, that the hole viewport is taken, after changing to the mobile mode the width of the items is set to something lot higher than you actually need, but bxslider operates with wrong values here.
Maybe it helps to fully destroy and re-init bxslider. If bxslider is written bad here, you manually  have to remove all inline styles.
Here is a "better" working fiddle:
https://fiddle.jshell.net/tas9fvp6/10/

No problem, I, myself, took a long time to answer back :)
After your post, I put Lazysizes out of the picture and I focused on debugging bxSlider.
In fact, it returns an error at load when there are images without "src", which my case due to lazysizes requirements.
I changed that to look for image without "src" and "data-srcset" and now it works fine !
Thanks for your time.


Anchors are not parsed via CLI

An example compose file:
services:
  hello-world:
    image: nginx
    env: &base
      DATABASE_USER: postgres
      DATABASE_PASSWORD: postgres
  redis:
    image: redis
    env: *base

x-uffizzi-ingress:
  service: hello-world
  port: 80

Result: Psych::BadAlias: Unknown alias: base

HOW TO TEST:
use compose file:
x-service: &base-service
    image: nginx

services:
  hello-world: *base-service

x-uffizzi-ingress:
  service: hello-world
  port: 80

Tested according to the check list

TESTED - OK
definitions:
    environment: &env-var
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres

services:
  web:
    image: uffizzitest/test_env:latest
    environment: *env-var
        # i.e., value should be added in the Uffizzi Dashboard

x-uffizzi:
  ingress:
    service: web
    port: 80

  continuous_preview:
    delete_preview_after: 1h

bash-5.1# bundle exec uffizzi preview create -o=json test-compose/uffizzi-compose-env-achor.yml
[✔] Creating containers...
[✔] Deploying preview...
  [✔] test_env
{"id":"deployment-9","url":"https://deployment-9-default.app.qa-gke.uffizzi.com"}
bash-5.1#

["POSTGRES_USER"]=> string(8) "postgres"
["POSTGRES_PASSWORD"]=> string(8) "postgres"

TESTED on production - OK
CLI + app.uffizzi.com
definitions:
    environment: &env-var
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres

services:
  web:
    image: uffizzitest/test_env:latest
    environment: *env-var

x-uffizzi:
  ingress:
    service: web
    port: 80

  continuous_preview:
    delete_preview_after: 1h

# uffizzi preview create -o=json compose.yml
[✔] Creating containers...
[✔] Deploying preview...
  [✔] test_env
{"id":"deployment-4505","url":"https://deployment-4505-zakary-tillman-1-1e4m0u.app.uffizzi.com"}
#

["POSTGRES_PASSWORD"]=> string(8) "postgres"
["POSTGRES_USER"]=> string(8) "postgres"


Fix/nested references nonrebased

I restored this branch because we thought that maybe some code was lost in rebase, but the bug was elsewhere.