[SPARK-45580][SQL] Handle case where a nested subquery becomes an existence join

What changes were proposed in this pull request?
In RewritePredicateSubquery, prune existence flags from the final join when rewriteExistentialExpr returns an existence join. This change prunes the flags (attributes with the name "exists") by adding a Project node.
For example:
Join LeftSemi, ((a#13 = c1#15) OR exists#19)
:- Join ExistenceJoin(exists#19), (a#13 = col1#17)
:  :- LocalRelation [a#13]
:  +- LocalRelation [col1#17]
+- LocalRelation [c1#15]

becomes
Project [a#13]
+- Join LeftSemi, ((a#13 = c1#15) OR exists#19)
   :- Join ExistenceJoin(exists#19), (a#13 = col1#17)
   :  :- LocalRelation [a#13]
   :  +- LocalRelation [col1#17]
   +- LocalRelation [c1#15]

This change always adds the Project node, whether rewriteExistentialExpr returns an existence join or not. In the case when rewriteExistentialExpr does not return an existence join, RemoveNoopOperators will remove the unneeded Project node.
Why are the changes needed?
This query returns an extraneous boolean column when run in spark-sql:
create or replace temp view t1(a) as values (1), (2), (3), (7);
create or replace temp view t2(c1) as values (1), (2), (3);
create or replace temp view t3(col1) as values (3), (9);

select *
from t1
where exists (
  select c1
  from t2
  where a = c1
  or a in (select col1 from t3)
);

1	false
2	false
3	true

(Note: the above query will not have the extraneous boolean column when run from the Dataset API. That is because the Dataset API truncates the rows based on the schema of the analyzed plan. The bug occurs during optimization).
This query fails when run in either spark-sql or using the Dataset API:
select (
  select *
  from t1
  where exists (
    select c1
    from t2
    where a = c1
    or a in (select col1 from t3)
  )
  limit 1
)
from range(1);

java.lang.AssertionError: assertion failed: Expects 1 field, but got 2; something went wrong in analysis

Does this PR introduce any user-facing change?
No, except for the removal of the extraneous boolean flag and the fix to the error condition.
How was this patch tested?
New unit test.
Was this patch authored or co-authored using generative AI tooling?
No.

Merged to master. There is a conflict on code part in old branches.
Could you make backporting PRs for all live release branches, please, @bersprockets ?

@dongjoon-hyun working on the back-ports.

Thank you!


Correction for raw affiliation BRGM‐RNSC,  Orléans, France; UMR LISAH,, INRA-IRD-SupAgro,, Montpellier, , France

Correction needed for raw affiliation BRGM‐RNSC,  Orléans, France; UMR LISAH,, INRA-IRD-SupAgro,, Montpellier, , France
raw_affiliation_name: BRGM‐RNSC,  Orléans, France; UMR LISAH,, INRA-IRD-SupAgro,, Montpellier, , France
new_rors: 03rnk6m14;05q3vnk25;04yvax419;05deqk823
previous_rors: 03rnk6m14;05q3vnk25;04yvax419
works_examples: W2139856800
contact: f248e12d5670212f8966d87f1ce39965:0840c5aa431b3ec91c43d4f8d3956b76050d @ inrae.fr

This issue was accepted and ingested by the OpenAlex team on 2024-11-18. The new affiliations should be visible within the next 7 days.


switching between character delay or not registering

When switching between character i have to press the number key again for it to detect that it is on that character.

; 弱网络环境
; 0 | 1
weak-network = 0
; 性能
; When your device performance is insufficient, or the network environment is not good, please turn down this option
; 当你的设备性能不足，或网络环境较差时，请调低该选项
; high | medium | low
performance = medium
whats the difference between these two settings?
the bottom one is from the NA config and top is CN config
which one do you use?

basic/weak-network is correct.
basic/performance has been deprecated.


Make debugging of invalid config easier when duplicated key is found

Description:
Print more info when a duplicated key in the config is found.
Ideally the line number in the configuration file should also be printed, but I am not sure if that is available?
yaml.error.YAMLError: ERROR: duplicate keys: ['platform'] in configuration of:
platform: template
platform: numeric_state
entity_id: sun.sun
value_template: {{ state.attributes.elevation }}
below: -2

I don't like the use of new lines in exception messages. I do like the idea of showing the config block. So let's make a dict out of it and use repr_helper instead and make that part of the config message.

I just realized that we're having the potential here to print out the whole configuration because the most likely place to have a duplicate key is in the root (switch: and switch: for example). In that case, we should try to limit the text that we generate or else it's going to be too much text.

Another try.
Will not lose duplicate keys, and the text will be limited since only the keys are showed
yaml.error.YAMLError: ERROR: duplicate keys: ['temp_control'] in configuration of: homeassistant=..., temp_control=..., temp_control=..., sun=..., http=..., automation=...,

I think adding a line number would be infinitely more helpful and avoids any accidental dumping of the whole config. Looks like by the time we get called the pyyaml loader, which has the line number information, is already at the end of the file, but there is a stackoverflow post that shows how to annotate the parsed nodes with their respective line numbers.

In that stackoverflow post someone mentioned a yaml parser that is forked off of pyyaml, which already keeps track of line numbers. https://pypi.python.org/pypi/ruamel.yaml

Variant using existing pyyaml parser that adds both filename and line numbers.
https://github.com/balloob/home-assistant/compare/dev...jaharkes:configuration_error_msg

I'll be closing this PR in favor of #1750 which mentions the lines that we find duplicate keys. Hurray for better reporting! This release will be awesome.


How to display custom label with angles in iOS?

Click to expand!
Issue Type
Feature Request
Source
source
Tensorflow Version
pod 'TensorFlowLiteSwift', '~> 0.0.1-nightly', :subspecs => ['CoreML', 'Metal']
Custom Code
Yes
OS Platform and Distribution
No response
Mobile device
No response
Python version
No response
Bazel version
No response
GCC/Compiler version
No response
CUDA/cuDNN version
No response
GPU model and memory
No response
Current Behaviour?
How to display degree of angle on every keyPoint.

Standalone code to reproduce the issue
Below code for to create degree of angle I used

func angle(
              firstLandmark: CGPoint,
              midLandmark: CGPoint,
              lastLandmark: CGPoint
          ) -> CGFloat {
              let radians: CGFloat =
                  atan2(lastLandmark.y - midLandmark.y,
                            lastLandmark.x - midLandmark.x) -
                    atan2(firstLandmark.y - midLandmark.y,
                            firstLandmark.x - midLandmark.x)

               var degrees = radians * 180.0 / .pi
              //var degrees = radians * .pi / 180 /// .pi
                degrees = abs(degrees) // Angle should never be negative
               if degrees > 180.0 {
                  degrees = 360.0 - degrees // Always get the acute representation of the angle
               }

               let roundedValue1 = round(degrees)

              return roundedValue1
          }

Relevant log output
No response

Required like this,  is it possible?

Hi @IBSApple! Currently we don't have these features in TF-lite itself. Feel free to visit these solutions from mediapipe.
Facemesh- for Face detection
Pose detection - for other detection of other body parts.
Won't be this issue again a duplicate of previous feature requests. #56265 , #56294.
Thank you!

Ok thanks

Is it possible to add custom keyPoints in TF pose estimation?

@IBSApple ! You can upload your own dataset and train it to get custom keypoints.  Also found a link to use mediapipe's facemesh detection directly in Colab. Thank you!


Adding labels

Hey team,
I have gone ahead and added Labels into syncSession after months of the public requesting this feature.
Note: I have updated the Asset ID of the public source code to my profile as (of course) it is the latest version - however feel free to sync it to yours and edit the ID.

Also, for the love of god, I have added linting and styling.


Add support for having one external ID

What

Add support for having one external ID to behave like the locationID parameter

How

Add condition when having one external ID to interpret that as a locationID parameter

As agreed, I have created a ticket for refactoring the useEffect inside the Sidebar and Bottom sheet components. @ammapspeople


Improve exception when object is not found

Now just says Exception raise or something.

Fixed in #59 .

Fixed in #59 .


bluetooth: build error with extended advertising

When extended advertising is enabled without CONFIG_BT_CTLR_PRIVACY=y the following build error is produced:
/mnt/devel/zp/zephyr/subsys/bluetooth/controller/ll_sw/ull_adv_aux.c: In function 'ull_adv_aux_hdr_set_clear':
/mnt/devel/zp/zephyr/subsys/bluetooth/controller/ll_sw/ull_adv_aux.c:502:25: error: 'struct ll_adv_set' has no member named 'own_addr_type'
  502 |   sec_pdu->tx_addr = adv->own_addr_type & 0x1;

Unless using extended advertising features requires BT_CTLR_PRIVACY by specification this should be fixed.  (I really hope it isn't required, because my use case is sensor beacons that are well-served with the default id from FICR, and very poorly served by ids that change.)
Reproducing code at: https://github.com/pabigot/zephyr/tree/issue/28090

@andrzej-kaczmarek Please address this. And, if there is no address in the previous primary PDU, then with/without privacy the device address has to be supplied by the caller of the ull_adv_aux_hdr_set_clear function in the value length encoded parameter.

See this on nightly build:
INFO    -  1/12 nrf51dk_nrf51422          tests/bluetooth/init/bluetooth.init.test_controller_dbg_ll_sw_split FAILED Build failure (build)
 ```


feat: switch base image

switch to official ubuntu base image
add tools from base image
small env renaming

closes #111

build succeed 🎉

build succeed 🎉


Multiple media file in a message

Hello,
First of all, I'm thankful for your great effort in creating this excellent library. I'm looking forward to contributing to it.
My question about the "media" field. I checked twice before creating this issue. (also, I did deep research on StackOverflow)
When checking message content on a channel chat, I realized it only returns one media file for each message, but the message contains two different image files.
I'm not sure if this is a miss or my mistake. Please advise?
{TL.Message}     Date: {11/09/2022 12:49:01 PM}     From: {user<PHONE_NUMBER>}     ID: 82894     Peer: {channel<PHONE_NUMBER>}     ReplyTo: null     TtlPeriod: 0     date: {11/09/2022 12:49:01 PM}     edit_date: {01/01/0001 12:00:00 AM}     entities: null     flags: has_from_id | has_media | has_grouped_id | has_replies     forwards: 0     from_id: {user<PHONE_NUMBER>}     fwd_from: null     grouped_id:<PHONE_NUMBER>8633317     id: 82894     media: {TL.MessageMediaPhoto}     message: ""     peer_id: {channel<PHONE_NUMBER>}     post_author: null     reactions: null     replies: {TL.MessageReplies}     reply_markup: null     reply_to: null     restriction_reason: null     ttl_period: 0     via_bot_id: 0     views: 0

To send a media group (album of photos/documents), you can look at this example
When retrieving it, a media group consists of a series of (usually) consecutive messages in a chat (one per media) having the same grouped_id value, so you would need to collect all of these. There is no way to know how many medias/messages the album is composed of.


Docs: Titles are switched on "Address space example exposing endpoints externally" subsections

Under "Address space example exposing endpoints externally", the two subsections, "OpenShift route example" and "OpenShift LoadBalancer service example" need their titles to be swapped to reflect the content contained in each.

PR: https://github.com/EnMasseProject/enmasse/pull/3060/

PR merged, so closing


Avoid showing the Key added modal when the user first enters the key management screen on fresh install

Avoid showing the Key added modal when the user first enters the key management screen on fresh install

verified this issue on keeper app v 1.2.18(412)


More literature on Python is needed

not enough helpful information about Python

You did not use the GitHub methods (assignee, labels, linked PR)


rubocop: Fix RSpec/VerifiedDoubles

I'm not really good in verified doubles so did my best guessing which kind of them is right :)


Changed bottom padding on single page for shop

When shop introduce text (lots of words) was set on data, we can't see 'お店までの道順' where behind footer navigation.
So added bottom padding for container on single shop page.

Thanks 😊


added some passwords

Thank you for this! This is the first time I've actually figured out how to submit a pull request! (I'm Nathan's brother--he told me about it.)

This is so wonderful! Congrats on your first PR :-)


mssql select stmt special generation

fix #56
adding mssql select generation

Changes look okay to me. Can you double check @ryanemerson ?

Thanks @rigazilla


Fix more term dates in historical legislators file

This continues the work from #385. It makes the following changes:
J. Bennett Johnston's first Senate term began Nov. 14, 1972; not Jan. 1, 1972. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=J000189
Allen Ellender's last Senate term ended July 27, 1972; not Jan. 3, 1973. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=E000112
Elaine Edwards was in the Senate from Aug. 1, 1972, to Nov. 13, 1972; not Jan. 1, 1972, to Jan. 3, 1973. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=E000068
Thomas Eagleton's first Senate term began Dec. 28, 1968; not Jan. 1, 1968. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=E000004
Edward V. Long's first Senate term began Sept. 23, 1960; not Jan. 1, 1960. His second term ended Dec. 27, 1968; not Jan. 3, 1969. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=L000415
Thomas C. Hennings Jr.'s third House term ended Dec. 31, 1940; not Jan. 3, 1941. His last Senate term ended Sept. 13, 1960; not Jan. 3, 1961. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=H000502
William B. Spong Jr.'s Senate term began Dec. 31, 1966; not Jan. 1, 1966. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=S000739
A. Willis Robertson's first Senate term began Nov. 6, 1946; not Jan. 1, 1946. His last Senate term ended Dec. 30, 1966; not Jan. 3, 1967. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=R000317
Carter Glass's first Senate term began Feb. 20, 1920; not Jan. 1, 1920. His last term ended May 28, 1946; not Jan. 3, 1947. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=G000232
Thomas S. Martin's last Senate term ended Nov. 12, 1919; not March 3, 1919. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=M000200
Maurine Neuberger's first Senate term began Nov. 9, 1960; not Jan. 1, 1960. http://bioguide.congress.gov/scripts/biodisplay.pl?index=N000052
Richard Neuberger's Senate term ended March 9, 1960; not Jan. 3, 1961. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=N000053
Robert P. Griffin's last House term ended May 10, 1966; not Jan. 3, 1965. His first Senate term began May 11, 1966; not Jan. 1, 1966. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=G000465
Patrick V. McNamara's last Senate term ended April 30, 1966; not Jan. 3, 1967. http://bioguide.congress.gov/scripts/biodisplay.pl?index=M000582
Joseph Montoya's last House term ended Nov. 3, 1964; not Jan. 3, 1965. His first Senate term began Nov. 4, 1964, not Jan. 1, 1964. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=M000876
Dennis Chavez's first Senate term began May 11, 1935; not Jan. 1, 1935. His last Senate term ended Nov. 18, 1962; not Jan. 3, 1963. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=C000338
Bronson M. Cutting's first Senate term was Dec. 29, 1927, to Dec. 6, 1928; not Dec. 5, 1927, to March 3, 1929. His second Senate term ended May 6, 1935; not Jan. 3, 1935. http://bioguide.congress.gov/scripts/biodisplay.pl?index=C001028
Octaviano Ambrosio Larrazolo's Senate term began Dec. 7, 1928; not Jan. 1, 1928. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=L000101
Andrieus A. Jones's last Senate term ended Dec. 20, 1927; not Dec. 31, 1927. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=J000210
Charles Potter's first House term began Aug. 26, 1947; not Jan. 3, 1947. His last House term ended Nov. 4, 1952; not Jan. 3, 1951. His first Senate term began Nov. 5, 1952; not Jan. 1, 1952. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=P000460
Frederick Van Ness Bradley's last House term ended May 24, 1947; not Jan. 3, 1949. http://bioguide.congress.gov/scripts/biodisplay.pl?index=B000742
Arthur H. Vandenberg's first Senate term began March 31, 1928; not Jan. 1, 1928. His first Senate term ended Jan. 3, 1929; not Jan. 3, 1935. His second Senate term began April 15, 1929. His last Senate term ended April 18, 1951; not Dec. 31, 1951. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=V000025
Woodbridge Ferris's Senate term ended March 23, 1928; not March 3, 1929. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=F000092
Spessard Holland's first Senate term began Sept. 25, 1946; not Jan. 1, 1946. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=H000720
Charles O. Andrews's first Senate term began Nov. 3, 1936; not Jan. 1, 1936. His last Senate term ended Sept. 18, 1946; not Jan. 3, 1947. http://bioguide.congress.gov/scripts/biodisplay.pl?index=A000244
Park Trammell's last Senate term ended May 8, 1936; not Jan. 3, 1937. http://bioguide.congress.gov/scripts/biodisplay.pl?index=T000353
Harry P. Cain's Senate term began Dec. 26, 1946; not Jan. 1, 1946. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=C000021
Hugh B. Mitchell's Senate term was Jan. 10, 1945, to Dec. 25, 1946; not Jan. 3, 1945, to Jan. 3, 1947. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=M000814
Monrad C. Wallgren's last House term ended Dec. 19, 1940; not Jan. 3, 1939. His first Senate term began Dec. 19, 1940; not Jan. 1, 1940. His last Senate term ended Jan. 9, 1945; not Dec. 31, 1945. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=W000088
Lewis B. Schwellenbach's Senate term ended Dec. 16, 1940; not Jan. 3, 1941. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=S000160
Ralph Flanders's first Senate term began Nov. 1, 1946; not Jan. 1, 1946. http://bioguide.congress.gov/scripts/biodisplay.pl?index=F000190
Warren R. Austin's last Senate term began ended Aug. 2, 1946; not Jan. 3, 1947. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=A000342
Frank L. Greene's first House term began July 30, 1912; not April 4, 1911. His Senate term ended Dec. 17, 1930; not March 3, 1931. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=G000425
David J. Foster's last House term ended March 21, 1912; not March 3, 1913. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=F000300
Earle Clements served a second House term, from Jan. 3, 1947, until Jan. 6, 1948. His Senate term began Nov. 27, 1950; not Jan. 1, 1950. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=C000506
Alben Barkley's penultimate Senate term ended Jan. 19, 1949; not Dec. 31, 1949. His last Senate term ended April 30, 1956; not Jan. 3, 1957. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=B000145
Garrett Withers's first Senate term was Jan. 20, 1949, to Nov. 26, 1950; not Jan. 3, 1949, to Jan. 3, 1951. His first House term began Aug. 2, 1952; not Jan. 3, 1951. His second House term ended April 30, 1953; not Jan. 3, 1955. http://bioguide.congress.gov/scripts/biodisplay.pl?index=W000658
John A. Whitacker's first House term began April 17, 1948; not Jan. 3, 1947. His last House term ended Dec. 15, 1951; not Jan. 3, 1953. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=W000348
Karl E. Mundt's last House term ended Dec. 30, 1948; not Jan. 3, 1947. His first Senate term began Dec. 31, 1948; not Jan. 1, 1948. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=M001078
Harlan J. Bushfield's Senate term ended Sept. 27, 1948; not Jan. 3, 1949. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=B001168
Warren Magnuson's last House term ended Dec. 13, 1944; not Jan. 3, 1943. His first Senate term was Dec. 14, 1944, to Jan. 3, 1945; not Jan. 1, 1944, to Jan. 3, 1951. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=M000053
Homer T. Bone's last Senate term ended Nov. 13, 1944; not Jan. 3, 1945. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=B000614
Frank Carlson's first Senate term began Nov. 29, 1950; not Jan. 1, 1950. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=C000154
Clyde M. Reed's last Senate term ended Nov. 8, 1949; not Dec. 31, 1949. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=R000112
Joseph C. O'Mahoney's first Senate term was Dec. 18, 1933, to Jan. 3, 1935; not Jan. 1, 1934, to Jan. 3, 1941. He had another Senate term starting Nov. 29, 1954; not Jan. 1, 1954. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=O000088
Lester C. Hunt's Senate term ended June 19, 1954; not Jan. 3, 1955. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=H000975
John B. Kendrick's last term in the Senate ended Nov. 3, 1933; not Dec. 31, 1933. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=K000098
Prentiss Brown had a second House term, from Jan. 3, 1935, to Nov. 18, 1936. His first Senate term started Nov. 19, 1936; not Jan. 1, 1936.
James Couzens's first Senate term began Nov. 29, 1922; not Jan. 1, 1922. His last Senate term ended Oct. 22, 1936; not Jan. 3, 1937. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=C000812
Truman H. Newberry's Senate term ended Nov. 18, 1922; not March 3, 1923. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=N000062
Robert Rice Reynolds's first Senate term started Dec. 5, 1932; not Jan. 1, 1932. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=R000179
Lee S. Overman's last Senate term ended Dec. 12, 1930; not March 3, 1931. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=O000140
Robert D. Carey's first Senate term began Dec. 1, 1930; not Jan. 1, 1930. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=C000146
Francis E. Warren's first Senate term began Dec. 1, 1890; not Jan. 1, 1890. His last Senate term ended Nov. 28, 1929; not Dec. 31, 1929. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=W000164
Patrick J. Sullivan's Senate term was Dec. 5, 1929, to Nov. 20, 1930; not April 15, 1929, to March 3, 1931. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=S001060
Jesse H. Metcalf's first Senate term began Nov. 5, 1924; not Jan. 1, 1924. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=M000670
LeBaron B. Colt's last Senate term ended Aug. 18, 1924; not March 3, 1925. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=C000651
J. Lister Hill's last House term ended Jan. 11, 1938; not Jan. 3, 1939. His first Senate term began Jan. 10, 1938; not Jan. 1, 1938. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=H000598
Dixie Bibb Graves' Senate term was Aug. 20, 1937, to Jan. 10, 1938; not Jan. 5, 1937, to Jan. 3, 1939. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=G000391
Hugo Black's last Senate term ended Aug. 19, 1937; not Dec. 31, 1937. Source: http://bioguide.congress.gov/scripts/biodisplay.pl?index=B000499

+1

Just outstanding.


Prepare NPM release of React component and Web component

What

Make it possible to release the Map Template as a single React component and a Web component on NPM.

How

Updated the Stencil components changelog and a component readme so it can be published.
Replaced the readme in the Map Template package with one that is targeted the NPM releases
Moved the old readme in the Map Template package into a .github folder to make it show up on Github.
Change thre name fo the Map template package to @mapsindoors/map-template.
Added script to a distribution folder for the NPM release. This builds both a React component and a Web component.

Looks good to me. Well done with the documentation 👏


[Project Moon TRPG] Re-release v0.7

Submission Checklist
Required

[x] The pull request title clearly contains the name of the sheet I am editing.
[x] The pull request title clearly states the type of change I am submitting (New Sheet/New Feature/Bugfix/etc.).
[x] The pull request makes changes to files in only one sub-folder.
[x] The pull request does not contain changes to any json files in the translations folder (translation.json is permitted)

New Sheet Details

The name of this game is: <   >  (i.e. Dungeons & Dragons 5th Edition, The Dresden Files RPG)

The publisher of this game is: <   > (i.e. Wizards of the Coast, Evil Hat)

The name of this game system/family is: <   > (i.e. Dungeons & Dragons, FATE)

[x] I have followed the Character Sheets Standards when building this sheet.

[ ] I have authorization from the game's publisher to make this an official sheet on Roll20 with their name attached.
[x] This game is not a traditionally published game, but a copy of the game rules can be purchased/downloaded/found at: https://drive.google.com/drive/folders/15nMYLz0sB5xWUiSOG4oN6IsteRJjpdZk
[x] This sheet is for an unofficial fan game, modification to an existing game, or a homebrew system.

Changes / Description
Release v0.7 of this sheet was initially released three weeks ago (Pull request 1), before being reverted after multiple bugs were discovered (Pull request 2)
With all new features now being fully tested and all known bugs having been resolved the update is ready for re-release
The re-release also includes four new features:

Any property of the sheet that can be Imported/Exported can now be set through Default Sheet Settings. This notably includes settings in the top-right cogwheel menu
The Auto Update setting now displays an indicator inside the Update button showing if the functionality is active (green), searching for the initiative item (yellow) or failed to find the item and is in standby mode (red)
The token instance display is now resizable
Damage Resistance and Protection now apply to after resistance damage

A full illustrated changelog for the release can be found here: Roll20 - v0.7 Changelog
Important Info!
I've tested merging this update into the master branch locally and found that the merge does not function properly unless the [Project Moon RPG] Revert from v0.7 back to v0.6.3 merge is first reverted

I've added a second pull request Revert "[Project Moon RPG] Revert from v0.7 back to v0.6.3" which does this
All the merge conflicts on this request should resolve after the other pull request is merged
I'll be available tomorrow for the review period in case any issues or questions arise over this. Apologies for the mess!

Character Sheet Info Roll20 Internal Use only.

@BeautiDemise looks like there are a few more merge conflicts to sort through

Understood! Give me a few minutes to give them a look over

@NorWhal The changes from Release v0.7 are correct for both of the conflicts

@BeautiDemise please verify this is the correct code - just being extra cautious.

Believe I figured it out. Getting a couple of differences at there two locations (the leftmost code is correct)

@NorWhal Learning a lot on the go here! Added a commit that corrects the last few conflicts. It should be all good to go!


Adjust load_images Method to Ensure Correct File Paths

This pull request addresses an issue found in the load_images method of the BatchImageLoader class, where the method could potentially generate incorrect and duplicated file paths.
The load_images function used the glob.glob function to get the filenames from a specific directory, and then it appended the directory path to the filenames. However, since glob.glob already returns the full path (including the directory path), appending the directory path again resulted in duplicated paths.
To fix this issue, the code was refactored to add the result from glob.glob directly to self.image_paths, as glob.glob already provides a complete path. Furthermore, to handle all potential edge cases, the absolute path of each file is ensured using os.path.abspath.
The revised load_images method is as follows:
def load_images(self, directory_path, pattern):
    for file_name in glob.glob(os.path.join(directory_path, pattern), recursive=True):
        if file_name.lower().endswith(ALLOWED_EXT):
            # Ensure the file path is absolute
            abs_file_path = os.path.abspath(file_name)
            self.image_paths.append(abs_file_path)

With this change, we can now ensure that the load_images method always generates correct file paths, regardless of whether glob.glob returns relative or absolute paths.
Please let me know if you have any questions or concerns about this change.

Looks good. Thanks for the PR.


weedFS init Daemon !!

Hi,
As i was unable to find any daemon to start weedFS. Here is the daemon script to enable it on boot.

Paste following contents into /etc/init.d/weed
You can edit parameters like mdir ,ip etc by simply editing the code.

http://pastebin.com/wMdqy8xw

This script is tested on centos-6 with weedFS master. However, to make it compatible with volume, few parameters would be required to replace.

Nice. This is master server only. Can we also have the init script for volume server? You can fork the wiki repo https://github.com/chrislusf/seaweedfs/wiki and add a section about the deployment script.

We're using init on both master/volume right now. I'll surely create a section with deployment init daemons.


Remove the Rate struct

The declaration of the small Rate struct is more costly than not using it at all

This may save some gas, but we really want to keep Rate structure to make sure instances of this structure states immutable (a order never changes its rate). There are so many variables named amountBlah, keeping rate makes the code more readable.


Extract the results url from cloud build runs and post it as status

This should make it easier to access the results of the bazel test runs. Specifically, two different status messages are posted:

Required builds
Required tests

A thought: since we can see right here that the cloud builds are working, want to get rid of the Github Actions stuff in the same PR?

A thought: since we can see right here that the cloud builds are working, want to get rid of the Github Actions stuff in the same PR?

I considered it, but I think it is prudent to leave the GitHub Actions in place until we have enough confidence in the new set up. We could remove the "required" constraint on that test run though.

Fair point on the confidence thing. IDK, if we're going to keep it around, I think it might as well be blocking. With a time difference of 2m vs 25m, we'll be tempted to just ignore it if it's not required anyway. Keeping it required encourages us to keep considering while it is still around.

Fair point on the confidence thing. IDK, if we're going to keep it around, I think it might as well be blocking. With a time difference of 2m vs 25m, we'll be tempted to just ignore it if it's not required anyway. Keeping it required encourages us to keep considering it while it is still around.

Makes sense. Let us leave it in place for now. We can remove it in two weeks or so if we don't run into any major issues with the new set up.


Fix typos in 2.2.0

+1

:+1:

:+1:

:+1:

:+1:

Oh damn!
Sorry for this. merging and deploying ASAP!

Published on NPM as 2.2.1
Sorry again for the typos. Should've never been published without more testing.


[FEATURE] WINNF.FT.S.REL2.RI.EAP.1 Test Case Development

Description
Test case WINNF.FT.S.REL2.RI.EAP.1 is for testing of antenna gain calculation using EAP methods.
Requirements
Test data needs to be created as specified in the spec
Files
A list of files or test code related to this issue.
TS-4006, pp. 23
Tasks

[x] Review spec for WINNF.FT.S.REL2.RI.EAP.1.
[x] Write code
[x] Create test data based on spec
[x] Test code with the data created in Task 3

Created test data.
Reviewed test specification.

Partially implemented, code for antenna gain calculation using reference model is done, need to add code for comparing this with the value provided by the testing SAS . Code needs to be reviewed.


Ubuntu 22.04 / 22.10 images are missing riscv64 architecture

Reported by @mtelvers the Ubuntu images on docker hub are missing riscv 64 support in the latest versions. See https://hub.docker.com//ubuntu/tags?page=1&name=22.10 and https://hub.docker.com//ubuntu/tags?page=1&name=22.04.
Causing the docker pipeline to fail and our riscv based images to be out of date. Based on the scarcity of riscv hardware we are only really using the riscv hardware for building these images and for some minor testing. We could one of:

Use the oldest published version that did include riscv support
Use an alternative image like https://hub.docker.com/r/riscv64/ubuntu/
Remove publishing images until we get more hardware online and Ubuntu starts publishing docker images again.

Option 1 seems reasonable for now. Other than the ocluster-worker and base-image-builder, is anyone else relying on these images? @dra27

For the latest ocluster worker deployment, we went with option 1, using FROM ubuntu:jammy-20221130 as that is the latest version which supports riscv.  Dockerfile.worker

What's the "correct" thing here? Should we be using riscv64/ubuntu in general?

i.e. is there any information on the change in support of riscv64

Not much apart from "this is intentional and we're talking about it to possibly bring it back later" https://github.com/docker-library/official-images/pull/13950#issuecomment-1412111952

I'm suggesting to use the images from riscv64/ubuntu:

https://github.com/ocurrent/ocaml-dockerfile/pull/150
https://github.com/ocurrent/docker-base-images/pull/214

What's the "correct" thing here? Should we be using riscv64/ubuntu in general?

I've pushed this change, enabling the riscv64 Ubuntu images again (and the multiarch tags as well).

@benmandrew With the 23.10 upgrade docker base images is missing RiscV support again. See https://images.ci.ocaml.org/job/2023-10-31/060912-ocluster-build-fd0e13

The Ubuntu build page shows that Mantic has been built successfully on riscv64 and is up-to-date: https://launchpad.net/~cloud-images-release-managers/+livefs/ubuntu/mantic/ubuntu-oci.
The riscv64/ubuntu Docker page hasn't had updates in 10 months. I sent an email a few weeks ago to Steve Langasek (listed on the Launchpad page) about this but I've not had a response. I'll try contacting other members of the team, but it's a little difficult to find who's responsible.

@mtelvers has work to address this on https://github.com/mtelvers/docker-base-images/pull/1, which PR will to target this repo soon.

@mtelvers opened https://github.com/ocurrent/docker-base-images/issues/206, which tracks work on a general solution to this problem, so I'll close this in favor of that.


sdk-hsm-thingy53/sample/02_rgb_led

Can you give me the detailed configuration of the device tree?

I'm using a predefined device tree, did not create one in this repo. See the declaration here
https://github.com/HomeSmartMesh/sdk-hsm-thingy53/blob/17828cb5aef531865aab6b7b1e8462148d3e42c7/samples/02_rgb_led/CMakeLists.txt#L10
the board  and the device tree you're looking for can be found in the Zephyr repo here
https://github.com/zephyrproject-rtos/zephyr/tree/main/boards/arm/thingy53_nrf5340
Closing as this is not an issue, rather a request for support. For future support requests, it is better to open a github discussion
https://github.com/HomeSmartMesh/sdk-hsm-sensortag/discussions


Add dtensor support for TE optimizers

Description
Please include a brief summary of the changes, relevant motivation and context.
multi_tensor_apply needs to work directly on the local shards of the DTensors in the optimizer states.
Fixes # (issue)
Type of change

[ ] Documentation change (change only to the documentation, either a fix or a new content)
[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Infra/Build change
[ ] Code refractor

Changes
Please list the changes introduced in this PR:

If tensor is DTensor, replace it with its local shard.

Checklist:

[ ] I have read and followed the contributing guidelines
[ ] The functionality is complete
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes

/te-ci pytorch

@timmoon10 Hi could you merge this? I don't have the permission to do so.


[Not an issue] http4s-armeria reference in the docs

Do you mind If I add a reference to the http4s-armeria project on the Scala integration page on your website?

I'm happy to add a link for http4s-armeria. ❤ Please send a PR.


CMake Support

I've taken the liberty of moving @mnunberg's cmake branch towards feature parity with the current Makefile.
Supports:

Building
Installing
Running unit tests
Generating & installing pkg-config file

Missing:

Running unit tests through Valgrind
Building certain examples

Original Makefile is dropped. MSVC support, if desired, should be done in a subsequent PR.
Fixes #611. Fixes #641.

I'm going to revert the change of removing the Makefile and then merge this

I've merged this PR, sans the removal of the original Makefile


Update to use package:observable

This shouldn't be merged until polymer is updated, or we'll get version lock in our pubspec


OP-1857: NCTL - Validator Ejection Scenario

Implements itst13 from https://casperlabs.atlassian.net/wiki/spaces/OPS/pages/1133707277/Casper+Network+Infra+Test+Scenarios.
Also refactor nightly ci script to take a non-default chainspec.

No blockers from me :)

thanks! I'll update the points you mentioned!

bors r+


Name being displayed as "(name)"

In some of the dialogue where she mentions my name, instead of my name displaying it just displays what I assume is a generic placeholder ("(name)") or something.

Fixed in newest update!


Make it compatible with Deno

Make it compatible with Deno, and publish it wherever Deno users pick their libraries from.

[x] Resolve issue #11 first.
[x] Ensure that imports have extensions
[x] https://denopkg.com/
[x] https://deno.land/x/
[ ] https://nest.land/

More info here: https://dev.to/craigmorten/how-to-publish-deno-modules-2cg6

Done, we won't provide support for nest.land until they improve their tooling.


KAFKA-6911: Fix dynamic keystore/truststore update check

Fix the check, add unit test to verify the change, update DynamicBrokerReconfigurationTest to avoid dynamic keystore update in tests which are not expected to update keystores.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

retest this please


Create an alternative diagram for Kubernetes version of MMS

I don't know what is meant by an alternative diagram. At the moment we've got the deployment diagram.

It's a redesigned MMS architecture. What it will look like when it's deployed to Kubernetes. The main change being the move of MMSD out of each production hub and creating a centralized MMSD and the second big change is moving NATS from inside MMSD to an external container.


MWPW-146848: Repo Sync Workflow for FG

Floodgate prep for MAX

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 98.05%. Comparing base (3000832) to head (93bae33).

Additional details and impacted files
@@           Coverage Diff           @@
##            stage     #666   +/-   ##
=======================================
  Coverage   98.05%   98.05%
=======================================
  Files          34       34
  Lines        4168     4168
=======================================
  Hits         4087     4087
  Misses         81       81

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Docs


Provide better error descriptions for CSI-S3 access and connection failures

Issue:
Currently, CSI-S3 emits this error string in the logs most of the time: failed to initialize S3 client: Endpoint:  does not follow ip address or domain name standards. However, the root causes of this error can  be different e.g. #160 #90 The dataset is left pending.
Requirements:
Provide an error log message corresponding to the root cause and use it to populate the status field of the Dataset.

I would like to take this @srikumar003
/assign

i am seeing the invalid endpoint errors when running in Kind.  i have tried a bunch of variants for the endpoint field of my DataSet. none seem to work. e.g. "codeflare-s3.codeflare-system.svc.cluster.local:9000", with and without the leading http://... with and without the svc... part, etc. nothing works.
how does one go about diagnosing/debugging/fixing these kinds of issues?

@ashutosh887 apologies for not responding and thanks for taking on this issue!

Thanks @srikumar003
Is there a public channel to discuss if I get stuck somewhere

I have the same issue and cannot understand what's wrong:
I1218 00:06:07.573611       1 controller.go:1332] provision "mattermost/mm-db-dump" class "csi-s3": started
I1218 00:06:07.573990       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"mattermost", Name:"mm-db-dump", UID:"66e7a933-4e0d-4389-b1af-f187372ac075", APIVersion:"v1", ResourceVersion:"178444495", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "mattermost/mm-db-dump"
I1218 00:06:07.581240       1 connection.go:182] GRPC call: /csi.v1.Controller/CreateVolume
I1218 00:06:07.581273       1 connection.go:183] GRPC request: {"capacity_range":{"required_bytes":5368709120},"name":"pvc-66e7a933-4e0d-4389-b1af-f187372ac075","parameters":{"mounter":"goofys"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":5}}]}
I1218 00:06:07.583155       1 connection.go:185] GRPC response: {}
I1218 00:06:07.583264       1 connection.go:186] GRPC error: rpc error: code = Unknown desc = failed to initialize S3 client: Endpoint:  does not follow ip address or domain name standards.
I1218 00:06:07.583350       1 controller.go:753] CreateVolume failed, supports topology = false, node selected false => may reschedule = false => state = Finished: rpc error: code = Unknown desc = failed to initialize S3 client: Endpoint:  does not follow ip address or domain name standards.
I1218 00:06:07.583480       1 controller.go:1099] Final error received, removing PVC 66e7a933-4e0d-4389-b1af-f187372ac075 from claims in progress
W1218 00:06:07.583524       1 controller.go:958] Retrying syncing claim "66e7a933-4e0d-4389-b1af-f187372ac075", failure 7
E1218 00:06:07.583574       1 controller.go:981] error syncing claim "66e7a933-4e0d-4389-b1af-f187372ac075": failed to provision volume with StorageClass "csi-s3": rpc error: code = Unknown desc = failed to initialize S3 client: Endpoint:  does not follow ip address or domain name standards.
I1218 00:06:07.583635       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"mattermost", Name:"mm-db-dump", UID:"66e7a933-4e0d-4389-b1af-f187372ac075", APIVersion:"v1", ResourceVersion:"178444495", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "csi-s3": rpc error: code = Unknown desc = failed to initialize S3 client: Endpoint:  does not follow ip address or domain name standards.

The CRD  is the next:
    apiVersion: com.ie.ibm.hpsys/v1alpha1
    kind: Dataset
    metadata:
      name: mm-db-dump
    spec:
      local:
        type: "COS" # Cloud Object Storage
        secret-name: "s3-dump-secret"
        #secret-namespace: "{SECRET_NAMESPACE}" #optional if the secret is in the same ns as dataset
        endpoint: "http://rgw-slow-dev01.ti.local"
        bucket: "mm-db-dump"
        readonly: "false" # default is false
        provision: "false" # DLF will allocate bucket on the COS if it doesn't exist [Default: false]
        #region: "" #it can be empty

The issue was that https://github.com/datashim-io/datashim/issues/90#issuecomment-808296672
I'd edited the endpoint and that was not catched.


Button position change after upgrade 3.1.1

[x] This is not a v0.x issue.
[x] I have searched the issues of this repository and believe that this is not a duplicate.

Expected Behavior

Current Behavior
after yarn upgrade @material-ui/core upgrade to 3.1.1

Is it expected?
Steps to Reproduce

Link:

before upgrade commit
after upgrade commit
button code

Context

Your Environment

Reproduce in Travis CI

Tech
Version

Material-UI
v3.1.1

@quanbrew Thank you for the report. The change was most likely introduced by #12906. I have a bad news for you, the previous behavior was most likely already broken. I would suggest you to use the flexbox if you need a fine grain tuning on the alignement.

@oliviertassinari Thanks, If the alignment behavior is consistent, it is acceptable.

Yes, it was done for consistency.


Pub trait store references are deprecated. Updated the content.

Fixes https://github.com/substrate-developer-hub/substrate-docs/issues/2020


Breakage from 0.4.0 to 0.5.0

With version 0.4.0 and
source_suffix = [".rst", ".md"]
source_parsers = {".md": "recommonmark.parser.CommonMarkParser"}

Works fine, markdown files are added to the build.
With version 0.5.0 and
source_suffix = [".rst", ".md"]
source_parsers = {".md": "recommonmark.parser.CommonMarkParser"}

Extension error:
source_parser for 'md' is already registered

With version 0.5.0 and
source_suffix = [".rst", ".md"]
# source_parsers = {".md": "recommonmark.parser.CommonMarkParser"}

Markdown files not added:
checking consistency... /media/Data/dev/pawamoy/aria2p/docs/changelog.md: WARNING: document isn't included in any toctree
/media/Data/dev/pawamoy/aria2p/docs/code_of_conduct.md: WARNING: document isn't included in any toctree
/media/Data/dev/pawamoy/aria2p/docs/configuration.md: WARNING: document isn't included in any toctree
/media/Data/dev/pawamoy/aria2p/docs/contributing.md: WARNING: document isn't included in any toctree
/media/Data/dev/pawamoy/aria2p/docs/credits.md: WARNING: document isn't included in any toctree
/media/Data/dev/pawamoy/aria2p/docs/readme.md: WARNING: document isn't included in any toctree
/media/Data/dev/pawamoy/aria2p/docs/usage.md: WARNING: document isn't included in any toctree

I'm guessing the sphinx version is the problem, you should use
extensions = ['recommonmark']

For sphinx >= 1.4

Sorry, I forgot to precise, I already have "recommonmark" in the extensions list.

Can you link to the project? Or a reproducible project?

Of course: aria2p
You'll need poetry.
git clone https://github.com/pawamoy/aria2p
cd aria2p
poetry install

make docs  # should work

poetry run pip install recommonmark==0.5.0
make docs  # error above

Hi again, it seems recommonmark 0.5.0 is now installed by default on readthedocs. It forces me to pin recommonmark==0.4.0 in a requirements.txt file (mine is docs/requirements.txt).
Have you had any time to take a look at the issue?
I created a new repository to reproduce the issue at https://github.com/pawamoy/repro-recommonmark-146.

source_parser for 'md' is already registered

Seems to be a misreading of the changes in Sphinx 1.8 where source_parsers has been replaced by a source_suffix dict. So the change for Sphinx 1.8 and recommonmark 0.5 should be:
 # Documentation in Markdown
-source_suffix = [".rst"]
-source_parsers = {".md": "recommonmark.parser.CommonMarkParser"}
+source_suffix = {
+    '.rst': 'restructuredtext',
+    '.md': 'markdown',
+}

It wasn't immediately obvious to me and hope that helps

A-ha! It seems the culprit was in fact url_resolver. Trying your fix only did not work, but when I also commented the url_resolver line, it worked.
# Documentation in Markdown
-source_suffix = [".rst"]
-source_parsers = {".md": "recommonmark.parser.CommonMarkParser"}
+source_suffix = [".rst", ".md"]

@ conf.py:39 @ def setup(app):
-            "url_resolver": lambda url: doc_root + url,
+            #"url_resolver": lambda url: doc_root + url,

Based on the docs:

Auto Doc Ref
This option is deprecated. This option has been superseded by the default linking behavior, which will first try to resolve as an internal reference, and then as an external reference.

So, shouldn't my markdown files be resolved internally first, and then externally to GitHub (with url_resolver)? It seems it's doing the opposite, since keeping the url_resolver line uncommented results in these two warnings:

/media/Data/dev/pawamoy/repro-recommonmark-146/docs/contributing.md: WARNING: document isn't included in any toctree
/media/Data/dev/pawamoy/repro-recommonmark-146/docs/readme.md: WARNING: document isn't included in any toctree

...and the links in the generated docs then point to GitHub (with a 404).

Closing in favor of https://github.com/readthedocs/recommonmark/issues/137


Can the destination patching be optional?

Hi, still working on the CI setup 👋🏻
I have quite a specific setup I believe, in which I have multiple clusters within Argo with ApplicationSets which generate the Application based on the cluster's labels, e.g:
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
   [OMITED]
spec:
  generators:
    - matrix:
        generators:
          - list:
              elements:
                - environment: production
                  server_name: <application_instance_identifier>
          - clusters:
              selector:
                matchLabels:
                  argocd.argoproj.io/secret-type: cluster
                  "production_{{server_name}}": enabled
  syncPolicy:
    preserveResourcesOnDeletion: true
  template:
    metadata:
      [OMITED]
    spec:
      [OMITED]
      destination:
        server: "{{server}}"

While the cluster secret looks like this:
apiVersion: v1
kind: Secret
metadata:
  name: cluster_name
  namespace: argocd
  annotations:
    managed-by: argocd.argoproj.io
  labels:
    argocd.argoproj.io/secret-type: cluster
    production_id: enabled
type: Opaque
stringData:
  [OMITED]

This approach gives us a lot of control over what is deployed on the K8s cluster with a single look at the cluster's labels (which work like a kind of feature flag here). It's a very handy abstraction, I'd like not to resign from. However, it seems to be not compatible with ArgoCD Diff Preview as the ADP patches all Application/ApplicationSets destinations to in-cluster and in my case, it generates false diffs due to those labels selectors/labels templating (as mentioned in #29 ).
I made a fork locally, where I simply commented out this patching, I manually patched cluster secrets to override .stringData.server = "https://kubernetes.default.svc" and it works like a charm, generating diffs with respect to label selectors.
I'd be extremely grateful if this patching could be disabled with a flag. I believe I can provide a PR with this change.
EDIT: nah, I can't deliver such a PR, I don't know Rust enough to write it idiomatically enough (and simply passing a flag via arguments three times doesn't feel right, I believe a sort of global configuration object would work fine), I resort to you, Dag 🙏🏻

Hi again :)
And again, Thank you once more for your interest in the project 🚀
Could you help me understand the problem in a bit more detail?
Yes, argocd-diff-preview will update your destination.server: "{{server}}", but i am not sure why that is a problem.

in my case, it generates false diffs due to those labels selectors/labels templating

Does Argo CD produce a different result if the destination.server field is set directly vs. if it is populated by the  generators.clusters?
I just want to make sure I understand your request correctly. You would like to disable patching specifically for
spec.template.spec.destination.server, but keep all the other patches on that file? (e.g. disabling of syncPolicy and setting spec.project: default)

ADP patches all Application/ApplicationSets destinations

... and just to avoid any confusion - It only patches Applications' sources where .spec.sources[*].repoURL.contains(<your-org/your-repo>). Not sources which point to other URLs :)

Does Argo CD produce a different result if the destination.server field is set directly vs. if it is populated by the generators.clusters?

Yes, as you might remember we're utilizing templating ApplicationSets with cluster's secrets labels for setting specific Helm values (e.g. for selecting the valid DataDog instance). We even use the Server's name in the Application name (like deployment-{{name}}).
Take this live example of our DataDog agent deployment:
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: datadog
  namespace: argocd
spec:
  syncPolicy:
    preserveResourcesOnDeletion: true
  generators:
    - clusters: {}
  template:
    metadata:
      name: "datadog-{{name}}"
    spec:
      project: "datadog"
      sources:
        - repoURL: https://helm.datadoghq.com
          chart: datadog-operator
          targetRevision: 1.2.2
          helm:
            releaseName: "datadog"
            valueFiles:
              - $values/deployments/production/datadog/values/{{metadata.labels.datadog_instance}}.yaml
        - repoURL: git@github.com:company/platform.git
          targetRevision: main
          ref: values
        - repoURL: git@github.com:company/platform.git
          targetRevision: main
          path: deployments/production/datadog/manifests
          directory:
            recurse: true
            include: "{credentials,agents}/{{metadata.labels.datadog_instance}}.yaml"
      destination:
        server: "{{server}}"
        namespace: monitoring
      syncPolicy:
        automated: {}
        syncOptions:
          - CreateNamespace=true

As you can see this ApplicationSet will generate different Applications based on the value of the server's name and the value of cluster's label datadog_instance.

Sorry, I confused myself while working on this 😅 I made a simple PoC with dummy resources and it works just fine, disabling patching wasn't necessary.
I believe I went off the rails while figuring out how not to spawn extra application resources due to extra in-cluster cluster secrets (an empty cluster generator will produce additional resources for that).


Leak in term writing?

The following program aborts with a malloc error (likely from a buffer overflow):
:- op(201, fy, '@').

test :-
	Result = dict('.'(:(valid,@(true)),'.'(:(fields,dict('.'(:(revision,dict('.'(:(valid,@(true)),'.'(:(ignored,@(true)),[])))),'.'(:(creator,dict('.'(:(valid,@(true)),'.'(:(ignored,@(true)),[])))),'.'(:(updated_at,dict('.'(:(valid,@(true)),'.'(:(ignored,@(true)),[])))),'.'(:(created_at,dict('.'(:(valid,@(true)),'.'(:(ignored,@(true)),[])))),'.'(:(qux_abbreviation,dict('.'(:(valid,@(true)),[]))),'.'(:(qux_name,dict('.'(:(valid,@(false)),'.'(:(error,'256文字以下の値を入力してください'),[])))),'.'(:(baz_id,dict('.'(:(valid,@(false)),'.'(:(error,'256文字以下の値を入力してください'),[])))),'.'(:(owner_id,dict('.'(:(valid,@(true)),'.'(:(ignored,@(true)),[])))),'.'(:(qux_type_code,dict('.'(:(valid,@(true)),[]))),'.'(:(foo,dict('.'(:(fields,dict('.'(:('foo/bar_groups',dict('.'(:(valid,@(true)),[]))),'.'(:('foo/acme_notification_url',dict('.'(:(valid,@(false)),'.'(:(error,'「https://」で始まらなければなりません'),[])))),'.'(:('foo/qux_api_version',dict('.'(:(valid,@(true)),[]))),'.'(:('foo/xyz_unit_id',dict('.'(:(valid,@(false)),'.'(:(error,'256文字以下の値を入力してください'),[])))),'.'(:('foo/abc_label',dict('.'(:(fields,dict('.'(:('foo/abc_label/print_abc_list',dict('.'(:(valid,@(true)),[]))),'.'(:('foo/abc_label/report_id',dict('.'(:(valid,@(true)),[]))),'.'(:('foo/abc_label/report_setting_key',dict('.'(:(valid,@(true)),[]))),[]))))),'.'(:(valid,@(true)),[])))),'.'(:('foo/xyz_notes',dict('.'(:(valid,@(true)),[]))),[])))))))),'.'(:(valid,@(true)),[])))),'.'(:(system_id,dict('.'(:(valid,@(false)),'.'(:(error,'256文字以下の値を入力してください'),[])))),'.'(:(qux_status,dict('.'(:(valid,@(true)),[]))),[])))))))))))))),'.'(:(spec,dict('.'(:(type,model),'.'(:(model,qux_config),[])))),[])))),
	write(Result).

$ tpl leak.pl -g test,halt
malloc(): invalid size (unsorted)
Aborted

Writing with ignore_ops(true) seems to avoid it, so my guess would be it has to do with space insertion for the @ operator.
Valgrind log
==2116== Memcheck, a memory error detector
==2116== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==2116== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==2116== Command: tpl leak.pl -g test,halt
==2116==
==2116== Invalid write of size 2
==2116==    at 0x4842B33: memmove (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2116==    by 0x1AA12D: plain (print.c:232)
==2116==    by 0x1AEF4F: print_term_to_buf (print.c:844)
==2116==    by 0x1B1344: print_term_to_buf (print.c:1033)
==2116==    by 0x1ABDA3: print_iso_list (print.c:520)
==2116==    by 0x1AE067: print_term_to_buf (print.c:741)
==2116==    by 0x1AF47E: print_term_to_buf (print.c:895)
==2116==    by 0x1B21D8: print_term_to_buf (print.c:1127)
==2116==    by 0x1ABDA3: print_iso_list (print.c:520)
==2116==    by 0x1AE067: print_term_to_buf (print.c:741)
==2116==    by 0x1AF47E: print_term_to_buf (print.c:895)
==2116==    by 0x1B2EFE: print_term_to_stream (print.c:1317)
==2116==  Address 0x59f1930 is 1,200 bytes inside a block of size 1,201 alloc'd
==2116==    at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2116==    by 0x1B2E8C: print_term_to_stream (print.c:1313)
==2116==    by 0x1D4E88: fn_iso_write_1 (streams.c:1998)
==2116==    by 0x1BC369: start (query.c:1749)
==2116==    by 0x1BCB84: execute (query.c:1949)
==2116==    by 0x1654CA: run (parser.c:3615)
==2116==    by 0x1B393B: pl_eval (prolog.c:122)
==2116==    by 0x1164F2: main (tpl.c:274)
==2116==
==2116== Invalid write of size 1
==2116==    at 0x4842B63: memmove (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2116==    by 0x1AA12D: plain (print.c:232)
==2116==    by 0x1AEF4F: print_term_to_buf (print.c:844)
==2116==    by 0x1B1344: print_term_to_buf (print.c:1033)
==2116==    by 0x1ABDA3: print_iso_list (print.c:520)
==2116==    by 0x1AE067: print_term_to_buf (print.c:741)
==2116==    by 0x1AF47E: print_term_to_buf (print.c:895)
==2116==    by 0x1B21D8: print_term_to_buf (print.c:1127)
==2116==    by 0x1ABDA3: print_iso_list (print.c:520)
==2116==    by 0x1AE067: print_term_to_buf (print.c:741)
==2116==    by 0x1AF47E: print_term_to_buf (print.c:895)
==2116==    by 0x1B2EFE: print_term_to_stream (print.c:1317)
==2116==  Address 0x59f1932 is 1 bytes after a block of size 1,201 alloc'd
==2116==    at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2116==    by 0x1B2E8C: print_term_to_stream (print.c:1313)
==2116==    by 0x1D4E88: fn_iso_write_1 (streams.c:1998)
==2116==    by 0x1BC369: start (query.c:1749)
==2116==    by 0x1BCB84: execute (query.c:1949)
==2116==    by 0x1654CA: run (parser.c:3615)
==2116==    by 0x1B393B: pl_eval (prolog.c:122)
==2116==    by 0x1164F2: main (tpl.c:274)
==2116==
==2116== Invalid write of size 1
==2116==    at 0x1AA13B: plain (print.c:233)
==2116==    by 0x1AEF4F: print_term_to_buf (print.c:844)
==2116==    by 0x1B1344: print_term_to_buf (print.c:1033)
==2116==    by 0x1ABDA3: print_iso_list (print.c:520)
==2116==    by 0x1AE067: print_term_to_buf (print.c:741)
==2116==    by 0x1AF47E: print_term_to_buf (print.c:895)
==2116==    by 0x1B21D8: print_term_to_buf (print.c:1127)
==2116==    by 0x1ABDA3: print_iso_list (print.c:520)
==2116==    by 0x1AE067: print_term_to_buf (print.c:741)
==2116==    by 0x1AF47E: print_term_to_buf (print.c:895)
==2116==    by 0x1B2EFE: print_term_to_stream (print.c:1317)
==2116==    by 0x1D4E88: fn_iso_write_1 (streams.c:1998)
==2116==  Address 0x59f1933 is 2 bytes after a block of size 1,201 alloc'd
==2116==    at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2116==    by 0x1B2E8C: print_term_to_stream (print.c:1313)
==2116==    by 0x1D4E88: fn_iso_write_1 (streams.c:1998)
==2116==    by 0x1BC369: start (query.c:1749)
==2116==    by 0x1BCB84: execute (query.c:1949)
==2116==    by 0x1654CA: run (parser.c:3615)
==2116==    by 0x1B393B: pl_eval (prolog.c:122)
==2116==    by 0x1164F2: main (tpl.c:274)
==2116==
==2116== Invalid write of size 1
==2116==    at 0x4E05F81: __vsnprintf_internal (vsnprintf.c:112)
==2116==    by 0x4DDBDF5: snprintf (snprintf.c:31)
==2116==    by 0x1AEFD2: print_term_to_buf (print.c:848)
==2116==    by 0x1B1344: print_term_to_buf (print.c:1033)
==2116==    by 0x1ABDA3: print_iso_list (print.c:520)
==2116==    by 0x1AE067: print_term_to_buf (print.c:741)
==2116==    by 0x1AF47E: print_term_to_buf (print.c:895)
==2116==    by 0x1B21D8: print_term_to_buf (print.c:1127)
==2116==    by 0x1ABDA3: print_iso_list (print.c:520)
==2116==    by 0x1AE067: print_term_to_buf (print.c:741)
==2116==    by 0x1AF47E: print_term_to_buf (print.c:895)
==2116==    by 0x1B2EFE: print_term_to_stream (print.c:1317)
==2116==  Address 0x59f1933 is 2 bytes after a block of size 1,201 alloc'd
==2116==    at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2116==    by 0x1B2E8C: print_term_to_stream (print.c:1313)
==2116==    by 0x1D4E88: fn_iso_write_1 (streams.c:1998)
==2116==    by 0x1BC369: start (query.c:1749)
==2116==    by 0x1BCB84: execute (query.c:1949)
==2116==    by 0x1654CA: run (parser.c:3615)
==2116==    by 0x1B393B: pl_eval (prolog.c:122)
==2116==    by 0x1164F2: main (tpl.c:274)
==2116==
==2116== Invalid write of size 1
==2116==    at 0x4E05FAB: __vsnprintf_internal (vsnprintf.c:117)
==2116==    by 0x4DDBDF5: snprintf (snprintf.c:31)
==2116==    by 0x1AEFD2: print_term_to_buf (print.c:848)
==2116==    by 0x1B1344: print_term_to_buf (print.c:1033)
==2116==    by 0x1ABDA3: print_iso_list (print.c:520)
==2116==    by 0x1AE067: print_term_to_buf (print.c:741)
==2116==    by 0x1AF47E: print_term_to_buf (print.c:895)
==2116==    by 0x1B21D8: print_term_to_buf (print.c:1127)
==2116==    by 0x1ABDA3: print_iso_list (print.c:520)
==2116==    by 0x1AE067: print_term_to_buf (print.c:741)
==2116==    by 0x1AF47E: print_term_to_buf (print.c:895)
==2116==    by 0x1B2EFE: print_term_to_stream (print.c:1317)
==2116==  Address 0x59f1933 is 2 bytes after a block of size 1,201 alloc'd
==2116==    at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2116==    by 0x1B2E8C: print_term_to_stream (print.c:1313)
==2116==    by 0x1D4E88: fn_iso_write_1 (streams.c:1998)
==2116==    by 0x1BC369: start (query.c:1749)
==2116==    by 0x1BCB84: execute (query.c:1949)
==2116==    by 0x1654CA: run (parser.c:3615)
==2116==    by 0x1B393B: pl_eval (prolog.c:122)
==2116==    by 0x1164F2: main (tpl.c:274)
==2116==
==2116== Invalid write of size 1
==2116==    at 0x4842B63: memmove (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2116==    by 0x1AA12D: plain (print.c:232)
==2116==    by 0x1B19DE: print_term_to_buf (print.c:1084)
==2116==    by 0x1ABDA3: print_iso_list (print.c:520)
==2116==    by 0x1AE067: print_term_to_buf (print.c:741)
==2116==    by 0x1AF47E: print_term_to_buf (print.c:895)
==2116==    by 0x1B21D8: print_term_to_buf (print.c:1127)
==2116==    by 0x1ABDA3: print_iso_list (print.c:520)
==2116==    by 0x1AE067: print_term_to_buf (print.c:741)
==2116==    by 0x1AF47E: print_term_to_buf (print.c:895)
==2116==    by 0x1B2EFE: print_term_to_stream (print.c:1317)
==2116==    by 0x1D4E88: fn_iso_write_1 (streams.c:1998)
==2116==  Address 0x59f1933 is 2 bytes after a block of size 1,201 alloc'd
==2116==    at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2116==    by 0x1B2E8C: print_term_to_stream (print.c:1313)
==2116==    by 0x1D4E88: fn_iso_write_1 (streams.c:1998)
==2116==    by 0x1BC369: start (query.c:1749)
==2116==    by 0x1BCB84: execute (query.c:1949)
==2116==    by 0x1654CA: run (parser.c:3615)
==2116==    by 0x1B393B: pl_eval (prolog.c:122)
==2116==    by 0x1164F2: main (tpl.c:274)
==2116==
==2116== Invalid write of size 1
==2116==    at 0x1AA13B: plain (print.c:233)
==2116==    by 0x1B19DE: print_term_to_buf (print.c:1084)
==2116==    by 0x1ABDA3: print_iso_list (print.c:520)
==2116==    by 0x1AE067: print_term_to_buf (print.c:741)
==2116==    by 0x1AF47E: print_term_to_buf (print.c:895)
==2116==    by 0x1B21D8: print_term_to_buf (print.c:1127)
==2116==    by 0x1ABDA3: print_iso_list (print.c:520)
==2116==    by 0x1AE067: print_term_to_buf (print.c:741)
==2116==    by 0x1AF47E: print_term_to_buf (print.c:895)
==2116==    by 0x1B2EFE: print_term_to_stream (print.c:1317)
==2116==    by 0x1D4E88: fn_iso_write_1 (streams.c:1998)
==2116==    by 0x1BC369: start (query.c:1749)
==2116==  Address 0x59f1934 is 3 bytes after a block of size 1,201 alloc'd
==2116==    at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2116==    by 0x1B2E8C: print_term_to_stream (print.c:1313)
==2116==    by 0x1D4E88: fn_iso_write_1 (streams.c:1998)
==2116==    by 0x1BC369: start (query.c:1749)
==2116==    by 0x1BCB84: execute (query.c:1949)
==2116==    by 0x1654CA: run (parser.c:3615)
==2116==    by 0x1B393B: pl_eval (prolog.c:122)
==2116==    by 0x1164F2: main (tpl.c:274)
==2116==
==2116== Invalid write of size 1
==2116==    at 0x4E05F81: __vsnprintf_internal (vsnprintf.c:112)
==2116==    by 0x4DDBDF5: snprintf (snprintf.c:31)
==2116==    by 0x1AEB60: print_term_to_buf (print.c:806)
==2116==    by 0x1B21D8: print_term_to_buf (print.c:1127)
==2116==    by 0x1ABDA3: print_iso_list (print.c:520)
==2116==    by 0x1AE067: print_term_to_buf (print.c:741)
==2116==    by 0x1AF47E: print_term_to_buf (print.c:895)
==2116==    by 0x1B21D8: print_term_to_buf (print.c:1127)
==2116==    by 0x1ABDA3: print_iso_list (print.c:520)
==2116==    by 0x1AE067: print_term_to_buf (print.c:741)
==2116==    by 0x1AF47E: print_term_to_buf (print.c:895)
==2116==    by 0x1B2EFE: print_term_to_stream (print.c:1317)
==2116==  Address 0x59f1934 is 3 bytes after a block of size 1,201 alloc'd
==2116==    at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2116==    by 0x1B2E8C: print_term_to_stream (print.c:1313)
==2116==    by 0x1D4E88: fn_iso_write_1 (streams.c:1998)
==2116==    by 0x1BC369: start (query.c:1749)
==2116==    by 0x1BCB84: execute (query.c:1949)
==2116==    by 0x1654CA: run (parser.c:3615)
==2116==    by 0x1B393B: pl_eval (prolog.c:122)
==2116==    by 0x1164F2: main (tpl.c:274)
==2116==
==2116== Invalid write of size 1
==2116==    at 0x4E05FAB: __vsnprintf_internal (vsnprintf.c:117)
==2116==    by 0x4DDBDF5: snprintf (snprintf.c:31)
==2116==    by 0x1AEB60: print_term_to_buf (print.c:806)
==2116==    by 0x1B21D8: print_term_to_buf (print.c:1127)
==2116==    by 0x1ABDA3: print_iso_list (print.c:520)
==2116==    by 0x1AE067: print_term_to_buf (print.c:741)
==2116==    by 0x1AF47E: print_term_to_buf (print.c:895)
==2116==    by 0x1B21D8: print_term_to_buf (print.c:1127)
==2116==    by 0x1ABDA3: print_iso_list (print.c:520)
==2116==    by 0x1AE067: print_term_to_buf (print.c:741)
==2116==    by 0x1AF47E: print_term_to_buf (print.c:895)
==2116==    by 0x1B2EFE: print_term_to_stream (print.c:1317)
==2116==  Address 0x59f1934 is 3 bytes after a block of size 1,201 alloc'd
==2116==    at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2116==    by 0x1B2E8C: print_term_to_stream (print.c:1313)
==2116==    by 0x1D4E88: fn_iso_write_1 (streams.c:1998)
==2116==    by 0x1BC369: start (query.c:1749)
==2116==    by 0x1BCB84: execute (query.c:1949)
==2116==    by 0x1654CA: run (parser.c:3615)
==2116==    by 0x1B393B: pl_eval (prolog.c:122)
==2116==    by 0x1164F2: main (tpl.c:274)
==2116==
==2116== Invalid write of size 2
==2116==    at 0x4842B33: memmove (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2116==    by 0x1AA12D: plain (print.c:232)
==2116==    by 0x1AEF4F: print_term_to_buf (print.c:844)
==2116==    by 0x1B21D8: print_term_to_buf (print.c:1127)
==2116==    by 0x1ABDA3: print_iso_list (print.c:520)
==2116==    by 0x1AE067: print_term_to_buf (print.c:741)
==2116==    by 0x1AF47E: print_term_to_buf (print.c:895)
==2116==    by 0x1B21D8: print_term_to_buf (print.c:1127)
==2116==    by 0x1ABDA3: print_iso_list (print.c:520)
==2116==    by 0x1AE067: print_term_to_buf (print.c:741)
==2116==    by 0x1AF47E: print_term_to_buf (print.c:895)
==2116==    by 0x1B2EFE: print_term_to_stream (print.c:1317)
==2116==  Address 0x59f1934 is 3 bytes after a block of size 1,201 alloc'd
==2116==    at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2116==    by 0x1B2E8C: print_term_to_stream (print.c:1313)
==2116==    by 0x1D4E88: fn_iso_write_1 (streams.c:1998)
==2116==    by 0x1BC369: start (query.c:1749)
==2116==    by 0x1BCB84: execute (query.c:1949)
==2116==    by 0x1654CA: run (parser.c:3615)
==2116==    by 0x1B393B: pl_eval (prolog.c:122)
==2116==    by 0x1164F2: main (tpl.c:274)
==2116==
==2116== Invalid write of size 1
==2116==    at 0x1AA13B: plain (print.c:233)
==2116==    by 0x1AEF4F: print_term_to_buf (print.c:844)
==2116==    by 0x1B21D8: print_term_to_buf (print.c:1127)
==2116==    by 0x1ABDA3: print_iso_list (print.c:520)
==2116==    by 0x1AE067: print_term_to_buf (print.c:741)
==2116==    by 0x1AF47E: print_term_to_buf (print.c:895)
==2116==    by 0x1B21D8: print_term_to_buf (print.c:1127)
==2116==    by 0x1ABDA3: print_iso_list (print.c:520)
==2116==    by 0x1AE067: print_term_to_buf (print.c:741)
==2116==    by 0x1AF47E: print_term_to_buf (print.c:895)
==2116==    by 0x1B2EFE: print_term_to_stream (print.c:1317)
==2116==    by 0x1D4E88: fn_iso_write_1 (streams.c:1998)
==2116==  Address 0x59f193e is 13 bytes after a block of size 1,201 alloc'd
==2116==    at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2116==    by 0x1B2E8C: print_term_to_stream (print.c:1313)
==2116==    by 0x1D4E88: fn_iso_write_1 (streams.c:1998)
==2116==    by 0x1BC369: start (query.c:1749)
==2116==    by 0x1BCB84: execute (query.c:1949)
==2116==    by 0x1654CA: run (parser.c:3615)
==2116==    by 0x1B393B: pl_eval (prolog.c:122)
==2116==    by 0x1164F2: main (tpl.c:274)
==2116==
==2116== Invalid write of size 1
==2116==    at 0x4E05F81: __vsnprintf_internal (vsnprintf.c:112)
==2116==    by 0x4DDBDF5: snprintf (snprintf.c:31)
==2116==    by 0x1AEFD2: print_term_to_buf (print.c:848)
==2116==    by 0x1B21D8: print_term_to_buf (print.c:1127)
==2116==    by 0x1ABDA3: print_iso_list (print.c:520)
==2116==    by 0x1AE067: print_term_to_buf (print.c:741)
==2116==    by 0x1AF47E: print_term_to_buf (print.c:895)
==2116==    by 0x1B21D8: print_term_to_buf (print.c:1127)
==2116==    by 0x1ABDA3: print_iso_list (print.c:520)
==2116==    by 0x1AE067: print_term_to_buf (print.c:741)
==2116==    by 0x1AF47E: print_term_to_buf (print.c:895)
==2116==    by 0x1B2EFE: print_term_to_stream (print.c:1317)
==2116==  Address 0x59f193e is 13 bytes after a block of size 1,201 alloc'd
==2116==    at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2116==    by 0x1B2E8C: print_term_to_stream (print.c:1313)
==2116==    by 0x1D4E88: fn_iso_write_1 (streams.c:1998)
==2116==    by 0x1BC369: start (query.c:1749)
==2116==    by 0x1BCB84: execute (query.c:1949)
==2116==    by 0x1654CA: run (parser.c:3615)
==2116==    by 0x1B393B: pl_eval (prolog.c:122)
==2116==    by 0x1164F2: main (tpl.c:274)
==2116==
==2116== Invalid write of size 1
==2116==    at 0x4E05FAB: __vsnprintf_internal (vsnprintf.c:117)
==2116==    by 0x4DDBDF5: snprintf (snprintf.c:31)
==2116==    by 0x1AEFD2: print_term_to_buf (print.c:848)
==2116==    by 0x1B21D8: print_term_to_buf (print.c:1127)
==2116==    by 0x1ABDA3: print_iso_list (print.c:520)
==2116==    by 0x1AE067: print_term_to_buf (print.c:741)
==2116==    by 0x1AF47E: print_term_to_buf (print.c:895)
==2116==    by 0x1B21D8: print_term_to_buf (print.c:1127)
==2116==    by 0x1ABDA3: print_iso_list (print.c:520)
==2116==    by 0x1AE067: print_term_to_buf (print.c:741)
==2116==    by 0x1AF47E: print_term_to_buf (print.c:895)
==2116==    by 0x1B2EFE: print_term_to_stream (print.c:1317)
==2116==  Address 0x59f193e is 13 bytes after a block of size 1,201 alloc'd
==2116==    at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2116==    by 0x1B2E8C: print_term_to_stream (print.c:1313)
==2116==    by 0x1D4E88: fn_iso_write_1 (streams.c:1998)
==2116==    by 0x1BC369: start (query.c:1749)
==2116==    by 0x1BCB84: execute (query.c:1949)
==2116==    by 0x1654CA: run (parser.c:3615)
==2116==    by 0x1B393B: pl_eval (prolog.c:122)
==2116==    by 0x1164F2: main (tpl.c:274)
==2116==
==2116== Invalid write of size 1
==2116==    at 0x4E05F81: __vsnprintf_internal (vsnprintf.c:112)
==2116==    by 0x4DDBDF5: snprintf (snprintf.c:31)
==2116==    by 0x1AC98D: print_iso_list (print.c:586)
==2116==    by 0x1AE067: print_term_to_buf (print.c:741)
==2116==    by 0x1AF47E: print_term_to_buf (print.c:895)
==2116==    by 0x1B21D8: print_term_to_buf (print.c:1127)
==2116==    by 0x1ABDA3: print_iso_list (print.c:520)
==2116==    by 0x1AE067: print_term_to_buf (print.c:741)
==2116==    by 0x1AF47E: print_term_to_buf (print.c:895)
==2116==    by 0x1B2EFE: print_term_to_stream (print.c:1317)
==2116==    by 0x1D4E88: fn_iso_write_1 (streams.c:1998)
==2116==    by 0x1BC369: start (query.c:1749)
==2116==  Address 0x59f193e is 13 bytes after a block of size 1,201 alloc'd
==2116==    at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2116==    by 0x1B2E8C: print_term_to_stream (print.c:1313)
==2116==    by 0x1D4E88: fn_iso_write_1 (streams.c:1998)
==2116==    by 0x1BC369: start (query.c:1749)
==2116==    by 0x1BCB84: execute (query.c:1949)
==2116==    by 0x1654CA: run (parser.c:3615)
==2116==    by 0x1B393B: pl_eval (prolog.c:122)
==2116==    by 0x1164F2: main (tpl.c:274)
==2116==
==2116== Invalid write of size 1
==2116==    at 0x4E0C134: _IO_default_xsputn (genops.c:394)
==2116==    by 0x4E0C134: _IO_default_xsputn (genops.c:370)
==2116==    by 0x4DF10FB: __vfprintf_internal (vfprintf-internal.c:1688)
==2116==    by 0x4E05F99: __vsnprintf_internal (vsnprintf.c:114)
==2116==    by 0x4DDBDF5: snprintf (snprintf.c:31)
==2116==    by 0x1AC98D: print_iso_list (print.c:586)
==2116==    by 0x1AE067: print_term_to_buf (print.c:741)
==2116==    by 0x1AF47E: print_term_to_buf (print.c:895)
==2116==    by 0x1B21D8: print_term_to_buf (print.c:1127)
==2116==    by 0x1ABDA3: print_iso_list (print.c:520)
==2116==    by 0x1AE067: print_term_to_buf (print.c:741)
==2116==    by 0x1AF47E: print_term_to_buf (print.c:895)
==2116==    by 0x1B2EFE: print_term_to_stream (print.c:1317)
==2116==  Address 0x59f193e is 13 bytes after a block of size 1,201 alloc'd
==2116==    at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2116==    by 0x1B2E8C: print_term_to_stream (print.c:1313)
==2116==    by 0x1D4E88: fn_iso_write_1 (streams.c:1998)
==2116==    by 0x1BC369: start (query.c:1749)
==2116==    by 0x1BCB84: execute (query.c:1949)
==2116==    by 0x1654CA: run (parser.c:3615)
==2116==    by 0x1B393B: pl_eval (prolog.c:122)
==2116==    by 0x1164F2: main (tpl.c:274)
==2116==
==2116== Invalid write of size 1
==2116==    at 0x4E05FAB: __vsnprintf_internal (vsnprintf.c:117)
==2116==    by 0x4DDBDF5: snprintf (snprintf.c:31)
==2116==    by 0x1AC98D: print_iso_list (print.c:586)
==2116==    by 0x1AE067: print_term_to_buf (print.c:741)
==2116==    by 0x1AF47E: print_term_to_buf (print.c:895)
==2116==    by 0x1B21D8: print_term_to_buf (print.c:1127)
==2116==    by 0x1ABDA3: print_iso_list (print.c:520)
==2116==    by 0x1AE067: print_term_to_buf (print.c:741)
==2116==    by 0x1AF47E: print_term_to_buf (print.c:895)
==2116==    by 0x1B2EFE: print_term_to_stream (print.c:1317)
==2116==    by 0x1D4E88: fn_iso_write_1 (streams.c:1998)
==2116==    by 0x1BC369: start (query.c:1749)
==2116==  Address 0x59f193f is 14 bytes after a block of size 1,201 alloc'd
==2116==    at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2116==    by 0x1B2E8C: print_term_to_stream (print.c:1313)
==2116==    by 0x1D4E88: fn_iso_write_1 (streams.c:1998)
==2116==    by 0x1BC369: start (query.c:1749)
==2116==    by 0x1BCB84: execute (query.c:1949)
==2116==    by 0x1654CA: run (parser.c:3615)
==2116==    by 0x1B393B: pl_eval (prolog.c:122)
==2116==    by 0x1164F2: main (tpl.c:274)
==2116==
==2116== Invalid write of size 1
==2116==    at 0x4E05F81: __vsnprintf_internal (vsnprintf.c:112)
==2116==    by 0x4DDBDF5: snprintf (snprintf.c:31)
==2116==    by 0x1AF5E0: print_term_to_buf (print.c:909)
==2116==    by 0x1B21D8: print_term_to_buf (print.c:1127)
==2116==    by 0x1ABDA3: print_iso_list (print.c:520)
==2116==    by 0x1AE067: print_term_to_buf (print.c:741)
==2116==    by 0x1AF47E: print_term_to_buf (print.c:895)
==2116==    by 0x1B2EFE: print_term_to_stream (print.c:1317)
==2116==    by 0x1D4E88: fn_iso_write_1 (streams.c:1998)
==2116==    by 0x1BC369: start (query.c:1749)
==2116==    by 0x1BCB84: execute (query.c:1949)
==2116==    by 0x1654CA: run (parser.c:3615)
==2116==  Address 0x59f193f is 14 bytes after a block of size 1,201 alloc'd
==2116==    at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2116==    by 0x1B2E8C: print_term_to_stream (print.c:1313)
==2116==    by 0x1D4E88: fn_iso_write_1 (streams.c:1998)
==2116==    by 0x1BC369: start (query.c:1749)
==2116==    by 0x1BCB84: execute (query.c:1949)
==2116==    by 0x1654CA: run (parser.c:3615)
==2116==    by 0x1B393B: pl_eval (prolog.c:122)
==2116==    by 0x1164F2: main (tpl.c:274)
==2116==
==2116== Invalid write of size 1
==2116==    at 0x4E05FAB: __vsnprintf_internal (vsnprintf.c:117)
==2116==    by 0x4DDBDF5: snprintf (snprintf.c:31)
==2116==    by 0x1AF5E0: print_term_to_buf (print.c:909)
==2116==    by 0x1B21D8: print_term_to_buf (print.c:1127)
==2116==    by 0x1ABDA3: print_iso_list (print.c:520)
==2116==    by 0x1AE067: print_term_to_buf (print.c:741)
==2116==    by 0x1AF47E: print_term_to_buf (print.c:895)
==2116==    by 0x1B2EFE: print_term_to_stream (print.c:1317)
==2116==    by 0x1D4E88: fn_iso_write_1 (streams.c:1998)
==2116==    by 0x1BC369: start (query.c:1749)
==2116==    by 0x1BCB84: execute (query.c:1949)
==2116==    by 0x1654CA: run (parser.c:3615)
==2116==  Address 0x59f1940 is 15 bytes after a block of size 1,201 alloc'd
==2116==    at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2116==    by 0x1B2E8C: print_term_to_stream (print.c:1313)
==2116==    by 0x1D4E88: fn_iso_write_1 (streams.c:1998)
==2116==    by 0x1BC369: start (query.c:1749)
==2116==    by 0x1BCB84: execute (query.c:1949)
==2116==    by 0x1654CA: run (parser.c:3615)
==2116==    by 0x1B393B: pl_eval (prolog.c:122)
==2116==    by 0x1164F2: main (tpl.c:274)
==2116==
==2116== Invalid write of size 1
==2116==    at 0x4E05F81: __vsnprintf_internal (vsnprintf.c:112)
==2116==    by 0x4DDBDF5: snprintf (snprintf.c:31)
==2116==    by 0x1AF5E0: print_term_to_buf (print.c:909)
==2116==    by 0x1B2EFE: print_term_to_stream (print.c:1317)
==2116==    by 0x1D4E88: fn_iso_write_1 (streams.c:1998)
==2116==    by 0x1BC369: start (query.c:1749)
==2116==    by 0x1BCB84: execute (query.c:1949)
==2116==    by 0x1654CA: run (parser.c:3615)
==2116==    by 0x1B393B: pl_eval (prolog.c:122)
==2116==    by 0x1164F2: main (tpl.c:274)
==2116==  Address 0x59f1941 is 16 bytes after a block of size 1,201 alloc'd
==2116==    at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2116==    by 0x1B2E8C: print_term_to_stream (print.c:1313)
==2116==    by 0x1D4E88: fn_iso_write_1 (streams.c:1998)
==2116==    by 0x1BC369: start (query.c:1749)
==2116==    by 0x1BCB84: execute (query.c:1949)
==2116==    by 0x1654CA: run (parser.c:3615)
==2116==    by 0x1B393B: pl_eval (prolog.c:122)
==2116==    by 0x1164F2: main (tpl.c:274)
==2116==
==2116== Invalid write of size 1
==2116==    at 0x4E05FAB: __vsnprintf_internal (vsnprintf.c:117)
==2116==    by 0x4DDBDF5: snprintf (snprintf.c:31)
==2116==    by 0x1AF5E0: print_term_to_buf (print.c:909)
==2116==    by 0x1B2EFE: print_term_to_stream (print.c:1317)
==2116==    by 0x1D4E88: fn_iso_write_1 (streams.c:1998)
==2116==    by 0x1BC369: start (query.c:1749)
==2116==    by 0x1BCB84: execute (query.c:1949)
==2116==    by 0x1654CA: run (parser.c:3615)
==2116==    by 0x1B393B: pl_eval (prolog.c:122)
==2116==    by 0x1164F2: main (tpl.c:274)
==2116==  Address 0x59f1942 is 17 bytes after a block of size 1,201 alloc'd
==2116==    at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2116==    by 0x1B2E8C: print_term_to_stream (print.c:1313)
==2116==    by 0x1D4E88: fn_iso_write_1 (streams.c:1998)
==2116==    by 0x1BC369: start (query.c:1749)
==2116==    by 0x1BCB84: execute (query.c:1949)
==2116==    by 0x1654CA: run (parser.c:3615)
==2116==    by 0x1B393B: pl_eval (prolog.c:122)
==2116==    by 0x1164F2: main (tpl.c:274)
==2116==
dict([valid: @true,fields:dict([revision:dict([valid: @true,ignored: @true]),creator:dict([valid: @true,ignored: @true]),updated_at:dict([valid: @true,ignored: @true]),created_at:dict([valid: @true,ignored: @true]),qux_abbreviation:dict([valid: @true]),qux_name:dict([valid: @false,error:256文字以下の値を入力してください]),baz_id:dict([valid: @false,error:256文字以下の値を入力してください]),owner_id:dict([valid: @true,ignored: @true]),qux_type_code:dict([valid: @true]),foo:dict([fields:dict([foo/bar_groups:dict([valid: @true]),foo/acme_notification_url:dict([valid: @false,error:「https://」で始まらなければなりません]),foo/qux_api_version:dict([valid: @true]),foo/xyz_unit_id:dict([valid: @false,error:256文字以下 の値を入力してください]),foo/abc_label:dict([fields:dict([foo/abc_label/print_abc_list:dict([valid: @true]),foo/abc_label/report_id:dict([valid: @true]),foo/abc_label/report_setting_key:dict([valid: @true])]),valid: @true]),foo/xyz_notes:dict([v==2116== Invalid read of size 1
==2116==    at 0x4E0C01E: _IO_default_xsputn (genops.c:399)
==2116==    by 0x4E0C01E: _IO_default_xsputn (genops.c:370)
==2116==    by 0x4E096F9: _IO_new_file_xsputn (fileops.c:1265)
==2116==    by 0x4E096F9: _IO_file_xsputn@@GLIBC_2.2.5 (fileops.c:1197)
==2116==    by 0x4DFD3C0: fwrite (iofwrite.c:39)
==2116==    by 0x15286D: net_write (network.c:330)
==2116==    by 0x1B2F27: print_term_to_stream (print.c:1321)
==2116==    by 0x1D4E88: fn_iso_write_1 (streams.c:1998)
==2116==    by 0x1BC369: start (query.c:1749)
==2116==    by 0x1BCB84: execute (query.c:1949)
==2116==    by 0x1654CA: run (parser.c:3615)
==2116==    by 0x1B393B: pl_eval (prolog.c:122)
==2116==    by 0x1164F2: main (tpl.c:274)
==2116==  Address 0x59f1931 is 0 bytes after a block of size 1,201 alloc'd
==2116==    at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2116==    by 0x1B2E8C: print_term_to_stream (print.c:1313)
==2116==    by 0x1D4E88: fn_iso_write_1 (streams.c:1998)
==2116==    by 0x1BC369: start (query.c:1749)
==2116==    by 0x1BCB84: execute (query.c:1949)
==2116==    by 0x1654CA: run (parser.c:3615)
==2116==    by 0x1B393B: pl_eval (prolog.c:122)
==2116==    by 0x1164F2: main (tpl.c:274)
==2116==
alid: @true])]),valid: @true]),system_id:dict([valid: @false,error:256文字以下の値を入力してください]),qux_status:dict([valid: @true])]),spec:dict([type:model,model:qux_config])])==2116==
==2116== HEAP SUMMARY:
==2116==     in use at exit: 11,526 bytes in 68 blocks
==2116==   total heap usage: 5,701 allocs, 5,633 frees, 56,428,199 bytes allocated
==2116==
==2116== 3,299 (2,960 direct, 339 indirect) bytes in 10 blocks are definitely lost in loss record 3 of 4
==2116==    at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2116==    by 0x154DC9: directives (parser.c:397)
==2116==    by 0x161B9A: process_term (parser.c:2887)
==2116==    by 0x16283E: tokenize (parser.c:3046)
==2116==    by 0x14FF6A: load_text (module.c:1276)
==2116==    by 0x156E29: directives (parser.c:678)
==2116==    by 0x161B9A: process_term (parser.c:2887)
==2116==    by 0x16283E: tokenize (parser.c:3046)
==2116==    by 0x14FF6A: load_text (module.c:1276)
==2116==    by 0x156E29: directives (parser.c:678)
==2116==    by 0x161B9A: process_term (parser.c:2887)
==2116==    by 0x16283E: tokenize (parser.c:3046)
==2116==
==2116== 8,227 (7,104 direct, 1,123 indirect) bytes in 24 blocks are definitely lost in loss record 4 of 4
==2116==    at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2116==    by 0x154DC9: directives (parser.c:397)
==2116==    by 0x161B9A: process_term (parser.c:2887)
==2116==    by 0x16283E: tokenize (parser.c:3046)
==2116==    by 0x14FF6A: load_text (module.c:1276)
==2116==    by 0x1B553E: pl_create (prolog.c:533)
==2116==    by 0x115C14: main (tpl.c:165)
==2116==
==2116== LEAK SUMMARY:
==2116==    definitely lost: 10,064 bytes in 34 blocks
==2116==    indirectly lost: 1,462 bytes in 34 blocks
==2116==      possibly lost: 0 bytes in 0 blocks
==2116==    still reachable: 0 bytes in 0 blocks
==2116==         suppressed: 0 bytes in 0 blocks
==2116==
==2116== For lists of detected and suppressed errors, rerun with: -s
==2116== ERROR SUMMARY: 54 errors from 23 contexts (suppressed: 0 from 0)

Thanks a bunch, works now!


fix: avoid endless loop when handling symlinks

This resolves #108 for me. I tried to run tests locally to see if this breaks anything, but I guess the tests need to run in CI as they were failing with or without this change on my environment.
Fixes #108

@q0rban I'm not sure that just removing code is the correct way to handle this. You should handle the case of infinite loop like keeping track of paths to be sure you don't hit the same path twice to break the loop

The interesting thing is that the code is already keeping track of the paths. I couldn't figure out a scenario where this code wouldn't result in an infinite loop, which is why I deleted it. But then I realize it is assuming that the paths are not nested at the same level. So I believe the code will work if path/to/some/symlink/index.js is symlinked to path/to/real/index.js (since they have different nesting levels), but I believe it will always loop if path/to/symlink/index.js is linked to path/to/real/index.js since they have the same nesting depth.
Furthermore, I don't believe this function will properly handle the scenario where there are multiple symlinks involved. For example, let's say module chicken is symlinked wholesale into another module:
path/to/node_modules/some-module/node_modules/symchicken -> path/to/node_modules/some-module/node_modules/chicken

but then, what happens if chicken has some links in it as well?
chicken/lib/link.js -> chicken/deeply/nested/index.js

This would result in
path/to/node_modules/some-module/node_modules/symchicken/lib/index.js -> path/to/node_modules/some-module/node_modules/chicken/deeply/nested/index.js

Does that mean this logic needs to recurse? The mind reels.

Incidentally, do you have any tips on running tests locally? I don't see any documentation on that in the repo.

@robertsLando updated the approach and would appreciate your thoughts. The existing logic seems error prone, and thus difficult to ascertain the intent of it.

See https://github.com/yao-pkg/pkg/pull/111/files

Also please fix lint issues: yarn fix

Could you also add a test that covers this?
test-99-#108

Whoops, apologies, I missed this. Will take a look at what this entails.

Okay, test added. I can confirm that the test just hangs on the main branch, and builds properly on this branch.

See review above

I know, I requested another review after pushing fixes based on that review. :)

Found improvements on how test works? 😆 Spent 2 days trying to improve them


Write should return a token that is used as input to read

Currently the design is always operating on a single active call operation per session. Is that really sufficient? Will we never need parallel calls? From my perspective, it also makes the API difficult to understand ("Read what?").
If actual parallel calls is too complicated to implement right now, would it work to use a token but explicitly only allow one call to be in progress (i.e., if token != _active_token return BAD_TOKEN)

You're also not supporting multiple simultaneous sessions, but at least it looks like that could be supported with the current API (only implementation changes).

Errors could have a similar problem. Our driver APIs are implemented with something like:

Get the last error for the given session.
If no session is provided, get the last error for the current thread.

Ideally (2) is never needed or only needed on failed Initialize. And (1) is pretty important if you have simultaneous sessions.
So, you could add a session param to GetError to address that from an API perspective. And you could just assert that the session in is the one-and-only globally active session if you want to restrict that (for now).
This seems like less of an issue. Creating a future alternate API for GetErrorForSession is probably fine.

Combining this feedback with #6 we can make a StartCall method that returns a token to be passed to FinishCall. I also think a `BlockingCall' method would be nice to have.

closing as resolved


Can't set host price via HTTP API

Connecting to http://localhost:9980/host/config?price=5 returns a malformed price error.
I think it's happening during the casting of the string from the url ("5" in this case) to whatever type price actually is, relevant code in apihost.go...
func (d *daemon) hostConfigHandler(w http.ResponseWriter, req *http.Request) {
	// load current settings
	config := d.host.Info().HostSettings

	// map each query string to a field in the host announcement object
	qsVars := map[string]interface{}{
		"totalstorage": &config.TotalStorage,
		"minfilesize":  &config.MinFilesize,
		"maxfilesize":  &config.MaxFilesize,
		"minduration":  &config.MinDuration,
		"maxduration":  &config.MaxDuration,
		"price":        &config.Price,
		"collateral":   &config.Collateral,
	}

	for qs := range qsVars {
		// only modify supplied values
		if req.FormValue(qs) != "" {
			_, err := fmt.Sscan(req.FormValue(qs), qsVars[qs])
			if err != nil {
				writeError(w, "Malformed "+qs, 400)
				return
			}
		}
	}

	d.host.SetConfig(config)
	writeSuccess(w)
}

Yep, scanning broke when the Currency type was implemented. I've fixed this on my own development branch, so we should be able to close this once my branch is merged.


feat: update drive sync and allow file deletion

Pre-Flight checklist

[x] Testing

[x] Code is tested
[x] Changes have been manually tested

What are the relevant tickets?
Closes https://github.com/mitodl/ocw-studio/issues/1555
Relevant https://github.com/mitodl/hq/discussions/359
What's this PR do?
This PR:

Updates the website GDrive sync task
Deletes files that no longer exist in GDrive and aren't being used on any page

It was necessary to delete existing files to achieve the effect expected in the linked ticket. Since I was writing the delete logic, I also considered https://github.com/mitodl/hq/discussions/359 in the implementation.
Note: Files that are used in content other than page will also be deleted.

The rest of the details can be read from the issue comments.

Any background context you want to provide?
Might be irrelevant but in case you wonder, I experimented with the changes API and decided not to use it for two reasons.

The API does not provide any filtering options.
A Change can be removed from the list because of a user action. The documentation does not specify a strict set of reasons.
In the end, even if possible, it would've been more complicated to implement.

How should this be manually tested?

Download the folder here
Head to your local ocw-studio setup
Checkout this branch hussaintaj/1555-gdrive-sync
Start ocw-studio and create a test course.
Open the folder you downloaded in step 1. Open the step01 subdirectory. Upload these files to the associated course's GDrive folder final_files.
In ocw-studio, open the resources tab of your test course and hit Sync w/ GDrive.
Expected Behaviour: You should see 5 files after a successful sync.
Click on image3.png and add something in the description field. This will act as a marker that we will use later.
Create a page in your test course. In the body, embed image5.png. Save the page.
Finish adding the metadata for the course and then publish it to draft or live, or both.
Open your local instance of MinIO. Log in and navigate to your course files.
Expected Result: You should see the synced files in the static_resources, and content in the resources directories.
Open the GDrive folder from step 5. Delete the following files image3.png, image4.png, image5.png.
Open the folder you downloaded in step 1. Open the step02 subdirectory.

Upload image2.png. Gdrive will ask you if you want to replace it. Say yes.
Upload image3.png.

Repeat step 6
After sync. Expected Result

image4.png is removed from the resources file.
image5.png is not removed. Because it was used in step 9.
Click image3.png, the description should have the content you added in step 8.

 Restore image3.png from step 13.
Repeat step 6
After sync. Expected Result: You should see two image3.png.
Publish the course again.
Repeat step 11 and 12. Make sure all the changes in the files are reflected properly. Note: One file will be named image32.png
That's it for this systematic testing. Feel free to go crazy and try other things that you can think of.

Codecov Report

Merging #1724 (8da2bfb) into master (a747fc3) will decrease coverage by 2.03%.
The diff coverage is 86.66%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@            Coverage Diff             @@
##           master    #1724      +/-   ##
==========================================
- Coverage   87.43%   85.40%   -2.03%
==========================================
  Files         247      123     -124
  Lines       10608     6961    -3647
  Branches     1796      942     -854
==========================================
- Hits         9275     5945    -3330
+ Misses       1171      874     -297
+ Partials      162      142      -20

Impacted Files
Coverage Δ

gdrive_sync/api.py
91.38% <82.97%> (-1.20%)
:arrow_down:

gdrive_sync/tasks.py
97.22% <90.69%> (-0.36%)
:arrow_down:

... and 125 files with indirect coverage changes
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Accidentally put on hold, so removed it

Thank you for testing this.

When videos are deleted, should the corresponding Youtube videos be deleted or made private? Should the video files (including transcodes) and transcripts on S3 be deleted?

Ideally, yes, they should be deleted since we would have no way to track them after a resource is deleted. I'm triggering the delete logic that already existed, I'll take a look at the implementation to verify if youtube videos and transcripts are deleted.

When videos are deleted, should the corresponding Youtube videos be deleted or made private? Should the video files (including transcodes) and transcripts on S3 be deleted?

Ideally, yes, they should be deleted since we would have no way to track them after a resource is deleted. I'm triggering the delete logic that already existed, I'll take a look at the implementation to verify if youtube videos and transcripts are deleted.

I did a bit of digging, the youtube video should be deleted with the video object thanks to the:
https://github.com/mitodl/ocw-studio/blob/b6c2a4df416d18afdfd45456892b850d582c292b/videos/signals.py#L11
If captions or transcripts are added manually through drive sync, those will be deleted when the user deletes the files from the drive.
For 3play generated transcripts, I'm not sure if they are being deleted. I'll look into it a bit more.

I've added a receiver to delete transcript files from Video file fields.
I don't think we can delete entities from 3play since it provides no API from what I can tell from the docs.
I don't see any other location a transaction/caption might be kept.

Thank you for the review @mbertrand. I've implemented your suggestions and transcript deletion (as described here https://github.com/mitodl/ocw-studio/pull/1724#issuecomment-1511177607). Please have another look.
I've tried to cover as many cases as I could think of. It's a complex piece of code and we want to make sure we don't miss something. I'd appreciate it if you could specifically test the video files to the best of your knowledge.


[9.x] DOC BLOCK params fix on orderBy methods at Query Builder stub file

[ ] Added or updated tests
[ ] Documented user facing changes
[ ] Updated CHANGELOG.md

As per the laravel 9.18 updates, they changed the comment of the orderBy and orderByDesc method you can check
Changes
Just changed the wrong comment.

I think Eloquent Builder's genericness causes unit tests to fail.

I am new to this package and I try to check the test flow it's hard to understand directly.
Can anyone tell me how we are managing tests in this package?
Otherwise, I need to create an issue and close this PR accordingly.

Can anyone tell me how we are managing tests in this package?

Simply run PHPUnit on your local machine.

Hi,
You can just change \Illuminate\Database\Eloquent\Builder to \Illuminate\Database\Eloquent\Builder<Illuminate\Database\Eloquent\Model This should be enough to fix the errors.
To run tests locally you try using composer test:unit

@canvural
I did this already but still, 1 test is failing so I am researching for that.
request-object.php file line number 18
Error response
-'Illuminate\Foundation\Auth\User|null'
+'App\User|null'

That is fine locally.

Thank you!