reload_data on time intervals

Hi there,
First, thanks for this project, I really like the clean UI 👍
I'm importing my calendar via ics_event_files and want to keep calcure running during the day. Is there any functionality to refresh the data in certain time intervals (e.g. every 15mins) so changed events are reflected automatically?

Hi, thank you.
No, but you can manually reload with Q.
Due to the design of the software, it is difficult to implement good auto refresh. Plus, even if it was there, it would cause some troubles if there's no internet or freeze when the ics file is large. So we decided to keep it as manual update.
P.s. if you or someone else would like to work on it, I'll gladly consider good implementations of this feature.

Without doing a bigger change to the current code base, what do you think about the following idea to solve the freeze problem?

a new class icalsync.py takes care about downloading all ics files that are located remotely and stores it into ~/.config/calcure/ical, eventually separated in additional subfolders calendars and tasks

this process could run as an async thread
it takes also a reference of screen to signal that a reload_data is needed

the purpose of your existing loader.py slightly changes, it will only read the files that have been stored into the new cache directory ~/.config/calcure/ical, this ensures that the loader will always execute fast enough

open for discussion if "local" ics files that are referenced in the config will still be handled by loader.py or if icalsync.py will also take care about those and copy such files into the new cache directory

refreshing (downloading again) the data via icalsync.py can be repeated and the reload_data status set when this process has finished in the background

Sorry for the delay, I've been thinking about this solution. And while it's a reasonable solution, basically I think it's a bit overkill for a rather niche problem. I'd say, let's just make a simple auto reloading at set time intervals, with 0 (no reload) as the default. I imagine not many people need this feature and even fewer people have such a huge ics files that they need caching, and even if they have, they can wait once in an hour, as it happens passively on the background. Moreover, such people can solve freezing problem by using vdisyncer or other sync solutions. I think calcure should not take advanced syncing functionality at least at this stage.
So, I'll come from vacation next week and will try a simple implementation of this feature, then we can see that maybe it's enough.

Okay, I implemented the data reload on intervals, so that not there is a parameter data_reload_interval = 0 (in minutes, with 0 to switch off) that controls how often data reload. Actually, it's not precise, as the program itself is reloading every 20 seconds, but for our purpose it is precise enough, I think. It is now in the main branch, and will be available in the next release.

Thanks, that's enough for my working dashboard to update in regular intervals :)


Flags for Database.open()

Please, allow to use other flags in Database.open() method like SQLITE_OPEN_FULLMUTEX.

Do you only need the mutex flags? I'm reluctant to expose the flag int as a parameter since the library sets most flags internally (for the open mode and uri vs file). AFAIK parameters controlling visibility of the cache can also be set via a uri parameter, so mutex and and SQLITE_OPEN_NOFOLLOW are the only ones remaining.

For my purposes, yes, i'd like to be sure my sqlite has serialized access because it is used from different isolates.

I added a mutex option for this, I'll release the next version soon.

i'd like to be sure my sqlite has serialized access because it is used from different isolates

Unless your sharing a database pointer across isolates (don't do this!), the default multi-threading mode is fine. You won't need a full mutex if each isolate opens a database by itself, even if they all point to the same file.

I added a mutex option for this, I'll release the next version soon.
Thanks, awesome.

i'd like to be sure my sqlite has serialized access because it is used from different isolates

Unless your sharing a database pointer across isolates (don't do this!), the default multi-threading mode is fine. You won't need a full mutex if each isolate opens a database by itself, even if they all point to the same file.

That would solve my problem, but i am not sure it is easy to do. I can run only global or static functions with simple args in the isolate, so no state. Does an isolate have access to a static object property created in other isolates?

Does an isolate have access to a static object property created in another isolate?

It does not, isolates have separate memory and only communicate by sending messages.

I can run only global or static functions with simple args in the isolate, so no state

Exactly, that's why you won't accidentally use a Database object across multiple isolates. So you might not need the full mutex option at all. Regardless, it's available in version 0.18 of this package.


What is the difference between the white balance and the color constancy?

Hi, thanks for your question.
Color constancy aims at rendering original object colors regardless of scene lighting conditions. The white balance goal, on the other hand, is to render achromatic colors (i.e., greyish) properly. White balance is commonly assumed to achieve the goal of color constancy, but this is just an assumption. Because it is typically implemented by a diagonal correction (i.e., a linear scale of each color channel independently), which apparently can't fix all colors perfectly. Hope you find this helpful.


The images have disappeared again

Say no more..even the repository is missing...days wasted from my miserable life

Hi
Try images on dockerhub for replacement.
https://github.com/Doublemine/gcr.io-mirror/blob/master/cloud-solutions-images.md


Updating to Wrong Version Bug

I left the title kind of vague because the bug is rather strange and hard to explain.
Luma3DS v5.5 just released today, and I attempted to switch from the unstable build (which includes the commit in the version) to the stable, and doing so, it seems to switch me to v5.4-859944dc, which is not the latest stable, and is, in fact, not the stable branch at all.
However, when I chose unstable, it then updated me to the latest stable build without the commit in the version, so I'm not exactly what the cause of this, possibly something server side? ¯_(ツ)_/¯

Almost definitely something server side. Downloading the latest release from my site manually gives me v5.4-859944dc.
It's definitely something server side, probably something borked with lastVer checking. Either that or Aurora commited something weirdly and caused my (rather hacky) lastVer checking to fail.
I'll fix it manually for now and close this, I'd prefer this issue to be re-opened on my site's github.


cmd/vet: warn about slice being compared to nil

What version of Go are you using (go version)?
go version go1.12.7 windows/amd64
Does this issue reproduce with the latest release?
yes
What did you do?
I reported bugs #33103, #33104 and #33105. In the changes to fix these bugs, I noticed a common pattern: a slice is checked for being nil instead of checking its length.
I had thought that len(slice) == 0 would be the canonical way of testing a slice for emptiness, and I was surprised to find nothing about this topic in Effective Go.
I was also surprised that this bug has occurred in go/printer, which is written by the core go team. Is there a specific reason for using the nil check instead of the len check?
To prevent this kind of bugs in the future, there should be some tool that warns about this situation. My first idea was to integrate this check into vet. Since vet reports suspicious constructs, this seems to fit perfectly.

https://play.golang.org/p/MwAH4Y57HL3
Good catch!  Could be a regression from earlier go versions though.

@av86743 there is no regression here, everything is working as expected.
@rillig I don't think that this is something that vet can flag without causing false positives. Empty slices and nil slices are semantically different. A lot of places do not (or should not) care about the distinction, but some places do. For example, a nil slice can denote the difference between an empty list and the absence of a list, or a message with an empty body vs a message with no body.

@rillig welcome to golang.

@rillig I don't think that this is something that vet can flag without causing false positives. Empty slices and nil slices are semantically different. A lot of places do not (or should not) care about the distinction, but some places do. For example, a nil slice can denote the difference between an empty list and the absence of a list, or a message with an empty body vs a message with no body.

Yes, I know that this is difficult to decide. But there should at least be some tool that can flag this. For humans this is difficult since expr == nil needs type analysis to see whether expr is a slice or not. And even if the tool merely says "hey, dear human, have a look at these few code snippets, they might be wrong", that would already help.
The point is that this oversight caused some real bugs, and there are more bugs of this kind lying around, waiting to be discovered.

I quickly hacked together the nilslice check, which found these possible issues:
$ ~/git/go/bin/go vet
# go/printer
.\nodes.go:76:5: comparison of slice comments == nil should be len(comments) == 0
.\nodes.go:395:16: comparison of slice Names == nil should be len(Names) == 0
.\nodes.go:1231:6: comparison of slice Results != nil should be len(Results) > 0
.\nodes.go:1277:6: comparison of slice List != nil should be len(List) > 0
.\nodes.go:1393:6: comparison of slice Values != nil should be len(Values) > 0
.\nodes.go:1429:5: comparison of slice Values != nil should be len(Values) > 0
.\nodes.go:1511:6: comparison of slice Values != nil should be len(Values) > 0
.\printer.go:1081:5: comparison of slice comments != nil should be len(comments) > 0
.\printer.go:1120:22: comparison of slice comments == nil should be len(comments) == 0

The one in line 1277 is the one from #33103.
The one in line 395 looks a lot like #33104.
The one in line 1511 is the one from #33105.
I don't know how to trigger the other comparisons, but these examples alone look like it would be worthwhile to apply this check to the whole go tree, at least once.

On the topic of #33103, I do not believe that your fix in go/printer is correct. Rather, this looks like a bug in go/parser, as the list really should be nil, not empty. In a similar vein, all the comparisons with nil in nodes.go should be correct (assuming no other bugs in go/parser.)
nodes.go:76 is a false positive independent of the AST parsing issue, it lazily allocates a slice, it really does mean to check for nil.

... It seems likely that a -> a is incorrectly turning the nil slice into an empty slice, thus breaking an invariant of the AST. ...

@dominikh
So you do have AST invariants (for golang.)  Do you also have a tool/package which validates that these are satisfied?


Truncate Operator failing for Bigquery

Found below error when running Truncate Op. for Bigquery table.
FAILED    [ 25%][2022-04-13 03:22:07,787] {base_executor.py:82} INFO - Adding to queue: ['<TaskInstance: qzf6u3usex1b9ig6.test_stats_check_1_1649800307_truncate backfill__2016-01-01T00:00:00+00:00 [queued]>']
[2022-04-13 03:22:12,805] {taskinstance.py:1429} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=qzf6u3usex1b9ig6
AIRFLOW_CTX_TASK_ID=test_stats_check_1_1649800307_truncate
AIRFLOW_CTX_EXECUTION_DATE=2016-01-01T00:00:00+00:00
AIRFLOW_CTX_DAG_RUN_ID=backfill__2016-01-01T00:00:00+00:00
[2022-04-13 03:22:12,811] {credentials_provider.py:312} INFO - Getting connection using `google.auth.default()` since no key file is defined for hook.
[2022-04-13 03:22:16,637] {credentials_provider.py:312} INFO - Getting connection using `google.auth.default()` since no key file is defined for hook.
[2022-04-13 03:22:21,021] {taskinstance.py:1718} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/Users/utkarsharma/sandbox/astronomer/astro/.nox/dev/lib/python3.8/site-packages/google/cloud/bigquery/dbapi/cursor.py", line 203, in _execute
    self._query_job.result()
  File "/Users/utkarsharma/sandbox/astronomer/astro/.nox/dev/lib/python3.8/site-packages/google/cloud/bigquery/job/query.py", line 1498, in result
    do_get_result()
  File "/Users/utkarsharma/sandbox/astronomer/astro/.nox/dev/lib/python3.8/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
    return retry_target(
  File "/Users/utkarsharma/sandbox/astronomer/astro/.nox/dev/lib/python3.8/site-packages/google/api_core/retry.py", line 189, in retry_target
    return target()
  File "/Users/utkarsharma/sandbox/astronomer/astro/.nox/dev/lib/python3.8/site-packages/google/cloud/bigquery/job/query.py", line 1488, in do_get_result
    super(QueryJob, self).result(retry=retry, timeout=timeout)
  File "/Users/utkarsharma/sandbox/astronomer/astro/.nox/dev/lib/python3.8/site-packages/google/cloud/bigquery/job/base.py", line 728, in result
    return super(_AsyncJob, self).result(timeout=timeout, **kwargs)
  File "/Users/utkarsharma/sandbox/astronomer/astro/.nox/dev/lib/python3.8/site-packages/google/api_core/future/polling.py", line 135, in result
    raise self._exception
google.api_core.exceptions.BadRequest: 400 DELETE must have a WHERE clause at [1:1]

closed by https://github.com/astro-projects/astro/pull/363


Added image url of product to output.

I needed images of products and spent about 3 hours figuring it out. Image url is one of 4 in the images key. I have chosen the first index wich has the highest resolution.
There is mostlikely a better way of doing this, but this works, feel free to do whatever you think would be a better addition.

Very nice @HakanKartal! Does every product have a product image? If not, perhaps it should account for the case that the field is undefined for some products. If all products have an image, it should be fine like this.

Hey @rkalis!
Sorry for my delayed response!
I've since noticed that indeed (for some reason) not every response contains an image, or any sort of static link for said product (despite the product having one on ah.nl). However, every possible response should have one. That being said, 50 out of 21909 products dont seem to contain an image, although this number decreases by the day. I've added a case for it nonetheless.

Thanks for looking into it and adding these changes to the PR. I'll merge and publish them.

It took a bit longer to publish it to PyPI. But v0.2.1 is live (https://pypi.org/project/appiepy/0.2.1/).


Rephrase comment about Local for clarity. (Adopted)

Objective

Finish PR #10322

Solution

Rebase changes in PR #10322 and format the changes.

Several unrelated changes had to be made to please the CI. I do not know why.

I guess new rust version just released or something like that.
Can you remove these changes please? They are handled in the separate PR: https://github.com/bevyengine/bevy/pull/11123


Rules with exceptions

Hello!
I got a question and can not find the resolution I would not doubt in.
The situation is following:
Different hosts has different "normal" values of generic metrics, let's say Load Average:

worker-class hosts LA should always be above "1.0 per cpu"
storage-class hosts LA should always be relatively low
database-class hosts LA should be low too but high peaks are possible
supervisor-class hosts LA is notably higher than "1.0" and that's OK
etc

Which is the correct way to form LA alert then?
Simple .warn(lambda: "mean" > 1.0) doesn't seems suitable cause of different hosts classes. Backup hosts would generate nighly false-positive alerts when backup process occur.
It's also annoying to see tens of alerts when overall load of worker-class hosts is high.
So the questions are:

What is best practice to separate different alert profiles? Should I just send load_profile value for each host or probably include some smart separation logic into TICK scripts?
Is it possible to somehow unite notable amount of alerts into single meta-alert? Like, not 10 times "web server is overloaded" but "web cluster looks overloaded", which would suppress "childs".

Appreciate any ideas. Thank you for your time.

Based on your description of the differences between the different classes of hosts I would recommend using different tasks.
Are your hosts tagged with their class? If so then creating separate tasks for each would be simple.
An alternative would be to use templates, see https://docs.influxdata.com/kapacitor/v1.1/examples/template_tasks/ Templates allow for creating many similar tasks by specifying variables for each task. If the only things that differed across classes where basic threshold I would recommend templates. But since you also want to deal with differences like peaks then using different tasks provides to most flexibility to address those issues.
As for your second question yes, it is possible to alert on aggregates across groups of servers instead of single servers. Currently there is not easy way to suppress the child alerts other than joining the alerts together which adds a bit of complexity.

@nathanielc
Thank you for brilliantly clear answer again, you guys are great!
Templates are cool for simple things and I used them at the very beginning, but since now I'm separating alerts not with just different limits - flexibility becomes complexity. :(
Looks like I should end up adding some kind of "class" tag to hosts to simplify the tasks at kapacitor level. Otherwise tasks would be too complex. After couple days of working on another things, this idea seems to be fairly elegant. At first sight I was in doubt.
Thank you again, you added enough confidence!

@agrrh
Were you able to accomplish the goal.
I too have a situation where i want different thresholds for different hosts.
Should we use sideload node or Templates?

@agrrh
Were you able to accomplish the goal.
I too have a situation where i want different thresholds for different hosts.
Should we use sideload node or Templates?


Update Dockerfile to include curl in final build image

As per discussion on https://github.com/skrashevich/double-take/discussions/130#discussioncomment-7176323

OK, I'll admit to being confused as in the Docker Build logs it does reference curl being installed, but it very definitely doesn't seem to be anywhere when I connect to the image...

Could you tell me the path to curl in the release image?


Broken anchor links for headings in CONTRIBUTE.md

Describe your problem and how to reproduce it:
Anchor links such is not working properly in several languages.
For sure is not working in: Russian, Spanish, Chinese languages.
Add a Link to the page with the problem: wiki/russian/contribute.md
Tell us about your browser and operating system:

Browser Name: Chrome
Browser Version: Version 74.0.3729.169 (Official Build) (64-bit)
Operating System: Windows 10 Pro 1809

If possible, add a screenshot here (you can drag and drop, png, jpg, gif, etc. in this box):

These 3/4 PRs (exlude one - which I PR'd to my own forked-repo) may close this issue.
Please review them.


Add comments for solutions

Codecov Report

Merging #172 (a097f24) into master (3dea991) will not change coverage.
The diff coverage is n/a.

:exclamation: Current head a097f24 differs from pull request most recent head 4f6a0f0. Consider uploading reports for the commit 4f6a0f0 to get more accurate results

@@            Coverage Diff            @@
##             master     #172   +/-   ##
=========================================
  Coverage     71.14%   71.14%
  Complexity      783      783
=========================================
  Files           111      111
  Lines          2596     2596
  Branches        340      340
=========================================
  Hits           1847     1847
  Misses          639      639
  Partials        110      110

Impacted Files
Coverage Δ

.../seedu/address/logic/parser/AddressBookParser.java
53.84% <ø> (ø)

...n/java/seedu/address/model/person/Appointment.java
91.08% <ø> (ø)

...du/address/model/person/UniqueAppointmentList.java
79.31% <ø> (ø)

...a/seedu/address/model/person/UniquePersonList.java
94.44% <ø> (ø)

src/main/java/seedu/address/ui/CommandBox.java
0.00% <ø> (ø)

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Default Theme

Is your feature request related to a problem? Please describe.
Problem: When installing rimble-ui for the first time as a dependency into a project there is not a default theme included.
Describe the solution you'd like
When installing rimble for the first time implement with a default theme.
Describe alternatives you've considered
Update install directions to include theme.

Document how to extend theme, import {theme}
https://github.com/ConsenSys/rimble-ui/blob/master/example/.storybook/config.js


Change the bond options from hashmap to struct members.

Allowing arbitrary bond options in hashmap is not a good API.
Should use Option<> to hide the invalid options.

@cathay4t @ffmancera I would like to work on this issue.

Sure!

Oops it seems there has been a little bit of overlap on this issue. https://github.com/nispor/nispor/commit/6ebf64d4ee6f45ef0015212d4b917fe9054ae6c1 @cathay4t


⚠ AGEG website (prod) has degraded performance

In a624295, AGEG website (prod) (https://www.ageg.ca/) experienced degraded performance:

HTTP code: 200
Response time: 2254 ms

Resolved: AGEG website (prod) performance has improved in 19f6978.


Exit code when no tests discovered/executed

Hi.
I am from vstest team.
Recently we introduced new RunConfiguration parameter called TreatNoTestsAsError - TreatNoTestsAsError
The idea here – if no tests were found during test execution, so vstest.console.exe returns non-zero code - if that parameter has value true.
I was testing today that logic in my testing pipeline and I saw that return code automatically setting to zero if there were not some flaky tests (see the attachment) Is it any new logic that you have in AzDo, because month ago when I tested the same scenario I did not see such behaviour (previous.png in attachements).
Currently:

Previously:

I expected my pipeline to fail, but it passed. Is it some new settings where we can change that option? Is it hard coded logic?

Hi. Has anyone looked at this issue?

Hi @Sanan07
This repo is for pipeline tasks mostly, but as I see this error comes from Azure DevOps side - could you please create ticket on https://developercommunity.visualstudio.com/ to get right eyes on it?

@anatolybolshakov Thank you for response. I created the one : ticket.
This can be closed then.


This repo starts off with a generic schema about people. Document the future intentions if the schema isn't changed ASAP

people schema a has been replaced with MimsSoil


Add link to Microsoft Office documentation

I'm on the government cloud and I have feedback on this statement:
Power BI entry points for the add-in aren't available in national/regional, government, and air-gapped clouds. See the Microsoft Office documentation for information about the availability of the add-in in PowerPoint in these clouds.
Would like a link to the "Microsoft Office documentation" to get more information or updates on when Power BI entry points will be available for me.

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 6a5d14ba-66ff-7447-76d6-4f4d38e5b382
Version Independent ID: 6a5d14ba-66ff-7447-76d6-4f4d38e5b382
Content: Power BI add-in for PowerPoint - Admin info - Power BI
Content Source: powerbi-docs/collaborate-share/service-power-bi-powerpoint-add-in-admin.md
Service: powerbi
Sub-service: pbi-collaborate-share
GitHub Login: @paulinbar
Microsoft Alias: painbar

Thanks for your comment, @elue-wrd. This issue has been assigned to the doc author for review.

Hi @elue-wrd, thanks for your feedback, and sorry about the long delay in replying. I am looking for a link and will put it into the documentation if I find one. Closing this issue now and hoping that I'll be able to find the link you're looking for.


Node js SSL error {“code”:“CERT_UNTRUSTED”}

I am using node version 0.12.7 with npm (2.11.3) and sails (0.11.5). So in the project i consuming REST Api using request (npm Module). API is on another domain so I enable CORS in sails, Everything is working fine On QA server but when i move my code to production (API is hosted with SSL) at API call it says
{"code":"CERT_UNTRUSTED"}.
Below is API call with request module.
request({
        url: sails.config.constant.baseApiUrl+'login',
        method: "POST",
        json: true,
        body: data
    }, function (error, response, body){
            console.log('error:- '+JSON.stringify(error));
            console.log('response:- '+JSON.stringify(response));
            console.log('body:- '+JSON.stringify(body));

         // Other code to handle response

    });

Response:-
error:- {"code":"CERT_UNTRUSTED"}
response:- undefined
body:- undefined
Is that any way to solve this issue in sails js?

@abdulrashid9 This doesn't sound like a Sails issue at all, but an issue with making a request to a secure server without providing any credentials.  See the request docs for info on how to specify a certificate to use when making a request.  You can also try adding rejectUnauthorized: false to the request options.


Native multi-arch ClusterBuildStrategy

Changes
This PR proposes a sample multi-arch native ClusterBuildStrategy. The ClusterBuildStrategy runs as a main orchestrator pod. It creates one "agent" job for each architecture requested by the Build. The agent jobs are responsible for building the container image and coordinate with the orchestrator pod via messages in FIFO pipes sent through the kubectl client.
The agent jobs run through the following coordination barriers:

Wait for the assets to be uploaded (source code, pull secrets, etc...)
Waiting for the image download to start
Waiting for the image download to finish

The orchestrator pod:

Create the agent jobs
Upload the assets and signal the completion of the assets upload through a write to the FIFO pipe in each agent job
Start the download of the oci archive
Inform the agent jobs the download is completed (so that they can complete)
Create a manifest-list and push it to the final registry.

The service account that runs the strategy must be able to create, list, get and watch Jobs and Pods. It also needs to be allowed the exec verb for pods.
Relates to #1119
/kind feature
Submitter Checklist

[ ] Includes tests if functionality changed/was added
[ ] Includes docs if changes are user-facing
[x] Set a kind label on this PR
[x] Release notes block has been filled in, or marked NONE

See the contributor guide
for details on coding conventions, github and prow interactions, and the code review process.
Release Notes
Add sample build strategy to orchestrate multi-arch container image builds. The service account executing this build strategy must have the ability to manage Jobs and Pods, as well as have the ability to exec into Pods.

/cc @adambkaplan

🤔 I'll need to check if our test suite automatically runs a smoke test for each sample build strategy, or if that is something we need to add to the e2e test code directly.

Needs to be added specifically. Though, what would we test in GitHub actions, there is no way to create a KinD cluster with nodes of different architectures ...

🤔 I'll need to check if our test suite automatically runs a smoke test for each sample build strategy, or if that is something we need to add to the e2e test code directly.

Needs to be added specifically. Though, what would we test in GitHub actions, there is no way to create a KinD cluster with nodes of different architectures ...

I think for the matter of testing, it is good enough to build a single-arch image into a manifest-list, i.e., having only one element in the architectures list of the Build object.

/hold
To allow other reviews

/unhold
(added by mistake)

This failure on e2e (v1.27.11, v0.59.2)  seems unrelated:
   Copying blob sha256:df2c3b2eb7cc63351bb32f26457bbe0402af8082548f26975f0c329bc7841881
  Copying blob sha256:efe636eac583776a8a114d50fef15bc65b648f3d2bb53326cf1f21cc5ef2b3ae
  Copying blob sha256:fe17849545bb51455d3f7c8773ded2dbb1d6668a85bd00564573a4b88afd36f6
  Error: creating build container: copying system image from manifest list: writing blob: adding layer with blob "sha256:fe17849545bb51455d3f7c8773ded2dbb1d6668a85bd00564573a4b88afd36f6": creating read-only layer with ID "fdd09facb36803af0795aedc41a5c13fdb1c835d7ff211ddfedac7813bd2dd80": no space left on device

This other on e2e (v1.27.11, v0.53.5), too, as the same test seemed to pass with v1alpha1.
• [FAILED] [95.119 seconds]
Using One-Off Builds Embed BuildSpec in BuildRun [It] should build an image using Buildpacks and a Git source
/home/runner/work/build/build/test/e2e/v1beta1/e2e_one_off_builds_test.go:64

  Timeline >>
  2024-07-25T14:06:34Z 1 Still waiting for build run 'onoff-fjrst' to succeed.
  [FAILED] in [It] - /home/runner/work/build/build/test/e2e/v1beta1/validators_test.go:134 @ 07/25/24 14:07:09.039
  2024-07-25T14:07:09Z 1 Print failed BuildRun's log
  [PANICKED] in [AfterEach] - /opt/hostedtoolcache/go/1.21.12/x64/src/runtime/panic.go:261 @ 07/25/24 14:07:09.041
  << Timeline

  [FAILED] BuildRun status doesn't move to Succeeded
  Expected
      <v1.ConditionStatus>: False
  not to equal
      <v1.ConditionStatus>: False
  In [It] at: /home/runner/work/build/build/test/e2e/v1beta1/validators_test.go:134 @ 07/25/24 14:07:09.039

  Full Stack Trace
    github.com/shipwright-io/build/test/e2e/v1beta1_test.validateBuildRunToSucceed.func1()
    	/home/runner/work/build/build/test/e2e/v1beta1/validators_test.go:134 +0x322
    reflect.Value.call({0x1a327a0?, 0xc000688390?, 0xc0007f5a20?}, {0x1d9a3e0, 0x4}, {0x3060d60, 0x0, 0xc0007f5ab2?})
    	/opt/hostedtoolcache/go/1.21.12/x64/src/reflect/value.go:596 +0xce7
    reflect.Value.Call({0x1a327a0?, 0xc000688390?, 0x0?}, {0x3060d60?, 0x1?, 0x3?})
    	/opt/hostedtoolcache/go/1.21.12/x64/src/reflect/value.go:380 +0xb9
    github.com/onsi/gomega/internal.(*AsyncAssertion).buildActualPoller.func3()
    	/home/runner/work/build/build/vendor/github.com/onsi/gomega/internal/async_assertion.go:325 +0x11f
    github.com/onsi/gomega/internal.(*AsyncAssertion).match(0xc0001f6d90, {0x2090c18?, 0xc0004e2b80}, 0x1, {0xc0004e2b90, 0x1, 0x1})
    	/home/runner/work/build/build/vendor/github.com/onsi/gomega/internal/async_assertion.go:540 +0x82c
    github.com/onsi/gomega/internal.(*AsyncAssertion).Should(0xc0001f6d90, {0x2090c18, 0xc0004e2b80}, {0xc0004e2b90, 0x1, 0x1})
    	/home/runner/work/build/build/vendor/github.com/onsi/gomega/internal/async_assertion.go:145 +0x86
    github.com/shipwright-io/build/test/e2e/v1beta1_test.validateBuildRunToSucceed(0xc0001a04d0, 0xc0002d8f00)
    	/home/runner/work/build/build/test/e2e/v1beta1/validators_test.go:144 +0x362
    github.com/shipwright-io/build/test/e2e/v1beta1_test.glob..func4.2.2()
    	/home/runner/work/build/build/test/e2e/v1beta1/e2e_one_off_builds_test.go:80 +0x450

  There were additional failures detected.  To view them in detail run ginkgo -vv

@SaschaSchwarze0 do you know if GitHub actions run the matrixed tests on the same machine? That could explain the out of disk issue.

@SaschaSchwarze0 do you know if GitHub actions run the matrixed tests on the same machine? That could explain the out of disk issue.

I would assume not given a common use case for matrix is also to use it for the os itself.
Disk space had not been an issue since we started to use https://github.com/shipwright-io/build/blob/v0.13.0/.github/workflows/ci.yml#L135-L144. I might not have looked too carefully, but did we have it somewhere else outside of this PR? Running concurrent jobs, letting them running until the result is streamed over (and therefore duplicated) to the buildrun pod and not deleting them (not seeing a kubectl delete for them and relying on ttlSecondsAfterFinished of one day) certainly can be disk-intensive.

@SaschaSchwarze0 do you know if GitHub actions run the matrixed tests on the same machine? That could explain the out of disk issue.

I would assume not given a common use case for matrix is also to use it for the os itself.
Disk space had not been an issue since we started to use https://github.com/shipwright-io/build/blob/v0.13.0/.github/workflows/ci.yml#L135-L144. I might not have looked too carefully, but did we have it somewhere else outside of this PR? Running concurrent jobs, letting them running until the result is streamed over (and therefore duplicated) to the buildrun pod and not deleting them (not seeing a kubectl delete for them and relying on ttlSecondsAfterFinished of one day) certainly can be disk-intensive.

Builds and BuildRuns should be deleted after each case, according to  https://github.com/shipwright-io/build/blob/433160ecf08c84cb7075910e9ea01d2b25d2abc8/test/e2e/v1alpha1/e2e_test.go#L30-L48
I'm running it a few times locally to see if I find anything that can be related to my changes... I can't re-run it here without your help though.

Builds and BuildRuns should be deleted after each case

Indeed they are. I missed the ownerreference in the job you are creating.


Prevent JGit from loading local configuration in tests

Without any preparations JGit defaults to read system / user / project
configuration from local file system. Users running tests locally
might see test failures due to their config diverging from the test
author config.
For me JGitEnvironmentRepositoryIntegrationTests.pullDirtyRepo() was
constantly failing, because JGit calculated a different hash for the
created commit than on the build server.

Codecov Report

Merging #1272 into master will decrease coverage by 0.22%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #1272      +/-   ##
============================================
- Coverage     79.47%   79.25%   -0.23%
+ Complexity      993      992       -1
============================================
  Files           117      117
  Lines          3528     3528
  Branches        490      490
============================================
- Hits           2804     2796       -8
- Misses          549      559      +10
+ Partials        175      173       -2

Impacted Files
Coverage Δ
Complexity Δ

...fig/client/ConfigServicePropertySourceLocator.java
66.19% <0%> (-3.53%)
25% <0%> (ø)

...rk/cloud/config/client/ConfigClientProperties.java
89.65% <0%> (-1.73%)
40% <0%> (-1%)

...cloud/config/monitor/FileMonitorConfiguration.java
51.58% <0%> (-0.8%)
21% <0%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2e606de...5b67a25. Read the comment docs.

Codecov Report

Merging #1272 into master will decrease coverage by 0.22%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #1272      +/-   ##
============================================
- Coverage     79.47%   79.25%   -0.23%
+ Complexity      993      992       -1
============================================
  Files           117      117
  Lines          3528     3528
  Branches        490      490
============================================
- Hits           2804     2796       -8
- Misses          549      559      +10
+ Partials        175      173       -2

Impacted Files
Coverage Δ
Complexity Δ

...fig/client/ConfigServicePropertySourceLocator.java
66.19% <0%> (-3.53%)
25% <0%> (ø)

...rk/cloud/config/client/ConfigClientProperties.java
89.65% <0%> (-1.73%)
40% <0%> (-1%)

...cloud/config/monitor/FileMonitorConfiguration.java
51.58% <0%> (-0.8%)
21% <0%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2e606de...5b67a25. Read the comment docs.

Does anything prevent the merge?

Nope. It will happen in a few days

Rebased on top of master.

@ewirch would you mind submitting this against the 2.0.x branch and we will take care of merging it forward?  I think it would be useful across all the active branches.

Sure thing. Rebased.


Books/search within notebook

Initial Find in a notebook implementation:
Find in Markdown cell:
This still needs work to highlight the exact work and currently it highlights the entire line when a match is found. This means if there are multiple matches in the same line, users might not see any difference on the UI since moving to the matches still highlights the same line.
Find in Code cell:
Works as expected.
Note: Find in output cell is not in scope of find for now.
Implementation summary:
Find in code cell re-uses the vscode editor decorations implementation. We've implemented the following as part of the notebookModel: searching cell contents and finding ranges w.r.t the search results as well as creating decorations of those ranges. For applying those decorations, we get the editor associated with the cell and pass-in the decoration to be applied.
Find in markdown is somewhat a crude implementation since it doesn't have an editor associated with it. We pass in the range of the search result to the markdown cell component and manipulate the dom to add a highlight style to the html element that's referred by the range.

@MaddyDev did you see the LGTM alerts? lemme know if you need some help

Yeah, I looked into them. I haven't paid any attention to the code that's already added before I took over the find. I'm working on addressing them, will probably check with you if they have wider effect than just find.


Settings menu

Requires finalized bind system
Related: https://github.com/ForLoveOfCats/SkyOfSteel/issues/11


OCL: added errors processing in OpenCL code generator, resolve space issue

Replacement for https://github.com/Itseez/opencv/pull/4113

:+1:


Add support for native package builds to the build container

What would you like to be added:
We should extend the container build with a possibility to build native Debian packages. One would use native Debian packages if they are the maintainer of the upstream source code. There is then no need to pull external source code but the original source code can already been placed in the package repo.
For this however, the build container and its scripts must support native builds. As of now, they only support builds based on sources coming from Debian testing or other Git repos.
Why is this needed:
There are some Garden Linux packages which are not build out of external sources but are build with locally stored sources like this one: https://gitlab.com/gardenlinux/gardenlinux-package-gardenlinux-selinux-module
For those cases, we must be able to build Debian native packages which basically already include the upstream code.

Update the title to native package builds to avoid mistake. :)


some functions have the wrong type of comments

for example:
/**
 * Extension method to provide show keyboard for View.
 */
fun View.setPaddingHorizontal(value: Int) = setPaddingRelative(value, paddingTop, value, paddingBottom)

@Dreamersoul Please open the PR with fixes.

@ravidsrk https://github.com/VictorChow/KotlinAndroidLib/blob/master/ext/src/main/java/pers/victor/ext/ViewExt.kt
Do you know this repo?


Fix incorrect URL in 'it-always-loads'

The URL is referencing a page that does have a code sample, but it should be referencing the page that does not have code samples, i.e. the one in the screenshot.

Thanks.


[Minor][SpecInfer] Fix Optional FC Bias for Mixtral Eagle Model

This PR added an optional fc bias option so that mixtral eagle model could be loaded.
Fix referenced from https://github.com/SafeAILab/EAGLE/blob/08dc06ccb428f2d3cf8380933ea50ed3e4178168/eagle/model/ea_model.py#L42
CC: @KnowingNothing @tqchen

LGTM.


Add Receptor fixes

Fixes #704

From the call:

Bidirectional links between antigen and receptor may not be necessary.

From the call: @bcorrie and @bussec are in general ok with this PR, @bussec will make some final clarification in the description of Cell.reactivity_measurement. Will put this up for a final discussion in January, then merge.

From the call:

Need to review, but seems mostly fine, except we don't think we need reactivity_measurements in Cell. There wouldn't be a prohibition on it, but it would not be the preferred schema path.

From the call:

Sorry I missed the call, looks like my calendar entry ended end of 2023. I am still traveling, returning home tomorrow.

Need to review, but seems mostly fine, except we don't think we need reactivity_measurements defined in Cell. There wouldn't be a prohibition on it, but it would not be the preferred schema path.

reactivity_measurements in both Cell and Receptor are arrays of IDs that refer to ReceptorReactivity objects (essentially the RecceptorReactivity.receptor_reactivity_id of the object). So reactivity_measurements are not defined in the Cell object per se.
My understanding of Cell having a receptor_reactivity is that this would capture when a specific experiment measured a reactivity for an epitope for a specific Cell (e.g. in a 10X experiment). See https://github.com/airr-community/airr-standards/issues/704#issuecomment-1641133894
Is that the general understanding? Would this not be the preferred thing to do in such an experiment?

We were thinking the normal usage would be Cell -> Receptor -> ReceptorReactivity. If you wanted to jump over Receptor and put a reactivity_measurements array into Cell you could, in the same way that you can add sample_id to Rearrangement (semantics are clear even though there's no sample_id defined in Rearrangement), but it wouldn't be the model.

Hey gang,
I wanted to toss thoughts from a B cell perspective. The above discussion seems very focused on T cells so I'm trying to figure out if we need another Object to cover Antibody specificity or make a more general.
Random thoughts:

ReceptorReactivity is more of a T cell concept since I think of "reactivity" as a (T) cell's response when it's receptor binds something. For antibodies the concept of reactivity is not on-point since there is no cell.
The above and that BCRs (on B cells) can also interact with antigen (like T cells), begs the question is if we should have the distinction between antibodies and BCRs on B cells? We sequence B cells but often test binding/affinity on antibodies. Maybe that distinction should be encapsulated in the description of how the BCR/antibody specificity was measured?
peptide_aa_string is something that don't have much applicability in B cell world.

-krish

I agree that reactivity doesn’t make sense for Abs. I haven’t kept up with all the discussion on this, but doesn’t ReceptorSpecificity make more sense and cover BCR, TCR, and Ab (even though Abs aren’t receptors)?

I think that name change covers my name issue. The only thing I could see falling outside that is if we had protein sequence from protein sequencing of antibodies, like Georgiou does. In IRAD we've thought about going after that data type because there is a lot of it in the patent Genbank. But, as far as I know, all our tool chains are very focused on nucleotide sequences, so that's an issue.

@krishnaroskin @lgcowell The Receptor object was developed with both AB/BCR and TCR in mind. In general, it can exist independent of a Cell, i.e., you can use it to annotate reactivities of antibodies that were not observed in cells derived from a Subject, e.g., a recombinantly expressed antibody in which somatic hypermutations were added or removed. However, as @javh described, we assume that the typical way how it would be used is in the context of single-cell experiment.

ReceptorReactivity is more of a T cell concept since I think of "reactivity" as a (T) cell's response when it's receptor binds something. For antibodies the concept of reactivity is not on-point since there is no cell.

This is basically a question about the semantics of the term "reactity". For me, reactivity implies that the receptor or antibody can interact with a given target in a way that can be experimentally detected. Using this definition, activiation of the cell that bears a given receptor/antibody is a sufficient, but not a necessary criterion. The reason why we use reactivity instead of specificity is that the later one - similar to its use in statistics - is relative to something else. Polyreactive antibodies for example are exactly that: They bind everything from polystyrol to shoe laces (i.e. they are "reactive") but they are not specific for any of these things.

The above and that BCRs (on B cells) can also interact with antigen (like T cells), begs the question is if we should have the distinction between antibodies and BCRs on B cells? We sequence B cells but often test binding/affinity on antibodies. Maybe that distinction should be encapsulated in the description of how the BCR/antibody specificity was measured?

In general this is the idea behind the Receptor/ReceptorReactivity split: A given receptor shoud always have the same intristic reactity (otherwise there would be no chance of reproducible results), but different assays will measure it by different means. For example, take an experiment where a fluorescently-labeled protein is used as bait to select for target-specific B cells during sorting. The sorted cells are then subjected to single-cell sequencing and the sequence information of the IGH/IGK transcripts is then used to produce a recombinant antibody that can be tested in vitro (e.g. ELISA). In this case a single Receptor record would have one or more ReceptorReactivity record that describe the results obtained using the recombinant antibody and one that describes the binding in the initial fluorescent bait assay. The latter ReactivityRecord could be directly referrenced from a Cell record (@bcorrie's use-case), while the prior ones would not. The reactivity_method and reactivity_readout properties can be used to provide further information on the experimental approach.

peptide_aa_string is something that don't have much applicability in B cell world.

Fully agree, therefore the schema does not require you to provide this property.

We were thinking the normal usage would be Cell -> Receptor -> ReceptorReactivity. If you wanted to jump over Receptor and put a reactivity_measurements array into Cell you could, in the same way that you can add sample_id to Rearrangement (semantics are clear even though there's no sample_id defined in Rearrangement), but it wouldn't be the model.

Is this implying that Cell.receptor_reactivity should be removed from the object schema? If so I think that is a problem. When receptor_reactivity is measured in an experiment, that reactivity is associated with a specific Cell. If we go Cell -> Receptor -> ReceptorReactivity we lose the information that that specific receptor_reactivity is associate with that Cell and that the receptor_reactivity was measured in a specific experiment.
Remember that since Receptor is a "global object" (there is only one Receptor with a specific paired VDJ/CDR3) it is quite likely for a specific Receptor to have more than one measured receptor_reactivity. This is even likely to happen in a single experiment if I am not mistaken, but it would certainly happen across multiple experiments. Multiple Cells will be assigned the same Receptor and have different measured receptor_reactivity. In the above model a single Receptor will have multiple receptor_reactivity measurements, but it won't be possible to tell which receptor_reactivity measurement came from which Cell. This would be even worse if the receptor_reactivity measures came from different experiments. In this case we lose information about which Study a receptor_reactivity measure came from.
Bottom line is I believe we need to be able to capture a link between Cell and a specific receptor_reactivity directly in the model.

Hmm, in thinking about this a bit more, the AIRR Standard is primarily about capturing how an experiment is done, no? So a direct link from the Cell to the ReceptorReactivity that was measured in the experiment is actually the critical path. It is what is actually measured in the experiment.
Mapping Cells to Receptors is a processing step that may or may not be done in an experiment.

Hmm, in thinking about this a bit more, the AIRR Standard is primarily about capturing how an experiment is done, no?

I agree with this sentiment, which makes me wonder how to get the set of receptor reactivities that were done in a single experiment?
{"digression rant": true}
The underlying issue that I feel we continually run into with these discussions is how to represent relations. How do we resolve the tension between an efficient data structure and our semantic model of the biology (with the ADC often as a third wrinkle). This is where JSON schema fails us because it doesn't have an explicit representation for relations (unlike say SQL DDL), instead it is implicit semantics (like ids and arrays) without any syntactic enforcement.
I tend to always advocate for the efficient data structure. That means in 1-to-n relations, I advocate that the "link" always goes on the "n" side where a single field can be used to point to the "1" object. Versus if you put the relation on the "1" side then you need an array holding "n" objects/ids.
Given this, having reactivity_measurements in Receptor is "wrong" (inefficient), even though semantically it sounds just fine: "a receptor has a set of reactivities". Instead ReceptorReactivity should have a receptor_id field that points to its one and only Receptor. With the current model, every time there is a reactivity measurement then the "global Receptor object" needs to updated with a new object added to the array. Who does that? How do we even know when to do that? Sounds messy to me. As an extreme analogy, imagine instead of Rearrangement pointing to its V allele, the V allele in the germline set had an array of rearrangements.
So I understand the current discussion is around Cell. A Cell can have multiple Receptors, so it's an n-to-n relation with ReceptorReactivity. However if you consider a Cell/Receptor combo, then nominally its a 1-to-n relation. Using the "efficiency" logic, the ReceptorReactivity should have a cell_id that points to the Cell. If the ReceptorReactivity does not involve a cell then that field is null. With ReceptorReactivity having both receptor_id and cell_id then I believe you can represent the true n-to-n relation by having multiple ReceptorReactivity objects with various combos of receptor_id and cell_id values.
Finally, coming back to where I started, maybe we should consider adding some formalism, like some JSON-LD attributes, that more explicitly defines the relations in the schema. Not really sure if that helps though, though maybe it helps to document the arity of the relation. Looking at some of the new draft objects, I feel we've introduced a number of "data inefficient" relations.
{"digression rant": false}

Using the "efficiency" logic, the ReceptorReactivity should have a cell_id that points to the Cell. If the ReceptorReactivity does not involve a cell then that field is null. With ReceptorReactivity having both receptor_id and cell_id then I believe you can represent the true n-to-n relation by having multiple ReceptorReactivity objects with various combos of receptor_id and cell_id values.

I don't object to this implementation if we feel that is a better way to go. As long as we maintain the ability to have a link between ReceptorReactivity to both Cell and Receptor if desired I think we are fine. Both can be null if the experiment does not have such links.
The array stems from the original Receptor model where their was an Array of reactivity objects embeded in the Receptor object. This was changed to have a separate ReceptorReactivity object an array of ReceptorReactivity IDs instead and the same was added to Cell.
I am OK to change ReceptorReactivity to have a cell_id and receptor_id to make these links. We can then remove the arrays receptor_reactivity from Cell and Receptor.

With the current model, every time there is a reactivity measurement then the "global Receptor object" needs to updated with a new object added to the array. Who does that? How do we even know when to do that? Sounds messy to me.

This is a really good reason to change our model as @schristley suggests...

I just pushed a set of changes to the schema that should address some of the points raised.
Some explanations on these changes and thoughts on the recent discussion regarding the relations between the objects, with a focus on experimental aspects:
From my point of view, there are two distinct types of reactivity information that we are trying to annotate:

Reactivity of recombinant receptors: In this case the receptor is reconstructed based on the sequences from an scAIRR-seq experiment and recombinantly expressed, e.g., as an soluble antibody or a T cell line expressing it as TCR. In such a setup we look at one defined receptor at a time, so we can simplify the Cell:Receptor:Reactivity relation as N:1:N, i.e., multiple cells can have the same receptor and a given receptor can have multiple reactivity measurements. Following our discussion on the representation of 1:N relations, this means that both Cell and ReceptorReactivity should reference to Receptor (Note: The fact that a cell can have multiple receptors would IMO not change in this decision, as the number of receptors per cell is usually 1 or 2 (and rarely large than 3), while cells per receptor can easily go into the range of hundreds. However, it means that Cell needs to contain an array of receptor IDs (as it does)). Importantly, in such an experiment there is no direct relation between the cell observed in the scAIRR-seq experiment and the reactivity measurement, so there is no reason to have a direct reference between the respective records here. This leads to the situation that the origin of a ReceptorReactivity is lost, which is something that - as @bcorrie critizied - we need to avoid. However, I don't see any obvious entry point below the level of the study, which is why my edits contain the addition of a study_id property to ReceptorReactivity. This also addresses the problem of receptors that are tested within a study, but are not observed in the scAIRR-seq data (e.g., antibodies in which some or all hypermutations are reverted).

Reactivity of individual cells: In this case cells are incubated with a bait (e.g., a protein or carbohydrate for B cells or an MHC:peptide multimere for T cells) that is labeled, e.g., with a fluorescent or DNA barcode tag. The presence of the tag on the cells (measured, e.g., as fluorescence or as barcode counts in the sequencing data to stick with the previous examples) is then interpreted as reactivity that can be attributed to the Ig/TCR expressed by the cell. We therefore had the idea to use ReceptorReactivity to capture this information, which then gave rise to the notion that there should be a direct reference between Cell and ReceptorReactivity as otherwise it would be impossible to attribute a given ReceptorReactivity to a Cell (given the N:1:N relation described above). After careful consideration, I have decided against adding a cell_id property to ReceptorReactivity and I would like us to reconsider the original decision to use ReceptorReactivity for this type of information, as I think that the ways these experiments are usually conducted do not allow for a proper attribution of reactivity information to an individual receptor:

If multiple receptors are expressed, binding cannot be directly attributed to any of them
Binding experiments are primarily used to select high-binding cells, not to measure reactivity. Therefore often the level of receptor surface expression in not measured, but would be necessary for normalization
Combining the previous two points, multiple receptor might be expressed at different levels but it is very challenging with the methods described above to normalize for this
Finally, the exact amount of background binding of the bait by an individual cell is unknown as it is difficult to control for it

Therefore, while not impossible, most experiments will provide data that should not be considered to be receptor reactivity data, but rather cell reactivity data.

The most obvious solution would IMO be to leave Receptor and ReceptorReactivity as it is now, and add a separate CellReactivity object, but I am open to other suggestions.

@bussec as a non-expert, your argument makes sense to me.
Am I correct in stating that:

If we "reused" ReceptorReactivity to represent your case 2 above, we could simply add a cell_id and it would work. But in this case we are "confusing" or "conflating" the two types of reactivity when you are suggesting we probably shouldn't.
If we created a new CellReactivity it would have similar fields to ReceptorReactivity but would not be related at all to the Receptor. It would have a cell_id to link it to the Cell, but would not have a link to a Receptor. We would then remove the array of receptor_reactivity in Cell also.

If I am not mistaken you have not made that change yet, correct?
For the record, we are trying to curate a study that has CellReactivity using MHC:peptide multimers. That is where issue #704 "Receptor object issues when used in real life..." came from. There is no Receptor inference done in this study and the reactivity is to the Cell. So to capture this study accurately from the experimental perspective we would have Cell and CellReactivity but no Receptor and ReceptorReactivity.
If the study actually inferred that Cell C was actually an instance of Receptor R, we would then create a Receptor object and link it to the Cell.
So I think you rationale of adding CellReactivity and having two different types of "Reactivity" make sense to me.

@bcorrie Yes, your two points are correct. I think that a separate CellReactivity record is the cleaner solution, although it will be very similar to the ReceptorReactivity record. In haven't made these changes yet, be we would need to clean up some of the experimental keywords in addition.

Thank you for the detailed description @bussec

Reactivity of recombinant receptors:

A somewhat naive question. This overlaps (same, different?) with the annotation that IEDB is doing?

Reactivity of individual cells: In this case cells ...

Hi @bpeters42 , this is an interesting case that Christian brings up. Is this something that IEDB is currently annotating in some way? While the reactivity is not tied to a one specific receptor, it is limited to a small possible set.
@bussec Are you aware of any publications that have reported this type of data?

See my use case here about how one might annotate a Cell with two paired chain Rearrangements with Receptor information that is linked to IEDB: https://github.com/airr-community/airr-standards/issues/704#issuecomment-1891063546

@bussec Are you aware of any publications that have reported this type of data?

We are working on curating this paper: https://doi.org/10.1038/s41590-022-01184-4 which uses MHC/peptide dextramer tags to determine cells that have specificity to specific MHC/peptide complexes.
We have loaded the Rearrangement and Cell data for this study already (https://gateway.ireceptor.org/samples?query_id=93707) and are working on the Cell/CellReactivity linking currently. We also have similar specificity data for some of our T1D studies that we are working on.
In fact, it is the curation of this study that caused me to create #704 "Receptor object issues when used in real life...".

@bussec I have added a start of CellReactivity. Has a cell_id rather than study_id, and I removed receptor_hash as I don't think it applies??? I also removed receptor_reactivity from Cell.

@bussec Are you aware of any publications that have reported this type of data?

Lots of this going on in the B cell world:
https://pubmed.ncbi.nlm.nih.gov/34848871/
https://pubmed.ncbi.nlm.nih.gov/37935199/
https://pubmed.ncbi.nlm.nih.gov/36708513/
https://pubmed.ncbi.nlm.nih.gov/37440409/
...though many of those don't do more than an SRA deposit, in terms of reporting.

I'm a bit confused here.  If there's reactivity data without any associated AIRR-seq data, then isn't this out of scope for the AIRR schema?
The COVID-19 paper with the oligo tagged MHC:peptide assay is measuring both the TCR and the reactivity of the TCR simultaneously (via the 10x protocol). Wouldn't this be a case for Receptor:ReceptorReactivity?

2. Reactivity of individual cells: In this case cells are incubated with a bait (e.g., a protein or carbohydrate for B cells or an MHC:peptide multimere for T cells) that is labeled, e.g., with a fluorescent or DNA barcode tag. The presence of the tag on the cells (measured, e.g., as fluorescence or as barcode counts in the sequencing data to stick with the previous examples) is then interpreted as reactivity that can be attributed to the Ig/TCR expressed by the cell. We therefore had the idea to use ReceptorReactivity to capture this information,

@bussec @bcorrie isn't this already covered by CellExpression though? In particular, we have "antigen_bait_binding_by_fluorescence_intensity" and "antigen_bait_binding_by_dna_barcode_count" as recommended values for CellExpression.property_type.
I could see how it might make sense to separate this into a new CellReactivity object, since antigens are not "expressed," but (a) we should make the boundaries/use cases clear and (b) the above should be added to CellReactivity.reactivity_method along with corresponding changes to readout.

I'm a bit confused here. If there's reactivity data without any associated AIRR-seq data, then isn't this out of scope for the AIRR schema?
The COVID-19 paper with the oligo tagged MHC:peptide assay is measuring both the TCR and the reactivity of the TCR simultaneously (via the 10x protocol). The purpose of these assays is to determine the reactivity of individual TCRs. Wouldn't this be a case for Receptor -> ReceptorReactivity?

I guess that "measuring both the TCR and the reactivity of the TCR simultaneously" is not quite true since (a) you can have a reactivity measurement for a cell from which a TCR was not captured (and vice versa) and (b) an allelic inclusion (or cell doublet) would result in a single reactivity measurement but >1 possible receptor.

@bussec Are you aware of any publications that have reported this type of data?

We are working on curating this paper: https://doi.org/10.1038/s41590-022-01184-4 which uses MHC/peptide dextramer tags to determine cells that have specificity to specific MHC/peptide complexes.

This study is curated in IEDB.  It has 891 receptors, 44 assays, 18 epitopes and 5 antigens. What is AIRR trying to curate that isn't already in IEDB?

I guess that "measuring both the TCR and the reactivity of the TCR simultaneously" is not quite true since (a) you can have a reactivity measurement for a cell from which a TCR was not captured (and vice versa) and (b) an allelic inclusion (or cell doublet) would result in a single reactivity measurement but >1 possible receptor.

@scharch good point. Hrm. Is the main challenge here that the global Receptor object is too ambitious? Hard to update, will require inferences that aren't simple, etc? In which case, I think the proposed CellReactivity would end up being the default route people take because of convenience.

I suppose that this would mean that if we wanted to be complete in the curation of this study, we would need to create 891 Receptors and link the Cells we have loaded (https://gateway.ireceptor.org/samples/cell?query_id=93707) to each of the related Receptor object, which in turn would have a link to the related IEDB object. I suspect there is an N to 1 relationship between Cell and Receptor and it is likely that all Cells will have a Receptor as we are only curating the epitope specific Cells from the study.

So for this study's reactivity data it seems like the minimal would be:

Create a CellReactivity object for each measured reactivity between a Cell and an epitope target. This is linked back to the Cell through the cell_id field. These are the actual measured experimental results.

In this way:

One can find reactivity data in the ADC that is experimentally measured at the Cell level by looking at CellReactivity
One can find reactivity data in the ADC at the Receptor level with a more complex query. You would need to do something like "give me the set of Cells C that are linked to Receptor R and then give me all the CellReactivity for all the Cells in set C"
One can find reactivity data in the ADC at the Receptor level by looking a ReceptorReactivity.
One can find reactivity data from external sources at the Receptor level through querying IEDB about known reactivity from Receptors in the ADC

This is non-trivial, but at least "semantically" distinct???
In the AKC world, this is all going to be trivial, right 8-)

One can find reactivity data in the ADC at the Receptor level with a more complex query. You would need to do something like "give me the set of Cells C that are linked to Receptor R and then give me all the CellReactivity for all the Cells in set C"

I wonder if CellReactivity should have a receptor_id so that a Cell->Epitope reaction can be associated with both a Cell and a Receptor, which would avoid the above issue... It can be null if no Receptor is created, but if  there is a Cell->Receptor association, should we maintain this in the CellReactivity? Possibly not, as this makes internal consistency challenging.

I have added repertoire_id and data_processing_id to CellReactivity similar to how we have it with CellExpression. I think we need to link the reactivity back to the Repertoire, no?

@scharch good point. Hrm. Is the main challenge here that the global Receptor object is too ambitious? Hard to update, will require inferences that aren't simple, etc? In which case, I think the proposed CellReactivity would end up being the default route people take because of convenience.

It seems to me that the experiments are measuring CellReactivity. Isn't the fact that there is ReceptorReactivity inferred in some sense? If we define a Receptor in this case aren't we inferring that the reactivity is between the Receptor and the epitope.

@bussec Are you aware of any publications that have reported this type of data?

We are working on curating this paper: https://doi.org/10.1038/s41590-022-01184-4 which uses MHC/peptide dextramer tags to determine cells that have specificity to specific MHC/peptide complexes.

This study is curated in IEDB. It has 891 receptors, 44 assays, 18 epitopes and 5 antigens. What is AIRR trying to curate that isn't already in IEDB?

The AIRR-seq, Cell, and GEX data from the study.
This seems like a perfect use case for linking the ADC and IEDB since the study is currently curated in both. Currently the AIRR-seq/Cell/Gex data is in the ADC and the Epitope specificity data is in IEDB. There is currently no way to determine that there is related data in both repositories except through searching for the publication in each platform. In fact I didn't know it was curated in IEDB. Isn't that the crux of the problem 8-)

Yes, though I meant specifically about reactivity. My concern was duplication of curation between the two resources about receptors and reactivity, especially if there isn't coordination, then you get two separate curations, leading to confusion for users. The link that you want between the two resources doesn't happen automatically, so yeah that's the crux of the problem but ADC doing it's own curation that duplicates IEDB doesn't sound like a good idea.

I suppose that this would mean that if we wanted to be complete in the curation of this study, we would need to create 891 Receptors and link the Cells we have loaded (https://gateway.ireceptor.org/samples/cell?query_id=93707) to each of the related Receptor object, which in turn would have a link to the related IEDB object.

Or you can just point the AIRR Study to the IEDB Study and then you know there is information in both. Or have the AIRR Rearrangements point to the IEDB records. Or have AIRR Cells point to IEDB records. Or what you said... There are many options, it all depends upon what your goals are based upon some driving use cases.

In the AKC world, this is all going to be trivial, right 8-)

If not trivial then at least easier and more consistent. Part of my confusion with this whole thread is understanding where the boundaries are. Many things seem to be exactly what the AKC data harmonization effort is doing. If AIRR is trying to create standards for the integration effort and data harmonization that AKC is doing, then it feels a bit like the cart before the horse.

@schristley  said:

Or you can just point the AIRR Study to the IEDB Study and then you know there is information in both. Or have the AIRR Rearrangements point to the IEDB records. Or have AIRR Cells point to IEDB records. Or what you said... There are many options, it all depends upon what your goals are based upon some driving use cases.

We (the iReceptor team) chose the Minervina et al. study because it provides a good scientific use case that tested all of the new AIRR Standard constructs - Subject HLA, Rearrangement, Cell, CellExpression, Receptor, and ReceptorReactivity that were available after the release of AIRR v1.4/iReceptor v4.0. Thus in a way it was chosen to help us both poke holes in the AIRR model as well as to understand the boundaries between the ADC and other resources better.
In order to poke holes in the AIRR model we wanted to curate the study as completely as possible. So we want to curate all of the HLA, Rearrangement, Cell, CellExpression, Receptor, and CellReactivity data that was produced in the study in the appropriate ADC constructs. Although we have done all of the others, we have only recently been trying to curate the Minervina Receptor, ReceptorReactivity, or CellReactivity data - and that is what started this whole discussion 8-)
@schristley also said:

If not trivial then at least easier and more consistent. Part of my confusion with this whole thread is understanding where the boundaries are. Many things seem to be exactly what the AKC data harmonization effort is doing. If AIRR is trying to create standards for the integration effort and data harmonization that AKC is doing, then it feels a bit like the cart before the horse.

For me the AIRR Receptor and Reactivity objects are designed to enable data harmonization and integration, not necessarily perform that integration. That is why Receptor has receptor_refs which point to external sources of information (such as IEDB) that might have information about that object that doesn't exist in the ADC. There is no intent to duplicate data needlessly.
The reason there is overlap in terms of Reactivity data between the ADC and IEDB in this case is that the data from the Minervina study has been reused for both purposes. This is a good thing, and to me these uses are complementary but different. The links between Cell, CellExpression, and CellReactivity are all part of the Minverina immune response study, and are only found in the ADC, so I do believe that the CellReactivity (linked back to the actual Cell) from the study should be captured in the ADC.
This study also provide evidences for specificity of known Receptors in IEDB, so the data has also been curated and captured there.
Our challenge, like usual, is to figure out how to link the two so that the user doesn't get confused... 8-)

Minor tweak in CellReactivity and ReceptorReactivity - shouldn't reactivity_unit be a Unit Ontology (UO) field?

Minor tweak in CellReactivity and ReceptorReactivity - shouldn't reactivity_unit be a Unit Ontology (UO) field?

Answering this myself. I don't think it can because there is a need to have a Reactivity as a true/false - which I can't find in UO. But I am also not sure how to represent the boolean this cell binds to this antigen/epitope/peptide?
Do we really know the fields reactivity_readout and reactivity_method to have these as enums? Should they be strings with a recommended limited vocabulary instead?

From the call:

Keep lean version of Receptor.
Remove ReceptorReactivity.

Done

Receptor.reactivity_measurements to reactivity_ref.

This was already removed. There is now only a Receptor.receptor_ref
It seems to me like receptor_ref is a better name for this field. At least when it points to an IEDB record, it is pointing to a IEDB.Receptor object which likely records many documented reactivity records for that Receptor.

Can we rename CellReactivity to Reactivity.

This seems more appropriately named CellReactivity to me, since it is actually linked to the Cell. I don't think this is particularly critical since there is a cell_id field that makes the implicit link to Cell clear.
But it seems to me that since we have CellExpression as an object it seems like CellReactivity is more consistent.

Note to self: Changes in the docs are still pending, will wait for #628 to  complete.

But it seems to me that since we have CellExpression as an object it seems like CellReactivity is more consistent.

No objection from me.

It seems to me like receptor_ref is a better name for this field. At least when it points to an IEDB record, it is pointing to a IEDB. Receptor object which likely records many documented reactivity records for that Receptor. I think the intent here is we are pointing to external definitions of the Receptor that provide more info like "receptor reactivity".

I don't think that was the goal. I thought we were replacing the functionality of the reactivity object itself. @bussec?

I don't think that was the goal. I thought we were replacing the functionality of the reactivity object itself. @bussec, am I mistaken?

The Receptor object now points to a bunch of external entities that contain information about that Receptor, noted with CURIEs. Those entities (e.g. IEDB_RECEPTOR:10) contain information about receptor reactivity, but they currently are not pointing to different reactivity records, you are correct.
        receptor_ref:
            type: array
            description: Array of receptor identifiers defined for the Receptor object
            title: Receptor cross-references
            items:
                type: string
            example: ["IEDB_RECEPTOR:10"]
            x-airr:
                nullable: true
                adc-query-support: true

I think we could link to individual external reactivity records, but that would be moving more towards replicating information that is in those other repositories. The above IEDB Receptor (https://www.iedb.org/receptor/10) has three assays that show positive results:
https://www.iedb.org/assay/1650226
https://www.iedb.org/assay/1650227
https://www.iedb.org/assay/1755178
So we could have:
        receptor_ref:
            type: array
            description: Array of receptor identifiers defined for the Receptor object
            title: Receptor cross-references
            items:
                type: string
            example: ["IEDB_ASSAY":"1650226","IEDB_ASSAY":"1650227","IEDB_ASSAY:"1755178"]
            x-airr:
                nullable: true
                adc-query-support: true

But as I say, you can get that from IEDB by linking to a single record, and you are now replicating all of the links to specific assay data that is curated in IEDB. Some Receptors would have MANY assays, so that would be painful to manage.
It makes more sense to me to be linking to the external object at the same conceptual level (e.g. Receptor).

It seems to me like receptor_ref is a better name for this field. At least when it points to an IEDB record, it is pointing to a IEDB. Receptor object which likely records many documented reactivity records for that Receptor. I think the intent here is we are pointing to external definitions of the Receptor that provide more info like "receptor reactivity".

I don't think that was the goal. I thought we were replacing the functionality of the reactivity object itself. @bussec, am I mistaken?

The way I interpreted our discussions in the last call was to replace the (AIRR-C-defined) ReceptorReactivity object with links to objects defined and hosted by other repositories. This is different from removing reactivity information completely from the Receptor object. IMO there is value for users to be able to directly assess how much (or how little) information there is about a given receptor, without having to do lookups with other repos.

@bcorrie @javh I had another look at this and did some minor fixes with regard to enums of experimental properties in CellReactivity. Otherwise:

I would be ok with removing the reactivity_ref property from the Receptor object and only retain the links to an "receptor-like" object in an external database. While I would still prefer to have the reactivity information in there, I also think that @bcorrie's concerns with regard to logical structure and update frequency of reactivity information are valid.
In which scenario do we need CellReactivity.repertoire_id and CellReactivity.data_processing_id, given that this information is already present in the Cell object?
If we agree on dropping reactivity information altogether, the Docs need some more work-up to reflect the changes regarding *Reactivity objects.
The original ReceptorReactivity object was designed with a low-throughput experimental setup in mind than what will get from experiments that would use a CellReactivity record (i.e., 10,000-1,000,000 cells stained with the same set of baits). Irrespective of normalization in the backend database of a repo, do we need to split CellReactivity into a "reagent" and a "measurement" object? Or will file compression save us?

As the last point is not directly connected to the original topic of this PR, I would like to suggest that we make the changes for 1-3 and merge and then have a new issue for 4.

I would be ok with removing the reactivity_ref property from the Receptor object and only retain the links to an "receptor-like" object in an external database. While I would still prefer to have the reactivity information in there, I also think that @bcorrie's concerns with regard to logical structure and update frequency of reactivity information are valid.

This is currently the situation, I had not yet added reactivty_ref into the Receptor object, so there is currently only a receptor_ref. So this is "done" unless we decide otherwise.

2. In which scenario do we need CellReactivity.repertoire_id and CellReactivity.data_processing_id, given that this information is already present in the Cell object?

Currently, unless I am mistaken (gasp), all observed objects are connected to their respective Repertoire and the relative _id fields.
Although the use case might not be obvious, making it possible for the data consumer to find all CellReactivity information related to a Repertoire without having to find all Cells and the search CellReactivity for a large number of Cells seems like a good idea to me. It future proofs us from things that we might not think would be common.
In fact, now that I think about it from an ADC perspective, one use case would be I want to download all of the Cells, CellExpression, and CellReactivity data from a specific Sample. I would do this by processing the related repertoire_id fields from that Sample and query the appropriate end point once.

As the last point is not directly connected to the original topic of this PR, I would like to suggest that we make the changes for 1-3 and merge and then have a new issue for 4.

Good idea, since I don't understand 4 8-)

@bussec I think this is back in your court. I am OK with the way it is, I think we should have repertoire_id and data_processing_id in the CellReactivity object.

3. If we agree on dropping reactivity information altogether, the Docs need some more work-up to reflect the changes regarding *Reactivity objects.

Would you prefer to update the Reactivity docs in this PR or update the docs in the general Docs PR (which is yet to be created).

@bcorrie I now updated the docs, we should be good to go.

Nice... @javh I added you as reviewer - @bussec and I are happy with this, any objections to merge?

Minor comment:
The language of cell_reactivity_id wasn't quite consistent with other _id fields which are currently recommended to be globally unique, but not required to be so.  I push a change for that.
More bigger comment (I missed this earlier, sorry):
I'm confused about the directionality of the records here. Cell has no reference to CellExpression or CellReactivity. And there are "back references" to Repertoire, DataProcessing, and Cell inside CellExpression and CellReactivity.
Shouldn't it be the opposite? Shouldn't CellExpression and CellReactivity be members of Cell?

More bigger comment (I missed this earlier, sorry): I'm confused about the directionality of the records here. Cell has no reference to CellExpression or CellReactivity. And there are "back references" to Repertoire, DataProcessing, and Cell inside CellExpression and CellReactivity.
Shouldn't it be the opposite? Shouldn't CellExpression and CellReactivity be members of Cell? I'm missing something here.

@javh see the discussion starting here for more info around this: https://github.com/airr-community/airr-standards/pull/705#issuecomment-1951531948

@bcorrie I read those, but I'm still confused. How do you retrieve the expression and reactivity data for a single cell?

@bcorrie I read those, but I'm still confused. How do you retrieve the expression and reactivity data for a single cell?

ADC wise:
# Get a cell ID from the first Cell in the repository
curl -d '{"size":1}' https://covid19-1.ireceptor.org/airr/v1/cell | grep \"cell_id\"
    "cell_id": "641e1abd42e3dd9c961f2d8b",

# Get all of that Cell's expression data
curl -d '{"filters":{"op":"=","content":{"field":"cell_id","value":"641e1abd42e3dd9c961f2d8b"}}}' https://covid19-1.ireceptor.org/airr/v1/expression

Rearrangements for a different cell:
curl -d '{"filters":{"op":"=","content":{"field":"cell_id","value":"641e1adb4e75a9e20c57407e"}},"fields":["v_call","junction_aa","cell_id"]}' https://covid19-1.ireceptor.org/airr/v1/rearrangement | egrep "junction|v_call|cell_id"
    "cell_id": "641e1adb4e75a9e20c57407e",
    "v_call": "TRAV9-2",
    "junction_aa": "CAPKPGNAGGTSYGKLTF"
    "cell_id": "641e1adb4e75a9e20c57407e",
    "v_call": "TRBV7-9",
    "junction_aa": "CASSFRGTGELFF"

Reactivity would be the same, except there is no Reactivity data in the ADC yet...

This is essentially exactly what the iReceptor Gateway displays on the Cell page:

Minor comment:
The language of cell_reactivity_id wasn't quite consistent with other _id fields which are currently recommended to be globally unique, but not required to be so. I push a change for that.

I note that this is true for CellExpression, Receptor, and Cell as well, I have made similar changes...

Okay, thanks @bcorrie.  Essentially what you have here is searching all expression data for cells instead of the model where you search all cells for expression data. Which, might be the better approach, but seems contradictory with what I thought the intent behind the Cell object was - to be a container for observations about individual cells.
In this model, CellExpression and CellReactivity are akin to Rearrangement in that they are top-level-ish tables/matrices of observations keyed by cell_id.  This makes a lot of sense to me and is consistent with how these observations are stored as files.
But, then the Cell object serves no purpose because the only part of it doing any work is cell_id which is used as a key to link expression, reactivity, and V(D)J sequence (rearrangement) observations.  In which case, we can drop Cell altogether and move the fields in it elsewhere (eg, virtual_pairing -> Receptor, expression_study_method -> CellExpression, etc).
What this looks like to me is two competing logical structures. Cell contains CellExpression vs CellExpression contains Cell (represented by cell_id).  I think we need to either pick one or decide we're going to support two competing representations and design/document them accordingly.
If I were to have an opinion today, I like the model where expression, reactivity, and rearrangement are the primary observation objects. But, I dunno. It's a big change.
Does this make any sense?

@javh I agree with you on the general points, although I would rather lean towards having Cell as a container. I will try to have a look at the discussions we had on this topic before (i.e. at the point where we decided to implement it in the current fashion) before the next call.
However, I would like to note that this is becoming a discussion about revisiting the overall structure of the Cell object, while the original question of this PR - the structure of Receptor and related objects - has IMO been solved. While I agree that Receptor will likely be affected again by the forthcoming changes we are now starting to discuss, I would like to opt for merging this PR and create a new issues around the structure of Cell.

In this model, CellExpression and CellReactivity are akin to Rearrangement in that they are top-level-ish tables/matrices of observations keyed by cell_id. This makes a lot of sense to me and is consistent with how these observations are stored as files.
But, then the Cell object serves no purpose because the only part of it doing any work is cell_id which is used as a key to link expression, reactivity, and V(D)J sequence (rearrangement) observations. In which case, we can drop Cell altogether and move the fields in it elsewhere (eg, virtual_pairing -> Receptor, expression_study_method -> CellExpression, etc).

That isn't how I think of it... I agree Rearrangement, CellExpression, and CellReactivity are the base level observations. But I think Cell is also observed, no? And it is an important concept that I think we want to track as an entity. I would think of the aggregation aspect of it more along the lines of Clone. Yes, I suppose we could live without it, but it certainly seems important as a class of object to me.
Currently the Cell object doesn't have a significant number of attributes currently, but if we don't have Cell we have no way of capturing either observed or computed attributes of that object. I would imagine that the Rearrangement, CellExpression, and CellReactivity are not the only things that are important about a Cell? I think providing a solution to things like #317 and #477 need a Cell object, no?

What this looks like to me is two competing logical structures. Cell contains CellExpression vs CellExpression contains Cell (represented by cell_id).

I don't see these as two logical structures at all... Cell and CellExpression are two fundamental objects, and there is a relationship between them. The relationship is captured through cell_id and it is 1:N from Cell to CellExpression.
We can choose to store that relationship by having links to N CellExpression objects in every Cell or a single link to a Cell in every CellExpression object. We are choosing to store the relationship in CellExpression. I should note that this is the way we normally choose to store 1:N relationships when the N is large.


Enhancement Request: Add List and Gantt View to Dashboard

It would be useful to add linked icons for both the List and Gantt views from the main project Dashboard and My Projects pages.
Currently, there is only an icon for the Calendar view and clicking the text defaults to Board view.
The rationale behind this request is my attempt to migrate users to a single project management tool - Kanboard - from a diverse range of other tools including ToDoist (List view) and Microsoft Project (Gantt view). It would be useful to allow users who prefer these alternative task management approaches to be able to view their preference screens in a single click.
EDIT:
An acceptable, and maybe better, alternative to the above request would be to set a default view during new user creation, and obviously allow users to change this view from their profile page themselves.
The default view would then load the associated page any time the project name is clicked - on the dashboard, in My Project, in My Tasks etc.
The set default drop down would then of course default to Board view.

I just added direct links to the list and gantt chart in both pages. (only in master at the moment)


XPath returns null, works in Chrome developer tools

Description
If I go to the URL https://www.zima.at/wohnen/wien/ and execute the following in the Chrome Developer Tools:
$x('//div[contains(@class, "wpb_tab") and @aria-hidden="false"]//div[contains(@class, "row-inner-wrap")]')

It returns 2 nodes. If I perform the same XPath with HtmlAgilityPack it returns 0 nodes.
Fiddle or Project
HTML from: https://www.zima.at/wohnen/wien/
var selector = "//div[contains(@class, "wpb_tab") and @aria-hidden="false"]//div[contains(@class, "row-inner-wrap")]";
var nodes = html.DocumentNode.SelectNode(selector);
// Unfortunately, nodes is null

Further technical details
.NET Framework 4.8, up-to-date HtmlAgilityPack (downloaded from NuGet today).

Oh boy, can be closed, aria-hidden attribute is dynamically added by JavaScript! Totally missed that.


[Spark] Support external DSV2 catalog in RESTORE command

Which Delta project/connector is this regarding?

[x] Spark
[ ] Standalone
[ ] Flink
[ ] Kernel
[ ] Other (fill in here)

Description

How was this patch tested?

A new end-to-end test
A parser test case

Does this PR introduce any user-facing changes?

Yes, users can use RESTORE command on the tables of their external DSV2 catalogs.

FYI I am closing this and continuing my works on https://github.com/delta-io/delta/pull/2036


It's LPCWSTR and not LPWSTR

So it can work with its example: https://docs.microsoft.com/en-us/windows/win32/dlls/using-run-time-dynamic-linking

@shd8698766 : Thanks for your contribution! The author(s) have been notified to review your proposed change.


Add new to Proxy instancialization

The 'new' keyword needed to remain compatible with node v6 Proxy implementation

Thanks for still sending pull requests even though I forgot to respond to your first one!