feat(arviz): add arviz import and export

This PR implements a very basic import and export scheme with ArviZ.
Checks for new file types
If it finds netcdf ".nc" or zarr ".zarr" files, it tries to read them with ArviZ.
https://github.com/nanograv/la_forge/blob/6d230691559db4afc9ce28d75e23278432980337/la_forge/core.py#L118-L126
Converts ArviZ data and metadata into a single "chain" for La Forge to consume
It filters out any parameters that do not correspond to samples in the chain. This means that the number of parameters always equals the number of columns in the chain. It therefore skips the check later on that would add extra PTMCMC parameters, such as lnpost.
In this implementation, you need to have already named your variables in the ArviZ InferenceData object to their desired final names. I would recommend naming them to their usual PTMCMC values for backwards compatibility.
https://github.com/nanograv/la_forge/blob/6d230691559db4afc9ce28d75e23278432980337/la_forge/core.py#L127-L129
Adds an arviz cached_property to the Core class
Assuming you have a Core named my_core, calling my_core.arviz will return an ArviZ InferenceData object populated with the Core's data.
https://github.com/nanograv/la_forge/blob/6d230691559db4afc9ce28d75e23278432980337/la_forge/core.py#L687-L713
Self contained example
import arviz as az
from la_forge.core import Core
from pathlib import Path

chain_dir = Path("test_chain/")
chain_dir.mkdir(parents=True, exist_ok=True)

inf_data = az.load_arviz_data("regression1d")
inf_data.to_netcdf(chain_dir/"chain.nc")

az_core = Core(chain_dir.as_posix())

>>> print(az_core.chain)
[[ 1.5665029  -1.33202579  1.8831507  -1.34775906  1.21947951]
 [ 1.80178445 -1.18480163  1.8831507  -1.34775906  1.10626457]
 [ 1.84332941 -1.22758633  1.8831507  -1.34775906  1.0078494 ]
 ...
 [ 1.68224315 -1.18911818  1.8831507  -1.34775906  0.91507343]
 [ 2.01824417 -1.34356813  1.8831507  -1.34775906  1.14519821]
 [ 1.94768056 -1.37682251  1.8831507  -1.34775906  0.99698405]]

>>> print(az_core.params)
['slope', 'intercept', 'true_slope', 'true_intercept', 'eps']

>>> print(az_core.arviz)
Inference data with groups:
	> posterior

>>> print(az_core.arviz.posterior)
<xarray.Dataset> Size: 96kB
Dimensions:         (chain: 1, draw: 2000)
Coordinates:
  * chain           (chain) int64 8B 0
  * draw            (draw) int64 16kB 0 1 2 3 4 5 ... 1995 1996 1997 1998 1999
Data variables:
    slope           (chain, draw) float64 16kB 1.567 1.802 1.843 ... 2.018 1.948
    intercept       (chain, draw) float64 16kB -1.332 -1.185 ... -1.344 -1.377
    true_slope      (chain, draw) float64 16kB 1.883 1.883 1.883 ... 1.883 1.883
    true_intercept  (chain, draw) float64 16kB -1.348 -1.348 ... -1.348 -1.348
    eps             (chain, draw) float64 16kB 1.219 1.106 1.008 ... 1.145 0.997
Attributes:
    source:      la_forge_core
    created_at:  2024-10-04T00:00:57+00:00

Also I should raise a warning if an inference data object is imported that doesn't have all of the usual PTMCMC fields defined. This is just so users are aware some methods may fail that depend on those fields existing


Does Azure Management includes Graph API, MCAS...

Does this policy also protects other portal like Graph Explorer, Microsoft 365 admin portal or Cloud App Security?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: c6262808-898e-6196-de6c-d76f649f20fa
Version Independent ID: 2075ceab-b773-db35-a79d-7044e839bc17
Content: Conditional Access - Require MFA for Azure management - Azure Active Directory
Content Source: articles/active-directory/conditional-access/howto-conditional-access-policy-azure-management.md
Service: active-directory
Sub-service: conditional-access
GitHub Login: @MicrosoftGuyJFlo
Microsoft Alias: joflore

@Martony78 Thanks for the feedback! We are currently investigating and will update you shortly.

@Martony78, Yes, Conditional Access Policy does protect Graph Explorer, Microsoft 365 Admin Portal and the Cloud App Security.

@souravmishra-msft so you confirm this specific policy also protects Graph explorer..?

@Martony78 , Graph explorer can also be protected by using the Conditional Access Policies. Conditional Access Policies can be applied for all the service principals that are registered in Azure AD.

Hope this helps.

@Martony78 Hope the above answered your query. Should you still have any query , please feel free to tag any of us to your reply and we will be glad to help .

Of course I know Graph API can be protected by CA, I asked if it was included in "Azure Management". I suppose not since you are not able to clearly answer this simple question.
That means protecting "Azure Management" is pretty useless since I have never seen a hacker using the Azure portal... he would prefer using Graph API and/or PowerShell and in that case this Baseline Policy (or Security Default) will be helpless.


[SPARK-26762][SQL][R] Arrow optimization for conversion from Spark DataFrame to R DataFrame

What changes were proposed in this pull request?
This PR targets to support Arrow optimization for conversion from Spark DataFrame to R DataFrame.
Like PySpark side, it falls back to non-optimization code path when it's unable to use Arrow optimization.
This can be tested as below:
$ ./bin/sparkR --conf spark.sql.execution.arrow.enabled=true

collect(createDataFrame(mtcars))

Requirements

R 3.5.x
Arrow package 0.12+Rscript -e 'remotes::install_github("apache/arrowapache-arrow-0.12.0", subdir = "r")'

Note: currently, Arrow R package is not in CRAN. Please take a look at ARROW-3204.
Note: currently, Arrow R package seems not supporting Windows. Please take a look at ARROW-3204.
Benchmarks
Shall
sync && sudo purge
./bin/sparkR --conf spark.sql.execution.arrow.enabled=false --driver-memory 4g

sync && sudo purge
./bin/sparkR --conf spark.sql.execution.arrow.enabled=true --driver-memory 4g

R code
df <- cache(createDataFrame(read.csv("500000.csv")))
count(df)

test <- function() {
  options(digits.secs = 6) # milliseconds
  start.time <- Sys.time()
  collect(df)
  end.time <- Sys.time()
  time.taken <- end.time - start.time
  print(time.taken)
}

test()

Data (350 MB):
object.size(read.csv("500000.csv"))
350379504 bytes

"500000 Records"  http://eforexcel.com/wp/downloads-16-sample-csv-files-data-sets-for-testing/
Results
Time difference of 221.32014 secs

Time difference of 15.51145 secs

The performance improvement was around 1426%.
Limitations:

For now, Arrow optimization with R does not support when the data is raw, and when user explicitly gives float type in the schema. They produce corrupt values. In this case, we decide to fall back to non-optimization code path.

Due to ARROW-4512, it cannot send and receive batch by batch. It has to send all batches in Arrow stream format at once. It needs improvement later.

How was this patch tested?
Existing tests related with Arrow optimization cover this change. Also, manually tested.

@Arnoldosmium do you still need this PR?

Hi @yifeih , yes, significant perf improvements will be brought in via this.


Enhancement : Desktop Notif Through toast handler

Hi,
I have created a PR following #165.
This feature will

add cross platform desktop notifications feature through toast handler.
toast handler supports 2 modes

notify
alert (notif with a beep)

upon addition

enables user to receive events as desktop notifications

changes can be viewed here feature/desktop-notif-integration
Working of the toast handler can be visulaised here
$ kubewatch
INFO[0000] Starting kubewatch controller                 pkg=kubewatch-pod
INFO[0000] Starting kubewatch controller                 pkg=kubewatch-service
INFO[0000] Processing add to service: default/kubernetes  pkg=kubewatch-service
INFO[0000] Processing add to service: kube-system/tiller-deploy  pkg=kubewatch-service
INFO[0000] Processing add to pod: default/demo-585cfcd7c9-9n5sb  pkg=kubewatch-pod
INFO[0000] Processing add to pod: kube-system/tiller-deploy-6d6cc8dcb5-6h4qr  pkg=kubewatch-pod
INFO[0000] Kubewatch controller synced and ready         pkg=kubewatch-pod
INFO[0000] Kubewatch controller synced and ready         pkg=kubewatch-service
INFO[0008] Processing add to service: default/demo       pkg=kubewatch-service
INFO[0029] Processing delete to service: default/demo    pkg=kubewatch-service

Upon creation of service demo we get desktop notification, as shown below

Upon deletion of service demo we get desktop notification, as shown below

entire collection of notifications can be viewed in the notifacation tray.

as #165 gets merged with master, this PR can be raised.
Thoughts and comments ?

@jbianquetti-nami  and @jjo kindly have a look at this feature and share your view and comments

@tylerauerbeck , can you share you views and comments about this feature.
Can this be helpful..?
I would like to discuss on this.