Summary of GSOC2019 calls

21MAY2019
Relevant Information -

What should be xgb dmatrix for Interval regression ?
Ans :- Create two column for all the times
2-column matrix representation of outputs=labels.
un-censored output, event is obserrved, e.g. y_i = 5. (5, 5)

survival::Surv(10, 10, type="interval2")
[1] 10

left-censored output, event not observed, but we know it happened some time before t_i=5. y_i = (-Inf = \underline y_i, 5=t_i=\overline y_i)

survival::Surv(-Inf, 10, type="interval2")
[1] 10-

Right-censored output, event not observed , but we know it happened some time after t_i =5. y_i = (5,inf)

survival::Surv(5, Inf, type="interval2")
[1] 5+

Interval-censored output, event not observed, but we know it happened some time in y_i =  (5,10)

survival::Surv(5, 10, type="interval2")
[1] [5, 10]

There is no need for the survival object. This is more of a legacy.

How to handle sigma in AFT?
Ans:- Treat as a hyperparameter.

What is the dimension of the predicted value of interval regression?
Ans:- It is always one real value, not interval.

Relevant Document
https://github.com/tdhock/aft-poster/blob/master/HOCKING-AFT.pdf
http://members.cbio.mines-paristech.fr/~thocking/survival.pdf
https://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-9-269
Please Check.
@tdhock, @hcho3

May 28, 2019 Update

Done: re-created the loss charts from Toby’s paper to show that loss functions are reasonable
Done: submitted RFC to public XGBoost repo
In-Progress: Python PoC for end-to-end gradient boosting with survival loss
In-Progress: Derive formulas for AFT

hi @avinashbarnwal  can you please post the source code and rendered images for the loss charts you created?

@tdhock See https://github.com/avinashbarnwal/gsoc/tree/master/AFT/R

looks like a good start but the plot is incorrect for uncensored data. it should look like the square loss around the label for the normal distribution. also you should try creating a facetted ggplot, with one panel per censoring type. for inspiration here is the code I used for the 1-page AFT poster, https://github.com/tdhock/aft-poster/blob/master/figure-loss.R

Thanks, Prof. @tdhock  I am looking into it.

looks better @avinashbarnwal ! glad to see that you got the facetted ggplots working.
however it looks like there is a problem with your computation of the logistic loss for the uncensored output -- it should be minimal at the label.

the x axis title says log scale but the ticks indicate linear scale (i.e the distance between 2.5 and 5.0 is the same as between 5.0 and 7.5)
the x axis title also says that you are plotting the predicted survival time in days but I don't think that is right -- aren't you actually plotting the predicted center of the gaussian/logistic = log(predicted time) ?
the legend says logLogistic and logNormal but you may want to change that to normal / logistic for simplicity -- it is usually easier to deal with the real valued labels rather than the positive valued labels. technically if t_i>0 is the survival time and y_i = log(t_i) \in R is the real-valued log(survival time) then the model assumes t_i  ~ lognormal( f(x_i), \sigma^2) which is the same thing as y_i ~ normal( f(x_i), \sigma^2)
please increase the number of grid points at which you evaluate the loss functions (the curves are pretty choppy)
please increase the size of the interval censored output, so we can see the loss function approach zero and flatten out at the center of the limits.

also for next week's homework please check your formulas for the gradient in the overleaf. add a row of plots to the figure that shows the loss functions. use facet_grid(fun ~ type) -- rows for different functions (loss, gradient, hessian)

first row should be loss function
second row should be gradient

@tdhock @hcho3
I have made the loss function with all the changes required.
Here is the link for the Plot - https://github.com/avinashbarnwal/GSOC-2019/blob/master/AFT/R/loss_aft.png
I have changed the code as well.
Code Link - https://github.com/avinashbarnwal/GSOC-2019/blob/master/AFT/R/assigment1.R
@ Please check the formula for the Normal - Censored part. I have used different formula compared to
formula given in this https://github.com/avinashbarnwal/GSOC-2019/blob/master/paper/HOCKING-AFT.pdf, rather than this i have used formula given in this document. (http://home.iitk.ac.in/~kundu/paper146.pdf)

looks better Avinash. but why doesn't the uncensored loss go to zero? (it
should...)
about the normal - censored loss, you should double check your work by
using the normal CDF  (pnorm in R)
On Wed, Jun 5, 2019 at 6:47 AM Avinash Barnwal<EMAIL_ADDRESS>wrote:

@tdhock https://github.com/tdhock @hcho3 https://github.com/hcho3
I have made the loss function with all the changes required.
Here is the link for the Plot -
https://github.com/avinashbarnwal/GSOC-2019/blob/master/AFT/R/loss_aft.png
I have changed the code as well.
Code Link -
https://github.com/avinashbarnwal/GSOC-2019/blob/master/AFT/R/assigment1.R
@ Please check the formula for the Normal - Censored part. I have used
different formula compared to
formula given in this
https://github.com/avinashbarnwal/GSOC-2019/blob/master/paper/HOCKING-AFT.pdf,
rather than this i have used formula given in this document. (
http://home.iitk.ac.in/~kundu/paper146.pdf)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/avinashbarnwal/xgboost/issues/1?email_source=notifications&email_token=AAHDX4QWXCGCFMV5LJLUBYDPY67WLA5CNFSM4HOPO5U2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODW7YEAI#issuecomment-499089921,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAHDX4T5DS6SKKWT47LROO3PY67WLANCNFSM4HOPO5UQ
.

Prof. @tdhock, The uncensored
Loss function Normal- -log(1/(t.lowersigmasqrt(2pi))exp((log(t.lower/y.hat))**2/(-2sigmasigma)))
We have a constant term -log(1/(t.lowersigmasqrt(2*pi)) irrespective of y.hat. This makes it non-zero, this is based on my thinking.
Similarly for Logistic.
Sorry, I meant Normal Uncensored formula to check.

right but we usually subtract away the constant terms. if you do that you
should recover the square loss which is 0 at the min.
On Wed, Jun 5, 2019 at 10:22 AM Avinash Barnwal<EMAIL_ADDRESS>wrote:

Prof. @tdhock https://github.com/tdhock, The uncensored
Loss function Normal- -log(1/(t.lowersigmasqrt(2
pi))exp((log(t.lower/y.hat))**2/(-2sigmasigma)))
We have a constant term -log(1/(t.lowersigmasqrt(2*pi)) irrespective of
y.hat. This makes it non-zero, this is based on my thinking.
Similarly for Logistic.
Sorry, I meant Normal Uncensored formula to check.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/avinashbarnwal/xgboost/issues/1?email_source=notifications&email_token=AAHDX4UKPXEFEYOE3EVFZBTPY7Y6ZA5CNFSM4HOPO5U2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXANOZQ#issuecomment-499177318,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAHDX4T65PX5QDPTNE52GYDPY7Y6ZANCNFSM4HOPO5UQ
.

Hi Prof. @tdhock, @hcho3
Please find the updated document for AFT(https://github.com/avinashbarnwal/GSOC-2019/blob/master/doc/Accelerated_Failure_Time.pdf). I think I have made mistake here as I am taking gradient and hessian with respect to y.hat, not X\beta.
For least square loss function-gradient and hessian with respect to y.hat and X\beta are same. But it would start mattering when we have link functions involved. Similarly, for classification, we take gradient and hessian with respect to  X\beta, not y.hat.
For reference, please check doc - https://cran.r-project.org/web/packages/gbm/vignettes/gbm.pdf
Bernoulli part.
This leads to more decision on notations as in survival document we are using f for pdf and in the above document, we have f(x_i) for X\beta.
Please let me know your thoughts.

I dont think there is an issue because the link function is always the identity with the normal and logistic models. again we should be able to use the formulas in the survival manual.

Prof. @tdhock, I think we need to calculate gradient with respect to log(y.hat) not y.hat as log(y.hat) = X\beta. Please let me know.

section 6.8 of survival manual gives derivatives with respect to eta, which is the real-valued prediction.

in your notation you use log(y.hat) for the real valued prediction, so that is the same as eta from the survival manual.
in your PDF please do not use x\beta as that is only valid for linear models

Prof. @tdhock, Thanks. I will use eta for log(y.hat) and take gradient and Hessian based on that. In my document, I was taking gradient and Hessian based on y.hat, not on eta. I will make the correction.

Prof. @tdhock, @hcho3, I have created the document and plot for loss, negative gradient, and hessian.
Document - https://github.com/avinashbarnwal/GSOC-2019/blob/master/doc/Accelerated_Failure_Time.pdf
Plot - https://github.com/avinashbarnwal/GSOC-2019/blob/master/AFT/R/loss_grad_hess_aft.png
Please let me know your thoughts.

the plot looks good except for the hessian for the interval censored outputs for small predictions -- it seems to be too large. can you please double check? it should look like right censored outputs (i.e. with finite lower limit) on the left side of the plot

Thanks, Prof. @tdhock.  Please check it again.

Thanks, Prof. @tdhock. I have changed the code and plot both. I had one sign in the formula wrong. Please recheck the plot (https://github.com/avinashbarnwal/GSOC-2019/blob/master/AFT/R/loss_grad_hess_aft.png). Do you think Interval hessian is correct for Normal distribution?

Hi @hcho3 and Prof. Toby,
Please find the python implementation of gradient boosting for AFT.
Code  - https://github.com/avinashbarnwal/GSOC-2019/blob/master/AFT/py/gb_aft.ipynb
Plot    - https://github.com/avinashbarnwal/GSOC-2019/blob/master/AFT/py/Nesterov_False.png
and let me know your thoughts.

We'll need to check the POC implementation, since the training loss should trend down, not up.
Also, let's plot mean(abs(log(Y) - eta)), since the predicted score eta should try to match log(Y).

Hi @hcho3 and Prof. @tdhock,
As discussed with @hcho3, I am working on documentation for binomial loss and POC for the same.

Hi @hcho3 and Prof. @tdhock,
Please find the document for binomial loss and code below:-
Document -  https://github.com/avinashbarnwal/GSOC-2019/blob/master/doc/Binomial_Loss.pdf.
Code          - https://github.com/avinashbarnwal/GSOC-2019/blob/master/AFT/R/binomial_loss.R.
Plot            - https://github.com/avinashbarnwal/GSOC-2019/blob/master/AFT/R/binomial_loss.png.

there is still something wrong for the interval hessian for small predicted values on https://github.com/avinashbarnwal/GSOC-2019/blob/master/AFT/R/loss_grad_hess_aft.png -- it should be constant as the prediction goes to zero, log(prediction) goes to -Inf

binomial loss looks reasonable, except for the x axis label.
please (1) use facet_grid and (2) use more grid points and (3) maybe have different columns or colors for different labels

Hi Prof. @tdhock  and @hcho3 ,
Please check the R plots and Python Plots again -
R Plots -

AFT - https://github.com/avinashbarnwal/GSOC-2019/blob/master/AFT/R/loss_grad_hess_aft.png
Binomial Loss - https://github.com/avinashbarnwal/GSOC-2019/blob/master/AFT/R/binomial_loss.png

Python Plots-

Log loss, Data Type - Mixed - https://github.com/avinashbarnwal/GSOC-2019/blob/master/AFT/py/Nesterov_False_Loss_LogLoss_Data_Mixed.png
Mae, Data Type      - Mixed - https://github.com/avinashbarnwal/GSOC-2019/blob/master/AFT/py/Nesterov_False_Loss_Mae_Data_Mixed.png
Log loss, Data Type - Uncen - https://github.com/avinashbarnwal/GSOC-2019/blob/master/AFT/py/Nesterov_False_Loss_Logloss_Data_Uncensored.png
Mae, Data Type      - Uncen - https://github.com/avinashbarnwal/GSOC-2019/blob/master/AFT/py/Nesterov_False_Loss_Mae_Data_Uncensored.png

aft hessian looks good now but binomial hessian has a problem: should be
zero as prediction goes to Inf
On Tue, Jul 2, 2019 at 12:48 PM Avinash Barnwal<EMAIL_ADDRESS>wrote:

Hi Prof. @tdhock https://github.com/tdhock and @hcho3
https://github.com/hcho3 ,
Please check the R plots and Python Plots again -
R Plots -

AFT -
https://github.com/avinashbarnwal/GSOC-2019/blob/master/AFT/R/loss_grad_hess_aft.png
Binomial Loss -
https://github.com/avinashbarnwal/GSOC-2019/blob/master/AFT/R/binomial_loss.png

Python Plots-

Log loss, Data Type - Mixed -
https://github.com/avinashbarnwal/GSOC-2019/blob/master/AFT/py/Nesterov_False_Loss_LogLoss_Data_Mixed.png
Mae, Data Type - Mixed -
https://github.com/avinashbarnwal/GSOC-2019/blob/master/AFT/py/Nesterov_False_Loss_Mae_Data_Mixed.png
Log loss, Data Type - Uncen -
https://github.com/avinashbarnwal/GSOC-2019/blob/master/AFT/py/Nesterov_False_Loss_Logloss_Data_Uncensored.png
Mae, Data Type - Uncen -
https://github.com/avinashbarnwal/GSOC-2019/blob/master/AFT/py/Nesterov_False_Loss_Mae_Data_Uncensored.png

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/avinashbarnwal/xgboost/issues/1?email_source=notifications&email_token=AAHDX4VZ2MJ5RUWZJTB6DRTP5OWIBA5CNFSM4HOPO5U2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZCLNYA#issuecomment-507819744,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAHDX4X2M3WDCMXBXZ2NZU3P5OWIBANCNFSM4HOPO5UQ
.

paper with an example of a real-world problem with only left, right, and interval censored labels: http://proceedings.mlr.press/v28/hocking13.html

discussed:

compute loss via dnorm or log/exp? in C++ normal distn functions are provided in std lib, but logistic dist functions are not. logistic provided in boost but we don't want to depend on that.
check work on binomial loss via duplicate data trick.

Hi, Prof @tdhock and @hcho3,
Please check the updated plot and code for binomial loss.
Plot - https://github.com/avinashbarnwal/GSOC-2019/blob/master/AFT/R/binomial_loss.png
Code - https://github.com/avinashbarnwal/GSOC-2019/blob/master/AFT/R/binomial_loss.R
Doc - https://github.com/avinashbarnwal/GSOC-2019/blob/master/doc/Binomial_Loss.pdf

that looks more reasonable
suggestions: (1) more grid points, (2) no need to use aes(color) because
you already put the different functions in different panels, (3) add more
columns for different labels, e.g. y=5, n=10; y=0, n=10;, y=2, n=10
On Tue, Jul 2, 2019 at 9:44 PM Avinash Barnwal<EMAIL_ADDRESS>wrote:

Hi, Prof @tdhock https://github.com/tdhock and @hcho3
https://github.com/hcho3,
Please check the updated plot and code for binomial loss.
Plot -
https://github.com/avinashbarnwal/GSOC-2019/blob/master/AFT/R/binomial_loss.png
Code -
https://github.com/avinashbarnwal/GSOC-2019/blob/master/AFT/R/binomial_loss.R
Doc -
https://github.com/avinashbarnwal/GSOC-2019/blob/master/doc/Binomial_Loss.pdf
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/avinashbarnwal/xgboost/issues/1?email_source=notifications&email_token=AAHDX4RFXVQLPHGHKXV7VRTP5QVD7A5CNFSM4HOPO5U2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZDH2QY#issuecomment-507936067,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAHDX4TPGGDI6EMLZVEHBVDP5QVD7ANCNFSM4HOPO5UQ
.

Hi Prof. @tdhock ,
Please find the updated binomial loss:-
R
Plot - https://github.com/avinashbarnwal/GSOC-2019/blob/master/AFT/R/binomial_loss.png.
Py
Updated Plots
LogLoss - Mixed - https://github.com/avinashbarnwal/GSOC-2019/blob/master/AFT/py/Nesterov_False_Loss_LogLoss_Data_Mixed.png
Mae        - Mixed - https://github.com/avinashbarnwal/GSOC-2019/blob/master/AFT/py/Nesterov_False_Loss_Mae_Data_Uncensored.png
LogLoss - Uncensored - https://github.com/avinashbarnwal/GSOC-2019/blob/master/AFT/py/Nesterov_False_Loss_Logloss_Data_Uncensored.png
Mae        - Uncensored - https://github.com/avinashbarnwal/GSOC-2019/blob/master/AFT/py/Nesterov_False_Loss_Mae_Data_Uncensored.png
C++
Functions for Pdf and cdf of Normal and Logistic Distribution
https://github.com/avinashbarnwal/GSOC-2019/tree/master/AFT/C%2B%2B
I have tested with x=0 for each distributions and it was giving correct results compared to R inbuilt functions.
Thanks, @hcho3 for changing the formula hessian for interval data.

@avinashbarnwal The AFT loss plots now look good. I'll take a look at the C++ code soon.

Hi  Prof. @tdhock and @hcho3
I have been working on implementing loss, negative gradient and hessian functions for AFT in C++ and test it through plots on Python.
For details- Please find the link.
https://github.com/avinashbarnwal/GSOC-2019/tree/master/AFT/C%2B%2B
Please let me know your thoughts.

@avinashbarnwal I modified your notebook slightly to plot the uncensored AFT loss with yhat in the X axis:

The curve for the normal should not cliff off like that.

In general, let's try to reproduce the R plots using C++ code. That way, we have assurance that C++ code does what R code does.

right, the loss function should be convex (for a fixed scale parameter)
On Tue, Jul 16, 2019 at 10:25 PM Philip Hyunsu Cho<EMAIL_ADDRESS>wrote:

In general, let's try to reproduce the R plots using C++ code. That way,
we have assurance that C++ code does what R code does.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/avinashbarnwal/xgboost/issues/1?email_source=notifications&email_token=AAHDX4X4322NHO4MS7I3Z7DP7YVDXA5CNFSM4HOPO5U2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2CBFBI#issuecomment-511971973,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAHDX4SXEVZXCIGAMZ6LGNLP7YVDXANCNFSM4HOPO5UQ
.

hi there @avinashbarnwal is the R learning algo for AFT losses working yet?
When it is I would suggest modifying https://github.com/tdhock/neuroblastoma-data/blob/master/iregnet.R which I wrote for benchmarking a different learning algorithm for censored outputs, on 33 different labeled data sets.

@tdhock Thanks for the link. The datasets should come in handy.

Prof. @tdhock ,
For both AFT and binomial loss, R learning algorithms(loss, negative gradient and hessian) are working.
I am currently writing C++ functions for binomial loss. Soon we will have everything in the xgboost package.

ok where is the fork/branch with the AFT models? can I install it?
Here is a new benchmark script https://github.com/tdhock/neuroblastoma-data/blob/master/xgboost.R#L41
The benchmark includes 33 different data sets, each with several train/test splits (designated by fold ID numbers in folds.csv file).
To run the code please fork then clone that repo, and then modify the "xgboost.R" script so xgboost works on these learning problems. (run the script / start R in the neuroblastoma-data directory)
The script also runs baselines penaltyLearning::IntervalRegressionCV (penaltyLearning.scale1) and constant (always predict 0).

@avinashbarnwal

R learning algorithms(loss, negative gradient and hessian) are working

I don't think you have boosting PoC in R? We do have boosting PoC in Python, however.

Prof. @tdhock ,
We haven't add the AFT loss function in xgboost package yet. We are testing it first in C++ and then I will add to the package. Please let me know if you want proof of concept in R.

@avinashbarnwal For now, can we run Python boosting PoC on the 33 datasets?

no problem if it is not ready yet but you may consider using those data sets to test /debug the algo.
if you don't use that R script then you need to know that

input/feature matrix is in data/DATA_NAME/inputs.csv.xz -- some features are missing (NA) or infinite in some rows -- you can ignore those features/columns.
output/label matrix is in data/DATA_NAME/outputs.csv.xz
would be good to compare to another learner, at least a baseline constant prediction, would be better to compare against regularized linear model / computing test AUC as in my R script.
suggested cross-validation folds are given in folds.csv files, e.g.

> folds.csv.vec <- Sys.glob("data/*/cv/*/folds.csv")
> folds.csv.vec
 [1] "data/ATAC_JV_adipose/cv/equal_labels/folds.csv"
 [2] "data/CTCF_TDH_ENCODE/cv/equal_labels/folds.csv"
 [3] "data/H3K27ac-H3K4me3_TDHAM_BP/cv/equal_labels/folds.csv"
 [4] "data/H3K27ac_TDH_some/cv/equal_labels/folds.csv"
 [5] "data/H3K27me3_RL_cancer/cv/equal_labels/folds.csv"
 [6] "data/H3K27me3_TDH_some/cv/equal_labels/folds.csv"
 [7] "data/H3K36me3_AM_immune/cv/equal_labels/folds.csv"
 [8] "data/H3K36me3_TDH_ENCODE/cv/equal_labels/folds.csv"
 [9] "data/H3K36me3_TDH_immune/cv/equal_labels/folds.csv"
[10] "data/H3K36me3_TDH_other/cv/equal_labels/folds.csv"
[11] "data/H3K4me1_TDH_BP/cv/equal_labels/folds.csv"
[12] "data/H3K4me3_PGP_immune/cv/equal_labels/folds.csv"
[13] "data/H3K4me3_TDH_ENCODE/cv/equal_labels/folds.csv"
[14] "data/H3K4me3_TDH_immune/cv/equal_labels/folds.csv"
[15] "data/H3K4me3_TDH_other/cv/equal_labels/folds.csv"
[16] "data/H3K4me3_XJ_immune/cv/equal_labels/folds.csv"
[17] "data/H3K9me3_TDH_BP/cv/equal_labels/folds.csv"
[18] "data/detailed/cv/R-3.6.0-chrom/folds.csv"
[19] "data/detailed/cv/R-3.6.0-profileID/folds.csv"
[20] "data/detailed/cv/R-3.6.0-profileSize/folds.csv"
[21] "data/detailed/cv/R-3.6.0-sequenceID/folds.csv"
[22] "data/detailed/cv/chrom/folds.csv"
[23] "data/detailed/cv/profileID/folds.csv"
[24] "data/detailed/cv/profileSize/folds.csv"
[25] "data/detailed/cv/sequenceID/folds.csv"
[26] "data/systematic/cv/R-3.6.0-chrom/folds.csv"
[27] "data/systematic/cv/R-3.6.0-profileID/folds.csv"
[28] "data/systematic/cv/R-3.6.0-profileSize/folds.csv"
[29] "data/systematic/cv/R-3.6.0-sequenceID/folds.csv"
[30] "data/systematic/cv/chrom/folds.csv"
[31] "data/systematic/cv/profileID/folds.csv"
[32] "data/systematic/cv/profileSize/folds.csv"
[33] "data/systematic/cv/sequenceID/folds.csv"
>

Thanks. Prof. @tdhock I will show the results for given datasets.

Also if we have time we should consider supporting the Extreme Value distribution (in addition to logistic and normal), since in stats people often like to use Weibull/Exponential models, and they are special cases of the EV dist.

Hi Prof. @tdhock and @hcho3,
I have added the C++ code for binomial loss and notebook for validating the same.
** Link **
** C++ ** - https://github.com/avinashbarnwal/GSOC-2019/tree/master/BinomialLoss/C%2B%2B
** Notebook ** - https://github.com/avinashbarnwal/GSOC-2019/blob/master/BinomialLoss/Python Notebook/Visualizing distributions.ipynb
I am in the process of testing the AFT for 33 datasets.
** Link ** - https://github.com/avinashbarnwal/GSOC-2019/blob/master/AFT/test/data/neuroblastoma-data-master/src/notebook/001_data_massage.ipynb.

@avinashbarnwal I left a comment about the binomial loss implementation: avinashbarnwal/GSOC-2019#5. We do not want to use n!/ ((n-r)! r!)  because 1) it has many redundant computation and 2) it may cause overflows.

@tdhock I modified @avinashbarnwal's notebook a bit to generate a combined plot with the C++ AFT loss:

So we're pretty confident that his AFT implementation is correct.

that looks reasonable

I added some stub code in the XGBoost codebase, so that we can store y_lower and y_upper as part of the data matrix (DMatrix) object: https://github.com/avinashbarnwal/xgboost/commits/survival_analysis
@avinashbarnwal will be working to graft the AFT loss into XGBoost.

Hi @hcho3  and Prof. @tdhock,
We have been able to test the xgboost for aft for dataset - detailed, systematic and simulated datasets. Detailed and Systematic datasets are mentioned above and Simulated dataset is generated by me.
We have tested for aft normal.
Please find the plots here - https://github.com/avinashbarnwal/GSOC-2019/tree/master/AFT/test/data/neuroblastoma-data-master/src/notebook/xgboost plots.
But this is not working for first few datasets.
Link for notebook - https://github.com/avinashbarnwal/GSOC-2019/blob/master/AFT/test/data/neuroblastoma-data-master/src/notebook/002_xgboost_test.ipynb
Please let me know your thoughts.

@avinashbarnwal Nicely done. This is first time AFT loss is implemented in XGBoost.

But this is not working for first few datasets

Can you be more specific? Why do you think XGBoost is not “working”?

I think we have not put the overflow in the formula as we have here https://github.com/avinashbarnwal/GSOC-2019/blob/master/AFT/py/_aft_loss.py in the denominator.

@avinashbarnwal I’m still curious about your line of reasoning here. What indicators or symptoms do you see that you think indicate deficiencies in the current implementations?
Once we know the indicators (“bad smells”), then we can discuss possible diagnosis for that

I have been trying to print the first order and second order  https://github.com/avinashbarnwal/GSOC-2019/blob/master/xgboost/src/objective/aft_obj.cc. But it is not printing.

Use LOG(CONSOLE) in place of std::cout.

Hi,
Please find the first order and second order for each iteration.

LOG(CONSOLE) was not giving the right answer. I have attached the std::cout.

@avinashbarnwal Can you add a space between two numbers?

Please find with spaces.

I don't see what is the problem with the plots.
train error decreasing and validation error going up after some iterations
= looks ok
I would suggest computing test error in K-fold CV, and compare with test
error of baselines which have been computed in the neuroblastoma-data repo
On Fri, Aug 2, 2019 at 6:12 AM Avinash Barnwal<EMAIL_ADDRESS>wrote:

Please find with spaces.
[image: Log]
https://user-images.githubusercontent.com/6061417/62372509-a9d0ad00-b505-11e9-9265-2f20b97b30c0.png
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/avinashbarnwal/xgboost/issues/1?email_source=notifications&email_token=AAHDX4X2IUL4V4U7VZKTXS3QCQXEZA5CNFSM4HOPO5U2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3NWP7A#issuecomment-517695484,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAHDX4V5KHLJYM4NZDNWEMLQCQXEZANCNFSM4HOPO5UQ
.

@tdhock The issue is that we have infinities in the first and second order gradients.

please double check your C++ code with the R code
is it specific to one loss function?
On Fri, Aug 2, 2019 at 10:54 AM Philip Hyunsu Cho<EMAIL_ADDRESS>wrote:

@tdhock https://github.com/tdhock The issue is that we have infinities
in the first and second order gradients.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/avinashbarnwal/xgboost/issues/1?email_source=notifications&email_token=AAHDX4SXLMKMGQAKDTHUOVLQCRYC7A5CNFSM4HOPO5U2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3ON26Q#issuecomment-517791098,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAHDX4XU5DD2E2F5HBUQ2MLQCRYC7ANCNFSM4HOPO5UQ
.

In C++ code, we have denominators in the first-order gradient going to 0. We have a problem similar to this. https://github.com/dmlc/xgboost/issues/2955. We can use a similar idea to prevent the overflow.

Hi @hcho3 and Prof. @tdhock,
I have added the plot for first data -  "data/ATAC_JV_adipose/cv/equal_labels/folds.csv"
Plots - https://github.com/avinashbarnwal/GSOC-2019/tree/master/AFT/test/data/neuroblastoma-data-master/src/notebook/xgboost plots/ATAC_JV_adipose
I have changed the overflow problem as mentioned above. Please find the updated code.
Code - https://github.com/avinashbarnwal/xgboost/blob/survival_analysis/src/objective/aft_obj.cc
Please let me know what you think?

plots look good
this is the first time I look at your C++ implementation and I think it
would be better to treat left/right as special cases of interval -- do you
think that would be possible? it would reduce duplication in the code...
On Fri, Aug 2, 2019 at 7:14 PM Avinash Barnwal<EMAIL_ADDRESS>wrote:

Hi @hcho3 https://github.com/hcho3 and Prof. @tdhock
https://github.com/tdhock,
I have added the plot for first data -
"data/ATAC_JV_adipose/cv/equal_labels/folds.csv"
Plots -
https://github.com/avinashbarnwal/GSOC-2019/tree/master/AFT/test/data/neuroblastoma-data-master/src/notebook/xgboost plots/ATAC_JV_adipose
I have changed the overflow problem as mentioned above. Please find the
updated code.
Code -
https://github.com/avinashbarnwal/xgboost/blob/survival_analysis/src/objective/aft_obj.cc
Please let me know what you think?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/avinashbarnwal/xgboost/issues/1?email_source=notifications&email_token=AAHDX4X35WNTAKUR7PAW6I3QCTSZFA5CNFSM4HOPO5U2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3PFEFA#issuecomment-517886484,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAHDX4UXPWOJD6KT46F42KTQCTSZFANCNFSM4HOPO5UQ
.

Hi Prof. @tdhock,
I fear, I won't be able to combine left/right-censored to the interval as we have a different formulation for each censored data types.
Hi @hcho3,
As discussed, Please find the negative gradients plots - https://github.com/avinashbarnwal/GSOC-2019/blob/master/AFT/Python Notebook/Visualizing distributions.ipynb and check the cell number 11. I have changed the range from
#x_vals   = np.linspace(np.log(21), np.log(215), 200)
to
x_vals   = np.linspace(np.log(2**1), 1000, 200)
I see that it is going to nan very quickly.

you are right that "we have a different formulation for each censored data types" but the left and right can be expressed as special cases of interval, which would simplify the code.
e.g.
 grad  = (pdf_u-pdf_l)/(sigma*std::max(cdf_u-cdf_l,eps)); //interval
    grad = pdf/(sigma*cdf); //left
// could be instead:
if(y_lower== -INFINITY) {pdf_l=0;cdf_l=0}

Also I would suggest simplifying aft_obj.cc by creating a class for each different AFT distribution. This would reduce duplication between the _interval and _uncensored functions e.g.
// right now:
grad_interval(){
      case AFTNoiseDistribution::kNormal:
      pdf_u  = common::aft::dnorm(z_u,0,1);
      pdf_l  = common::aft::dnorm(z_l,0,1);
      cdf_u  = common::aft::pnorm(z_u,0,1);
      cdf_l  = common::aft::pnorm(z_l,0,1);
      break;
grad_uncensored(){
     case AFTNoiseDistribution::kNormal:
      pdf  = common::aft::dnorm(z,0,1);
      _grad = grad_norm(z,0,1);
      break;
//suggestion:
class AFTLoss: // base class
  double grad_uncensored(...){
    pdf = pnorm(z, 0, 1);
    _grad = compute_gradient(z, 0, 1);
    grad = _grad/(sigma*pdf);
class AFTNormal: AFTLoss
  dnorm = common::aft::dnorm;
  pnorm = common::aft::pnorm;
  double compute_gradient(...){
    // same as your current grad_norm function
  }
class AFTLogistic: AFTLoss
  dnorm = common::aft::dlogistic
 // etc

Hi Prof. @tdhock,
Thanks for explaining more. I have got it. I will make the changes accordingly.

@tdhock Given that GSOC end is coming, can we postpone code re-factoring until after we can get AFT and binomial loss working? Currently @avinashbarnwal needs to finish putting in the binomial loss into XGBoost.

@avinashbarnwal @tdhock And ideally, I'd like to see pull request for AFT at dmlc/xgboost pretty soon.

Hi @hcho3 and Prof. @tdhock,
I was testing the algorithm with different eps for different datasets like first and second dataset mentioned above.
eps = 1e-12f gave the best result. I have kept that as final eps to be used.

I wouldn't suggest to compromise on code quality just because the end of GSOC is near. I think at least one PR (with good quality code) on the main xgboost repo should constitude a successful GSOC project. (IMO no problem if there is not time for the binomial loss)

Hi Prof. @tdhock and @hcho3,
I am doing my best to make changes suggested by Prof. @tdhock. As suggested by @hcho3, we won't be able to test the class in python through c++. We can directly test with datasets and manual testing.
I am going to maintain whatever we have in survival_analysis branch in xgboost and push all the changes to the new branch I am going to create named as "survival_analysis1".

@avinashbarnwal Is it necessary to create a new branch? We can always revert changes later.

Hi @hcho3,
I was thinking not to touch whatever is working. Please let me know what you think?

@avinashbarnwal On the second thought, the second branch makes sense. We want to easily compare two implementations so that the new implementation can be validated against the current one. Go ahead and create new branch.

Hi @hcho3 and Prof. @tdhock,
I have made the suggested changes in https://github.com/avinashbarnwal/xgboost/blob/survival_analysis1/src/objective/aft_obj.cc.
Please let me know your thoughts.
Now i need to change in the metric file. As we can change the loss as well left/right as interval.
I can create the class object AFTloss in the objective.cc and then call in metric survival_metric.cc.
But i am not sure how to do it?
Please let me know your thoughts.
Link of metric file :- https://github.com/avinashbarnwal/xgboost/blob/survival_analysis1/src/metric/survival_metric.cc

@avinashbarnwal You need to mark functions virtual if you want child classes override these functions from the parent class.

Yes. I was thinking the same. Thanks for the suggestions. I am making the changes now.

for consistency of naming I would suggest something like "extreme value" rather than "weibull" because the other distributions you consider are normal and logistic, which are for real-valued numbers (the correspoonding positive valued distributions are called log-normal and log-logistic). Weibull is a positive-valued distribution whose real-valued counterpart is called the extreme value distribution.

Yes. I will make the changes in the name straightaway. Please let me know if you think plots look good.

why do you introduce the AFTEventType? seems overly complicated to me; would be simpler to just test if y_upper==y_lower then uncensored, otherwise censored.
again I don't think you should have four cases for computing the grad/hessian. instead just two cases (uncensored or interval) with interval having special cases for left/right censoring.
I would suggest moving grad_norm, hess_norm to members of the normal sub-class of the AFTloss class.

Please check https://github.com/avinashbarnwal/xgboost/blob/survival_analysis1/src/objective/aft_obj.cc. We have only 2 cases. Event variable is important to categorize left/right and interval.

name should be "extreme" to be consistent with R:
> names(survival::survreg.distributions)
 [1] "extreme"     "logistic"    "gaussian"    "weibull"     "exponential"
 [6] "rayleigh"    "loggaussian" "lognormal"   "loglogistic" "t"
>

the plots look reasonable, but the code is extremely unusual. exp(1)**z should be exp(z)

I looked at aft_obj.cc but I don't see any reason to keep the event variable.
right now you have three cases, one for each censoring type, and each of them makes the same call,
        first_order_grad  = aft::grad_interval(y_lower[i], y_higher[i], yhat[i],
                                              param_.aft_sigma,event);

I would suggest removing the event variable, removing the three cases, and simply call aft::grad_interval whenever there is censoring. Inside aft::grad_interval you can test if lower/upper limit is infinite.

@tdhock We re-organized the code to incorporate your suggestion: https://github.com/avinashbarnwal/xgboost/commit/f52c96708ce54f12f594ec2a38537166b07d253f. We used polymorphism to massively simplify the metric and objective functions.

ok but please further simplify https://github.com/avinashbarnwal/xgboost/commit/f52c96708ce54f12f594ec2a38537166b07d253f#r34598175

I am looking into it.

Hi Prof. @tdhock and @hcho3,
I have pushed the extreme distribution to xgboost and also made changes to R files and plot with changed naming convention to "Extreme".

how will that be specified in the user interface?
right now the "survival:cox" objective is documented on https://xgboost.readthedocs.io/en/latest/parameter.html as "negative values are considered right censored" so that implies to me that the "survival:" prefix means all outputs are positive-valued.
to avoid confusion with that objective / output, I would suggest a new prefix for the three loss functions, "aft:gaussian" "aft:extreme" and "aft:logistic" -- does that make sense to you @hcho3  ? in each of these cases the output should be specfied by an interval (a,b) where both a and b are potentially negative. (log survival times) in other words, if you have an output vector c(5.5, -0.1) for survival:cox, that means the first output is un-censored, and the second is right censored (0.1, INFINITY) on the positive scale (survival times). The corresponding outputs that we would use are (1.7, 1.7) and (-2.3, INFINITY) --- on the real-valued scale (log survival times)

@tdhock I think "survival:cox" uses survival time, not log survival time. So the output would be positive in that case. The current AFT implementation uses survival time as well.

ok in that case the -0.1 output for survival:cox becomes (0.1, INFINITY) for us.
and in that case  "survival:lognormal"  "survival:loglogistic" "survival:weibull" may make more sense.
however I would suggest specfying outputs as real valued (potentially negative) log survival times, which has two advantages in my opinion

can use the entire double precision number range for storage (negative numbers can not be used if the output is positive / survival time)
consistency with the underlying real-valued distributions (gaussian, logistic, extreme)

maybe both aft:gaussian (using real valued log survival times) and survival:loggaussian (using positive valued un-logged survival times) etc can be supported?

@tdhock That's reasonable suggestion. We will need to revise survival:cox implementation to use log survival times.

Hi Prof @tdhock, @hcho3
I am going through the document https://github.com/avinashbarnwal/GSOC-2019/blob/master/paper/AFT/survival.pdf shared by you. In page number 82, we have Extreme distribution section where it says
"If y is Weibull then log(y) is distributed according to the (least) extreme
value distribution."
In previous cases - we have normal and logistic, where y is following normal and logistic.
Please let me know if i am missing anything.
I am trying to test extreme distribution using a notebook for the first few datasets. I am finding inf in errors.
Link for notebook - https://github.com/avinashbarnwal/GSOC-2019/blob/master/AFT/test/data/neuroblastoma-data-master/src/notebook/002_xgboost_test.ipynb
ScreenShot of Gradient and hessian

I have used all the formulas mentioned in the document.

the loss function for the extreme value dist goes to Inf very fast as the predicted value gets smaller.
if it helps, there is an alternative way to compute the loss (my.cost in code below) which is more numerically stable (results in Inf less often):
## extreme value CDF.
F <- function(z)1-exp(-exp(z))
max.pred <- 5
min.pred <- -15
curve(F, min.pred, max.pred)
s <- 1
y.lo <- -5
library(data.table)
pred.dt <- data.table(pred=seq(min.pred, max.pred, l=101))
pred.dt[, z.lo := (y.lo-pred)/s]
pred.dt[, p.lo := F(z.lo)]
pred.dt[, e.lo := 1/(exp(exp(z.lo))-1)]
pred.dt[, my.cost := log(1+e.lo)-log(e.lo)]
pred.dt[, naive.cost := -log(1-p.lo)]
plot(my.cost ~ pred, pred.dt)
pred.dt[, lines(pred, naive.cost, col="red")]

Hi Prof @tdhock,
Please let me know what kind of censoring we have in the above formula? Looks like it is right censoring.
Do we have different stabilizing techniques for a different kind of censoring?
Please let me know what we are using finally for y in 3rd distribution extreme or Weibull?

Hi Prof. @tdhock and @hcho3 ,
I have added comment to https://github.com/avinashbarnwal/xgboost/commit/f52c96708ce54f12f594ec2a38537166b07d253f#r34598175.

in my opinion the y values (at least internally) should be on the real-valued (potentially negative) scale, for all distributions, so the 3rd distribution should be "extreme"
that being said if you run into issues with extreme I think it would be more useful to just have a working prototype for the other two distributions.

Hi Prof. @tdhock and @hcho3,
I have kept "extreme" as 3rd distribution. I have changed the conditions to detect the censoring type as follows:-

Uncensored - Before -> y_lower == y_higher and After ->  y_lower == y_higher
Left - Before -> std::isinf(y_lower) and After -> std::abs(y_lower) <=1e-12f
Right - Before -> std::isinf(y_higher) and After -> std::isinf(y_higher)
Interval - Before -> !std::isinf(y_lower) && !std::isinf(y_higher) and After ->std::abs(y_lower)>1e-12f && !std::isinf(y_higher)

Please let me if you think this is correct.
Rest of the code is same and I haven't made any changes to "Extreme" Distribution.

i'm not sure that is correct.
y_lower and y_higher should be log survival times (possible negative).
if y_lower == y_higher then un-censored (use pdf)
otherwise censored

if y_lower == -INFINITY then p_lower=0 else p_lower = cdf(z_lower)
if y_higher == INFINITY then p_higher=1 else p_higher = cdf(z_higher)
loss = -log(p_higher - p_lower)

in particular I would suggest using INFINITY which is defined in math.h
I don't think you should use the 1e-12 constant which is totally arbitrary.

Prof. @tdhock,
We have been using y as un-logged values. Therefore, a comparison with 0 is done. I would change the y_lower and y_higher in logged format internally.
I saw that in your datasets you have log-transformed responses and to test I took anti-log of responses.
Here, we expect users to enter survival times or logged transformed response. I am not sure about this?

in survival::survreg the user can specify either kind of outputs. For me the logged outputs make more sense because anyway that is what we use internally.

Thanks. I will change the y_lower and y_higher to log-transformed values and the conditions to detect the censoring types.

Hi Prof. @tdhock and @hcho3,
I have pushed the following changes:-

Code refactoring for left/right/interval for loss,gradient and hessian.
Removing x, mu, and sd and adding z.
Now the conditions to detect censoring is same as before.
Now we are passing log(y_lower) and log(y_higher) internally to do all the calculated.

I have tested here - https://github.com/avinashbarnwal/GSOC-2019/blob/master/AFT/test/data/neuroblastoma-data-master/src/notebook/002_xgboost_test.ipynb. We have similar results as before.

good. is there a PR where we can view a summary of all changes you want to merge? (files changed)

Hi Prof. @tdhock and @hcho3,
Please find the pull request - https://github.com/dmlc/xgboost/pull/4763.

Hi Prof. @tdhock and @hcho3 ,
I am trying to write the R vignette and i am finding this error. -
Error in setinfo.xgb.DMatrix(dtrain, "label_lower_bound", y_lower) :
setinfo: unknown info name label_lower_bound
Please find the R code - https://github.com/avinashbarnwal/GSOC-2019/blob/master/AFT/R/doc.R
Any help would be great.

sounds like an xgboost error, @hcho3 should know

is it normal in xgboost to specify the outputs/labels as attributes of the input/feature matrix? at least in R that is pretty confusing. the usual ways in R are

via formula, e.g. lm(y ~ x1 + x2, data=some.data.frame)
via separate input/output matrix objects e.g. glmnet(X, y) where X is the feature/input matrix and y is the output/label vector.

@tdhock The mistake was due to a badly chosen sigma parameter. When I set sigma to 10 or 20, the metrics are no longer infinity. In general, we should choose a reasonable value for sigma depending on the range of log survival times.

@tdhock I do not think XGBoost package supports the use of a formula. Currently, users are asked to create an object of type xgb.DMatrix with data and labels supplied.

Hi Prof. @tdhock,
@hcho3 helped in resolving the issue. We have a working vignette here - https://github.com/avinashbarnwal/GSOC-2019/blob/master/AFT/R/doc.R. I am structuring more.

red flag!
why doesn't sigma=1 work? I think it should be fine for the data sets I have provided.
sigma=10 seems too big to me -- there is almost no flat region for interval-censored outputs.
s <- 1;curve(-log(pnorm(10,x,s)-pnorm(5,x,s)),0, 20)

Hi Prof. @tdhock,
I think we need to see the standard deviation of log(survival times) and we should use something similar sigma as starting point. We tested it with very less sigma and it was giving very less predicted value with very high log(survival times). For example - log(survival times) ~ 10 and y hat was 0.03. I think this is not correct.

@tdhock @avinashbarnwal Maybe we need set a bigger value for base_score, the initial guess for yhat. By default, it is set to 0.5, which may be too low.

more generally you could set it to the mean of all the finite limits?

Hi Prof. @tdhock and @hcho3,
I have included base_score as average and sigma as the standard deviation of the log(survival_times).
We have very less number of Infinite observations now.
NoteBook - https://github.com/avinashbarnwal/GSOC-2019/blob/master/AFT/test/data/neuroblastoma-data-master/src/notebook/002_xgboost_test.ipynb


[Task]: Document Methods in the HealthcareApiClient

What needs to happen?
Several methods are not documented in the Healthcare API Client (https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/healthcare/HealthcareApiClient.java#L238).
We should add documentation for these methods.
Issue Priority
Priority: 3 (nice-to-have improvement)
Issue Components

[ ] Component: Python SDK
[X] Component: Java SDK
[ ] Component: Go SDK
[ ] Component: Typescript SDK
[X] Component: IO connector
[ ] Component: Beam examples
[ ] Component: Beam playground
[ ] Component: Beam katas
[ ] Component: Website
[ ] Component: Spark Runner
[ ] Component: Flink Runner
[ ] Component: Samza Runner
[ ] Component: Twister2 Runner
[ ] Component: Hazelcast Jet Runner
[ ] Component: Google Cloud Dataflow Runner

.take-issue


[FLINK-27707] Implement ManagedTableFactory#onCompactTable

This PR implements onCompactTable to perform the scan on the latest snapshot and (pick data files to compact for non-rescale bucket condition) and serialize the meta to JSON string.

Update
For non-rescale bucket compaction, we don't perform a scan at the planning phase. Instead, we put a flag along with part spec to indicate it is ordinary manual trigger compaction.


Some validations issues at Form

1.- Go to http://allready-test.azurewebsites.net/Account/Register
2.- Complete the form

3.- There are some missing validations, like: Name/last name should allow only alpha values
4.- Phone number text should allow only numbers

Raised https://github.com/HTBox/allReady/pull/2160 for this issue

Closing as a PR was merged for this issue.


Only one broken attachment displayed if multiple blob container attachments with the same name

Storage Explorer Version: 1.28.0
Build Number: 20230208.3
Branch: rel/1.28.0
Platform/OS: Windows 10/Linux Ubuntu 20.04/MacOS Ventura 13.2 (Apple M1 Pro)
Architecture: ia32/x64/arm64
How Found: Exploratory testing
Regression From: Previous release (1.27.2)
Steps to Reproduce

Azure AD attach two blob containers (make sure already given the correct RBAC role).
Rename the two attachments using a same name.
Switch to the 'ACCOUNT MANAGEMENT' panel -> Remove the azure account that associated with the Azure AD attachments.
Switch to the 'EXPLORER' panel.
Check whether displays two broken blob container attachments.

Expected Experience
Display two broken blob container attachments.
Actual Experience
Only display one broken blob container attachment.
Additional Context
This issue doesn't reproduce for queues/tables.

Verified this issue on the build 20230209.2. Fixed. So, we are going to close it.


Bump to<EMAIL_ADDRESS>

May fix #30

Wohoo :+1:

:)

Too fast! :+1:
@moul : do you have any image reference of scw command I can use to test it on our architecture as a (maybe) drop in replacement for docker 1.7?

Build detail: https://travis-ci.org/scaleway/qa/builds/89229379
I released the image with name Docker 1.9 BETA, I let the Docker 1.7 image alive.
To give a try, you can follow this link:

http://cloud.scaleway.com/#/servers/new?image=a2333c67-1605-4c45-96e0-5d9f707f61ec

Or use scw: scw run docker-1.9-beta

[FAILED] Failed to start LXC network bridge setup.
See 'systemctl status lxc-net.service' for details.
[...]
[FAILED] Failed to start FUSE filesystem for LXC.
See 'systemctl status lxcfs.service' for details.

Nice start :)

Ok, so I fixed this issue by using:
$ insmod /lib/modules/4.2.5-260/kernel/fs/fuse/fuse.ko
$ systemctl restart lxcfs.service lxc-net.service docker
$ docker version
Client:
 Version:      1.9.0
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   76d6bc9
 Built:        Tue Nov  3 22:16:18 UTC 2015
 OS/Arch:      linux/arm

Server:
 Version:      1.9.0
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   76d6bc9
 Built:        Tue Nov  3 22:16:18 UTC 2015
 OS/Arch:      linux/arm

I'm building a new kernel which will prevent this hack
I will hot-replace the kernel for this image

Ok fixed, now you can try out this new image, docker starts at boot automatically

:+1: thank you, I will test in few hours now.


What about UTF8 support?

Hi, great job!
I'm wondering if the keys would be in utf8 encoding?
Many thanks!

The keys are full binary, thats why you have to specify the key len. So, the keys are safe to use with UTF-8. You just have to count bytes not letters.


Unified filters/predicates interface for Parquet

Currently we support filters on row-group level based on an AND of predicates. I would like to implement this in the next 4-6 weeks also for pyarrow. I have a pure Python implementation working but for my use case I required more than simply  AND. Therefore I would propose to change the current interface to accept predicates in disjunctive normal form. With this representation, we are able to represent all possible predicates.
This change would mean that we would also support list of list of tuples in
https://github.com/dask/dask/blob/e823ecf51032ea7e4d57687a91d788cfb28e36ac/dask/dataframe/io/parquet.py#L915-L919
@martindurant @wesm @mrocklin Would that sound like an acceptable extension of the interface?

No objection from me, though I'm probably not the most up-to-date on the state of parquet reading/writing in dask.dataframe.  @martindurant would probably have more detailed thoughts.

I am not opposed.
Changes in the fastparquet side would have to go into fastparquet.api.filter_*, I think.

Does this need to be implemented in fastparquet in order for it to be
implemented in dask.dataframe?  Or can we provide an informative error if
the feature only exists in some engines, but not all.
On Fri, Aug 3, 2018 at 10:03 AM, Martin Durant<EMAIL_ADDRESS>wrote:

I am not opposed.
Changes in the fastparquet side would have to go into
fastparquet.api.filter_*, I think.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/3847#issuecomment-410258648, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AASszGAsBzod1SJsn_I-TBF49yWLKUZqks5uNFgfgaJpZM4Vt9R7
.

No, it could be implemented for arrow alone, with an error message, or (since the logic isn't too complicated), be implemented for both in dask. At the moment, there are two functions in fastparquet for the filter functionality, and it would be nice if any implementation became available to users of the direct fastparquet API.

@xhochy can you create a tracking JIRA for this so we can tackle in the 0.11.0 release cycle?

@wesm there is https://issues.apache.org/jira/plugins/servlet/mobile#issue/ARROW-1796 for this.


Uncaught ValueError in base_coder.py line 156

Aider version: 0.60.2.dev43+ged4ad45e
Python version: 3.11.9
Platform: Windows-10-10.0.22631-SP0
Python implementation: CPython
Virtual environment: No
OS: Windows 10 (64bit)
Git version: git version 2.47.0.windows.2
An uncaught exception occurred:
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "__main__.py", line 7, in <module>
    sys.exit(main())
             ^^^^^^
  File "main.py", line 710, in main
    coder.run(test_errors)
  File "base_coder.py", line 724, in run
    self.run_one(with_message, preproc)
  File "base_coder.py", line 773, in run_one
    list(self.send_message(message))
  File "base_coder.py", line 1197, in send_message
    self.reply_completed()
  File "architect_coder.py", line 33, in reply_completed
    editor_coder = Coder.create(**new_kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "base_coder.py", line 156, in create
    raise ValueError(f"Unknown edit format {edit_format}")
ValueError: Unknown edit format json

Thanks for trying aider and filing this issue.
This looks like a duplicate of #1835. Please see the comments there for more information, and feel free to continue the discussion within that issue.
I'm going to close this issue for now. But please let me know if you think this is actually a distinct issue and I will reopen this issue.


Local dev scripts: installation, docker refresh

Installation script ensures nothing but code winds up in the resulting Docker images.
Refresh script automatically keeps track of what's running, shutdown/build/run procedures, and test bindmounts. We've got people copy-pasting whole paragraphs of shell commands and losing track of which copypastes do what, and that's just not necessary.
Prerequisites:

[x] Unless it is a documentation hotfix it should be merged against the dev branch
[x] Branch is up-to-date with the branch to be merged with, i.e. dev
[x] Build is successful
[x] Code is cleaned up and formatted

(rebased onto dev to incorporate patches added there since October)


os/switching-channels: clarify usr and etc config files

This clarifies the different roles of /usr/share/coreos/update.conf
and /etc/coreos/update.conf, and the override semantics between them.
It also adds an explicit note about a valid transitory state.

/cc @enxebre @squat to clarify some installer doubts
/cc @crawford @zbwright for review

lgtm thanks @lucab

@bgilbert thanks for spotting typos and fixing my en_IT :)
Keeping the other reviews for the overall content, merging.