Ag-Net: building a customized deep neural network for recognizing crop categories based on spectral characteristics

ESIP Member Organization
CSISS/LAITS, George Mason University
Alaska Ocean Observing System (AOOS) and Axiom Data Science
Mentors
Ziheng Sun
Jesse Lopez
Project Ideas
Ag-Net: building a customized deep neural network for recognizing crop categories based on spectral characteristics
Information for students
See ESIP general guidelines
Abstract
How many kinds of crops can you recognize? It is hard to say many. In most time of growing season, they are all green plants. Dent corn and sweet corn, black bean and red bean, barley and wheat, grass and weeds, etc. Distinguishing them takes a ton of knowledge and experiences. Agriculture scientists have struggled for years to figure out an automated way to recognize them. Deep learning is a powerful tool for non-linear classification problems. The critical part for deep learning is training dataset, which can be extracted from the reports and map products of U.S. department of agriculture. However, the existing deep neural networks are not performing as well as expected on crop classification because of their learned representation features in the back propagation are not common enough to tell the small differences among crops with similar external look. A customized network with special filters may help tell those minor differences in high spectral characteristics for more accurate recognition results.
Technical Details
Python; Keras; Geoweaver; numpy; scikit-learn; matplotlib; GDAL.
Helpful Experience
Machine learning knowledge; satellite image manipulation; python programming.
First steps
Start to get familiar with DeeplabV3, U-net or any other state-of-art deep neural network and test them on a sample training dataset.

@Juhi-Purswani Sure. Could you please send me an email<EMAIL_ADDRESS>and we can start discussion. Thank you!

@ZihengSun Sir, I am also very interested in working on this project. I have sent you an email. Can you please check and guide me further?  Thank you!

@hdsingh Welcome! Thanks for your response! I have sent some tutorials for you to get started.

Hi @ZihengSun ,
I am very interested in working on the project and making a contribution to it. I have sent and email to you stating my interest in the project. Please check the mail and guide me for the project.
Thank You!

Hi! I'm really really interested in this project. This is very in line to one of my previous projects.
Correct me if I'm wrong, the underlying task is to build a robust Semantic Image Segmentation model specific for agriculture land Classification.
I'll get started with the literature below and write summary blogs to keep a track of major contributions and the key points for each paper.
Rethinking Atrous Convolution for Semantic Image Segmentation
U-Net: Convolutional Networks for Biomedical Image Segmentation
I've also sent you an email regarding a few doubts and specific questions, probably we can have detailed discussion on slack or through email.

Thank you very much for your responses!
We are still waiting for Google official announcement on Feb 26. Before that, make sure you read the ESIP participation guidelines. Any additional questions please let @abburgess or me know.
We have a general tutorial for excising deep learning in agriculture on FigShare. Hope it can help you get started.

Hello Sir @ZihengSun, I am really interested in contributing to this project. I have previously worked on projects demanding health status of plants which required a fair bit of knowledge in Image Processing, Classification and working with various standard python libraries.
I have sent you a mail in order to get some context, and understand how to get started. In addition, I shall also be doing work on my side of understanding the technologies, including reading latest papers on Deep Learning.
Thank You.

I am really interested in this project , and I have already been doing similar kind of work earlier on also........could you please guide me on how I can contribute.

Hi all -
Quick reminder that we should hold off on discussing projects at-depth
until February 26th; that is the date mentor organizations are officially
announced. It's okay to say 'hi' though!
AB
Annie Burgess, PhD
Lab Director | Earth Science Information Partners (ESIP)
esipfed.org/lab http://esipfed.org/lab |<PHONE_NUMBER>
Sign up for the monthly ESIP Lab update here http://eepurl.com/dtKL8z.
On Thu, Feb 14, 2019 at 8:12 AM navyasingh002<EMAIL_ADDRESS>wrote:

I am really interested in this project , and I have already been doing
similar kind of work earlier on also........could you please guide me on
how I can contribute.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ESIPFed/gsoc/issues/13#issuecomment-463621707, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AG_XwMlDaPRWW6mgJ94WnOGnVX46gQ4yks5vNWDFgaJpZM4afBCr
.

Congrats ESIP!
As it is 26th February, I guess it is okay to discuss ideas now!
Hello Ziheng, Hope you are doing well. I was following this organisation and was quite thinking about the ideas to implement this NN. I have mailed them to you. Please let me know how you feel about them.
Thanks and Regards,
Aniruddh

Hi @all
Congratulations ESIP is officially GSoC mentor org!
Let's start to think about how to write your applications!
Slack would be a great place to discuss details. Please join in ESIP slack workspace and send me a message about your plans.

Hello, I want to contribute myself to this project and want to solve the issues. I have mailed to @ZihengSun and want further comments. Thank you!

Hello @ZihengSun  I am interesting in contributing to the project.I would like to ask you about the dataset that we have to work upon

I created a sample dataset for you to start.
https://github.com/ZihengSun/Ag-Net-Dataset

Thanks for this @ZihengSun, it will be really helpful :)

Hello @ZihengSun , I'm Aditya, final year engineering student, I would like to contribute to this project. Please guide me further.

Hi @ZihengSun , I made my own implementation of U-net and another network called segNet on land change prediction problem (pixel wise semantic segmentation), my paper on that is in the final revisions. I think I'd be able to help on this project. Please let me know if you have patches of labeled data. If I can help at this project I would be glad to.

Hi @ZihengSun, I am interested in contributing towards the project. I have previous worked on a similar problem on Kaggle. I also a fair share of experience with segmentation modelling. Please let me know what do do further!

Hi @PariyaPm ,
Thank you for the response. That was great progress! U-Net and SegNet are both promising networks for land cover classification. I have created a sample training dataset. I believe it is trainable for both nets (I tried it on SegNet a little bit). Let me know if you meet any problems when using them.
Best, Ziheng

Hi @thePairedElectron Please take a look at the sample dataset and general tutorial above to help you get started.

UNet with attention gives better results than SegNet.

@sinAshish It might be true in some cases. However, we need experiment results before making final conclusion statement in general scope.

ok, I'll post the experimental results here after my model gets trained!

I would like to contribute to this project. I have some experience with Deep Neural Networks, UNet, Autoencoders, Keras, PyTorch and Tensorflow and with training models. Can you help me get started?

Respected Sir,
I have sent you an email stating the previous experience I have in the field and have asked for help to get me started further. I have also stated that I have read about U-Net architecture and Image segmentation. I have also seen the dataset which you have posted here.
My email is<EMAIL_ADDRESS>Can you help me figure out what to do next?
Thank you.

Hello sir@ZihengSun, I am currently pursuing my pg at IIIT-Hyderabad and I am really interested in working on this project. Can you provide more information about the project and resources so that I can get more familiar?
Mail<EMAIL_ADDRESS>

Hi, thank you for the responses! Please start with using the network you are familiar with like unet autoencoder, segnet, yolo, rcnn, etc to test run on the sample dataset above. From the results you will be able to see the drawbacks and think about how to customize the network to improve the accuracy. Write your thought into an application. Any difficulties please let me know.

Sir, Can I submit you my work after some days (like 3-4) ?

@ZihengSun  You mentioned having experimented with Segnets.Did You also Try to Experiment with number of Channels you were passing.Instead of passing all the 7 channels maybe Passing Each Channel Individually or Grouping 2-3 Channels together and then on final Prediction Taking a Weighted Average For The Final Segmentation Mask.I think that should Make The model converge More Fast

@ZihengSun  is it ok to use the dataset provided by you on other platforms like kaggle or colab for GPU services?

A very very basic EDA  of the sample dataset. Fellow participants would find it useful to see the various channels with the mask.

@ZihengSun You mentioned having experimented with Segnets.Did You also Try to Experiment with number of Channels you were passing.Instead of passing all the 7 channels maybe Passing Each Channel Individually or Grouping 2-3 Channels together and then on final Prediction Taking a Weighted Average For The Final Segmentation Mask.I think that should Make The model converge More Fast

No, I didn't try that. It could be interesting. The visible bands are very limited in distinguishing crops in growing season. Fully taking advantage of all the 7 bands could make it easier for neural network to recognize them.

A very very basic EDA of the sample dataset. Fellow participants would find it useful to see the various channels with the mask.

Thank you for sharing this!

Sir, do you mean we may consider all channels at once, average them and pass the newly created dataset of images as training data? This is because #input images = 7*#output images. This may benefit the network I think. Will experiment on this and tell you the result in 3 days. Thank you.

@ZihengSun I think Averaging All  channels to Combine Them Would Result in Loss Of lots of Information.I think Creating a model that can directly start with 7 Channels is an optimal solution.I am working on the same and would let you know the results soon

@ZihengSun  Sir, so I implemented the U-Net on the dataset you gave. The implementation was done by me on Google Colab. Hopefully the GPU was able to handle dataset read all at once. I also started with all the 7 channels directly (taking full effect of pixel values).
The code: https://colab.research.google.com/drive/1WflXioKXcg1SCi_rd2KR60Yq1i6ZO-xc#scrollTo=yNre1WZTmm4L
If you or anyone else is not able to access, please do let me know.
I have implemented the U-net from this paper: http://arxiv.org/abs/1505.04597
Problems I am facing:

Learning is way too slow (even if I don't pay any heed to accuracy since I don't think it makes any sense in regression problems (we are generating an image)).

Loss is way too much.

If anyone can see the code and give me suggestions on how to improve my model to minimize loss, it will be very helpful. I am using batch size = 256 and minimum learning rate = 0.00001. Also if anyone can implement the dataset (train) on any other network like DeepLabv3, and how it fares, it will be very beneficial to my learning.
Okay so I am posting losses (apologies if the post is way too big):
Train on 9478 samples, validate on 1673 samples
Epoch 1/100
9478/9478 [==============================] - 79s 8ms/step - loss: 0.6259 - acc: 0.0000e+00 - val_loss: 0.6914 - val_acc: 0.0000e+00
Epoch 00001: val_loss improved from inf to 0.69144, saving model to model.h5
Epoch 2/100
9478/9478 [==============================] - 64s 7ms/step - loss: 0.6042 - acc: 0.0000e+00 - val_loss: 0.6110 - val_acc: 0.0000e+00
Epoch 00002: val_loss improved from 0.69144 to 0.61097, saving model to model.h5
Epoch 3/100
9478/9478 [==============================] - 64s 7ms/step - loss: 0.6033 - acc: 0.0000e+00 - val_loss: 0.6176 - val_acc: 0.0000e+00
Epoch 00003: val_loss did not improve from 0.61097
Epoch 4/100
9478/9478 [==============================] - 64s 7ms/step - loss: 0.5981 - acc: 0.0000e+00 - val_loss: 0.6384 - val_acc: 0.0000e+00
Epoch 00004: val_loss did not improve from 0.61097
Epoch 5/100
9478/9478 [==============================] - 64s 7ms/step - loss: 0.5947 - acc: 0.0000e+00 - val_loss: 0.6272 - val_acc: 0.0000e+00
Epoch 00005: ReduceLROnPlateau reducing learning rate to 0.00010000000474974513.
Epoch 00005: val_loss did not improve from 0.61097
Epoch 6/100
9478/9478 [==============================] - 64s 7ms/step - loss: 0.5901 - acc: 0.0000e+00 - val_loss: 0.6142 - val_acc: 0.0000e+00
Epoch 00006: val_loss did not improve from 0.61097
Epoch 7/100
9478/9478 [==============================] - 64s 7ms/step - loss: 0.5887 - acc: 0.0000e+00 - val_loss: 0.6051 - val_acc: 0.0000e+00
Epoch 00007: val_loss improved from 0.61097 to 0.60511, saving model to model.h5
Epoch 8/100
9478/9478 [==============================] - 64s 7ms/step - loss: 0.5881 - acc: 0.0000e+00 - val_loss: 0.5992 - val_acc: 0.0000e+00
Epoch 00008: val_loss improved from 0.60511 to 0.59922, saving model to model.h5
Epoch 9/100
9478/9478 [==============================] - 64s 7ms/step - loss: 0.5879 - acc: 0.0000e+00 - val_loss: 0.6010 - val_acc: 0.0000e+00
Epoch 00009: val_loss did not improve from 0.59922
Epoch 10/100
9478/9478 [==============================] - 64s 7ms/step - loss: 0.5879 - acc: 0.0000e+00 - val_loss: 0.5975 - val_acc: 0.0000e+00
Epoch 00010: val_loss improved from 0.59922 to 0.59746, saving model to model.h5
Epoch 11/100
9478/9478 [==============================] - 64s 7ms/step - loss: 0.5872 - acc: 0.0000e+00 - val_loss: 0.5953 - val_acc: 0.0000e+00
Epoch 00011: val_loss improved from 0.59746 to 0.59533, saving model to model.h5
Epoch 12/100
9478/9478 [==============================] - 64s 7ms/step - loss: 0.5872 - acc: 0.0000e+00 - val_loss: 0.5948 - val_acc: 0.0000e+00
Epoch 00012: val_loss improved from 0.59533 to 0.59482, saving model to model.h5
Epoch 13/100
9478/9478 [==============================] - 64s 7ms/step - loss: 0.5867 - acc: 0.0000e+00 - val_loss: 0.5957 - val_acc: 0.0000e+00
Epoch 00013: val_loss did not improve from 0.59482
Epoch 14/100
9478/9478 [==============================] - 64s 7ms/step - loss: 0.5860 - acc: 0.0000e+00 - val_loss: 0.5964 - val_acc: 0.0000e+00
Epoch 00014: val_loss did not improve from 0.59482
Epoch 15/100
9478/9478 [==============================] - 64s 7ms/step - loss: 0.5864 - acc: 0.0000e+00 - val_loss: 0.5919 - val_acc: 0.0000e+00
Epoch 00015: val_loss improved from 0.59482 to 0.59195, saving model to model.h5
Epoch 16/100
9478/9478 [==============================] - 64s 7ms/step - loss: 0.5857 - acc: 0.0000e+00 - val_loss: 0.5920 - val_acc: 0.0000e+00
Epoch 00016: val_loss did not improve from 0.59195
Epoch 17/100
9478/9478 [==============================] - 64s 7ms/step - loss: 0.5862 - acc: 0.0000e+00 - val_loss: 0.5984 - val_acc: 0.0000e+00
Epoch 00017: val_loss did not improve from 0.59195
Epoch 18/100
9478/9478 [==============================] - 64s 7ms/step - loss: 0.5851 - acc: 0.0000e+00 - val_loss: 0.5921 - val_acc: 0.0000e+00
Epoch 00018: ReduceLROnPlateau reducing learning rate to 1.0000000474974514e-05.
Epoch 00018: val_loss did not improve from 0.59195
Epoch 19/100
9478/9478 [==============================] - 64s 7ms/step - loss: 0.5850 - acc: 0.0000e+00 - val_loss: 0.5915 - val_acc: 0.0000e+00
Epoch 00019: val_loss improved from 0.59195 to 0.59152, saving model to model.h5
Epoch 20/100
9478/9478 [==============================] - 64s 7ms/step - loss: 0.5847 - acc: 0.0000e+00 - val_loss: 0.5904 - val_acc: 0.0000e+00
Epoch 00020: val_loss improved from 0.59152 to 0.59040, saving model to model.h5
Epoch 21/100
9478/9478 [==============================] - 64s 7ms/step - loss: 0.5846 - acc: 0.0000e+00 - val_loss: 0.5897 - val_acc: 0.0000e+00
Epoch 00021: val_loss improved from 0.59040 to 0.58970, saving model to model.h5
Epoch 22/100
9478/9478 [==============================] - 64s 7ms/step - loss: 0.5845 - acc: 0.0000e+00 - val_loss: 0.5895 - val_acc: 0.0000e+00
Epoch 00022: val_loss improved from 0.58970 to 0.58949, saving model to model.h5
Epoch 23/100
9478/9478 [==============================] - 64s 7ms/step - loss: 0.5846 - acc: 0.0000e+00 - val_loss: 0.5889 - val_acc: 0.0000e+00
Epoch 00023: val_loss improved from 0.58949 to 0.58886, saving model to model.h5
Epoch 24/100
9478/9478 [==============================] - 64s 7ms/step - loss: 0.5845 - acc: 0.0000e+00 - val_loss: 0.5879 - val_acc: 0.0000e+00
Epoch 00024: val_loss improved from 0.58886 to 0.58790, saving model to model.h5
Epoch 25/100
9478/9478 [==============================] - 64s 7ms/step - loss: 0.5846 - acc: 0.0000e+00 - val_loss: 0.5885 - val_acc: 0.0000e+00
Epoch 00025: val_loss did not improve from 0.58790
Epoch 26/100
9478/9478 [==============================] - 64s 7ms/step - loss: 0.5845 - acc: 0.0000e+00 - val_loss: 0.5892 - val_acc: 0.0000e+00
Epoch 00026: val_loss did not improve from 0.58790
Epoch 27/100
9478/9478 [==============================] - 64s 7ms/step - loss: 0.5844 - acc: 0.0000e+00 - val_loss: 0.5894 - val_acc: 0.0000e+00
Epoch 00027: ReduceLROnPlateau reducing learning rate to 1e-05.
Epoch 00027: val_loss did not improve from 0.58790
Epoch 28/100
9478/9478 [==============================] - 64s 7ms/step - loss: 0.5843 - acc: 0.0000e+00 - val_loss: 0.5893 - val_acc: 0.0000e+00
Epoch 00028: val_loss did not improve from 0.58790
Epoch 29/100
9478/9478 [==============================] - 64s 7ms/step - loss: 0.5843 - acc: 0.0000e+00 - val_loss: 0.5891 - val_acc: 0.0000e+00
Epoch 00029: val_loss did not improve from 0.58790
Epoch 30/100
9478/9478 [==============================] - 64s 7ms/step - loss: 0.5844 - acc: 0.0000e+00 - val_loss: 0.5896 - val_acc: 0.0000e+00
Epoch 00030: val_loss did not improve from 0.58790
Epoch 31/100
9478/9478 [==============================] - 64s 7ms/step - loss: 0.5842 - acc: 0.0000e+00 - val_loss: 0.5888 - val_acc: 0.0000e+00
Epoch 00031: val_loss did not improve from 0.58790
Epoch 32/100
9478/9478 [==============================] - 64s 7ms/step - loss: 0.5841 - acc: 0.0000e+00 - val_loss: 0.5896 - val_acc: 0.0000e+00
Epoch 00032: val_loss did not improve from 0.58790
Epoch 33/100
9478/9478 [==============================] - 64s 7ms/step - loss: 0.5840 - acc: 0.0000e+00 - val_loss: 0.5891 - val_acc: 0.0000e+00
Epoch 00033: val_loss did not improve from 0.58790
Epoch 34/100
9478/9478 [==============================] - 64s 7ms/step - loss: 0.5842 - acc: 0.0000e+00 - val_loss: 0.5893 - val_acc: 0.0000e+00
Epoch 00034: val_loss did not improve from 0.58790
Epoch 00034: early stopping

Hi, @sankalpmittal1911-BitSian you might want to look into the loss function you are using for the task. From what I am guessing, you are considering this as regression problem and using mean squared error (correct me if I am wrong), while in fact this is not a regression problem, rather classification problem. You have to do classification(segmentation) at pixel level (think of predicting probabilities for all classes at each pixel location and selecting one with maximum probability at all pixel location). Usually for segmentation, CrossEntropyLoss is used and you might want to try it out. Let me know if you have any doubts regarding implementation.
Best,
Shivam

Hi, @shivamsaboo17
In the end we are generating masked images which are also images. Input is (128,128,1) and output is also (128,128,1). Yes the problem in itself is classification, I get it, since we are reading the output cdl images as a table: https://github.com/ZihengSun/Ag-Net-Dataset/blob/master/cdlvalue.csv
But the output is still an image right? I have not used MSE but have instead used Binary Cross Entropy Loss only as my last activation function is sigmoid on each pixel of output image.
Thanks and Best!
Sankalp Mittal

Hi @sankalpmittal1911-BitSian thanks for replying. I understand your intuition of using sigmoid on each pixel to create a image and then use BCELoss. However, the output is more accurately a segmentation map rather than an image. We use BinaryCrossEntropyLoss when we have only 2 classes to predict (say background and foreground in image). In this case there are more than 2 classes to choose from (hint: load all target images and create a dictionary of unique values). Let's say we have to classify into n classes, so the output which you should predict would be of shape (128, 128, n) and then you will select most probable class (i.e argmax) after using softmax over these classes. So actually, the actual values are not important (as in case of image). You can assign any color to any class and replace the class which you predicted with those values so you can distinguish during visualization. See an example belo for multi-class classification. You would realize that our aim is to classify different objects and the intensity value which you predict are not important.

Let me know if you have any doubts.
Best,
Shivam

Thanks a lot for the help! I will update the sigmoid with the softmax layer and use the categorical cross entropy loss as loss function and train the U-Net again. From the dataset, there are 255 classes.
Will post those results soon.

Sir, do you mean we may consider all channels at once, average them and pass the newly created dataset of images as training data? This is because #input images = 7*#output images. This may benefit the network I think. Will experiment on this and tell you the result in 3 days. Thank you.

Averaging is not a recommended practice. As the band values are surface reflectance in different spectral regions, some bands like band 5 have very large numbers, and the others are small. If using average, the variation pattern in the small-number bands will be hard to learn.

@ZihengSun Sir,
So from what I gather, every pixel in the output layer will be like a array of 255 length (i.e. number of classes).  [0.5 0.4 ....]. This becomes in Keras (batch,128,128,255). Now we need to apply softmax to every pixel of image over the number of classes, that is along axis = 2. Then we need to take the maximum of each pixel along the number of classes so the final image is again (batch,128,128,1) which is then compared with output .tif file. Please correct me if I am wrong. Thank You. Also, every output label image is the segmentation of only one of the 255 crops right?

@ZihengSun  I implemented a toy UNet model just to test whether it works.Here are the Results.I didnt take any validation set(I understand its a bad practice but just wanted to get started),Also There are no Image Augmentations Used.Here is the log after training for 10 epochs
Epoch 0===> trn_loss=1.4780788052082061 trn_acc=0.5332283782958984
Epoch 1===> trn_loss=1.4878936338424682 trn_acc=0.5299695205688476
Epoch 2===> trn_loss=1.4736434078216554 trn_acc=0.5325695419311524
Epoch 3===> trn_loss=1.466643956899643 trn_acc=0.5359280395507813
Epoch 4===> trn_loss=1.4540286827087403 trn_acc=0.5391661834716797
Epoch 5===> trn_loss=1.4555853509902954 trn_acc=0.5378150558471679
Epoch 6===> trn_loss=1.4375684547424317 trn_acc=0.5455305099487304
Epoch 7===> trn_loss=1.4241751718521118 trn_acc=0.5499917984008789
Epoch 8===> trn_loss=1.427647979259491 trn_acc=0.5488113784790039
Epoch 9===> trn_loss=1.4385879373550414 trn_acc=0.5430210113525391
I used Categorical Cross Entropy as Loss Function and for calculating accuracy I took out the final predictions from model,128128 in this case,Compared them with Base Labels and did a summation of all the times it correctly predicted
acc=some_num
fin_acc=acc/(128128)batch_sizetrain_loader_length

@ZihengSun I used a modified the U-Net by adding Batch Normalizations and used sparse_crossentropy as the loss, after applying softmax as the last layer. Also I normalized the pixel values of the band images.
Considering the labels are between 1-247, I plotted the loss graph. It has been attached. As you can see, the loss values aren't very low after 50 epochs. Any suggestions?

@ZihengSun I trained a  UNet model with resnet as encoder for 10 epochs, and get a dice score of around 0.48. I used all the 7 channels at once. But I need some help in loading the mask to appropriate no. of classes

Hi @sinAshish,
By dice score do you mean IoU or something similar? So did you use that as a custom metric instead of just accuracy. I will update my model based on suggestions here. I have this one problem:
So I have this output image of the form (batch,128,128,255). How will I apply softmax to this keras API layer along the axis such that it is applied over number of classes. And how we will select the maximum pixel as probability since maximum can belong to different classes?
u9 = Conv2DTranspose(n_filters1, (3, 3), strides=(2, 2), padding='same') (c8)
u9 = concatenate([u9, c1], axis=3)
u9 = Dropout(dropout)(u9)
c9 = conv2d_block(u9, n_filters=n_filters1, kernel_size=3, batchnorm=batchnorm)
#c9 = conv2d_block(u9, n_filters=255, kernel_size=3, batchnorm=batchnorm)
outputs = Conv2D(255, (1, 1)) (c9)
#outputs = core.Reshape((128,128 ,  255))(outputs)
#conv6 = core.Permute((2,1))(outputs)

outputs = core.Activation('softmax')(outputs)
#outputs = softmax(outputs, axis = 3)(outputs)

If I need to change the labeled output images in form of (128,128,255) as well, how can we modify our masks then?
Thank you all

@ZihengSun Sir,
So from what I gather, every pixel in the output layer will be like a array of 255 length (i.e. number of classes). [0.5 0.4 ....]. This becomes in Keras (batch,128,128,255). Now we need to apply softmax to every pixel of image over the number of classes, that is along axis = 2. Then we need to take the maximum of each pixel along the number of classes so the final image is again (batch,128,128,1) which is then compared with output .tif file. Please correct me if I am wrong. Thank You. Also, every output label image is the segmentation of only one of the 255 crops right?

That is correct. The final predicted class is the one with the highest probability.

@ZihengSun I implemented a toy UNet model just to test whether it works.Here are the Results.I didnt take any validation set(I understand its a bad practice but just wanted to get started),Also There are no Image Augmentations Used.Here is the log after training for 10 epochs
Epoch 0===> trn_loss=1.4780788052082061 trn_acc=0.5332283782958984
Epoch 1===> trn_loss=1.4878936338424682 trn_acc=0.5299695205688476
Epoch 2===> trn_loss=1.4736434078216554 trn_acc=0.5325695419311524
Epoch 3===> trn_loss=1.466643956899643 trn_acc=0.5359280395507813
Epoch 4===> trn_loss=1.4540286827087403 trn_acc=0.5391661834716797
Epoch 5===> trn_loss=1.4555853509902954 trn_acc=0.5378150558471679
Epoch 6===> trn_loss=1.4375684547424317 trn_acc=0.5455305099487304
Epoch 7===> trn_loss=1.4241751718521118 trn_acc=0.5499917984008789
Epoch 8===> trn_loss=1.427647979259491 trn_acc=0.5488113784790039
Epoch 9===> trn_loss=1.4385879373550414 trn_acc=0.5430210113525391
I used Categorical Cross Entropy as Loss Function and for calculating accuracy I took out the final predictions from model,128 * 128 in this case,Compared them with Base Labels and did a summation of all the times it correctly predicted
acc=some_num
fin_acc=acc/((128 * 128) * batch_size * train_loader_length)
I will be uploading the file soon with validation and some data Augmentations tried out and training for a little longer time
Also is there any baseline accuracy that we can compare our model to

@at1998 I would say give it another 100 epochs and see if the training accuracy rise significantly? If not, it might have something in the output layer not matching the CDL training labels.

@ZihengSun I used a modified the U-Net by adding Batch Normalizations and used sparse_crossentropy as the loss, after applying softmax as the last layer. Also I normalized the pixel values of the band images.
Considering the labels are between 1-247, I plotted the loss graph. It has been attached. As you can see, the loss values aren't very low after 50 epochs. Any suggestions?

@rishi-s8 The curves look reasonable. As it is still on the big slope down trend, the loss should be expected to decrease more if the training continues.

@ZihengSun I trained a UNet model with resnet as encoder for 10 epochs, and get a dice score of around 0.48. I used all the 7 channels at once. But I need some help in loading the mask to appropriate no. of classes

Sorry , I am not very clear about the question. Do you want to shrink the number of classes by removing those empty classes?

@ZihengSun I trained a UNet model with resnet as encoder for 10 epochs, and get a dice score of around 0.48. I used all the 7 channels at once. But I need some help in loading the mask to appropriate no. of classes

Sorry , I am not very clear about the question. Do you want to shrink the number of classes by removing those empty classes?

I stacked all the 7 channels to create a input tensor of (128,128,7), but by mistake I loaded the masks as only ('RGB') for training so the output tensor was (128,128,3). I need help in creating a mask that has the shape of (128,128,255), like how t o do it?

u9 = Conv2DTranspose(n_filters*1, (3, 3), strides=(2, 2), padding='same') (c8)
    u9 = concatenate([u9, c1], axis=3)
    u9 = Dropout(dropout)(u9)
    c9 = conv2d_block(u9, n_filters=n_filters*1, kernel_size=3, batchnorm=batchnorm)
    #c9 = conv2d_block(u9, n_filters=255, kernel_size=3, batchnorm=batchnorm)

    outputs = Conv2D(255, (1, 1)) (c9)
    #outputs = core.Reshape((128,128 ,  255))(outputs)
    #conv6 = core.Permute((2,1))(outputs)

    outputs = core.Activation('softmax')(outputs)
    #outputs = softmax(outputs, axis = 3)(outputs)

This is my code. The final output is of form (128,128,255). How to apply softmax to this so that I can select image belonging to class having maximum probability?
@ZihengSun  Sir can you help me regarding this? Applying softmax along an axis in Keras API. Thank you. I am assuming that input image is grayscale.

@sankalpmittal1911-BitSian  if you are using Keras, I assume the output of softmax layer would be automatically calculated from (width, height, classes) into (width, height, 1) which will match the dimension of CDL image. Please refer to this code for building U-net in Keras.

@ZihengSun  Why are codes from 1-254 when there are just 133 categories in total in cdlvalues.csv

@at1998 the missing numbers have no corresponding crop classes so they are skipped. There are only ~130 kinds of crops in US ag statistics.

@ZihengSun  Sir, the person has reshaped the output from (batch,img_width,img_height,n_classes) to (batch,img_width*img_height,n_classes) and then applied softmax on it. Should I do the same? And then I will have to change the outputs in the same format as well.

@ZihengSun  I replaced the values in the mask from 1-254 o 1-132 and it loss seems to steadily go down.Dice Scores are Around 0.48 while accuracy is in range 0.5-0.6.NOte that I converted the image to 3 channel by just adding all the values together.Will now increase the initial layer channels to 11 and observe how our accuracy goes up

@ZihengSun  Followed your Advice for observing the effect of training it for more epochs and here's The result.

Epoch 1/100
50/50 [==============================] - 1705s 34s/step - loss: 5.1689 - acc: 0.1686 - val_loss: 4.9105 - val_acc: 0.2356

Epoch 00001: val_loss improved from inf to 4.91052, saving model to model.h5
Epoch 2/100
50/50 [==============================] - 1021s 20s/step - loss: 4.4849 - acc: 0.2629 - val_loss: 4.4510 - val_acc: 0.1839

Epoch 00002: val_loss improved from 4.91052 to 4.45103, saving model to model.h5
Epoch 3/100
50/50 [==============================] - 1117s 22s/step - loss: 3.7226 - acc: 0.2872 - val_loss: 4.1678 - val_acc: 0.2061

Epoch 00003: val_loss improved from 4.45103 to 4.16783, saving model to model.h5
Epoch 4/100
50/50 [==============================] - 1109s 22s/step - loss: 3.0371 - acc: 0.2992 - val_loss: 3.4495 - val_acc: 0.2427

Epoch 00004: val_loss improved from 4.16783 to 3.44954, saving model to model.h5
Epoch 5/100
50/50 [==============================] - 1116s 22s/step - loss: 2.6087 - acc: 0.3151 - val_loss: 2.9204 - val_acc: 0.2475

Epoch 00005: val_loss improved from 3.44954 to 2.92044, saving model to model.h5
Epoch 6/100
50/50 [==============================] - 1119s 22s/step - loss: 2.4059 - acc: 0.3271 - val_loss: 2.5952 - val_acc: 0.2727

Epoch 00006: val_loss improved from 2.92044 to 2.59517, saving model to model.h5
Epoch 7/100
50/50 [==============================] - 1108s 22s/step - loss: 2.3171 - acc: 0.3289 - val_loss: 2.3975 - val_acc: 0.3254

Epoch 00007: val_loss improved from 2.59517 to 2.39755, saving model to model.h5
Epoch 8/100
50/50 [==============================] - 1117s 22s/step - loss: 2.2777 - acc: 0.3291 - val_loss: 2.4233 - val_acc: 0.3114

Epoch 00008: val_loss did not improve from 2.39755
Epoch 9/100
50/50 [==============================] - 1120s 22s/step - loss: 2.2546 - acc: 0.3312 - val_loss: 2.3257 - val_acc: 0.3173

Epoch 00009: val_loss improved from 2.39755 to 2.32571, saving model to model.h5
Epoch 10/100
50/50 [==============================] - 1130s 23s/step - loss: 2.2417 - acc: 0.3330 - val_loss: 2.3650 - val_acc: 0.3170

Epoch 00010: val_loss did not improve from 2.32571
Epoch 11/100
10/50 [=====>........................] - ETA: 4:10 - loss: 2.2265 - acc: 0.3341

For 10 epochs. Batch Size is 32. Pixel-wise cross entropy loss function.
Input: (Batch,128,128,7)
Output: (Batch,128,128,num_classes)
Should I continue training for 100 epochs or should I change something because learning is too slow?
Thanks.

Epoch 1/100
50/50 [==============================] - 1705s 34s/step - loss: 5.1689 - acc: 0.1686 - val_loss: 4.9105 - val_acc: 0.2356

Epoch 00001: val_loss improved from inf to 4.91052, saving model to model.h5
Epoch 2/100
50/50 [==============================] - 1021s 20s/step - loss: 4.4849 - acc: 0.2629 - val_loss: 4.4510 - val_acc: 0.1839

Epoch 00002: val_loss improved from 4.91052 to 4.45103, saving model to model.h5
Epoch 3/100
50/50 [==============================] - 1117s 22s/step - loss: 3.7226 - acc: 0.2872 - val_loss: 4.1678 - val_acc: 0.2061

Epoch 00003: val_loss improved from 4.45103 to 4.16783, saving model to model.h5
Epoch 4/100
50/50 [==============================] - 1109s 22s/step - loss: 3.0371 - acc: 0.2992 - val_loss: 3.4495 - val_acc: 0.2427

Epoch 00004: val_loss improved from 4.16783 to 3.44954, saving model to model.h5
Epoch 5/100
50/50 [==============================] - 1116s 22s/step - loss: 2.6087 - acc: 0.3151 - val_loss: 2.9204 - val_acc: 0.2475

Epoch 00005: val_loss improved from 3.44954 to 2.92044, saving model to model.h5
Epoch 6/100
50/50 [==============================] - 1119s 22s/step - loss: 2.4059 - acc: 0.3271 - val_loss: 2.5952 - val_acc: 0.2727

Epoch 00006: val_loss improved from 2.92044 to 2.59517, saving model to model.h5
Epoch 7/100
50/50 [==============================] - 1108s 22s/step - loss: 2.3171 - acc: 0.3289 - val_loss: 2.3975 - val_acc: 0.3254

Epoch 00007: val_loss improved from 2.59517 to 2.39755, saving model to model.h5
Epoch 8/100
50/50 [==============================] - 1117s 22s/step - loss: 2.2777 - acc: 0.3291 - val_loss: 2.4233 - val_acc: 0.3114

Epoch 00008: val_loss did not improve from 2.39755
Epoch 9/100
50/50 [==============================] - 1120s 22s/step - loss: 2.2546 - acc: 0.3312 - val_loss: 2.3257 - val_acc: 0.3173

Epoch 00009: val_loss improved from 2.39755 to 2.32571, saving model to model.h5
Epoch 10/100
50/50 [==============================] - 1130s 23s/step - loss: 2.2417 - acc: 0.3330 - val_loss: 2.3650 - val_acc: 0.3170

Epoch 00010: val_loss did not improve from 2.32571
Epoch 11/100
10/50 [=====>........................] - ETA: 4:10 - loss: 2.2265 - acc: 0.3341

For 10 epochs. Batch Size is 32. Pixel-wise cross entropy loss function.
Input: (Batch,128,128,7)
Output: (Batch,128,128,num_classes)
Should I continue training for 100 epochs or should I change something because learning is too slow?
Thanks.

@sankalpmittal1911-BitSian It is not slow to me.

@at1998 It guess you do need write all the details according to the proposal writing guidelines.

@all A color map of the crop values is uploaded. Use it to render your results into the standard crop map style.

@ZihengSun  In the color map all values (till 255) are available whereas the model that I have been training I have converted values from 1 to 133 and not till 255 since they are not available in the Masks.
Should The final Layer of the model be 133 or 255 ?

Should be 255. 133 should be reverse mapped to 255 during map production phase.

@ZihengSun  Because as per csv table this 133rd class is actually the pixel value 254?

@ZihengSun Just To clarify ,I should have 133 output in the last layer of UNet as in shape will be (133,128,128) and then reverse map the 133 values to original Values like 255=133 in this model?Am i Assuming correct

It actually has 256 values (8 bit). The best practice is directly using (360, 360, 256) as output shape, using (360, 360, 7) as input shape.
133 is only needed when your network requires consecutive output values. I suggest creating a map variable to remember the relationship between real crop category values and your projected values specially designed for your network, so you won't get confused about what the predicted values mean.

@ZihengSun  I am saving the mapping to a dictionary and will use that to reverse it.Thanks for clarification.

@ZihengSun  Images are (128,128) right?

@ZihengSun I have made my network as such. Input tensor of (128,128,7) and output tensor of (128,128,255)

@sankalpmittal1911-BitSian  I tink he is talking about the original Dataset that we will be working upon in the program.Right now we are only given a sample dataset to try out.

The size is 128, not 360. I downsized the tiles to make the required memory smaller for each batch.

@ZihengSun Could we have access to the original Dataset of size 360 . In almost all cases,Increasing The size will lead to better results when it comes to Segmentation.

They should have the same results. The impacted region is the region area which should be fine. Some time ago I used FCN (I forget which code I used) on the 128 dataset and have this result:

The accuracy is >80%. You approach at least achieve this accuracy. Carefully inspect your code and network to figure out why your model is not working.

@ZihengSun  I am having trouble in setting up rotation for both image masks and input.Right now I am using the 4 dihedral transform(90 degree rotation with flips) as part of my Data Augmentation.Also Did you Normalise the values to ImageNet Stats(using a pretrained model as feature extractor).Right now I am Normalising the data to have 0 mean and 1 std dev but at the same time I am using a Resnet as a feature in Middle Layers.I will try different Normalisaion to see the effect.

@ZihengSun  One more thing I needed to confirm,Do I need to be restricted to Keras

It is not restricted to Keras. Pytorch should work.

@ZihengSun I will also try pytorch if keras does not do the job. Will get back soon.

@sankalpmittal1911-BitSian  Let me know if you need any help and usually choice of a framework doesnt have that much affect on a model's accuracy since base operations are very much similar among  frameworks

@ZihengSun  @at1998
I am having hard time improving the accuracy after certain epoch. Here is my training-validation log:
Epoch 1/100
675/675 [==============================] - 5518s 8s/step - loss: 2.1004 - acc: 0.3985 - val_loss: 6.5754 - val_acc: 0.2246

Epoch 00001: val_loss improved from inf to 6.57537, saving model to model.h5
Epoch 2/100
675/675 [==============================] - 1119s 2s/step - loss: 1.9145 - acc: 0.4472 - val_loss: 5.7538 - val_acc: 0.3389

Epoch 00002: val_loss improved from 6.57537 to 5.75385, saving model to model.h5
Epoch 3/100
675/675 [==============================] - 1119s 2s/step - loss: 1.8592 - acc: 0.4621 - val_loss: 2.8144 - val_acc: 0.2887

Epoch 00003: val_loss improved from 5.75385 to 2.81440, saving model to model.h5
Epoch 4/100
675/675 [==============================] - 1132s 2s/step - loss: 1.8234 - acc: 0.4741 - val_loss: 3.4980 - val_acc: 0.2567

Epoch 00004: val_loss did not improve from 2.81440
Epoch 5/100
675/675 [==============================] - 1121s 2s/step - loss: 1.7949 - acc: 0.4820 - val_loss: 4.3436 - val_acc: 0.2417

Epoch 00005: ReduceLROnPlateau reducing learning rate to 0.0009999999776482583.

Epoch 00005: val_loss did not improve from 2.81440
Epoch 6/100
675/675 [==============================] - 1128s 2s/step - loss: 1.7152 - acc: 0.5067 - val_loss: 1.8597 - val_acc: 0.4699

Epoch 00006: val_loss improved from 2.81440 to 1.85970, saving model to model.h5
Epoch 7/100
675/675 [==============================] - 1127s 2s/step - loss: 1.6981 - acc: 0.5113 - val_loss: 1.9781 - val_acc: 0.4349

Epoch 00007: val_loss did not improve from 1.85970
Epoch 8/100
675/675 [==============================] - 1144s 2s/step - loss: 1.6885 - acc: 0.5142 - val_loss: 1.9789 - val_acc: 0.4414

Epoch 00008: ReduceLROnPlateau reducing learning rate to 9.999999310821295e-05.

Epoch 00008: val_loss did not improve from 1.85970
Epoch 9/100
675/675 [==============================] - 1143s 2s/step - loss: 1.6717 - acc: 0.5197 - val_loss: 1.7216 - val_acc: 0.5034

Epoch 00009: val_loss improved from 1.85970 to 1.72164, saving model to model.h5
Epoch 10/100
675/675 [==============================] - 1143s 2s/step - loss: 1.6682 - acc: 0.5208 - val_loss: 1.7112 - val_acc: 0.5082

Epoch 00010: val_loss improved from 1.72164 to 1.71118, saving model to model.h5
Epoch 11/100
675/675 [==============================] - 1134s 2s/step - loss: 1.6665 - acc: 0.5212 - val_loss: 1.7099 - val_acc: 0.5086

Epoch 00011: val_loss improved from 1.71118 to 1.70989, saving model to model.h5
Epoch 12/100
675/675 [==============================] - 1143s 2s/step - loss: 1.6650 - acc: 0.5218 - val_loss: 1.7065 - val_acc: 0.5100

Epoch 00012: val_loss improved from 1.70989 to 1.70651, saving model to model.h5
Epoch 13/100
675/675 [==============================] - 1136s 2s/step - loss: 1.6636 - acc: 0.5221 - val_loss: 1.7085 - val_acc: 0.5087

Epoch 00013: val_loss did not improve from 1.70651
Epoch 14/100
675/675 [==============================] - 1141s 2s/step - loss: 1.6619 - acc: 0.5227 - val_loss: 1.7054 - val_acc: 0.5100

Epoch 00014: val_loss improved from 1.70651 to 1.70538, saving model to model.h5
Epoch 15/100
675/675 [==============================] - 1120s 2s/step - loss: 1.6607 - acc: 0.5232 - val_loss: 1.7036 - val_acc: 0.5106

Epoch 00015: val_loss improved from 1.70538 to 1.70361, saving model to model.h5
Epoch 16/100
675/675 [==============================] - 1114s 2s/step - loss: 1.6593 - acc: 0.5235 - val_loss: 1.7008 - val_acc: 0.5118

Epoch 00016: val_loss improved from 1.70361 to 1.70075, saving model to model.h5
Epoch 17/100
675/675 [==============================] - 1125s 2s/step - loss: 1.6581 - acc: 0.5239 - val_loss: 1.7030 - val_acc: 0.5105

Epoch 00017: val_loss did not improve from 1.70075
Epoch 18/100
675/675 [==============================] - 1123s 2s/step - loss: 1.6569 - acc: 0.5243 - val_loss: 1.6997 - val_acc: 0.5120

Epoch 00018: val_loss improved from 1.70075 to 1.69970, saving model to model.h5
Epoch 19/100
675/675 [==============================] - 1119s 2s/step - loss: 1.6557 - acc: 0.5246 - val_loss: 1.6994 - val_acc: 0.5118

Epoch 00019: val_loss improved from 1.69970 to 1.69941, saving model to model.h5
Epoch 20/100
675/675 [==============================] - 1127s 2s/step - loss: 1.6544 - acc: 0.5251 - val_loss: 1.6981 - val_acc: 0.5122

Epoch 00020: val_loss improved from 1.69941 to 1.69810, saving model to model.h5
Epoch 21/100
675/675 [==============================] - 1125s 2s/step - loss: 1.6530 - acc: 0.5256 - val_loss: 1.6984 - val_acc: 0.5119

Epoch 00021: val_loss did not improve from 1.69810
Epoch 22/100
675/675 [==============================] - 1125s 2s/step - loss: 1.6518 - acc: 0.5259 - val_loss: 1.6961 - val_acc: 0.5130

Epoch 00022: val_loss improved from 1.69810 to 1.69612, saving model to model.h5
Epoch 23/100
675/675 [==============================] - 1128s 2s/step - loss: 1.6505 - acc: 0.5264 - val_loss: 1.6945 - val_acc: 0.5136

Epoch 00023: val_loss improved from 1.69612 to 1.69453, saving model to model.h5
Epoch 24/100
675/675 [==============================] - 1127s 2s/step - loss: 1.6496 - acc: 0.5266 - val_loss: 1.6949 - val_acc: 0.5133

Epoch 00024: val_loss did not improve from 1.69453
Epoch 25/100
675/675 [==============================] - 1130s 2s/step - loss: 1.6483 - acc: 0.5271 - val_loss: 1.6930 - val_acc: 0.5140

Epoch 00025: val_loss improved from 1.69453 to 1.69305, saving model to model.h5
Epoch 26/100
675/675 [==============================] - 1133s 2s/step - loss: 1.6471 - acc: 0.5275 - val_loss: 1.6942 - val_acc: 0.5132

Epoch 00026: val_loss did not improve from 1.69305
Epoch 27/100
675/675 [==============================] - 1126s 2s/step - loss: 1.6459 - acc: 0.5279 - val_loss: 1.6903 - val_acc: 0.5151

Epoch 00027: val_loss improved from 1.69305 to 1.69033, saving model to model.h5
Epoch 28/100
675/675 [==============================] - 1048s 2s/step - loss: 1.6447 - acc: 0.5283 - val_loss: 1.6903 - val_acc: 0.5150

Epoch 00028: val_loss improved from 1.69033 to 1.69027, saving model to model.h5
Epoch 29/100
675/675 [==============================] - 1097s 2s/step - loss: 1.6436 - acc: 0.5286 - val_loss: 1.6897 - val_acc: 0.5151

Epoch 00029: val_loss improved from 1.69027 to 1.68966, saving model to model.h5
Epoch 30/100
675/675 [==============================] - 1057s 2s/step - loss: 1.6424 - acc: 0.5290 - val_loss: 1.6876 - val_acc: 0.5160

Epoch 00030: val_loss improved from 1.68966 to 1.68756, saving model to model.h5
Epoch 31/100
675/675 [==============================] - 1049s 2s/step - loss: 1.6412 - acc: 0.5294 - val_loss: 1.6841 - val_acc: 0.5177

Epoch 00031: val_loss improved from 1.68756 to 1.68409, saving model to model.h5
Epoch 32/100
675/675 [==============================] - 1015s 2s/step - loss: 1.6400 - acc: 0.5298 - val_loss: 1.6859 - val_acc: 0.5167

Epoch 00032: val_loss did not improve from 1.68409
Epoch 33/100
254/675 [==========>...................] - ETA: 8:44 - loss: 1.6348 - acc: 0.5302

First I tried with this model:
def get_unet(input_img, n_filters=16, dropout=0.15, batchnorm=True):
    # contracting path
    c1 = conv2d_block(input_img, n_filters=n_filters*1, kernel_size=3, batchnorm=batchnorm)
    p1 = MaxPooling2D((2, 2)) (c1)
    p1 = Dropout(rate=dropout*0.5)(p1)

    c2 = conv2d_block(p1, n_filters=n_filters*2, kernel_size=3, batchnorm=batchnorm)
    p2 = MaxPooling2D((2, 2)) (c2)
    p2 = Dropout(rate=dropout)(p2)

    c3 = conv2d_block(p2, n_filters=n_filters*4, kernel_size=3, batchnorm=batchnorm)
    p3 = MaxPooling2D((2, 2)) (c3)
    p3 = Dropout(rate=dropout)(p3)

    c4 = conv2d_block(p3, n_filters=n_filters*8, kernel_size=3, batchnorm=batchnorm)
    p4 = MaxPooling2D(pool_size=(2, 2)) (c4)
    p4 = Dropout(rate=dropout)(p4)

    c5 = conv2d_block(p4, n_filters=n_filters*16, kernel_size=3, batchnorm=batchnorm)

    # expansive path
    u6 = Conv2DTranspose(n_filters*8, (3, 3), strides=(2, 2), padding='same') (c5)
    u6 = concatenate([u6, c4])
    u6 = Dropout(rate=dropout)(u6)
    c6 = conv2d_block(u6, n_filters=n_filters*8, kernel_size=3, batchnorm=batchnorm)

    u7 = Conv2DTranspose(n_filters*4, (3, 3), strides=(2, 2), padding='same') (c6)
    u7 = concatenate([u7, c3])
    u7 = Dropout(rate=dropout)(u7)
    c7 = conv2d_block(u7, n_filters=n_filters*4, kernel_size=3, batchnorm=batchnorm)

    u8 = Conv2DTranspose(n_filters*2, (3, 3), strides=(2, 2), padding='same') (c7)
    u8 = concatenate([u8, c2])
    u8 = Dropout(rate=dropout)(u8)
    c8 = conv2d_block(u8, n_filters=n_filters*2, kernel_size=3, batchnorm=batchnorm)

    u9 = Conv2DTranspose(n_filters*1, (3, 3), strides=(2, 2), padding='same') (c8)
    u9 = concatenate([u9, c1], axis=3)
    u9 = Dropout(rate=dropout)(u9)
    c9 = conv2d_block(u9, n_filters=n_filters*1, kernel_size=3, batchnorm=batchnorm)

    c10 = conv2d_block(c9, n_filters=255, kernel_size=1, batchnorm=batchnorm)

    outputs = Conv2D(255, (1, 1), activation='softmax') (c10)

    model = Model(inputs=[input_img], outputs=[outputs])
    return model

Then I changed to:
def get_unet(input_img, n_filters=256, dropout=0.15, batchnorm=True):
    # contracting path
    c1 = conv2d_block(input_img, n_filters=n_filters*1, kernel_size=3, batchnorm=batchnorm)
    p1 = MaxPooling2D((2, 2)) (c1)
    p1 = Dropout(rate=dropout*0.5)(p1)

    c2 = conv2d_block(p1, n_filters=n_filters*2, kernel_size=3, batchnorm=batchnorm)

    # expansive path
    u3 = Conv2DTranspose(n_filters*1, (3, 3), strides=(2, 2), padding='same') (c2)
    u3 = concatenate([u3, c1])
    u3 = Dropout(rate=dropout)(u3)
    c3 = conv2d_block(u3, n_filters=n_filters*1, kernel_size=3, batchnorm=batchnorm)

    c4 = conv2d_block(c3, n_filters=255, kernel_size=1, batchnorm=batchnorm)

    outputs = Conv2D(255, (1, 1), activation='softmax') (c4)

    model = Model(inputs=[input_img], outputs=[outputs])
    return model

Layer (type)                    Output Shape         Param #     Connected to
==================================================================================================
img (InputLayer)                (None, 128, 128, 7)  0
__________________________________________________________________________________________________
conv2d_64 (Conv2D)              (None, 128, 128, 256 16384       img[0][0]
__________________________________________________________________________________________________
batch_normalization_61 (BatchNo (None, 128, 128, 256 1024        conv2d_64[0][0]
__________________________________________________________________________________________________
leaky_re_lu_61 (LeakyReLU)      (None, 128, 128, 256 0           batch_normalization_61[0][0]
__________________________________________________________________________________________________
conv2d_65 (Conv2D)              (None, 128, 128, 256 590080      leaky_re_lu_61[0][0]
__________________________________________________________________________________________________
batch_normalization_62 (BatchNo (None, 128, 128, 256 1024        conv2d_65[0][0]
__________________________________________________________________________________________________
leaky_re_lu_62 (LeakyReLU)      (None, 128, 128, 256 0           batch_normalization_62[0][0]
__________________________________________________________________________________________________
max_pooling2d_13 (MaxPooling2D) (None, 64, 64, 256)  0           leaky_re_lu_62[0][0]
__________________________________________________________________________________________________
dropout_25 (Dropout)            (None, 64, 64, 256)  0           max_pooling2d_13[0][0]
__________________________________________________________________________________________________
conv2d_66 (Conv2D)              (None, 64, 64,<PHONE_NUMBER>     dropout_25[0][0]
__________________________________________________________________________________________________
batch_normalization_63 (BatchNo (None, 64, 64, 512)  2048        conv2d_66[0][0]
__________________________________________________________________________________________________
leaky_re_lu_63 (LeakyReLU)      (None, 64, 64, 512)  0           batch_normalization_63[0][0]
__________________________________________________________________________________________________
conv2d_67 (Conv2D)              (None, 64, 64,<PHONE_NUMBER>     leaky_re_lu_63[0][0]
__________________________________________________________________________________________________
batch_normalization_64 (BatchNo (None, 64, 64, 512)  2048        conv2d_67[0][0]
__________________________________________________________________________________________________
leaky_re_lu_64 (LeakyReLU)      (None, 64, 64, 512)  0           batch_normalization_64[0][0]
__________________________________________________________________________________________________
conv2d_transpose_13 (Conv2DTran (None, 128, 128,<PHONE_NUMBER>     leaky_re_lu_64[0][0]
__________________________________________________________________________________________________
concatenate_13 (Concatenate)    (None, 128, 128, 512 0           conv2d_transpose_13[0][0]
                                                                 leaky_re_lu_62[0][0]
__________________________________________________________________________________________________
dropout_26 (Dropout)            (None, 128, 128, 512 0           concatenate_13[0][0]
__________________________________________________________________________________________________
conv2d_68 (Conv2D)              (None, 128, 128,<PHONE_NUMBER>     dropout_26[0][0]
__________________________________________________________________________________________________
batch_normalization_65 (BatchNo (None, 128, 128, 256 1024        conv2d_68[0][0]
__________________________________________________________________________________________________
leaky_re_lu_65 (LeakyReLU)      (None, 128, 128, 256 0           batch_normalization_65[0][0]
__________________________________________________________________________________________________
conv2d_69 (Conv2D)              (None, 128, 128, 256 590080      leaky_re_lu_65[0][0]
__________________________________________________________________________________________________
batch_normalization_66 (BatchNo (None, 128, 128, 256 1024        conv2d_69[0][0]
__________________________________________________________________________________________________
leaky_re_lu_66 (LeakyReLU)      (None, 128, 128, 256 0           batch_normalization_66[0][0]
__________________________________________________________________________________________________
conv2d_70 (Conv2D)              (None, 128, 128, 255 65535       leaky_re_lu_66[0][0]
__________________________________________________________________________________________________
batch_normalization_67 (BatchNo (None, 128, 128, 255 1020        conv2d_70[0][0]
__________________________________________________________________________________________________
leaky_re_lu_67 (LeakyReLU)      (None, 128, 128, 255 0           batch_normalization_67[0][0]
__________________________________________________________________________________________________
conv2d_71 (Conv2D)              (None, 128, 128, 255 65280       leaky_re_lu_67[0][0]
__________________________________________________________________________________________________
batch_normalization_68 (BatchNo (None, 128, 128, 255 1020        conv2d_71[0][0]
__________________________________________________________________________________________________
leaky_re_lu_68 (LeakyReLU)      (None, 128, 128, 255 0           batch_normalization_68[0][0]
__________________________________________________________________________________________________
conv2d_72 (Conv2D)              (None, 128, 128, 255 65280       leaky_re_lu_68[0][0]
==================================================================================================
Total params: 7,302,647
Trainable params: 7,297,531
Non-trainable params: 5,116

Can anyone point out what I should do to eliminate this problem? Thanks.

@sankalpmittal1911-BitSian  From the logs of your Training It seems that validation Loss Is going down after every epoch.So I dont See a Problem in that.Try training it for more time.

@sankalpmittal1911-BitSian Thanks for the network. Can you use the current model to predict an image using a random input tile from the test dataset? Let's first check if the setup will work correctly before training it with more epoches.

@ZihengSun Okay. I will input any one of the (128,128,7) from the validation set and post the predicted (128,128,1) segmented image after the argmax of (128,128,255) output. I am right now using model trained for only 35 epochs. Thanks.

@ZihengSun Sir, So I used my model trained for only 30 epochs first. Got the accuracy of 53% (I said that it was not improving after certain epoch). Then I took (128,128,7) from the validation set and these 7 input images are:

Then I loaded my model and used it to predict the output:
ans=new_model.predict(X_test)
print(ans.shape)

ans_final=np.zeros((128,128,1))
index=-1
maxim=-1

for i in range(128):
  for j in range(128):
    for l in range(255):
      if(ans[0][i][j][l]>maxim):
        maxim=ans[0][i][j][l]
        index=l
    ans_final[i][j][0]=index
    index=-1
    maxim=-1

print(ans_final)

The predicted array:
[[[194.]
  [195.]
  [195.]
  ...
  [  5.]
  [  5.]
  [  4.]]

 [[195.]
  [195.]
  [195.]
  ...
  [  5.]
  [  5.]
  [  5.]]

 [[195.]
  [195.]
  [195.]
  ...
  [  5.]
  [  5.]
  [  5.]]

 ...

 [[176.]
  [176.]
  [176.]
  ...
  [  5.]
  [  5.]
  [  5.]]

 [[ 23.]
  [176.]
  [176.]
  ...
  [  5.]
  [  5.]
  [  5.]]

 [[175.]
  [176.]
  [176.]
  ...
  [  5.]
  [  5.]
  [  4.]]]

The predicted output:

The true output:

I can see that it needs more training but at least something is matching. Shall I use the color map to render my predicted output? Thanks.

@sankalpmittal1911-BitSian yeah, it is better for observing if rendered using colors.  The distribution shape starts to match but the values seem wrong.

@ZihengSun  Distribution shapes starts to match because I guess that set up of the model is actually making sense as far as the segmentation goes. Values i.e. pixel values predicted are wrong because accuracy is very less.
I tried training model for 10 epochs more and accuracy was literally hovering around 53% and is not improving. I have to increase accuracy to get the values right. What should I do to decrease loss further in the above model? I tried changing layers, optimizers, learning rate... Shall I add more layers?
Also I will post the color rendered image output. Thanks.

@sankalpmittal1911-BitSian I found that the way you get the results is a little weird to me.
for i in range(128):   for j in range(128):     for l in range(255):       if(ans[0][i][j][l]>maxim):         maxim=ans[0][i][j][l]         index=l     ans_final[i][j][0]=index     index=-1     maxim=-1
Is this the implementation of softmax? What is your code for compiling and training the model?

@ZihengSun  I managed to get training accuracy of 82 but on the validation set I cannot go beyond 70% accuracy.I tried various Learning Rates  and DropOut and Shifted from UNet to FCN.Should I try Ensembling.

@ZihengSun No sir, this is the prediction. Alternatively I could have employed numpy.argmax(axis=3). Here is my code for compiling and training model:
input_img = Input((im_height, im_width, 7), name='img')
model = get_unet(input_img, n_filters=512, dropout=0.15, batchnorm=True)
#model = get_unet()

model.compile(optimizer=Adam(lr=0.01), loss="categorical_crossentropy", metrics=["accuracy"])
model.summary()

Here is training:
callbacks = [
    EarlyStopping(patience=10, verbose=1),
    ReduceLROnPlateau(factor=0.1, patience=3, min_lr=0.00000001, verbose=1),
    ModelCheckpoint('model.h5', verbose=1, save_best_only=True, save_weights_only=False)
]

new_model=load_model('/content/model.h5')
results = new_model.fit_generator(get_data(train_ids, batch_size=2), steps_per_epoch=675 , epochs=20, verbose=1 , callbacks=callbacks, validation_data = get_data(valid_ids,batch_size=1) , validation_steps = 243)
I used model.fit_generator instead of model.fit because I was getting memory crash and thus I had to load data in batches.
I was getting validation accuracy almost same as training: 51% around.
Thank you.

@ZihengSun Can we try ensembling or are we restricted to a single Model

@at1998 Did the conversion of FCN from UNet made any difference? Also which type of ensembling technique are you referring?

@sankalpmittal1911-BitSian  I tried both the models and both are almost getting stuck at the same validation score.So In my case it didnt make any difference

Ensembling is allowed if it is making progress on the validation accuracy.

Training accuracy of 82 is very promising. Could you increase the ratio of training samples over testing samples? Normally for complex classification task on big data, the suggested ratio is 98:1:1. Since our sample dataset is small, I would say 90:5:5 might be good. I will add more samples to improve the feature diversity of the sample pool and reduce bias.

@sankalpmittal1911-BitSian ~50% and no longer increasing is not a good sign after ~30 epochs. It must have some places wrongly configured which makes the network not learning.

@ZihengSun I am using 90% for Training and 10% for validation.Havent split out the test set yet.Should i try to train  on 95% of the data.One thing that I have noticed is that averaging all the seven channels into 3 channels is producing and converging the Model much Faster rather than passing all the seven channels as Separate inputs.It could be due to poor Augmentation.

@at1998 Are you using (128,128,3) as input? Did you calculate average of 7 channels over 3 channels?

@ZihengSun In the AgNet Dataset We know that the 7 channels are 7 bands from LandSat8. Could we get an information on which band corresponds to which color spectrum.I searched on the Web and most articles mention LandSat8 having 11 channels.Could you please help me out.

@sankalpmittal1911-BitSian  I tried (1281283) as well as (1281287)

The bands in sample data come from Landsat 8 OLI sensor. More details about the band and its spectral region could be found here. Band 8 ~ 11  are rarely used in land cover classification. The multi spectral bands will help increase the feature space than only RGB bands. Neural network is supposed to take advantage of the more spectral features to get higher accuracy.

@ZihengSun I tried combining the 2,3 and 4th band to form RGB image and i will be posting soon,it doesnt seem like an image from satellite.Yes I understand that RGB wont be enough and I would be leveraging all the channels.Could you also clarify my doubts regarding The Proposal as i mentioned above?

You can include any idea that might improve the model in the proposal and state the reasons aside.

@ZihengSun This is what I am getting on stacking RGB channels Together.

Here is the code to combine the channels 👍
def visualise3chnnels(file_name):
  file_name=file_name.split("_")
  file_name=file_name[0]+"_"+file_name[1]+"_"+file_name[2]+"_"+file_name[3]
  path=f'{im_input}/'+file_name
  b2_link = gdal.Open(path + '_2.tif') # blue band
  b3_link = gdal.Open(path + '_3.tif') # green band
  b4_link = gdal.Open(path + '_4.tif') # red band
  b2 = norm(b2_link.ReadAsArray().astype(np.float))
  b3 = norm(b3_link.ReadAsArray().astype(np.float))
  b4 = norm(b4_link.ReadAsArray().astype(np.float))
  rgb = np.dstack((b4,b3,b2))
  plt.imshow(rgb)
  del b2, b3, b4
  sm.toimage(rgb,cmin=np.percentile(rgb,2),
               cmax=np.percentile(rgb,98)).save("RGB/"+file_name+'.png')

@ZihengSun I tried debugging my code. Softmax is performed correctly. All pixels over 255 classes sum to one. Data is being loaded correctly using the custom loader batchwise.  Now I am truly lost. Tried increasing the layers as well but the accuracy does not seem to improve. Changed metric to categorical_accuracy but again no improvement.

@ZihengSun Finally some improvement. Took only training this time.
->Increased number of layers.
->Trained for less data first, saved the model and gradually increased the data to 1593.
->Keras has issues with softmax. That is why I created a custom layer using backend.
Epoch 69/200
375/375 [==============================] - 1214s 3s/step - loss: 0.6522 - categorical_accuracy: 0.7840
Epoch 70/200
375/375 [==============================] - 1324s 3s/step - loss: 0.6465 - categorical_accuracy: 0.7900
Epoch 71/200
375/375 [==============================] - 1152s 3s/step - loss: 0.6459 - categorical_accuracy: 0.7921
Epoch 72/200
375/375 [==============================] - 1223s 3s/step - loss: 0.6426 - categorical_accuracy: 0.7944
Epoch 73/200
375/375 [==============================] - 1198s 3s/step - loss: 0.6457 - categorical_accuracy: 0.7921
Epoch 74/200
375/375 [==============================] - 1201s 3s/step - loss: 0.6468 - categorical_accuracy: 0.7909

The Landsat band value ranges are not 0~255. They are 16-bit numbers. Valid range is 0~10000. The creation of true-color image is not straightforward. Here is a guide how to create true-color Landsat image. My suggestion is rendering your results using the CDL color table first. That is easier.

The training looks promising. You need check the validation accuracy at the same time to avoid overfitting. Early stop might also be required, e.g., stop if no improvement after 10 consecutive epochs.

I am Swati Kanchan, a final year B. Tech undergraduate from NIT Durgapur, India. I have been working with Deep Learning in biomedical imaging for last one year and also published a work on Brain tumor classification using CNN, achieving the best accuracy, in proceedings of ICACCP'19. Recently, I along with my team won in Smart India Hackathon'19 where I developed two deep learning models and a user-friendly GUI application for Brain image segmentation and visualization.
I have started training on the sample training dataset without any kind of preprocessing provided by you on a slightly modified version of LinkNet for 13 epochs which is giving an accuracy of around 55%.
Below is graph for accuracy and Loss.

I have also started training this dataset using the FCN8 model and it will be complete 40 epochs by night. I will keep you informed about the accuracy and send the graphs for the same as soon as possible.

@ZihengSun

Here is what I have been able to go to in the training after training for about 150 epochs.The validation accuracy is still around 70% So I am going to work on that.Increase Dropout and try various methods.It seems to be stuck in this region.In the first 50 epochs it was able to go from 53-80% but then it too about 50 more epochs to get above 85%.

@ZihengSun I have kept dropout as 0.15 and I am getting 73% validation categorical accuracy.

@sankalpmittal1911-BitSian  I am not using Keras

LinkNet seems interesting. Looking forward to see if it can produce the similar results as the other nets used here.

@sankalpmittal1911-BitSian I will read it and get back to you as soon as I can.
Do you mean the left 93 images are used as validation datasets?

@at1998 I suggest you could print the validation accuracy as well in the log. It seems the training accuracy is still rising but very slow. Maybe it is already overfitting.

@ZihengSun  OKay I will do that and share it.I will also try adding some weight decay to try to stabilise it more and get back to you.

@ZihengSun Yes, those 93 images are in validation set.

@ZihengSun  Is there any kind Of Limit on How Lengthy the proposal Can be.I have almost finished my first draft of the proposal and its almost 8 pages Long.Do I need to Reduce it?

There is no length limit for the whole proposal. However, the project abstract should be no more than ten sentences. Precisely describe your plan. It must be doable within the project time window.

My draft is 13 pages. What should I modify in that? Can you please comment?
Thank you.

The length is fine.
I have read your draft. It has all the required sections. That is very good. You might need more emphasize your idea and work plan. The background and challenge introduction should be very brief in abstract. The abstract must have no more than 10 sentences.
Customizing deep learning model is not equal to switching among deep learning models. Trying different models and comparing their results is very important. But more importantly, we need know why. Why does one model perform better than others? If the reasons are figured, we will know how to improve it to next level. You need be more specific about how you are going to customize models and why it will improve the accuracy, what is the expected ideal outcome? You need figure out which part of the model is hindering the learning and proposing a solution like adding a customized layer, apply customized loss function, change the input features, etc.
The deliverable part is too brief. Not sure what results will be generated and how they can be used by the community. Please be more specific and rational.

Okay, I will get back with the updated draft as soon as I can. Thank you for the feedback.

@ZihengSun  I have mailed you my proposal at<EMAIL_ADDRESS>you please go through it and provide me the feedback

@ZihengSun  Can you please give me suggestions to improve the draft.Thanks

@at1998  It reads really good. The plan is clear and doable. The network configuration is promising. My advice is adding some core equations to help explain how the model will converge well on training datasets with highly complex semantics. It is also better if the deliverables are explicitly itemized with expected outcome (e.g., source code, software release, web installation package, documentation, etc).

@ZihengSun I can build up the equations and can explain why the model will converge faster however I didnt quite get what you meant in the latter part.Can you please give me an example.

Like where the code will reside, how to manage the version of models, what software will be used to create the documentation, what license will be used for the source code and the generated data results respectively, etc. These information is basic but essential in a proposal.

@ZihengSun  Okay I will do that.One more thing that i needed to discuss was that I have only recently begun with Open Source Projects I have done a lot of My own personal Projects and Have worked at companies as an Intern and took part in Various Deep Learning Competitions.But I havent contributed much to the the open source community.Will that pose a problem ?

@ZihengSun  I will Edit the Proposal And get back to you by tomorrow

Open source contributors' experiences will add credit to your proposal. No or less experiences are also fine, as long as you are willing to contribute to open source project in this summer.

@ZihengSun  Thank You for the clarification

@ZihengSun  I tried various methods for a UNet and after 85% it starts to overfit.If i train it for more the training accuracy goes up by 1-2% but then validation accuracy starts jumping between 76-78%.I am going to try FCN and observe if similar things happen.

@ZihengSun I am also working on the equations and updating my proposal and will be sending the next version in a day or two.Any Suggestions on the above problem ?

@ZihengSun  I was thinking about writing a custom loss function and then multiplying it by some factor to penalise it even more.
lets say loss=1.5 and no.of inaccurate predictions were 34.We can get 34 between a fixed range say[1,4] and multiply the loss by that.The more inaccurate predictions the more heavily the model is penalised and the more accurate predictions it makes the less.Would that work out

@ZihengSun  I noticed that the dataset is imbalanced, so I guess we could replace the loss with the weighted loss function. I am trying the same right now. I will also send updated proposal in 1-2 days.

I am also going to try PSPNet or LinkNet because for UNet I cannot go beyond 80% training and 73% validation.

If the network starts to overfit when validation accuracy is ~80%, I would say its capability cap is already reached. Customization is the only way to go beyond that line.

The dataset is imbalanced because the land cover is naturally biased. The pixels of roads are much fewer than the pixels of crop fields. You could try class weight.

Hi all - a friendly reminder that there is ONE WEEK LEFT to submit your proposals for this project! Best of luck and we're excited to see what is submitted!

@ZihengSun  I know that very less time is left. I will send you updated proposal by today's end. Shall I also share that on the portal or wait for it?

@abburgess Hi Annie, where should the proposals be submitted to?

@ZihengSun  please check your email. I've sent something regarding proposal :smile:

Draft proposals should be submitted through the GSoC portal.
Annie Burgess, PhD
Lab Director | Earth Science Information Partners (ESIP)
esipfed.org/lab http://esipfed.org/lab |<PHONE_NUMBER>
Sign up for the monthly ESIP Lab update here http://eepurl.com/dtKL8z.
On Thu, Apr 4, 2019 at 7:43 AM Ziheng Sun<EMAIL_ADDRESS>wrote:

@abburgess https://github.com/abburgess Hi Annie, where should the
proposals be submitted to?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ESIPFed/gsoc/issues/13#issuecomment-479926811, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AG_XwIfbcdCWA3cx0muteSOshLN2cM23ks5vdg-rgaJpZM4afBCr
.

@ZihengSun  Please check your email for updated draft proposal. What can I do to improve it further?

Also shall I share it with the organization?

Hi @ZihengSun !
I am very excited to contribute to this project. I know that it's way too late to get started, but I think that I can understand much in this one week.
So, can you please provide me with the required details to get started.
Thanks.

@ZihengSun Also, I found a very interesting publication: https://www.mdpi.com/2072-4292/10/8/1217/pdf
It makes use of temporal dependencies, as in it uses GRUs to classify crops. Maybe I can make use of that. Right now I am trying to find segmentation models which can achieve accuracy greater than 80% and I am thinking of accomplishing this by the end of 6th May (Pre-GSoC).

@ZihengSun  I have Sent you the Updated Proposal.Can you please go through it and please provide me with FeedBack before i upload it on the official GSOC Site.ThankYou

@ZihengSun  I trained my model for about 100 epochs(takes about an hour ) and while my training loss started going down very slowly but my validation loss started Stabilising to 78%.I will add The Custom Layer and Try On some more Customisation Itself.Meanwhile Can you please help me on with the proposal .Thanks

@ZihengSun I have already tried using custom lambda layer and it doesn't seem to make any difference. I have already used UNet. I will FPN/PSPNet/LinkNet and get back with the result as soon as I can.
Also please give feedback to the draft proposal.

Hi @ZihengSun !
I am very excited to contribute to this project. I know that it's way too late to get started, but I think that I can understand much in this one week.
So, can you please provide me with the required details to get started.
Thanks.

Most information is available in this thread.

@ZihengSun I've sent the proposal for review on your mail :smile:

@ZihengSun  I have Updated The Proposal Added Some Things.Can you please go through it.I understand that at this point you might be flooded with reviewing proposals so make things simpler I have Highlighted Whatever Changes I Made So You wont have to Go through Entire Document.I really Hope That you can Give it One Final Read Before I submit on the portal tomorrow

@ZihengSun Should I give link to the Visualization Notebook Used in the proposal?

@ZihengSun I have uploaded the final proposal. Please see other sections like training steps where I have delved deeper regarding eliminating overfitting, underfitting etc. Under current approach, I have added reasons why approach is not working and possible steps to eliminate the issues. I hope I have covered mathematically and logically the schedule. Thank you.

Thank you all for your efforts!


Run unit tests in the same process

Purpose
$subject.
Fixes #20205
Check List

[x] Read the Contributing Guide
[ ] Updated Change Log
[ ] Checked Tooling Support (#)
[x] Added necessary tests

[ ] Unit Tests
[ ] Spec Conformance Tests
[ ] Integration Tests
[ ] Ballerina By Example Tests

[ ] Increased Test Coverage
[ ] Added necessary documentation

[ ] API documentation
[ ] Module documentation in Module.md files
[ ] Ballerina By Examples

Has the time improved with this change?

@Kishanthan Yes, back to about 12 mins now (in MacOS) :)

Codecov Report

Merging #20216 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #20216   +/-   ##
=======================================
  Coverage    19.9%    19.9%
=======================================
  Files          55       55
  Lines        1437     1437
  Branches      218      218
=======================================
  Hits          286      286
  Misses       1138     1138
  Partials       13       13

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 96e43b9...edaf8b8. Read the comment docs.


duplicate entries in $HOME/.aws/credentials crashes cli

Description
If you have duplicate aws profile entries in $HOME/.aws/credentials cli app wont work
Steps to reproduce
copy paste a profile in your $HOME/.aws/credentials
run cli hello world
sam local invoke HelloWorldFunction --event event.json
Observed result
$ sam local invoke HelloWorldFunction --event event.json
Traceback (most recent call last):
  File "/usr/local/Cellar/aws-sam-cli/0.12.0/libexec/lib/python3.7/site-packages/botocore/configloader.py", line 149, in raw_config_parse
    cp.read([path])
  File "/usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/python3.7/configparser.py", line 696, in read
    self._read(fp, filename)
  File "/usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/python3.7/configparser.py", line 1065, in _read
    lineno)
configparser.DuplicateSectionError: While reading from '/Users/homeboy/.aws/credentials' [line 332]: section '156562137045' already exists

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/sam", line 11, in <module>
    load_entry_point('aws-sam-cli==0.12.0', 'console_scripts', 'sam')()
  File "/usr/local/Cellar/aws-sam-cli/0.12.0/libexec/lib/python3.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/Cellar/aws-sam-cli/0.12.0/libexec/lib/python3.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/Cellar/aws-sam-cli/0.12.0/libexec/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/Cellar/aws-sam-cli/0.12.0/libexec/lib/python3.7/site-packages/click/core.py", line 1064, in invoke
    sub_ctx = cmd.make_context(cmd_name, args, parent=ctx)
  File "/usr/local/Cellar/aws-sam-cli/0.12.0/libexec/lib/python3.7/site-packages/click/core.py", line 621, in make_context
    self.parse_args(ctx, args)
  File "/usr/local/Cellar/aws-sam-cli/0.12.0/libexec/lib/python3.7/site-packages/click/core.py", line 880, in parse_args
    value, args = param.handle_parse_result(ctx, opts, args)
  File "/usr/local/Cellar/aws-sam-cli/0.12.0/libexec/lib/python3.7/site-packages/click/core.py", line 1404, in handle_parse_result
    self.callback, ctx, self, value)
  File "/usr/local/Cellar/aws-sam-cli/0.12.0/libexec/lib/python3.7/site-packages/click/core.py", line 78, in invoke_param_callback
    return callback(ctx, param, value)
  File "/usr/local/Cellar/aws-sam-cli/0.12.0/libexec/lib/python3.7/site-packages/samcli/cli/options.py", line 55, in callback
    state.profile = value
  File "/usr/local/Cellar/aws-sam-cli/0.12.0/libexec/lib/python3.7/site-packages/samcli/cli/context.py", line 69, in profile
    self._refresh_session()
  File "/usr/local/Cellar/aws-sam-cli/0.12.0/libexec/lib/python3.7/site-packages/samcli/cli/context.py", line 78, in _refresh_session
    profile_name=self._aws_profile)
  File "/usr/local/Cellar/aws-sam-cli/0.12.0/libexec/lib/python3.7/site-packages/boto3/__init__.py", line 34, in setup_default_session
    DEFAULT_SESSION = Session(**kwargs)
  File "/usr/local/Cellar/aws-sam-cli/0.12.0/libexec/lib/python3.7/site-packages/boto3/session.py", line 80, in __init__
    self._setup_loader()
  File "/usr/local/Cellar/aws-sam-cli/0.12.0/libexec/lib/python3.7/site-packages/boto3/session.py", line 120, in _setup_loader
    self._loader = self._session.get_component('data_loader')
  File "/usr/local/Cellar/aws-sam-cli/0.12.0/libexec/lib/python3.7/site-packages/botocore/session.py", line 679, in get_component
    return self._components.get_component(name)
  File "/usr/local/Cellar/aws-sam-cli/0.12.0/libexec/lib/python3.7/site-packages/botocore/session.py", line 902, in get_component
    self._components[name] = factory()
  File "/usr/local/Cellar/aws-sam-cli/0.12.0/libexec/lib/python3.7/site-packages/botocore/session.py", line 150, in <lambda>
    lambda:  create_loader(self.get_config_variable('data_path')))
  File "/usr/local/Cellar/aws-sam-cli/0.12.0/libexec/lib/python3.7/site-packages/botocore/session.py", line 233, in get_config_variable
    logical_name)
  File "/usr/local/Cellar/aws-sam-cli/0.12.0/libexec/lib/python3.7/site-packages/botocore/configprovider.py", line 226, in get_config_variable
    return provider.provide()
  File "/usr/local/Cellar/aws-sam-cli/0.12.0/libexec/lib/python3.7/site-packages/botocore/configprovider.py", line 323, in provide
    value = provider.provide()
  File "/usr/local/Cellar/aws-sam-cli/0.12.0/libexec/lib/python3.7/site-packages/botocore/configprovider.py", line 382, in provide
    config = self._session.get_scoped_config()
  File "/usr/local/Cellar/aws-sam-cli/0.12.0/libexec/lib/python3.7/site-packages/botocore/session.py", line 324, in get_scoped_config
    profile_map = self._build_profile_map()
  File "/usr/local/Cellar/aws-sam-cli/0.12.0/libexec/lib/python3.7/site-packages/botocore/session.py", line 218, in _build_profile_map
    self._profile_map = self.full_config['profiles']
  File "/usr/local/Cellar/aws-sam-cli/0.12.0/libexec/lib/python3.7/site-packages/botocore/session.py", line 362, in full_config
    cred_file)
  File "/usr/local/Cellar/aws-sam-cli/0.12.0/libexec/lib/python3.7/site-packages/botocore/configloader.py", line 152, in raw_config_parse
    path=_unicode_path(path))
botocore.exceptions.ConfigParseError: Unable to parse config file: /Users/homeboy/.aws/credentials

Expected result
working cli app
Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

OS: OS X 10.11
sam --version: 0.12.0

Add --debug flag to command you are running

I was not able to replicate this with a duplicate profile, can you provide structure of your credentials file?

This is used by aws cli as well, and read by boto too, and since this was not reproducible. I'm closing it out, please re-open if deemed necessary.

I agree to rojomisin. I was able to replicate the issue and It got fixed after deleting the duplicate entry in credentials file.


pipewire: 0.3.76 -> 0.3.77

Diff: https://gitlab.freedesktop.org/pipewire/pipewire/-/compare/0.3.76...0.3.77
Changelog: https://gitlab.freedesktop.org/pipewire/pipewire/-/releases/0.3.77
Description of changes

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[x] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Oh, I missed the merge, but I was rebuilding due to rustc with this included, so it's also in staging-next already as 9e7a159d8c15df


Invalid code generated for parameters of array types with empty array as default value

Problem statement
The code generator emits invalid go code for parameters of array types that have an empty array as default value, e.g.:

sample input

swagger: "2.0"
info:
  title: Sample API
  description: API description in Markdown.
  version: 1.0.0
host: api.example.com
basePath: /v1
schemes:
  - https
paths:
  /users:
    get:
      summary: Returns a list of users in certain locations.
      description: Optional extended description in Markdown.
      produces:
        - application/json
      responses:
        200:
          description: OK
      parameters:
        - name: locations
          in: query
          type: array
          description: Locations filter
          default: []
          items:
            type: string

trying to generate a client:

$ swagger generate client -f swagger.yml
2021/03/09 10:04:16 validating spec /tmp/swagger-test/swagger.yml
2021/03/09 10:04:16 preprocessing spec with option:  minimal flattening
2021/03/09 10:04:16 building a plan for generation
2021/03/09 10:04:16 generation target ./
2021/03/09 10:04:16 planning definitions (found: 0)
2021/03/09 10:04:16 planning operations (found: 1)
2021/03/09 10:04:16 grouping operations into packages (packages: 1)
2021/03/09 10:04:16 operations for package packages "operations" (found: 1)
2021/03/09 10:04:16 planning meta data and facades
2021/03/09 10:04:16 rendering 2 templates for operation
2021/03/09 10:04:16 name field GetUsers
2021/03/09 10:04:16 package field operations
2021/03/09 10:04:16 creating generated file "get_users_parameters.go" in "client/operations" as parameters
2021/03/09 10:04:16 executed template asset:clientParameter
2021/03/09 10:04:16 source formatting failed on template-generated source ("client/operations/get_users_parameters.go" for parameters). Check that your template produces valid code
2021/03/09 10:04:16 unformatted generated source "get_users_parameters.go" has been dumped for template debugging purposes. DO NOT build on this source!
source formatting on generated source "parameters" failed: client/operations/get_users_parameters.go:96:37: expected operand, found ',' (and 10 more errors)

inspecting client/operations/get_users_parameters.go:

[...]
/ SetDefaults hydrates default values in the get users params (not the query body).
//
// All values with no default are reset to their zero value.
func (o *GetUsersParams) SetDefaults() {
  var (
        locationsDefault = []string{,}

  )
[...]

Fixed by #2534


update embedded WG repos

Now that most of our teams have OK-ed the use of highfive let's add more repos

r? @rust-highfive
(rust_highfive has picked a reviewer for you, use r? to override)

Thanks! You need to run the script to update the webhooks now. Redeploying...

@pietroalbini thanks! I ran the script locally (and saw all the repos get "added") but highfive is not working (rust-embedded/cortex-m-rt#175). Should I wait a bit more?

Did you give highfive write access to all those repos?

I forgot! It's working after fixing the perms. Thanks again.


Add the notion of tolerance

This removes the issue of skipping periods by adding a 10% tolerance.
The reason we'd previously skip periods is because we would wait strictly for at least the collection period before exporting.
This doesn't work because:
-Sometimes parts of code execute faster or slower
-The application is multithreaded
-The ticker probably gets a few microseconds off
We add a tolerance of 10% of the collection period. Since it's only 10%, we shouldn't have issues with exporting significantly too early.
Say we have a schedule with a collection period of 10 minutes.
-At t0, we export.
-At t0+9.59 minutes, we would export, where previously we wouldn't.

Sounds pretty neat! I was wondering though, the issue with skipping periods seems to occur mainly with smaller periods -- 1 second, 5 seconds -- and potentially would not occur for higher periods. In the event someone instrumented a metric to be collected every 10 days, say, would tolerance mean that it would in fact be collected every 9 days? Would this alter the timing a little too far in the opposite direction?
As an alternative, what do you think about queuing metrics to be exported? So when they ought to be collected, they are instead added a queue to be exported. When the load is low and the periods are long, the queue should be mostly empty, so metric collection would occur at the same rate. If the load is high and the periods are short, our timing might not work out to the period exactly, but averaged over a long period we still have the same amount of data.

Sorry if I'm not getting it, but how does the queuing approach work? Is it like:
-We have a period with 5s
-When we export, we put time + 5s into a queue
-Every time we tick, we check the front of the queue for which metrics we ought to export?
This might run into the same issue involving slight time differences.
Otherwise, this was actually my first implementation idea, back in the yonder days of June! I think the current method was favoured because it was less complicated to understand.

Otherwise, this was actually my first implementation idea, back in the yonder days of June! I think the current method was favoured because it was less complicated to understand.

Nice! I think I misunderstand the mechanics of how exporting works -- is the bottle neck in exporting here, where the periods are being checked, or in the collect call, where the metrics are being matched to periods? I originally assumed it might've been the latter, so in the event metric collection can't quite match the period, the export data can be added to a queue to be exported as soon as possible, rather than dropped in favor of a new incoming collect.

Otherwise, this was actually my first implementation idea, back in the yonder days of June! I think the current method was favoured because it was less complicated to understand.

Nice! I think I misunderstand the mechanics of how exporting works -- is the bottle neck in exporting here, where the periods are being checked, or in the collect call, where the metrics are being matched to periods? I originally assumed it might've been the latter, so in the event metric collection can't quite match the period, the export data can be added to a queue to be exported as soon as possible, rather than dropped in favor of a new incoming collect.

I thought so too initially! In turns out the matching is pretty fast. Even if it wasn't, it'd have been okay, because it only happens once before.


[DISE-6114] Updated get status API docs

(1) Updated status/enum to make this field consistent across all services is.
(2) Updated status code from 202 to 200 to match the actual response code.

Just signed Adobe CLA. Will open a new one.


SWS-349 SWSUI deploy job fails after GitHub organization renaming

Attempt to fix the Travis deploy job

I don't know if it fixes the problem, but I guess it won't hurt...


Fix Europeana success test, activate reingestion workflows

Description

This PR addresses a number of items with Europeana. I'm happy to split this out into multiple PRs if that would be most warranted, but all of the commits are focused on Europeana so this seemed easiest.

Fixed API responses
Updated the start date of the daily DAG
Updated the reingestion DAG

API responses
After testing this DAG after the merge of #821, I noticed that we were receiving no data for the run. The logs had Request failed with success = "False" in each of the data pull tasks. After some investigation, it appeared that the value for success that we were receiving was actually the Python/JSON native true, rather than "True". @stacimc mentioned that she was able to test the DAG successfully while the PR was being reviewed. It looks like this was because the initial check for success was in get_should_continue rather than get_batch. Based on the lifecycle of the ProviderDataIngester, the result of get_should_continue is only used after at least one batch has been processed:
https://github.com/WordPress/openverse-catalog/blob/91ab102df69a004224e339c1a86d6ee8f552f8a0/openverse_catalog/dags/providers/provider_api_scripts/provider_data_ingester.py#L183-L190
This meant that during testing, data was still being processed even though the success check was causing should_continue to be False. Moving the check into get_batch right before merging caused the DAG to no longer process any data.
To rectify this, I've updated the values stored in our response JSONs for testing and updated the check itself to use Python native boolean types.
Updated start date
Since we have the reingestion workflow, we don't need this DAG to run over the last decade's worth of data. This is in-line with other changes we've made to dated DAGs.
Updated reingestion workflow
I've also updated the reingestion workflow to use the new provider data ingester base class. I was able able to test this and have it run successfully!
Testing Instructions

Add the Europeana API key to your local environment if you haven't already
Run the daily europeana_workflow
Run the reingestion europeana_ingestion_workflow

Checklist

[x] My pull request has a descriptive title (not a vague title like Update index.md).
[x] My pull request targets the default branch of the repository (main) or a parent feature branch.
[x] My commit messages follow best practices.
[x] My code follows the established code style of the repository.
[x] I added or updated tests for the changes I made (if applicable).
[ ] I added or updated documentation (if applicable).
[x] I tried running the project locally and verified that there are no visible errors.

Developer Certificate of Origin

Developer Certificate of Origin
Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.

Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

I noticed the reingestion DAG doesn't have docs in the Airflow UI. Do we have an issue to add those yet?

They'll get added in https://github.com/WordPress/openverse-catalog/pull/849, albeit just the same doctstrings as are on the regular workflows.