Complement crawled data

The result of the crawler made by @douglasferlini in #5 is a JSON array with the overall information of the dresses os La Fiancee. Now, we need now to get the details of the dresses and generate a more detailed JSON array.
To do so, you will read the attached JSON and will make a request for the product API providing the urlPartin the following format:
const productID = <urlPart_from_json_array>
await fetch("https://www.lafiancee.com.br/_api/wix-ecommerce-storefront-web/api", {
    "credentials": "include",
    "headers": {
        "Accept": "*/*",
        "Authorization": "brUTfgwc9eaqQ4m_KjbIkjnR-MRt9rGfCLGikGEPiRU.eyJpbnN0YW5jZUlkIjoiMWI0OTQ1ODItZDg5Zi00MmY2LTg0YzAtNTAxOGE3NzI1Y2MyIiwiYXBwRGVmSWQiOiIxMzgwYjcwMy1jZTgxLWZmMDUtZjExNS0zOTU3MWQ5NGRmY2QiLCJtZXRhU2l0ZUlkIjoiN2RlM2ExNjgtNDEyNC00NDljLTg4ZDYtZmViNjkzYWY3NzRjIiwic2lnbkRhdGUiOiIyMDIwLTA5LTIzVDEyOjI3OjE4LjUyOVoiLCJ2ZW5kb3JQcm9kdWN0SWQiOiJQcmVtaXVtMSIsImRlbW9Nb2RlIjpmYWxzZSwiYWlkIjoiOWE0ZjJjNDAtMTIzNC00ZGM3LTg3OWEtMjIzZDMxMzI0N2E1IiwiYmlUb2tlbiI6IjY2YWFlNGVhLTk5YmItMDY2YS0wYzE2LWFlYWUzNGRkMmI4ZSIsInNpdGVPd25lcklkIjoiZmI0Y2Y2ODQtODZkZS00N2E0LWE2NjUtZjE4ZDcxYzA3YzUxIn0",
        "Content-Type": "application/json; charset=utf-8",
    },
    "body": `{"query":"query getProductBySlug($externalId: String!, $slug: String!, $withPricePerUnit: Boolean!, $withCountryCodes: Boolean!) {
          appSettings(externalId: $externalId) {
            widgetSettings
      }
      catalog {
            product(slug: $slug, onlyVisible: true) {
                id
                description
                isVisible
                sku
                ribbon
                price
                comparePrice
                discountedPrice
                formattedPrice
                formattedComparePrice
                formattedDiscountedPrice
                pricePerUnit @include(if: $withPricePerUnit)
                formattedPricePerUnit @include(if: $withPricePerUnit)
                pricePerUnitData @include(if: $withPricePerUnit) {
                baseQuantity
                baseMeasurementUnit
          }
          seoTitle
          seoDescription
          createVersion
          digitalProductFileItems {
                fileId
                fileType
                fileName
          }
          productItems {
                price
                comparePrice
                formattedPrice
                formattedComparePrice
                pricePerUnit @include(if: $withPricePerUnit)
                formattedPricePerUnit @include(if: $withPricePerUnit)
                optionsSelections
                isVisible
                inventory {
                status
                quantity
            }
            sku
            weight
            surcharge
            subscriptionPlans {
                list {
                id
                price
                formattedPrice
                pricePerUnit @include(if: $withPricePerUnit)
                formattedPricePerUnit @include(if: $withPricePerUnit)
              }
            }
          }
          name
          isTrackingInventory
          inventory {
            status
            quantity
          }
          isVisible
          isManageProductItems
          isInStock
          media {
            id
            url
            fullUrl
            altText
            thumbnailFullUrl: fullUrl(width: 50, height: 50)
            mediaType
            videoType
            videoFiles {
                url
                width
                height
                format
                quality
            }
            width
            height
            index
            title
          }
          customTextFields {
            title
            isMandatory
            inputLimit
          }
          nextOptionsSelectionId
          options {
            title
            optionType
            selections {
                id
                value
                description
                linkedMediaItems {
                    altText
                    url
                    fullUrl
                    thumbnailFullUrl: fullUrl(width: 50, height: 50)
                    mediaType
                    width
                    height
                    index
                    title
                    videoFiles {
                        url
                        width
                        height
                        format
                        quality
                    }
                }
            }
          }
          productType
          urlPart
          additionalInfo {
                id
            title
            description
            index
          }
          subscriptionPlans {
                list(onlyVisible: true) {
                  id
              name
              tagline
              frequency
              duration
              price
              formattedPrice
              pricePerUnit @include(if: $withPricePerUnit)
              formattedPricePerUnit @include(if: $withPricePerUnit)
            }
            oneTimePurchase {
                  index
            }
          }
          discount {
                mode
            value
          }
          currency
          weight
          seoJson
        }
      }
      localeData(language: "en") @include(if: $withCountryCodes) {
            countries {
              key
          shortKey
        }
      }
    }","variables":{"slug":productID,"externalId":"","withPricePerUnit":true,"withCountryCodes":false},"source":"WixStoresWebClient","operationName":"getProductBySlug"}`,
    "method": "POST",
});

This request will return a JSON that has the product options with title and selections which will be the features.
With this enhanced JSON Array we can build the database to serve this data.

Hi! I believe I should be able to tackle this. I've already forked the repository to https://www.github.com/afiorentino/stop-analyzing-api

Awesome, @afiorentino !! Just assigned to you this task.
If you think it would be better to handle it in another repository or using another programming language, feel free to do so, because this is not the core of stop-analyzing-api. But if you are comfortable doing it here, it is perfect too!
Thanks!

Oh I forgot to attach the JSON crawled by @douglasferlini. Here it goes:
lafiancee.json.zip

Hi, @tiagostutz
I'be been working on this task and I believe that I'm almost finished. I I'm curious if what I'm building is intended to be specific to the lafiancee site, or if I should try and be more generic so that part(s) of my contribution can be reused.

This case will be La Fiancee specific because this payload is from its website. But, as this site is made using Wix I think this would be an easy path to use this work with Wix based sites, but it is not a rule.

Hi @tiagostutz
I appreciate your patience with me regarding the completion of this task. I'm nearly complete, but I'm getting a 400 response code for the URL in the above code snippet. My repository is located at https://github.com/afiorentino/stop-analyzing-lafiancee-enhanced-json
I would love any feedback you have for me at this current stage and also a solution to the 400 response code I'm getting from the lafiancee website.

Hi there @afiorentino.!
You mean, 400 from this link: https://github.com/bancodobrasil/stop-analyzing-api/files/5276983/lafiancee.json.zip ?
Check it please, if you can go through.
I'll take a look at your repo and will place my observations as issues on your repo. Sounds good?

@KarineValenca could you tackle this?

@KarineValenca could you tackle this?

Sure!

Sure!

@tiagostutz I'm developing the code in this repo https://github.com/KarineValenca/stop-analyzing-enhanced-json
However, I have some questions. For each one of the urlPart in the attached JSON, I'm generating receiving the following response:
{"data":{"appSettings":{"widgetSettings":{}},"catalog":{"product":{"id":"1f48a12b-6dc9-4a1f-be94-e0b215a77210","description":"","isVisible":true,"sku":"cod316","ribbon":"","price":0.0,"comparePrice":0.0,"discountedPrice":0.0,"formattedPrice":"R$ 0,00","formattedComparePrice":"","formattedDiscountedPrice":"R$ 0,00","pricePerUnit":null,"formattedPricePerUnit":null,"pricePerUnitData":null,"seoTitle":null,"seoDescription":null,"createVersion":1568930104742000,"digitalProductFileItems":[],"productItems":[],"name":"Marca Melissa Sweet","isTrackingInventory":false,"inventory":{"status":"in_stock","quantity":0},"isManageProductItems":false,"isInStock":true,"media":[{"id":"daf591_e1a483b66a084ceea6c8e225e395538c~mv2.jpg","url":"daf591_e1a483b66a084ceea6c8e225e395538c~mv2.jpg","fullUrl":"https://static.wixstatic.com/media/daf591_e1a483b66a084ceea6c8e225e395538c~mv2.jpg/v1/fit/w_500,h_500,q_90/file.jpg","altText":null,"thumbnailFullUrl":"https://static.wixstatic.com/media/daf591_e1a483b66a084ceea6c8e225e395538c~mv2.jpg/v1/fit/w_50,h_50,q_90/file.jpg","mediaType":"PHOTO","videoType":null,"videoFiles":[],"width":540,"height":810,"index":0,"title":"melissa sweet 2 (Copy).jpg"},{"id":"daf591_35439db0ade24cb49c16a27c9832ac56~mv2.jpg","url":"daf591_35439db0ade24cb49c16a27c9832ac56~mv2.jpg","fullUrl":"https://static.wixstatic.com/media/daf591_35439db0ade24cb49c16a27c9832ac56~mv2.jpg/v1/fit/w_500,h_500,q_90/file.jpg","altText":null,"thumbnailFullUrl":"https://static.wixstatic.com/media/daf591_35439db0ade24cb49c16a27c9832ac56~mv2.jpg/v1/fit/w_50,h_50,q_90/file.jpg","mediaType":"PHOTO","videoType":null,"videoFiles":[],"width":540,"height":810,"index":1,"title":"melissa sweet 1 (Copy).jpg"}],"customTextFields":[],"nextOptionsSelectionId":6,"options":[{"title":"Estilo","optionType":"DROP_DOWN","selections":[{"id":1,"value":"Boho Chic","description":"Boho Chic","linkedMediaItems":null}]},{"title":"Modelagem","optionType":"DROP_DOWN","selections":[{"id":2,"value":"Evasê","description":"Evasê","linkedMediaItems":null}]},{"title":"Corpete","optionType":"DROP_DOWN","selections":[{"id":3,"value":"Alcinhas","description":"Alcinhas","linkedMediaItems":null},{"id":4,"value":"Decote em coração","description":"Decote em coração","linkedMediaItems":null}]},{"title":"Tecido","optionType":"DROP_DOWN","selections":[{"id":5,"value":"Renda 3D","description":"Renda 3D","linkedMediaItems":null}]}],"productType":"physical","urlPart":"marca-melissa-sweet","additionalInfo":[],"subscriptionPlans":{"list":[],"oneTimePurchase":null},"discount":{"mode":"PERCENT","value":0.0},"currency":"BRL","weight":0.0,"seoJson":null}}}}
This is right?
Another question, do I need to aggregate the JSON responses in one single file? If yes, how should I aggregate this?

@tiagostutz I'm developing the code in this repo https://github.com/KarineValenca/stop-analyzing-enhanced-json
However, I have some questions. For each one of the urlPart in the attached JSON, I'm generating receiving the following response:
{"data":{"appSettings":{"widgetSettings":{}},"catalog":{"product":{"id":"1f48a12b-6dc9-4a1f-be94-e0b215a77210","description":"","isVisible":true,"sku":"cod316","ribbon":"","price":0.0,"comparePrice":0.0,"discountedPrice":0.0,"formattedPrice":"R$ 0,00","formattedComparePrice":"","formattedDiscountedPrice":"R$ 0,00","pricePerUnit":null,"formattedPricePerUnit":null,"pricePerUnitData":null,"seoTitle":null,"seoDescription":null,"createVersion":1568930104742000,"digitalProductFileItems":[],"productItems":[],"name":"Marca Melissa Sweet","isTrackingInventory":false,"inventory":{"status":"in_stock","quantity":0},"isManageProductItems":false,"isInStock":true,"media":[{"id":"daf591_e1a483b66a084ceea6c8e225e395538c~mv2.jpg","url":"daf591_e1a483b66a084ceea6c8e225e395538c~mv2.jpg","fullUrl":"https://static.wixstatic.com/media/daf591_e1a483b66a084ceea6c8e225e395538c~mv2.jpg/v1/fit/w_500,h_500,q_90/file.jpg","altText":null,"thumbnailFullUrl":"https://static.wixstatic.com/media/daf591_e1a483b66a084ceea6c8e225e395538c~mv2.jpg/v1/fit/w_50,h_50,q_90/file.jpg","mediaType":"PHOTO","videoType":null,"videoFiles":[],"width":540,"height":810,"index":0,"title":"melissa sweet 2 (Copy).jpg"},{"id":"daf591_35439db0ade24cb49c16a27c9832ac56~mv2.jpg","url":"daf591_35439db0ade24cb49c16a27c9832ac56~mv2.jpg","fullUrl":"https://static.wixstatic.com/media/daf591_35439db0ade24cb49c16a27c9832ac56~mv2.jpg/v1/fit/w_500,h_500,q_90/file.jpg","altText":null,"thumbnailFullUrl":"https://static.wixstatic.com/media/daf591_35439db0ade24cb49c16a27c9832ac56~mv2.jpg/v1/fit/w_50,h_50,q_90/file.jpg","mediaType":"PHOTO","videoType":null,"videoFiles":[],"width":540,"height":810,"index":1,"title":"melissa sweet 1 (Copy).jpg"}],"customTextFields":[],"nextOptionsSelectionId":6,"options":[{"title":"Estilo","optionType":"DROP_DOWN","selections":[{"id":1,"value":"Boho Chic","description":"Boho Chic","linkedMediaItems":null}]},{"title":"Modelagem","optionType":"DROP_DOWN","selections":[{"id":2,"value":"Evasê","description":"Evasê","linkedMediaItems":null}]},{"title":"Corpete","optionType":"DROP_DOWN","selections":[{"id":3,"value":"Alcinhas","description":"Alcinhas","linkedMediaItems":null},{"id":4,"value":"Decote em coração","description":"Decote em coração","linkedMediaItems":null}]},{"title":"Tecido","optionType":"DROP_DOWN","selections":[{"id":5,"value":"Renda 3D","description":"Renda 3D","linkedMediaItems":null}]}],"productType":"physical","urlPart":"marca-melissa-sweet","additionalInfo":[],"subscriptionPlans":{"list":[],"oneTimePurchase":null},"discount":{"mode":"PERCENT","value":0.0},"currency":"BRL","weight":0.0,"seoJson":null}}}}
This is right?
Another question, do I need to aggregate the JSON responses in one single file? If yes, how should I aggregate this?

Hi @KarineValenca ! Sorry for the late response
Yes, this is the JSON you posted is for a single product, right?
The resulting aggregated JSON for that would be something like:
[
{
   "id":"5f355a24ccb4180025ee98ab",
   "title":"Marca Melissa Sweet",
   "subtitle":"",
   "choiceURL":"https://www.lafiancee.com.br/product-page/marca-melissa-sweet",
   "contentURL":[
      "https://static.wixstatic.com/media/daf591_e1a483b66a084ceea6c8e225e395538c~mv2.jpg/v1/fit/w_500,h_500,q_90/file.jpg",
      "https://static.wixstatic.com/media/daf591_35439db0ade24cb49c16a27c9832ac56~mv2.jpg/v1/fit/w_500,h_500,q_90/file.jpg"
   ],
   "attributes":[
      {
         "estilo":"Boho Chic"
      },
      {
         "modelagem":"Evasê"
      },
      {
         "corpete":"Alcinhas"
      },
      {
         "corpete":"Decote em coração"
      },
      {
         "tecido":"Renda 3D"
      }
   ]
},
...
]

There was a discussion about this JSON format at https://github.com/bancodobrasil/stop-analyzing-api/issues/29#issuecomment-702963899 and https://github.com/bancodobrasil/stop-analyzing-api/issues/16#issuecomment-676524806, but now that we are working on a real use case we can see that there were some pitfalls there.
We probably have to update the Database modeling to reflect some changes here, like:

changing the contentURL attribute, that was a single URL and now is an array
adding the choiceURL attribute that has the product URL

Another important thing to note is that there will be one value in the attributes field for each value in selections array of the options attribute in the LaFiancee JSON, as you can see with the corpete attribute.
Have I filled all the gaps or do you have some more doubts? Please share here if so.
Thanks!

Hi @KarineValenca ! Sorry for the late response
Yes, this is the JSON you posted is for a single product, right?
The resulting aggregated JSON for that would be something like:
[
{
   "id":"5f355a24ccb4180025ee98ab",
   "title":"Marca Melissa Sweet",
   "subtitle":"",
   "choiceURL":"https://www.lafiancee.com.br/product-page/marca-melissa-sweet",
   "contentURL":[
      "https://static.wixstatic.com/media/daf591_e1a483b66a084ceea6c8e225e395538c~mv2.jpg/v1/fit/w_500,h_500,q_90/file.jpg",
      "https://static.wixstatic.com/media/daf591_35439db0ade24cb49c16a27c9832ac56~mv2.jpg/v1/fit/w_500,h_500,q_90/file.jpg"
   ],
   "attributes":[
      {
         "estilo":"Boho Chic"
      },
      {
         "modelagem":"Evasê"
      },
      {
         "corpete":"Alcinhas"
      },
      {
         "corpete":"Decote em coração"
      },
      {
         "tecido":"Renda 3D"
      }
   ]
},
...
]

There was a discussion about this JSON format at https://github.com/bancodobrasil/stop-analyzing-api/issues/29#issuecomment-702963899 and https://github.com/bancodobrasil/stop-analyzing-api/issues/16#issuecomment-676524806, but now that we are working on a real use case we can see that there were some pitfalls there.
We probably have to update the Database modeling to reflect some changes here, like:

changing the contentURL attribute, that was a single URL and now is an array
adding the choiceURL attribute that has the product URL

Another important thing to note is that there will be one value in the attributes field for each value in selections array of the options attribute in the LaFiancee JSON, as you can see with the corpete attribute.
Have I filled all the gaps or do you have some more doubts? Please share here if so.
Thanks!

@tiagostutz Your explication was clear! I'm just not sure how to fill the id, subtitle, and contentURL. Could you help me with that? After knowing that, I'll finish the enhanced JSON.

@tiagostutz Your explication was clear! I'm just not sure how to fill the id, subtitle, and contentURL. Could you help me with that? After knowing that, I'll finish the enhanced JSON.

Hi @KarineValenca

id: will be generated by the local persistence (like a MongoDB)
contentURL: will be the link to the content itself. In the La Fiancee case, it will be "https://www.lafiancee.com.br/product-page/"+ the attribute urlPart.
subtitle: In this case it will be empty.

If you have any more questions, please post here.! =)

Hi @KarineValenca

id: will be generated by the local persistence (like a MongoDB)
contentURL: will be the link to the content itself. In the La Fiancee case, it will be "https://www.lafiancee.com.br/product-page/"+ the attribute urlPart.
subtitle: In this case it will be empty.

If you have any more questions, please post here.! =)

Hi, again @tiagostutz. This is the enhanced JSON I built:
https://github.com/KarineValenca/stop-analyzing-enhanced-json/blob/master/enhanced.json
Please check it out. Any problems, let me know.

Hi, again @tiagostutz. This is the enhanced JSON I built:
https://github.com/KarineValenca/stop-analyzing-enhanced-json/blob/master/enhanced.json
Please check it out. Any problems, let me know.

Perfect.!! That's the expected JSON. Thanx! =)
Is this project runnable locally? I mean, if we need to download an updated JSON from La Fiancee site and parse again, is it possible? If so, could you provide a README on how to do so?

Perfect.!! That's the expected JSON. Thanx! =)
Is this project runnable locally? I mean, if we need to download an updated JSON from La Fiancee site and parse again, is it possible? If so, could you provide a README on how to do so?

Yes, it is possible. I wrote the instructions here:
https://github.com/KarineValenca/stop-analyzing-enhanced-json/blob/master/README.md
We just need to update the lafiancee.json file with the new data, and it will fetch the new data.

Yes, it is possible. I wrote the instructions here:
https://github.com/KarineValenca/stop-analyzing-enhanced-json/blob/master/README.md
We just need to update the lafiancee.json file with the new data, and it will fetch the new data.


Error about isomeric SMILES to molecules and stereo type of bonds

**Hi, everyone:
When I queried the stereo type of bonds in a isomeric molecule, GetStereo function returned “STEREONONE” .
My code is:
m=Chem.MolFromSmiles('C[C@@H]1CCC[C@@H]1C')
for bond in m.GetBonds():
      print(bond.GetStereo())

I think I have given the isomeric SMILES, why rdkit can not return the stereo types of all bonds ?
I tried the function Chem.DetectBondStereochemistry(m), but rdkit still returned "STEREONONE".
Moreover,
When I transferred the molecule to the isomeric SMILES by rdkit:
print(Chem.MolToSmiles(m,isomericSmiles=True))
rdkit returned: C[C@H]1CCC[C@H]1C
the stereo type has changed.
**
who can help me about this problem? thanks a lot

RDKit Version: 2<IP_ADDRESS>
Platform: Windows 10 x64 python 2.7.14

m=Chem.MolFromSmiles('C[C@@H]1CCC[C@@H]1C')
for bond in m.GetBonds():
      print(bond.GetStereo())
print(Chem.MolToSmiles(m,isomericSmiles=True))

Sorry for the extremely slow reply.
In the RDKit's terminology the stereochemistry of bonds refers to E/Z (or CIS/TRANS) stereochemistry of double bonds, so there is no double bond stereochemistry in this molecule.
The molecule does have two atomic stereocenters that have stereo information associated with them:
In [15]: m=Chem.MolFromSmiles('C[C@@H]1CCC[C@@H]1C')
    ...: for atom in m.GetAtoms():
    ...:       print(atom.GetChiralTag())
    ...:
CHI_UNSPECIFIED
CHI_TETRAHEDRAL_CCW
CHI_UNSPECIFIED
CHI_UNSPECIFIED
CHI_UNSPECIFIED
CHI_TETRAHEDRAL_CCW
CHI_UNSPECIFIED

Could it be that you're interested in the wedging of the bonds around chiral centers? This is not present by default, but you can add it:
In [17]: from rdkit.Chem import rdDepictor

In [18]: m=Chem.MolFromSmiles('C[C@@H]1CCC[C@@H]1C')
    ...: rdDepictor.Compute2DCoords(m)
    ...: Chem.WedgeMolBonds(m,m.GetConformer())
    ...: for bond in m.GetBonds():
    ...:       print(bond.GetBondDir())
    ...:
BEGINWEDGE
NONE
NONE
NONE
NONE
BEGINWEDGE
NONE

Note that I needed to generate a set of 2D coordinates as well, since bond wedging information only makes sense in the context of a conformer.
Does that help?


IncrementalIndex's in ThreadLocal doesn't need to be instance-level

IncrementalIndex's ThreadLocal in can be static final. This ThreadLocal doesn't store anything specific to a given IncrementalIndex: rather, it's used to set a Row temporarily, and there is no situation when an IncrementalIndex could call recursively into another IncrementalIndex, so this temporary assignment cannot be disturbed.
As a safety net, we could add a check if (in.get() != null) throw ISE(..); in.set(row);
Furthermore, it would be ideal to get rid of ThreadLocal altogether (which would be beneficial for performance, because every Aggregator wouldn't need to query ThreadLocal upon every indexed row (ThreadLocal.get() is like Map.get(), not super cheap), and for reducing complexity, but this requires a significant refactoring of Aggregator interface and/or indexing logic, so saving this idea for a better time.

@leventov can you please have a look at PR?


Short-term roadmap

Short-term roadmap
Now that F# 4.0 is out the door, we wanted to outline plans for Visual F# spanning the coming weeks and months.  Below represents a short-term roadmap, covering work that is in-progress or will happen in the coming weeks.  @KevinRansom is putting together a longer-term roadmap and will post that when it's ready.  We are eager to get regular code flow moving again!
CoreCLR
As you can see in #499, we are working on standing up the F# compiler and runtime on CoreCLR.  When CoreCLR hits 1.0, it's important that F# is ready and supported.  In particular, getting the core F# compilation toolchain working on CoreCLR for cross-platform development is something we really want to invest in. We consider this to be one of, if not the, top priority item for us in the coming months.
We are starting with just the basics -

Compiler
Runtime
Console apps

Then we will add in more and more as the platform stabilizes and we can build it -

Interactive
Type providers - binary compatible? Need re-compile?
Cross-platform IDE tools based on .NET Core/CoreCLR

Clear out backlog of PRs
We put a lot of PRs on hold while F# 4.0 was locking down. We even have a few lingering PRs on our old CodePlex page! We are now looking forward to clearing the backlog and making room for new contributions.
In particular, there are a number of pending "cleanup" PRs that do large-scale re-arrangement and cleanup of the codebase. It will be disruptive, but we think it's best to merge these first, getting the hard part out of the way. Once this is done we will clear the other bugfix and performance PRs.  This will likely require PR authors to assist with resolving merge conflicts, so please bear with us.
Branches
Here are the branches we want to use for this upcoming work.

Branch
Purpose
Flow

master
VS vNext/Dev15 work
Send all general PRs here

OOB
VS 2015 Update/OOB update work
Cherry-pick fixes from master

coreclr
CoreCLR work
Takes merges from OOB

We will use git tags to mark the final commits of various releases.
In picture form:

The existing fsharp4 branch will turn into OOB and we will delete fsharp4.  Once the PR backlog has been cleared, we will merge everything back into master, which will be where most contributions occur in the near future. coreclr will track with OOB.
With F# 4.0 development we had a confusing split where we asked some PRs to be made against master and others against fsharp4. This time it should be simpler - send all PRs to master, and we will simply cherry-pick the appropriate changes into the OOB branch for the next update release.
master develepment will ultimately turn into "F# 4.1" or whatever ships in the next major Visual Studio release.
OOB development will ultimately become the next update release. This might be timed with VS Update, or might be totally out-of-band.

+1 for merging cleanup PRs first. It's a huge win in the long term.

Thanks for putting this out on the open. As part of the coreCLR work, will there be support for creating 100% F# UWP apps and ASPvNext, which I beleive are "profiles" of coreCLR without needing to have a C# entrypoint?

@ovatsus There are a lot of moving parts, which are easy to confuse (pretty sure I'm still confused).
The F# compiler can already output portable libraries which in theory should be consumable and runnable by ASP vNext.  But if we want support for on-the-fly compilation within ASP nNext, the compiler itself needs to be runnable on .NET Core (that's what the current work is).  Even once that's possible, there is still nontrivial plumbing to be done within ASP.NET in order for pure-F# entrypoints to be possible.
For UWP, the compiler doesn't need to run on .NET Core, it just needs to output compatible assemblies. Technically we can do that today via portable libraries. However .NET Native is a required part of the UWP pipeline, and as it stands .NET Native can't handle F# assemblies. We are working on addressing that.  But once again, there are more steps before pure-F# entrypoints are possible.

Oh, I didn't realize this was about getting the compiler itself to run on CoreCLR. What I'm mostly interested in is on having a CoreCLR FSharp.Core and having the F# compiler target them, so I can have a full UWP F# app without needing the PCLs. The problem with the PCLs is that you cannot access any of the platform specific APIs from F#, forcing you to use C# for a big chunk of the functionality.
I didn't know UWP required .NET Native, thought that was an optional optimization. That alone will make sure we won't see UWP F# apps anytime soon :(

I'm happy with whatever path you choose. Just tell me what to you need me to do (happy to collapse changed into single check-in, probably not interested in my rambling paths to solutions anyway!)

I know it's really important for some people that we follow all the platform trends and all the super innovative ASP.Net vnext project ideas, but what about real language features? Or things that improve the daily life of a F# developer.

@forki   The above is the near-term roadmap for the Visual F# Tools team, specifically w.r.t. branches.  Tooling, cross-platform reach and .NET Core are, in the long run, very important to Visual F#'s viability and I believe the above are the right thing for the Visual F# Tools team to be focusing on.
The F# language itself is in good shape and the vast majority of requests/dissatisfaction/needs we see are around tooling, integration into broader programming scenarios (cross-plat, mobile, UWP, cloud etc.). The Visual F# Tools team are always open to PRs in areas such as bug-fixing, performance, cleanup, tooling etc..
At this moment I think its best that improvements to the F# language be a separate track of work on a different delivery timeline. Currently, given the above branches, I'd recommend submitting PRs to OOB (or master) for later integration without there being a specific branch for integrating those as yet.  Right now, there's no specific timeline for F# 4.1 or 5.0 as a language design. To be honest I'm happy to let things settle for a while after getting F# 4.0 out the door. We are still, for example, in the process of seeing F# 4.0 out into the wild on Linux, Xamarin, FSharp.Compiler.Service etc.
For the language design process, my personal approach is to devote part of my time to that area (but not all).  My personal current work in this area has been to go through the http://fslang.uservoice.com suggestions to refine them to a more meaningful set of propositions. I've been marking some minor things as approved/planned. Certainly people can get started on those and submit them as independent PRs. However I can't give a specific timeline for how language improvements would be integrated into the repo branches outlined above.

thanks @dsyme. I know that the language design side takes time, but it would still be interesting to see what the next steps could be. But you are right that's probably not "short term".
The second thing that I meant is tooling. Now that F# 4.0 is released, I wonder what the direction is? I know there were discussions about omnisharp, atom, VFPT, FCS and so on. There is not much about this in this short term agenda. I hope to learn more about the plans in @KevinRansom's longer-term roadmap.

@ovatsus Yes, the current work is to get the compiler itself (and ultimately interactive, IDE tooling, etc) to target .NET Core, so that it can run on CoreCLR, and thus cross-platform and (in theory) within ASP vNext.
Accessing platform APIs in modern Windows apps in orthogonal to CoreCLR/.NET Core. That requires WinRT support. Supporting WinRT is yet another big ball of wax and at this point we aren't looking to prioritize any work in that area.
Like ASP, getting support for a pure-F# entrypoint in UWP (assuming current .NET Native blockers are resolved) is additional plumbing and tooling, it's not just a matter of targeting the right platform.

@latkin Does any of the UWP include shared projects for F#?
Having to rely on linked files or duplicated code is less than ideal for mobile applications.

@7sharp9 expanding the IDE support to include shared projects is certainly on the table, but not currently considered a top priority.  We would of welcome a PR that implemented this, though.

@latkin are you going to change the default branch to OOB or where do you expect most PR atm?

I was looking at that last week, but GitHub warned be that it could have an impact on pending PRs.  I didn't want to disrupt any of those.  Do you happen to know if changing the default branch is safe in this regard?
Once I get through enough backlog I'll merge OOB into master, then everything should go against master.

yes using NAV would even be better
2015-08-11 18:18 GMT+02:00 Lincoln Atkinson<EMAIL_ADDRESS>
I was looking at that last week, but GitHub warned be that it could have
an impact on pending PRs. I didn't want to disrupt any of those. Do you
happen to know if changing the default branch is safe in this regard?
Once I get through enough backlog I'll merge OOB into master, then
everything should go against master.
—
Reply to this email directly or view it on GitHub
https://github.com/Microsoft/visualfsharp/issues/563#issuecomment-129952756
.

Do you happen to know if changing the default branch is safe in this regard?

I can't remember any issues, but who knows.

I don't think changing the default branch has any negative impact, it just means GitHub will be set up to direct new PRs to that branch by default.

Ok I went ahead and changed the default to OOB

The F# compiler can already output portable libraries which in theory should be consumable and runnable by ASP vNext. But if we want support for on-the-fly compilation within ASP vNext, the compiler itself needs to be runnable on .NET Core (that's what the current work is). Even once that's possible, there is still nontrivial plumbing to be done within ASP.NET in order for pure-F# entrypoints to be possible.

This plumbing is trivial (unless you include tooling, then I have no clue), and I've already done it more than once. To get F# support in DNX today (no CoreCLR though) see https://github.com/YoloDev/YoloDev.Dnx.FSharp.

closing old discussion


Speeding up PyTests

Reducing iterations of K-H dataloader
Reducing latent variable size in RNNs
Reduction of field size in metric test

/blossom-ci


Do not emit log if attestation time is within the minute

Fixes #1966.
What do you think, @bonomat? Alternatively I would just not bother, but I don't know how often we get misled by this log.

I think we overcomplicated our life with what is happening in olivia.rs.
Imho best is to just deserialize to Response and check it attestation is Some, else ignore and fetch again later.
https://github.com/itchysats/itchysats/blob/b4b41f2bc5d14a21d785544584eea2009c02d3c9/model/src/olivia.rs#L145-L149

I think we overcomplicated our life with what is happening in olivia.rs. Imho best is to just deserialize to Response and check it attestation is Some, else ignore and fetch again later.
https://github.com/itchysats/itchysats/blob/b4b41f2bc5d14a21d785544584eea2009c02d3c9/model/src/olivia.rs#L145-L149

I see what you mean, but then we would never learn if Olivia has stopped attesting: we would continue to ignore and retry.

I think that's better than now. We are tracking how often we fetch announcements and if this keeps failing, e.g. nothing was successfully fetched in the last 1h, then we know that something is wrong.

I think that's better than now. We are tracking how often we fetch announcements and if this keeps failing, e.g. nothing was successfully fetched in the last 1h, then we know that something is wrong.

What do you think of returning a custom error if attestation is null in here:
https://github.com/itchysats/itchysats/blob/b4b41f2bc5d14a21d785544584eea2009c02d3c9/model/src/olivia.rs#L176
Then we can check on this particular error and ignore/debug print it instead.

I think that's better than now. We are tracking how often we fetch announcements and if this keeps failing, e.g. nothing was successfully fetched in the last 1h, then we know that something is wrong.

I think it's important to note that Olivia can successfully produce announcements and fail to update the corresponding endpoints with an attestation. What you describe would miss this sort of problem if I understand it correctly.

What do you think of returning a custom error if attestation is null in here:
[snip]
Then we can check on this particular error and ignore/debug print it instead.

It's good that you suggest this, because my approach in this PR misses the fact that deserialising into an attestation can fail for multiple reasons. I think we want to emit a custom error as you suggest; match against it when we deserialise; and then decide what to do.
I still think it might be good to silence any logs if the deserialisation error comes within the minute, but we don't need to be that clever.

I'm gonna close this since it's not that important and it should matter less with https://github.com/itchysats/itchysats/issues/1966.


fix: #[component] unknown attributes propagation

Currently, when you mark a component as deprecated, the meta attribute is propagated to the dummy model, when it should be propagated to the "real" model.
To check the behaviour, you can just mark as #[deprecated] the SimpleCounter component of the counter example and run trunk serve:
#[deprecated="Hello"]
#[component]
pub fn SimpleCounter( ... )

Before:
warning: use of deprecated function `__SimpleCounter`: Hello
 --> src/lib.rs:8:8
  |
8 | pub fn SimpleCounter(
  |        ^^^^^^^^^^^^^
  |
  = note: `#[warn(deprecated)]` on by default

After:
warning: use of deprecated function `counter::SimpleCounter`: Hello
 --> src/main.rs:1:14
  |
1 | use counter::SimpleCounter;
  |              ^^^^^^^^^^^^^
  |
  = note: `#[warn(deprecated)]` on by default

warning: use of deprecated function `counter::SimpleCounter`: Hello
 --> src/main.rs:8:18
  |
8 |         view! { <SimpleCounter initial_value=0 step=1/> }
  |                  ^^^^^^^^^^^^^

warning: `counter` (bin "counter") generated 2 warnings

Initially I started implementing for #[deprecated] attributes but I realize that can be extended to every attribute that is not #[doc =]. Not totally sure about the implications but it seems safe to me.

Thanks!


Allow filtering components in source structure

This PR allows app developers to filter the source structure (components tree) by component type (all, visible and non-visible). This change is brought from Kodular sources.

@barreeeiroo @SusanRatiLane I have rebased this onto master now that the UI binder refactor has been merged. Tested both the classic and new forms. It works but I'm not super excited about having to pass the parent to the child so that the child can manipulate the parent. We'll have to think about that more.


Make "show modified" option of the settings editor more prominent

With the new settings editor UI, the current way to filter the list of settings to only show previously modified settings is found in the ubiquitous ellipsis/more menu on the top right corner of the editor tab.
It wasn't obvious to me that the ellipsis menu options vary with each editor–usually it only contains the same three options–so it took me a moment to find the "show modified settings" option.
Although I understand the reasoning behind placing the option there, and I value a simple UI, I think the ellipsis menu can be overlooked when looking for an option specific to the settings search box, especially by users who are unaware that the option exists in the first place.

Regular editor ellipsis menu:

Settings editor ellipsis menu:

An alternative would be create another category for modified settings on level with "Commonly Used":

The ... button used to be in the editor, but we moved it up there to be consistent with other editors: https://github.com/Microsoft/vscode/issues/61590#issuecomment-432305306

Yes, and it's something that I'm still paying attention to. But we decided

Moving things always causes some small amount of pain but people will get used to it
We documented it in the release notes for people who read those at least
Many people didn't find it before either...

I can't speak to the previous version of the GUI editor, but I'm sure users will adapt.
With the earlier JSON editor in mind, I think there's a case to be made for the sake of new users. With the editable column exclusively listing modified settings, you reaped the following advantages:

Immediate access to frequently altered settings.
Time saved not searching for forgotten or cryptic settings names.
A fast way to rule out settings-related issues.
A concise presentation of the way you use the editor.

Obviously the feature is still present, but I think the above when discovered early on is especially valuable to users who haven't yet familiarized themselves with the editor and the various settings that influence its behavior. To reap the benefits, there is now a discoverability challenge in requiring the user to explore and avoid making a wrong assumption about the ellipsis menu options.
On an unrelated note, there seems to be a discrepancy in the terms used between the two current UIs to describe modified and default settings.
I understand this isn’t a high priority and that it's a result of a compromise, so I appreciate that you're still paying attention to the development. I just want to make sure my concerns are represented in the feedback since I came too late for #50249.

You're not wrong, but navigating the JSON still proved to be enough of a hurdle for many new users that we've found that the GUI is still a better default experience for new users.

On an unrelated note, there seems to be a discrepancy in the terms used between the two current UIs to describe modified and default settings.

Not sure what you mean by that?

I prefer the GUI too, I'm just concerned with the discoverability of the "modified settings" in its current location.
In 1.30.0 on the special JSON settings editor, "User Settings" is juxtaposed with "Default User Settings" whereas in the GUI, "User Settings" seems to include both modified and default values and appears instead in contrast to "Workplace Settings".
I just discovered #62964 in Insiders, so I guess it's no longer relevant.


Use node 16 in typescript dockerfile

The LTS support for Node 12 stops by the end of April 2022. Node 16 is the latest version which is mostly a requirement for current frontend frameworks, bundlers etc.
We should make typescript-toolset work with Node 16.
/cc @henninggross @m-apsolon

will sneak in a proposal to have nodejs version for LTS as parameter, defaulting to 14? :)

Had a small test derived from the ods-pipeline Dockerfile.typescript-toolset
# removed some stuff:
ods-pipeline$ docker images | grep nvm
test-nvm-only                          1.06GB
test-nvm-16                            1.31GB
test-nvm-14-16                         1.58GB
test-nvm-12-14-16                      1.77GB

These are not actually working at all but I wanted to get a size estimate.
For me some questions are:

Are these sizes acceptable?
The image without node is still at 1GB. Is there something in there we would not need?
How are images updated. If the dockerfile uses nvm install 1.16 --lts to install the latest 1.16 LTS version this will be the version at build time. How would this be updated?

Before going into the details, I want to explain my general viewpoint on dealing with versions and backwards compatibility.
I think that an open source project like this should not strive to cater for all use cases. Older versions fall in this category for me because I believe it is actually harder than most people believe to deliver well-working, well-tested solutions for a matrix of versions. I would rather focus on delivering one rock-solid offering that can be relied upon than delivering multiple things which "kind of work" or "likely work".
Therefore when I see nvm, it sounds intriguing, but I approach it with a lot of skepticism. Introducing it is probably easy, but how easy is it to guarantee all offered versions work with all other things in the image? How much does this increase the chance that something breaks? Or that we end up in difficult situations because it is hard to get multiple things work at the same time in the future?
For me the cost of maintaining older versions should be "paid" closer to the place where the benefits are gained. Only then can you weigh cost against benefit. For example, if a company has many projects that use old versions and want to continue doing so instead of upgrading, that company needs to bear the maintenance effort as well ...
Alright, general considerations aside :)
I would highly recommend to at most support two versions. For me that should be 14 and 16, because 12 is reaching its EOL in a few months already as noted above.
Keeping with the "two versions available" theme, I would see the next update to include 16 and 18 (assuming that is the next LTS). This would happen in my eyes whenever 18 is available and can be integrated into ODS pipeline. As to ODS pipeline versions, this might require a new major version for the task as removing Node.js 14 is breaking.
In terms of image size, @henrjk I think you also mentioned that the current size is 1.14GB, right? Going to 1.58GB is not great but maybe still acceptable as typically it would be cached on the nodes preforming the builds?

I agree with the sentiment that supporting flexible tooling comes at a high cost and would also favor instead of aiming to support specific (major) versions only instead of giving too much flexibility which might cause unforeseen issues.
I am agreeing with the "2 version available" in general. However as ods-pipeline is just being conceived perhaps it would make sense to only support 2 versions going forward.
In this sprit toolset typescript-toolset-node-16 could be provided and the current typescript-toolset (based on 12) could be removed.
In ~ May 2022 typescript-toolset-node-18 could be added and when 20 is supported 16 could be removed.
Regarding the current image I wonder wether it would make sense to remove cypress prereqs?
Would it be possible to package to package this separately so that e2e tests can be done either as another step in a pipeline or independently?

@michaelsauter @gerardcl @henninggross would you agree the first two paragraphs of the previous comment or should we be more backwards compatible? Anybody else who should have a chance to chime in here?

I agree with a single typescript-toolset-node-16 task for now. When Node.js 18 is available we can introduce it as a new build task. Then we can have both tasks for some time overlap, and eventually retire typescript-toolset-node-16.
I would love to hear some frontend devs confirm that Node.js 16 is the right choice for "right now". @m-apsolon @henninggross @netzartist @cschweikert?

Oh actually, can we name it node16-typescript-toolset? At least ending in -toolset for consistency with other images.

The latest stable versions of popular frameworks require an active LTS version (which is 16) and from my experience there are no issues with Node.js 16.

In reference to the information Node.js has on their website about their releases the timeline they provide clearly indicates that Node.js 16 is the current Active version while 14 stays in 'Maintenance' at least until Node.js 18 is released (and even longer).
Therefore I would say the best solution is to go with 2 versions at max to always offer the Current and the latest version in Maintenance.
👉  Node.js 16 and Node.js 14 for the time being.

@henninggross Agree going forward.
However to keep effort low right now, I suggest we go with @henrjk  suggestion to start with a node16-typescript-toolset. Once Node.js 18 is released we can add a second node18-typescript-toolset, and then always keep two versions around.
Who wants to take a stab at this?

I'd be happy to support but perhaps someone with more frontend experience could lead here?

I could start trying to implement this, but I will definitely come back to you for support @henrjk!


Add Getting Started section to README

README Update
Please add a_getting started_section to the README so folks know how to start with this project.

Add Get Started section in README file


Generated files should follow OS line endings

I'm submitting a...
[ ] Regression (a behavior that used to work and stopped working in a new release)
[ ] Bug report  <!-- Please search GitHub for a similar issue or PR before submitting -->
[ ] Performance issue
[x] Feature request
[ ] Documentation issue or request
[ ] Support request => https://github.com/ngxs/store/blob/master/CONTRIBUTING.md
[ ] Other... Please describe:

Current behavior
While running checking in the files generated by schematics on my linux machine, noticed that all line-ending were DOS style. See attached image.

Expected behavior
The generated files should follow the default line-endings of the OS.
Minimal reproduction of the problem with instructions
Add schematics to an existing Angular project on a Linux machine.
ng g @ngxs/schematics:store --name=app --path=app/store

Check the line endings on generated files.
What is the motivation / use case for changing the behavior?
While making changes to generated files, too many EOL related changes show up - depending upon the editor & git configuration.
For now, I am running dos2unix on generated files.
Environment
Libs:
- @angular/core version: 8.2.0
- @ngxs/store version: 3.5.0

For Tooling issues:
- Node version: v10.16.0
- Platform:  Linux

Others:
Fedora release 29 (5.1.18-200.fc29.x86_64)

+1


Site.css not found

I am trying to add some custom styling to this sample.
Tried adding the code as per documentation
var viewOptions = new DefaultViewServiceOptions();
viewOptions.Stylesheets.Add("/Content/Site.css");
var factory = new IdentityServerServiceFactory();
factory.ConfigureDefaultViewService(viewOptions);
when I step through the code "viewOptions" has it listed
idSvrFactory.ConfigureDefaultViewService(viewOptions);
I keep getting a 404 error.
Thanks

Do you have a ~/content/site.css on your server? Also, I'd look at the rendered HTML to ensure the custom .css file is listed in the HTML.

So should a custom stylesheet have its own <link> entry in the html page?


Cannot verify installation

Thank you for developing such a great plugin. However, unfortunately, I am having a difficult time attempting to verify the installation of the sdk.
I have added following to my bundle.gradle
subprojects{
    repositories {
        mavenCentral()
        maven { url 'http://devrepo.kakao.com:8088/nexus/content/groups/public/' }
    }
}

Then also added following to app/bundle.gradle
dependencies {
    implementation "org.jetbrains.kotlin:kotlin-stdlib-jdk7:$kotlin_version"
    testImplementation 'junit:junit:4.12'
    androidTestImplementation 'androidx.test:runner:1.1.1'
    androidTestImplementation 'androidx.test.espresso:espresso-core:3.1.1'
    implementation 'com.facebook.android:facebook-login:[5,6)'
    // 카카오 로그인 sdk를 사용하기 위해 필요.
    implementation group: 'com.kakao.sdk', name: 'usermgmt', version: '1.27.0'

    // 카카오링크 sdk를 사용하기 위해 필요.
    implementation group: 'com.kakao.sdk', name: 'kakaolink', version: '1.27.0'

    // 카카오톡 sdk를 사용하기 위해 필요.
    implementation group: 'com.kakao.sdk', name: 'kakaotalk', version: '1.27.0'

    // 카카오내비 sdk를 사용하기 위해 필요.
    implementation group: 'com.kakao.sdk', name: 'kakaonavi', version: '1.27.0'

    // 카카오스토리 sdk를 사용하기 위해 필요.
    implementation group: 'com.kakao.sdk', name: 'kakaostory', version: '1.27.0'

    // push sdk를 사용하기 위해 필요.
    implementation group: 'com.kakao.sdk', name: 'push', version: '1.27.0'
}

Then I have registered my ids like below in the main function
  KakaoContext.clientId = 'APP KEY';
  KakaoContext.javascriptClientId = 'JS KEY';

Then in the onClick listener I am calling below
print('kakao')
final installed = await isKakaoTalkInstalled(); // THIS IS ALWAYS FALSE
print('is kakao installed ${installed}');

However, the installed variable is always false.
What am I doing wrong?

You can reply in Korea if you want. I have written the post in English just in case you don't speak Korean.

And I also registered my debug and release hashes to kakao developers account

Never mind... It was because my dev android phone doesn't have kakaotalk installed.


Roles: default roles not being assigned to new users

Think this was caused by a batch process we have running which was removing the roles.


Set selected value inside custom multiselect component

Hello, I'm using the 2.0 beta 13 version of vue-multiselect. I would like to know how to set the selected value of multiselect when you use it in a custom component. If i try to do it via props vue js warn me about mutating a prop directly.
Case 1: In a add new line modal I would like to reset the value
Case 2: In an edit line model I would like to set all the selected value to a specific value.

for case 1 : use :value attribute in multiselect component and make it's initial value null or empty object(if you are using object).
for case 2 : just assign value to the :value attribute property that you want to set as selected.

Thanks!


error about squeeze

'''
InvalidArgumentError (see above for traceback): Tried to explicitly squeeze dimension 1 but dimension was not 1: 2
'''

Sorry, where does the error occur?

'''
InvalidArgumentError (see above for traceback): Tried to explicitly squeeze dimension 1 but dimension was not 1: 2
'''

This could be a version issue for python and TensorFlow.
You can try to change it with the README.

I met the same issue. Did you solve it?

Sorry, where does the error occur?
Here:
with slim.arg_scope(inception_v3.inception_v3_arg_scope()):
logits_v3, end_points_v3 = inception_v3.inception_v3(
input_diversity(x), num_classes=num_classes, is_training=False)

I met the same issue

I met the same issue. Did you solve it?

solved

Hi,Yinpeng. a question. why do you set the prediction of inception_resnet_v2 as the target label? and the loss is computed bu the (y,pred_logits). so is this means that you want to genarate a adv with target label as inception_resnet_v2 predicted rather than a adv that can fool all the 4 models you gave in the model? if it is convient, hope to get you reply? thx


Support all of the Arduino installs - Now including ARM64

Now that Arduino has released official installations for Arm64 installs, it would be great of Robotis supplied packages for the board installs that run on these platforms.
That issue has come up recently on the Trossen Robotics forum: http://forums.trossenrobotics.com/showthread.php?16001-Open-CM9-04-Anyone-had-luck-programming-using-the-Arduno-IDE&p=95621#post95621
Likewise for OpenCR.

Hi, @KurtE
I will check about this issue.
Thank you.


Auto import elements to scope

Hi! I don't know if this is the right place to ask, but Is there a way to import automatically some element when it gives this error below? Something similar to quick fix in vscode.
cannot find derive macro `Component` in this scope
consider importing one of these items:
bevy::ecs::component::Component
crate::Component

Thank you!

I believe you can find the import in the code actions ( vim.lsp.buf.code_action() )

Nice! Thank you!


HotReload - provide a Sub!

Ah seems so simple! Provide a little Sub that just automatically grabs, serialises and stores the model. After all the model is available at the point of sub creation, what could go wrong?
Doesn't work though.
The problem is that if you make a Sub like HotReload.autosave(model, ...) then the model as it was at the point of sub registration (i.e. initialisation) is the only model the sub ever sees/saves. So unless you make a sub with a different id each time (based on hashcode maybe), then it only ever saves the initial model.
Oh well, nice idea. :smile:


Cherry pick PR:   extends path in init.py for SDK correctly (#1531)

PR is cherry picked from master

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
11 unchanged lines in 2 files lost coverage.
Overall coverage decreased (-0.2%) to 36.823%

Files with Coverage Reduction
New Missed Lines
%

pkg/controller.v1/mpi/mpijob_controller.go
2
75.35%

pkg/controller.v1/pytorch/pytorchjob_controller.go
9
61.65%

Totals

Change from base Build<PHONE_NUMBER>:
-0.2%

Covered Lines:
2272

Relevant Lines:
6170

💛  - Coveralls


Password Rotation support - Add a CredentialProvider interface/class and set your own provider

Hello,
Feature Request
Is your feature request related to a problem? Please describe
I'm using AWS SecretsManager to fetch my Redis credentials and provide them to Lettuce upon startup
However, in case I rotate the password (in Redis and in SecretsManager), I must restart my application.
Because, if the connection to Redis is lost, it will use the previous old password rather than the current one.
Describe the solution you'd like
Instead of providing a simple pair of username+password, I'd like to provide my own class, implementing a CredentialsProvider interface, and have Lettuce call it whenever a new connection is created.
This way, I will be able to always fetch the current password in my provider.
This interface just implements getUserName() / getPassword() .
Similar example for RabbitMQ:
https://github.com/rabbitmq/rabbitmq-java-client/blob/master/src/main/java/com/rabbitmq/client/impl/CredentialsProvider.java
Describe alternatives you've considered
I know Redis 6 with ACL can support multiple passwords.
However, the Lettuce client doesn't support that either as far as I know.
In addition, even if multiple passwords are supported by Lettuce - I will need to know ahead all the passwords for the near future.
This is not the most convenient and secured solution.
Thank you

Thanks for your feature request. Are you interested in submitting a pull request? While such a change can touch a lot of places, I think it makes absolutely sense to introduce a CredentialsProvider. I'm not sure we want refresh() methods in the actual interface. Representing username/password as Credentials object makes more sense in terms of a consistent username/password pair than individual calls to CredentialsProvider.getUsername/getPassword.

@mp911de , I think that refresh() is indeed not required. Only getUsername/getPassword should be enough.
Unfortunately, I'm not really familiar with the internals of Lettuce and I think such a change (as you wrote) can affect a lot of places.
Any chance someone else who is more familiar with Lettuce can take it?

@mp911de , since you self-assigned this, does it mean the task is on progress?
If so, I'd implement a DefaultCredentialsProvider (implementing interface CredentialsProvider) that contains the regular username/password pair as today.

since you self-assigned this, does it mean the task is on progress?

As per your request.

Any chance someone else who is more familiar with Lettuce can take it?

I haven't started yet and probably it will take another week or two until I get there.

Hi @mp911de ,
I don't seem to have the time and enough knowledge of Lettuce project to handle this at the moment.
Please take ownership back if possible.
Thanks.

@mp911de ,
Is this feature under development ?

No, it's not. I don't have the bandwidth for features right now. Likely will have more time at the end of this year.


[#24] simple summaries of inst data

Daily summaries (min/mean/max) of instantaneous data.
closes #24

Do we want to do any filtering or flagging of the instantaneous data that inform daily summaries? For example, at least x% of the unit values should be present to calculate a mean/min/max? This is often the reason that single days are missing from the reported daily values on NWIS, for example.

Do we want to do any filtering or flagging of the instantaneous data that inform daily summaries? For example, at least x% of the unit values should be present to calculate a mean/min/max? This is often the reason that single days are missing from the reported daily values on NWIS, for example.

Do you know the rule used by NWIS? It would be great to adopt the same rules

Do you know the rule used by NWIS? It would be great to adopt the same rules

It seems like the local user/data maintainer can use discretion based on site knowledge and streamflow conditions, but basic guidelines for SC are that at least 50% of the unit values are present to publish the daily min/max (for DO this is 25%). No specific guidelines are given for reporting the mean other than to say that it's a judgement call when less than 100% of the expected unit values are present. (My reference for these sort of questions has been this 2006 technical report on SOPs for continuous water quality monitoring.)
The number of daily expected unit values is not constant across sites or sometimes even within a site (e.g. a site might record for 10 min for 2 years, then switch to 15 min for 1 year, then switch to 5 min intervals for a period...). So to add this flagging step we'd need to edit the data download functions to retain NA values.

The number of daily expected unit values is not constant across sites or sometimes even within a site (e.g. a site might record for 10 min for 2 years, then switch to 15 min for 1 year, then switch to 5 min intervals for a period...). So to add this flagging step we'd need to edit the data download functions to retain NA values.

Could we work with the hourly aggregated timeseries instead of the raw data? Then we can say that we require 12/24 to report values. The max and min would correspond to max/min of hourly average SC, so a little different of an interpretation, but that may help to remove noise within the instantaneous max/min data, anyway.

@jds485, @lekoenig - I might have missed this in a previous conversation/decision - do we not care about daily minimums (just looking at means/maxs)? I ask b/c in the p1_daily_data we only have the Value and Value_Max.

@lekoenig and @jds485 - are you good with me merging this?

Alright, @jds485 and @lekoenig - I think I resolved all of the comments/questions. Let me know if you see anything else that I should do before merge.

Thanks for taking on the daily summaries, Jeff! The pipeline builds for me. I made a couple small formatting comments/suggestions, so you can incorporate them if they resonate with you, or leave as is. I'm not sure if @jds485has is planning on running the code, but I approve the PR and am happy to merge.

I'm not going to run the code for this PR. I reviewed the diff of changes and they look good to me


Trusted domain editor: Looks like text file, but isn't

Testing #80595

invoke Manage Trusted Domains
any changes to the file are lost, only the values are kept. e.g. my formatting, commented out domains.

To verifiers:
Verify that:

Manage Trusted Domains setting should preserve your commented out parts
When you have removed the header or end, for example by deleting the first line // Links matching one or more entries in the list below can be opened without link protection. or last line // Save this file to apply the trusted domains rules., the next time you run Manage Trusted Domains it'll be restored.


Unable to find field "field" on type "Type"

Describe the bug
There is an error
Unable to find field "fieldA" on type "AType"
query SomeQuery($input: SomeInput!) {
    some(input: $input) {
        a
        b {
            c {
                ...on AType {
                    d {
                        e
                        f {
                            g
                        }
                    }
                }
            }
            fieldA
            fieldB
        }
    }
}

If I change order, put fieldA and fieldB before c, error might disappear.
If before d there is another field the problem gone.
Pretty weird. I spent several days to investigate the problem :(
To Reproduce
Steps to reproduce the behavior:
https://github.com/ivansky/graphql-codegen-test-repo
Expected behavior

Success generating files.
Environment:

OS: MacOS
@graphql-codegen/...:  1.9.1
NodeJS: v13.5.0

Additional context
If I change fields order the error disappears. Magic parsing error probably.

Looks similar to #3217 . We'll take a look soon!

1.7.0 - works
1.8.0+ - doesn't work

@ivansky Did you try the latest version 1.10.0?

@ardatan Yes, even alpha from Fixes PR

It seems like the same issue in https://github.com/dotansimha/graphql-code-generator/issues/3217 . It's a bug with isUsingTypes that only effects developers who use preset feature.
I fixed it in: https://github.com/dotansimha/graphql-code-generator/pull/3243


Raise SSL_R_QUIC_PROTOCOL_ERROR on any QUIC protocol error

QUIC error code, frame type and reason is in error data
Fixes #21337

This pull request is ready to merge

Merged to master branch. Thank you for the reviews.


openliberty_microprofile_tck_0 fails with hamcrest compilation error

https://ci.adoptopenjdk.net/job/Test_openjdk11_j9_sanity.external_x86-64_linux_openliberty-mp-tck/390/consoleFull
02:14:12  Build projects and create images done
02:15:00  /open-liberty/dev/com.ibm.ws.componenttest/src/componenttest/matchers/Matchers.java:13: error: cannot find symbol
02:15:00  import static org.hamcrest.Matchers.containsString;
02:15:00                            ^
02:15:00    symbol:   class Matchers
02:15:00    location: package org.hamcrest

Dup of https://github.com/AdoptOpenJDK/openjdk-tests/issues/1857


Add API Method getRdf for retrieving RDF representation of structures

Summary
Request to add a getRdf API method to Ketcher for retrieving the RDF representation of chemical structures, with support for specifying V2000 and V3000 formats similar to how it is done in SDF
Solution
Introduce a getRdf method in the Ketcher API that allows users to:

Retrieve the RDF representation of the current structure on the canvas.
Specify the desired format (V2000 or V3000) for the RDF output, similar to how it's done in SDF export.

Need merge into 2.26

Tested.

Desktop:

OS: Windows 11
Browser Chrome
Version 128.0.6613.120 (Official Build) (64-bit)

Ketcher version
[Version 2.26.0-rc.6]
Indigo version
[Version 1.25.0-rc.4]


✨(frontend) add jitsi buttons for instructors

Purpose
We want to allow the instructors to use jitsi breakout rooms and to share a video.
Proposal
Add those buttons for instructors.

[x] Share a video
[x] Participants list (in which we can use breakout rooms)

It misses something, in DashboardVideoLive with the picture in picture we are overriding toolbarButtons (to try to remove the bottom bar when jitsi is in the picture, I'm not sure it is really working tho). And since we can not get current config in jitsiapi there is probably some refacto to do. Like, updating toolbarButtons to be a function

Good catch, thanks!
Let me know what you think of my new proposal.

Not a big fan but it does the trick

What is the problem ?

Not a big fan but it does the trick

What is the problem ?

I wouldn't say 'problem' but I don't find it really futur proof

After a test the breakout room feature doesn't work at all. The user enter in the room, leave it and rejoin the main room. I didn't investigate why.


Feature Request: Fuzzy Search

An innate feature to search through the Terminal text is quite limited and basic:

it doesn't highlight several entries in the window;
it doesn't support using regular expressions;
it can't filter output to a fewer number of lines (much like grep does).
it only supports strict match;

With the power of fuzzy search, users gain a boost to their productivity of being able to find a text through history very fast. It's implemented in many text editors (JetBrains IDEA, Visual Studio Code) for their project's files search. There's a solution specifically for the command line (bash/zsh) as well - junegunn/fzf.
If we could integrate the feature into the terminal (via its UI) without relying on the bash settings, it would be a great capability!

Thanks for the suggestion! There are certainly parts of your request that are already being tracked in #3920, but I'll use this issue to specifically track implementing "fuzzy search". Thanks!

Hmm, I'm not sure. Fuzzy matching would sort results by their closeness to the needle after all and so we would need to still implement a different searcher so that we can get the closeness and rank it. fzf uses a variant of the Smith–Waterman algorithm which we could also implement. I think something like that would be a requirement for this issue.
But #15858 still makes this a lot simpler. Since it presents a view into the TextBuffer as if it was a single contiguous string, while also providing facilities to map character positions back to row/column indices, we can efficiently and with only little effort implement such string search algorithms now. ICU offers utext based APIs like UTEXT_NEXT32 to return the next UTF-32 character, u_foldCase to turn it into a consistent casing and then we can just feed it into the Smith–Waterman algorithm. Alternatively there's also grapheme cluster iterators, etc.


add blog for Keycloak at KubeCon Chicago, Nov. 2023 announcement.

@ahus1 thank you. I think the changes improve the blog. Looks good to me.

@stianst - could please do the final review and merge it? I know that @abstractj already review the Google Docs draft. Thank!


How can I debug my node?

I have written some code for a node. Is there any way that I can easily try it out by sending some locally stored dicom files and see if there are any errors?

You can use DCM-tk to send dimse messages
storescu --scan-directories -nh -xs --recurse -aec $YourAETitle $localhost $port $path/to/dicoms
If an exception is raised while processing, the traceback is stored in pynetdicom logger and not the dicomnode logger. Therefore it can be helpful to enable the logger with AbstractPipeline attribute: disable_pynetdicom_logger=False

Sorry to comment on a closed issue, but is it possible to get a minimal example of how to send files using storescu? I have the following AbstractInput:
class MRInput(AbstractInput):     def validate(self):         return True
which should accept any dicom dataset if I understand correctly. However, I get "2023/04/20 20:36:48 dicomnode INFO Received dataset is not accepted by any inputs" response from the server when I sent the directory with dicom files. I also tried with the PlusOneNode (erasing requirements for tags), and the results are the same.

#4 I believe this PR fixes it :)


fix: avoid create proxieds for watch only

use two arrays for accounts to add proxy and proxied accounts

Task linked: CU-8693j0p5y Proxy. Proxied account should not be added for WOW


Add coverage recursive

Codecov Report

:exclamation: No coverage uploaded for pull request base (main@ffe985c). Click here to learn what that means.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main       #2   +/-   ##
=======================================
  Coverage        ?   36.17%
=======================================
  Files           ?        2
  Lines           ?       47
  Branches        ?        0
=======================================
  Hits            ?       17
  Misses          ?       30
  Partials        ?        0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ffe985c...6e48bb4. Read the comment docs.


Compiling a crytsal for interpreter

I have no idea if this is the right repo for this bug report, or if it should go at https://github.com/crystal-lang/distribution-scripts Perhaps someone here can pull back the curtain enough for me to either solve the problem or at least find the right place to go for better help.
I'm on an M* mac, and I'm curious to try out the interpreter.  So I pulled down the repo and make interpreter=1 to spin it off.
It fails, and this is the error message it emits:
ld: warning: ignoring file /Users/robert/.asdf/installs/crystal/1.6.2/embedded/lib/libffi.a, building for macOS-arm64 but attempting to link with file built for macOS-x86_64

Crystal is using the currently available crystal to compile itself -- the currently available crystal was installed with asdf-crystal, and it seems like the bundled libffi.a file is the wrong architecture for the M-series arm chip. What should my next step be?

cf. https://github.com/crystal-lang/crystal/issues/9285 the install script could use the bundled tarball (https://github.com/crystal-lang/crystal/issues/9285#issuecomment-1124534438) to have the static libraries.


Expecting value: line 1 column 1 (char 0)

What is the reason for this issue that pops up after opening a webpage and inputting images for local deployment

Can you send me the screen where the pop-up occurred?


Travis

Added travis and most of sonatype deployment; still waiting for the credentials

This is blocked on releasing api-build-resources to Sonatype. There will be a similar pr for that shortly; however, we are still waiting for the credentials to Sonatype (so it will also be incomplete).


add R.sequence and R.traverse

This pull request deprecates R.commute and R.commuteMap in favour of R.sequence and R.traverse respectively.
Reasons to rename the functions:

My understanding is that sequence and traverse are the most common names for these functions. I read and reread Understanding traverse and sequence in order to come to terms with this concept, so I may be biased by the names used there. Is commute the name used anywhere but in Ramda?

Fantasy Land defines Traversable, which requires that a Traversable provide a sequence method. It also shows that traverse can be derived from map and sequence.

Currently each Ramda functions which dispatches has the same name as the method to which it dispatches. Having R.commute dispatch to sequence would break this convention.

Changes:

R.sequence dispatches; R.commute does not.

R.traverse takes (of, f, traversable) rather than (f, of, traversable). It's more likely one would apply several different f transformations to a single :twisted_rightwards_arrows: type inversion than apply the same transformation to several different type inversions. I also like having of off to the side as it conveys type information that would be unnecessary in the presence of type inference.

I'm particularly interested in hearing from @scott-christopher, who has a better grasp of this than I do. We have discussed these changes (though Scott is happy to keep the names) but I'm not sure that I've done everything correctly.
I'm about to :airplane: from Auckland to San Francisco, so I'll be offline for the next 12 hours or so.

R.sequence(R.of, []) evaluates to [], whereas R.commute(R.of, []) evaluates to [[]]

I don't think this is correct. Going down this path would also require the applicative instance to have a Monoid instance for empty. Calling sequence on an empty traversable should result in a pure instance of the applicative type containing the empty traversable.
e.g.
sequence(R.of, []) //=> [[]]
sequence(Maybe.of, []) //=> Just ([])
sequence(Maybe.of, Nothing) //=> Just (Nothing)

There are a few laws that should confirm this.
For example:
var identityL = compose(sequence(Identity.of), map(Identity));
var identityR = Identity;

identityL([]) == identityR([]); //=> false

identityL([]); //=> []
identityR([]); //=> Identity([])

I'm happy with this PR pending the comments above. The name change doesn't bother me too much, providing the behaviour doesn't drift too much in the process.

I don't have any objections to the name change, so long as we understand the reasons (#441) why we chose not to use traverse in the first place, and are certain they no longer hold.
I will have to read through that article or something similar to get enough of a feel to comment on the content and the fundamentals of the API change.  Maybe I'll have time tomorrow.

R.sequence(R.of, []) evaluates to [], whereas R.commute(R.of, []) evaluates to [[]].

I don't think this is correct. Going down this path would also require the applicative instance to have a Monoid instance for empty. Calling sequence on an empty traversable should result in a pure instance of the applicative type containing the empty traversable.
e.g.
sequence(R.of, []) //=> [[]]
sequence(Maybe.of, []) //=> Just ([])
sequence(Maybe.of, Nothing) //=> Just (Nothing)

I've removed the special case for [], but I'm still uncertain. Can you explain the following?
> R.sequence(R.of, [])
[[]]
> R.sequence(R.of, [[]])
[]

Haskell and R.commute/R.sequence appear not to agree:
Prelude Data.Traversable> sequenceA []
[]
Prelude Data.Traversable> sequenceA [[]]
[]
Prelude Data.Traversable> sequenceA Nothing
Nothing

/cc @DrBoolean

I'm assuming it's something funky going on with GHCi and IO there.
Prelude> let a = [] :: [IO a]
Prelude> sequenceA a
[]

If you explicitly declare their types they behave as I would expect.
Prelude> let b = [] :: [[a]]
Prelude> sequenceA b
[[]]
Prelude> let c = [] :: [Maybe a]
Prelude> sequenceA c
Just []
Prelude> let d = Nothing :: Maybe [a]
Prelude> sequenceA d
[Nothing]

I'm assuming it's something funky going on with GHCi and IO there.

Oh! This would never have occurred to me! I'm pleased Haskell agrees with us after all. :)
Do you have an explanation as to why [] becomes [[]] but [[]] becomes []? I'm now convinced there is an explanation; I'd love to learn of it.

Probably the best intuition I could try to offer (it still isn't entirely intuitive to me) is that if the inner applicative exists, the result will be based on how that applicative behaves with ap.
> sequence(of, [[1, 2], [3, 4]]);
[ [ 1, 3 ], [ 1, 4 ], [ 2, 3 ], [ 2, 4 ] ]

// compare to

> var a = R.ap(R.map(R.prepend, [3, 4]), [[]])
[ [ 3 ], [ 4 ] ]
> R.ap(R.map(R.prepend, [1, 2]), a)
[ [ 1, 3 ], [ 1, 4 ], [ 2, 3 ], [ 2, 4 ] ]

> sequence(of, [[1, 2], []]);
[]

// compare to

> R.ap(R.map(R.prepend, []), [[]])
[]

> sequence(of, [[], [3, 4]]);
[]

// compare to

> R.ap(R.map(R.prepend, []), [[3], [4]])
[]

it still isn't entirely intuitive to me

I'm pleased to hear this, in a way. :)
Perhaps this is a concept that reveals itself over time rather than being learnable in one sitting.
Have you any thoughts, @CrossEye, on dispatching to sequence or on taking of before f?

I will try to look at it this evening.  I'm not comfortable enough with any of this yet to comment.

:-1: on the name traverse for this. would be nice to reserve traverse for, you know, tree traversal et al.

:-1: on the name traverse for this. would be nice to reserve traverse for […] tree traversal et al.

If we use the name at all we should give it to the traverse function described in the Fantasy Land spec. One option is to provide R.sequence only. Those who want the map + sequence behaviour can write:
R.sequence(of, R.map(f, traversable))

I see it as the same issue as always: common names across FP libraries or more expressive names in general. I really like the name commute, with a possible alias of sequence for fantasyland compliance.

@DrBoolean: Unfortunately, or more likely, fortunately, Ramda has forsworn aliases altogether.

We could keep the name commute (which I also like) but have R.commute dispatch to sequence.

I'm anti-alias anyways :)

I'm anti-alias anyways :)

that explains why you're so smooth :stuck_out_tongue:

zing!

#1367 might be coming back to life, which could mean all sorts of extensions to our delegation techniques.  But I would rather not by default delegate to a different name, even if that turns out to be a part of an integration mechanism.
I originally raised an objection to "Traverable" in #441, saying much as @buzzdecafe does here that it really should be reserved for something else.  But I don't see us actually including that something else, and this is now part of FantasyLand, so I'm all for including it here.  :herb:
I still haven't read that article, and once again, won't have time this evening.  But at first glance, this implementation looks sensible to me.

Maybe we propose fantasyland update to commute and drop the mention of traverse. I'm down to do that.

Maybe we propose fantasyland update to commute and drop the mention of traverse. I'm down to do that.

Would the type class still be named Traversable or would we change it to Commutable at the same time? Perhaps such a discussion should happen in fantasy land. Fancy raising the issue over there, Brian?

Certainly

https://github.com/fantasyland/fantasy-land/issues/115

Thanks, DrB!

:-1:  on the name traverse for this. would be nice to reserve traverse for, you know, tree traversal et al.

What is a list if not a degenerate tree? All the branches are on the right.
[1,2,3] == 1:[2,3] == 1:2:[3] == 1:2:3:[] ==   :
                                              / \
                                             1   :
                                                / \
                                               2   :
                                                  / \
                                                 3  []

Looks like a tree to me. :)

@joneshf, are you in favour of this pull request as it stands?

What is a list if not a degenerate tree?

@joneshf i can't argue with that. since fantasyland/fantasy-land#115 was closed, i think it makes sense to go with the prevailing wind in this case, IOW :cow2:

And I'll say again, :herb:.  My original arguments are not relevant any more, especially as this version does delegate to the type's versions of traverse / sequence.


validation-test: refactor the test invocation (NFC)

This just uses the %target-build-swift-dylib instead of
%target-build-swift substitution along with %target-library-name to
ensure that we are target independent.

Replace this paragraph with a description of your changes and rationale. Provide links to external references/discussions if appropriate.

Resolves SR-NNNN.

CC: @jrose-apple @slavapestov @mikeash

@swift-ci please smoke test

@swift-ci please smoke test Linux platform


⚠ Samsung has degraded performance

In be7f718, Samsung (https://samsung.com) experienced degraded performance:

HTTP code: 200
Response time: 2722 ms

Resolved: Samsung performance has improved in 48185d7.


messageInfo.customizedMessageBody

Hello,
Seems if messageInfo.customizedMessageBody is in another language, like French, then encoding not really working..
Maybe good to add info about supported language for this?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 488fdbc7-ca6a-0356-35a2-c3de26392fc4
Version Independent ID: 0bc17388-55fa-2f0b-46bd-5c33620c68f7
Content: B2B collaboration code and PowerShell samples - Azure AD
Content Source: articles/active-directory/b2b/code-samples.md
Service: active-directory
Sub-service: b2b
GitHub Login: @msmimart
Microsoft Alias: mimart

@lightupdifire
Thanks for your feedback! We will investigate and update as appropriate.

@lightupdifire  thanks for your feedback, if you're interested in this feature being added please submit it against the feedback forums here : https://feedback.azure.com/forums/169401-azure-active-directory?query=b2b localized info and if there's enough community support the product team will look into implementing it accordingly.
That being said as this has been addressed within the scope of the github issue I'll be closing out this issue by EOD tomorrow, please let us know if you have anymore concerns in regards to this.

Thanks, submitted.


pre-commit: add basic hooks

Signed-off-by: Maciej Pijanowski<EMAIL_ADDRESS>

@pietrushnic There are no problems with these basic hooks, so I will merge it, so they can already be used.


How to obtain Root certificate?

I'm trying to run the esp8266 mqtt sample but  cannot get my device to connect. I assume this is because I did not change the ca.crt file that is in the data folder before uploading it to my microcontroller. Which root crt should I be using?

@gguuss  Thanks for your reply. I never used TLS handshakre in miccontroller before. I re-checked the MD5SUM from the fresh one and the file in  Esp8266-lwmqtt. They had different checksum value.
$ md5sum ca.crt
497904b0eb8719ac47b0bc11519b74d0  ca.crt
$ md5sum ca.crt
821aefd4d24af29fe23d970614707285  ca.crt

Does that mean we must replace the ca.crt periodically ?  and better kept in our program to check ?

The right thing to do would be to use an over-the-air mechanism for securely updating the certificate if it expires or changes using a platform-specific solution for your device. This degree of functionality isn't currently planned for this library.
I haven't run the example for a little while, I'll see if it works for me with the library cert / GH master cert.

I just verified the code is working for me using the older cert as well as the one I just generated for the GitHub master version.
A few troubleshooting suggestions:

Open the serial monitor while running (Ctrl+shift+m) Are you seeing an error code?
Double-check that you have added the public EC certificate for your device to the device registry, that you are using the correct device identifier, project id, and that your cloud region is correct. Basically any incorrect value in the ciotc_config.h file will result in the device not connecting.
Verify the JWT is valid using a tool like jwt.io

Just out of curiosity, which 8266 board are you using? I'm verifying on an Adafruit Huzzah board.

@gguuss  after recheck,  The problem look like came from the ca.crt not upload successful from Arduino IDE.  I use 2.5.0-beta3 vesion. Arduino IDE SPIFFS upload tool just report upload successful. but the error code -10002 still shown.
then turn to use SPIFFS library and  download the ca.crt from http and write to filesystem, That;s work. The SPIFFS could find and open ca.crt. I could not make sure the problem came from the too old IDE plugin or not. Because i also could not use the up-to-date MKSPIFFS to make BIN image and upload.
the looping look like came from the lack of ca.crt during handshake with Google IoT Core. This maybe borard-specific / Arduino-Core version issue. Need more check and test.

Thanks for the help. I've tried with the old and new cert, with the http and mqtt sample, and still can't connect. Ine the serial output I'm seeing a 0 error with mqtt, and a "connection failed" message with http. I double checked project name, secret key, pubsub subscription, device name, device public key. I'm really not sure what I'm doing wrong, but at this point it looks like it has nothing to do with the cert.

@alexdufetel  Did you try the debug level with TLS/SSL/HTTP Client Level in the board manager ? Maybe the result could show more detail information ? In the begging, my test  show the ca.crt load failed using the Arduino IDE upload tool plugin.

@gguuss both the mqtt and http example worked after re-create key-pairs and uploaded.  even the old keys was checked using your suggestion in the jwt.io tool. but not accept by Google IoT Core. Not sure why
the modified code -

the NTP to the local one and Google NTP for more quicker response.  maybe could relocate to the ciotc_config.h ? in some place, NTP server response
LED_BUILTIN was different related to ESP-modue/board, and that was selected in the board-config tool. if someone could not make sure. just remove that's part. Incorrect hardware configuration will make code halt there.

void getConfig() {
  // TODO(class): Move to library
  String header =
      String("GET ") + device->getLastConfigPath().c_str() + String(" HTTP/1.1");
  String authstring = "authorization: Bearer " + String(jwt.c_str());

  client.setInsecure();  // To check if ca.crt issue or not

  if (!client.connect(host, httpsPort)) {
    Serial.println("connection failed");
    return;
  } else {
    Serial.print("connection success, TCP Status: ");
    Serial.println(client.status());
  }

  // Connect via https.
  client.println(header.c_str());
  client.println(authstring.c_str());
  client.println("host: cloudiotdevice.googleapis.com");
  client.println("method: get");
  client.println("cache-control: no-cache");
  client.println();

  while (client.connected()) {
    String line = client.readStringUntil('\n');
    Serial.println(line);  // the read the https handshake status - if authorization problems easy to check
    if (line == "\r") {
      Serial.println("headers received");
      break;
    }
  }
  while (client.available()) {
    String line = client.readStringUntil('\n');
    if (line.indexOf("binaryData") > 0) {
      String val = line.substring(line.indexOf(": ") + 3, line.indexOf("\","));
      Serial.println(val);
      if (val == "MQ==") {
        Serial.println("LED ON");
        //digitalWrite(LED_BUILTIN, HIGH);
      } else {
        Serial.println("LED OFF");
        //digitalWrite(LED_BUILTIN, LOW);
      }
      resetBackoff();
    }
  }
  client.stop();
}

client.setInsecure();  // To check if ca.crt issue or not
Serial.println(line);  // printout the https handshake status - if authorization problems easy to check
this two step might make the debug more easily.  if Google IoT Core configuration issue, the HTTP response code and message will provide the information to debug.

@alexdufetel  Do you get any JWT String shown in the serial output??  like
eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1NDkzNzgyNjUsImV4cCI6MTU0OTM4MTg2NSwiYXVkIjoiYXBpLXByb2plY3QtNDUxNzA5MjIyNjAwIn0=.4hl6Gyz+aGQWuix0eWzhlMsq4vkJv6tt19bWarKVMSV32uhFLc7AeUeFCfdZ1lRBswLAsR+lyIEVKj5K5jXYSg==
Also you could try to add some code to check https response to check the handshake procedure.
in the getConfig() section, add one line in the while (client.connected()) { ... } section, like
'Serial.println(line);'
if any server-side configuration issue, this will give some http reponse code to check.

@bertrandlo Thanks for your reply. I do get a JWT, but then I can't establish the connection. Adding a debug line in client.connected() does not help unfortunately, because it does not get called. Would that mean my JWT is invalid?
When I paste my JWT to jwt.io, I get a "Invalid Signature" message. selected algorithm is ES256, and I pasted my public key in the "verify token" section
My JWT:
eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1NDk0Njk0MjcsImV4cCI6MTU0OTQ3MzAyNywiYXVkIjoiaW90LWRvZ2Zvb2QifQ==.w4h2EolYpqQcuPlO/MLovznI3El83TG+KlT0Lpu3C9/bEQuH1vLbHkqBMTwqpt27KO9GKkvFweYd0MUU32evdA==
My public key:
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAERusopVBMclQdyu7SjXIBwme8hmRB
pB7X41kQNkK0Uu5te0wULHhR9eKCPYQZD5WXqt5aWuRcwh3LijeJ7XwJqQ==
-----END PUBLIC KEY-----
My Serial output:
Waiting on time sync...
Refreshing JWT
eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1NDk0Njk0MjcsImV4cCI6MTU0OTQ3MzAyNywiYXVkIjoiaW90LWRvZ2Zvb2QifQ==.w4h2EolYpqQcuPlO/MLovznI3El83TG+KlT0Lpu3C9/bEQuH1vLbHkqBMTwqpt27KO9GKkvFweYd0MUU32evdA==
Success to open ca file
loaded
Waiting: 5117
pm open,type:2 0
connection failed
connection failed

also, for FYI : the device I'm using:
https://www.amazon.com/HiLetgo-Internet-Development-Wireless-Micropython/dp/B010N1SPRK/ref=sr_1_3?ie=UTF8&qid=1549470092&sr=8-3&keywords=esp8266+nodemcu

@bertrandlo /FYI - adds definitions for generic board support

@alexdufetel  Happy Chinese New Year~  ! ^^
that Jwt.io result should verify your jwt token, you also have to past the private key to make sure the jwt work well. could you try that again?  or regenerate a new pair.  your last two messages "connection failed" it look like the error of JWT

Don't know if this is relevant to your issues, but we also just ran into an MQTT connect issue (on the IBM Watson IoT Platform) with the esp8266 version 2.5.0 board support.  Reverting to 2.4.2 fixed the problem.  I'm using a Wemos D1 Mini.

@alexdufetel Let us know if the tip from @johnhmacleod helps or if trying alternative hardware fixes your issue. If it was related, we can update the readme and close this issue.

Don't know if this is relevant to your issues, but we also just ran into an MQTT connect issue (on the IBM Watson IoT Platform) with the esp8266 version 2.5.0 board support. Reverting to 2.4.2 fixed the problem. I'm using a Wemos D1 Mini.

I'm in the same conditions: fail with 2.5.0 (with lastError=0,returnCode=0), no problem with 2.4.2 (on Wemos D1 Mini Lite)

I'll add a note to the readme.  Maybe open a new issue for determining working versions of the Espressif Community SDK.

Enable debug mode when compiling and see datailed reason of failed handshake

@Gior80 Is this a configuration within the Arduino IDE?
E.g. Tools > Core Debug Level > Verbose

Yes exactly ,
This way u can get what’s happening at protocol level . I resolved some issues in this way looking what’s passing inside ssl handshaking

Just to be clear on usage , just activate it and open serial monitor while running . All message gonna come out

HiGus, I am also trying to connect my ESP8266 with GCP. From serial monitor output, I figured out that I am able to get the JWT string and ca file also successfully loaded. Globals created for MQTT. MQTT started. And then checking wifi
Connecting.... <IP_ADDRESS>.0.......
Series of zeros. What might be the issue?
Kindly help me with this

I'm not sure, have you tried increasing the debugging settings / verbosity as described in the README - this may provide a more clear bug report.
Please include the version of the library you're using, the version of the Espressif SDK, and the type of ESP developer board.

Adding this issue for tracking these types of issue

Hi,
I'm trying this project on MKR1000 but before this I have to update the firmware. It seems I have to use the Firmware updater described on https://www.arduino.cc/en/Tutorial/FirmwareUpdater to update the certificate.  What address should I add exactly in the Add domains text box? google.com:443 ?

@jsmith173 mqtt.googleapis.com:443 or mqtt.googleapis.com:8883

I think better when you use the FirmwareUpdater for MKR1000 (and not the hackster.io tool) to add the certificates
https://www.arduino.cc/en/Tutorial/FirmwareUpdater
When you'd have getDefaultSensorJSON (publish data as JSON), you can visualise the published data on Google Sheets (using dataflow).
I think a bit unusual that the topic must be specifed with /  (for example "/my-topic"). When I not add the / , the program restarts many times (multiple wifi and mqtt reconnects)
My final question: how to receive incoming telemetry data for a specified topic? (I'd read back the data published by publishTelemetry)

@alexdufetel
alex,
we are having the exact same issue as you describe (invalid signature), that ends with "=="
jwt.io shows an invalid signature.
also, ES256 signature actually should be 86 characters,
any ideas?

Sen kullanmak ESP8266 Veri Yükleme programı CERT yüklemek için.

Hi please share for esp32


How to bind text to the RichTextbox from view model.

I would like to bind the string(Combination of emoji  and some string) as a text from view model to richtext box. Is there any way to bind it.
Also, how to get entered text from richtextbox

Hi.
No. But...
The RichTextBox uses a "FlowDocument" instead of a "pure" Text. (Strings)
There is a way to Bind to a  "extended" RichTextBox-Control in the "Extended WPF Toolkit".
They already solved the "bindability" to a String-Property.
If this is not for You, you can extend a RichTexBox with a DependencyProperty.
I hope it helps...
c.u.

@sahithi89 in the latest release you can bind Text on the RichTextBox. See the new version of the sample application:
            <emoji:RichTextBox FontSize="20" x:Name="EmojiRichTextBox"/>

            <emoji:TextBlock FontSize="12" Text="{Binding Text, ElementName=EmojiRichTextBox, Mode=TwoWay}"/>


Run npm

Npm start

I believe this issue should be closed as it lacks context or information. Can a project maintainer or administrator please close this issue?


job.status and job.log returns error for certain domain name pattern

The job.status() and job.logs() Are failing, they're showing
gaierror                                  Traceback (most recent call last)
File /opt/app-root/lib64/python3.8/site-packages/urllib3/connection.py:174, in HTTPConnection._new_conn(self)
    173 try:
--> 174     conn = connection.create_connection(
    175         (self._dns_host, self.port), self.timeout, **extra_kw
    176     )
    178 except SocketTimeout:
....
ConnectionError: Failed to connect to Ray at address: http://ray/.

Cause:
In torchx The app_handle is a string. On my cluster it looks like
ray://torchx/ray-dashboard-mnisttest-default.tedchang-codeflare-test-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.us-south.containers.appdomain.cloud-mnisttest-trwmgjb2s041jc
This method uses a regx that not parse the app_id(mnisttest-trwmgjb2s041jc) and ray dashboard(ray-dashboard-mnisttest-default.tedchang-codeflare-test-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.us-south.containers.appdomain.cloud) property
":\d+-|.com-|.org-|.net-|.gov-|.io-"
/opt/app-root/lib64/python3.8/site-packages/torchx/schedulers/ray_scheduler.py
330         def _parse_app_id(self, app_id: str) -> Tuple[str, str]:
331             # find index of '-' in the first :\d+-
332             m = re.search(r":\d+-|.com-|.org-|.net-|.gov-|.io-" , app_id)
333             if m:
334                 sep = m.span()[1]
335                 addr = app_id[: sep-1]
336                 app_id = app_id[sep:]
337                 return addr, app_id
338 ~~~~~~~~~~~~~~~~
339             addr, _, app_id = app_id.partition("-")
340             return addr, app_id

something like this worked for my cluster but may not be robust
":\d+-|.com-|.org-|.net-|.gov-|.io-|.cloud-"

@KPostOffice @jbusche @MichaelClifford Our _parse_app_id looks different from the one in the [upstream] (https://github.com/pytorch/torchx/blob/d452a8ccf6341bb854f8fab93a866560dc2f652c/torchx/schedulers/ray_scheduler.py#L330-L337). Is it a fork which we can make it work for now ?

The one upstream was more broken than the one in our custom version. We can definitely make changes, I'm not sure how to make it more generic exactly. Maybe something including an reverse parse for . since I think the name cannot have periods in it, so the from last period to the - must indicate the top level domain.

@KPostOffice Reverse parsing sounds like a great idea!. Where is the custom torchx repo located ? I will see if can create a PR there.

Thanks for pointing this out @tedhtchang. I recall we ran into this issue before, and we could not determine a regex that provide stable results, so we added  a list of reasonable suffixes.
Likely worth revisiting to see if we can implement a general solution.
In the mean time a quick fix would be just to add .cloud- to our fork of torchx (https://github.com/project-codeflare/torchx/tree/OCP)


Add SpEL support to RU field in @Container annotation

This change add Spring Expression Language (SpEL) support to the RU field of the @Container field.

@Container(containerName= "\${azure.cosmosdb.container-name}", ru="\${azure.cosmosdb.ru}")
data class Timeshift()

Then we can configure the ru value via application.yaml to make it easily configurable via environment variables.
azure:
  cosmosdb:
    ru: ${COSMOS_DB_CONTAINER_RU:-400}

This is a nice feature since we can the provisioned throughput without changing the code.

Thanks for the feature @Duske. Can you please add some test cases verifying this scenario and then update the readme with details on this feature ?

@Duske  - were you able to add any test cases for this?
Worth covering all the scenarios for testing for this new feature.


Correct the style of brackets in spark array function document

Description
As discussion here, we need to correct the style of brackets. For example
SELECT array_remove(ARRAY (2, 1, NULL), 1); -- [2, NULL]

cc @rui-mo


Stop building/publishing heroku/buildpacks:20 and heroku/builder-classic:22

Since these images were:

superseded by the heroku/builder:* images in June 2022
officially deprecated in Oct 2023 (#429)
made to error by default in March 2024 (#474)

Builds can still continue using the last version of these builders published to Docker Hub, and the resultant app images can still be rebased onto newer run images if needed.
I've also cleaned up the available images table in the README, to make it display a little better. (There was too much content before, causing the table to wrap awkwardly when viewed in the narrow GitHub repo-root view.)
Closes #512.

Before:
https://github.com/heroku/cnb-builder-images/blob/d74b3df909399e28e5ab39fad894ee86d3830a83/README.md
After:
https://github.com/heroku/cnb-builder-images/blob/c3f980b53ee913661a4325328735972660839b0a/README.md

I've also opened https://github.com/heroku/languages-github-actions/pull/220 so the CNB release automation doesn't try to update manifest files that will no longer exist in this repo once this PR merges.