Build pages from data source

Currently Hugo handles internal and external data source with getJSON/getCSV which is great for using  data source in a template.
But Hugo cannot, using a data set of items, build a page for each of them plus related list pages like it does from the content directory files.
Here is a fresh start on specing this important step in the roadmap.
As a user, I can only see the configuration aspect of the task.
I don’t see many configuration tied issues except for the mapping of the key/values collected from the data source and the obvious external or internal endpoint of the data set. The following are suggestions regarding how those configurations could be managed by the users followed by a code block example.
Endpoint/URL/Local file
Depending on use cases there may be a need of one or several url/path.
For many projects, not every page types (post, page etc…) may be built from the same source. Type could be defined from a data source key or as a source parameter.
I suppose there could be other parameters per sources.
Front Matter mapping
User must be able to map the keys from the data source to Hugo’s commonly used Front Matter Variables (title, permalink, slug, taxonomies,  etc…).
Every keys not referenced in the map configuration could be stored as is a user-define Front Matter available in the .Params object but this should not be default as there maybe way too many.
Example:
This is a realtor agency, a branch of a bigger one.
Their pages are built with hugo's local markdown
They have an old wordpress site whose > 100 blog posts they did not want to convert to markdown. So they load those blog posts from a local data file on top of the local Hugo's own markdown posts.
They use a tier service to create job posts when they need to fill a new position. They want to hosts those job listing on their site though. Their jobs are served by https://ourjobs.com/api/client/george-and-son/jobs.json
The most important part of the website are their realty listings. They add their listing to their mother company's own website whose API in turn serves those at https://api.mtl-realtors/listings/?branch=george-and-son&status=available
Configuration
title: George and Son (A MTL Realtors Agency)

dataSources:
  - source: data/old_site_posts.json
    contentPath: blog
    mapping:
      Title: post_title
      Date: post_date
      Type: post_type
      Content: post_content
      Params.location.city: post_meta.city
      Params.location.country: post_meta.country

  - source: https://ourjobs.com/api/client/george-and-son/jobs.json
    contentPath: jobs
    mapping:
      Title: job_label
      Content: job_description

  - source: https://api.mtl-realtors/listings/?branch=george-and-son&status=available
    contentPath: listings/:Type/
    grabAllFrontMatter: true
    mapping:
      Type: amenity_kind
      Title: name
      Content: description
      Params.neighbourhood: geo.neighbour
      Params.city: geo.city

Content structure
This results in a content "shadow" structure. Hard lines dir/files are local, while dashed ones are remote.
content
├── _index.md
├── about.md
├── contact.md
├── blog
│     ├─── happy-halloween.md
│     ├─── merry-christmas.md
│     ├- - nice-summer
│     └- - hello-world
├- -listings
│     ├- - appartment
│     │   ├- - opportunity-studio
│     │   ├- - mile-end-condo
│     │   └- - downtown-tower-1
│     └- - house
│         └- - cottage-green
└- - jobs
      ├- - marketing-director
      └- - accountant-internship

Thanks for starting this discussion. I suspect we have to go some rounds on this to get to where we want.
Yes, we need field mapping. But when I thought about this problem, I imagined something more than a 1:1 mapping between an article with a permalink and some content in Hugo. I have thought about it as content adapters. I think it even helps to think of the current filesystem as a filesystem Hugo content adapter.
So, if this is how it looks like on disk:
content
├── _index.md
├── blog
│   └── first-post
│       ├── index.md
│       └── sunset.jpg
└── logo.png

What would the above look like if the data source was JSON or XML? Or even WordPress?
It should, of course, be possible to set the URL "per post" (like it is in content files), but it should also be possible to be part of the content section tree with permalink config per section etc. So, when you have 1 content dir + some other data sources, it ends up as one view.

As most data sources are usually a flat list of items, I suppose building the content directory structure will require some more mapping.
There are the type and section keys to be used as well as maybe others which would help position the item in the content structure.
There is also be urlsource parameter designed the same way as the global config one except it would take one of the mapped key's as pattern (I'll update my example after this):
url: /:Section/:Title/
I suppose there is no way around having many source configuration params/mapping which Hugo may need to best adapt the data source the desired structure. Maybe even having to use some pattern/regex/glob to best adapt those like the url suggestion above.
As for default structure. If there is no configured data source with a type parameter of blog, then Hugo will build it from content, the rest would be build from data source (supposing we a Page Bundle toggle, media mapping). See this real content merged with data source "phantom" structure:
content
├── _index.md
├── blog
│   └── first-post
│       ├── index.md
│       └── sunset.jpg
├ - - - recipe  (from data source)
│        └- - - first-recipe
│               ├ - - - index
│               └ - - - cake-frosting.jpg
└── logo.png

@bep now I understand more fully what you meant (I think). The config needs to tell Hugo how to model the content structure so it can build its pages from that.
In a sense we are not building pages from data source we are building a content structure from both local content and remote data source.
To reflex this I added to the desc a better project example to illustrate both configuration possibilities and resulting "content" structure.
This is a project we can add to in order to maybe better spec what this feature should achieve.

@regisphilibert I have been thinking about this, and I think the challenge with all of this isn't the mapping (we can experiment until we get a "working and good looking scheme"), but more the practical workflow -- esp. how to handle state/updates.

As an editor, I would love it if my site (including content) was as static as possible at commit time (v1.3.0 of Hugo Times is this).
That is, if I the editor, looked at the Netlify preview on GitHub and push merge, I would be sadly disappointed if I then ended up with something completely different.
I think this is an often overlooked quality of static sites: Versioned content.

I understand that in a dynamic world with JS APIs etc., the above will not be entirely true, always. But it should be a core requirement whenever possible.
A person in another thread mentioned GatsbyJS's create-source-plugin.
I don't think their approach of emulating the file system is a good match for Hugo, but I'm more curious about how they pull in data.

Ensure local data is synced with its source and 100% accurate. If your source allows you to add an updatedSince query (or something similar) you can store the last time you fetched data using setPluginStatus.

This is me guessing a little, but if I commit my GatsbyJS with some create-source-plugin sources to GitHub and build on Netlify, those sources will be pulled completely on every build (which I guess also is sub-optimal in the performance department). I suspect setPluginStatus is a local thing and the updatedSince is a way to speed up local development.
Given the above assumptions, the Gatsby approach does not meet the "static content" criteria above. I'm not sure how they can assure that the data is "100% accurate", but the important part here is that you have no way of knowing if the source has changed.
So, I was thinkering about:

Adding a sgllite3 database as a "build cache"
Adding a "prepare step" (somehow) that exports the non-file content sources out into a merge-friendly text format (i.e. consistent ordering etc.)

The output of 2) is what we use to build the final site.
There are probably some practical holes in the above. But there are several upsides. sqllite3 has some very interesting features (which could enable more cool stuff), so if you would want to make that the "master", you could probably edit your data directly in the DB, and you could probably drop the "flat file format" and put your DB into source control ... This is me thinking out loud a little.

That is, if I the editor, looked at the Netlify preview on GitHub and push merge, I would be sadly disappointed if I then ended up with something completely different

I'm not sure about this. And I already apologize if some of my lack of understanding of the technology/feature at hands bias my view.
I guess most of the use cases for this will be using contentful or WordPress Rest API or FireBase to manage your content, and let Hugo build the site from this remote source plus maybe a few other ones (remote and local).
In this use case, the editor will not see markdown and probably not the Netlify preview or that merge button, but only the contentful or WordPress dashboard and create/edit their content from there.
When a new page is published out of the draft zone, the editor will expect it to be visible on the site with few regard to the repo status. On bigger sites where several editors work at the same time, Hugo's built speed will help making sure the website can be "refreshed" often in order to keep up with content editing.
But this does not change the fact that we need caching and being able to tell the difference between the cached source and the remote one efficiently.
In order to handle the "when", by this I mean the decision between calling the remote source or using the cached one, I was thinking about a setting per source indicating at which rate it should be checked.
If setting is one hour, then Hugo would check cached source time and if older than one hour, call remote. It would then use and cache the remote source only if it differs from the cached one. (Maybe using a hash to compare cached vs remote ?)
I'm not sure I understand the process described with sqlite3, would this mean having a database inside Hugo ? 🤔

My talk about "database etc." clutters the discussion. This process cannot be stateless/dumb, was my main point. With 10 remote resources, some of them possibly out of your control, you (or I) would want some kind of control over:

If it should be published.
Possibly also when it should be published.

None of the above allows for a simple "pull and push". So, if you do your builds on a CI server (Netlify), but do your editing on your local PC, that state must be handled somehow so Netlify knows ... what. Note that the answer to 1) and 2) could possibly be to "publish everything, always", if that's your cup of tea.

Note that the answer to 1) and 2) could possibly be to "publish everything, always", if that's your cup of tea.

Yeah, maybe some people want it or default to it but offering more control is definetly a must have I think.

So, if you do your builds on a CI server (Netlify), but do your editing on your local PC, that state must be handled somehow so Netlify knows ...

True in my mind, a CI pipeline would have to be put into place above Hugo.
So when the source is edited (using contentful or other) the CI is notified and can run something like  hugo --fetch-source="contentful".
Or a simple cronjobs (don't know what to call those in the modern JAMstack) could be set in place so website is build every hour with hugo --fetch-source="contentful" and every day with hugo --fetch-source="events"

OK, I'm possibly overthinking it (at least for a v1 of this). But for the above to work at speed and for big sites, you need a proper cache you can depend on. I notice the GatsbyJS WordPress plugin saying that "this should work for any number of posts", but if you want this to work for your 10K WP blog, you really need to avoid pulling down everything all the time. I will investigate this vs Netlify and CircleCI.

but if you want this to work for your 10K WP blog, you really need to avoid pulling down everything all the time

Yes. Time is essence!
I can't imagine how long Gatsby would take to build a 10K WP blog considering it already takes 18s to build the hello-world starter kit.
And this is precisely why big content projects want to turn to Hugo.

After spending time with playing with the friendly competition and its data source solutions.
It becomes apparent that one of the biggest challenges of the current issue (now that Front Matter mapping will be taking care of by #5455) will be how the user can define everything Hugo needs to know in order to

efficiently connect to remote or local a data source,
retrieve the desired data,
and merge it into its content system (path etc...).

3 will be unique to each project and potentially source.
On the other hand 1 and 2 will be for the most part, constant for many data sources, like WordPress API or Contentful.
For example, for a source of type WordPress REST API, Hugo will always use the same set of endpoints plus a few custom ones potentially added by the user.
It will also systematically uses the same parameter to fetch paginated items.
We could group the settings of 1 and 2 into one Data Source Type (DST).
Then, in the line of Output Formats and MediaTypes, any newly defined Data Source could use X or Y Data Source Type.
This way any DST could be potentially:

Reusable among one project without repeating same lengthy settings (2 different WordPress APIs for one website)
Shared among users as setting files.
Built-in

Rough example of DataSourceType/DataSources settings:
DataSourceTypes
  - name: wordpress
    endpoint_base: wp-json/v2/
    endpoints: ['posts', 'page', 'listings']
    pagination: true
    pagination_param: page=:page
    [...]

DataSources:
  - source: https://api.wordpress.blog.com/
    type: wordpress
    contentPath: blog/
    [...]

I wanted to throw this into the discussion because it's a demonstration of how I generated temporary .md files from two merged sets of JSON data (Google Sheets API). These .md files are only generated and used during compilation and are not saved into the repository.
https://github.com/IslandViewPTA/islandview-pta-site/blob/master/src/scripts/gspread2md.py
This is a fairly simple script, but you can see that I needed to do some filtering of the data source and mapping the 2 source JSON data sets to front matter parameters per page.

@bwklein Thanks for this very informative input but... this belongs in a "tips and tricks" thread in the discourse which could mention this issue. Not the other way around :)
PS: This really belongs there, people would love to read this I'm sure.

@regisphilibert I'm exactly looking for the same thing!
Having json parts in a page is simple, but generate post from json... Headless CMS to Hugo 👍

Will having some logics in md files also be a feature on the roadmap:
For example:
{{ $jsonResponse := getJSON $apiUrl "https://example.com/xyz/blabla }}
{{ range $jsonResponse.data }}
---
title: "{{ .id | "default title" }}"
---
{{- end -}}
So this way, the title can be dynamic for routes, permalinks, etc.

👍

So this thing has been bugging me for a while now (especially as I wanted to make it easier for contributors on my website to add content without manually creating folders & files).
So I wrote a temporary solution: kidsil/hugo-data-to-pages
It's a JS wrapper that generates pages from data (and it cleans after itself!).
I'm currently using this to generate pages from YAML files and it seems to be working perfectly.
Would definitely appreciate some feedback.
Cheers

With #6041, it seems convenient that any data source could be, like a Hugo mod assigned to a "directory" in the file system.
If this route is chosen, we could, in order to add the remote "jobs" source from this issues's desc example, simply add a content/jobs/_index.md to the project and handle any config/FM/data source info from there.
Using same example as above:
# config.yaml
DataSourceTypes
  - name: wordpress
    endpoint_base: wp-json/v2/
    endpoints: ['posts', 'page', 'listings']
    pagination: true
    pagination_param: page=:page
    [...]

# content/jobs/_index.md
DataSource:
    source: https://api.wordpress.blog.com/
    type: wordpress
cascade:
   permalinks: /our-job-offers/:title

I really wish this would be bumped up in priority. Would be an absolute game-changer

@jbigot and @chris-79 I'm also eagerly waiting for this functionality. You might want to keep an eye on issue #6310 which is the current proposal for building pages from data. If you read though the updates, there have been quite a few changes in the background preparing for this

I really wish this would be bumped up in priority. Would be an absolute game-changer

I agree, if this was possible we would literally be able to connect Hugo to every headless CMS source e.g. craft, wordpress, contentful, etc! Relying on forestry.io has been a pain..
This would help teams and organizations to be able to work in their native cms to organize content in any way possible. :)

@regisphilibert @bep  The Paginator object can convert an index page into multiple pages. One simple solution to have pages from data might be extend this to be able to paginate not just a list of pages but an arbitrary list. Then with page size 1, we have created one page per entry in a JSON.
We might need enhancements to the paginator object to be able to customize the front matter of of the generated pages. But I think enhancements like those can be incremental. I don't know the internals of Hugo to be able to say how difficult is the task, but as a user, this would fall inline with the current way of using Hugo and we will not need to learn a new concept.

This feature would be a game changer for Hugo. I hate having to use Netlify CMS to achieve wh.at I want, Strapi would be my go to CMS

fwiw, I ended up making my own scripts to process custom JSON output from our Wordpress.com site.
It's not pretty code, and may not follow correct Go conventions (as I'm super new to it), but it's working for us.
Hope this helps someone until we get official support within Hugo :)
Articles
For building Markdown files in a flat, single directory for one of our Hugo sites.

get-articles.go
articles.json

Vuepress-Pages
For building files into hierarchical directories based on the "link" key. Yes, this is for VuePress, but it should work just fine with Hugo with minor tweaking (like changing /README.md to /_index.md).

get-vuepress-pages.go
vuepress-pages.json

Example Yarn build command (runs in Netlify):
{
  "scripts": {
    "build": "curl -s -H 'Cache-Control: no-cache' \"https://example.com/wp-json/custom-routes/v1/internal-resources\" --output pages.json && curl -s -H 'Cache-Control: no-cache' -sL -o main.go \"https://gitlab.com/snippets/*******/raw\" && go run main.go && vuepress build docs"
  }
}

My proposal for this feature is that it should be three features:

Define a format for content files in JSON. Eg you’d have sources/myfile.json with a new line delimited JSON stream of objects with keys “filepath”: “stories/whatever.md” and “body”: “whatever” and that gets represented as though it’s a bunch of files in the file system.

Allow STDOUT plugins that have a line in Hugo’s conf like “plugin-cmd = ‘myscript.py’” and those can spit out JSON in the format above. It gets rerun at some interval, either per request or per time duration.

FCGI gateway that works like the plugin except as requests come in, they get passed to the app which can respond with JSON.

@carlmjohnson Could you clarify what you mean when you mention "requests" here? As Hugo is a static site generator, it's unclear what per-request behavior would mean.

Good question. Per request means using hugo serve. For example, if you want to be able to create a live preview, you could have hugo serve running somewhere and the plugin intercepts the request, generates some content, and then passes that to Hugo to build the actual page.
I noticed that @bep starred https://github.com/natefinch/pie on Github the other day. I think that approach (using JSON-RPC over STDIN) would work for plugins. Apparently LSP plugins like Gopls etc. work on a similar basis.

Let me talk through some sites I have built with Hugo and how I did them and how I wish I could have done them to give more background for this.
I made one site that listed a lot (several hundred) of candidates for local elections. We emailed the candidates and had them fill out a Survey Monkey survey. We cleaned up the data into a big Google Sheet. I wrote a script that downloaded the Google Sheet and turned it into a JSON file for each candidate, which we put into the content directory as a .md file (because content files have to end in .md). Code here: https://github.com/baltimore-sun-data/voter-guide-2018
I more recently did a similar site that lists expert sources in Pennsylvania. Again, survey, Google Sheet, JSON files in the content directory. Code here: https://github.com/spotlightpa/sourcesdb
Writing the script to download a Google Sheet and turn it into a bunch of small JSON files was not a big task for me, but it was some amount of work, and I imagine a lot of people who know HTML/CSS might not be comfortable doing it. It would have been easier if I could have dropped a single CSV/JSON into the repo, instead of many small files. It would be even better if I could write a plugin that automatically connects to Google Sheets when you run hugo or hugo serve.
I have another site where we schedule stories to publish at a certain time. We do this by saving the stories in a database and then a task runs periodically to see if any stories are ready to publish, and if so, it adds them to Github at the proper location, which in turn triggers Netlify to deploy the site. It would be nice if I could just tell Hugo to get the stories out of my database, and then to do a scheduled publish, I would just trigger a Netlify build at the proper time (no middle step with Github saving the content).

@carlmjohnson I'm trying something similar (without much luck for the past few weeks). Its a personal project, and I am trying to build a repository of assignments and work done by students in our college. I was planning on having a front-end Google Form for user-submitted content where each row is a new submission. After this, we set up a script to read that as a JSON file and hopefully break it into individual content files, which can be pushed to Github, which triggers a Netlify deploy.
Unfortunately, my experience with Python (which I'm assuming you also used?) so far has been limited to NLP and not so much in this area. Your last website implementation seems most ideal to me, where the Sheets document could act as the database and I only build new content pages if there are new rows. I don't mind doing a manual trigger for connecting to Sheets every now and then.
I tried going through the repos you linked but I can't seem to be able to find the code that does the conversion from Sheets to JSON in the content folder. Would you be able to help me out? If it is okay with you and as time permits, I would like to take your help with this since you seem to have done it before.

@carlmjohnson I'm trying something similar (without much luck for the past few weeks). Its a personal project, and I am trying to build a repository of assignments and work done by students in our college. I was planning on having a front-end Google Form for user-submitted content where each row is a new submission. After this, we set up a script to read that as a JSON file and hopefully break it into individual content files, which can be pushed to Github, which triggers a Netlify deploy.
Unfortunately, my experience with Python (which I'm assuming you also used?) so far has been limited to NLP and not so much in this area. Your last website implementation seems most ideal to me, where the Sheets document could act as the database and I only build new content pages if there are new rows. I don't mind doing a manual trigger for connecting to Sheets every now and then.
I tried going through the repos you linked but I can't seem to be able to find the code that does the conversion from Sheets to JSON in the content folder. Would you be able to help me out? If it is okay with you and as time permits, I would like to take your help with this since you seem to have done it before.

@bwklein This is perfect! Thank you so much for sharing it. I've been poking around with it for the past half hour, the pipeline looks wonderful. Would it be okay if I could reach out to you via some channel, since I had some questions and I don't want to clutter this thread? I'm very, very new to this and I would appreciate your help.

@bwklein This is perfect! Thank you so much for sharing it. I've been poking around with it for the past half hour, the pipeline looks wonderful. Would it be okay if I could reach out to you via some channel, since I had some questions and I don't want to clutter this thread? I'm very, very new to this and I would appreciate your help.

https://github.com/spotlightpa/sourcesdb/blob/main/scripts/clean_data.py converts the CSV file into JSON .md files.

https://github.com/spotlightpa/sourcesdb/blob/main/scripts/clean_data.py converts the CSV file into JSON .md files.

Here is another usecase for a plugin architecture. This Hugo reader wants his em-dashes handled a certain way: https://www.sidewalken.com/non-breaking-em-dashes/ There could be some kind of custom template function plugins to do this.

Here is another usecase for a plugin architecture. This Hugo reader wants his em-dashes handled a certain way: https://www.sidewalken.com/non-breaking-em-dashes/ There could be some kind of custom template function plugins to do this.

@bep Is this feature still scheduled? I see it's marked for milestone 0.78 but we are on 0.8. Thank you for the great job

@bep Is this feature still scheduled? I see it's marked for milestone 0.78 but we are on 0.8. Thank you for the great job

Here is a one-liner, that splits a JSON array into different markdown pages.
The filename is based on the value of the first parameter in each element.
The script depends  on jq and it should work in any UNIX environment:
jq -cr '.[] | .filename, .' *.json | awk 'NR%2{f=$0".md";next} {print >f;close(f)}'
Optional step: Script to convert Markdown files to Hugo Page Bundles
Obviously this little script does not address the scope of this issue.
But I am posting it here because it can help those who need a quick and simple way to split a data file and consume its contents as Markdown content files in Hugo, while we wait for a native way to handle Page from Data.

Here is a one-liner, that splits a JSON array into different markdown pages.
The filename is based on the value of the first parameter in each element.
The script depends  on jq and it should work in any UNIX environment:
jq -cr '.[] | .filename, .' *.json | awk 'NR%2{f=$0".md";next} {print >f;close(f)}'
Optional step: Script to convert Markdown files to Hugo Page Bundles
Obviously this little script does not address the scope of this issue.
But I am posting it here because it can help those who need a quick and simple way to split a data file and consume its contents as Markdown content files in Hugo, while we wait for a native way to handle Page from Data.

Direct import from a database into Hugo would IMO be not only a gamechanger. It would also make Hugo absolutely central (almost inevitable...) in the Jamstack world.
Just a couple of sources, I came across. For inspiration/motivation.

Daptin Database API seems to be able Create and Build HUGO static sites
Cyrill Schumacher has developed his own Built-in SQL support for GoHugo.io

Since this thread has become of one of workarounds, it's worth mentioning that Sanity.io, an excellent editor, that uses an interesting open source query language, GROQ (developed by the same team), has posted a how-to on pulling data from their API to Hugo using Netlify plugins: https://www.sanity.io/guides/sanity-and-hugo-with-netlify-plugins

Hello, we are also searching some kind of workaround to generate "page" from JSON files.
Currently, it's easy to parse the data on a specific page, but we are not able to "generate" lots of page.
We are (with @DavidVergison) heading to a go script that will generate *.md with correct Front Matter.
Question is : better to put all necessary in the *.md, or only the ID and add a Hugo processing based on this id (retrieved in the json data) ?

@Frackher if you generate the pages at build time from the JSON files, then you don't need to store anything in the repository for those files, you just use the script to write the markdown files into the appropriate 'content' location before running the Hugo command to build the site.

Yeah, put the data in front matter. I'd rather write {{ .Title }} than {{ index site.Data.pages .Params.id "title" }}.

I'm doing just what @bwklein said and grabbing the data from a headless CMS in a build script (in Netlify) and using a .go script to split the JSON into individual .md files
In the process, I discovered that you don't even have to put the data in YAML format in the generated page files. I have .md files with just JSON inside like this:
{
  "id": 42920,
  "link": "covid-19-updates-and-resources",
  "title": "COVID-19 Updates and Resources",
  "image": "https://example.com/AMU4Wgn.png",
  "page_blocks": [
    {
      "section": {
        "section_content": "The College of Education has created this page to provide our faculty, staff, and students with the latest updates and resources on the coronavirus (COVID-19) pandemic. Stay informed by visiting this page regularly.\r\n\r\nThe main source of updates and information is the [University's COVID-19 page](https://www.uga.edu/coronavirus/info.php), so please check it regularly. Also, [University Libraries](https://guides.libs.uga.edu/COVID19/) includes library resources for online classes."
      }
    }
  ]
}

@chris-79 the only downside to content in front matter is that Shortcodes don't work in there, they have to be in the "Content" section of the file.

@bwklein Yeah, true 👍

@chris-79
Since this issue has become about workarounds, please share the Go script to split the JSON, otherwise for general discussion please use the Hugo Forums.

I am using Hugo with Directus 9 https://directus.io/ and Netlify.
It's not that hard to create separated md-files from the json-data into Hugo.
With Netlify onPrebuild its easy to setup. You can use it to fetch the json and create the md-files.
Here is my code, maybe someone find it useful.
module.exports = {
    onPreBuild: async ({ utils, packageJson }) => {
        const fs = require("fs-extra");
        const fetch = require("node-fetch");

        fs.readdir("./content/blog/", (err, files) => {
            if (err) console.log(err);
            else {
                files.forEach((file) => {
                console.log(`Deleting: ${file}`);
                fs.unlink(`content/blog/${file}`, (err) => {
                    if (err) throw err;
                });
                });
            }
        });

        //fs.remove('./public/blog');

        try {
            await fetch('https://URL/')
            .then(response => {console.log('response: ',response);
                return response.json(); })
            .then((data => {

                for (let value of Object.values(data.data)){
                    let filename = "./content/blog/"+value.slug+".md"

                    fs.writeFile(filename, JSON.stringify(value), function (err) {
                      if (err) return console.log(err);
                        console.log(`file ${filename} written`);
                      });
                }

              }))
        }
        catch (error) {
            utils.build.failBuild('Failure message', { error })
        }
      },
    }

Would this problem be at all simplified by removing the need to deal with remote data? Say there was a way to autosync a remote data source (Sanity, Wordpress, etc) to your Hugo project's /data directory via git or something. Then, if Hugo could just provide a way to output pages based on data files, a CI tool could rebuild/deploy whenever external data was updated and pushed to github repo. Am I at all understanding where the complexity lies here?

@bbarr many of us are already doing that, the hope here is that we can setup direct connections to content sources and build pages without downloading everything into the local content folder before build to generate pages. Hugo already has the ability to pull data from remote sources but not to build pages, only to generate content within pages that are represented in the content directory.

Then, if Hugo could just provide a way to output pages based on data files, a CI tool could rebuild/deploy whenever external data was updated and pushed to github repo

This issue is supposed to be in two phases.

Build from data (generating pages from a data file)
Build from data directly from a remote source (generating pages from a remote source, API etc...)

An issue raised since 2018 still isn't implemented and we're almost in 2022?!
I'm working on an ecommerce store that pulls products from Shopfiy via the API. NextJS can do this, but was hoping to come back to Hugo. Problem is Hugo doesn't support it and I can't understand why it's taken so long to implement. Is Hugo's architecture that messed up that this can't be implemented?

@JR2Media Wow, what an impressively entitled and condescending attitude. Comments like these are actively harmful in OSS as they sap the spirits of maintainers and encourage burnout.
This comment is also factually wrong. It says absolutely nothing about the quality of the Hugo codebase that this hasn't been implemented yet. Why hasn't it been implemented? I'll suggest a few possible reasons.

Hugo is maintained by a very small group of people
A feature like this would take a ton of work and require lots of planning and discussion. If that hasn't transpired yet, it's because other things have been prioritized.
As it stands, it is already possible to use other tools to cover this functionality. I've done so myself. Having built-ins in Hugo would be nice, but probably better to prioritize Hugo's core.

Get bent.

This has been an issue since 2013! https://github.com/gohugoio/hugo/issues/140#issuecomment-67767381

@lucperkins The point is this has been asked for since 2013. The developers don't listen to what the users want, people will drop eventually. Probably why NextJS has now taken over. If you can't adapt the changes of the users, expect them to leave.
And issue from 2013, hasn't been implemented. Says a lot about the Hugo developers really.

@JR2Media Last time I checked, the project is OSS. Perhaps you'd like to take a crack at it?

@lucperkins that lame old response again. It's the classic rebuttal given when a user questions why a feature hasn't been implemented.
Luckily I'm a Go developer. Yes most certainly I would implement this if I knew:

That it wasn't going get rejected (as others attempts have)
I can be briefed on the architecture either by Bep, or other developers who know the system better than I do.

Can we have a voice chat to discuss the architecture and ways of possibly implementing this?

Anyone looking at the https://github.com/gohugoio/hugo/graphs/contributors page can see that Hugo is basically a labor of love by @bep. It's very entitled to flame him, when he's been adding tons of features that expand Hugo over the years. As a long time Hugo user, I would never want to go back to the Hugo of 2018 or 2013. It's particularly weird to flame him now, since he just released v0.90, which adds support for remote resources.

Truth be told I love Hugo, used it since 2015. I've had to use NextJS for the last year and it's alright, but I'm not a fan of being stuck to React to develop a static site.  I've been developing static sites using NextJS to pull data from Shopify and generating the product pages. Something like this has to exist in Hugo because of the current trend with "HeadlessCMS" platforms.
Without it, developers are forced to use alternatives. When they learn an alternative such as NextJS, they likely won't come back. And it's a shame, because Hugo is an otherwise awesome SSG.
Such an implementation would be a game changer.
@carlmjohnson definitely not flaming @Bep. I've used Hugo for a long time too. I checked back today to see if this was possible yet so I could look at using Hugo over NextJS.

Can we have a voice chat to discuss the architecture and ways of possibly implementing this?

Any ticket is a good start for a discussion.

Luckily I'm a Go developer. Yes most certainly I would implement this if I knew...

Usually you simply comment a ticket with your offer to help and then ask your questions. (Rather than starting with a negative comment). Maybe not too late (all help is welcome) you could start fresh on this ticket: https://github.com/gohugoio/hugo/issues/6310

#6310 sounds quite complex; I think that we can utilize most of the existing functions if we would allow users to add a page as follows:
.Site.AddPage MARKDOWN FRONTMATTER

Example:
{{ with resources.Get "https://api.example.com/pages" }}
{{ range . }}
    {{ .Site.AddPage .Content (dict
        "date" .Date
        "draft" "false"
        "title" .Title
        "description" "This is frontmatter"
        ) }}
{{ end }}

If I understand it correctly, the Site.AddPage function could create a markdown file in the virtual /content directory based on the parameters.
Do you think this would be possible and a good approach @bep?

This needs to wait until after what I'm in the process of doing (which is a little more than I can write down in a simple comment here).

well said

@bep I suspect this feature will become increasingly important as a major HUGO based CMS (Forestry.io) has announced an imminent EOL and many developers will be seeking (scrambling for) an alternative to Forestry.
Forestry's successor product Tina.io has announced per site pricing in their Discord, and lets just say it feels very premium.  I know that Forestry is not the only game in town, but certainly it is a large part of our HUGO ecosystem. In the Forestry Slack there are over 1000 developers in the HUGO channel.
So anyway, myself, I am awaiting this feature with much anticipation, as I evaluate other CMS options for my clients.  I really like the idea of adapting Directus.io with a HUGO extension which my team could develop and offer in the Directus marketplace.  It would be an excellent model for a HUGO CMS, and this feature of mapping data sources to pages would be integral.
It's such an important feature,I would perhaps even offer to sponsor it!  Do we have any new thoughts about what is feasible?  I wish I had the Go development experience in my shop to contribute more meaningfully.  If a feature sponsorship is possible, I am here to discuss.
With gratitude for everyone's efforts!

CloudCannon and FrontMatter CMS are viable alternatives to Forestry.io and do not require data driven pages.  Having said that, there are other situations where data based pages would be a significant advantage for Hugo over other systems like SvelteKit.

@bwklein I appreciate your thoughts.  I am desperately seeking the next great CMS for HUGO.  I thought Forestry was close. It would be great if there were a no compromises solution. FrontMatter CMS is hard to give to non-technical users due to VSC requirements, and I have found CloudCannon to be somewhat mixed.  Menu management, for instance.  In this use case, I want to achieve ultimate usability for small business clients.
But I don't want to derail the discussion.  Data driven pages are something that I think could create massive possibilities. It could achieve a best-of-both-worlds flexibility to adapt to both GIT-backed, document driven SSG, and to leverage APIs creatively. @vanbroup's solution just leapt off the page to me as something inspired.  Finding this thread and seeing that the feature is in discussion gives me great hope and optimism.  I hope it can be included in a near-term 2022 release.  Again, I would sponsor this.

I would sponsor it too. Using Strapi for alot of APIs and Changing Texts on VueJS Web Apps. I'm using it also to manage the landingpage texts and images in hugo. I would love to have this feature for Blogposts.

I can't believe nobody has solved this yet. It's absurd. Where is the sponsorship money going?! I'm calling @bep out as a noob. I'm only responding to this because I keep getting email notifications. Folks, go with NextJS, or Gatsby. Hugo is now just a turd, that can't keep up with what the community wants.
I await the "Oh but it's open source you can contribute if you want ...". Many have tried. Bep rejects the PRs.
 I await the hate.

So you call out the author/maintainer/programmer of a project:

that you're not sponsoring
that you're not maintaining or otherwise helping
have 0 public github activity
the 1-2 issues that you've closed on other repos have SUPER minimal info and when you fix them the answer is "fixed it"
YOU ARE A COMPANY THAT USES THIS SOFTWARE

In general what you do is like calling a random person and nagging that they're "noobs" because they didn't have the calling tune that you liked.
You don't get the whole open source idea, in a whole other level. You're lightyears away of how things work around here. Be nicer. Nobody has to do ANYTHING. Heck, they can delete this repo just because they feel like it

Hello, dear colleagues!
For many years I have been using Wordpress to create websites, without using themes, only the Wordpress platform, only my own PHP on the server and JS on the client.
Recently, I made one site on HUGO, and I really liked this concept - speed, reliability, simplicity.
However, the customer didn't like editing the .MD files on disk to make changes to the site. He wants to go into the admin panel on the site and edit pages and posts using the input boxes and "save" button.
Can you please tell me if you can now (or in the near future) use Wordpress (or another CMS) as a headless CMS for HUGO?
It is IMPORTANT that CMS should not be a service on vendor's site or depend on some services (like Ntlify), but should be completely independent CMS and free, installed on customer's VIRTUAL hosting (Wordpress meets these conditions).
Thank you!

@geshov I'm not sure this is the place for your question. I can answer though because I use both Hugo and Wordpress. You're going to need to use headless Wordpress. You can then pull the content from Wordpress and save them as markdown files for Hugo. I don't know if there's an off the shelf solution for this. I have written my own tools to do this.

James, thank you for your prompt reply!
I'll think about your suggestion, but that's not exactly what I asked.
My question wasn't about that:

You can then pull the content from Wordpress and save them as markdown files for Hugo

It was about this:

Build pages from data source (title of this topic)

More specifically:

But Hugo cannot, using a data set of items, build a page for each of them plus related list pages like it does from the content directory files.

These are quotes from the first post of this thread, so that's where my question is most relevant.
Thank you!

Please head over to https://discourse.gohugo.io/ and ask your question. I will reply to you there.

@geshov I have an example of using Notion as hugo backend, if it is of any help: https://whynot.fail/coding/notion-blog/

What is the current progress of this feature?
I have the similar problem - generate multiple pages from CSV file with one page from one line of original csv

@knutov I ended up just writing a small python script that generates the pages for me.

and eventually I switched to another SSG generator that can do this out of the box,
what exactly am I using now - I can answer in a personal message so that it is not regarded here as advertising

In my mind, the amazing Regis has solved this, and for my company it's not an issue any more.  I give the strongest recommendation to Regis's article on the subject on The New Dynamic.  It's the "monster spotting" example.
https://www.thenewdynamic.com/article/toward-using-a-headless-cms-with-hugo-part-2-building-from-remote-api/
My team implemented this on 50+ sites.  It's no more complicated than Forestry ever was.  We can set up a modest site in about a day or less.  It's massively flexible, which we value, and provides both a "built" solution in line with the use case of HUGO as well as a strong CMS-agnostic integration path.
We trigger builds either from our CMS on change, using a Netlify build hook, or we have a manual "publish" action which cUrl's the web hook.  Many clients seem to prefer this idea of "wait and publish all at once" as opposed to every change being instant.
Anyway. Regis solved it.  It's not a problem any more. I promise, just follow his lead.  This is the path.

+1 for the amazing @regisphilibert

Before searching for documentation on how to do this, my expectation was that any support would be incredibly basic/rudimentary and I would probably be dealing with some special index file that explodes data into a directory of markdown files.
It took me a couple times reading that "headless cms" solution, but now I understand that it's just using hugo to generate content for hugo; it's literally, "code generating code," which is the same thing we're doing with smaller "stub generator" scripts. We just have to add logic to read hugo configuration data, which hugo has built-in.
To be honest, the hardest part of reading (and understanding) that post was my assumption that it was showcasing a new hugo feature that would just work out of the box. The existing options still seem "more enjoyable," but the solution here is actually pretty elegant and seems to answer a lot of existing questions, particularly with regard to mapping data.
Is it possible to "add a new phase" to hugo execution that processes templates in a _stubs/ (or _prebuild/) directory using a format such as (_stubs/albums.md):
FS_FLUSH: "/jazz_albums/*.md"
{{ range $.Site.Data.jazz.albums }}
FS_PATH: "/jazz_albums/{{ .ID }}.md"
---
title: {{ .ID }}
layout: album-demo
---
{{ .Summary  }}
<!--more-->
{{ end }}

I'm sure the "FS_" flags are wildly inaccurate, but what about the rest of that model?

For anyone else stumbling across this, I created a simple example a couple of weeks ago:
git clone --single-branch -b hugo-forum-topic-45433 https://github.com/jmooring/hugo-testing hugo-forum-topic-45433
cd hugo-forum-topic-45433
rm -rf prebuild/public && hugo -s prebuild && hugo server

Reference https://discourse.gohugo.io/t/45433

Maybe good to link this issue onthe Hugo roadmap page?

In my mind, the amazing Regis has solved this, and for my company it's not an issue any more. I give the strongest recommendation to Regis's article on the subject on The New Dynamic. It's the "monster spotting" example.
https://www.thenewdynamic.com/article/toward-using-a-headless-cms-with-hugo-part-2-building-from-remote-api/
My team implemented this on 50+ sites. It's no more complicated than Forestry ever was. We can set up a modest site in about a day or less. It's massively flexible, which we value, and provides both a "built" solution in line with the use case of HUGO as well as a strong CMS-agnostic integration path.
We trigger builds either from our CMS on change, using a Netlify build hook, or we have a manual "publish" action which cUrl's the web hook. Many clients seem to prefer this idea of "wait and publish all at once" as opposed to every change being instant.
Anyway. Regis solved it. It's not a problem any more. I promise, just follow his lead. This is the path.

Used this solution also and it worked perfect !!
I am using it together with directus and netlify... love it !
Thank you !

Regis' solution with a "nested" Hugo project is quite amazing and saved me lots of time. :pray: Properly integrating pages generated from data into Hugo would be great, because running two instances of Hugo during development is very error-prone. :sweat_smile:


Request: Port i3-gnome to work with Sway

It'd be great if this was compatible with sway wm. Any chance on porting it to work with it?

Well, I think that there's a way to get sway compatible with GNOME but you'll have to read a lot.
On another issue (#28) someone were trying to get it work with BSPWM, I've not knew anything new after closing the issue (due lack of activity).
I friendly invite you to try and implement sway-gnome by your self, you can use all the documentation available and everything related with the scripts.
The main challenge you'll face will be get to understand how wayland works and how can relate it with the binaries of GNOME.
These days i3-gnome have been a reference to customize and relate i3 and GNOME.
I hope the wiki we have here can give you a hand on this.
I'm looking forward with other projects so my time is limited on dealing with the issues of i3-gnome and update it in case GNOME updates as well.


Fix mixup between httplib and urllib3

Trying to fix https://github.com/kevin1024/vcrpy/issues/315, I found this, that I hoped would have fixed the bug, but alas it just moves it a little further (so I'm still struggling to find the root cause of #315)
At least, even if it's not enough for #315, I think this is still a typo in the code that should be fixed.
My two cents,

Can you provide a test for this?

I honestly have no idea how to test vcrpy itself, I got this by scrutiny...

@graingert It seems reset_patchers is not tested at all, or I don't understand how it's tested.

@graingert I pushed a test, that I confirmed breaks in pyXX-urllib3121 before the patch I propose, but work after.
I still fight with pyXX-requests216 that for some reason fails on Travis for test_urllib3.

@graingert Travis is green, youhou! So this PR fixes actually two bugs.
Little summary:

reset_patchers that is used in force_reset was resetting incorrectly urllib3 HTTPConnection class to httlib.HTTPConnection and not urllib3.connection.HTTPConnection. Easy part.
Starting requests 2.16.3, requests.packages is a black magic module that says requests.packages.urllib3 == urllib3. This becomes an actual namespace mess for mock.patch, when the code was trying to unmock requests.packages.urllib3.connection.HTTPConnection AND urllib3.connection.HTTPConnection. Meaning that force_reset was still not returning the correct HTTPConnection class, but the fake one from VCR this time. So this PR also ask force_reset to do not try to unmock if requests.packages is the black magic file.
I added a unittest and requests 2.18.4 to Travis just in case

nice, can you rebase and clean up your commits.

Haha hacks causing hacks all the way up the stack.  Thanks for this, hopefully will solve a lot of the issues people are seeing.

nice, can you rebase and clean up your commits.

I'd prefer to keep all of these commits, since they aren't doing different things.

@lmazuel can you provide the new stack trace that youa re getting?

@IvanMalison those commits were post cleanup.

@IvanMalison I like atomic commit as well, so I rebased 6 commits to 3 (do some fixup on commit named like "typo", etc.)


Won't reach internet from created subnet

Got here so best open new ticket as so different than earlier issue which was fixed already.
Now i have configs which works so that it does provide IP's. But i don't get internet connection. Not sure does it get send but feels like not result back in..?
-WinXP laptop still fails get IP-#Other problem
Maybe i have little problem still with config.. i have tried done:
sudo route add -host <IP_ADDRESS> -interface en4
No network (yet..)
I get to send request of ping from inside this network but it doesn't get answer (or other way around.. i can't say myself). There is no response to ping <IP_ADDRESS> dns even.
wireshark picture under..

Trying just:
I thought fake how InternetSharing does it.. combine dns to router address
Now i get port unreachable.. so better or worse?

Any ideas? #Update InternetSharing does create bridge0 too.
Do i need have manual config in SystemPreferences and what style? if it makes differences..
Update:
Been log how InternetShare does all.
At DHCP offer it gives
Internet Protocol Version 4, Src: <IP_ADDRESS>, Dst: <IP_ADDRESS>
So there is Src: Router IP, Dst: New device IP
this dhcp gives when i try WinXP laptop:
Internet Protocol Version 4, Src: <IP_ADDRESS>, Dst: <IP_ADDRESS>
So it is.. Router IP but the Broadcast ip? Hmm..
from sudo tcpdump -i en0
17:40:56.796391 IP <IP_ADDRESS>.bootps > <IP_ADDRESS>.bootpc: BOOTP/DHCP, Reply, length 290
17:40:56.796760 IP <IP_ADDRESS>.bootpc > broadcasthost.bootps: BOOTP/DHCP, Request from 00:1f:PS3 (oui Unknown), length 300
17:40:56.797664 IP <IP_ADDRESS>.bootps > <IP_ADDRESS>.bootpc: BOOTP/DHCP, Reply, length 290
17:40:56.797891 ARP, Probe <IP_ADDRESS>, length 46
17:40:57.297002 ARP, Announcement <IP_ADDRESS>, length 46
17:40:57.297162 ARP, Announcement <IP_ADDRESS>, length 46
17:40:57.485661 ARP, Request who-has <IP_ADDRESS> tell <IP_ADDRESS>, length 46
17:40:57.485694 ARP, Reply <IP_ADDRESS> is-at 8c:ethernet (oui Unknown), length 28
17:40:57.485949 IP <IP_ADDRESS>.49556 > dns.google.domain: 55586+ A? feu01.ps3.update.playstation.net. (50)

This was more likely likely a feature than issue. It doesn't connect subnet to network on macOS Sierra. It needs a extra steps which InternetSharing does like creating completely new Interface. InternetSharing is disabled while using this if it matters.

I confirm this was OS issue on newer macOS.. needed do some trickery to get internet shared but now works internet on subnet i create with this too. Hit message if anyone needed it.


Security Current Day OHLC

Expected Behavior
Be able to access the security OHLC values of the current trading day.
Actual Behavior
We need to implement this manually in each algorithm.
Potential Solution
Implement a system that keeps track of the current trading day OHLC.
Checklist

[x] I have completely filled out this template
[x] I have confirmed that this issue exists on the current master branch
[x] I have confirmed that this is not a duplicate issue by searching issues

Duplicate of https://github.com/QuantConnect/Lean/issues/1608


Frontend rewrite

[x] Rewrite in React
[x] Responsive search (remove lag while typing) - Fixes #120, and is a better fix for #115

[x] Improve rendering speed (by virtualizing)
[x] Improve .filter() speed (e.g. by removing regexes)
[ ] (?) "Proper OR implementation for searches" - See #104

[ ] Retain existing features

[ ] 'Space' to stop all
[x] Hold ctrl/shift to spam/loop
[x] Context menu (spam / loop / copy ~URL~ url)

[x] Context menu
[x] Play / Pause
[x] Spam
[x] Loop
[x] Copy URL
[x] Download - Fixes #121

[x] Random play on enter.
[-] ~TTS ("Japanese intern")~
[-] ~"Whether you're looking for things that do what ..."~
[ ] Themes

[x] Fixes #132
[ ] See #116
[ ] See #114

[x] Adjust volume
[x] Version information
[x] "CONTRIBUTIE" modal/link
[x] "No search results" message
[x] User interaction prompt for immediate media playback
[x] Play from URL

[ ] Add useless (but highly necessary) new features

[x] Sine wave while playing - Fixes #133
[x] Show progress bar for all playing instances of a sample
[ ] More...

@villermen Are you still missing anything in this PR? If not, I can start working towards getting it merged!


Pass the decryption key to the video directly

Hello,
We are working on a streaming application and we want to have AES encrypted streaming. The video source is already streaming in an encrypted format using all streaming protocols DASH, HLS, and Smooth. We have the encryption key and the initialize vector but I couldn't find a way to pass them to the Video component.
We have discovered the DRM support but we would prefer not to go with a license server that seems to be more time-consuming.
Any help would be appreciated.
Many thanks,
Florin

Any update on this?


Call lang of module not working

I had called the function dd( \Lang::get( 'products::table.action' ) );, but not working.
My module lang file:
App/Products/Resource/lang/table.php:
return [     'action' => 'Ação' ];
what this problem?

Did you tell laravel to find translations in that folder ?

Disregard.
I did not insert it in the language folder.
I so sorry;
Thanks for your help


Inflating heavy nodes

I am trying to draw hierarchy using treeview and the nodes contain photo of the user. Due to this the tree view takes time to inflate the graph and the UI freezes for that time. I tried implementing AsynLayoutInflater in the adapter but that did not work.
Can you provide any way to optimize the inflation?

How many nodes does your graph have? Did you consider Glide for loading the images?

Thanks for the suggestion , it works perfectly now bdw great work!


How to test a package/library like NumPy?

Describe the bug
Hi! Pynguin is a fantastic automated tool for Python. I want to use it to test a library like NumPy. But I met some failures and got some results. I want to share the information I collected and ask for help.
To Reproduce
I leverage numpy/matlib.py as an example. It contains many common NumPy APIs (e.g., empty and ones)
First round

Download NumPy source.
Checkout a release version (v1.26.4, the latest version of NumPy 1).
enter to numpy and run pynguin --project-path . --output-path ../output/numpy/ --module-name numpy.matlib  --maximum-search-time 60 -v
Then I get the following error from Pynguin:

Error importing numpy: you should not try to import numpy from  its source directory; please exit the numpy source tree, and relaunch your python interpreter from there.

Second round
According to the error you should not try to import numpy from  its source directory, I tried to do the second try:

pip install numpy==1.26.4
pynguin --project-path . --output-path ../output/numpy/ --module-name numpy.matlib  --maximum-search-time 60 -v (workdir is not the numpy source dir)

Then I get a test file, but all tests are annotated with @pytest.mark.xfail(strict=True).
I leverage one test as an example:
@pytest.mark.xfail(strict=True)
def test_case_3():
    list_0 = []
    none_type_0 = None
    var_0 = module_0.empty(none_type_0)
    assert f"{type(var_0).__module__}.{type(var_0).__qualname__}" == "numpy.matrix"
    assert len(var_0) == 1
    module_0.randn(*list_0)

After removing the @pytest.mark.xfail(strict=True), the test triggers an error in the last statement module_0.randn(*list_0). I remove the statement and get the new test, and can run successfully:
def test_case_3():
    list_0 = []
    none_type_0 = None
    var_0 = module_0.empty(none_type_0)
    assert f"{type(var_0).__module__}.{type(var_0).__qualname__}" == "numpy.matrix"
    assert len(var_0) == 1

However, Pynguin generates only None as the inputs for the empty and other functions under test, which are far from the common inputs. For example, empty's first parameter is a tuple denoting the output ndim array's shape.
So, my first question is, currently is possible to help Pynguin generate more meaningful inputs?
Third round
Since I have generated tests successfully in the second round, I try to generate more tests. For example, I want to generate test for numpy.linspace located in numpy/core/function_base.py#L25:

pynguin --project-path . --output-path ../output/ --module-name numpy.core.function_base  --maximum-search-time 60 -vvv

Then I get a file with only one test:
import pytest
import platform as module_0
import numpy.core.function_base as module_1

@pytest.mark.xfail(strict=True)
def test_case_0():
    var_0 = module_0.architecture()
    module_1.add_newdoc(var_0, var_0, var_0)

add_newdoc is a NumPy internal function that the end-users do not use.
I reviewed the Pynguin's log and found that it ignored the linspace function:
DEBUG    Skipping method linspace from analysis

I located this log is printed in module.py
It seemed that linspace satisfies the skipping requirement:
if (
        __is_private(method_name)
        or __is_protected(method_name)
        or __is_constructor(method_name)
        or not __is_method_defined_in_class(type_info.raw_type, method)
    ):
    LOGGER.debug("Skipping method %s from analysis", method_name)
        return

Here is the signature of linspace:
@array_function_dispatch(_linspace_dispatcher)
def linspace(start, stop, num=50, endpoint=True, retstep=False, dtype=None,
             axis=0):

, and it is not in a class. Hence, I guess linspace satisfies not __is_method_defined_in_class.
So, my second question is, why Pynguin treats linspace as a method instead of a function?
Expected behavior
Pynguin can generate tests for NumPy.
Screenshots
If applicable, add screenshots to help explain your problem.
Software Version (please complete the following information):

OS: Ubuntu 22.04.3 LTS
Python version: 3.10.15
Pynguin Version 0.38.0

To be honest, numpy is very challenging for various reasons.  Please note that Pynguin at its current state is a research prototype.  When I tried to replicate your issue I got errors from Python's import library (used by Pynguin) that there might be a circular import in numpy's source code, however, running import numpy in a REPL works fine.  There is definitely space for improving here for Pynguin, however, I do not really have a good suggestion that works reliably here.


namespace restricted backendRefs

The purpose of this issue is to enforce namespace restrictions for backendRefs in the admission webhook controller, under the httproute validation (which lives in internal/validation/gateway/httproute.go).
Pre-Requisites

[ ] #2080

added AC to reflect this need and closing as a duplicate of https://github.com/Kong/kubernetes-ingress-controller/issues/2080


Strange Behavior in exe, possible threat?

The file "Setup.exe" appears to be classified as a STEALER, indicating that it may have functionalities related to information theft.
Suspicious Behaviors:
Defense Evasion (TA0005):

Obfuscated files or information (T1027) using XOR encoding.
Registry modifications (T1112) to delete registry values.

Discovery (TA0007):

Performs various registry queries (T1012).
Discovers session username (T1033).
Conducts file and directory discovery (T1083).
Discovers user account information (T1087).

Anti-Analysis:

References anti-VM strings to detect virtual machine environments such as Xen, VMWare, and VirtualBox.
Also references strings of analysis tools.

Could you explain the reason behind these suspicious behaviors?
Behavior Virus Total

Perhaps I'm a little bit confused about what exactly did you expect by opening this issue. How am I supposed to explain false positives? You literally have the whole code at your disposal and the pipelines explicitly generating the releases from it.

"My concern isn't the false positive, but rather the behavior. Why is it conducting an analysis to determine if it's running on a VM? And what's the reason for obfuscating certain files and information? I'm not accusing, just asking."

How am I even supposed to know in-depth why the binary is flagged by an arbitrary antivirus software? There's no VM detection in the code, there's not a single form of obfuscation HENCE the project being open-source; “possible threat?” does sound a lot like an accusation over a question, that's why I see no point in this issue. If you feel like there's any shady behaviour going on, please keep it straight and technical as it should be on this platform: POINT IT IN THE CODE, DON'T SPECULATE.


Exported scenes cannot be edited!

The exported scene file can not be edited, the position of the transformation can be modified, and the rotation is invalid.

Upload a file

Here is the address and extraction code of the resource. https://pan.baidu.com/s/1GFaRzU9F6weaIPeMGh5CcA code: rssw

or here
scenes_city02_01.zip

I would suggest you to learn Unity a little before trying to work in it. You can't modify static mesh's parts in Editor.


[Help] Got an error when try using integrations with harpoon

Got this error E5108: Error executing lua: ...er/start/portal.nvim/lua/portal/integrations/harpoon.lua:7: Invalid buffer id: 1 when using command require("portal").jump_backward

Thanks for raising the issue!
This should now be fixed in main with 5cd642baab708b89be157569e328d683a45ca8d8. Let me know if you find that the issue persists

Hm, that's odd. I can't seem to reproduce. Is it the same error as above?

I've created a branch fix_harpoon_integration to test a couple other changes. Feel free to switch to it and see if it resolves the issue.
main..fix_harpoon_integration

Sorry for the late reply; I'm playing around with it, and it seems everything works @cbochs

@cbochs I found another issue in the fix_harpoon_integration branch; when I don't have any harpoon marks, the plugin will stop working ( it shows nothing when executing the command). It must have 1 item or will break the whole plugin

E5108: Error executing lua: .../.local/share/nvim/lazy/portal.nvim/lua/portal/query.lua:53: Unable to find registered query item:
harpoon

Why you require grapple in harpoon integration module ?
https://github.com/cbochs/portal.nvim/blob/6198977abbfe30e19075edf106a507a6a708402c/lua/portal/integrations/harpoon.lua#L16

Explicit support for plugin integration has been dropped. I'm happy to try and add examples in the README for harpoon integration, but am hesitant for now due to it's pending rewrite (see: https://github.com/ThePrimeagen/harpoon/issues/255).

Update took another look into harpoon and found that it should be relatively simple to filter for harpoon tags now. A couple of examples:
Filter the jumplist
require("portal.builtin").jumplist.tunnel_backward({
  filter = function(v) return require("harpoon.mark").status(v.buffer) ~= "" end
})

Select from jumplist (like the previous "harpoon" query")
require("portal.builtin").jumplist.tunnel_backward({
  slots = { function(v) return require("harpoon.mark").status(v.buffer) ~= "" end }
})


Model load takes a very long time!

I am using the Word2VecModel.fromBinFile(modelFile) API to open the default word2vec model, GoogleNews-vectors-negative300.bin (https://code.google.com/p/word2vec/). The model is ~3.6 gigs. My program has been stuck loading this model for more than an hour now. Heap usage is at 11 gigs.
What is the expected load time for a model this size? Why is it taking 11 gigs for a 3.6 gig model?
Would converting the model to the Thrift format be a solution?
Appreciate your help!
Regards,
Sumithra

Loading that model takes a lot of time in SwappedDataInputStream.readFloat(). 99%, according to YourKit.


Qualimap error message: BAM file is empty or no read alignments are located in exonic regions.

Hello,
I have used STAR to align paired-end reads from this sample and the output looks good when I view with samtools. However, I get this error message when I try to run Qualimap:
Failed to run rnaseq java.lang.RuntimeException: BAM file is empty or no read alignments are located in exonic regions. 	at org.bioinfo.ngs.qc.qualimap.process.ComputeCountsTask.run(ComputeCountsTask.java:559) 	at org.bioinfo.ngs.qc.qualimap.process.RNASeqQCAnalysis.run(RNASeqQCAnalysis.java:68) 	at org.bioinfo.ngs.qc.qualimap.main.RnaSeqQcTool.execute(RnaSeqQcTool.java:188) 	at org.bioinfo.ngs.qc.qualimap.main.NgsSmartTool.run(NgsSmartTool.java:190) 	at org.bioinfo.ngs.qc.qualimap.main.NgsSmartMain.main(NgsSmartMain.java:111)
Thanks for your help!

Thanks @prxrsp, sorry for the problem. It sounds like something is up with your BAM file, is it empty?

Hi @roryk,
No, it has a decent size and I have viewed it with samtools. It looks just fine.

Thanks, what genome is this on? Does this issue apply to you at all? https://github.com/chapmanb/bcbio-nextgen/issues/2203

I don't think so, as I have a proper BAM file and am running everything on O2. I use hg19.

Oh nice, you are on O2. Thanks, could you email me the directory you are running everything in so I can take a look?

Looks like this isn't bcbio related, so closing it here. Thanks!

I have a similar issue, same error on bam file that is not empty.. any suggestion?

Hi Roy,
Is this from running bcbio, or are you running qualimap by yourself?

Hi @roryk this happens when running Qualimap by myself on STAR aligner output.

Hiya!
Sorry, this repo is for bcbio; we use qualimap but aren't the authors. The qualimap Google groups is probably a better place to ask questions about it: https://groups.google.com/forum/#!forum/qualimap
My guess is, though, if you are seeing "Failed to run rnaseq java.lang.RuntimeException: BAM file is empty or no read alignments are located in exonic regions." on a BAM file that actually has alignments, is that the BAM file you are using has chromosomes named differently than the GTF file you are giving qualimap, so qualimap isn't finding any exons.
Qualimap also needs a specific GTF format which has gene_id and transcript_id in each entry, along with some other specifics. So your GTF file may not conform to the specifications. The qualimap mailing list would be a good place to get more help though. Hope that helps!

Thanks so much for your reply! I will checkout the group and mailing list.


getting error

Traceback (most recent call last):
File "c:\Users\pc.vscode-insiders\extensions\ms-python.python-2019.9.35938\pythonFiles\ptvsd_launcher.py", line 43, in
main(ptvsdArgs)
File "c:\Users\pc.vscode-insiders\extensions\ms-python.python-2019.9.35938\pythonFiles\lib\python\old_ptvsd\ptvsd_main_.py", line 432, in main
run()
File "c:\Users\pc.vscode-insiders\extensions\ms-python.python-2019.9.35938\pythonFiles\lib\python\old_ptvsd\ptvsd_main_.py", line 316, in run_file
runpy.run_path(target, run_name='main')
File "C:\Users\pc\AppData\Local\Programs\Python\Python37-32\lib\runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "C:\Users\pc\AppData\Local\Programs\Python\Python37-32\lib\runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "C:\Users\pc\AppData\Local\Programs\Python\Python37-32\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "f:\pythonprograms\main\product.py", line 23, in
File "f:\pythonprograms\main\product.py", line 17, in product_price_calcuation
product1.product_info_display()
UnboundLocalError: local variable 'product_price' referenced before assignment
PS F:\pythonprograms>

Closing this, please open a new issue following the issue template describing the error as well as your environment and steps to replicate the issue.


fix: unexpected num_frames during v2v inference

This is a good suggestion because the default value for this parameter is 49, however, I recommend that you comment out this code instead of deleting it. This is the simplest standard code, and we hope that developers without a foundation or those who are just starting out can understand the meaning of each parameter.


Getting "404: Not Found" when trying to use get_opensnp_datadump_filenames()

I am getting 404 error when trying to access filenames using
from snps.resources import Resources
r = Resources()
files = r.get_opensnp_datadump_filenames()

Code Link:
https://github.com/apriha/snps/blob/d69dc0fed4bb14b6dfde7de1372c717bdfbdf3ca/src/snps/resources.py#L265

Thanks for the issue. Sometimes the openSNP data dump returns 404... see here.
As a temporary fix, the @openSNP team can can kickoff a process to recreate the archive. Can you help @gedankenstuecke? 🙂 Thanks again.

Yep, I kicked our server to generate a new file. Sorry for that!


The Colored compatibility banner should be tapeable (more explanations + access to food preferences )

[ ] The Colored compatibility banner should offer access to the food preferences page
[ ] The Colored compatibility banner should offer access to a page that explains

[ ] why the product is incompatible when it's an allergen
[ ] why the product is incompatible when it's a dealbreaker
[ ] why the product is incompatible when it's a low score

Proposed mockup

Part of

#1354

Hi @teolemon, Can i get this one?

Shouldn't this be assigned to @Pualoo or is @cli1005 already working on it


Fixes Issue #3 - clicking labels should focus fields

Labels and ids added to the form. This should fix issue #3

Looks great to me, thanks Alistair!
Honza


add the ability to customize the auditable type

If you have a different project writing in the same models, the model class may have a different name. This pull request add the ability to write the audit with the right auditable_type.

@lrgalego We no-op set_version in the monolith for performance - would be good to do this for fulfillment as well.  We should probably update the attribute before it's inserted to prevent an extra query.

Slightly stale


ssr-dom-shim in reactive-element is breaking test cases in vitest/happy-dom

Which package(s) are affected?
Lit Core (lit / lit-html / lit-element / reactive-element)
Description
as of v1.6.0 of reactive-element, it imports srr-dom-shim when environment is node.
When testing using dom emulation like happy-dom, it is loading the shim, which is causing test cases to fail.
Reproduction
https://stackblitz.com/edit/vitest-dev-vitest-7rrc4s?file=package.json,test%2Fbasic.test.ts
Workaround
force @lit/reactive-element dependency version to 1.5.0
Is this a regression?
Yes. This used to work, but now it doesn't.
Affected versions
Failing with @lit/reactive-element 1.6.0, working with 1.5.0
Browser/OS/Node environment
OS: Windows 11
Node Version: 14.19
npm version: 6.14.17

Thanks for filing!
Looks like this line from the Rollup config is not working as intended: https://github.com/lit/lit/blob/65e56655b73d22172647c1a748e7a907ad0227c0/rollup-common.js#L279
The published code simply has extends HTMLElement instead of extends (globalThis.HTMLElement ?? HTMLElement).
Will try to get a fix today.

The fix for this is released with<EMAIL_ADDRESS>Updating to the latest version in the stackblitgz repro above alleviates the original issue.
One of the assertion still fails
  it('should increment the count on each click', () => {
    getInsideButton()?.click()
    expect(getInsideButton()?.innerText).toContain('1')
  })

with the error
TypeError: childNode.textContent.replace is not a function

I think this is due to something with how happy-dom implements .innerText. Changing to
  it('should increment the count on each click', () => {
    getInsideButton()?.click()
    expect(getInsideButton()?.textContent).toContain('1')
  })

makes the test pass.


WIP: overhaul profile overview tab UI

There was some love shown for this idea in the Discord so I thought I'd make a draft PR for it. This is a patch to make the overview tab of user profiles display not just a profile readme but also a users 6 most popular repos (sorted by stars), along with their activity. This PR also does the same for organization overview pages (albeit without activity) building off of my other PR.
People suggested on Discord that pinning repositories would be a better solution than displaying the 6 most starred repositories. I completely agree, and I'll look into adding that, but as the pinned repositories list would need to be stored in the database some guidance on how best to do that would be appreciated!

I've marked this as a WIP because as multiple others mentioned, pinning repositories would be better. The feature works as-is, however. If it's preferred for pinning repositories to be added as a separate follow-up PR, I'd be open to that as well :)

I've marked this as a WIP because as multiple others mentioned, pinning repositories would be better. The feature works as-is, however. If it's preferred for pinning repositories to be added as a separate follow-up PR, I'd be open to that as well :)

Pinning would be fantastic!  The best place to look is the previous pin-issue PR ( https://github.com/go-gitea/gitea/pull/19831 ) as there is alot of discussion as well as some code ideas.
Likewise the merged "pin issues" PR ( https://github.com/go-gitea/gitea/pull/24406 ) should proivde alot of reuse as well as what can be done with the DB

I've marked this as a WIP because as multiple others mentioned, pinning repositories would be better. The feature works as-is, however. If it's preferred for pinning repositories to be added as a separate follow-up PR, I'd be open to that as well :)

I has try it in https://github.com/go-gitea/gitea/pull/12498, may be helpfull for you

and #19831

You can mostly reuse the approach of #19831 if you use a table in 1NF.

@hazycora i did resolve the conflicts ... and if you make it a "profile overview" enhancement only e.g. remove the grid and do that in another pull. then we can get that enhancemen in sooner :)


Cache and reuse Docker containers (Warm Start)

Currently a new Docker container is created and cleanup on every request to the local api gateway, which means a cold start.
We need a better mechanism to cache and reuse containers so we don't take the performance penalty that causes bad latency on every request, similar to warm start.

Duplicate of #239 Closing


security group particle

particle, test, example
small update to sqs to use pcf_util.param_filter
deepdiff to requirements.txt

ah what i meant was to abstract this method.
            vpc_parents = list(filter(lambda x: x.flavor == vpc.flavor, parents))
            if len(vpc_parents) == 1:
                vpc_parents[0].sync_state()
                return vpc_parents[0].vpc_id
        raise InvalidConfigException("You need to specify either a VpcId or have a vpc as a parent")

into util but that it would for any flavor.
so the function would be get_flavor snd would take in flavor and a list. and it could even take in a 3rd param that would return an attr like ("vpc_id) in this case or otherwise it would just return the list


On Android in Talkbalk (screenreader) it is not possible to select a suggestion from the list

Tested this on a Android phone width Talkback  14.1 (currently the latest) on Chrome .
I can type something in the input field, but if I want to select something from the suggestions it selects the whole UL listbox, and it is not possible to select one item.
Turns out the the LI items in the suggestion list must be put in the tab sequence.
If you add the following in the _ITEM function in awesomplete.js, below line 381, it works fine:
"tabindex": "0",
Although Awesomplete works great, the whole project could use a bit of LOVE.

95% of the issues can be closed with the remark: 'functions as designed'   or   'closed due to 2 years of inactivity'   or 'won't fix'  or 'workaround can be made with a little bit of effort'. Currently the project looks abandoned.
The listLabel text can be configured, but the other a11y text (status.textContent) not. This would be useful if you want tot use Awesomplete in a different language with a screenreader. Not every blind person understands English, if you pardon my French :-)
And a minor bug: in the destroy function, the attribute 'aria-autocomplete' is removed (line 227), but this attribute is never set by Awesomplete. Looks like it is missing around line 22. And shouldn't the other attributes (aria-owns for example) be removed in the destroy function as well?

TIA,
  Nico

PR is merged, so this ticket can be closed.
Thanks


Put bad attribute back

The current fix for backwards compatibility breaks the logout flow.
traitlets will ignore any attribute in jupyterhub_config.py that's not defined on the class. When the typo was fixed, it meant that the bad attribute name couldn't be set on the class anymore.
Currently, when the user logs out, the authenticator will check it and raise an AttributeError, sending a 500: Internal Server Error to the UI.
[I 2020-04-21 21:05:01.922 JupyterHub samlauthenticator:732] User logged out: test.user
[E 2020-04-21 21:05:01.927 JupyterHub web:1792] Uncaught exception GET /hub/logout (::ffff:<IP_ADDRESS>)
    HTTPServerRequest(protocol='http', host='localhost:8000', method='GET', uri='/hub/logout', version='HTTP/1.1', remote_ip='::ffff:<IP_ADDRESS>')
    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/dist-packages/tornado/web.py", line 1703, in _execute
        result = await result
      File "/usr/local/lib/python3.6/dist-packages/samlauthenticator/samlauthenticator.py", line 754, in get
        forwad_on_logout = True if authenticator_self.slo_forwad_on_logout else False
    AttributeError: 'SAMLAuthenticator' object has no attribute 'slo_forwad_on_logout'

Developer Certificate of Origin Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors. 660 York Street, Suite 102, San Francisco, CA 94110 USA

Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed.

Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I have the right to submit it under the open source license indicated in the file; or

(b) The contribution is based upon previous work that, to the best of my knowledge, is covered under an appropriate open source license and I have the right under that license to submit that work with modifications, whether created in whole or in part by me, under the same open source license (unless I am permitted to submit under a different license), as indicated in the file; or

(c) The contribution was provided directly to me by some other person who certified (a), (b) or (c) and I have not modified it.

(d) I understand and agree that this project and the contribution are public and that a record of the contribution (including all personal information I submit with it, including my sign-off) is maintained indefinitely and may be redistributed consistent with this project or the open source license(s) involved.

Signed-off-by: JP Obley<EMAIL_ADDRESS>

Good catch. Is there a way that we can check this in a test?

Good catch. Is there a way that we can check this in a test?

Hmmm. A test for the configuration or the handler? Testing configuration feels like we'd be testing traitlets. Your instinct is right, though: this bug definitely would have been caught by a test case for the handler.
Writing a test for any of the handlers is made complicated by the handlers themselves being defined in the SAMLAuthenticator.get_handlers() method. We could move the handler definitions out of that function, either to a new module or keep them in the same one, and then have their __init__ methods take the authenticator instance as an argument.
What do you think? It would increase the size of this PR, but I can take a stab at it between changing diapers.

Nah - I think this is fine. Thanks for the work. I'll try to get this up later today.


Remove "turtle1" from lookup_transform_full

Thanks for noticing that! The same errors appears to be in the rolling and galactic distros too. We will need to backport it there afterwards.

@Mergifyio backport rolling galactic


Unable to change the padding between stars

Version 2.0.0 is now published with a better rating bar and proper default padding


AttributeError: 'PGVector' object has no attribute 'CollectionStore'. Did you mean: 'collection_name'?

System Info
Langchain version. - 0.0.326
Python - 3.9.6
Who can help?
No response
Information

[X] The official example notebooks/scripts
[X] My own modified scripts

Related Components

[ ] LLMs/Chat Models
[ ] Embedding Models
[ ] Prompts / Prompt Templates / Prompt Selectors
[ ] Output Parsers
[ ] Document Loaders
[X] Vector Stores / Retrievers
[ ] Memory
[ ] Agents / Agent Executors
[ ] Tools / Toolkits
[ ] Chains
[ ] Callbacks/Tracing
[ ] Async

Reproduction
embeddings = HuggingFaceEmbeddings(model_name='all-MiniLM-L6-v2')
CONNECTION_STRING = "postgresql+psycopg2://hwc@localhost:5432/test3"
COLLECTION_NAME = "state_of_the_union_test"
vectorestore = PGVector.from_documents(
embedding=embeddings,
documents=docs,
collection_name=COLLECTION_NAME,
connection_string=CONNECTION_STRING,
)
Getting below error
line 221, in get_collection
return self.CollectionStore.get_by_name(session, self.collection_name)
^^^^^^^^^^^^^^^^^^^^
AttributeError: 'PGVector' object has no attribute 'CollectionStore'. Did you mean: 'collection_name'?
Expected behavior
The embeddings should be stored in the database using pgvector.

seems like they deleted post_init by mistake see issue:
https://github.com/langchain-ai/langchain/issues/12616

After I downgraded my version to 0.0.229, it is working. So Fix is needed for latest versions.

upgrading to 0.0.327 didn't fix it either. only downgrading worked.

Also facing same issue with 0.0.327

Can confirm the above mentioned error:
AttributeError: 'PGVector' object has no attribute 'CollectionStore'
with langchain version 0.0.327 and Python 3.9.18
definitely worked with version 0.0.315

Can confirm the above mentioned error:
AttributeError: 'PGVector' object has no attribute 'CollectionStore'
with langchain version 0.0.327 and Python 3.9.18
definitely worked with version 0.0.315

I can confirm that with the upgrade to v0.0.329 this problem is SOLVED.


Added periodic offset committing

One of the things I found during the course of performance testing was that, for my use case at least, the consumers performed better if committing the offset was done periodically vs for every single message.  So I wrote this pretty self-contained object to provide that capability.  May be useful to add to the library.

This is great but I really think this is out of scope of the library. It is quite easy to do the same in the application code. Thanks!


codeowners -o '(unowned)' or something similar should work too

-o '' shows all

Good idea – I added a new flag (-u / --unowned). Included in the v1.0.0 released which you can get from the homebrew tap.


Abhi580 migrate microsoft.migrate nonbreaking 2023 05 01 preview

ARM (Control Plane) API Specification Update Pull Request

[!TIP]
Overwhelmed by all this guidance? See the Getting help section at the bottom of this PR description.

[!NOTE]
As of January 2024 there is no PR assignee. This is expected. See https://aka.ms/azsdk/pr-arm-review.

PR review workflow diagram
Please understand this diagram before proceeding. It explains how to get your PR approved & merged.

Click here to see the details of Step 1, Breaking Changes review
If you are in purview of Step 1 of the diagram, follow the Breaking Changes review process.
IMPORTANT! This applies even if you believe your PR was mislabeled, for any reason, including tool failure.

Click here to see the details of Step 2, ARM review
See https://aka.ms/azsdk/pr-arm-review.

 Click here to see the diagram footnotes
Diagram footnotes
[1] See ARM review queue (for PR merge queues, see [2]).
[2] public repo merge queue, private repo merge queue (for ARM review queue, [1])
The ARM reviewer on-call engineer visits the merge queue twice a day, so the approximate ETA for merges is 12 - 24 hours.

Purpose of this PR
What's the purpose of this PR? Check the specific option that applies. This is mandatory!

[ ] New resource provider.
[x] New API version for an existing resource provider. (If API spec is not defined in TypeSpec, the PR should have been generated using OpenAPI Hub).
[ ] Update existing version for a new feature. (This is applicable only when you are revising a private preview API version.)
[ ] Update existing version to fix OpenAPI spec quality issues in S360.
[ ] Other, please clarify:

edit this with your clarification

Due diligence checklist
To merge this PR, you must go through the following checklist and confirm you understood
and followed the instructions by checking all the boxes:

[x] I confirm this PR is modifying Azure Resource Manager (ARM) related specifications, and not data plane related specifications.
[x] I have reviewed following Resource Provider guidelines, including
ARM resource provider contract and
REST guidelines (estimated time: 4 hours).
I understand this is required before I can proceed to the diagram Step 2, "ARM API changes review", for this PR.

Additional information

 Viewing API changes
For convenient view of the API changes made by this PR, refer to the URLs provided in the table
in the Generated ApiView comment added to this PR. You can use ApiView to show API versions diff.

Suppressing failures
If one or multiple validation error/warning suppression(s) is detected in your PR, please follow the
suppressions guide to get approval.

Getting help

First, please carefully read through this PR description, from top to bottom. Please fill out the Purpose of this PR and Due diligence checklist.
To understand what you must do next to merge this PR, see the Next Steps to Merge comment. It will appear within few minutes of submitting this PR and will continue to be up-to-date with current PR state.
For guidance on fixing this PR CI check failures, see the hyperlinks provided in given failure
and https://aka.ms/ci-fix.
For help with PR workflow diagram Step 2 (ARM review), see https://aka.ms/azsdk/pr-arm-review.
If the PR CI checks appear to be stuck in queued state, please add a comment with contents /azp run.
This should result in a new comment denoting a PR validation pipeline has started and the checks should be updated after few minutes.
If the help provided by the previous points is not enough, post to https://aka.ms/azsdk/support/specreview-channel and link to this PR.

Please do not merge the PR. Service changes are yet to go in.

We are being flagged for 2 reasons in this PR:

Typespec file needed for review : We are not checking in typespec files previously from Migrate and only the json
files. Will we be able to get an exception on this from ARM reviewers.

There is an issue with Assessment Project modeling that is coming from previous versions, looks like a legacy
implementation but being flagged in this Version. Will we be able to get exception for this as well from ARM
reviewers.

We wanted to get an exception for the above issues since we just own this feature for which we have to publish swaager and not the entire RP. Can we get the exception ?

We are being flagged for 2 reasons in this PR:

Typespec file needed for review : We are not checking in typespec files previously from Migrate and only the json
files. Will we be able to get an exception on this from ARM reviewers.
There is an issue with Assessment Project modeling that is coming from previous versions, looks like a legacy
implementation but being flagged in this Version. Will we be able to get exception for this as well from ARM
reviewers.

We wanted to get an exception for the above issues since we just own this feature for which we have to publish swaager and not the entire RP. Can we get the exception ?

Any Update on this ?

LintDiff error TagsAreNotAllowedForProxyResources might be a false positive:
https://github.com/Azure/azure-openapi-validator/issues/741

/pr RequestMerge


Performance in the Wild

Several of us have brought up runtimes as an issue, either with our own work or something that could hurt adoption. This is a large topic that will be addressed by several efforts with time, so some patience is warranted. I'm starting this thread to capture the various discussions.
Marcel (@mpvl) has put exceptional effort into Cue and we should recognize this first. We owe you many thanks. Marcel is also the most knowledgeable about and the primary implementor of the algorithmic side of the answer to performance issues. Part of the goal here is to help him understand where the issues are.
There are two main ways to look at understanding performance issues:

Helping developers of Cue understand where performance issues arise
Helping users of Cue understand best practices for performant Cue

I believe we should sideline (2) for the time being as the guidance will likely change as we address (1).
Insofar as (1) is concerned, we can note the following:

The current performance situation is well known and expected given the rewrite of the evaluator. There are also some known performance improvements to be reimplemented.
Memory use is a primary driver right now. The additional bookkeeping is the main culprit here.
Most performance issues have been noticed in large configurations, typically reported by users in slack.

Benchmarking (via the Go tool) has not been implemented yet. This is a good first step. We have been mentally building a list of known issues and we should start collecting them in the repository. We will likely want to benchmark at both the function level and by looking at complex configurations run by the tool, though we likely don't want to include large configurations that are more typical in the wild.
Beyond benchmarking, we will want a way to help users identify and report issues with their own Cue. Some things that come to mind here:
A. Enable Go's perf tools from the Cue CLI / API
B. Expose some Cue logical level statistics from the CLI / API
a. Marcel has some stats objects in the evaluator that could be printed (?)
b. Some higher level stats like input / output nodes
C. Have some documentation on the above and how to contribute benchmarks for the repo
I'll leave it there for now and look forward to your additional insights. Please correct me anywhere I have misspoken.

I've just checked in a series of CLs that should add another big performance boost. For some larger configurations this let to about a 95% reduction in run time. Moreover, though, these CLs allow for a host more of improvements that will lead to better runtimes and memory consumption.
Things still to do:

use merge sort for sorting Arcs
special-case list processing
push-based, instead of demand-based updates
reduce size of Vertex
don't fully process fields if it is not strictly necessary:

while evaluating disjuncts, stop processing on error
when evaluating definitions, just do enough to collect conjuncts.
stop processing disjuncts when it is known they cannot become concrete

discriminator fields
various other control flow optimizations
optimize boolean computation
special-case arithmetic of small numbers
less memory allocation

The main order of business now is avoid the big slowdowns. Really the best way to help there is to come with complete reproducers to reproduce any performance issues.

I've just checked in a series of CLs that should add another big performance boost. For some larger configurations this let to about a 95% reduction in run time. Moreover, though, these CLs allow for a host more of improvements that will lead to better runtimes and memory consumption.
Things still to do:

use merge sort for sorting Arcs
special-case list processing
push-based, instead of demand-based updates
reduce size of Vertex
don't fully process fields if it is not strictly necessary:

while evaluating disjuncts, stop processing on error
when evaluating definitions, just do enough to collect conjuncts.
stop processing disjuncts when it is known they cannot become concrete

discriminator fields
various other control flow optimizations
optimize boolean computation
special-case arithmetic of small numbers
less memory allocation

The main order of business now is avoid the big slowdowns. Really the best way to help there is to come with complete reproducers to reproduce any performance issues.

Memory use is a primary driver right now. The additional bookkeeping is the main culprit here.

This is not correct: the additional bookkeeping has been vastly improved earlier and now it is merely a constant overhead. The large memory overhead that is/was witnessed is most likely a consequence of algorithmic inefficiencies and duplicating work.

Benchmarking (via the Go tool) has not been implemented yet.

Benchmarks have been implemented. If you mean profiling, then this is implemented through the Go testing framework. We have been manually adding the few lines to profile it from a command line, but it makes sense to enable this for everyone. Same for stats.

Memory use is a primary driver right now. The additional bookkeeping is the main culprit here.

This is not correct: the additional bookkeeping has been vastly improved earlier and now it is merely a constant overhead. The large memory overhead that is/was witnessed is most likely a consequence of algorithmic inefficiencies and duplicating work.

Benchmarking (via the Go tool) has not been implemented yet.

Benchmarks have been implemented. If you mean profiling, then this is implemented through the Go testing framework. We have been manually adding the few lines to profile it from a command line, but it makes sense to enable this for everyone. Same for stats.

Go 1.16's new runtime/metrics looks nice once Cue is 1.16+ only
https://golang.org/pkg/runtime/metrics/

This issue has been migrated to https://github.com/cue-lang/cue/issues/630.
For more details about CUE's migration to a new home, please see https://github.com/cue-lang/cue/issues/1078.


Release version (0.2.6) of VSCode plugin not working

In the current (at time of writing, v0.2.6) release version of the VSCode plugin, the example providers do not work. Debug log shows:
[providerClient(https://openctx.org/npm/@openctx/provider-hello-world)]: checking provider capabilities
[providerClient(https://openctx.org/npm/@openctx/provider-links)]: checking provider capabilities
[providerClient(https://openctx.org/npm/@openctx/provider-hello-world)]: failed to get provider capabilities: TypeError: (intermediate value).capabilities is not a function
failed to call provider: TypeError: (intermediate value).capabilities is not a function
[providerClient(https://openctx.org/npm/@openctx/provider-links)]: failed to get provider capabilities: TypeError: (intermediate value).capabilities is not a function
failed to call provider: TypeError: (intermediate value).capabilities is not a function

Upgrading to the preview version (v0.3.1716303872 (pre-release)) of the plugin seems to work though!

Thank you @cmars I will get someone to check this out.

@cmars Thanks for the report. I just published a new version of the VS Code extension (0.4.0). We made some changes to the OpenCtx provider API to add rename capabilities to meta, and I forgot to publish a new release extension version.
I just confirmed it's fixed in the release version on the VS Code marketplace, although I still need to fix it on the Open VSX marketplace (https://github.com/sourcegraph/openctx/actions/runs/9219687004/job/25365267401).


Avoid exceptions when return value is a list

This prevents exceptions where lists get interpreted incorrectly as expressions.
Before:

After:

Hi Paul, good catch and thank you for fixing it. I added you to the contributors so you can commit directly to the repository.