Allow theme = ["base", "my-theme" ] (aka theme composition and inheritance)

I know this has been requested before, in the form of some extends = some-other-theme.
This isn't "theme inheritance" in its purest form (which would be "theme1 extends theme2 extends theme3"), more like theme composition. Which is much simpler to understand/implement, but with most of the added benefits.
This relates to my work on https://github.com/bep/html5up-to-hugo -- which, with the current Hugo, becomes less elegant than it could be.
The simple rule is that the themes' files will form a big overlay/union file system from left to right.
So:
theme = ["base", "my-theme" ]

All files in base will be used unless there is a file with the same name in my-theme.
This will work with layouts, data, i18n, static (and in the future: content etc.)

What do you say?

Note that this is a "proposal", I'm not done with my part of the thinking here -- maybe this belongs in the theme itself etc...

One thing I see on the forum a lot is, "how can I make page X look different from the rest of the site". Could this facilitate styling a page like that, kind of overriding the theme applied to most of the site?

From @bep's description, yes it does. You could for example have your own version of single.html or of a shortcode.
I can see how this is useful. It's the same mechanic that Wordpress uses in creating 'child-themes', but I am not a fan. I remember losing track of whether I was editing the right files.
It's a personal perspective. Others will make it useful and may help produce more themes for Hugo. I don't have the knowledge to comment on the implementation. So I will abstain from voting and may even end up using it anyway.

+1 if I can have my personal "theme" of partial and shortcode collection, and can keep that theme as the secondary theme.
Thinking of something like:
theme = ["actual-theme", "partial-shortcode-collection-theme" ]

I've wanted this feature for a while. I had tried symlinking my commonly used partials/shortcodes from a common git repo, but symlinks don't work.

Child themes are a great idea. They should make it easier to get started with Hugo for beginners. And they probably also make it a lot easier for people to update the original, parent theme.

@RickCogley not sure about your example, but @kaushalmodi 's example is a primary use case, and I think this becomes super-powerful when we get proper dependency management, i.e:
theme = ["https://github.com/spf13/hyde/v2", "https://github.com/kaushalmodi/shortcode-pack/v1" ]

When Russ Cox is done with his brilliant thinking, I'm going to steal his thoughts: https://research.swtch.com/vgo
To me, this is mostly about DRY. On the project side, we can already today extend the theme by adding a single.html or overriding a partial etc. But that is not an option. So we get themes like hydex and hydey -- only slightly different ports of the same theme. If you want to create a new hyde theme with some blue colour, this should be possible.
Thinking about it, I think I will try to get this to work both on theme and in project.
So in theme.toml for my new Hyde theme hyde32:
# This theme is based on SPF13's port of Hyde, but with prettier colours.
theme = ["hyde"]

And then in my config.toml:
theme = ["hyde32", "partial-shortcode-collection-theme"]

The ordered set of file collections will then be:
hyde, hyde32, partial-shortcode-collection-theme

In its first iteration, the end user is responsible for pulling the above 3 themes into /themes -- but the ultimate goal here is to do:
# Shallow clone, no themes (only referenced in config.toml)
git clone https://github.com/bep/my-site
cd my-site
# hugo will download dependencies if not cached.
hugo

I think it will be very useful especially with output formats. If want to add an "api" dimension to your content, all you'd have to do is add a special "api" theme and reference it. Rather than single handedly adding the json templates in your layouts/_default or other.

Also it will allow theme developer to release several "extension" for their theme which not everyone will need. Like a "photo gallery" extension or a "Dentist" extension for a theme whose base is made for broader "health" entities.

This would be very useful. My use case is similar to @kaushalmodi  in that I keep a set of really basic files (_default layouts, etc.) to reuse on all projects and then layouts/themes for specific types of projects on top of that.

A development tip: Features like this are hard to get going unless you have a solid and failing test up and running:
const (
		themeStandalone = `
			name = "Theme Standalone"
`
		themeCyclic = `
			name = "Theme Cyclic"
			theme = "theme3"
`
		theme1 = `
			name = "Theme #1"
`

		theme2 = `
			name = "Theme #2"
			theme = "theme1"
`

		theme3 = `
			name = "Theme #3"
			theme = ["theme2", "themeStandalone", "themeCyclic"]
`

		theme4 = `
			name = "Theme #4"
			theme = "theme3"
`

		site1 = `
			theme = "theme4"
`
		site2 = `
			theme = ["theme2", "themeStandalone"]
`
	)

With a test that fails to build site1 and site2 -- then the fun starts! It is a threshold of boring work to get there.

On more use case is "shortcode" bundles. Your theme layout files would only contains shortcode files and people could use them by just adding your theme to their "theme set" and easily upgrade with future updates.

I just figured out that this would also be really cool for bundling of shortcodes!
:-)

Or bundling of output formats templates ! 🎉
You could even design an "Algolia" search theme "addon" along with script, css, template, output format template etc...
So much possibilities!

Yep, all those things that we copy from theme to theme, often without changing a single line.  Shortcodes, output formats. Heck even "base" for many.
Note if it makes dealing with static asset customization (like a site font file or landing image

Just a quick note that this is still pretty much priority.
In Hugo 0.38 themes got config.toml with namespaced .Params etc. and this will, of course, extend to these composites. Which will make something really powerful.

Can't wait for @kaushalmodi to make a mini-theme out of his debug functionalities! (debugprint and so forth...)

@regisphilibert Thanks! By now, I have couple of "mini-themes" I'd like to roll out once this feature is out:

Debugging (debugprint.html partial, Kind/Type/etc. debug info, etc.)
ATOM as custom output format
Better versions of internal shortcodes and templates: figure, opengraph, twitter_cards, ..

@kaushalmodi @regisphilibert coming back to this issue after some months, I was a little big confused about my own writing above.
Given:
theme = ["base", "my-theme" ]

On duplicates, what theme would you say should win? base or my-theme?

👍 my-theme

@bep Of course my-theme.
I can visualize that setup as this pseudo code:
theme my-theme extends base-theme
  function single.html
    """
    Only single.html from my-theme overriding that from base-theme
    """

If my-theme is overriding unwanted layouts and config params, user should trim them out.

I guess the theme names masked my real question, how about this:
theme = ["theme1", "theme2" ]

Who should win? Is it now obvious that "theme1" is the base theme? I obviously thought so back in February, just wanted to make sure ...

Be consistent with #4436

how about this:

theme = ["theme1", "theme2" ]

I think either direction (left to right, or right to left precedence) would work; just needs to be documented.
But I would suggest left to right increasing precedence order.. here's a reason:
A user could have this initially:
theme = ["some-theme"]

Then they choose to steal certain layouts from a different theme. So they then have:
theme = ["some-theme", "other-theme"]

i.e. they just need to append that to the right. So the delta of change only happen towards the right of that line.
Later, they tweak some portion of that "other-theme" and override just the RSS layout, so it will look like..
theme = ["some-theme", "other-theme", "my-rss"]

What I am getting at is that the initial theme = ["some-theme" part stays constant throughout. If the precendence is reversed, the base theme position will keep moving around.

theme-2 👍
It is my understanding that in any system, hardware of software, the layer on top will hide/override the layers below. As a left to right reader, it therefore makes sense to me that theme-2 is the layer on top.
Ultimately of course and that goes without saying, project's layouts should remain the utlimate rulers of this ecosystem :)

One last question. Since I wrote my test on this, a theme can now have confit.toml (i.e. site configuration). I also has a theme.toml, which is information about the theme -- we use this info to build the theme site etc.
So,
With this new feature both the project and the themes can be ... themes.
The question is:
Where to put the themes' theme configuration. theme.toml or config.toml?

With this new feature both the project and the themes can be ... themes.

Wait a minute, why?

Wait a minute, why?

I added a missing "d" to the above, which may clear things up.
The obvious motivation behind this is to create a new hyde port without having to copy-and-paste everything.
So I create my hyde-z theme and configure it to be based on some other theme: theme = hyde-x

theme = ["some-theme", "other-theme", "my-rss"]

Yes, that makes sense. Another obvious use case would be shortcode collections, so:
theme = ["my-theme", "cool-shortcodes"]

In the above, on duplicate shortcodes, the right will win. If the shortcode exists in the project, that will of course win (as it is today).

I see two ways to do this.
The old way:
Add your layouts files in layouts directory and set your theme as hyde-x:
themes = 'hyde-x'
The new way:
Create a /themes/hyde-z directory alongside themes/hyde-x and set theme both in your config.toml:
themes = ['hyde-x', 'hyde-z']
In both case, your project is not a theme, so I am still confused about your previous sentence :/

Where to put the themes' theme configuration. theme.toml or config.toml? (and please don't suggest that we merge those files into one ...)

I honestly think of theme.toml less of a functional config and more of a meta-data config.
I think the theme should go in config.toml. Now.. with that, I believe that if the theme's config.toml has:
# theme-x's config.toml
theme = [ "theme-x", "cool-shortcodes"]

and user's site config.toml has:
# user's site config.toml
theme = [ "_this_", "my-shortcodes"]

Looks like we will need a special placeholder like "_this_"? Otherwise, how will "theme-x" refer just to its own stuff without including the extensions?
Does that effectively become:
# user's site config.toml
theme = [ "theme-x", "cool-shortcodes", "my-shortcodes"]

I understand now. I always assumed the user would add themes’ port himself in his config.toml.
I don’t think a theme should be able to set itself as a layer of another theme. This is prone to inheritance being unknown to the user.
The port should let the user know he/she needs both themes in his themes directory and reference them both in his project’s config.toml this order.

@regisphilibert your project is not a theme, but it can be themed ... Which in my English means that you can configure it to use a theme.
@kaushalmodi
If project is:
theme = [ "my-theme", "my-shortcodes"]

And theme is:
theme = [ "theme-x", "even-cooler-shortcodes"]

The final graph will be:
[ "theme-x", "even-cooler-shortcodes", "my-shortcodes"]
The above assumes that "theme-x" is shallow.

@regisphilibert maybe ... I'm not totally convinced either way. We have lots of very similar themes as it is today. People create a new theme just to change some minor thing. And since you currently would have to do manual clones, It would not be "unknown to the user".

I'm on a move, so cannot articulate properly right now.. may be do something like:
# theme config
theme_extensions = ["a", "b"]

If user uses that theme without modifying theme-extensions, they inherit the theme's settings as they do now.
If user wants to add their own extensions, they need to do:
# user site config
theme_extensions = ["a", "b","c"]

In both case, your project is not a theme,

But it can be themeD?

... that said, I see enough questions in the above conclude what we can wait with the "inheritance" part, and start with the simpler setup where a project only can use themes (i.e. composition).

Though, don't you think that having an extra theme_extensions for list config param will make this easier and leave the theme as a single string value.

Though, don't you think that having an extra theme_extensions

What is the difference between a theme and a theme_extension? Why create one more term if they are the same? "Themes can be composed" is simpler to explain to people than "a project can have a theme. You can also add theme_extensions, which is the same as a theme, but just named differently."

It will blend into the theme extension inheritance that I talked about in my comment earlier.
With just one theme param, it will get difficult if users want to remove one or more of the extensions from the original theme.
Having the extensions, using the same example as earlier, user can set the extensions param as follows:

[] # empty
a
a,b,c
c
if they don't set, they get the extensions the theme defined (a,b).

@kaushalmodi you have a point, but if we start with "theme in project only", this can wait. It is a simpler model.
If we somehow need to control order at some point, we could allow the theme as a more complex structure:
[themes]
[themes.my-theme]
weight = 1

Which then could override the default right-to-left order you get.

What if the user wants to disable one or more of the theme's default extension? Are you planning to support that?

No (or maybe, but certainly not in the first iteration).
You started the talk about extensions and extensions in the scope as something optional, not me.
We may leave the "allow theme to set themes" in the first round, but my general view on this is that:
If a theme theme-a says that it depends on theme-b then  theme = theme-a ∪ theme-b. You cannot remove theme-b from the theme and expect it to work. This brings in a relevant discussion about transitive dependencies in Go programs (see #4754). It becomes hard when you want to version them etc., but you cannot ignore the problem. And you cannot remove dependencies and expect the program to compile.
But we will have to start simple here. If Hugo gets really popular we can maybe adopt vgo's model ad get versioned imports and hugo get etc.
That said, I have been thinking a little about this when sleeping. And I think we're going to reverse the order of precedence in the theme slice compared to what we talked about above.
If may be slightly less intuitive, but it will be consistent with #4436 (/cc @vassudanagunta) and it will solve another problem.
So in the project's config.toml:

theme = ["cool-shortcodes", "the-theme", "my-base-theme"]

The precedence is from left to right.

I have put cool-shortcodes first, because I want my shortcodes to be used and not the collection provided by the-theme.
my-base-theme is to the far right with the lowest precedence.
This also works mentally when you think about the project as the ultimate winner.

Also note that we currently do not have a "global identifier" for the themes, so the theme name whatever you name it when you clone it on disk. So you can certainly exclude a theme by putting some empty theme in the named folder. This is another big question that we need to keep out of this discussion.

What if the user wants to disable one or more of the theme's default extension? Are you planning to support that?

No (or maybe, but certainly not in the first iteration).
You started the talk about extensions and extensions in the scope as something optional, not me.
We may leave the "allow theme to set themes" out in the first round, but my general view on this is that:
If a theme theme-a says that it depends on theme-b then  theme = theme-a ∪ theme-b. You cannot remove theme-b from the theme and expect it to work. This brings in a relevant discussion about transitive dependencies in Go programs (see #4754). It becomes hard when you want to version them etc., but you cannot ignore the problem. And you cannot remove dependencies and expect the program to compile.
But we will have to start simple here. If Hugo gets really popular we can maybe adopt vgo's model and get versioned imports and hugo get etc.
That said, I have been thinking a little about this when sleeping. And I think we're going to reverse the order of precedence in the theme slice compared to what we talked about above.
It may be slightly less intuitive, but it will be consistent with #4436 (/cc @vassudanagunta) and it will make it more clear when you add "extension type themes" to the mix.
So in the project's config.toml:
theme = ["cool-shortcodes", "the-theme", "my-base-theme"]`

The theme's config.toml:
theme = ["my-cool-shortcodes", "cool-shortcodes"]`

The precedence is from left to right.

I have put cool-shortcodes first, because I want my shortcodes to be used and not the collection (my-cool-shortcodes) provided by the-theme.
cool-shortcodes is also included in the theme, but will not be included twice.
my-base-theme is to the far right with the lowest precedence.
This also works mentally when you think about the project as the ultimate winner (far left, before the = sign).

Also note that we currently do not have a "global identifier" for the themes, so the theme name whatever you name it when you clone it on disk. So you can certainly exclude a theme by putting some empty theme in the named folder. This is another big question that we need to keep out of this discussion.

So, I'm confused now, as of this commit:
Can themes declare their own "dependencies"?

Here comes another question, sorry.
If I understand correctly, theme's config.toml will bear its own theme declaration, we'll call it theme.theme. So will, as always, the project, we'll call it project.theme.
Now, If I understand the current state of inheritance/overriding of config.toml, the project's overrides the theme's.
So no matter what theme.theme is set to, will it not be systematically overridden by project.theme? This is undeniably what the user will expect anyway regarding any setting present in the theme's config.toml.

Can themes declare their own "dependencies"?

We may wait with that until a later release, but I think that will eventually happen.

So no matter what theme.theme is set to, will it not be systematically overridden by project.theme?

project.theme = something
theme.theme = something_totally_different

There is no systematic override in the above?

There is no systematic override in the above?

That is what I thought. Unless the slices will be merged somehow? Every project needing a theme will set a project.theme, thus overriding the theme.theme of its needed theme (which was used to declare some sort of theme dependencies)

That is what I thought. Unless the slices will be merged somehow?

The slices will not be merged, but there 3 rules to be aware of. If you think of the slices from left to right, and continue that thought into the theme.

If both the project and the theme defines myshortcodes  as a dependency, it will only be included once (the first one)
If myshortcodes includes someother-myshortcodes which then includes myshortcodes it is of course also only included once.
All other overrides are done on file level. So layouts/shortcodes/myshort.html will, on duplicates, the left-most version win.

And note that myshortcodes is supposed to refer to the same theme, so it should not matter where it is defined. You just need it. In practice, it may be harder, with versioning etc., but we need to solve that problem later.

I don't know how these themes relate to each other, it seems really complicated. I like what I thought was the basic flow of left to right inheritance.
theme = ["theme-made-for-project", "maiki-shortcode-collection", "that-one-override-for-the-gallery"]

In a CMS like Drupal or WordPress the themes have a direct "lineage", meaning they refer to a single theme as a "parent", though they can be chained along. Anything not fitting in that chain is added as a plugin.
I like that, it is easy to grok and easy to remember. And I already kinda build themes that way, except for using git to layer them on top of each other.
I wanted to point this use-case, but also to comment on the ordering: it should be consistent with the logic behind template inheritance. The default archetype comes to mind, where it makes sense that it isn't overridden, but it appears to be inconsistent with how other templates work.
I feel if the project/base theme is listed last it will be harder to remember and explain )as if it were easy to explain now! Ha!). :smile:

it should be consistent with the logic behind template inheritance.

I think my last proposal is that. If we keep the archetype logic out of this (we really should), then the current order is 1. project 2. theme.
Adding plural theme support you get 1. project 2. themes (and also potentially themes' themes if we go down that route).
A single parent theme is probably easier to understand, but it gives away a ton of flexibility. I'm currently not a big theme user. But I see big needs in re-using and compose my project from components in a simple way (shortcodes etc.). This is the reason I spend time on this feature, any theme inheritance will be an added bonus.
I say that even a component with 1 shortcode is a theme. We could come up with some other name for that, but I suspect that would make this even harder to grasp.

I cannot decide what is the best presedence order.
Given the theme list below:
theme = ["theme1", "theme2", "theme3"]

I have deliberately chosen very generic theme names, but you could imagine base-theme, my-shortcode-collection or whatever.
The question is: When two themes provide the same file (e.g. layouts/_default/single.html, who should win? Left or right?

I did not expect that outcome :) Oh well.

I did not expect that outcome :) Oh well.

My head was the other way around when I first thought about this, but revisiting it some months later, it just looked backwards... Reading from left makes project extends theme1 extends theme2 extends theme3 work as a sentence.

Yes with the "extends" notion, it makes a lot of sense.
Here is a thought, I don't think it derails the issue, but stop me if it does.
Can we question the usage of the word "theme" for this config parameter? A theme is really something people take as a skin for their website. But now, it appears a lot of our themes will be something different. Like our already popular exemple of shortcode bundle, or output formats bundle.
I feel the word theme fails to encompass what this layout inheritance will really bring along. I think we need a word used as an umbrella for themes/plugins/addons/extension as it would make more sense in the long term.

Can we question the usage of the word "theme" for this config parameter?

You may question it ...
I think

It works if you think a theme as a composite of components and other themes:

theme = ["component", "theme2", "another-component"]

Put all of those together and you got your theme.
Also, I don't want 2 terms for this, and changing it would be a fair amount of work, and you have to be really convincing to get me down that road.

I think it works if you think a theme as a composite of components and other themes

Again put this way it makes more sense. This should be the way to put it in the doc when the time comes.
Thanks.

I've been thinking of it as everything is a set of one or more templates that operate on your content following a "theme". It could be visual layout, or shortcodes, or an output format, or all of those. So if we explain how we use "theme", it should be fairly straightforward.
I've read this conversation in a lot of projects over the years (naming and "theme"), and it is recurring issue that new users will likely know about themes solely via a PHP CMS. So they have to explain it a bit more. And we kinda have to explain everything a bit more with Hugo, it seems; a lot of folks don't quite grok the simplicity of a  "template + content" workflow. ^_^

Just a little heads up to people waiting for this: I'm very close to finishing this implementing this now. To get this done, I had to refactor and get real control of all the theme related to filesystem handling. With this issue closed, we will have a fully virtualized filesystem interaction without all the path logic sprinkled around.
And it seems to make Hugo faster, at least for sites with themes. The hugo docs builds about 15% faster on my MacBook.

https://github.com/gohugoio/hugo/issues/2639 cross-linking for reference

I'd like to help with the documentation of this, mainly because I'm still having a hard time wrapping my mind around useful cases. :-D

Is there any tutorial on how to use this function? I'm trying to use specific theme (hugo-resume) for 'about' page and common theme (AllinOne) for others. Will this function works for this situation?
I've tried this method to use two themes. But the structure is different and results in the wrong 'about' page.

theme:
  - hugo-resume
  - allinone

The first homonymous template/data/i18n match will overwrite the later.

@regisphilibert
Thank you for your reply! I'm the beginner of hugo and focus on contents only.
Not quite understand your comments above.

Do you mean put these in the config.toml?

theme:
  - hugo-resume
  - allinone

However, that's unexpected token.

What's the meaning of The first homonymous template/data/i18n match will overwrite the later.?
Do you mean that i18n can be used to specify different theme for different menu?
If so, how to do that?


Building server needs @todesktop/cli

So when trying to build the server of the last version 0.0.19, it shows:
$ yarn run build-ts
$ tsc --sourceMap false
/bin/sh: línea 1: todesktop: orden no encontrada
error Command failed with exit code 127.

So I installed the lib as global and it asks for a login.

Hey try using yarn package instead, I'm going to remove toDesktop

yarn make is also gone. I'm testing your solution now.
Quick answer!


Amplify CLI Plug-In: Amplify push doesn't push all resources copied over from amplify.copyBatch()

Describe the bug
When writing a plug-in Amplify push doesn't upload all the files (only the cloudformation templates) found in the backend directory created for the resource. This should be a desired behavior to upload anything that is written using amplify.copyBatch() to S3.
To Reproduce

Create a new amplify plugin not associated with any of the default Amplify support services.
Copy a few files over to the resource directory using copyBatch()
Observe only the template files being uploaded to S3.

Expected behavior
Upload all resources to S3 so they can be used. For example a zip file for deploying a lambda function.
Desktop (please complete the following information):

OS: Mac

Work around
Resources hosted in another bucket not great but used until this is fixed.

That's the intended behavior of the awscloudformation provider plugin.
For the third party plugin, if you want to upload other contents into the deployment bucket, you have two choices:

you can write the logic into the category plugin
you can write another provider plugin to upload the contents, and your category plugin uses the new provider plugin instead of the default awscloudformation provider plugin.

Should that be the intended behavior if we want to encourage users to use cloudformation as the default provider?

If moving forward on this it is decided that this is intended behavior then it should be documented.
Also, it should then be documented on how to communicate with AWS services. Including where to get credentials and should we use the default node-js AWS sdk.

@wizage I agree, we need code-specific documentation for our context/extensions which we expose to our plugins. Is there anything you're specifically blocked on?

@wizage That's awesome! Would be great to know about your plugin. Would probably be our first 3rd-party Amplify CLI plugin :D Super excited to learn more.


exercise API js added

round 2

We know this gets a response that provides workout data. Still need to go through the data passed back to pull out relevant information.


Correction for raw affiliation UMR MoISA (Montpellier Interdisciplinary centre on Sustainable Agri-food systems), Univ Montpellier, CIRAD, CIHEAM-IAMM, INRAE, Institut Agro Montpellier, IRD, Montpellier, France

Correction needed for raw affiliation UMR MoISA (Montpellier Interdisciplinary centre on Sustainable Agri-food systems), Univ Montpellier, CIRAD, CIHEAM-IAMM, INRAE, Institut Agro Montpellier, IRD, Montpellier, France
raw_affiliation_name: UMR MoISA (Montpellier Interdisciplinary centre on Sustainable Agri-food systems), Univ Montpellier, CIRAD, CIHEAM-IAMM, INRAE, Institut Agro Montpellier, IRD, Montpellier, France
new_rors: 05kpkpg04;04abkkz03;051escj72;05q3vnk25;03rnk6m14;0005r2j17;003vg9w96;011xg1225
previous_rors: 05kpkpg04;04abkkz03;051escj72;05q3vnk25;03rnk6m14;0005r2j17;003vg9w96
works_examples: W4399049632;W4402901640
contact: e8cea33d1d0af290b07f36ca349ca0d1:1725dcfe15303ae1 @ supagro.fr

This issue was accepted and ingested by the OpenAlex team on 2024-12-19. The new affiliations should be visible within the next 7 days.


AZCopy hangs under Windows Subsystem for Linux (WSL)

Which version of the AzCopy was used?
Note: The version is visible when running AzCopy without any argument
azcopy_linux_amd64_10.0.8
Which platform are you using? (ex: Windows, Mac, Linux)
Debian GNU/Linux 9 (stretch) under WSL 17134.950
What command did you run?
Note: Please remove the SAS to avoid exposing your credentials. If you cannot remember the exact command, please retrieve it from the beginning of the log file.
copy https://container.blob.core.windows.net/path/L008_001.R1.fastq.gz?REDACTED ./downloads/path/L008_001.R1.fastq.gz
What problem was encountered?
When specifying relative path "downloads" without a dot slash:  AZCopy prints Scanning and shows nothing, AZCopy consumes 30% more CPU and does nothing. After killing the process, CPU is returned and no files were started or queued
When specifying relative path "./downloads" with a dot slash as above:  AZCopy prints Scanning, creates a Job that does nothing, logs show pending job but does nothing, consumes 30% more CPU and does nothing. After killing the process, CPU is returned and no files were started or queued
How can we reproduce the problem in the simplest way?
run the out-of-the-box copy operation under WSL environment (as specified in https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-v10)
Have you found a mitigation/solution?
run the same script and sequence of steps under Debian GNU/Linux 9 (stretch) in a Google cloud VM with no issues
run the same file request using AZCopy embedded in Azure Storage Explorer

Thanks for the bug report @barry-at-technome-io.  We are currently putting the finishing touches on a re-write of the directory handling logic, which should (in theory) fix up inconsistencies like the one you report where it acts slightly differently depending on how the path is specified.
As for the 30% CPU issue, we'll test that and see if we can reproduce it.   I'm hoping, since you're on quite and old version, that it will turn out to be something that we have already fixed.  But we won't know that until we test with the new release that's currently in development.  (If you want, you can test with version 10.2.1 , which contains a lot of improvements over 10.0.8.  But the up-coming 10.3 will have even more).

Actually, I think the issue is very probably a known issue. The fix is coming: https://github.com/Azure/azure-storage-azcopy/pull/519
The fix to the known issue will be released in 10.3.  So I'd suggest you retest in 10.3 (once its released, in a small number of weeks from now), and then comment here to let us know if the problem is resolved.

@JohnRusk Having same issue on Windows Server 2012 R2 (2 CPUs) on VMWare. Details as follows ...

SET AZCOPY_SPA_CLIENT_SECRET=
AZCOPY LOGIN --service-principal --application-id  --tenant-id=
returns
Failed to perform login command: adal: Failed to execute the refresh request. Error = 'Post https://login.microsoftonline.com/e1nnnn-nnnnn-nnnn-nnnn-nnnnn/oauth2/token?api-version=1.0: dial tcp <IP_ADDRESS>:443: i/o timeout'

Windows Event Viewer ...
==> REQUEST/RESPONSE (Try=1/21.0252026s[SLOW >3s], OpTime=21.0272058s) -- REQUEST ERROR GET https://aka.ms/azcopyv10-version-metadata?timeout=901 User-Agent: [AzCopy/10.2.1 Azure-Storage/0.7 (go1.12; Windows_NT)] X-Ms-Client-Request-Id: [c7499e49-44c3-4766-7307-769f25b98352] X-Ms-Version:

@barry-at-technome-io Version 10.3 is now released. As noted above, I think if you retest on 10.3 the problems should be gone.  I'll close this thread on that assumption, but if they're not gone please comment and we'll reopen.


Accepting an invitation from a now-deactivated user produces a server error

It doesn't show any errors to the user registering, happily.  But we should add a check in process_new_human_user that prereg_user.referred_by isn't deactivated before trying to send the message:
https://github.com/zulip/zulip/blob/c97956c5ff9457e01e642a8f0695e97b44921f3a/zerver/lib/actions.py#L481-L490

@alxmv Is it that a user is considered deactivated if the is_active attribute of the User is set to False?

@Parth-Mittal-NITK: Yup!

@alexmv Ready for review

@zulipbot claim

Hey @alfredoosauce! Thanks for your interest!
However, I have been working on this issue and there is a PR linked to it. It's failing some tests and hence has taken a lot of time.

@mittal-parth Sounds good, how do I get unsigned to the project? Thanks

I've adjusted the assignees.  Thanks for being excited about the task, @alfredoosauce, and thanks for your ongoing work on it, @mittal-parth!

Thanks @alexmv

@alexmv ,  I want to help with this issue, can I claim it? Not sure if @mittal-parth is still working in this!!

Thanks for your enthusiasm, @Akshat1903!  However, #20206 by @mittal-parth was essentially done, so I fixed the one minor issue with it, rebased it, and I expect it will merge shortly.


Try out ghc 8.10.2

Addressed by https://github.com/input-output-hk/cardano-prelude/pull/136, closing.


note that JSON encoding is not sufficient

Arg, encodeURIComponent doesn't escape single quotes, so it's not good enough.

I will try to improve this issue. Give me couple of days. I have in tight deadline. I will get back you. And thanks for the help.


More detail for REST exceptions

We have written our own front-end for submitting items to DSpace (using VueJS). This works well except when there are exceptions, in which case we only get an HTTP code or a generic message such as "Request is invalid or incorrect". This makes it difficult for us to debug the error. We have our server hosted externally so we do not have easy access to the logs - and the logs don't show us the cause of the error unless debug/stacktrace is enabled.
For example, a recent issue was that the metadata registry had not been updated, and we were trying to write to non-existent meta fields. The REST response just gave us 500, and it took a long time to track down the error. Recently we have been getting 422 exceptions for some items, and simply have no idea what the problem is.
Describe the solution you'd like
For REST requests, I would like the cause exception message to be provided in the REST response. Often the message of the original exception is more than enough for us to fix the problem. (This message wouldn't be displayed to the user, but rather logged for us to investigate.)
Describe alternatives or workarounds you've considered
The only workaround we have currently is to ask our provider to scan the logs - in most cases this is also insufficient due to lack of detail in exceptions, especially 422 exceptions.

As an example, I've added the following to DSpaceApiExceptionControllerAdvice::sendErrorResponse():
boolean showCause = true; //TODO: fetch from settings

String adjustedMessage = message;
if (showCause) {
    Throwable rootCause = ex;
    while (rootCause.getCause() != null && rootCause.getCause() != rootCause) {
        rootCause = rootCause.getCause();
    }
    adjustedMessage = String.format("%s [CAUSE: %s]", message, rootCause.getMessage());
}

//Exception properties will be set by org.springframework.boot.web.support.ErrorPageFilter
response.sendError(statusCode, adjustedMessage);

This gives me a very useful error message:
An exception has occurred [CAUSE: bad_dublin_core schema=a.fake.key. Metadata field does not exist!]

Or even better - send a header to indicate you would like cause messages. That way no changes would be required to config.

As noted in your PR, this behavior is purposeful.  DSpace is avoiding "information exposure" by not including the exact messages in the stacktrace in our REST API responses.  See this comment on your PR: https://github.com/DSpace/DSpace/pull/8359#issuecomment-1162174580

We have deployed a patch to return the exception message for REST errors, and it is less satisfactory than what we had hoped. For example, we see things like this:
org.springframework.web.util.NestedServletException: Request processing failed; nested exception is javax.persistence.PersistenceException: org.hibernate.exception.SQLGrammarException: could not extract ResultSet
and
Error rolling back new item (xxx-xxxxx-xxx) files: Calling method 'get' is not valid without an active transaction (Current status: MARKED_ROLLBACK)
We do not know where or why these errors occurred. Stopping and restarting the server to enable stack tracing is not only infeasible for us, but also not useful - we cannot guarantee that we can reproduce the conditions that caused the error.
It seems to me that the solution, while painful, is to be more defensive before attempting an operation with known failure conditions. Writing to metadata is a good example; rather than waiting for an exception deep in the stack, an existence check could throw a more useful error regarding the absence of that field.


Addresses in Texas Like '# Country Road 212' should be parsed differently

So I am attempting to parse the address: "1 COUNTRY ROAD 212"
Which is giving the following output: (note I am parsing the result it into JSON format)
[{"AddressNumber": "1"}, {"StreetName": "COUNTRY"}, {"StreetNamePostType": "ROAD"}, {"OccupancyIdentifier": "212"}]

When in Texas it uses the Country Road # as a street name for the rural areas.
Output should be something like:
[{"AddressNumber": "1"}, {"StreetName": "COUNTRY ROAD 212"}]

I assume that something Country Rd 212 should be parsed with the same idea as above.
This is kind of a corner case, but I thought I should post an issue report anyways :+1:

thanks for reporting - we've run into issues with weird rural Texas roads before. what's the source of this address? do you have more addresses where this came from?

I can list out a bunch of addresses that are similar:
1901 Country Road 183 Carbon TX 76435
10 Country Road 3668 Splendora TX 77372
1954 Country Road 43 Greenville TX 75401
118 Country Road 46 Hondo TX 78861
14442 Country Road 546 Nevada TX 75173
It looks like from the data that I have there are also some with named country roads:
109 Country Road Ranchette Corsicana TX 75109
104 Country Road Ranchette Corsicana TX 75109
116 Country Road Ranchette Corsicana TX 75109
If I find any more, I will post them up in this same post.

didn't train on the country road ranchette addresses b/c I was unable to find the addresses on google maps, but now the parser is pretty good at recognizing 'country road' as StreetNamePreType. let us know if you run into other patterns that stump the parser, though

Awesome, looks great! Thanks a lot :+1: