Provide utilities for adding new files, modifying files, and adding directories

Many platforms require the addition of a platform-specific file that's not in a typical local Django project. For example, Fly.io requires the addition of a platform-specific Dockerfile.
Rather than leaving the mechanics of checking for an existing file and writing it, provide a utility. That way all platforms handle this task in a consistent manner, and it should be simpler to write plugins for new platforms.

[x] Sketch out the API for utility function add_file().
[x] Also consider top-level architecture around utils.

[x] self.sd.utils.<function-name>() is not very satisfying. Rather from ... import plugin_utils, plugin_utils.<function-name>().

[x] Write the function in core simple_deploy.
[x] Use the function in add_dockerfile()
[x] Use the function in all of Fly's platform_deployer module.
[x] Modify platforms to use this function:

[x] Fly.io
[x] Platform.sh
[x] Heroku

[x] Repeat for add_directory().

[x] Write function
[x] Update fly
[x] Platform.sh
[x] Heroku

[ ] Repeat for modify_file().

[ ] Write function
[ ] Update fly
[ ] Platform.sh
[ ] Heroku

[ ] Test locallly
[ ] e2e tests
[ ] Merge
[ ] Release?
[ ] Announce?
[ ] Open new issue for modifying existing file, ie settings.py.
[ ] Consider a utility function for adding a new directory; see platform.sh make_platform_dir().

Public function add_file()

Purpose: add a new file to the project.
Checks if the file already exists.
If existing file, gets permission to overwrite.
Exits if unable to add or overwrite file.
Input: path, contents
Build this out based on Fly's add_dockerfile().

Cleanup

[ ] Remove utils.write_file_from_template().

[ ] Make sure not used anywhere else.
[ ] Remove
[ ] Run tests

Reorganizing utils

[ ]  Start a plugin_utils module.
[ ] Remove references to sd_command where possible. For example, can I access messages without going through sd_command?
[ ] Remove functions from existing utils module.
[ ] Move utils.py to utils/sd_utils.py.

modify_file()

[ ] Write a utility function modify_file() that expects to act on an existing file. Just replace contents, and report the change in the output.


fix docs err

Description

Fixes zh-cn docs format
Checklist

[ ] I have run cargo make pr-flow
[x] I have reviewed my own code
[ ] I have added tests

Could you also add the missing </tr> tags?

Could you also add the missing </tr> tags?

ahhh, I mistakenly copy the code, I think it's fine now, thx


Remove ThePBone.GalaxyBudsClient 4.2-a0b1f7 to timschneeb.GalaxyBudsClient 4.2-a0b1f7

Microsoft Reviewers: Open in CodeFlow

URL: https://github.com/ThePBone/GalaxyBudsClient/releases/download/4.2/GalaxyBudsClient_Windows_64bit_Setup.exe
OK
(Automated message - build 887)


Java CI

Motivation and Context
Description
Opening a PR with initial CI changes to build and run tests against Java packages.
Runs:

Build Java Semantic Kernel
Run Java Integration Tests and Samples

Contribution Checklist

[ ] The code builds clean without any errors or warnings
[ ] The PR follows SK Contribution Guidelines (https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
[ ] The code follows the .NET coding conventions (https://learn.microsoft.com/dotnet/csharp/fundamentals/coding-style/coding-conventions) verified with dotnet format
[ ] All unit tests pass, and I have added new tests where possible
[ ] I didn't break anyone :smile:

All comments have been resolved. Talking with the team we decided that we will wait until our branch gets merge into upstream and we will create a new PR incorporating this changes.


Weaver Mobile Wallet

Clear and concise description of the problem
I think you should develop Weaver Mobile Wallet app. I think this is necessary.
Suggested solution
Weaver Mobile Wallet
Alternative
No response
Additional context
No response
Validations

[X] Read the docs.
[X] Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

haklisin orasi biraz karışık ,bakalım şans bizimle olsun

Weaver has wallet functionality, but to be precise, weaver is an integrated development environment.


Add quickfix support

I addeed loclist support on day 1, but not quickfix list support. Some people will probably like to use the quickfix list. Some support should be added for populating the quickfix list, and for automatically opening the list when there are errors.

This has been resolved already.


0.2.14 - another breaking change

Not sure if this is a bug, or something we need to reconfigure in our app (I assume bug since this is a patch), but https://github.com/angular-ui/ui-router/commit/da63b27b7bbd13d2633b68fa74606f977e6d4147#diff-6de0577dce2b7e2e060fb6f1b36a33e3R1628 ended up throwing errors on parameters that were previously working without issue.
We ended up locking ui-router to an older version.
As a noob I can't get too much into the details. The guru who would have dealt with this, @sdbondi, will be on leave until next week Monday.

Okay, unfortunately I can't really help you, since you've provided no details about the problem you're having.

ye, will get on this in a week or so

also experiencing this, will update once I figure out what on our end needed to be changed

pretty sure this is probably a duplicate of the other breaking bug. I suggest closing as such until more information is provided that it is a different bug

@larrybotha Doesn't this seem to be related to #1915 ?

Alright, closing unless any more information is supplied.


support streaming gzip middleware

Is your feature request related to a problem? Please describe.
Currently the gzip middleware does not support streaming.
repo: https://github.com/hertz-contrib/gzip
Describe the solution you'd like
Implementing gzip middleware based on writer hijacking. refer to https://www.cloudwego.io/zh/docs/hertz/tutorials/framework-exten/response_writer/.
Describe alternatives you've considered
Add streaming gzip middleware.
Additional context

https://github.com/gin-contrib/gzip

please assign to me

@chaoranz758 Any progress?


Call for maintainers

Similar to https://github.com/chjj/marked/issues/756, I would open a call for maintainers o collaborators on this project. I must to say in advance I'm sure if I would be a good candidate, both because I'm busy with NodeOS (and willing to have some collaborators there so I could have some life...) and because if I were one of the maintainers of blessed one of the first things I would do would be to move the widgets to independent projects and make a 0.5 release really minimalistic to allow more flexibility (like double buffering), and I know that chjj is not happy with both ideas in the current development status, but if the addition of collaborators would give some impetus to blessed I would definitely help to its development in some way or another.

I would be interested in helping maintain if that is a possibility (although I realize nothing is confirmed yet).  I love terminal-related projects, and I even composed interfaces like blessed with some raw methods before I knew it existed.  There is also lots of other JavaScript work on my profile if you are curious.  Thanks. :>

I would love to help maintain as well, although I am by no means an expert in the codebase. I use blessed actively for personal projects and would relish the chance to improve this library.

Why a 0.5 release? The next minor version is 0.2 :)
Anyway, seems like a good idea. Blessed could really use some love.

I am interested in maintening this project.

Thanks @Kakise :-) So where do you propose to start?

This repo seems dead. Did anyone ever start a separate fork for support purposes?

This repo seems dead. Did anyone ever start a separate fork for support purposes?

None that I would know, according to the forks network most of the alternatives are just pull-requests. Since I was the one that opened the issue I would start it, I'm a bit busy with NodeOS but as I've said before I have several ideas about how this module should move fordward. Do you agree?

@piranna What are your ideas? Although I lack the knowledge of the code base to be a maintainer at present 0 I've only used blessed casually in a couple of fun-time projects - I would like to see more activity around blessed in general, and would be willing to help make it happen. I think @chjj would generally sanction any reasonable plan to stimulate the ecosystem.
That being said, I would be willing to be a maintainer in the absence of any other volunteers - would also be willing to interview with other community members to assess my suitability.
I suggested in issue #256 the creation of an IRC channel - and chjj gave it his blessing on twitter - but I haven't created it yet as i'm not sure whether ##blessed or #blessed would be more appropriate - I think a single hash would be acceptable personally.

My main ideas are to split the widgets from the core on two or more different projects for clean-up both of code and APIs, and later move to a double buffer architecture where each widget draws on its own "screen" similar to how X11 and Plan9 rio works, that will make it easier to work with the widgets and also to do higher level tricks like transparency and alpha-blending, and finally add support CSS-like styles and HTML-like box model and CSS flexbox, so it will make it easier to program and use.

@piranna I've opened up ##blessed on freenode IRC for general discussion. Feel free to pop in and we can discuss more synchronously ideas for revitalizing this project.
One idea might be to set up an informal poll asking interested parties what they'd like to see happen with this project - better issue moderation, code cleanup, API restructuring, etc. Ideally a consensus could be reached about how to get things rolling without dividing potential contributors/users.

I agree on the pool, but I don't have availability for synchronous chat. I preffer to stick to async conversations over github issues.

@piranna Understandable, async is a good default.
In my mind the relevant questions re: revitalizing this project are who is willing to own a repo and coordinate activity (issues and PRs) while maintaining an open mind with regard to the direction of the project insofar as other contibutors are concerned?
I think it makes sense to defer to the existing contributors, so if @piranna or @Lambeaux want to create an organization or simply designate their fork as the de facto support fork, that ought to be sufficient, no?

I could be able to create a fork and an organization and manage the pull-requests, but due to lack of time I will not be able to lead the development. I could help to discuss the roadmap and the path of the project though, I think we need some guidelines about how it should move forward. What do you think?

I have created a new organization and moved my fork there, whoever that wants to be a collaborator or an owner please tell me and I'll add him there.

@piranna Apologies for the long silence, mind adding me as collaborator?

@piranna Apologies for the long silence, mind adding me as collaborator?

Done.

Any updates?


Not available to render example qmd file without installing additional components

Hi, thanks for nice work.
Problem
Recently I tried to use this template for practice my self.
Here's what I did.

install and load template using command quarto use tempalte dfolio/quarto-ieee
Just render ieee.qmd with Rstudio

However, this will return error message:

LaTeX Error: File orcidlink.sty not found.
Solution
and I found solution to solve this problem.

install tinytex via Console: tinytex::install_tinytex(), then check whether result of tinytex::is_tinytex() is TRUE
install orcidlink.sty via Console using tinytex

tinytex::parse_packages(
    text = "! LaTeX Error: File `orcidlink.sty' not found."
)

and it will return 'orcidlink' like below image

and then render qmd file again and it will work.

I think this process added into readme, this template will look much better. (If you don't mind i'll modify this and make PR)
Thanks.
Kim

Thank you for reporting this issue
As reported on quarto issue 877

Are you using TeX Live ? Automatic installation would be disabled if not using TeX Live as LaTeX distribution, or explicitly opt-out.
Quarto uses by default a Quarto engine that does try to install missing dependencies (https://quarto.org/docs/output-formats/pdf-engine.html#quarto-pdf-engine)

Initially, I didn't use TeX Live, but the packages that came with my Linux distribution. After switching to TeX Live, I don't have this problem anymore.
Second, to limit this problem, I explicitly set latex-auto-install: true in the _extension.yml. So it is surprising that such an issue still occurs.
So I guess this issue might be related to that.

My understanding is that
If I use quarto's default pdf engine (not TexLive or TinyTex), I won't have any problems.
Is that correct?
and As an additional dumb question 😅
Is it possible to just uninstall tinytex with a command
quarto uninstall tinytex

to use the default engine?
Thanks for kind response.
Kim.


new theme options

I have added the OneDark color scheme well-known as the color scheme for Atom. It's pretty popular, so I figured I would add it in.

Also edited the Jellyfin css to change the header color to --modal-header-color as well as the sidebar to match --modal-bg-color

Thanks for the new theme option!

Also edited the Jellyfin css to change the header color to --modal-header-color as well as the sidebar to match --modal-bg-color

What was wrong with the default blur on the side menu?

Thanks for the new theme option!

Also edited the Jellyfin css to change the header color to --modal-header-color as well as the sidebar to match --modal-bg-color

What was wrong with the default blur on the side menu?

I don't mind it; however, the header color doesn't change at all with this theme unless I make those changes in jellyfin-base.css. The same issue exists with the new Dracula theme, the header color just persists from the default Jellyfin theme if that makes sense. Theme likes Plex behave differently because of the blur, gradients, etc. Perhaps you can take a look at that if you have the time?

Also, once we figure out the next steps, I'll go ahead and compile screenshots for the theme to finalize the PR.

Thanks for the new theme option!

Also edited the Jellyfin css to change the header color to --modal-header-color as well as the sidebar to match --modal-bg-color

What was wrong with the default blur on the side menu?

I don't mind it; however, the header color doesn't change at all with this theme unless I make those changes in jellyfin-base.css. The same issue exists with the new Dracula theme, the header color just persists from the default Jellyfin theme if that makes sense. Theme likes Plex behave differently because of the blur, gradients, etc. Perhaps you can take a look at that if you have the time?
I'd say the sidebar change should for sure be added, but we need to figure out the header issue that exists with this theme and Dracula.

The header does change. It's partially transparent.
chrome

firefox

https://github.com/GilbN/theme.park/blob/master/CSS/themes/jellyfin/jellyfin-base.css#L134
Did you follow these install instructions or just standard subfilter?
https://docs.theme-park.dev/themes/jellyfin/#setup

Thanks for the new theme option!

Also edited the Jellyfin css to change the header color to --modal-header-color as well as the sidebar to match --modal-bg-color

What was wrong with the default blur on the side menu?

I don't mind it; however, the header color doesn't change at all with this theme unless I make those changes in jellyfin-base.css. The same issue exists with the new Dracula theme, the header color just persists from the default Jellyfin theme if that makes sense. Theme likes Plex behave differently because of the blur, gradients, etc. Perhaps you can take a look at that if you have the time?
I'd say the sidebar change should for sure be added, but we need to figure out the header issue that exists with this theme and Dracula.

The header does change. It's partially transparent.
chrome

firefox

https://github.com/GilbN/theme.park/blob/master/CSS/themes/jellyfin/jellyfin-base.css#L134
Did you follow these install instructions or just standard subfilter?
https://docs.theme-park.dev/themes/jellyfin/#setup

Oh. I was applying the style through nginx, not the app itself. I'll try it with that way.

Yup, this works as expected doing it directly through Jellyfin. I'm just curious if you know why that is?
Also, is it possible to fix the disparity between the rendering of firefox and chrome regarding the header? It's much better in Chrome, but I use Firefox

Okay so I looked into it and saw the bit about how this has to be changed manually in Firefox by setting layout.css.backdrop-filter.enabled to true. I just did that and it works as expected.
So now that we have figured that out, I don't at all mind if you discard my edit to jellyfin-base.css. What would you like me to do next for this PR? Screenshots for the docs?

Okay so I looked into it and saw the bit about how this has to be changed manually in Firefox by setting layout.css.backdrop-filter.enabled to true. I just did that and it works as expected.
So now that we have figured that out, I don't at all mind if you discard my edit to jellyfin-base.css. What would you like me to do next for this PR? Screenshots for the docs?

Great. Yes, screenshots would be great. And yeah you can just revert those jellyfin base changes.
I also need to add a fallback for firefox on the sidemenu as I see its just transparent without backdrop filter enabled.
If you want to add that too, go ahead.

Yup, this works as expected doing it directly through Jellyfin. I'm just curious if you know why that is?

iirc, the theme.park css gets loaded before the jellyfin css. They load some of it in the body not the head. So subfiltering the body would probably have worked just fine.

Alright I'll roll that back.
Also, just for giggles, I replaced </head> with </body> in the subfilter area for nginx and the same issue occurred, so I'm just gonna keep applying it through the app itself.
"I also need to add a fallback for firefox on the sidemenu as I see its just transparent without backdrop filter enabled."
Yeah that's something better suited for you than myself, I'm not much of a web dev at all.

Do you think the theme can be merged before screenshots are taken? I'm still getting my services up and running to be honest, though I have a friend with a lot and he could possibly take the screenshots for me.

And do you want me to make a tailored file for every service within the themes folder or is that done by you when themes are created? I noticed the current community themes do not have that yet.

If you don't run that many themable services just screenshot the ones you run and I'll fix the rest.
No, you just need to add it in the variables folder.

If you don't run that many themable services just screenshot the ones you run and I'll fix the rest.
No, you just need to add it in the variables folder.

Okay, how is it then invoked upon merging? Usually you just specify the theme in the themes folder which will link to the base file as well as its respective variable. Do I instead just specify two lines (the base file as well as the variable file) to bypass that for the time being?
I'll get started on the screenshots. Expect one for Pi-Hole, Portainer, Gitea, and Grafana.

Made some changes, where do you want the screenshots?

Here are some screenshots:
Pi-Hole

Portainer

Gitea

I'd save them locally for upload, not sure how long I'll keep them on imgur.

Here are some screenshots:
Pi-Hole

Portainer

Gitea

I'd save them locally for upload, not sure how long I'll keep them on imgur.

Thanks. Github has them now. So you can just remove them from imgur if you want.

I added "By Monklite" on the community page. If you dont want that let me know.
https://docs.theme-park.dev/community-themes/

Hello, How can this new theme can be installed/used ?

Depends on the install method. There are new installation docs.


Make Diagram deserializable

Right now, the train Deserialize is not implemented for Diagrams. This is preventing automatic derivation of the trait for AttachOption, which in turn is preventing us from replaying some labelled-related actions in the CLI. This should be fixed.

Nick wrote implementations of Deserialize for Diagram and Rewrite in #755. Unfortunately that will break on labelled because we need Label to implement Deserialize (and moreover the type of Label changes a few times in the commit history).

obake 👀👀👀

We discussed with Calin a way to add optional action dumps in .hom, with no stability guarantees.
I will obake the relevant data structures in homotopy-core to support backwards compatibility between beta and alpha as well, but only after the labelled data structures are confirmed as stable.


Update warning-explanations

made slight corrections to the use of English in the text.
IMO it reads slightly better, happy to discuss though :smile:

Thanks


hotfix/cougarCash -> dev

This change is

Reviewed 2 of 2 files at r1.
Review status: all files reviewed at latest revision, all discussions resolved.

Comments from Reviewable


Update UserGuide for V1.2

Pull Request Test Coverage Report for Build 165

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 83.799%

Totals

Change from base Build 163:
0.0%

Covered Lines:
2581

Relevant Lines:
3080

💛  - Coveralls


Label not number present in all training examples

Operating system (windows, osx, ...):Windows 10
Content of model configuration file:

Issue:
I am getting the error that i have written in title everytime when i fit my model to data

I would recommend trying to upload you're training data here: which should make it easier to see which utterance is missing a label. You probably would also get this error if you had mis-spelled the intent key name.
https://rasahq.github.io/rasa-nlu-trainer/

No
On Tue, Jul 31, 2018, 6:53 PM Akela Drissner-Schmid <
<EMAIL_ADDRESS>wrote:

@Sahil1313 https://github.com/Sahil1313 have you managed to figure out
the problem?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/RasaHQ/rasa_nlu/issues/1267#issuecomment-409219043,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AdNr4r0JAgWQVQvSZ7AfsW8Yi5Wu-sqUks5uMFpQgaJpZM4Vk3Z8
.

Can you provide some more details please? e.g. posting the full error trace

Closing for inactivity. @Sahil1313 please provide the full error trace and/or training data if you'd like us to re-open and try to troubleshoot. We're happy to help.


cherrypick 688 to fix image manifest read issue.

cherrypick https://github.com/open-cluster-management/multicluster-observability-operator/pull/688 to release-2.3

/hold

/lgtm

/unhold


Ability to remotely parse metadata over small HTTP requests

In this branch: https://github.com/huggingface/safetensors/compare/julien-c/js I pushed a proof-of-concept of how, given the simplicity of the format, one can fetch metadata about the weights over small (Range) HTTP requests.
The code is JS (can run in browsers or in Node) but it would be similar in any language.
Here's an example on how to fetch the header in a single file for instance:
async function parseSingleFile(url: URL): Promise<FileHeader> {
	const bufLengthOfHeaderLE = await (
		await fetch(url, {
			headers: {
				Range: "bytes=0-7",
			},
		})
	).arrayBuffer();
	const lengthOfHeader = new DataView(bufLengthOfHeaderLE).getBigUint64(
		0,
		true
	);
	/// ^little-endian
	const header: FileHeader = await (
		await fetch(url, {
			headers: {
				Range: `bytes=8-${7 + Number(lengthOfHeader)}`,
			},
		})
	).json();
	/// no validation for now, we assume it's a valid FileHeader.
	return header;
}

where a FileHeader type is defined as:
type TensorName = string;
type Dtype =
	| "F64"
	| "F32"
	| "F16"
	| "I64"
	| "I32"
	| "I16"
	| "I8"
	| "U8"
	| "BOOL";

interface TensorInfo {
	dtype: Dtype;
	shape: number[];
	data_offsets: [number, number];
}

type FileHeader = Record<TensorName, TensorInfo> & {
	__metadata__: Record<string, string>;
};

Results
As a fun first experiment, I compute the number of params per dtype for all models currently with a safetensors version on the HuggingFace Hub.
Here's the results:

model
safetensors
params

gpt2
single-file
{ 'F32' => 137022720 }

roberta-base
single-file
{ 'F32' => 124697433, 'I64' => 514 }

Jean-Baptiste/camembert-ner
single-file
{ 'F32' => 110035205, 'I64' => 514 }

roberta-large
single-file
{ 'F32' => 355412057, 'I64' => 514 }

bigscience/bloom-560m
single-file
{ 'F16' => 559214592 }

hf-internal-testing/tiny-random-bert-safetensors
single-file
{ 'F32' => 127463, 'I64' => 512 }

hf-internal-testing/tiny-random-bert-sharded-safetensors
index-file
{ 'F32' => 87929, 'I64' => 512 }

Narsil/small3
index-file
{ 'F32' => 59159, 'I64' => 512 }

Narsil/small2
single-file
{ 'F32' => 59159, 'I64' => 512 }

hf-internal-testing/tiny-random-bert-safetensors-tf
single-file
{ 'F32' => 87929 }

Thought it'd be fun to share! cc @mishig25 @osanseviero too

Super nice !
Actually I just thought , for the initial read, you could probably initial the first request directly on the first 100ko maybe ?
And refetch only if needed.
This would avoid launching 2 network calls in most settings (100ko is adjustable).
Just an optimization that might be worthwhile in production

Yes! I thought of that optimization too @Narsil. I'll probably implement it in a v2.

Update for top 100 most downloaded models: (currently 2486 models have the safetensors tag)

model
safetensors
params

bert-base-uncased
single-file
{ 'F32' => 110106428 }

jonatasgrosman/wav2vec2-large-xlsr-53-english
single-file
{ 'F32' => 315472545 }

gpt2
single-file
{ 'F32' => 137022720 }

xlm-roberta-base
single-file
{ 'F32' => 278885778 }

roberta-base
single-file
{ 'F32' => 124697433, 'I64' => 514 }

distilbert-base-uncased
single-file
{ 'F32' => 66985530 }

t5-base
single-file
{ 'F32' => 222903936 }

xlm-roberta-large
single-file
{ 'F32' => 561192082 }

bert-base-multilingual-cased
single-file
{ 'F32' => 178566653 }

bert-base-cased
single-file
{ 'F32' => 108932934 }

distilroberta-base
single-file
{ 'F32' => 82760793 }

albert-base-v2
single-file
{ 'F32' => 11842272 }

roberta-large
single-file
{ 'F32' => 355412057, 'I64' => 514 }

distilbert-base-uncased-finetuned-sst-2-english
single-file
{ 'F32' => 66955010 }

facebook/bart-large-mnli
single-file
{ 'F32' => 407344133 }

t5-small
single-file
{ 'F32' => 60506880 }

deepset/roberta-base-squad2
single-file
{ 'F32' => 124056578, 'I64' => 514 }

distilbert-base-multilingual-cased
single-file
{ 'F32' => 135445755 }

bigscience/bloom-560m
single-file
{ 'F16' => 559214592 }

bert-base-chinese
single-file
{ 'F32' => 102882442 }

distilgpt2
single-file
{ 'F32' => 88204032 }

camembert-base
single-file
{ 'F32' => 111246085 }

Jean-Baptiste/camembert-ner
single-file
{ 'F32' => 110035205, 'I64' => 514 }

bert-large-uncased
single-file
{ 'F32' => 336226108 }

gpt2-medium
single-file
{ 'F32' => 379988992 }

cambridgeltl/SapBERT-from-PubMedBERT-fulltext
single-file
{ 'I64' => 512, 'F32' => 109482240 }

facebook/bart-base
single-file
{ 'F32' => 139420416 }

bert-large-uncased-whole-word-masking-finetuned-squad
single-file
{ 'F32' => 335143938 }

distilbert-base-uncased-distilled-squad
single-file
{ 'F32' => 66364418 }

gpt2-large
single-file
{ 'F32' => 811778816 }

mrm8488/t5-base-finetuned-common_gen
single-file
{ 'F32' => 296926848 }

openai-gpt
single-file
{ 'F32' => 119680512 }

t5-large
single-file
{ 'F32' => 737668608 }

d4data/biomedical-ner-all
single-file
{ 'F32' => 66427476 }

distilbert-base-cased-distilled-squad
single-file
{ 'F32' => 65192450 }

Jean-Baptiste/roberta-large-ner-english
single-file
{ 'I64' => 514, 'F32' => 354315269 }

prompthero/openjourney
single-file
{ 'F32' => 123060480, 'I64' => 77 }

GanjinZero/UMLSBert_ENG
single-file
{ 'I64' => 512, 'F32' => 109482240 }

google/flan-t5-base
single-file
{ 'F32' => 247577856 }

google/flan-t5-large
single-file
{ 'F32' => 783150080 }

roberta-base-openai-detector
single-file
{ 'F32' => 125237762 }

mrm8488/t5-base-finetuned-summarize-news
single-file
{ 'F32' => 222903936 }

google/flan-t5-xxl
sharded
{ 'F32' =><PHONE_NUMBER>0 }

bert-base-multilingual-uncased
single-file
{ 'F32' => 168055961 }

bert-large-cased
single-file
{ 'F32' => 334661958 }

mrm8488/bert-multi-cased-finetuned-xquadv1
single-file
{ 'F32' => 177854978 }

facebook/wav2vec2-base-960h
single-file
{ 'F32' => 94395552 }

oliverguhr/german-sentiment-bert
single-file
{ 'F32' => 109083651 }

malteos/scincl
single-file
{ 'I64' => 512, 'F32' => 109918464 }

Dizex/InstaFoodRoBERTa-NER
single-file
{ 'I64' => 514, 'F32' => 124058115 }

bert-large-uncased-whole-word-masking
single-file
{ 'F32' => 336226108 }

ltg/norbert2
single-file
{ 'I64' => 512, 'F32' => 125164986 }

shahrukhx01/question-vs-statement-classifier
single-file
{ 'I64' => 512, 'F32' => 11171074 }

facebook/esm2_t6_8M_UR50D
single-file
{ 'I64' => 1026, 'F32' => 7840842 }

pszemraj/flan-t5-large-grammar-synthesis
single-file
{ 'F32' => 783150080 }

bigscience/bloomz-560m
single-file
{ 'F16' => 559214592 }

roberta-large-mnli
single-file
{ 'F32' => 356412419 }

Gustavosta/MagicPrompt-Stable-Diffusion
single-file
{ 'F32' => 124439808, 'U8' => 12582912 }

human-centered-summarization/financial-summarization-pegasus
single-file
{ 'F32' => 568796007 }

finiteautomata/beto-emotion-analysis
single-file
{ 'I64' => 512, 'F32' => 109859335 }

voidful/albert_chinese_small
single-file
{ 'F32' => 4812936 }

mrm8488/distilroberta-finetuned-financial-news-sentiment-analysis
single-file
{ 'I64' => 514, 'F32' => 82120707 }

mrm8488/t5-base-finetuned-question-generation-ap
single-file
{ 'F32' => 296926848 }

nbroad/ESG-BERT
single-file
{ 'I64' => 512, 'F32' => 109502234 }

impira/layoutlm-document-qa
single-file
{ 'I64' => 514, 'F32' => 127792898 }

bert-base-german-cased
single-file
{ 'F32' => 109705010 }

aubmindlab/bert-base-arabert
single-file
{ 'F32' => 135851010 }

deepset/tinyroberta-squad2
single-file
{ 'I64' => 514, 'F32' => 81529346 }

albert-base-v1
single-file
{ 'F32' => 11842272 }

beomi/kcbert-base
single-file
{ 'F32' => 109542194 }

Babelscape/wikineural-multilingual-ner
single-file
{ 'I64' => 512, 'F32' => 177269769 }

rinna/japanese-gpt-1b
single-file
{ 'F16' =><PHONE_NUMBER> }

setu4993/LaBSE
single-file
{ 'I64' => 512, 'F32' => 470926848 }

bigscience/bloom-1b1
single-file
{ 'F16' =><PHONE_NUMBER> }

sagorsarker/bangla-bert-base
single-file
{ 'F32' => 165092235 }

pszemraj/grammar-synthesis-small
single-file
{ 'F32' => 76961152 }

vicgalle/xlm-roberta-large-xnli-anli
single-file
{ 'I64' => 514, 'F32' => 559893507 }

typeform/distilbert-base-uncased-mnli
single-file
{ 'F32' => 66955779 }

distilbert-base-german-cased
single-file
{ 'F32' => 67431550 }

EleutherAI/gpt-neox-20b
sharded
{ 'F16' =><PHONE_NUMBER>8, 'U8' => 184549376 }

bigscience/bloom
sharded
{ 'BF16' =><PHONE_NUMBER>24 }

bigscience/bloom-3b
single-file
{ 'F16' =><PHONE_NUMBER> }

wavymulder/Analog-Diffusion
error
model id does not contain safetensors weights

FredZhang7/distilgpt2-stable-diffusion-v2
single-file
{ 'F32' => 81912576, 'U8' => 6291456 }

albert-xxlarge-v2
single-file
{ 'F32' => 223180256 }

cointegrated/rubert-tiny2
single-file
{ 'I64' => 2048, 'F32' => 29376502 }

KES/T5-KES
single-file
{ 'F32' => 222903552 }

cointegrated/LaBSE-en-ru
single-file
{ 'I64' => 512, 'F32' => 128993837 }

knkarthick/MEETING_SUMMARY
single-file
{ 'F32' => 406340696 }

rinna/japanese-roberta-base
single-file
{ 'I64' => 514, 'F32' => 110652416 }

xlm-clm-ende-1024
single-file
{ 'F32' => 208673979 }

oliverguhr/spelling-correction-english-base
single-file
{ 'F32' => 139470681 }

lidiya/bart-large-xsum-samsum
single-file
{ 'F32' => 406340696 }

dominguesm/bert-restore-punctuation-ptbr
single-file
{ 'I64' => 512, 'F32' => 108344079 }

patrickjohncyh/fashion-clip
single-file
{ 'I64' => 127, 'F32' => 151277312 }

mrm8488/bert-spanish-cased-finetuned-pos-16-tags
single-file
{ 'F32' => 109863953 }

MoritzLaurer/mDeBERTa-v3-base-mnli-xnli
single-file
{ 'I64' => 512, 'F16' => 278811651 }

blanchefort/rubert-base-cased-sentiment-rusentiment
single-file
{ 'I64' => 512, 'F32' => 177855747 }

elastic/distilbert-base-cased-finetuned-conll03-english
single-file
{ 'F32' => 65197833 }

cointegrated/rubert-tiny-toxicity
single-file
{ 'I64' => 512, 'F32' => 11785733 }

@julien-c How is the canonical order of tensors reconstructed as seen here via huggingface.co/gpt2?show_tensors=true

The above example shows the first two tensor names aren't following lexicographical order (as intended) whereas the response returns the safetensor layout which is not in order... so does this mean the the information can be retrieved/exists somewhere programmatically!? 🙏🏼

@sparverius that's a question for @mishig25 who implemented it, but yeah we have a few heuristics we use to order the layers on the frontend side – while the API exposes the logical on-disk order of the safetensors file (we had a lot of debate about this 🤣)
We can share some pseudo-code to demonstrate what we're doing on the frontend side maybe.

but yeah we have a few heuristics we use to order the layers on the frontend side – while the API exposes the logical on-disk order of the safetensors file (we had a lot of debate about this 🤣)

Interesting, what were the main takeaways?

We can share some pseudo-code to demonstrate what we're doing on the frontend side maybe.

That would be awesome, thank you!
All thanks to the safetensors format, I have been working on a little side project building on this vision of summarizing/representing a given concrete model architecture textually and visually... hoping the results will be insightful to compare/diff models side-by-side or for gaining individual insight at a glance between models for a given task 🎨

@sparverius here is the heuristic to order the layers:
1. Split a layer name. The splitters/seperators are [".", "-", "_"]. Example: h.0.attn.c_proj.bias -> ["h", 0 "attn", "c_proj", "bias"]
2. Compare layername objects. If the current element is string, do lexiocographical order. If they are numbers, do numbers order. Ex: ["h", 0, "attn", "c_proj", "bias"] will rank higher than ["h", 1, "attn", "c_proj", "bias"] because 0 < 1 in their second elements
3. Use the below heauristic names (copied mostly from transformers naming convention), to "overwrite" the lexiocographical order

const REGEX_FIRST_LAYERS = /(embed|wte|wpe|shared)/i;
const REGEX_LAST_LAYERS = /(head|classifier)/i;

summarizing/representing a given concrete model architecture textually and visually

This sounds super interesting, i'm sure many people would be interested in this

Thanks @mishig25! Interesting, any other existing efforts to catalog different architectures?

@julien-c thanks, I hope it will be useful!
Showing that one can retrieve model information from a safetensors checkpoint shows the beauty & transparency of the format and thankfully the inspiration for this side-project 🤗 ...
I'm cautious to depend upon this fact on a more widespread level since it costs precious requests for hf serverside (a few for larger sharded models think HuggingFaceM4/idefics-80b or even tiiuae/falcon-180B) and the whole ordering thing, even though it might be a good advertisement for safetensors...
Some thoughts from what i've been running into for one,  utilizing config.json seems helpful for ordering

EXPAND for details on how that might be useful for transformers, diffusers, timm models

for transformers
Certain heuristics for encoder-only, encoder-decoder, decoder only ...
Llama-2-7b-hf
config.json gives us some insights on shapes, # layers, etc
{
  "architectures": [
    "LlamaForCausalLM"
  ],
  ...
  "hidden_size": 4096,
  ...
  "max_position_embeddings": 4096,
  "model_type": "llama",
  "num_attention_heads": 32,
  "num_hidden_layers": 32,
  "num_key_value_heads": 32,
  ...
  "vocab_size": 32000
}

what about an architecture that hasn't been pulled into transformers lib yet
microsoft/phi-1_5
gives estimate of tensor shapes etc, but doesn't tell us what each layer is composed of unless one looks at custom code...
{
  ...
  "architecture": { ... "block_cls": "parallel", ... },
  "architectures": [ "MixFormerSequentialForCausalLM" ],
  "auto_map": { ... "AutoModelForCausalLM": ... },
  ...
  "model_type": "mixformer-sequential",
  "n_embd": 2048,
  "n_head": 32,
  "n_inner": null,
  "n_layer": 24,
  "n_positions": 2048,
  ...
  "rotary_dim": 32,
  ...
  "vocab_size": 51200
}

perhaps https://huggingface.co/microsoft/phi-1_5/blob/main/tokenizer_config.json gives us a hint?
{
  ...
  "tokenizer_class": "CodeGenTokenizer",
  ...
}

other interesting cases:
facebook/maskformer-swin-large-coco

for timm:
https://huggingface.co/timm/resnet50.a1_in1k/blob/main/config.json
{
    "architecture": "resnet50",
    ...
    "first_conv": "conv1",
    "classifier": "fc",
    ...
}

for diffusers:
it gets a bit more complicated
https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/raw/main/model_index.json
this one involves some pointer chasing ☞ ✦ ✧ ❂ 🏆
{
  "_class_name": "StableDiffusionXLPipeline",
  "text_encoder": [
    "transformers",
    "CLIPTextModel"
  ],
  ...
  "unet": [
    "diffusers",
    "UNet2DConditionModel"
  ],
  "vae": [
    "diffusers",
    "AutoencoderKL"
  ]
}

Other Errata:
Ambiguities arise: What is Big🐤?
https://github.com/huggingface/transformers/blob/dcbfd93d7aeb14f8ff08a48866d2a68950d4c69a/templates/adding_a_new_model/open_model_proposals/ADD_BIG_BIRD.md?plain=1#L217-L228
🤔 https://github.com/huggingface/transformers/blob/dcbfd93d7aeb14f8ff08a48866d2a68950d4c69a/templates/adding_a_new_model/open_model_proposals/ADD_BIG_BIRD.md?plain=1#L256-L257

TL;DR 🤖
Perhaps this discussion for this side-project is better suited for elsewhere though, I am wondering if a community effort in cataloging model summaries might be the best way ... all thoughts welcome 🤗 !

Always wanted a way to see model architecture info at a glance & the ability to remotely parse safetensors is the first ive seen as a way to do so
The labels in safetensors are not ordered and ordering with heuristics is helpful to some degree
Extending heuristics with other available information such as config.json might improve ordering
network requests to models is cheap but not free
caching results to file?

Problem Statement
How can one know the concrete architecture of a model at a glance without grokking paper, source code, or ultimately loading into memory?

Related existing tools:
Text-based

model.summary()
torchinfo/torch-summary https://pypi.org/project/torchinfo/

Visual

tensorboard graphs example
https://github.com/paulgavrikov/visualkeras
Torchlens https://github.com/johnmarktaylor91/torchlens
pytorchviz https://github.com/szagoruyko/pytorchviz

Ideas

Tool to summarize model in particular format

doesn't need to be complex just standardized
condense repeated information ie. layers 1-24 are identical
something akin to repr(model) or model.summary()

 Example
{
    "deberta": {
        "class": "DebertaV2Model",
        "embeddings": {
            "class": "DebertaV2Embeddings",
            "position_ids": "[1, 512]",
            "word_embeddings": {
                "class": "Embedding",
                "weight": "[128100, 768]",
            },
            ...
        },
        "encoder": {
            "class": "DebertaV2Encoder",
            "layer": {
                "class": "ModuleList",
                "N": {
                    "class": "DebertaV2Layer",
                    "attention": {
                        "class": "DebertaV2Attention",
                        "self": {
                            "class": "DisentangledSelfAttention",
                            "query_proj": {
                                "class": "Linear",
                                "weight": "[768, 768]", # deberta.encoder.layer.N.attention.self.query_proj.weight
                                "bias": "[768]", # deberta.encoder.layer.N.attention.self.query_proj.bias
                            },
                            "key_proj": { "class": "Linear", ... },
                            ...
                            "pos_dropout": { "class": "StableDropout" },
                            "dropout": { "class": "StableDropout" }
                        },
                        "output": {
                            "class": "DebertaV2SelfOutput",
                            "dense": {
                                "class": "Linear",
                                "weight": "[768, 768]", # deberta.encoder.layer.N.attention.output.dense.weight
                                ...
                            }
                            ...
                        }
                    },
                    "intermediate": {
                        "class": "DebertaV2Intermediate",
                        "dense": { ... },
                        "intermediate_act_fn": { "class": "GELUActivation" }
                    },
                    ...
                },
                ..
            },
           "rel_embeddings": {
                "class": "Embedding",
                "weight": "[512, 768]",
            },
            "LayerNorm": { ... }
        }
    },
    "pooler": { "class": "ContextPooler", ... },
    "classifier": { ... },
    "dropout": { "class": "StableDropout" }
}

thinking of create a repo & associated pip package

allowing users to pip install dependency

something like it runs after saving checkpoint / before push_to_hub (or in process before converting safetensor)

runs a souped up model-summary

sends pull-request to repo OR pushes file to hub?

🤩 just for fun: show lovely cli graphics think starship.rs because huggingface is fun!

or, find a workflow that works with hf hub?

Outcomes:
A catalog / repo / central place hosting model summaries

Discovery
Searchable hierarchy
Automagically get visualization for free
Automagically compare models
Promote insight & perhaps aid in research
could combine concrete architecture comparisons with benchmarks etc...

Great summary, @sparverius!!

A catalog / repo / central place hosting model summaries

IMO the best would be to place each model summary into its model repo on the HF Hub (through a Pull request or Discussion)
Also makes me think a bit of https://huggingface.co/spaces/hf-accelerate/model-memory-usage by @muellerzr
@mishig25 do you remember if we had noted somewhere public some of our thoughts about how to encode model architecture into an easy to use file format?

Great summary, @sparverius!!

Thanks!

IMO the best would be to place each model summary into its model repo on the HF Hub (through a Pull request or Discussion)

Good point, that would be the most accessible.

@mishig25 do you remember if we had noted somewhere public some of our thoughts about how to encode model architecture into an easy to use file format?

If you have that to share, that would be awesome!
Was thinking of something simple, a similar format as safetensors metadata, encoding intermediate labels as class names for example,
{
 "model": "Model",
 "model.embed_tokens": "Embedding",
 "model.embed_tokens.weight": {"shape": [32000, 4096], "dtype": "float16"},
 "model.layers": "ModuleList",
 "model.layers.0": "DecoderLayer",
 "model.layers.0.self_attn": "Attention",
 "model.layers.0.self_attn.rotary_emb": "RotaryEmbedding",
 "model.layers.0.self_attn.rotary_emb.inv_freq": {"shape": [64], "dtype": "float32"},
 "model.layers.0.self_attn.rotary_emb.cos_cached": {"shape": [1, 1, 4096, 128], "dtype": "float16"},
 "model.layers.0.self_attn.rotary_emb.sin_cached": {"shape": [1, 1, 4096, 128], "dtype": "float16"},
 "model.layers.0.self_attn.k_proj": "QuantLinear",
 "model.layers.0.self_attn.k_proj.qweight": {"shape": [512, 4096], "dtype": "int32"},
 "model.layers.0.self_attn.k_proj.qzeros": {"shape": [32, 512], "dtype": "int32"},
 "model.layers.0.self_attn.k_proj.scales": {"shape": [32, 4096], "dtype": "float16"},
 "model.layers.0.self_attn.k_proj.g_idx": {"shape": [4096], "dtype": "int32"},
 "model.layers.0.self_attn.k_proj.bias": {"shape": [4096], "dtype": "float16"},
 ...
 "model.layers.0.mlp": "MLP",
 "model.layers.0.mlp.act_fn": "SiLUActivation",
 ...
 "model.layers.0.mlp.up_proj": "QuantLinear",
 "model.layers.0.mlp.up_proj.qweight": {"shape": [512, 11008], "dtype": "int32"},
 "model.layers.0.mlp.up_proj.qzeros": {"shape": [32, 1376], "dtype": "int32"},
 "model.layers.0.mlp.up_proj.scales": {"shape": [32, 11008], "dtype": "float16"},
 "model.layers.0.mlp.up_proj.g_idx": {"shape": [4096], "dtype": "int32"},
 "model.layers.0.mlp.up_proj.bias": {"shape": [11008], "dtype": "float16"},
 "model.layers.0.input_layernorm": "RMSNorm",
 "model.layers.0.input_layernorm.weight": {"shape": [4096], "dtype": "float16"},
 "model.layers.0.post_attention_layernorm": "RMSNorm",
 "model.layers.0.post_attention_layernorm.weight": {"shape": [4096], "dtype": "float16"},
 ...
}

This has the advantage of being able to cross check against safetensors, complements safetensor metadata with a bit of added info and can easily be jsonized...

@mishig25 do you remember if we had noted somewhere public some of our thoughts about how to encode model architecture into an easy to use file format?

There was no public discussion. Internally, you've posted

Can ONNX refer to external weights, i.e. for instance could a ONNX file only represent the computation graph, but point to a safetensors file for the actual weights? (maybe through an extension)

btw @sparverius I assume you've seen this doc page https://huggingface.co/docs/safetensors/metadata_parsing ?

@ThiloteE we just sum the number of parameters in all the tensors
There's also a python implementation in case it is more readable: https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/utils/_safetensors.py


”Turcinești”, ”Cartiu”, "Horezu", "Rugi" are listed in the drop-down menu

Description:
”Turcinești”, ”Cartiu”, "Horezu", "Rugi" are listed in the drop-down menu alongside their corresponding county and commune, according to the law
Precondition:
The website is up an running.

Step 1
Write in the search bar "Turcinești"

Expected results
The location appears in the drop-down menu as "Sat Turcinești, GORJ (Turcinești)".

Step 2
Press "x" button.

Expected results
The location was deleted from the search bar.

Step 3
Write in the search bar "Cartiu"

Expected results
The location appears in the drop-down menu as "Sat Cartiu, GORJ (Turcinești)".

Step 4
Press "x" button.

Expected results
The location was deleted from the search bar.

Step 5
Write in the search bar "Horezu"

Expected results
The location appears in the drop-down menu as "Sat Horezu, GORJ (Turcinești)".

Step 6
Press "x" button.

Expected results
The location was deleted from the search bar.


outputReport not generating file

I've tried a bunch of things but cannot get the eslint-loader to output a file containing any warnings / errors from eslint.
Here's my webpack.config.js
const isDevBuild = !(require('yargs').argv.p || false);
const path = require('path');
const outputFolder = isDevBuild ? 'dist/dev' : 'dist/production';

const webpack = require('webpack');
const CleanWebpackPlugin = require('clean-webpack-plugin');
const HtmlWebpackPlugin = require('html-webpack-plugin');
const ExtractTextPlugin = require('extract-text-webpack-plugin');
// const BundleAnalyzerPlugin = require('webpack-bundle-analyzer').BundleAnalyzerPlugin;
const CopyWebpackPlugin = require('copy-webpack-plugin');

const CssExtractor = new ExtractTextPlugin({
      filename: './css/[name].[hash].min.css',
      allChunks: false
    });

module.exports = {
  devtool: 'source-map',
  entry: {
    app: './src/root.js',
  },
  output: {
    filename: './js/[name].[hash].bdl.js',
    path: path.resolve(__dirname, outputFolder),
    sourceMapFilename: './maps/[file].map'
  },
  module: {
    rules: [
      {
        test: /\.jsx?$/,
        exclude: /node_modules/,
        enforce: 'pre',
        loader: 'eslint-loader',
        options: {
          formatter: require('eslint-friendly-formatter'),
          outputReport: {
            filePath: 'checkstyle.xml',
            formatter: require('eslint-friendly-formatter')
          }
        }
      },
      // Handle js / jsx files
      {
        test: /\.jsx?$/,
        exclude: /node_modules/,
        loader: 'babel-loader'
      },
      // Handle stylus files
      {
        exclude: path.resolve(__dirname, './styles/common'),
        test: /\.styl$/,
        use: CssExtractor.extract({
          fallback: 'style-loader',
          use: [
            'css-loader',
            'stylus-loader'
          ],
        })
      },
    ]
  },
  plugins: [
    // Deletes the contents of output folder prior to build
    new CleanWebpackPlugin(outputFolder),
    // Injects the bundles into the HTML
    new HtmlWebpackPlugin({
      template: './src/index.html',
      filename: 'index.html',
      inject: 'body'
    }),
    // Split vendor specific js code (node_modules) into a separate file
    new webpack.optimize.CommonsChunkPlugin({
      name: 'vendor',
      filename: './js/vendor.[hash].bdl.js',
      minChunks(module, ) {
        let context = module.context;
        return context && context.indexOf('node_modules') >= 0;
      },
    }),
    // Extract stylus generated CSS to a separate file(s)
    CssExtractor,
    // Copy static content to the dist folder
    new CopyWebpackPlugin([{
      from: 'public',
      to: ''
    }])
    // new BundleAnalyzerPlugin()
  ]
};

I purposefully added errors and warnings in my code to try to generate the file, and it does display them in the terminal, but no file is being generated.

The file was being created in my build folder.


WHIP Server listening on all ports blocks Halcyon.exe

Okay hoping I am posting this for the right repo as y'all have more then one repository (halcyon and WHIP-Server).  I am testing on a local machine (not in full grid mode at the moment as I am just testing to see if this would work with my project).
When I run the Whip Server it appears it is listening on all ports and then when running Halcyon.exe (same machine per above testing) Halcyon says it doesn't have permission to listen on port 9000.  I have all ports necessary unblocked in firewalls and modem/ routers so I know the firewall isn't an issue.  Is there perhaps a way to define a listening port i.e. 32700 (as identified in the whip.cfg file)?
Thanks in advance.

Sounds like you have whip running as expected on 32700.  So I think your issue is more with the region server.  The INI file config uses 9000 as the default port for logins.  The message you're seeing is an indication that the network layer isn't allowing the connection to take place.  There's a comment in the code with how to address that but I'll paste it here:
        netsh http add urlacl url=http://+:9000/ user=mdickson
        netsh http add urlacl url=https://+:9016/ user=mdickson

You'd replace the "mdickson" with the account name you're running the server under.  The 9016 port address is for SSL connections to the server. You can skip that if you don't need it.  Run these command with an admin command prompt.

Good morning,
Thanks for the response.  I actually did try the netsh command as I had thought of that myself, changed different ports, checked firewall (for both windows 10 pro x64 and modem/ router) etc with no problems.  I also verified that the port was open using Opensim and some of its derivatives just to make sure I wasn't missing something in the network. Opensim and its derivatives (i.e. Aurora-Sim etc) worked on the port so I know its opened in the firewalls.  I actually went as far as disabling firewalls and adding the host name to the DMZ.  The problem still persisted even with changing the hostname etc to both the internal IP I give the machine on my network and the external IP as well as FQDN. However the error still interestingly comes up.
The WHIP server appears to see the Halcyon instance connect to it yet detects a forced shutdown of the client.  If that helps any.

Sorry for the delays in responding. We've been pretty busy getting a new Halcyon release together.

When I run the Whip Server it appears it is listening on all ports and then when running Halcyon.exe (same machine per above testing) Halcyon says it doesn't have permission to listen on port 9000.

I don't think this is in any way related to WHIP. It's just that there's already something listening on port 9000.  So the goal would be to identify the conflicting process.
Before you run Halcyon.exe, try running this:
netstat -aonp tcp | findstr 9000
It shouldn't match anything, otherwise something else is already listening on port 9000.  I seem to recall that I had the same port clash when I first loaded a Halcyon executable that listened on port 9000, but I forget what the clashing process was.
If that command returns something, the number on the end will indicate the process ID of the process listening on 9000.  You can use Task Manager's Details tab to identify which (if you have the Process ID column enabled). Alternatively, you can start a command prompt with "Run as Administrator" and add the b option which puts the process name (if available) on the next line.  But since it's on the next line, you can't use findstr so it would be something like:
netstat -abonp tcp >ports.txt
and then edit ports.txt and find the line with 9000, and see what process follows it on the next line.
Unfortunately, it may be process ID 4 (or some similar low number) which I believe is related to the process launch code in the system for .NET processes.  In other words, the process ID and name may not help at all here if it's a .NET app.

Also, the User process (the part listening on port 9000) is often on port 8002 anyway.  On my system, I first resolved this by changing the user login port from 9000 to 8002.
Be sure to enabled it with the netsh command if you use any other ports though.

Before you run Halcyon.exe, try running this:
netstat -aonp tcp | findstr 9000
It shouldn't match anything, otherwise something else is already listening on port 9000.

Actually that's not 100% foolproof in that the -p tcp is filtering out the IP6 listens.  If it doesn't return anything, you might want to try:
netstat -aon | findstr 9000
or
netstat -abon >ports.txt

The WHIP server appears to see the Halcyon instance connect to it yet detects a forced shutdown of the client. If that helps any.

This is something different. I think it's just saying a viewer is able to connect to the User/login part but not the region code.  Does that mean Halcyon is able to open port 9000 now, but the UDP and TCP ports are not permitted in netsh ?
Check both:

internal_ip_port (typically port 902x) in the Region.XML file in Regions, as well as
http_listener_port (typically 950x) in the Halcyon.ini file.

Make sure both are specified to an unused port, and ACLs enabled in netsh.

Actually I haven't even tried logging in with user/ login system.  The Whip Server appears to be assuming that Halcyon.exe (my assumption here) is a client of some sort.  When the Halcyon.exe terminates with the port error (after using netsh commands, etc) WHIP says Client closed due to forced termination.  I did check all ports are open even tried on fresh installation with just WIndows 10 x64 (V1607)  Halcyon.ini has the default listern ports set to 9000 under [Network]. Haven't even gotten as far as setting up a region.

@Fred2008Levasseur I think whip is most likely running OK here. When it says forced closed it is because the Halcyon process is terminating for some other reason I believe

@ddaeschler Thanks for that.  I was able to get the Whip server running however the strange thing even with all the netsh commands etc and nothing blocking the ports in firewall (in OS or Modem/ Router) Halcyon.exe still believes its a port permissions error when in fact the port is not in use by any other process.  The Whip server does see Halcyon.exe trying to connect to it but detects that it was a force shut down in the client (I am assuming this is referring to the Halcyon.exe as I am not even at the point of logging in via a viewer which is the actual client).  I also noticed there are no building tools i.e. sln files or anything of that nature in the whip-server repo as well.  Probably intentional though I do like the ability to be able to build locally for my testing purposes, as sometimes this will help.
@appurist I did check with the commands per your recommendations and found the port in fact is not being used by any other program, app, or process on either IPV4 or IPV6.  I do have one more idea I am going to try as it dawned on me I might need to run Halcyon.exe as administrator (even though I built it locally).

On Windows 10, you need to run Halcyon.exe as Administrator or you WILL get a permission denied to use port 9000 (or port whatever is used by the region when in grid mode) error.
You either need to right click and choose Run as Administrator for your CMD Prompt or in your Halcyon.exe, right click and choose the Compatibility Tab and check "Run this program as an administrator" to have that value forced at each run time.

First, I think you can forget about Whip, as that's not having any trouble or causing any trouble. The problem is permission to open port 9000.  I don't think we normally run Halcyon as Administrator.  I think the issue here may be that you're trying to start up a Halcyon server in standalone mode, which requires all of the ports and configuration options to all be configured correctly or it will fail to start, and this is just a startup error in the Halcyon process.  And in standalone mode, that includes a region.  So to diagnose that, it might be good to post your Halcyon.log file for us to see how far it's getting before it runs into trouble, and what the precise error is, and what information it includes.
I'm not sure if any of us are running in standalone mode anymore. It's not used by InWorldz and I think most of the Halcyon developers run in grid mode.  Grid mode would also allow you to check each service one at a time.  So you'd start Whip, then the User grid server, then the Grid grid server, then Messaging grid server.  And also at some point after Whip, start Aperture for textures.  I think the reason most of us use grid mode is because the Halcyon server can only support one region (with PhysX physics) per Windows instance, so we need to fire up more than one for multiple regions. Since we want them to be able to communicate, they need to share the same core services, so they can't be built-in to one instance and not the other.  Running each grid service seperately would also allow you to see how far you were getting in terms of starting User... ok it's up... Starting Grid... etc. But the Halcyon.log file may also answer that.

@appurist After testing with running as admin and non admin I concur that running as admin does not have any affect as I built Halcyon locally.  The same result applies as shown:
2017-01-22 08:19:08,000 [1] INFO  OpenSim.Application - [HALCYON MAIN]: configured log4net using default Halcyon.exe.config
2017-01-22 08:19:08,015 [1] INFO  OpenSim.Application - Performing compatibility checks...
2017-01-22 08:19:08,031 [1] INFO  OpenSim.Application - Environment is compatible.
2017-01-22 08:19:08,046 [ConsoleThread] INFO  OpenSim.ConfigurationLoader - [CONFIG] Reading configuration settings
2017-01-22 08:19:08,046 [ConsoleThread] INFO  OpenSim.ConfigurationLoader - [CONFIG] Reading configuration file C:\Users\virtu\Desktop\halcyon-master\bin\Halcyon.ini
2017-01-22 08:19:08,130 [ConsoleThread] INFO  OpenSim.Framework.Servers.BaseOpenSimServer - [STARTUP]: Beginning startup processing
2017-01-22 08:19:08,230 [ConsoleThread] INFO  OpenSim.OpenSim - ====================================================================
2017-01-22 08:19:08,230 [ConsoleThread] INFO  OpenSim.OpenSim - ========================= STARTING HALCYON =========================
2017-01-22 08:19:08,230 [ConsoleThread] INFO  OpenSim.OpenSim - ====================================================================
2017-01-22 08:19:08,230 [ConsoleThread] INFO  OpenSim.OpenSim - [HALCYON MAIN]: Running in sandbox mode
2017-01-22 08:19:08,230 [ConsoleThread] INFO  OpenSim.OpenSim - GC: Server mode: False, Interactive
2017-01-22 08:19:08,331 [ConsoleThread] INFO  OpenSim.Region.Framework.StorageManager - [DATASTORE]: Attempting to load OpenSim.Data.MySQL.dll
2017-01-22 08:19:08,416 [ConsoleThread] INFO  OpenSim.Region.Framework.StorageManager - [DATASTORE]: Added IRegionDataStore Interface
2017-01-22 08:19:08,431 [ConsoleThread] INFO  OpenSim.Region.ClientStack.ClientStackManager - [CLIENTSTACK]: Attempting to load OpenSim.Region.ClientStack.LindenUDP.dll
2017-01-22 08:19:10,482 [ConsoleThread] INFO  OpenSim.Region.ClientStack.ClientStackManager - [CLIENTSTACK]: Added IClientNetworkServer Interface
2017-01-22 08:19:10,482 [ConsoleThread] DEBUG OpenSim.OpenSimBase - [HALCYONBASE] Attempting to load asset server id=WHIP
2017-01-22 08:19:11,238 [ConsoleThread] INFO  OpenSim.Framework.PluginLoader1 - [PLUGINS]: Initializing addin manager 2017-01-22 08:19:11,873 [ConsoleThread] INFO  OpenSim.Framework.PluginLoader1 - [PLUGINS]: Loading extension point /OpenSim/AssetClient
2017-01-22 08:19:11,973 [ConsoleThread] ERROR OpenSim.Application - [APPLICATION]:
APPLICATION EXCEPTION DETECTED: System.UnhandledExceptionEventArgs
Exception: System.Exception: Asset server WHIP could not be loaded
at OpenSim.OpenSimBase.InitializeAssetCache() in C:\Users\virtu\Desktop\halcyon-master\OpenSim\Base\OpenSimBase.cs:line 324
at OpenSim.OpenSimBase.Initialize() in C:\Users\virtu\Desktop\halcyon-master\OpenSim\Base\OpenSimBase.cs:line 286
at OpenSim.Region.ClientStack.RegionApplicationBase.StartupSpecific() in C:\Users\virtu\Desktop\halcyon-master\OpenSim\Region\ClientStack\RegionApplicationBase.cs:line 107
at OpenSim.OpenSimBase.StartupSpecific() in C:\Users\virtu\Desktop\halcyon-master\OpenSim\Base\OpenSimBase.cs:line 201
at OpenSim.OpenSim.StartupSpecific() in C:\Users\virtu\Desktop\halcyon-master\OpenSim\Base\OpenSim.cs:line 163
at OpenSim.Framework.Servers.BaseOpenSimServer.Startup() in C:\Users\virtu\Desktop\halcyon-master\OpenSim\Framework\Servers\BaseOpenSimServer.cs:line 300
at OpenSim.Application.Main(String[] args) in C:\Users\virtu\Desktop\halcyon-master\InWorldz\Halcyon\Application.cs:line 145
Application is terminating: True
As a post note during compiling with runprebuild2015.bat the first 3 projects were not able to be found which is potentially a cause for the warnings shown my build output:
BuildMessage.txt
Is there a specific order that should be used for starting services for running in grid mode? perhaps this might have something to do with it as per your suggestion.

Maybe post your config files (Redact your password in the database lines).
Do you have  gridmode = false set and uncommented?
Do you have this block configured?
[StandAlone]
    ; StandAlone is often used for simple test setups with one region.
    ; Provide information to a standalone boot.
    ; If gridmode is true you should be looking in the various xml config files,
    ;  such as the UserServer_Config.xml

    accounts_authenticate = false
    welcome_message = "Welcome to Halcyon"

    **asset_source = "whip://password@localhost:32700"**

The password matching the password setup in the whip.cfg
And then again under [Network]
    ; The URL for the asset server, usually a WHIP server.
    ;  See https://github.com/InWorldz/whip-server
    **asset_server_url = "whip://password@localhost:32700"**

Be sure to start the Whip server, let it settle down, then start the Halcyon.exe.
From your log file, it sounds like Halcyon.exe is not communicating with Whip and is whigging out causing it to error out and close.

This looks like the issue specified here: https://github.com/InWorldz/halcyon/wiki/Development
Did you download the source from a zipfile? If so, you need to unblock the directories in windows since downloads are restricted from running on windows.
However the best way to get around this problem is to check out the source code with git.

Yes I checked out the source with git.  The other interesting thing I did find (even struck me as odd) is that the WHIP server asks for permission to be unblocked in firewall even though I gave all permissions.  Additionally I found two Whip.exe instances in the firewall one saying port 32702 and one just said Any for ports.  I suspect that there may actually be something going on even though the actual ports are open.  I am going to try to shut off smart screen and disable firewall and defender just to see if thats whats causing problems (though I hope not.)

Do you have whip.cfg setting the Whip server port to 32702 and not the default 32700?
And if so, did you change halcyon.ini to match?

No the ports are the defaults for Whip and do match.

And the results of turning off Windows Firewall and Smartscreen? Then running Whip.exe letting it settle, then running Halcyon.exe?

Good afternoon,
Sorry for the delayed response (my day time job and development projects for my work always take precedence for me).  I did get it to work this morning just before before work. (smart screen did actually prevent it from working so people would want to disable smartscreen).
The command suggested by @sonjamichelle would in fact work for people and is probably one that should be included in the documentation somewhere (replacing the username of course).  In my case however i had created a tool a while back that takes care of unblocking stuff like that so in my case that command wasn't necessary.    Thanks for the suggestion though as it is a good one :)
Halcyon now does communicate with WHIP however when logging in it could not communicate with the WHIP server.  Logging in via viewer).  During the login the following error occurs: 2017-01-23 13:01:32,306 [STP SmartThreadPool Thread #0] ERROR OpenSim.Region.CoreModules.Capabilities.AssetCapsModule - [APERTURE] Could not contact the aperture texture server to register caps on region Welcome Center. Server returned error Unable to connect to the remote server
It is possible I missed something in the ini file somewhere (which I will be checking when I get home).  I will also try to configure it for grid mode and give that a shot and see how that goes as well.
As a post note Fred2008Levasseur is actually my non work ID I use when I am at home and testing for my own personal projects so there is no confusion.

So alles gut now?
I always disable that micro$oft annoyance smartscreen. It causes all kinds of issues running programs.
The documentation is a bit lacking in some details. A lot of the issues I came across in setting up my grid I found in trial and error and going through the issues and posting my own issues and asking a lot of questions. the developers here are an amazing bunch!
I'm toying with the idea of making a video of the setup process, going from "zero to hero" so to speak. Maybe when RL things settle down a bit.
When you get into setting up a multi region grid mode, check out the issues #232 and #298. There is a lot of information in them that should help you out.

Good afternoon,
My apologies for not following up sooner.  I did manage to get things going and was able to take Halcyon for a quick test run.  A couple of questions I do have assuming I would consider running Halcyon as the base architecture for a grid however:

Does Halcyon have a working economy module i.e. one which provides stipend payments, can be connected to a website interface for purchasing of in-world currency, etc?

Is there a known working website interface for Halcyon currently or example files which can be used to create one?  (Preferably in PHP)

If I want to build the WHIP Server locally and run a built copy locally to avoid permissions issues, what are the steps to build the WHIP server?  NOTE: There are no instructions for this in the WHIP Server Repo.

Which Grid Management Program is recommended for use to manage a grid running on Halcyon?

Thanks.

Answers to the Qs I know answers to:

Yes. See notes in the Halcyon wiki.
There are a few projects in place, one of which is in VB ASP.NET spearheaded by Vinhold.
Building WHIP, at least under Linux, is covered in the WHIP wiki and with some additional notes in this pull request for it.
Please clarify what you would be expecting a Grid Management Program to accomplish and I might have some ideas, right now the concept means different things to different people.

I did see the notation on the wiki for quesiton 1.  However for it to be fully functional in grid mode it appears it needs at minimum landtools.php (or equivelant).

Are there any of which I can download and look at to give myself ideas as to what I would be dealing with for a website interface should I go with Halcyon as base code?.

I did not see the wiki notation on the WHIP-Server repo.  In reviewing it though there unfortunately is not instructions to build on Windows.

For the Grid Management Program I am looking at the prospects of a program that can manage grid and region instances, management and creation or removal of regions/ estates, etc without having to actually log into the server/ VM machine to manage the instances

Additional Question:  Is password hashing still using MD5 or has this now moved to SHA512?

1&2. At this point I'm going to see if I can nudge Vinhold to contact you as he's got more answers for the website interfaces than I do at this time.

I'm afraid that WRT to WHIP and Halcyon I'm a Linux guy.  I know that building under Windows was "easy" according to Vinhold and others - WHIP was initially designed to be built and run in Windows.  It's Linux that's the new target for the code.

Maestro was designed for that task, but it's development has stalled.  Vinhold, with my help, has developed a few procedures and website tools that make manual processing for smaller grids less painful.  InWorldz has their own systems, of which Maestro was slated to take over when the ability to take over existing instances managed by another system gets coded.  In theory Maestro should be able to handle a fresh grid fine, but I've no experience with it.

Password hashing is still using MD5.  Patches are welcome as long as the result still can handle existing hashed passwords.

@emperorstarfinder: A lot has happened and put in the Halcyon Wiki, Configuration page specifically the "Single server 1-10 regions" section for setup, since your prior last post on Jan. 23 and now.
I will be working on refining the grid setup and configuration models as I complete the setup of three Halcyon grids for another company this next week. I am also setting up a download website for my "My World" website. I will be posting in the Halcyon Wiki page the website location soon as I can get it set up. Perhaps later this week. It is a nearly complete website in VB.Net 4.0 to setup and run a grid using Halcyon as its core.
I am setting it up as an open source project so others may contribute other flavors of website programming or extend what I have already done. While it is a full working website for the grid, it is not a content management system, and can be extended using the supplied website page templates for Visual Studio 2015, which will also work with Visual Studio 2017. I have not yet updated, but I am informed it is a good upgrade.
I have been just swamped with this grid setup which is involving Linux for the website host, a Linux MySQL server, Linux Whip server, Linux Grid services server and Windows 2008 Web Server R2 as the Region server. The objective is to be able to run fully in Linux as soon as the PhysX port is in place for the Halcyon region processing. Kf6kjg has been doing the Linux configuration for the grid services setup and posting some minor corrections to get them running. Proof will be in what happens over the next two days as I bring up the first of the grids. Yes, my website will be tested in this process to run under Apache with mono as the website host.
I had helped sonjamichelle get her site / grid setup using the "My World" website with its earlier editions. A lot has been done to the programming since then and being applied to the grids I am creating this next week.
I will be adding more information to the Halcyon configuration for the config and Halcyon.ini as well as firewall settings for the different grid size setups. I have also completed the smallest grid configuration on my own Windows 2007 Pro desktop with the website running on my home server. The Whip, grid services and one region are all on one server. I am duplicating the same site / grid on a virtual server running the Win 2008 Web Server R2 where the website, MySQL, grid services and region server are all running on it. Having done this I will be refining the setup information on how it was done in the Halcyon Wiki pages. It is as close as it comes to the nearly mythical "Halcyon on a Stick" idea. It cannot be distributed with Windows as the OS for it. So hopefully soon the entire grid can run in Linux!

Good morning,
@Vinhold Thank you for that in-depth update on your website interface.  I look forward to giving it a test as I continue to test Halcyon to see if it will meet my standards for the projects I would be using it as base code for.
I upgraded recently to Visual Studio 2017 and can assure you it is a nice upgrade from VS2015.  The layout is pretty much unchanged though it runs much smoother then VS2015 seemed to.  It has support for .Net 4.0, 4.5, 4.5.1, 4.5.2, 4.6, 4.6.1, and 4.6.2 .  It also has support for development using the new .Net Core.  If your using Visual Studio 2015 I recommend upgrading to VS2017.  VS2017 also has integrated support for development using Xaramin as well which will make for nice cross platform development.
I did read the wiki pages, and one of the things I noticed is it appears to be a more general direction for install and configuring.  Perhaps an actual step by step guide would be more helpful to end-users who may not be necessarily familiar with the way Halcyon works.
I am glad to hear that work is happening to get PhysX running on Linux along with the ability to be able to run the entire grid on Linux.
A question I would have is:
does your My World website implementation have the ability to use a payment processor for purchasing of in-world currency and regions by users?

The MyWorld website implementation includes functionality for a no economy setup, limited economy setup and a full economy setup. This was a project idea that I brought to @Vinhold and he took off with it like a lion with a fresh kill. It's a very impressive piece of work.
Unfortunately my PC died Friday night so I have no way to bring my site online so you can take a look at what can be done with @Vinhold's MyWorld package. By tweaking the CSS and using the site templates you can customize the look and feel of your own site. You do need an IIS server and a separate MySQL database for the website. The database can be hosted on the same MySQL server as your grid if you have a small traffic site and grid.

@sonjamichelle This is good to hear.  I am jumping out on a limb here and assuming then that full economy setup with the MyWorld site would in fact actually support purchasing of regions as well.
Unfortunately I don't have a copy of the My World site code as of yet or id test it myself.  Setting up IIS usually isn't a hard thing to do but does sound like some great things have been done.
Im sorry to hear your PC died.  Hopefully it won't take much to get sorted.  They are just so important in this day and age.

As of the last update I was involved in the Economy Module didn't do things like allow users to go to a website where they could click on a map to purchase a region on the grid. The Economy Module module enabled in-world payments and transactions. Buying land, purchasing goods, renting property, etc.
To get "Region Sales" from the website would probably require some integration with the Anaximander world map software. @kf6kjg would be the one to bring in on that aspect as well as @Vinhold to integrate that functionality into the website code. Sounds like a major undertaking to me, but I'm no coder. We'll have to see what those guys say.

Okay so Anaximander is what runs the world map features.  That answers that question.  It would be a wise idea I think to add the ability to purchase regions via the website that way there isn’t some complicated process to do so.
I have started to dive into the code today to fully review and audit the code to determine if it is feasible for the projects I am working on and various grids I will be launching.  I have already identified certain things I will have to port into my local instance of Halcyon should I decide to use it as my base code.
But I will be interested to hear more on whether that is something that could in fact be implemented in the My World site as I dive into the code base and do my rieviews.

Anaximander is only a fast map generating process built by Kf6kjg posted at Anaximander2 Git hub that makes the map tiles which can be used in a world display page. I have not had a chance to create that page for the My World website. It is on the "to do" list. For region sales / purchases, a second map page would be used that would show reserve boundary around existing lands and spaces that are open to select.  Map page displays are an entire art of their own. lol
The My World website does use its own MySQL DB and tables only because it has its own page data to track as well as custom processing for remote region management all content that is not suitable to be in the grid DB. It is of small content and can be run on the same MySQL Serve as the grid DB with no problems, even for a large grid operation. Its traffic is very small in comparison to the grid.
I have noted the comment regarding specific implementation details for Halcyon setup. I want to do that but I have to get out my "mountain climbing gear" to tackle the steep learning curve I have to make for proper Wiki page authoring to contain all the information I have been collecting for it and get it all nicely formatted for easy reference. :)

Yes map art should be an industry unto itself lol.
Would it be possible to get a copy of the current My World website so I can have a peak at it as I continue my testing?  I am glad to hear that ability to provide sales/ purchasing of regions and renewals of regions will be added.  That is a good step.
ANd I would assume if it is able to run under apache it does not necessarily need IIS to host it.

The My World website was initially run in II7 on Windows. I am atm working on getting it to run in Ubuntu with Apache and mod-mono and being as confused as a good rookie in Linux can get. lol! It is easy in IIS7 to set up. But like any such stuff, depends on your experience. In IIS7 you have to set up redirects for the Landtool.php to point to LandTool.aspx instead or land sales wont work in the My World site.  I can get you started. Open my profile and my website page there. Use the Contact Us page to email me to Director. I will get you how to download the MyWorld zip file and instructions. :)

I assume you mean Project Director?

Yes. lol, I forgot which name was put in it since I dont spend time sending myself messages using that page. :p

Okay was double checking prior to sending.  Ill link this Issue on the form. Then I don't have to go about explaining a ton of things when I got about a billion tasks on the desk today.

Lol! I am expecting your comment, link not really needed other than some related identity.

Okay email just sent.   Though forgot to sign my name at the end but it should be pretty easy to figure out its me.

Just a clarification for the comments earlier on Anaximander "map software". It's not map software at all, and region sales would not be done through Anaximander.  What the software does is generate map tiles from region content (including terrain).  It is a graphics program that produces an image file for each region, and also merges those to represent zoom levels: each 2x2, 3x3, 4x4, etc.
Those tiles are then provided to a few consumers by a simple location fetch API. It takes X and Y coordinates and a zoom level and returns a texture.  That API is then used by the map web page code to display a map and respond to interaction, including clicks and panning.  Depending on which map it is, it may support region purchases.
Most of this is completely independent of Halcyon code.  Linden Lab documents the mapping software and API here, and obviously they are not connected to the Halcyon project (or Anaximander).  Viewers also pull the map tiles via the tile API for display in their internal World Map windows.
Second Life provides maps via the maps.secondlife.com domain, and InWorldz provides something similar at places.inworldz.com.  These domains are used for "SLURL"-like links to locations, for example, http://places.inworldz.com/InWorldz%20Desert%20Island/128/128/25 will open a small popup with teleport options.  This is part of the mapping website code and are recognized internally by viewers to pop open the World Map instead of launching a browser.  See the LL documentation for these URLs here.
Both InWorldz and Second Life used the Google Maps API to draw the actual maps, with tiles obtained separately (we hope to use Anaximander for that in InWorldz soon).  Last year, support for the v2 Google Maps API changed significantly, and both SL and IW replaced it with the much simpler, faster, and less Google-entrenched leaflet.js map library.

Thank you Appurist, I was going to ask you for the new setups for the map pages once I can get to working on it. Kf6kjg will also be helping me do it.

Good afternoon,
After a bit of playing around with the architecture and figuring out its operation (how it is supposed to be started up, etc), I have managed to get in a good test.
After a mini test locally, it became apparent there are a few things i would need to modify if I were to use Halcyon in the base of my projects.  Things such as:

Implementing Var Regions (variable region sizes)
Infinite Regions (this does not impact viewers but rather just removes the invisible wall around the region thus allowing users to walk outside their region even without a neighboring region being present without it crashing the user)
My own region restart implementation which would allow users on my projects to remain in their regions while the region restarts without having to leave the region or them being logged out.  Yes this actually does work and my current grid architecture i have it working.  Viewers will go grey for about 10 to 20 seconds and upon receiving packets from the region once it has restarted resumes normal operation with the user still being logged in.
Multiple different terrain types (i.e. aquatic, grassland, marshland, mountainous, etc)  I would also make it be rendered in a procedural way so the terrain looks different in each set for the default in each region. In fact this one I definitely would recommend being implemented in the core of Halcyon so you don't have the default terrain that you received originally from opensim.  Where your not Opensim it would be a good thing to help you continue defining your independence as a project.

Recommendations
As I tested this I found that the configuration files as they are now are a bit to intimidating and cumbersome (similar to the way the configuration files in Opensimulator are).  It would be a better approach to split the configuration files up.  For example configuration items relating to scripting would go in a file called Scripting.ini and physics related configurations would go in a file called Physics.ini.  Additionally I would encourage that configuration files be placed in their own directory preferably outside of the bin directory.  In doing this there would be less chance of users losing their configuration settings they have set when upgrading to a new version (i.e. copying and pasting the bin directory from an updated version to their grid and region servers).
It also might be a good idea to provide instructions for creating the virtual machines on nix and windows hardware on the wiki for those that may not be knowledgeable on how to do that.
I have been looking through the MyWorld site provided by @Vinhold and hope to be able to test it against a configured local grid this week.  (it looks good so far).
I will probably have more as I continue testing and reviewing the code this week.  However I have recommended Halcyon to the other members on my team for their review for our team projects along with adding it to consideration for my personal projects.

I most  say  this  is  an complicated task, a  normal user can never figure this all out, as for  me I  did  not get halcyon to work I  mean its  not  like  opensim  out of the  box install, and  I  can not  find  any halcyon.ini  file whats  complete and would work in grid  mode with some port  and IP changes, like  put here your IP address or domain, then whip and then this unblock sequence, I  have  all  server  software 180 km  away in a  server  farm and  my assess runs over a special login viewer. Well  has  somebody  a  working  halcyon.ini and  whip.cfg  for  me and  from what  I  understood if I  run 22 regions I  need  halcyon 22 time started  up?

Yes you would need to run 22 instances of Halcyon.  the Halcyon.ini file is in the bin folder and is named Halcyon.ini.example.  You need to save that as Halcyon.ini.  the Whip.cfg file you will find in the whip server repository.  It is better to set up virtual machines to do this.
I currently have for my team's grid(s) 1 VM for website services i.e. wiki, mantis, main website, etc. (200GB is a good size to start),  1 VM for the Database Server (1TB is good size to start), 1 VM for Grid Services i.e. Whip Server, Grid Server, User Server, Message Server, Inventory Server (1 TB is a good size to start), 1 VM per 10 Regions (Each Region VM you should be able to get away with 50GB to 100GB for VM machine size).  On each region VM you will need to run the Aperture Server which will connect to your whip server and handle the rest.
The order to start your grid up is as follows:
Grid start up Order:

Whip, 2. User Server, 3. Grid Server, 4. Message Server, 5. Aperture, 6. Start each region in order of preference.

VInhold has also done a website and open sourced it that will help you manage your grid complete with instructions for setting it all up at: My World Website.

It is true that Halcyon is a bit more complicated at first and can appear daunting to get the hang of but they have done things that are considerably better then the straight opensimulator.  Hopefully down the road they will make configuration a bit easier and less complicated for people who are not yet familiar with Halcyon.
Hopefully this helps you a little bit at least.

Greetings Xantisi,
I have been a bit sidetracked with a new business I set up, so some of the My World website development has stalled a bit: The world map display is about 80% complete. The land purchase map access for website is to be derived from the world map page once it is complete. The world map setup also implies an Events page which has not been designed yet. And so it it goes. However, aside from the new features, the core ability to set up and operate a grid of one to several hundred regions is able to be accomplished with the My World website as it is. All is explained in the set up instructions from start to finish. Eventually we will reach a point where you can install the website, set up server virtual machines and startup a management module and open the website to create your grid. It has been a long process to get to this point with the few of us working on it. But excellent progress is being made.
Please note the Halcyon Wiki Configuration page for several grid size options. I have build running grids for each grid size except the largest one, thus determining the steps for each level as documented. While the My World website will help with grid setup and management a lot, it cannot set up the basic grid and the virtual machines needed on your server to make a strong grid host for you. So my setup instructions deals with how to create and set up your server(s) to have the grid you desire that will support the number of users you would like to have in it.
Please contact me from the My World website for more direct help if you wish. Note: I am not one of the Halcyon developers, just a long time Halcyon user and doing all I can to help the project. :)

The initial issue for this seems to have been resolved. Thank you to all who have participated, as this has been friendly and informative for all.


Freshness work - PIM tutorial

Hi @mandardalvi, I made some minor edits as part of freshness work for this article. Requesting for your review of the same.

@timayabi2020, please merge this.


error LNK2038: mismatch detected for '_ITERATOR_DEBUG_LEVEL': value '2' doesn't match value '0'

Following the Windows instructions to build a sample native application using the C++ backend, I'm getting the following errors:
PortableRuntime.lib(RuntimeInstance.obj) : error LNK2038: mismatch detected for '_ITERATOR_DEBUG_LEVEL': value '2' doesn't match value '0' in corert-demo.obj
PortableRuntime.lib(RuntimeInstance.obj) : error LNK2038: mismatch detected for 'RuntimeLibrary': value 'MTd_StaticDebug' doesn't match value 'MT_StaticRelease' in corert-demo.obj
...

Turns out that the C++ code is compiled with /MT but I compiled CoreRT by just using build.cmd which defaults to debug mode compiled using /MTd. Compiling the C++ generated file using /MTd solves the linker error.
I'm not sure what is the best course of actions here. At least this is documented and someone else having the problem will see what needs to be done.

The instructions should be updated to add --cppcompilerflags /D_DEBUG for debug CoreRT builds.
Introduced by https://github.com/dotnet/corert/commit/650f99d78c2bc0c89a81e30d9c758f184514ba98#diff-adec353b8bdbfa8c549ed738939291e8R141

--cppcompilerflags are added by "dotnet compile --native --cpp" automatically. @manu-silicon How are you compiling your C++ code?

The issue is not that they are missing, it is that they are wrong when you are using a debug build of CoreRT. As @jkotas mentioned adding --cppcompilerflags /D_DEBUG (or --cppcompilerflags /MTd) solves the issue. Although using /D_DEBUG still generates a linker warning about conflicting C libraries, whereas /MTd does not.

This is not the common production scenario where we expect to link against release version of CoreRT. I think it is fair to document addition of how to use the switch - do you want to start a PR for the documentation update @manu-silicon ?

do you want to start a PR for the documentation update @manu-silicon

I have done that already #809

Just tried again today after updating and now it looks like I need to pass /MDd instead of /MTd. Did something changed in the way  PortableRuntime.lib is built in debug mode?

Yes, we had to rollback https://github.com/dotnet/corert/pull/819 to workaround issues with live version of cli - that also include the change how the runtime library is built.
I just submitted PR with the doc fix.

We will switch back to live version of cli again (rollback the rollback) in near feature once the issues with live version of cli are resolved  - so it will change again.


e2e: conditionally querying total escrow based on release version

Description

closes: #XXXX
Commit Message / Changelog Entry
type: commit message

see the guidelines for commit messages. (view raw markdown for examples)

Before we can merge this PR, please make sure that all the following items have been
checked off. If any of the checklist items are not applicable, please leave them but
write a little note why.

[x] Targeted PR against correct branch (see CONTRIBUTING.md).
[ ] Linked to Github issue with discussion and accepted design OR link to spec that describes this work.
[ ] Code follows the module structure standards and Go style guide.
[ ] Wrote unit and integration tests.
[ ] Updated relevant documentation (docs/) or specification (x/<module>/spec/).
[ ] Added relevant godoc comments.
[ ] Provide a commit message to be used for the changelog entry in the PR description for review.
[x] Re-reviewed Files changed in the Github PR explorer.
[ ] Review Codecov Report in the comment section below once CI passes.

Thanks for the comments, @chatton. I will address those in #3368.


Model logging, deleted users and custom properties

Hi there,
Firstly, I want to say this is a great logging solution, I have been using it for a few months and have found it very easy to use.
I have noticed that when a user model is deleted, this causes a bit of an issue as it obviously can't enumerate the user model. I don't really want to enable softDeletes on my user model and saw another post which suggested logging a custom property might be the solution.
I think I could make this work, I could log a custom property with the user name at the time of logging the model event. If the user model no longer exists, I could fallback to the logged user name custom property.
My question is; is it possible to perform model logging and log a custom property at the same time?
Ultimately, what I am trying to achieve here is to manage the logs in such a way that I can report on events even after a user has been deleted. If there's a better way, feel free to point that out too!
Thanks.

is it possible to perform model logging and log a custom property at the same time?

I think so yes. Here's how you can add custom properties: https://docs.spatie.be/laravel-activitylog/v2/basic-usage/logging-activity#setting-custom-properties

Thanks - maybe a silly question... will that work?
If we are logging events that are performed on a model [create, update, delete], the code to add a log entry isn't specifically called from the model - I assume it uses an observer?

OK - I've taken a different route with this.
For anyone else trying to add extra properties when logging model evens, the easiest way I have found is to customise the LogsActivity Trait.
To do this, simply take a copy of the LogsActivity trait, change the namespace and reference it in your model instead of the default.
Don't forget to use DetectsChanges in your new trait if you change the namespace.
Once done, you can customise the boot method to add your new properties as additional information to the withProperties function.
Here is an example:
protected static function bootCustomTrait()
    {
        static::eventsToBeRecorded()->each(function ($eventName) {
            return static::$eventName(function (Model $model) use ($eventName) {
                if (! $model->shouldLogEvent($eventName)) {
                    return;
                }

                $description = $model->getDescriptionForEvent($eventName);

                $logName = $model->getLogNameToUse($eventName);

                $prop1 = 'new property 1';

                $prop2 = 'new property 2';

                $properties = array_merge( $model->attributeValuesToBeLogged($eventName), ['prop1' => $prop1, 'prop2' => $prop2] );

                if ($description == '') {
                    return;
                }

                app(ActivityLogger::class)
                    ->useLog($logName)
                    ->performedOn($model)
                    ->withProperties($properties)
                    ->log($description);
            });
        });
    }


Error: connect EHOSTDOWN <IP_ADDRESS>:80 - Local

@syndbg @bw @youanden @chown9835
Error: connect EHOSTDOWN <IP_ADDRESS>:80 - Local
my webpack.config.js:
new WebpackGoogleCloudStoragePlugin({
      directory: './dist',
      // NOTE: Array of filenames to include in the uploading process
      include: ['.js', '.html', '.css', '.svg', '.png', '.jpg'],
      // NOTE: Array of filenames to exclude in the uploading process
      // exclude: ['cats.js'],
      // NOTE: Options passed directly to
      // Google cloud Node Storage client.
      // This is mostly authentication-wise.
      // For more information:
      // https://github.com/GoogleCloudPlatform/google-cloud-node/tree/master/packages/storage#authentication
      storageOptions: {
        projectId: 'project_id',
      },
      // NOTE: Options used by
      // WebpackGoogleCloudStoragePlugin
      // regarding uploading.
      uploadOptions: {
        // Where to upload files
        bucketName: 'bucket-with-my-domain.com',
        destinationNameFn: file => file.path.replace('dist/', ''),
        forceCreateBucket: true,
        gzip: true,
        makePublic: true,
      },
    }),

traceback:
events.js:163
      throw er; // Unhandled 'error' event
      ^

Error: connect EHOSTDOWN <IP_ADDRESS>:80 - Local (<IP_ADDRESS>:51072)
    at Object.exports._errnoException (util.js:1050:11)
    at exports._exceptionWithHostPort (util.js:1073:20)
    at internalConnect (net.js:889:16)
    at lookupAndConnect (net.js:977:5)
    at Socket.realConnect (net.js:945:5)
    at Agent.connect [as createConnection] (net.js:77:22)
    at Agent.createSocket (_http_agent.js:195:26)
    at Agent.addRequest (_http_agent.js:157:10)
    at new ClientRequest (_http_client.js:212:16)
    at Object.request (http.js:26:10)
    at Request.start (/node_modules/google-auto-auth/node_modules/request/request.js:740:32)
    at Request.end (/node_modules/google-auto-auth/node_modules/request/request.js:1394:10)
    at end (/node_modules/google-auto-auth/node_modules/request/request.js:567:14)
    at Immediate.<anonymous> (/node_modules/google-auto-auth/node_modules/request/request.js:581:7)
    at runCallback (timers.js:672:20)
    at tryOnImmediate (timers.js:645:5)
    at processImmediate [as _immediateCallback] (timers.js:617:5)

Changed to "@google-cloud/storage": "1.5.2"

Will do PR with that.

Can we consider this fixed after the PR was merged and 0.8.0 was released?

Sure! Thanks!


This is Zac Fowler and my introduction

Description
[Describe what this change achieves]
Issues Resolved
[List any issues this PR will resolve]
Check List

[ ] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the BSD-3-Clause License.

Hey @RealZacharyFowler actually could you remove the Instagram? That field can only handle twitter handles as it generates links automatically and the string there will break our build.

never heard back from @RealZacharyFowler - closing


Link to supplementary material

Hi, could you kindly include a link to the supplementary material.

sure, i have added the link in README.md.