Supervisor setup with 2 dumb processes

In Action
See it in action with these steps:

Build container:

# for Podman, substitute 'docker' for 'podman'
docker build --no-cache -t data_platform_main__local -f docker/Dockerfile.main .

Run container:

# for Podman, substitute 'docker' for 'podman'
docker run --rm --volume $PWD:/data_platform:ro data_platform_main__local supervisord -n -c /data_platform/supervisor/conf/dataplatform.conf

Stop container with CTRL-C. (Note: Container will not stop right away.)

Links

Asana

question: is supervisor/ a Python module, or a directory for scripts? If it's the latter, I don't believe you need a __init__.py.

when I used ^C to exit out of the container, it exited immediately: it didn't appear to wait for anything to shut down.

Overall, this seems like a lot of boilerplate for something that's pretty straightforward in Elixir, and it would match our other ECS applications. Given that this application isn't interfacing directly with Spark (or even the data at all), does it make sense for it to be in Python?

@paulswartz

The supervisor process command is this /usr/local/bin/python -m supervisor.cubic_ods__process_incoming, so I added the __init__.py because of the -m
I'm looking into the ^C to see if I missed anything.

Regarding your comment about Elixir. You're probably right. Elixir is more advanced in this sense. Although I have to be frank in that I would need ramp up on it to do this type of work. I mentioned this when we talked last week that this work can be done with Elixir, but I was choosing python so we can attract more talent in the data engineering area and make everything about the platform approachable by that talent. Maybe we can bring this up to a bigger group of folks, but we should make a decision soon, so I can continue to execute. Let me know what you think.

Given that this application does not need to interface with Spark, I believe it would be better to have it written in Elixir. The data engineering is changing, with new libraries like Polars in Rust and Explorer in Elixir. It's also the language all of our other ECS tasks are in, meaning that we have lots of institutional knowledge about how to deploy/monitor/maintain Elixir services.
If you still feel strongly that it should be in Python, an RFC would be the appropriate way to discuss with the larger engineering group.


'onmessage' and 'postMessage' missing from 'self' in 'lib.webworker.d.ts'

It's currently an error to use self.onmessage and self.postMessage.

tracked by https://github.com/Microsoft/TypeScript/issues/14877


uvx on old mac

I apologies if off topic or already answered.
I'm on an old mac with system python2.7
I do not want to modify the system python, so I'm thinking of using uv (uvx) to have a self contained environment independent from the system. I installed uv 0.5.5 (and I validated that it works perfectly if I create a venv with uv venv .venv and then I activate)
when I try to use uvx running the following:
uvx --python 3.13 python -c 'print("hello world")'
I got this error
  × No solution found when resolving tool dependencies:
  ╰─▶ Because python was not found in the package registry and you require python, we can conclude that your requirements are unsatisfiable.

is uvx not supposed to run on old systems where python2.7 is the default ?

uvx is trying to run a module, and python is not a module I think.
Could you try uv run python 'print("hello world")' ?

thanks,
uv run --python 3.13 python -c 'print("hello world")' works perfectly.
actually also running uvx ruff --version works as expected
I think I got tricked by the example in the documentation where the example is
uvx pycowsay 'hello world!'
that gives me the following error

~/Library/Caches/uv/archive-v0/Q3qULlnBMDe-tYneApmmI/bin/pycowsay: line 2: realpath: command not found
~/Library/Caches/uv/archive-v0/Q3qULlnBMDe-tYneApmmI/bin/pycowsay: line 2: ~/uv/python: No such file or directory
~/Library/Caches/uv/archive-v0/Q3qULlnBMDe-tYneApmmI/bin/pycowsay: line 2: exec: ~/uv/python: cannot execute: No such file or directory

@Stegallo Can you run uvx --verbose ruff and check which Python version it's using for you?

@FishAlchemist
uvx --verbose ruff --version

DEBUG uv 0.5.5 (95cd8b8b3 2024-11-27)
DEBUG Searching for default Python interpreter in managed installations or search path
DEBUG Searching for managed installations at `~/.local/share/uv/python`
DEBUG Found managed installation `cpython-3.13.0-macos-x86_64-none`
DEBUG Found `cpython-3.13.0-macos-x86_64-none` at `~/.local/share/uv/python/cpython-3.13.0-macos-x86_64-none/bin/python3.13` (managed installations)
DEBUG Acquired lock for `~/.local/share/uv/tools`
DEBUG Released lock at `~/.local/share/uv/tools/.lock`
DEBUG Caching via interpreter: `~/.local/share/uv/python/cpython-3.13.0-macos-x86_64-none/bin/python3.13`
DEBUG Using request timeout of 30s
DEBUG Solving with installed Python version: 3.13.0
DEBUG Solving with target Python version: >=3.13.0
DEBUG Adding direct dependency: ruff*
DEBUG Found fresh response for: https://pypi.org/simple/ruff/
DEBUG Searching for a compatible version of ruff (*)
DEBUG Selecting: ruff==0.8.1 [compatible] (ruff-0.8.1-py3-none-macosx_10_12_x86_64.whl)
DEBUG Found fresh response for: https://files.pythonhosted.org/packages/89/a8/a957a8812e31facffb6a26a30be0b5b4af000a6e30c7d43a22a5232a3398/ruff-0.8.1-py3-none-macosx_10_12_x86_64.whl.metadata
DEBUG Tried 1 versions: ruff 1
DEBUG marker environment resolution took 0.015s
Resolved 1 package in 16ms
DEBUG Running `ruff --version`
DEBUG Looking at `.dist-info` at: ~/Library/Caches/uv/archive-v0/yogUHl1xqVbsxkynXGQs0/lib/python3.13/site-packages/ruff-0.8.1.dist-info
ruff 0.8.1
DEBUG Command exited with code: 0

however the issue still happens with a command like black
uvx --verbose black --version

DEBUG uv 0.5.5 (95cd8b8b3 2024-11-27)
DEBUG Searching for default Python interpreter in managed installations or search path
DEBUG Searching for managed installations at `~/.local/share/uv/python`
DEBUG Found managed installation `cpython-3.13.0-macos-x86_64-none`
DEBUG Found `cpython-3.13.0-macos-x86_64-none` at `~/.local/share/uv/python/cpython-3.13.0-macos-x86_64-none/bin/python3.13` (managed installations)
DEBUG Acquired lock for `~/.local/share/uv/tools`
DEBUG Released lock at `~/.local/share/uv/tools/.lock`
DEBUG Caching via interpreter: `~/.local/share/uv/python/cpython-3.13.0-macos-x86_64-none/bin/python3.13`
DEBUG Using request timeout of 30s
DEBUG Solving with installed Python version: 3.13.0
DEBUG Solving with target Python version: >=3.13.0
DEBUG Adding direct dependency: black*
DEBUG Found fresh response for: https://pypi.org/simple/black/
DEBUG Searching for a compatible version of black (*)
DEBUG Selecting: black==24.10.0 [compatible] (black-24.10.0-cp313-cp313-macosx_10_13_x86_64.whl)
DEBUG Found fresh response for: https://files.pythonhosted.org/packages/d0/a0/a993f58d4ecfba035e61fca4e9f64a2ecae838fc9f33ab798c62173ed75c/black-24.10.0-cp313-cp313-macosx_10_13_x86_64.whl.metadata
DEBUG Adding transitive dependency for black==24.10.0: click>=8.0.0
DEBUG Adding transitive dependency for black==24.10.0: mypy-extensions>=0.4.3
DEBUG Adding transitive dependency for black==24.10.0: packaging>=22.0
DEBUG Adding transitive dependency for black==24.10.0: pathspec>=0.9.0
DEBUG Adding transitive dependency for black==24.10.0: platformdirs>=2
DEBUG Found fresh response for: https://pypi.org/simple/mypy-extensions/
DEBUG Found fresh response for: https://pypi.org/simple/pathspec/
DEBUG Found fresh response for: https://files.pythonhosted.org/packages/2a/e2/5d3f6ada4297caebe1a2add3b126fe800c96f56dbe5d1988a2cbe0b267aa/mypy_extensions-1.0.0-py3-none-any.whl.metadata
DEBUG Found fresh response for: https://files.pythonhosted.org/packages/cc/20/ff623b09d963f88bfde16306a54e12ee5ea43e9b597108672ff3a408aad6/pathspec-0.12.1-py3-none-any.whl.metadata
DEBUG Found fresh response for: https://pypi.org/simple/click/
DEBUG Found fresh response for: https://pypi.org/simple/packaging/
DEBUG Searching for a compatible version of click (>=8.0.0)
DEBUG Found fresh response for: https://pypi.org/simple/platformdirs/
DEBUG Selecting: click==8.1.7 [compatible] (click-8.1.7-py3-none-any.whl)
DEBUG Found fresh response for: https://files.pythonhosted.org/packages/00/2e/d53fa4befbf2cfa713304affc7ca780ce4fc1fd8710527771b58311a3229/click-8.1.7-py3-none-any.whl.metadata
DEBUG Searching for a compatible version of mypy-extensions (>=0.4.3)
DEBUG Selecting: mypy-extensions==1.0.0 [compatible] (mypy_extensions-1.0.0-py3-none-any.whl)
DEBUG Found fresh response for: https://files.pythonhosted.org/packages/88/ef/eb23f262cca3c0c4eb7ab1933c3b1f03d021f2c48f54763065b6f0e321be/packaging-24.2-py3-none-any.whl.metadata
DEBUG Searching for a compatible version of packaging (>=22.0)
DEBUG Selecting: packaging==24.2 [compatible] (packaging-24.2-py3-none-any.whl)
DEBUG Searching for a compatible version of pathspec (>=0.9.0)
DEBUG Selecting: pathspec==0.12.1 [compatible] (pathspec-0.12.1-py3-none-any.whl)
DEBUG Searching for a compatible version of platformdirs (>=2)
DEBUG Selecting: platformdirs==4.3.6 [compatible] (platformdirs-4.3.6-py3-none-any.whl)
DEBUG Found fresh response for: https://files.pythonhosted.org/packages/3c/a6/bc1012356d8ece4d66dd75c4b9fc6c1f6650ddd5991e421177d9f8f671be/platformdirs-4.3.6-py3-none-any.whl.metadata
DEBUG Tried 6 versions: black 1, click 1, mypy-extensions 1, packaging 1, pathspec 1, platformdirs 1
DEBUG marker environment resolution took 0.119s
Resolved 6 packages in 147ms
DEBUG Running `black --version`
DEBUG Looking at `.dist-info` at: ~/Library/Caches/uv/archive-v0/Z8sv2ZcObyjX3Wn2eFDXy/lib/python3.13/site-packages/black-24.10.0.dist-info
DEBUG Looking at `.dist-info` at: ~/Library/Caches/uv/archive-v0/Z8sv2ZcObyjX3Wn2eFDXy/lib/python3.13/site-packages/click-8.1.7.dist-info
DEBUG Looking at `.dist-info` at: ~/Library/Caches/uv/archive-v0/Z8sv2ZcObyjX3Wn2eFDXy/lib/python3.13/site-packages/mypy_extensions-1.0.0.dist-info
DEBUG Looking at `.dist-info` at: ~/Library/Caches/uv/archive-v0/Z8sv2ZcObyjX3Wn2eFDXy/lib/python3.13/site-packages/packaging-24.2.dist-info
DEBUG Looking at `.dist-info` at: ~/Library/Caches/uv/archive-v0/Z8sv2ZcObyjX3Wn2eFDXy/lib/python3.13/site-packages/pathspec-0.12.1.dist-info
DEBUG Looking at `.dist-info` at: ~/Library/Caches/uv/archive-v0/Z8sv2ZcObyjX3Wn2eFDXy/lib/python3.13/site-packages/platformdirs-4.3.6.dist-info
~/Library/Caches/uv/archive-v0/Z8sv2ZcObyjX3Wn2eFDXy/bin/black: line 2: realpath: command not found
~/Library/Caches/uv/archive-v0/Z8sv2ZcObyjX3Wn2eFDXy/bin/black: line 2: ~/uv/python: No such file or directory
~/Library/Caches/uv/archive-v0/Z8sv2ZcObyjX3Wn2eFDXy/bin/black: line 2: exec: ~/uv/python: cannot execute: No such file or directory
DEBUG Command exited with code: 126

@Stegallo I just want to say it's normal not to use Python 2.7.
But didn't this problem exist in older versions of uv? Or is this actually a problem specific to Python 3.13?

@FishAlchemist
I have not tried with older versions of uv.
maybe I have a misunderstanding of the use cases for uvx.
The way I understand it is that I can use uvx if I don't want to explicit install the newest version of python (3.13) on my system for using modern python tools, especially on an old machine where python2.7 is still used by the system.
I don't think this is critical as we are talking about old machines and I have a functioning workaround.
My workaround is to use pyenv and install the version of python3 that I am interested in.

I think the problem is realpath is not available on your system.
❯ which realpath
/bin/realpath

Can you confirm?

Thanks @zanieb
yes. you are correct
➜  ~ which realpath
realpath not found

I installed it via brew install coreutils and now uvx works perfectly!
Thank you!


Fix #24 #355, Add compile flag abstractions

Describe the contribution
Fixes #24, Fixes #355
Adds STRICT_NO_WARNINGS and OMIT_DEPRECATED
prep options for CI and as example build
Testing performed
Built with STRICT_NO_WARNINGS=true, false, and undefined
Built with OMIT_DEPRECATED=true, false, and undefined (note sample_app fails)
Confirmed flags get globally applied (not great scoping, but ensures flag coverage)
Expected behavior changes
Simplifies/enables CI based on abstracted flags, see https://github.com/nasa/cFS/issues/39
System(s) tested on:

cFS Dev Server 2
OS: Ubuntu 18.04
Versions: Master bundle with this change

Contributor Info
Jacob Hageman - NASA/GSFC

Preference is for complexity to be shifted to the build system vs CI to simplify user's burden to execute similar tests locally (and adapt the sample as needed).  Our CI has external dependencies that the build system doesn't have, if a user want's to try out the examples it's trivial and obvious when provided as part of the build system (just flags to prep).

Understood, but CI needs to be more extensive anyway, because it currently does not sufficiently cover the compile-time options available.  This has been discussed in the past and also I believe there are some related tickets to test with different configs.
My suggestion is to make more than one "defs" directory here, so the sample_defs (which the guides suggest new users clone as their starting point) can remain simple and straightforward.
The other "defs" directories can be used during CI as well as examples of more complex configs.
But the main issue is the logic being proposed is something very CI-specific, and absolutely NOT a good example of the right way to do things.

CCB - 20200108: discussed compiler flags in depth, scoping for mission/host vs target would improve things, Joe will submit an update

Took a shot at better scoping, and applied by default.  Note this just defines HOST_STRICT_COMPILE and TARGET_STRICT_COMPILE, the add_definitions would need to go in the appropriate locations.

Rejected in favor of #462 and #463


Add sections

Hello,
i am using the API key when sending and email.
I have added the following:
$sendgrid = new \SendGrid(env('SENDGRID_KEY', 'sendgrid'));
$email = new \SendGrid\Email();

$email
        ->addTo(xxxxxx)
        ->setFrom('xxxxxxxxx')
        ->setFromName('Web Services')
        ->setSubject('My subject')
        // set html or text to an empty space (see http://git.io/hCNy)
        ->setHtml($serial)
        ->setText('Text Hello!') // <-- triggers the plaintext version of the template
        ->setSections(array("%office%" => "an office", "%home%" => "your house"))
        ->addFilter('templates', 'enabled', 1)
        ->addFilter('templates', 'template_id','xxxxxxxxxxxxxx');

but the sections %office% and %home% are not inserted in the template.I cannot see the text "an office" and "your home" but i see just %office% and %home% when i send the email.
What am i doing wrong?

Ok seemed that i had to add the ->addSubtitution parameters among all others.

This still does not seem to work for me (using ->addSection())

HI @geekdevs,
Could you please reach out to our support team for assistance? https://support.sendgrid.com. Please provide the relevant source code so they can troubleshoot more easily. Thanks!

Hey @geekdevs,
Have you been able to resolve this "addSection" problem?
I seem to have the same problem here...
Thanks,

Hello @psavard,
Perhaps I can help you. Could you please provide the following:

Version of the library you are using.
Sample code.

Thanks!
Elmer

@psavard no, sorry. I had to solve it urgently at that time and found some workaround.

In fact, the problem seems to be with the sendgrid API, I have extract my API call from the code and injected it in Postman to trigger the sending of the Sendgrid template and I have the exact same problem, the template don't seem to interpret the sections.
 Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.",
		"-paragraph2-":"To continue, click on the link below",
		"-action_text-":"Click Here"
	},
	"template_id":"********-****-****-****-************"
}'

and my sendgrid template :

	<title></title>

-salutation-,

-paragraph1-

-paragraph2-

		-action_text-

With this result I see 2 possibilities :

Sections tag is supported with the template sytem
There is a bug in the Sendgrid API

What is your though on this one?
Thanks for your help!

Hi @psavard,
I have re-opened this issue so I can try to assist further. This is issue is now on our backlog.
With Best Regards,
Elmer

I also am having issues getting sections working. They seem to just not work. The code below is ripped from my Yii2. Once I get it pushed (very soon) the full code could be inspected. However, I have gotten everything else to work. So I am pretty sure it's on SendGrid's API end.
$mailer = Yii::$app->mailer;
$message = $mailer->compose()
    ->setTo($to)
<EMAIL_ADDRESS>=> 'Alerts'])
<EMAIL_ADDRESS>    ->setSubject('Hey -username-, Read This Email')
    ->setHtmlBody('Dear -username-,

My HTML message here

%section1%')
    ->setTextBody('Dear -username-,\n\nMy Text message here\n\n%section1%')
    ->addSection('%section1%', 'This is the header')
    ->addSubstitution('-username-', $user->username)
    ->send();

From my Message class:
public $sections;

public function addSection($key, $val)
{
    $this->sections[$key] = (string) $val;
    return $this;
}

// --- down where i build the message

if ( isset($this->sections) && is_array($this->sections) ) {
    foreach ($this->sections as $key => $val) {
        $this->getSendGridMail()->addSection($key, $val);
    }
}

I apologize for the Yii2 reference. My code will be viewable soon. However, I am confident that this isn't on my end and this library or the SendGrid API is not handling it properly.

Thanks for the additional feedback @WadeShuler, I've added your vote to this issue.
Since this likely not a library issue, you will get the best assistance from our support team. That said, this is still on our backlog and will continue to gain priority as we get more reactions and comments.

@thinkingserious : is there a way for me to view the sendgrid support ticket?

I've dug a little more on this problem and I've finally made it worked correctly. In order to work, the sections should be referenced in the substitution section.
So, the JSON should looks like this :
{
	"from":{
<EMAIL_ADDRESS>	},
	"personalizations":[
		{
			"to":[
				{
					"name":"test1",
<EMAIL_ADDRESS>				}
			],
			"substitutions":{
				"-name-":"Another One",
				"-abs_url-":"http:\/\/someplace.com\/",
				"-salutation_section-":"-salutation-",
				"-p1_section-":"-paragraph1-",
				"-p2_section-":"-paragraph2-",
				"-action_section-":"-action_text-",
			}
		}
	],
	"subject":"Some subject",
	"content":[
		{
			"type":"text\/html",
			"value":"Dummy content"
		}
	],
	"sections":{
		"-salutation-":"Hello -name-",
		"-paragraph1-":"Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.",
		"-paragraph2-":"To continue, click on the link below",
		"-action_text-":"Click Here"
	},
	"template_id":"********-****-****-****-************"
}

and the template shoud contains something like :

	<title></title>

-salutation_section-,

-p1_section-

-p2_section-

		-action_section-

ref : https://sendgrid.com/docs/Classroom/Build/Add_Content/substitution_and_section_tags.html#-Complex-Section-and-Substitution-01

Thanks for sharing the solution @psavard. Please email us at<EMAIL_ADDRESS>referencing this issue so we can send you some swag :)

@psavard This seems to be a work around hack, not "properly".
The sections should work simply by passing them as:
"sections":{
    "-section1-": "This is my section",
},

Then of course, referencing -section1- in the template/content.
While your way might work, SendGrid should actually fix it so they work like they are supposed to.


list the "debug" option

While debugging #20, I had to figure out the debug option from the code, since it's not listed in the README.
It should, shouldn't it ?

Dead.


button to force update trackers chapter count

Describe your suggested feature
I would really like to have a button in advanced settings of the app.
Just like force recheck downloads, it forces the trachers to update chapter counts for all entries with a tracker enabled.
Other details
When trackers stop working (like MAL does) and the chapter count gets stuck/stagnates, I have to manually update each entry once the tracking starts working again.
That's why I would really like a way to force all of them to be Synced
Acknowledgements

[X] I have searched the existing issues and this is a new ticket, NOT a duplicate or related to another open or closed issue.
[X] I have written a short but informative title.
[X] I have updated the app to version 0.16.5.
[X] I will fill out all of the requested information in this form.

I can almost guarantee that's not gonna happen as it would enable users to bombard the trackers with requests, which would at best cause a temporary block on the user for hitting a rate-limit, and at worst a ban or even them blocking the app altogether


How to install

Hi Sean,
tried to put slim_fast to work.
Created a blank Phoenix app.
Added package as dependency in mix.exs like all others.
Ran mix deps.get.
Then I tried a few things, ranging from simply changing my template’s suffix to .slim to adding SlimFast.evaluate(slim, site_title: "Website Title") to different files,  none of which worked :)
Can you give us some pointers on how to put it to work?
Once I understand it, I’ll be happy to add it to the README if that’s helpful.
Thanks

Hey @gusaiani!
SlimFast's first issue, hooray :tada:
I have not yet looked at Phoenix integration but I will do that this week and get back to you.  My goal has to get the parser as complete as possible, then focus on making the package easier to use with Phoenix.
Let's leave this open and I will update it soon.

I found this package for supporting HAML in Phoenix: https://github.com/chrismccord/phoenix_haml.  I'm going to read a bit more and figure out whether a separate package is necessary or not.

Okay @gusaiani, I believe I've got what you want.  When you have a chance could you take a peek and see if things work for you?  The Phoenix integration is in a new package here: phoenix_slim.
Please keep in mind there may still be issues with the parser.
Thanks!

Yup, it did work in a tiny app/template and I thank you very very much for that.
It seems that live reload is not happening at the moment.
Even though I might even trade live reload for phoenix_slim, I’ll test that further ;)
In fact, I'll begin porting my app and letting you know if I see crazy stuff (with fixes if I’m able).
Again: many thanks!

I'll ask around this week and see if I can't make progress on live reloading.  Thanks again for your interest!

@gusaiani I'm going to close this.  If you'd like to update the README with a note about Phoenix integration via phoenix_slim, I'd be glad to merge.

Awesome. Thanks!


Performance optimizations

inline some hot function
add 'ecma_copy_value_if_not_object' similer to 'ecma_value_free_if_not_object'
remove unnecessary helpers
improve 'do_number_bitwise_logic'

JerryScript-DCO-1.0-Signed-off-by: László Langó<EMAIL_ADDRESS>

Binaries
master: 182,684
branch: 183,088
Performance results on x86_64:
                                     Benchmark |                   Perf(+ is better)
                                    3d-cube.js |   0.245s ->   0.245s :  -0.136%  (+-0.694%) : [≈]
                        access-binary-trees.js |   0.120s ->   0.119s :  +0.973%  (+-5.341%) : [≈]
                            access-fannkuch.js |   0.652s ->   0.632s :  +3.069%  (+-2.168%) : [+]
                               access-nbody.js |   0.274s ->   0.269s :  +2.065%  (+-0.434%) : [+]
                   bitops-3bit-bits-in-byte.js |   0.185s ->   0.171s :  +7.996%  (+-1.451%) : [+]
                        bitops-bits-in-byte.js |   0.235s ->   0.214s :  +8.865%  (+-0.876%) : [+]
                         bitops-bitwise-and.js |   0.255s ->   0.249s :  +2.353%  (+-0.812%) : [+]
                         bitops-nsieve-bits.js |   2.870s ->   1.656s : +42.290%  (+-0.670%) : [+]
                      controlflow-recursive.js |   0.085s ->   0.084s :  +0.783%  (+-1.790%) : [≈]
                                 crypto-aes.js |   0.336s ->   0.314s :  +6.352%  (+-0.438%) : [+]
                                 crypto-md5.js |   0.443s ->   0.329s : +25.837%  (+-1.905%) : [+]
                                crypto-sha1.js |   0.252s ->   0.201s : +20.211%  (+-1.472%) : [+]
                          date-format-tofte.js |   0.213s ->   0.208s :  +2.349%  (+-2.498%) : [≈]
                          date-format-xparb.js |   0.074s ->   0.073s :  +0.679%  (+-2.035%) : [≈]
                                math-cordic.js |   0.272s ->   0.254s :  +6.614%  (+-1.835%) : [+]
                          math-partial-sums.js |   0.142s ->   0.144s :  -1.172%  (+-1.440%) : [≈]
                         math-spectral-norm.js |   0.142s ->   0.136s :  +4.000%  (+-4.007%) : [≈]
                              string-base64.js |   0.520s ->   0.506s :  +2.819%  (+-1.552%) : [+]
                               string-fasta.js |   0.273s ->   0.275s :  -0.732%  (+-0.441%) : [-]
                               Geometric mean: |                   Speed up: 7.872% (+-0.498%) : [+]

Performance results on RPi2:
                                     Benchmark |          RSS(+ is better) |                   Perf(+ is better)
                                    3d-cube.js |   76k ->   76k :  +0.000% |   2.547s ->   2.538s :  +0.393%  (+-0.809%) : [≈]
                        access-binary-trees.js |   76k ->   76k :  +0.000% |   1.355s ->   1.350s :  +0.369%  (+-1.240%) : [≈]
                            access-fannkuch.js |   32k ->   32k :  +0.000% |   7.490s ->   7.370s :  +1.602%  (+-1.351%) : [+]
                               access-nbody.js |   36k ->   36k :  +0.000% |   2.885s ->   2.810s :  +2.600%  (+-1.728%) : [+]
                   bitops-3bit-bits-in-byte.js |   24k ->   24k :  +0.000% |   1.705s ->   1.617s :  +5.132%  (+-1.270%) : [+]
                        bitops-bits-in-byte.js |   24k ->   24k :  +0.000% |   2.465s ->   2.388s :  +3.144%  (+-1.314%) : [+]
                         bitops-bitwise-and.js |   24k ->   24k :  +0.000% |   3.208s ->   3.265s :  -1.793%  (+-0.701%) : [-]
                         bitops-nsieve-bits.js |  148k ->  148k :  +0.000% |  20.065s ->  17.855s : +11.014%  (+-1.043%) : [+]
                      controlflow-recursive.js |  136k ->  136k :  +0.000% |   0.940s ->   0.950s :  -1.064%  (+-0.000%) : [-]
                                 crypto-aes.js |  104k ->  104k :  +0.000% |   3.440s ->   3.350s :  +2.616%  (+-1.368%) : [+]
                                 crypto-md5.js |  152k ->  152k :  +0.000% |   3.560s ->   3.295s :  +7.444%  (+-1.508%) : [+]
                                crypto-sha1.js |  112k ->  112k :  +0.000% |   2.208s ->   2.067s :  +6.342%  (+-2.926%) : [+]
                          date-format-tofte.js |   60k ->   60k :  +0.000% |   2.125s ->   2.087s :  +1.765%  (+-1.515%) : [+]
                          date-format-xparb.js |   52k ->   52k :  +0.000% |   0.830s ->   0.835s :  -0.602%  (+-2.031%) : [≈]
                                math-cordic.js |   32k ->   32k :  +0.000% |   2.888s ->   2.830s :  +1.991%  (+-0.496%) : [+]
                          math-partial-sums.js |   24k ->   28k : -16.667% |   1.577s ->   1.565s :  +0.792%  (+-1.409%) : [≈]
                         math-spectral-norm.js |   32k ->   32k :  +0.000% |   1.458s ->   1.455s :  +0.172%  (+-1.529%) : [≈]
                              string-base64.js |  152k ->  152k :  +0.000% |   5.070s ->   5.075s :  -0.099%  (+-0.744%) : [≈]
                               string-fasta.js |   40k ->   44k : -10.000% |   3.230s ->   3.208s :  +0.697%  (+-1.131%) : [≈]
                               Geometric mean: |    RSS reduction: -1.322% |                   Speed up: 2.289% (+-0.333%) : [+]

I've also been experimenting with getting rid of the second argument of ecma_value_copy but never had the time to get to the end of it. It's great to see this happening.
Some questions though:

Do you have code size results for ARM as well? (Or are the 'Binaries' already giving arm/linux code size?)
The PR seems to have some unrelated changes: ecma number casts, ecm_value_copy improvements, etc. Do all of them bring improvements on their own, or only together? What's the reason of boundling them together?

Do you have code size results for ARM as well? (Or are the 'Binaries' already giving arm/linux code size?)

Those are ARM binary sizes, measured on RPi2 with GCC 4.8

The PR seems to have some unrelated changes: ecma number casts, ecm_value_copy improvements, etc. Do all of them bring improvements on their own, or only together? What's the reason of boundling them together?

I tried various combinations of the them and every improvement had some small side effect, but in one patch they work quite well on every sunspider test.

LGTM (FWIW)
But please check all copyright headers (at least one file has old date: ecma-helpers-values-collection.c)

LGTM


revert changes in the signature of 'convertCompilerOptionsFromJson'

fixes breaking change in API reported here.
We need to enable tests that will verify that there are no unexpected changed in the public surface of the API. I'll do this in a separate PR;

Can you add a new APISample test for these ones. something like https://github.com/Microsoft/TypeScript/blob/master/tests/cases/compiler/APISample_compile.ts to make sure we do not break it again.

:+1:


create settings class in core library that allows to configure static values at runtime

Closing this from here and let's move to Core side, if needed. Good idea though.


updated the blog/views.py

#93

Info about Issue or bug

Fixes: #93

view profile error showing is resolved

#93

Resolves the issue #93

@Kajol-Kumari  and @harsh-9in  resolved the multiple commit isuue and opened a new PR  for the issue #93


Error:  "The given key 'cookie' was not present in the dictionary."

[ERROR] No cookie in header!
[ERROR] System.Collections.Generic.KeyNotFoundException: The given key 'cookie' was not present in the dictionary.
   at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
   at PayCheckServerLib.TokenHelper.ReadFromHeader(Dictionary`2 kv) in D:\a\PayCheck3\PayCheck3\PayCheckServerLib\Helpers\TokenHelper.cs:line 146

Followed the steps provided, yet still failing at nebula game config data fetch.
Installed certificates to the correct locations, as well as pasting them into the server folder, as well as editing the "hosts" file.

The "No cookie in header" comes from the time things. For me works ws expected.
Try deleting your save file and run again,dont know why is that happening for me too

Fixed in latest gs-mm Branch


Fun project

Fun project but always getting this just so you know :)
body: list[BodyFragment]
TypeError: 'type' object is not subscriptable

Hi, first of all: thank you for taking the time to report this issue.
I wrote this project in python 3.10, which allows me to reference the type of the items inside a list.
It seems like this feature isn't supported in the version you are using.
I'll update the project to work in older versions in the next few days.
Thank you for your help!


fixes for godot 4.2.1

I actually created a fork to adapt to 4.2.x new plugin structure

Merged! Thanks!


[zh] Localize a component image into Chinese

Localize a SVG by following the body text.
content/zh-cn/docs/concepts/architecture/cloud-controller.md

Current online page | Preview

looks like we need a localized image directory for the pr approval process.

looks like we need a localized image directory for the pr approval process.

Yes, create a new folder:
zh-cn/docs/architecture/images/components-of-kubernetes.svg

Not sure whether the image can be put into somewhere static/images/ while still be localized. Have you checked if Hugo support this image localization?

Verified, this works:
static/images/docs/components-of-kubernetes-zh.svg

![Kubernetes 组件](/images/docs/components-of-kubernetes-zh.svg)

This doesn't work:
zh-cn/docs/concepts/architecture/images/components-of-kubernetes.svg

![Kubernetes 组件](images/components-of-kubernetes.svg)

/lgtm
/approve


[BEAM-5357] Disable optimization for Go check IsWorkerCompatibleBinary

This disabled the IsWorkerCompatibleBinary optimization since its failing on some systems.
@herohde @lostluck
Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---
---
---
---
---

Java

Python

---

---
---
---
---

Thanks!

This change seems to be causing some issues - I cannot run the pipeline with a binary, it always resorts to cross compiling and looking for the path to the source on my local environment where the binary was built and thus fails


Condition "doesn't begin with" evaluates incorrectly

Bug in the expression evaluation function.

Fixed in version B.2.8.0


Allow for multiple sequences for Compose

What steps will reproduce the problem?
1. Initiate Compose ("°")
2. Type any vowel of either capital or lower-case
3. Type any accent symbol (`,'"^*)
4. Note that no exotic character is printed (nor is it present without being
visible)

Expected:
That all sensical sequences for exotic characters in Compose would be supported.

Actual:
That only one (perhaps more in certain circumstances) sequence exists per
character, even if there are other combinations which would still make sense.

App Version: 1.29.1288 2012-01-13

Applicable to all layouts which include Compose.

Additional Info:
I'm passionately in love with this keyboard. But some of the sequences which
are left out by the built-in Compose are the ones which feel more natural to me.

All the best,

-HG

Original issue reported on code.google.com by<EMAIL_ADDRESS>on 20 Jan 2012 at 4:13

N.B.: https://github.com/klausw/hackerskeyboard/blob/master/java/src/org/pocketworkstation/pckeyboard/ComposeSequence.java#L249 is where these sequences start.…


Add link to the Android sample.

We got a question https://github.com/GoogleCloudPlatform/java-docs-samples/issues/469 about using this sample code in an Android application. We should mention the Android samples whereever we have them. (I think this will become even more important when we add Google Cloud client library samples since I don't think Android is supported for those libraries)

Current coverage is 49.44% (diff: 100%)

Merging #471 into master will not change coverage

@@             master       #471   diff @@
==========================================
  Files            82         82
  Lines          2680       2680
  Methods           0          0
  Messages          0          0
  Branches        198        198
==========================================
  Hits           1325       1325
  Misses         1312       1312
  Partials         43         43

Powered by Codecov. Last update e0c8493...fb625d0


Add System.Containers.Tasks and CreateNewImage

Leaving this up for early reviewing.

Closing to replace with https://github.com/rainersigwald/containers/pull/41


Code used to benchmark

Thanks for this really useful library!
I'm trying to find the code you have used to compare ClipKIT with other methods (the graphs shown here).
if you have written any wrappers around trimAI, Gblocks (whose source code I cant seem to find) and BMGE, could you please share them too?

Hi @sen-pai,
Thank you for writing and for using ClipKIT!
With respect to the desirability-based framework for ranking, I used the general framework detailed, including R code, here: integRATE from Eidem et al. (2018) BMC Medical Genomics.
For trimAl, I used:
trimal -in <inputfile> -out <outputfile> -gappyout;
trimal -in <inputfile> -out <outputfile> -strict; and
trimal -in <inputfile> -out <outputfile> -strictplus.
For GBlocks, I used default parameters. I, too, am having trouble finding the source code since the original domain seems to be stalling. However, I found this link to a web portal version http://phylogeny.lirmm.fr/phylo_cgi/one_task.cgi?task_type=gblocks.
I hope this helps!


Server returns blank page with 404 status code

Describe the Bug
When attempting to access the application through my web browser, I only receive a blank page with an HTTP status of 404. I can confirm that this isn't a network configuration issue, as it actually establishes an HTTP connection, and I can also access all of the APIs without issue. I am not utilizing any reverse proxy, I am connecting directly to the configured port.
Reproduction Steps

Create a new server with minimal mods installed.
Start the server.
Connect to the locally created web server.
The application doesn't load and just shows a blank page with a 404 status code in the dev console.

Expected Result
The application should load.
Screenshots and Videos
HTTP request information from Firefox devtools:

Crash Report or Log
https://gist.github.com/Silveere/d6cc6e29f188c7c70985aa752c46fcab
Operating System
Arch Linux
CTM Version
1.4
Create Version
0.5.1d
Minecraft Version
1.20.1
Mod Loader
Quilt
Modloader Version
0.21.0
Instance Type
Server hosted on the same LAN
Other Mods
create-fabric-0.5.1-d-build.1161+mc1.20.1.jar
create-track-map-1.4+mc1.20.1-fabric.jar
qfapi-7.4.0_qsl-6.1.2_fapi-0.90.0_mc-1.20.1.jar
quilt-kotlin-libraries-2.1.3+kt.1.9.10+flk.1.10.10.jar

Additional Context
I was first experiencing this issue in a fairly large custom modpack. I uninstalled all mods besides those necessary to load the server, and I still have the issue.

I just tried hosting the static content (everything in src/main/resources/assets/littlechasiu/ctm/static) on my web server and only reverse proxying /api/* and everything seems to work fine like that. It just seems to be some issue with serving the static content.

Seeing this issue on 1.19.2 with CTM Fabric 1.4 (Create 0.5.1) on Quilt too

This issue appears to be to do with quilt. Running the exact same modpack on fabric and quilt and both can load apis, but only fabric can show the map


Access to pre-build model

Hi,
May I ask where I can access the pre-build model, the smiles.npy and the arguments.txt file?
model = darkchem.utils.load_model(‘/path/to/model_folder/’) # arguments.txt must be present in this folder, as well as respective network weights

# load data
x = np.load(‘/path/to/smiles.npy’)

# generate latent space
x_latent = model.encoder.predict(x)

# generate property predictions
y_pred = model.predictor.predict(x_latent)

# predict SMILES outputs
x_pred = model.decoder.predict(x_latent)

Thanks!

To discuss training/test data sharing, please contact us via email.


1 out of N runs starts successfully, others fail immediately

Hi,
I am trying to reproduce BERT pretraining example by following the steps here: https://github.com/mosaicml/examples/tree/main/examples/bert without any custom modifications.
I am trying to run it on 4xA100 80GB, inside a conda environment. A really weird and annoying problem I am facing now is that only 1 lucky run out of N attempts to start succeeds to start with training. All other (N-1) attempts fail with this error:
Training using config:
data_local: /ssdpool/eldar/bert_c4
data_remote: null
max_seq_len: 128
tokenizer_name: bert-base-uncased
mlm_probability: 0.15
run_name: hf-bert-base-uncased
model:
  name: hf_bert
  use_pretrained: false
  pretrained_model_name: ${tokenizer_name}
  tokenizer_name: ${tokenizer_name}
  model_config:
    num_attention_heads: 12
    num_hidden_layers: 12
    max_position_embedding: 512
    attention_probs_dropout_prob: 0.1
train_loader:
  name: text
  dataset:
    local: ${data_local}
    remote: ${data_remote}
    split: train
    tokenizer_name: ${tokenizer_name}
    max_seq_len: ${max_seq_len}
    shuffle: true
    mlm_probability: ${mlm_probability}
  drop_last: true
  num_workers: 8
eval_loader:
  name: text
  dataset:
    local: ${data_local}
    remote: ${data_remote}
    split: val
    tokenizer_name: ${tokenizer_name}
    max_seq_len: ${max_seq_len}
    shuffle: false
    mlm_probability: 0.15
  drop_last: false
  num_workers: 8
scheduler:
  name: linear_decay_with_warmup
  t_warmup: 0.06dur
  alpha_f: 0.02
optimizer:
  name: decoupled_adamw
  lr: 0.0005
  betas:
  - 0.9
  - 0.98
  eps: 1.0e-06
  weight_decay: 1.0e-05
max_duration: 286720000sp
eval_interval: 2000ba
global_train_batch_size: 4096
seed: 17
device_eval_batch_size: 128
device_train_microbatch_size: 128
precision: bf16
progress_bar: false
log_to_console: true
console_log_interval: 1ba
callbacks:
  speed_monitor:
    window_size: 500
  lr_monitor: {}

Initializing model...
n_params=1.0951e+08
Building train loader...
Building eval loader...

ERROR:composer.cli.launcher:Rank 1 crashed with exit code 1.
Waiting up to 30 seconds for all training processes to terminate. Press Ctrl-C to exit immediately.
/home/ekurtic/miniconda3/envs/mosaicml/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 6 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/home/ekurtic/miniconda3/envs/mosaicml/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/eb22f1_next_epoch': [Errno 2] No such file or directory: '/eb22f1_next_epoch'
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/home/ekurtic/miniconda3/envs/mosaicml/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/d016e8_next_epoch': [Errno 2] No such file or directory: '/d016e8_next_epoch'
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/home/ekurtic/miniconda3/envs/mosaicml/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/eb22f1_barrier': [Errno 2] No such file or directory: '/eb22f1_barrier'
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/home/ekurtic/miniconda3/envs/mosaicml/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/d016e8_barrier': [Errno 2] No such file or directory: '/d016e8_barrier'
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/home/ekurtic/miniconda3/envs/mosaicml/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/d016e8_shard_states': [Errno 2] No such file or directory: '/d016e8_shard_states'
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/home/ekurtic/miniconda3/envs/mosaicml/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/eb22f1_shard_states': [Errno 2] No such file or directory: '/eb22f1_shard_states'
  warnings.warn('resource_tracker: %r: %s' % (name, e))
Global rank 1 (PID 4033709) exited with code 1
----------Begin global rank 1 STDOUT----------
Training using config:
data_local: /ssdpool/eldar/bert_c4
data_remote: null
max_seq_len: 128
tokenizer_name: bert-base-uncased
mlm_probability: 0.15
run_name: hf-bert-base-uncased
model:
  name: hf_bert
  use_pretrained: false
  pretrained_model_name: ${tokenizer_name}
  tokenizer_name: ${tokenizer_name}
  model_config:
    num_attention_heads: 12
    num_hidden_layers: 12
    max_position_embedding: 512
    attention_probs_dropout_prob: 0.1
train_loader:
  name: text
  dataset:
    local: ${data_local}
    remote: ${data_remote}
    split: train
    tokenizer_name: ${tokenizer_name}
    max_seq_len: ${max_seq_len}
    shuffle: true
    mlm_probability: ${mlm_probability}
  drop_last: true
  num_workers: 8
eval_loader:
  name: text
  dataset:
    local: ${data_local}
    remote: ${data_remote}
    split: val
    tokenizer_name: ${tokenizer_name}
    max_seq_len: ${max_seq_len}
    shuffle: false
    mlm_probability: 0.15
  drop_last: false
  num_workers: 8
scheduler:
  name: linear_decay_with_warmup
  t_warmup: 0.06dur
  alpha_f: 0.02
optimizer:
  name: decoupled_adamw
  lr: 0.0005
  betas:
  - 0.9
  - 0.98
  eps: 1.0e-06
  weight_decay: 1.0e-05
max_duration: 286720000sp
eval_interval: 2000ba
global_train_batch_size: 4096
seed: 17
device_eval_batch_size: 128
device_train_microbatch_size: 512
precision: bf16
progress_bar: false
log_to_console: true
console_log_interval: 1ba
callbacks:
  speed_monitor:
    window_size: 500
  lr_monitor: {}

Initializing model...
n_params=1.0951e+08
Building train loader...
Building eval loader...

----------End global rank 1 STDOUT----------
----------Begin global rank 1 STDERR----------
Traceback (most recent call last):
  File "/home/ekurtic/github/eldarkurtic/mosaicml/examples/examples/bert/main.py", line 141, in <module>
    main(cfg)
  File "/home/ekurtic/github/eldarkurtic/mosaicml/examples/examples/bert/main.py", line 91, in main
    trainer = Trainer(
  File "/home/ekurtic/miniconda3/envs/mosaicml/lib/python3.9/site-packages/composer/trainer/trainer.py", line 975, in __init__
    dist.initialize_dist(device, dist_timeout)
  File "/home/ekurtic/miniconda3/envs/mosaicml/lib/python3.9/site-packages/composer/utils/dist.py", line 426, in initialize_dist
    dist.init_process_group(device_obj.dist_backend, timeout=timeout_timedelta)
  File "/home/ekurtic/miniconda3/envs/mosaicml/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 761, in init_process_group
    default_pg = _new_process_group_helper(
  File "/home/ekurtic/miniconda3/envs/mosaicml/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 909, in _new_process_group_helper
    pg = _create_process_group_wrapper(
  File "/home/ekurtic/miniconda3/envs/mosaicml/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 3232, in _create_process_group_wrapper
    helper_pg = ProcessGroupGloo(store, rank, world_size, timeout=timeout)
RuntimeError: [../third_party/gloo/gloo/transport/tcp/pair.cc:799] connect [<IP_ADDRESS>]:1927: Connection refused
Exception ignored in: <function StreamingDataset.__del__ at 0x1482b4fc11f0>
Traceback (most recent call last):
  File "/home/ekurtic/miniconda3/envs/mosaicml/lib/python3.9/site-packages/streaming/base/dataset.py", line 851, in __del__
    wait_for_local_leader(world)
  File "/home/ekurtic/miniconda3/envs/mosaicml/lib/python3.9/site-packages/streaming/base/util.py", line 62, in wait_for_local_leader
    wait_for_file_to_exist(dir_path,
  File "/home/ekurtic/miniconda3/envs/mosaicml/lib/python3.9/site-packages/streaming/base/util.py", line 49, in wait_for_file_to_exist
    raise RuntimeError(f'{err_msg}, bailing out: ' + f'{timeout:.3f} < {dt:.3f} sec.')
RuntimeError: Waiting for local rank 0, bailing out: 60.000 < 60.033 sec.
Exception ignored in: <function StreamingDataset.__del__ at 0x1482b4fc11f0>
Traceback (most recent call last):
  File "/home/ekurtic/miniconda3/envs/mosaicml/lib/python3.9/site-packages/streaming/base/dataset.py", line 851, in __del__
    wait_for_local_leader(world)
  File "/home/ekurtic/miniconda3/envs/mosaicml/lib/python3.9/site-packages/streaming/base/util.py", line 62, in wait_for_local_leader
    wait_for_file_to_exist(dir_path,
  File "/home/ekurtic/miniconda3/envs/mosaicml/lib/python3.9/site-packages/streaming/base/util.py", line 49, in wait_for_file_to_exist
    raise RuntimeError(f'{err_msg}, bailing out: ' + f'{timeout:.3f} < {dt:.3f} sec.')
RuntimeError: Waiting for local rank 0, bailing out: 60.000 < 60.034 sec.
/home/ekurtic/miniconda3/envs/mosaicml/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 6 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

----------End global rank 1 STDERR----------
ERROR:composer.cli.launcher:Global rank 0 (PID 4033708) exited with code -15

And the bash script I am using to run this:
export HF_DATASETS_CACHE=/ssdpool/eldar/hf_cache
export CUDA_VISIBLE_DEVICES=4,5,6,7
export MASTER_PORT=12345
export WORLD_SIZE=4
export TORCH_DISTRIBUTED_DEBUG=DETAIL

composer main.py yamls/main/hf-bert-base-uncased.yaml

What am I doing wrong here, any ideas? Why is GLOO complaining in error messages above, shouldn't we use NCCL as a backend for DDP runs?

First thing to check is probably that your GPU environment is working. If device is not provided, Trainer will automatically select gpu or cpu based on torch.cuda.is_available() (https://github.com/mosaicml/composer/blob/12b94d94696180dd1df2ba536fc2dab588b17786/composer/utils/device.py#L30-L31). The other thing I notice is that your rank 0 and rank 1 logs are printing a different value for device_train_microbatch_size, which is pretty weird, since the config you reference has that set to 128 (https://github.com/mosaicml/examples/blob/dd67dee70fc5cd1b69167a15a9ca819dd7cce28b/examples/bert/yamls/main/hf-bert-base-uncased.yaml#L80).

@dakinggg do you have any suggestions how to test if my "GPU environment is working"?
Btw, what is the correct way to run DDP experiments with composer library: composer main.py or composer -n 8 main.py?

you can just run python -c "import torch; print(torch.cuda.is_available(), torch.cuda.device_count()), which should print out True, N_GPUS if everything is set up correctly.
Running composer main.py will use all available gpus (gets defaults from LOCAL_WORLD_SIZE or torch.cuda.device_count() see https://github.com/mosaicml/composer/blob/20fb86b476d91749528a989265c3e211eb1d774d/composer/cli/launcher.py#L140-L151). whereas composer -n 8 main.py will explicitly launch 8 processes. Either should be fine.
You can also try adding device: gpu to the yaml you are running to explicitly tell composer to use gpu. That should error if something is set up incorrectly.

I finally got some time to debug, and it turns out the process just hangs at this specific line: dist.init_process_group(device_obj.dist_backend, timeout=timeout_timedelta) (https://github.com/mosaicml/composer/blob/38cec0937640ccee148ca4f6611f94e40e35ad22/composer/utils/dist.py#L426), and breaks with: RuntimeError: Waiting for local rank 0, bailing out: 60.000 < 60.033 sec.. I have played with different values for timeout_timedelta but no difference. A very interesting thing is that the only way that I am able to run the experiment is like this:
while true
do
    composer -n 8 main.py yamls/main/hf-bert-base-uncased.yaml
done

and after a number of attempts, it would somehow magically succeed  with distributed initialization...
The server and environment where I am running this seem to be fine, I have tried to run DDP experiments with two different repositories (FFCV and HF-transformers) and both of them work just fine, dist.init_process_group(...) never hangs. However, I have noticed one difference between mosaicml and the other two repos that run successfully: in the "main" python script, they both spawn distributed training with torch.multiprocessing.spawn(...), and I am not able to find any instances of similar command when searching through the mosaicml codebase. Am I missing something here?

Could you share the output of pip list? And also share the full trace and the full script you are running? The RuntimeError: Waiting for local rank 0, bailing out: 60.000 < 60.033 sec. error is just coming when our streaming dataset tries to clean up, and is not the cause of the issue.

Could you also pass python_log_level='debug' to trainer? It will print out the distributed environment that it is trying to initialize.

pip list output:
Package                Version           Editable project location
---------------------- ----------------- ---------------------------------------------------------------------
aiohttp                3.8.4
aiosignal              1.3.1
antlr4-python3-runtime 4.9.3
async-timeout          4.0.2
attrs                  22.2.0
bcrypt                 4.0.1
boto3                  1.26.113
botocore               1.29.113
Brotli                 1.0.9
brotlipy               0.7.0
certifi                2022.12.7
cffi                   1.15.1
cfgv                   3.3.1
chardet                5.1.0
charset-normalizer     2.0.4
circuitbreaker         1.4.0
click                  8.1.3
cmake                  3.26.3
contourpy              1.0.7
coolname               2.2.0
coverage               7.2.3
cryptography           39.0.2
cycler                 0.11.0
datasets               2.10.1
dill                   0.3.6
distlib                0.3.6
docker-pycreds         0.4.0
einops                 0.5.0
exceptiongroup         1.1.1
filelock               3.11.0
flit_core              3.8.0
fonttools              4.39.3
frozenlist             1.3.3
fsspec                 2023.4.0
gitdb                  4.0.10
GitPython              3.1.31
huggingface-hub        0.13.4
identify               2.5.22
idna                   3.4
importlib-metadata     6.3.0
iniconfig              2.0.0
jmespath               1.0.1
kiwisolver             1.4.4
matplotlib             3.7.1
mkl-fft                1.3.1
mkl-random             1.2.2
mkl-service            2.4.0
mosaicml               0.13.2
mosaicml-examples      0.0.3
mosaicml-streaming     0.3.0
multidict              6.0.4
multiprocess           0.70.14
nodeenv                1.7.0
numpy                  1.23.5
oci                    2.98.0
omegaconf              2.2.3
packaging              22.0
pandas                 2.0.0
paramiko               3.1.0
pathtools              0.1.2
Pillow                 9.5.0
pip                    23.0.1
platformdirs           3.2.0
pluggy                 1.0.0
pre-commit             2.21.0
promise                2.3
protobuf               4.22.3
psutil                 5.9.4
py-cpuinfo             9.0.0
pyarrow                11.0.0
pycparser              2.21
PyNaCl                 1.5.0
pynvml                 11.5.0
pyOpenSSL              23.1.1
pyparsing              3.0.9
pyright                1.1.296
PySocks                1.7.1
pytest                 7.3.0
pytest_codeblocks      0.16.1
pytest-cov             4.0.0
python-dateutil        2.8.2
python-snappy          0.6.1
pytorch-ranger         0.1.1
pytz                   2023.3
PyYAML                 6.0
regex                  2023.3.23
requests               2.28.1
responses              0.18.0
s3transfer             0.6.0
sentry-sdk             1.19.1
setproctitle           1.3.2
setuptools             65.6.3
shortuuid              1.0.11
six                    1.16.0
slack-sdk              3.21.1
smmap                  5.0.0
tabulate               0.9.0
tokenizers             0.13.3
toml                   0.10.2
tomli                  2.0.1
torch                  1.12.1
torch-optimizer        0.3.0
torchaudio             0.12.1
torchmetrics           0.11.3
torchvision            0.13.1
tqdm                   4.65.0
transformers           4.25.1
triton                 2.0.0.dev20221103
typing_extensions      4.4.0
tzdata                 2023.3
urllib3                1.26.15
virtualenv             20.21.0
wandb                  0.13.6
wheel                  0.38.4
xxhash                 3.2.0
yarl                   1.8.2
zipp                   3.15.0
zstd                   <IP_ADDRESS>

I am literally trying to reproduce training of the HF-BERT from scratch by following the steps here: https://github.com/mosaicml/examples/tree/main/examples/bert. No changes to the code, just running composer -n main.py yamls/main/hf-bert-base-uncased.yaml.
The full trace is here with python_log_level='debug':
Training using config:
data_local: ./mosaicml_c4_bert
data_remote: null
max_seq_len: 128
tokenizer_name: bert-base-uncased
mlm_probability: 0.15
run_name: debug
model:
  name: hf_bert
  use_pretrained: false
  pretrained_model_name: ${tokenizer_name}
  tokenizer_name: ${tokenizer_name}
  model_config:
    num_attention_heads: 12
    num_hidden_layers: 12
    max_position_embedding: 512
    attention_probs_dropout_prob: 0.1
train_loader:
  name: text
  dataset:
    local: ${data_local}
    remote: ${data_remote}
    split: train
    tokenizer_name: ${tokenizer_name}
    max_seq_len: ${max_seq_len}
    shuffle: true
    mlm_probability: ${mlm_probability}
  drop_last: true
  num_workers: 8
eval_loader:
  name: text
  dataset:
    local: ${data_local}
    remote: ${data_remote}
    split: val
    tokenizer_name: ${tokenizer_name}
    max_seq_len: ${max_seq_len}
    shuffle: false
    mlm_probability: 0.15
  drop_last: false
  num_workers: 8
scheduler:
  name: linear_decay_with_warmup
  t_warmup: 0.06dur
  alpha_f: 0.02
optimizer:
  name: decoupled_adamw
  lr: 0.0005
  betas:
  - 0.9
  - 0.98
  eps: 1.0e-06
  weight_decay: 1.0e-05
max_duration: 1ep
eval_interval: 2000ba
global_train_batch_size: 4096
seed: 17
device_eval_batch_size: 128
device_train_microbatch_size: 512
precision: bf16
device: gpu
progress_bar: true
log_to_console: true
console_log_interval: 1ba
callbacks:
  speed_monitor:
    window_size: 500
  lr_monitor: {}
loggers:
  wandb:
    project: mosaicml_bert_pretraining
    entity: ek

Initializing model...
n_params=1.0951e+08
Building train loader...
Building eval loader...
2023-04-24 16:03:12,379: rank0[1003409][MainThread]: DEBUG: composer.utils.dist: Initializing torch.dist: global_rank=0, local_rank=0, world_size=8, local_world_size=8, node_rank=0
ERROR:composer.cli.launcher:Rank 1 crashed with exit code 1.
Waiting up to 30 seconds for all training processes to terminate. Press Ctrl-C to exit immediately.
/home/ek/miniconda3/envs/mosaicml/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 6 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/home/ek/miniconda3/envs/mosaicml/lib/python3.10/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/af0626_barrier': [Errno 2] No such file or directory: '/af0626_barrier'
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/home/ek/miniconda3/envs/mosaicml/lib/python3.10/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/e9c924_next_epoch': [Errno 2] No such file or directory: '/e9c924_next_epoch'
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/home/ek/miniconda3/envs/mosaicml/lib/python3.10/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/e9c924_shard_states': [Errno 2] No such file or directory: '/e9c924_shard_states'
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/home/ek/miniconda3/envs/mosaicml/lib/python3.10/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/af0626_shard_states': [Errno 2] No such file or directory: '/af0626_shard_states'
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/home/ek/miniconda3/envs/mosaicml/lib/python3.10/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/af0626_next_epoch': [Errno 2] No such file or directory: '/af0626_next_epoch'
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/home/ek/miniconda3/envs/mosaicml/lib/python3.10/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/e9c924_barrier': [Errno 2] No such file or directory: '/e9c924_barrier'
  warnings.warn('resource_tracker: %r: %s' % (name, e))
Global rank 1 (PID 1003410) exited with code 1
----------Begin global rank 1 STDOUT----------
Training using config:
data_local: ./mosaicml_c4_bert
data_remote: null
max_seq_len: 128
tokenizer_name: bert-base-uncased
mlm_probability: 0.15
run_name: debug
model:
  name: hf_bert
  use_pretrained: false
  pretrained_model_name: ${tokenizer_name}
  tokenizer_name: ${tokenizer_name}
  model_config:
    num_attention_heads: 12
    num_hidden_layers: 12
    max_position_embedding: 512
    attention_probs_dropout_prob: 0.1
train_loader:
  name: text
  dataset:
    local: ${data_local}
    remote: ${data_remote}
    split: train
    tokenizer_name: ${tokenizer_name}
    max_seq_len: ${max_seq_len}
    shuffle: true
    mlm_probability: ${mlm_probability}
  drop_last: true
  num_workers: 8
eval_loader:
  name: text
  dataset:
    local: ${data_local}
    remote: ${data_remote}
    split: val
    tokenizer_name: ${tokenizer_name}
    max_seq_len: ${max_seq_len}
    shuffle: false
    mlm_probability: 0.15
  drop_last: false
  num_workers: 8
scheduler:
  name: linear_decay_with_warmup
  t_warmup: 0.06dur
  alpha_f: 0.02
optimizer:
  name: decoupled_adamw
  lr: 0.0005
  betas:
  - 0.9
  - 0.98
  eps: 1.0e-06
  weight_decay: 1.0e-05
max_duration: 1ep
eval_interval: 2000ba
global_train_batch_size: 4096
seed: 17
device_eval_batch_size: 128
device_train_microbatch_size: 512
precision: bf16
device: gpu
progress_bar: true
log_to_console: true
console_log_interval: 1ba
callbacks:
  speed_monitor:
    window_size: 500
  lr_monitor: {}
loggers:
  wandb:
    project: mosaicml_bert_pretraining
    entity: ek

Initializing model...
n_params=1.0951e+08
Building train loader...
Building eval loader...

----------End global rank 1 STDOUT----------
----------Begin global rank 1 STDERR----------
2023-04-24 16:03:12,065: rank1[1003410][MainThread]: DEBUG: composer.utils.dist: Initializing torch.dist: global_rank=1, local_rank=1, world_size=8, local_world_size=8, node_rank=0
Traceback (most recent call last):
  File "/home/ek/github/eldarkurtic/mosaicml/examples/examples/bert/acdc_main.py", line 459, in <module>
    main(cfg)
  File "/home/ek/github/eldarkurtic/mosaicml/examples/examples/bert/acdc_main.py", line 405, in main
    trainer = Trainer(
  File "/home/ek/miniconda3/envs/mosaicml/lib/python3.10/site-packages/composer/trainer/trainer.py", line 975, in __init__
    dist.initialize_dist(device, dist_timeout)
  File "/home/ek/miniconda3/envs/mosaicml/lib/python3.10/site-packages/composer/utils/dist.py", line 427, in initialize_dist
    dist.init_process_group(device_obj.dist_backend, timeout=timeout_timedelta)
  File "/home/ek/miniconda3/envs/mosaicml/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 627, in init_process_group
    _store_based_barrier(rank, store, timeout)
  File "/home/ek/miniconda3/envs/mosaicml/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 242, in _store_based_barrier
    worker_count = store.add(store_key, 0)
RuntimeError: Broken pipe
Exception ignored in: <function StreamingDataset.__del__ at 0x14c8a55131c0>
Traceback (most recent call last):
  File "/home/ek/miniconda3/envs/mosaicml/lib/python3.10/site-packages/streaming/base/dataset.py", line 851, in __del__
    wait_for_local_leader(world)
  File "/home/ek/miniconda3/envs/mosaicml/lib/python3.10/site-packages/streaming/base/util.py", line 62, in wait_for_local_leader
    wait_for_file_to_exist(dir_path,
  File "/home/ek/miniconda3/envs/mosaicml/lib/python3.10/site-packages/streaming/base/util.py", line 49, in wait_for_file_to_exist
    raise RuntimeError(f'{err_msg}, bailing out: ' + f'{timeout:.3f} < {dt:.3f} sec.')
RuntimeError: Waiting for local rank 0, bailing out: 60.000 < 60.053 sec.
Exception ignored in: <function StreamingDataset.__del__ at 0x14c8a55131c0>
Traceback (most recent call last):
  File "/home/ek/miniconda3/envs/mosaicml/lib/python3.10/site-packages/streaming/base/dataset.py", line 851, in __del__
    wait_for_local_leader(world)
  File "/home/ek/miniconda3/envs/mosaicml/lib/python3.10/site-packages/streaming/base/util.py", line 62, in wait_for_local_leader
    wait_for_file_to_exist(dir_path,
  File "/home/ek/miniconda3/envs/mosaicml/lib/python3.10/site-packages/streaming/base/util.py", line 49, in wait_for_file_to_exist
    raise RuntimeError(f'{err_msg}, bailing out: ' + f'{timeout:.3f} < {dt:.3f} sec.')
RuntimeError: Waiting for local rank 0, bailing out: 60.000 < 60.055 sec.
/home/ek/miniconda3/envs/mosaicml/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 6 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

----------End global rank 1 STDERR----------
ERROR:composer.cli.launcher:Global rank 0 (PID 1003409) exited with code -15

Could you try running a script that just calls dist.initialize_dist(device, dist_timeout)?

Closing due to inactivity.


Enable MPI coverage with Linux and reduce heap size hint

Experimenting with different settings to avoid #1590

First run: Ubuntu works, macOS fails
Second run: Ubuntu works, macOS works

If this works robustly, it would be a great addition 👍


Front camera distorts image in landscape

When ever I take an image with the front camera and hold my phone in landscape, the image is distorted. Please see the image attached to get a better understanding.

Here is how I fixed this issue:
In YPCameraVC look for func shoot() and replace the code with this code:
`func shoot() {
// Prevent from tapping multiple times in a row
// causing a crash
v.shotButton.isEnabled = false
    photoCapture.shoot { imageData in

        guard let shotImage = UIImage(data: imageData) else {
            return
        }

        self.photoCapture.stopCamera()

        var image = shotImage
        // Crop the image if the output needs to be square.
        if YPConfig.onlySquareImagesFromCamera {
            image = self.cropImageToSquare(image)
        }

        // Flip image if taken form the front camera.
        if let device = self.photoCapture.device, device.position == .front {

//                image = self.flipImage(image: image)
            let orientation: UIDeviceOrientation = YPDeviceOrientationHelper.shared.currentDeviceOrientation

            switch orientation {
            case .landscapeLeft, .landscapeRight:
                image = image.withHorizontallyFlippedOrientation()
            case .portrait:
                let imageSize: CGSize = image.size
                UIGraphicsBeginImageContextWithOptions(imageSize, true, 1.0)
                let ctx = UIGraphicsGetCurrentContext()!
                ctx.rotate(by: CGFloat(Double.pi/2.0))
                ctx.translateBy(x: 0, y: -imageSize.width)
                ctx.scaleBy(x: imageSize.height/imageSize.width, y: imageSize.width/imageSize.height)
                ctx.draw(image.cgImage!, in: CGRect(x: 0.0,
                                                    y: 0.0,
                                                    width: imageSize.width,
                                                    height: imageSize.height))
                let newImage: UIImage = UIGraphicsGetImageFromCurrentImageContext()!
                UIGraphicsEndImageContext()
                image = newImage
            default:
                image = image.withHorizontallyFlippedOrientation()
            }

        }

        let noOrietationImage = image.resetOrientation()

        DispatchQueue.main.async {
            self.didCapturePhoto?(noOrietationImage.resizedImageIfNeeded())
        }
    }
}`

Works perfectly well on my end.


[Bug] Stacking uniforms.

In widedomain, doing mm(uniform(10, 40), uniform(20, 30)) does not do what I'd expect (stack two uniforms one on top of another), but instead just presents one uniform from 20 to 30.
Ozzie Gooen:

The main part is fine, this is a problem with the visualization

I'm quite sure the problem here is in the d3; the min is showing at the bottom of the y-axis, while really that should be set to 0.

So a part of this is that in the multimodal aggregate distribution, points are repeated.
This can be found out by adding some code to my integrateContinuousShape function (in XYShape.re), which iterates over the points.
  let integrateContinuousShape =
      (
        ~indefiniteIntegralStepwise=(p, h1) => h1 *. p,
        ~indefiniteIntegralLinear=(p, a, b) => a *. p +. b *. p ** 2.0 /. 2.0,
        t: DistTypes.continuousShape,
      )
      : float => {
    let xs = t.xyShape.xs;
    let ys = t.xyShape.ys;

    E.A.reducei(
      xs,
      0.0,
      (acc, _x, i) => {
        let areaUnderIntegral =
          switch (t.interpolation, i) {
          | (_, 0) => 0.0
          | (`Stepwise, _) =>
            indefiniteIntegralStepwise(xs[i], ys[i - 1])
            -. indefiniteIntegralStepwise(xs[i - 1], ys[i - 1])
          | (`Linear, _) =>
            let x1 = xs[i - 1];
            let x2 = xs[i];
            let h1 = ys[i - 1];
            let h2 = ys[i];
            let b = (h1 -. h2) /. (x1 -. x2);
            let a = h1 -. b *. x1;

            if (Js.Float.isNaN(a)) {
              Js.log("Error, repeated (x,y): ");
              Js.log("x1: " ++ Js.Float.toString(x1));
              Js.log("x2: " ++ Js.Float.toString(x2));
              Js.log("h1: " ++ Js.Float.toString(h1));
              Js.log("h2: " ++ Js.Float.toString(h2));
            };

            indefiniteIntegralLinear(x2, a, b)
            -. indefiniteIntegralLinear(x1, a, b);
          };
        acc +. areaUnderIntegral;
      },
    );
  };

If I do that, I get a lot of repeated points:

Note that the mean and variance don't work, because they have a division by 0 error.
The fix would not be difficult (just skip repeated points, or do a first pass and delete them). But why do we have repeated points in the first place?

I think fixing how x-points are selected for uniforms has taken care of this issue. Here is what I see:

Cheers, it seems like this has been solved now; closing the issue.


Support goto def on https://github.com/microsoft/pylance-release/issues/4660 as well

this https://github.com/microsoft/pyrx/pull/4033 enables https://github.com/microsoft/pylance-release/issues/4660
but it only works for completion. go to def doesn't work for now. we should support it since that is only one missing among editor features such as hover, checker, completion, signature help.

once we have service provider for workspace/service/program and once we move extension.tryGetDefinition to some kind of definition provider service
we can add literal in __getattr__ as one of those definition provider, so things like find all referece, highlight reference, rename and go to def just work.

This issue has been fixed in prerelease version 2023.9.11, which we've just released. You can find the changelog here: CHANGELOG.md


Article not scrollable on mobile device

Where to find the issue
https://www.coronawarn.app/en/faq/results/
Describe the issue
https://www.coronawarn.app/en/faq/results/#further_details is truncated if selected through "App Features" > "Other features" and device is in portrait orientation.
Steps to reproduce
Google Chrome on Android

Open https://www.coronawarn.app/en/faq/results/ in Google Chrome with mobile device in portrait orientation
Tap right-arrow
Tap "Other features" under "App Features"
Tap "In-App statistics starting with release 1.1"
Note that article is displayed, however only the first paragraph is visible

The issue can also be simulated in Google Chrome Desktop. Press F12 (or Ctrl+Shift+I) for developer tools, toggle the device toolbar (Ctrl+Shift+M) and select Samsung Galaxy A51/A71.  Then follow the above steps to reproduce.
Expected results
The full text of the article https://www.coronawarn.app/en/faq/results/#further_details should be displayed in portrait mode.
Workaround
Rotate the device to landscape orientation.

Internal Tracking ID: EXPOSUREAPP-12452

Internal Tracking ID: EXPOSUREAPP-12452

This issue is no longer reproducible.

I suspect that https://github.com/corona-warn-app/cwa-website/pull/2600 may have solved the issue.


Problems with blueprint registration

Hi, i creating the modular system for my project on Flask. And i got problems with blueprint registration.
Example of my code:
for url_prefix, blueprint in blueprints.items():
    self.Gateway.register_blueprint(blueprint=blueprint, url_prefix=url_prefix)

Exception from Flask:
Traceback (most recent call last):
  File "D:\repos\UniServices\main.py", line 13, in <module>
  File "D:\repos\UniServices\main.py", line 10, in main
    Server(configuration_file='../configuration.json').run()
  File "D:\repos\UniServices\Server\Server.py", line 30, in __init__
    self.setup(configuration_file)
  File "D:\repos\UniServices\Server\Server.py", line 45, in setup
    self.Gateway.register_blueprint(blueprint=blueprint, url_prefix=url_prefix)
  File "D:\repos\UniServices\venv\lib\site-packages\flask\scaffold.py", line 57, in wrapper_func
  File "D:\repos\UniServices\venv\lib\site-packages\flask\app.py", line 1028, in register_blueprint
    blueprint.register(self, options)
TypeError: Blueprint.register() missing 1 required positional argument: 'options'

I have dictionary with blueprints for automatically adding blueprints.
Code from Blueprints module
from .Blueprints.IndexBlueprint import Blueprint as IndexBlueprint

blueprints = {
    '/': IndexBlueprint,
}

Environment:

Python version: 3.10.1
Flask version: 2.0.2

I can't reproduce this issue with the information provided.
Please use our Discord for discussion about your own code. This tracker is for issues related to the project itself.


How inject Bower dependencies for Angular 2?

I can't select Bower inject if choose Angular 2
This option is only availabe to Angular 1 option

It's a chosen limitation. Angular2+ is not published on Bower and they officially that they don't want  to support it. We followed their line and doesn't support this combination.


Custom component development

Hi,
Let's say I want to add support for another RTOS, but I don't want to edit pyOCD sources directly. Is pyOCD able to load external components?
e.g. pemicro probe is standalone python package, so I guess it should be possible...

Yup! To some extent, at least. Pyocd currently supports RTOS and debug probe plugins. So it sounds like you should be ok.
(I'm converting this issue to a discussion and will continue with more details there.)


Add support to check changes in function signatures

Created a utility scripts/signature.py to export and diff function signatures.
python ./scripts/signature.py --help
Usage: signature.py [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  diff    Diffs Velox function signatures.
  export  Exports Velox function signatures.

Signature Diffs
The diff tool will error out when :

A function / or signature is removed from the incoming PR.

python ./scripts/signature.py diff /tmp/presto_signatures_new.json /tmp/orig_sigs_change.json
Signature removed: [root['test_from_base64']]

A function/ signature is changed from the incoming PR (e.g type changed etc).

   python ./scripts/signature.py diff /tmp/orig_sigs_change.json /tmp/presto_signatures_new.json
Signature changed: {"root['to_utf8'][0]": {'new_value': '(varchar) -> varbinary', 'old_value': '(int) -> varbinary'}}

A signature is repeated from incoming PR.

python ./scripts/signature.py diff /tmp/presto_signatures_new.json /tmp/presto_signatures_new_change.json
Signature repeated: {"root['to_utf8'][0]": {'old_repeat': 1, 'new_repeat': 2, 'old_indexes': [0], 'new_indexes': [0, 1], 'value': '(varchar) -> varbinary'}}

Addition of signatures and functions is detected but is not considered an error.
NOTE
Templated functions which have a concrete signature added which will override the template is not considered an error.
User workflow
This is intended as a tool and as such there might be cases where we genuinely need to remove / modify signatures etc. In these cases even though this tool will fail the build the merge engineer is supposed to use discretion and override the failure.

@kgpai @pedroerp Does this change cover aggregate and window functions as well as scalar functions?

@kgpai @pedroerp Does this change cover aggregate and window functions as well as scalar functions?

I only remember seeing a pyvelox function to extract scalar signatures. I guess aggregate and window are planned next?

@mbasmanova @pedroerp This only covers scalar for now - I will create another PR with support for aggregate/window signatures.

Could someone take a look at the changes in https://github.com/facebookincubator/velox/pull/4794. This PR seems to incorrectly report breaking changes - for example, adding new types for a function should be allowed.


Skip internal field getters when building suggestions

Pull Request Description

close #6900
This is a follow-up to the discussion of the imports/exports meeting.
Right now we have no control over the visibility of atom constructor arguments. One way to hide them is a convention of filtering getters by an internal prefix or suffix.
Important Notes

Checklist
Please ensure that the following checklist has been satisfied before submitting the PR:

[x] The documentation has been updated, if necessary.
[x] All code follows the
Scala,
Java,
and
Rust
style guides. In case you are using a language not listed above, follow the Rust style guide.
All code has been tested:

[x] Unit tests have been written where possible.

We have no visibility rules right now. I howe the discussion on imports/exports should clarify that.


pass debitor id to cncrd-adapter

The cncrd adapter will need to ask the attendee service for some information, such as the email address, when initially creating paylinks.
In order to do this safely, we need to pass down the debitor id, because we allow to manually specify the referenceId, and thus cannot trust it to point to the correct attendee.
This is the payment service part of this implementation. Implementation in the adapter is done as part of https://github.com/eurofurence/reg-payment-cncrd-adapter/issues/34

what do you know, we already had it available.


Custom reserved word for repo names?

Hi,
I've just migrate my server from Gitlab (which was a big pain to maintain & update) to Gogs, and was very happy with the installation & configuration until I want to create, in an organisation, a repo called "api" and "admin", which, apparently, are "reserved words".
Since I can use these words on Gitlab, GitHub and Bitbucket, I really don't understand why I can't use these names, common names for web-application parts.
Why can't we, as administrators of our own Gogs installation, decide which words can or cannot be used as repo name, specially for such comman and widely-used words in development?
Thanks in advance,

I believe the reason for those being reserved is because they are part of gogs api and admin paths/views -- are you asking why such paths aren't configurable?

(sorry for my english, it's not my first language, if this sound aggressive it's because a lack of vocabulary)
My point of view is that, as an user, I don't understand why I can't use certain words as repositories name (specially for words like "admin" or "api", which are widely used by developers).
Whatever the reason, I think it's wrong. For tools like these, it creates a frustration for the user, and I think it's wrong.
And, as in GitHub, Gitlab and Bitbucket it's not a problem, I really doesn't understand why the Gogs users should be "restrained".

Isn't this the same as #2903 ?

Closing as duplicate of #2903


Investigate missing sensor data in IWearRemapper datadumper file

By checking the data dumped from the IWearRemapper, I found the following behaviour:
...
-1 27976.097323 IWearRemapper ... (("XsensSuit::fbAcc::Head" ("XsensSuit::fbAcc::Head" 1 -0.016373313533672 -0.0190550281415655 0.000531472733429794)) ("XsensSuit::fbAcc::LeftFoot" ("XsensSuit::fbAcc::LeftFoot" 1 0.00442949986499089 0.00220952626739518 0.01514176584128)) ...
-1 27976.105447 IWearRemapper ... (("XsensSuit::fbAcc::LeftFoot" ("XsensSuit::fbAcc::LeftFoot" 1 0.00722438204900699 0.00521013136724258 0.0151151371458731)) ...
-1 27976.110341 IWearRemapper ... (("XsensSuit::fbAcc::LeftFoot" ("XsensSuit::fbAcc::LeftFoot" 1 0.00722438204900699 0.00521013136724258 0.0151151371458731)) ...
-1 27976.119040 IWearRemapper ... (("XsensSuit::fbAcc::Head" ("XsensSuit::fbAcc::Head" 1 0.0190951423117848 -0.00909498756059363 -0.0739544957307423)) ("XsensSuit::fbAcc::LeftFoot" ("XsensSuit::fbAcc::LeftFoot" 1 -0.0328387372429226 -0.0157998343825085 -0.00982986474280878)) ...
-1 27976.131309 IWearRemapper ... (("XsensSuit::fbAcc::Head" ("XsensSuit::fbAcc::Head" 1 0.038067409596767 -0.00213547009733639 -0.018574056519352)) ("XsensSuit::fbAcc::LeftFoot" ("XsensSuit::fbAcc::LeftFoot" 1 -0.0300501728865379 -0.0128057789517119 -0.00983834574195441))...
...

The second and third samples don't have the XsensSuit::fbAcc::Head.

Please guys add the comments that we wrote on the chat. cc @Yeshasvitvs

As @diegoferigo highlighted,
"However, if this happens only in the first few samples, a possible explanation is that after the calibration, some getSensorData (or similar method) is called in the xsens driver, but the driver still has to properly initialize itself. We might be too quick to ask data right after the calibration is finished"
In addition, we should assert that the wearable data size is equal to the  number of sensors.

In another dumped file I found another missing data, this time for the shoes:
...
-1 0.000000 1312.897323 IWearRemapper () () () ((FTShoeLeftFTSensors (FTShoeLeftFTSensors 1 1.20477376898739 17.3058125032916<PHONE_NUMBER>06986 -3.00952605367758 -25.6593735630912 5.56826094317577)) (FTShoeRightFTSensors (FTShoeRightFTSensors 1 -0.176451816950106 19.1924397005572<PHONE_NUMBER>11674 -2.8651473915682 -23.6995854163627 5.19620440316723))) ...
-1 0.000000 1312.900958 IWearRemapper () () () ((FTShoeLeftFTSensors (FTShoeLeftFTSensors 1 0.810120448674894 17.6638752474322<PHONE_NUMBER>32377 -3.0077255165682 -25.7377736510734 5.60779743868969)))...
-1 0.000000 1312.918355 IWearRemapper () () () ((FTShoeLeftFTSensors (FTShoeLeftFTSensors 1 1.42492757758114 17.561519290401<PHONE_NUMBER>14018 -3.01276091695883 -25.6533310826225 5.58480510610912)))...
-1 0.000000 1312.922112 IWearRemapper () () () ((FTShoeLeftFTSensors (FTShoeLeftFTSensors 1 1.42492757758114 17.561519290401<PHONE_NUMBER>14018 -3.01276091695883 -25.6533310826225 5.58480510610912)) (FTShoeRightFTSensors (FTShoeRightFTSensors 1 -0.0464469341376059 19.601314212276<PHONE_NUMBER>42924 -2.86670378805258 -23.6856083655815 5.20430415762279))) ...
...

Sometimes, the values of one shoes are not streamed at all.
cc @lrapetti @diegoferigo @Yeshasvitvs

First check is to see if the number of sensors and the wearable data size are similar; the second check is to investigate the synchronisation of all the sensors.

I already discussed this with someone, not sure whom. Currently the IWearRemapper device performs two operations. It reads WearableData messages from multiple sources and:

Forwards the message to an output port
Exposes and IWear interface from which sensors can be gathered

The first point is implemented as follows:
https://github.com/robotology-playground/wearables/blob/f6999a0f5f36161b4aa4bbdc32ec7bbe2f508092/devices/IWearRemapper/src/IWearRemapper.cpp#L353-L356
Note that this callback is executed every time a single-source message is read. This means that only the data coming from that source is overridden, and the rest instead stays untouched. Actually, since the callbacks can be executed in parallel if two messages from two sources arrive in the same moment, the output message can be mixed.
I just realized that the 2. is thread safe (there are mutexes in the default sensors implementation),  but not 1. If we want to make it thread safe, we should protect with a mutex also this section:
https://github.com/robotology-playground/wearables/blob/f6999a0f5f36161b4aa4bbdc32ec7bbe2f508092/devices/IWearRemapper/src/IWearRemapper.cpp#L364-L365

However, my original point is that the data forwarding to the output port is not an optimal solution. It was developed in the early stages of this project, and now there is a better solution. In my opinion the remapper should only expose an IWear interface. The logic about the message forwarding should be removed from the IWearRemapper, and instead it should be substituted by attaching to it a IWearWrapper device (which rate can be also changed).

In my opinion the remapper should only expose an IWear interface. The logic about the message forwarding should be removed from the IWearRemapper, and instead it should be substituted by attaching to it a IWearWrapper device (which rate can be also changed).

We tried to dump data through the IWearRemapper and we didn't experience any missing data (further tests in the next days will eventually confirm it).

Since this investigation has been done, let's close it.


Double click on disabled undo/redo button toggles fullscreen

Issue
Doble click on disabled history button (when reaching end of history) toggles fullscreen as it would when just clicking on the title bar next to it, instead of just doing nothing, as it should. This is an issue when rapildly undoing, but it might not qualify as an issue.

Potential fix:
Still capturing the click event even though the  button is disabled

not really possible with gtk since it disables all input when a button is disabled


[Utils] ReflectionMark

Plot
When using some reflection-based functionality - it is not quite beautiful to hardcode names of accessed methods or fields right at the point of the call. Using some string constants also might be troublesome, especially when you accessing reflectively your own code, and your project is at the beginning stage, or a stage of active refactoring.
As a solution, right from the start you can create "marks" at methods or fields you are planning to access reflectively. For example:

You have a method that's to be referred reflectively by name
You add @ReflectionMark annotation to this method, and set some string name
You declare static constant of type Mark with the class, declaring method, and the same name
Any other part of the program might use this constant to gain "link" to the method.
Method migth be refactored in any way - for as long as long as caller code doesn't loose reference to the constant it will point directly to the method. And you can refactor Mark constant itself (for example, rename it, or move it to enother class) - with use of modern IDE's caller won't loose it.
Note: the only possible way to brake link between mark and it's target, is to move target to another class, and forget to change class, specified in the mark. Or to change name in either annotation, or mark, but it seems more malicious, than anyhow real useful usecase.

Example:
public class Some {

    /*
     * Mark pointing to the {@link #doSome()} method.
     */
    public static final Mark MARK_DO_SOME = new Mark(Some.class, "mark_do_some");

    @ReflectionMark("mark_do_some")
    public void doSome() {
    }

    public static void main(String[] args) {
        Method m = ReflectionMarks.getMarkedMethod(Some.MARK_DO_SOME);
    }
}

Requirements

Implement @ReflectionMark annotation with runtime retention and targets: [method, field]. Annotation should declare method value() of type String with no default value.
Implement immutable entity Mark, that contains one Class<?> instance and one String instance.
Implement ReflectionMarks class that would provide public static methods:

#getMarkedMethods(Mark):List<Method>
#getMarkedMethod(Mark):Method
#getMarkedFields(Mark):List<Field>
#getMarkedField(Mark):Field

Methods returning lists find *all methods of fields marked with the same mark. So there's the whole another kind of functionality this feature might be utilized for.
Methods returning single instance should throw an exception in case there's no marked target found, or in case there're more than one target found.
Optionally
Lookup methods might be placed in the Mark class itself:

#getMarkedMethods():List<Method>
#getMarkedMethod():Method
#getMarkedFields():List<Field>
#getMarkedField():Field

But then it seems kinda weird, like mark can do something, but it doesn't really know what it's pointing to, field or method. Optinally there might be two classes:

MethodMark

#getMarkedMethods():List<Method>
#getMarkedMethod():Method

FieldMark

#getMarkedFields():List<Field>
#getMarkedField():Field

Then it will be declarator's responsibility to use right mark type, so callers could access right kind of functionality.

Postponed from 0.0.9 release. May be included in the next release when we better understand issues and requirements.

Canceled. To vague and specific.


Devanagari to English

Hi !
I am trying to transliterate Hindi to English, I got the following error, when tried.
Traceback (most recent call last):
File "", line 1, in
File "/Library/Python/2.7/site-packages/indic_transliteration/sanscript.py", line 385, in transliterate
return func(data, scheme_map, **options)
File "/Library/Python/2.7/site-packages/indic_transliteration/sanscript.py", line 307, in _roman
return ''.join(buf)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe0 in position 0: ordinal not in range(128)

try python 3
On Sun, Dec 2, 2018, 00:31 khowal<EMAIL_ADDRESS>wrote:

Hi !
I am trying to transliterate Hindi to English, I got the following error,
when tried.
Traceback (most recent call last):
File "", line 1, in
File
"/Library/Python/2.7/site-packages/indic_transliteration/sanscript.py",
line 385, in transliterate
return func(data, scheme_map, **options)
File
"/Library/Python/2.7/site-packages/indic_transliteration/sanscript.py",
line 307, in _roman
return ''.join(buf)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe0 in position 0:
ordinal not in range(128)
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/sanskrit-coders/indic_transliteration/issues/24, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACipXQW5KXHh5Sd9Yl-5Fof-odg0lNxAks5u04_NgaJpZM4Y9TLo
.


update packages

:white_check_mark: Build TheIdServer 4.5.1-fix-dependencies1-0010 completed (commit https://github.com/Aguafrommars/TheIdServer/commit/63b57364d0 by @)


Dragonbane: initial Release

Submission Checklist
Required

[X] The pull request title clearly contains the name of the sheet I am editing.
[x] The pull request title clearly states the type of change I am submitting (New Sheet/New Feature/Bugfix/etc.).
[x] The pull request makes changes to files in only one sub-folder.
[x] The pull request does not contain changes to any json files in the translations folder (translation.json is permitted)

New Sheet Details

The name of this game is: Dragonbane

The publisher of this game is: Free League

The name of this game system/family is: Dragonbane

[x] I have followed the Character Sheets Standards when building this sheet.

[x] This sheet has been made on behalf of, or by, the game's publisher.
[ ] This game is not a traditionally published game, but a copy of the game rules can be purchased/downloaded/found at: <   >
[ ] This sheet is for an unofficial fan game, modification to an existing game, or a homebrew system.

Changes / Description

Official Dragonbane Character Sheet
This is the official sheet for the Dragonbane RPG system by Free League Publishing. The sheet supports the core rulebook and is integrated with the Dragonbane Roll20 compendium.
Character Sheet Features
The character sheet automates many aspects of the dragonbane system and accessing information from the Dragonbane compendium.
Skills
The sheet handles all aspects of the Dragonbane skill system:

Base chance based on attribute
Marking skills for advancement when they roll a dragon or demon
Advancement rolls
Secondary skills (aka spell schools)

Boon and Bane
The sheet automates the banes from conditions and worn armor. In addition, you can specify other miscellaneous boons and banes that your character has by entering a positive number in the boons and banes inputs at the top left of the sheet. The sheet will calculate your total boon/bane score and roll the appropriate amount of d20's.
Death Rolls
Clicking the death rolls text below the hit points container will roll a death roll, and the result of the death roll will automatically be applied to the character sheet, including reaching 3 successes resulting in gaining 5 hit points.
Spells
Casting a spell is dangerous in Dragonbane. If you roll a demon on your cast check for the spell, the sheet will roll on the magical mishaps table for you and output the result. Maybe you become a fox, cast a completely different spell, or attract the attention of a demon.
Of course spells are also powerful, and the sheet will also output reminder text of what to do when you roll a dragon on a spell roll.
Drag and Drop integration
The sheet supports dragging and dropping of NPCs & Monsters, spells, heroic abilities, weapons, armor, and basic gear. These do not need to be dragged to any specific area of the sheet, simply drag the compendium entry from the right sidebar onto your open sheet and the sheet will populate with the new information. NPCs & monsters can also be dragged directly to the tabletop to create a new journal entry with their statistics.
Summoned Creatures and Swarms
Some creatures, like summons and swarms, have variable stats. When you drop one of these to your game, the sheet will prompt you for information about the stats the creature should have. Additionally, some stats, like damage, can be dynamic changed at each roll to accomodate things like elemental damage being based on the power level they were cast at.
Party Sheet
The character sheet also provides a party sheet functionality to fully support the Dragonbane systems card drawn initiative. This can be used in addition to the Roll20 initiative tracker, or completely in place of it by toggling a setting on the sheet's setting page.
This page is meant to be used as a separate character sheet entry in the game's journal which all players should be given edit access to.
Rolling initiative
To roll initiative using the party sheet, simply select the token you wish to roll initiative for and click the roll initiative button to pull one of the ten initiative cards for that creature, add it to the initiative tracker, and sort the character sheet based initiative order.
Resetting the initiative
The character sheet's initiative order can be reset simply by clicking the reset initiative button.

Character Sheet Info Roll20 Internal Use only.


Incorrect number of parameters

Dear Authors,
In the README.md the number of parameters mentioned for SLaK-T with 51x51 kernels is ~30M however, when I ran your eval code for Imagenet with your trained model, line 352 of main.py printed "number of params: 50703838" which is way more than 30M.
I saw the same issue with the SLaK-S model which in fact had 91131820 parameters in contrast to the ~55M mentioned in the file, and also SLaK-B which had 157950792 parameters in contrast to the ~95M mentioned.
So basically, the accuracies are matching but not the number of parameters to those mentioned.
So, is there a typing mistake in your file or am I missing something?
Just for reference the command is ran was:
python main.py --model SLaK_tiny --eval true
--Decom True --kernel_size 51 49 47 13 5 --width_factor 1.3
--resume path/to/checkpoint
--input_size 224 --drop_path 0.2
--data_path /path/to/imagenet-1k
And I changed SLaK_tiny to the appropriate model names and gave the correct paths for the models and the dataset.

Hi, Thanks for your interest and question. Our models are actually sparse models where a large portion of their parameters are zero. Thus, the reported parameter count refers to the total number of parameters that are non-zero. Specifically, we used 40% sparsity for all the models so the overall parameter count of SLaK-T is 50703838*60% ~= 30M.

Hi,
Thanks a lot for the quick reply. This makes sense. Just maybe a small suggestion, instead of writing it as #params, in this case calling it #non-zero params or #effective params would make it a more accurate table.


Throttling message consumption

Description
I have a C# application which uses the Confluent kafka package to read the message from Azure Eventhub. I have the below major setting from the Confluent Kafka Package.
MaxPartitionFetchBytes =20971520,
CancellationDelayMaxMs = 200
FetchMaxBytes = 5242880
With this major settings while consuming the messages i see the Throttle Request from Azure Eventhub. I have azure eventhub with Standard tier with 1TU, which means Egress: Up to 2 MB per second or 4096 events per second.
How to match this constrain in confluent kafka package setting.

But I don't see the message loss. but it takes time to consume the message.
How to control the throttling?
How to reproduce
Have Azure Eventhub Namespace of Standard tier with 1 TU, Create EH and 1 partition.
Checklist
Please provide the following information:

[ ] A complete (i.e. we can run it), minimal program demonstrating the problem. No need to supply a project file.
[ ] Confluent.Kafka nuget version - 2.1.1
[ ] Apache Kafka version
[ ] Client configuration.
CancellationDelayMaxMs = 200
MaxPartitionFetchBytes =20971520 and 1048579
FetchMaxBytes = 5242880 and 1048579
[ ] Operating system.
[ ] Provide logs (with "debug" : "..." as necessary in configuration).
[ ] Provide broker log excerpts.
[ ] Critical issue.

@mhowlett @codyaray  @ijuma  @codyaray  It will be helpful if any input on this

Couple of observation now after tuning the below configuration.
FetchMaxBytes to 102400
message.max.bytes to 102399
With this I can see the Throttling Request happening in first 2 min when the application starts

Updating the question in this Comment
Problem Statement
Single consumer is consuming the message from the single topic partition. The consuming topic partition has 8578 messages The observation from Azure Eventhub Metrics, Outgoing message is high and seeing the Throttling Request.
This test is performed in Azure Eventhub with Standard Tier with 1 Throughput Unit.
Expected Result
Should see only the 8578 message in Outgoing message metrics and no Throttling Request should see.
Consumer Configuration
AutoOffsetReset =AutoOffsetReset.Earliest
EnableAutoOffsetStore = true
EnableAutoCommit = false
AutoCommitIntervalMs = 10000 //Used only when "EnableAutoCommit " is set to true
EnablePartitionEof = true
ConnectionsMaxIdleMs = 180000
MaxPartitionFetchBytes = 12518400 // 11.9 MB
PartitionAssignmentStrategy = PartitionAssignmentStrategy.CooperativeSticky
IsolationLevel = IsolationLevel.ReadUncommitted
SocketNagleDisable = true
SocketKeepaliveEnable = true
MetadataMaxAgeMs = 180000
SessionTimeoutMs = 30000
MaxPollIntervalMs = 300000
CancellationDelayMaxMs = 200
Consumer Topic Metrics

Note: The incoming messages is there is different time frame. Messages are accumulated once performing the test with different ConsumerGroup for each run for test
Producer Topic Metrics

@mhowlett and Confluent Inc Members Any update on this?

any update on this ?


join elimination

Pull instruction from join into predecessors when it can be eliminated in all but one predecessor.

Obsolete.  SSA has a related problem but there is a separate issue for that.


Allow hide text views

Please let Text views to be Gone or Invisible

https://github.com/GwonHyeok/StickySwitch/commit/1f06f0c4e73b00d2f396dc3791f4fa330a6312a6
i will update new version library as soon as possible