Support for multiple locations

We're going to make this multi-tenant, so different locations can use the same platform.
We'll introduce a concept of a Location, which has Members (relationship to user).
All Shifts will belong to a location.
A member can optionally be a Coordinator, which gives them the ability to add/remove other members and create/remove shifts.

Open question for now is how to handle the concept of several locations that share the same pool of volunteers.
One option is to have a Pool of members.
Let's skip that for the moment though.

We changed our mind on the language, going to call these "teams" with "members".

Also "admin" rather than "coordinator" because it's easier to spell.

Next steps for this:

[ ] Shifts belong to a team
[ ] When a member signs in, they are redirected to the calendar page for their team (if they are on multiple teams they get to pick which one to visit)
[ ] Newly created accounts are prompted to enter a 6 letter invite code to join a team
[ ] Team admins can manage shifts (add/delete)

I'm going to add all current active users to the Pillar Point team using a data migration.

Urgh. I should have called "team" "program".
It's just too hard to change the names of Django apps and models though, so I'm going to leave it. The user-facing name will be "program" though.

I'm going to disable automated deployments for the moment while everything is in flux.

I need to know the station ID closest to Duxbury. Used a SQL query via ChatGPT:
https://www.rockybeaches.com/data?sql=SELECT+*%2C
+++++++SQRT(POWER(37.9015078+-+lat%2C+2)+%2B+POWER(-122.7211458+-+lon%2C+2))+AS+distance
FROM+noaa_stations
ORDER+BY+distance
LIMIT+5%3B

SELECT *,
       SQRT(POWER(37.9015078 - lat, 2) + POWER(-122.7211458 - lon, 2)) AS distance
FROM noaa_stations
ORDER BY distance
LIMIT 5;

It's this one:
9414958

Added Duxbury as a location in my development environment:

And:
TidePrediction.populate_for_station(9414958)

And now I have predictions for both stations (Duxbury and Pillar Point):

I've broken the tests. https://github.com/natbat/pillarpointstewards/actions/runs/6436412869/job/17479721502
auth0_login/tests.py ..........                                          [ 32%]
homepage/tests.py FFF...                                                 [ 51%]
shifts/tests.py .....                                                    [ 67%]
tides/tests.py ...                                                       [ 77%]
weather/tests.py .                                                       [ 80%]
auth0_login/tests.py .....                                               [ 96%]
homepage/tests.py F                                                      [100%]

=================================== FAILURES ===================================
___________________________ test_homepage_logged_in ____________________________

admin_client = <django.test.client.Client object at 0x7f933ae72cd0>

    def test_homepage_logged_in(admin_client):
        response = admin_client.get("/")
        assert response.context["request"].user.is_authenticated
>       assert response.context["upcoming_shifts"] == []

homepage/tests.py:20:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = [[{'True': True, 'False': False, 'None': None}, {'csrf_token': <SimpleLazyObject: <function csrf.<locals>._get_val at ...>, 'DEFAULT_MESSAGE_LEVELS': {'DEBUG': 10, 'INFO': 20, 'SUCCESS': 25, 'WARNING': 30, 'ERROR': 40}}, {}, {'teams': []}]]
key = 'upcoming_shifts'

    def __getitem__(self, key):
        if isinstance(key, str):
            for subcontext in self:
                if key in subcontext:
                    return subcontext[key]
>           raise KeyError(key)
E           KeyError: 'upcoming_shifts'

/opt/hostedtoolcache/Python/3.11.5/x64/lib/python3.11/site-packages/django/test/utils.py:81: KeyError
_____________________ test_homepage_shows_assigned_shifts ______________________

admin_client = <django.test.client.Client object at 0x7f933aa9d6d0>
admin_user_has_shift = None, admin_user_has_past_shift = None

    def test_homepage_shows_assigned_shifts(
        admin_client, admin_user_has_shift, admin_user_has_past_shift
    ):
        response = admin_client.get("/")
>       assert len(response.context["upcoming_shifts"]) == 1

homepage/tests.py:27:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = [[{'True': True, 'False': False, 'None': None}, {'csrf_token': <SimpleLazyObject: <function csrf.<locals>._get_val at ...>, 'DEFAULT_MESSAGE_LEVELS': {'DEBUG': 10, 'INFO': 20, 'SUCCESS': 25, 'WARNING': 30, 'ERROR': 40}}, {}, {'teams': []}]]
key = 'upcoming_shifts'

    def __getitem__(self, key):
        if isinstance(key, str):
            for subcontext in self:
                if key in subcontext:
                    return subcontext[key]
>           raise KeyError(key)
E           KeyError: 'upcoming_shifts'

/opt/hostedtoolcache/Python/3.11.5/x64/lib/python3.11/site-packages/django/test/utils.py:81: KeyError
_____________________________ test_render_calendar _____________________________

admin_client = <django.test.client.Client object at 0x7f933aa61dd0>
admin_user = <User: admin>
rf = <django.test.client.RequestFactory object at 0x7f933ae38b10>

    def test_render_calendar(admin_client, admin_user, rf):
        # Test for just one month
        assert Shift.objects.count() == 0
        # Two other volunteers
        user1 = User.objects.create(username="one")
        user2 = User.objects.create(username="two")
        # Three shifts this month
        empty_shift = Shift.objects.create(
            dawn=make_datetime(2022, 4, 3, 6),
            dusk=make_datetime(2022, 4, 3, 20),
            shift_start=make_datetime(2022, 4, 3, 8, 30),
            shift_end=make_datetime(2022, 4, 3, 8, 30),
        )
        # Current user is on this one
        user_is_on_shift = Shift.objects.create(
            dawn=make_datetime(2022, 4, 7, 6),
            dusk=make_datetime(2022, 4, 7, 20),
            shift_start=make_datetime(2022, 4, 7, 8, 30),
            shift_end=make_datetime(2022, 4, 7, 8, 30),
        )
        user_is_on_shift.stewards.add(admin_user)
        # This one is full:
        full_shift = Shift.objects.create(
            dawn=make_datetime(2022, 4, 21, 6),
            dusk=make_datetime(2022, 4, 21, 20),
            shift_start=make_datetime(2022, 4, 21, 8, 30),
            shift_end=make_datetime(2022, 4, 21, 8, 30),
        )
        full_shift.stewards.add(user1)
        full_shift.stewards.add(user2)
        request = rf.get("/")
        request.user = admin_user
>       rendered = render_calendar(request, 2022, 4)
E       TypeError: render_calendar() missing 1 required positional argument: 'month'

homepage/tests.py:62: TypeError
________________________________ test_homepage _________________________________

client = <django.test.client.Client object at 0x7f933a965810>

    def test_homepage(client):
        response = client.get("/")
        html = response.content.decode("utf-8")
>       assert "<title>Pillar Point Tidepool Stewards</title>" in html
E       assert '<title>Pillar Point Tidepool Stewards</title>' in '<!DOCTYPE html>\n\n\n<title> Tidepool Stewards</title>\n<meta name="viewport" content="width=de...li>\n
\n\n          \n
 \n
\n\n
\n\n\n'

Next: the mechanism for providing 6 character codes to people in order for them to join a program.

Duxbury should use this logo:


change 'coersion' to 'coercion'

326,000 Google results for "type coercion", 5,100 for "type coersion" (where alan-lang.org is on the first page).

Thank you :)


MKV Converting

I have just discovered this project. And I really like it...
However, I have a question:
is there an exact reason why the downloads are always converted to mkv?
Is there a way to deactivate this? Possibly with an environment variable...
Many thanks

The MKV files take up more storage space for me. Without me seeing any advantage in this. Hence the question

MKV files should be a few megabytes at most. It is basically the same size, so I don't really see the argument to be honest

an option to deactivate would be great. it would also be nice to save on conversion. Since I use tdarr anyway. So that would be double anyway


change behavior of 'buildtest config compilers find'

Hey @shahzebsiddiqui, please let me know if you prefer a table format. Thank you!

I think output is good as is.


V4 1016 food index rebuild on changes

Added foodControllers for rebuilding all indexes and specified indexes only (using existing NodeJs Workers)
Added Jobs for rebuilding All and Specified only indexes  (using existing NodeJs Workers and 2 new Redis Jobs)
**Added Redis Set as a message buffer for the Scheduled Jobs to read from **
Added triggers for Specified Indexing Job (Admin Locales changes in Synonyms and Recipe Foods) to Check the Redis Set and executed reindexing for the "modified' locales
Added Redis Pub: Sub as a message broker between the Job and the food-indexing processes running in the cluster

Redis Stream: Adding messages is working, but the format of messages during the read() is weird. Doesn't work properly.

@lukashroch addressed identified issues and refactored the code.


Query: Organise capital cities by population from largest to smallest

All the capital cities in the world organised by largest population to smallest.
All the capital cities in a continent organised by largest population to smallest.
All the capital cities in a region organised by largest to smallest.

As A member of organization x
I Want to view all capital cities in the **world **by population largest to smallest
So That I can see which capital city has the largest population in the world
As A member of organization x
I Want to view all capital cities in a given **Continent **by population largest to smallest
So That I can see which capital city has the largest population in the Continent
As A member of organization x
I Want to view all capital cities in a **Region **by population largest to smallest
So That I can see which capital city has the largest population in the Region
Acceptance Criteria:
Given I run the query
When I have provided no parameters
Then I will see the capital cities listed in the **world **from largest to smallest
Given I run the query
When I have provided 1 parameter
Then I will see the capital cities listed in the **Continent **from largest to smallest
Given I run the query
When I have provided 2 parameters
Then I will see the capital cities listed in the **Region **from largest to smallest

Queries created for this functionality
2 New Methods written
getCapitalCitiesByPopulation
This Method can accept 0, 1, or 2 parameters and will return the list of Capital Cities from largest to smallest
e.g.
getCapitalCitiesByPopulation("Asia","Eastern Asia")
printCapCities()
This Method will print a list of Capital Cities provided to it. The columns included depends on the query run.


Replaced Java based nodes via server with Dynamic js nodes

Fixes #27
Functionality (new functionality is marked with *)

[x] Initial render shows molecules
[x] Selections, should replace hilite functionality

[x] Enable publish selection toggle
[x] Enable subscribe to selection toggle
[x] Hide/show toggles should generate selection column on output port (*)

[x] Store state, like Pocket selection radius, when view is reopened it should be set to radius from before (*)
[x] The ngl canvas should be saveable as a PNG, but don't have stage instance to call makeimage and KNIME Dynamic JS node only support SVG format. (*)
[x] Workflow tests should use new node implementation and pass
[x] Molecule canvas should be expanded to full width/height when using in a component

Uses https://github.com/3D-e-Chem/molviewer-tsx/tree/jsnode branch of app

Created issues (#29 and #30) for new unimplemented features.


Add IdentityFile and LimaHome fields to limactl list format

Based on the suggestion in https://github.com/lima-vm/lima/pull/324#issuecomment-942611305.
Update examples/docker.yaml to show how this can be used to construct the instance-specific path of forwarded sockets.
$ export DOCKER_HOST=$(limactl list docker --format 'unix://{{.LimaHome}}/{{.Name}}/sock/docker')
$ docker info

When the port forwards don't specify an absolute path:
portForwards:
  - guestSocket: "/run/user/{{.UID}}/docker.sock"
    hostSocket: docker

Example: The halog command from https://github.com/lima-vm/lima/pull/324#issuecomment-942633294 can now use the additional filtering by instance name:
$ function halog { tail -f "$(limactl ls ${1:-default} -f '{{.LimaHome}}/{{.Name}}/ha.stderr.log')"; }
$ halog docker
{"level":"info","msg":"Waiting for the optional requirement 1 of 1: \"user probe 1/1\"","time":"2021-11-29T11:58:44-08:00"}
{"level":"debug","msg":"executing script \"user probe 1/1\"","time":"2021-11-29T11:58:44-08:00"}
[...]

portForwards:
  - guestSocket: "/run/user/{{.UID}}/docker.sock"
    hostSocket: docker

Switching to a relative host socket name also exposed a bug in the limactl validate function: We must always use an absolute path to load a lima.yaml file because the hostSocket may be located relative to it, and the validation function makes sure it is absolute.

Probably want to use "docker.sock" also for the host socket, to stay consistent with "ssh.sock" and friends ?
And then use the same for "podman.sock", in the podman.yaml (again to make it more similar throughout lima)

Probably want to use "docker.sock" also for the host socket, to stay consistent with "ssh.sock" and friends ?

Ok, will do that (also to avoid the "reserved word" impression), but it is kind of redundant: The idea was that FOO.sock would be stored by default as sock/FOO inside the instance directory, but keeps the length the same. I didn't want user-forwarded sockets directly in the instance directory to avoid collision with the lima sockets (ha, ga, serial, ssh).

Can we rather use a form like hostSocket: "${{.InstanceDir}}/sock/docker" ?

I was intentionally using the relative name to show how it defaults to {{.LimaHome}}/{{.Instance}}/sock. I wondering if supporting that was a mistake, and we should have always required an absolute path?
Should we "undocument" the relative path case, and changes all samples and docs to use absolute paths instead?

Can we rather use a form like hostSocket: "${{.InstanceDir}}/sock/docker" ?

We didn't actually have {{.InstanceDir}}; you had to write {{.LimaHome}}/{{.Instance}}.
I have now added {{.InstanceDir}} to the limactl list fields, but not (yet) to the hostSocket. Ideally we should use the same field names in both places, but we already have the conflict that {{.Instance}} in hostSocket is called {{.Name}} in limactl list.
I didn't use {{.Name}} in hostSocket because it is ambiguous: it sounds like the name of the socket itself (but that doesn't make sense, as it would be a recursive reference).
My suggestion:

remove documentation for relative hostSocket and the {{.Instance}} fields (but keep them working).
add {{.Name}} and {{.InstanceDir}} fields to hostSocket.
change samples to use {{.InstanceDir}}/sock/FOO.sock everywhere
document that sockets should not go directly into the instance dir, but only into the sock subdir

Thoughts?

Thoughts?

I'm going to take the absence of objections as agreement. 😄
@AkihiroSuda PTAL

Having a variable for the dir sounds like a good idea, similar to $HOME vs /home/$USER (or something).
I like keeping the file extension on the unix socket, even if not "needed". Also makes it easier to filter out...


Enable use_build_context_synchronously linting rule

Is your feature request related to a problem? Please describe.
The rule use_build_context_synchronously seems quite important and may prevent bugs.
Describe the solution you'd like
Enable the rule, set its severity to error and fix all occurrences.

Looks promising, right now this is experimental lint though. So I would wait.

still experimental


External MIRQ some times missed

Hello Stephan,
i use an NEORV32 config with an external IRQ input of MIRQ
IO_MTIME_EN => false, -- implement machine system timer
mtime_irq_i     => std_ulogic(mtime_irq),  -- machine timer interrupt, available if IO_MTIME_EN = false

The belonging c code looks like follows:
/* Processor */
#include <neorv32.h>

void mtime_irq_handler(void)
{
	neorv32_uart0_putc('.'); // send tick symbol via UART
}

int main()
{
    /* init UART */
    neorv32_uart0_setup(115200, PARITY_NONE, FLOW_CONTROL_NONE);

	/* enable error capture */
	neorv32_rte_setup();

	/* install MTIME interrupt handler to RTE */
	neorv32_rte_exception_install(RTE_TRAP_MTI, mtime_irq_handler);

	/* enable MTIME IRQ */
	neorv32_cpu_irq_enable(CSR_MIE_MTIE); 		// enable MTIME interrupt

	/* enable global interrupt */
	neorv32_cpu_eint();

	/* end */
	while (1) {
		__asm("nop");
	}
}

The MIRQ is only asserted one clock cycle. The Waveform looks like follows:

There are three cursors, on the two first leads the IRQ to no UART interaction and the third starts an UART transfer. My expections were three UART transfers. The MIRQ is missing wenn der Signal mtime_irq toggles but there is no action on the UART.

Thanks for your support!
BR,
Andreas

Hey Andreas!
Seems like there is a "bug" in the documentation. Section [2.5.1. RISC-V Standard Interrupts just say something about the FIRQs, but there should be:

The RISC-V standard interrupts are level-triggered and high-active. Once set the signal has to stay high until the interrupt request is explicitly acknowledged (e.g. writing to a memory-mapped register). The RISC-V standard interrupts can NOT be acknowledged by writing zero to the according mip CSR bit.

I'll fix that.
So, your external MTI source has to keep the interrupt signal high until the program explicitly acknowledges the request.

I'll fix that.

Thanks for your help :)

Yes :)


Automated builds

Checklist

[X] I've run make from the root directory to see all Dockerfiles are created successfully.
[X] I've updated the documentation if necessary.

Description
replaces our Docker Hub repos with Automated Build repos (for staging env, for starters—https://hub.docker.com/r/ccistaging)
pulls a README & Dockerfile for each image & sends them to a separate github repo (https://github.com/circleci-public/example-images) which triggers the automated builds
other than that, images continue to be built in the same way (automated builds are just to get our Dockerfiles and READMES into Docker Hub automatically)
seems to be working as-expected for the staging env—after a week or so, I'd like to make the same changes @ https://hub.docker.com/r/circleci

🚀


Dependency package eslint has a new version available

We have identified a dependency on version 7.32.0 of eslint. A new version (8.15.0) is available for upgrade.
Following are the steps to upgrade package dependency.

Understand the breaking changes between the version being used and the version you want to upgrade to.

Identify all packages that take a dependency on this package.

Go to the root folder for each such package (/sdk/service-name/package-name) and update package.json to have the new version.

Run rush update to ensure the new version is pulled in.

Make relevant changes to absorb the breaking changes.

Repeat steps 3 to 5 for each of the packages that have a dependency on this package.

For anyone considering taking up this task, please take up #19160 at the same time in the same PR

Code pointers:

Set up your dev environment
Run rush lint to run the linter across all the packages. This should succeed without any errors. If you see any errors, try to fix them first in a separate pull request and then continue here
Update the version of eslint and @types/eslint used in all the package.json files in this repo
Run rush update to ensure the new versions gets installed
Run rushx build in the eslint-plugin-azure-sdk package folder to find and fix any build errors in the plugin due to the upgrade.
Run rush lint to run the linter across all the packages. This should succeed without any errors. If you see any errors, include the fix for them in the same pull request that made the dependency upgrade

upgrade to ^8.0.0 in PR #21733, and 83e13d88f9


Jump to page not working properly on Mozilla, Edge and IE.

There is an issue on Mozilla, Edge, and IE.
The problem begins if you want to jump to a page and your pdf has a different high of pages.
So, for example when you pass a parameter '#page=20' and there is a page before (Eg page 18) which is lower or higher than the rest of the pages, then 19, 20 and forward will appear not at the top of the page.
I've tried to combinate parameters but does not work on those Browsers, only work properly on Chrome.
#page=20
#page=20,0,0
#page=page-fit
...
Thanks.

Please make sure that you fill out the ISSUE_TEMPLATE.md with all of the requested information when opening an issue; as-is there's unfortunately no actionable information provided here.

Closing until the issue template information is provided, in particular an example PDF file and exact steps to reproduce. Thanks.


Feature request: offline_mode

I would like to use CPM in enterprise development.
For this purpose, it would be convenient if, in addition to CPM_SOURCE_CACHE, there was an option to define a folder to which artifacts are downloaded (initial, i.e. there is a zip archive, or a clone of the repository is performed, etc.),
as well as an option to prohibit attempts to download anything.
If such an option already exists, I would be glad to know, because I have the impression that there is no such thing.

i believe you can just copy paste the CPM_SOURCE_CACHE and it should work in offline mode as long as you didn't update the dependency.

Still, this is not quite it, because instead of archives/repositories and so on, you have to store the internal representation of FetchContent. I don't really understand how everything is arranged, but can't Fetch Content politely ask to download artifacts somewhere? Let's say some boost is downloaded by an archive, we would like to have it in the repository by the same archive.

Just make CPM_SOURCE_CACHE to some .cpm value. Like that you have a folder in your repo with all the downloaded dependency in a subfolder of your repo.
For example: .cpm/boost/somehashthatdefineyourversion

Is that really enough? Ideally there would be some kind of integration to automate this.
To me it just seems like bad design to just mid build download some sources on top of the upstream's pristine sources.
If you could for example integrate better with git by using submodules for sources then the whole issue regarding the offline mode would be solved as sources would be fetched and always up to date assuming the sources checked out were not modified.


Support ChainerX multi-node evaluation

As I mentioned at #7476, ChainerX doesn't support the buffer protocol.
In order to support multi-node evaluation on distributed ChainerX training,
I added typecasting before the communication of the evaluation result.
I think the evaluation may not be a performance bottleneck, so in this case, this is enough.
I checked this PR with #6387 so please test with this.

Since this PR depends on other chainerX-related PRs, we will test & review after they are merged.
Sorry for the delay, and thank you for your understanding. 🙏

Covered by #7844


eth-validator-watcher: init at 0.7.1

Kilnfi's validator watcher.  May be interesting to anyone who wants to roll their own notifier, for upcoming block proposals etc. rather than relying on an external service such as beaconcha.in
I have tried this with lighthouse and it works (using the generic beacon API)

@ekimber nice addition!


Added resources for Machine Learning

Added book on Machine Learning with Python
Added section for online courses (MOOCs) and added two entries from Coursera

@naman1901 Can you squash all the commits? As its just one file and you made three commits.
Once you squash, I'll merge quickly!
Thanks!

I don't know how to do that. Can you help me?

@naman1901 you can read about it here.
Hope it helps! :+1:

Hey @naman1901, are you still working on this? Should I close this?

I can fix this if needed.

I can fix this if needed.

I guess the PR is not active.
@havanagrawal you can go ahead for the fix.


GET /catalog/2022-04-01/items

Describe the bug
GET /catalog/2022-04-01/items doesn't coincide with Amazon sp-api docs, cannot pass sellerId, keywords are required (but in docs they are optional), not all includeData options are supported. To put it in a nutshell, I need to get products from specific seller by seller id
To Reproduce
Steps to reproduce the behavior:
Launch this code:
self.catalog_items_api = CatalogItems(marketplace, **credentials)

products = self.catalog_items_api.search_catalog_items(sellerId=SELLER_ID,
                                                                       includedData="relationships,attributes,dimensions,identifiers,images,productTypes,salesRanks,summaries",
                                                                       marketplaceIds=["ATVPDKIKX0DER"])

Expected behavior
To get products from specific seller by seller id with pagination
Desktop (please complete the following information):

ArchLinux

Hi, it should work by setting the correct version - does it?


feat: immersions and submersions

Update: this is only the correct definition in finite dimensions; needs generalisation to infinite-dimensional manifolds.
Using that local diffeomorphisms have bijective differential (cherry-picked from #8738).
Note that sphere-eversion used a different design for immersions (no differentiability required); I'm open to considering their design.

Reviewer question: what's the best argument order for immersions and submersions: I I' n f or I I' f n
Future possibilities:

define smooth embeddings (done locally)
show that being both an immersion and a submersion is a local diffeomorphism: this requires the inverse function theorem
show: a diffeomorphism is precisely a proper map which is an immersion and a submersion

Linking relevant Zulip thread. Summary: Definitions of immersion and submersion in this PR are not correct for infinite-dimensional manifolds. They are useful but should be called something like "infinitesimally injective/surjective", and they will be implied by the correct definitions in the context of Banach spaces.


MAINT: Revert addition of tox

Let's see whether the failing test comes from the tox insulation of the environment.

The nitransforms bug was related to adding per volume transforms to a chain. Should be fixed in the latest release. Are you able to fix this issue by pinning <24?

Are you able to fix this issue by pinning <24?

That was about to be my next commit :D
This is where I think we are hitting game over:
https://github.com/nipreps/eddymotion/blob/4a6cf72c5fa3dc905215c9db4e32b9298a3cebb1/test/test_integration.py#L44-L55

It is nitransforms.


applying a scale function to new input

I am seeking to jointly process "light" and "dark" integrated intensities previously processed using CrystFEL.

Careless appears to run fine on a single "dark" input MTZ generated from the stream files (e.g. job 24317437_9082). An existing structure can be refined against these dark structure factor amplitudes.

I am able to run a next Careless run on the dark data where I initialize the scales using the scales from the preceding run (--scale-file="./merge_crystfel_24317437_9082_mono_grid_3/dhfr_grid_scale")--see job id 24362624_115. The resulting amplitudes again permit for refinement in Phenix with similar statistics (a little better, FWIW).

When I run Careless on light and dark data (without DW), I get a numerical issues warning and early termination:

Encountered numerical issues, terminating optimization early!

and progress bar:
Training:   0%|          | 0/30000 [00:03<?, ?it/s, Grad Norm=nan, loss=nan, F KLDiv=6.07e+02, NLL=nan, NLL_val=nan]

I then tried to prevent this by providing careless with an approximate initial scale function based on just the dark data (using the same metadata keys). In this case, I get the following error (job ID 24368387_30733)

Traceback (most recent call last):
  File "/n/hekstra_lab/people/dhekstra/conda_envs/careless-12/bin/careless", line 8, in <module>
    sys.exit(main())
  File "/n/hekstra_lab/people/dhekstra/conda_envs/careless-12/lib/python3.9/site-packages/careless/careless.py", line 9, in main
    run_careless(parser)
  File "/n/hekstra_lab/people/dhekstra/conda_envs/careless-12/lib/python3.9/site-packages/careless/careless.py", line 41, in run_careless
    model.scaling_model.load_weights(parser.scale_file)
  File "/n/hekstra_lab/people/dhekstra/conda_envs/careless-12/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 70, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/n/hekstra_lab/people/dhekstra/conda_envs/careless-12/lib/python3.9/site-packages/tensorflow/python/ops/resource_variable_ops.py", line 720, in _restore_from_tensors
    raise ValueError(
ValueError: Received incompatible tensor with shape (2884,) when attempting to restore variable with shape (6292,) and name Variable:0.
WARNING:tensorflow:Detecting that an object or model or tf.train.Checkpoint is being deleted with unrestored values. See the following logs for the specific values in question. To sil
ence these warnings, use `status.expect_partial()`. See https://www.tensorflow.org/api_docs/python/tf/train/Checkpoint#restorefor details about the status object returned by the resto
re function.
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).mlp_scaler.network.layer_with_weights-0.kernel
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).mlp_scaler.network.layer_with_weights-0.bias
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).mlp_scaler.network.layer_with_weights-1.kernel
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).mlp_scaler.network.layer_with_weights-1.bias
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).mlp_scaler.network.layer_with_weights-2.kernel
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).mlp_scaler.network.layer_with_weights-2.bias
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).mlp_scaler.network.layer_with_weights-3.kernel
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).mlp_scaler.network.layer_with_weights-3.bias
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).mlp_scaler.network.layer_with_weights-4.kernel
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).mlp_scaler.network.layer_with_weights-4.bias
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).mlp_scaler.network.layer_with_weights-5.kernel
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).mlp_scaler.network.layer_with_weights-5.bias
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).mlp_scaler.network.layer_with_weights-6.kernel
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).mlp_scaler.network.layer_with_weights-6.bias
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).mlp_scaler.network.layer_with_weights-7.kernel
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).mlp_scaler.network.layer_with_weights-7.bias
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).mlp_scaler.network.layer_with_weights-8.kernel
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).mlp_scaler.network.layer_with_weights-8.bias
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).mlp_scaler.network.layer_with_weights-9.kernel
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).mlp_scaler.network.layer_with_weights-9.bias
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).mlp_scaler.distribution.layer_with_weights-0.kernel
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).mlp_scaler.distribution.layer_with_weights-0.bias

Args: mono
--separate-files
--mc-samples=10
--mlp-layers=10
--image-layers=0
--dmin=2.5
--iterations=30_000
--test-fraction=0.1
--seed=5165
--mlp-width=20

Sorry if I missed it, but can you process the light data alone?

let me try

oddly, the error on step 3 above appears to have been caused by some of the input MTZs missing metadata columns.

your last message seems to indicate this is an issue with the input data formatting and not with the careless CLI. i will close this issue for now.


Add test cases

Describe your changes
Added test cases for employee and potential command
Issue ticket number and link
closes #92
Checklist before requesting a review

[x] I have performed a self-review of my code.
[x] If it is a core feature, I have added thorough tests.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:
see 7 files with indirect coverage changes


unrecognized arguments: --custom-location

getting error while running -- az appservice kube create. Custom location is set but looks like cli is not accepting --custom-location as an argument.
PS /home> $customlocationId
/subscriptions/xxxxxxxxxxx/resourcegroups/arc-demo/providers/microsoft.extendedlocation/customlocations/dev-someapp
PS /home> az appservice kube create -g "arc-demo" -n "dev-do-someapp-dev-01" --custom-location $CustomLocationId --static-ip "<IP_ADDRESS>" -l "eastus" --tags app="someapp"
unrecognized arguments: --custom-location /subscriptions/xxxxxxxxxxxx/resourcegroups/arc-demo/providers/microsoft.extendedlocation/customlocations/dev-someapp

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 95bceeac-dc48-471b-2d60-b7442b60ee06
Version Independent ID: a0381e86-0a67-19a3-eb35-f758ce1c2f2c
Content: Set up Azure Arc for App Service, Functions, and Logic Apps - Azure App Service
Content Source: articles/app-service/manage-create-arc-environment.md
Service: app-service
GitHub Login: @cephalin
Microsoft Alias: cephalin

Thanks for brining this to our attention. We will investigate it further and update you shortly.

I looked somemore into it. See the below 2 images. I can see extension enabled on my remote k8 cluster. And then custom location also visible and looks ok. However in the left menu, I see no option to add new services on this custom location.

this may be related. I can see custom location setup with webapp extension, but when i try to setup webapp, I cant see custom location.

@manipuraco did you run the following commands first?
az extension remove --name appservice-kube
az extension add --yes --source "https://aka.ms/appsvc/appservice_kube-latest-py2.py3-none-any.whl"

When you run az -v you should see appservice-kube 0.1.21 listed as an extension.


Initial support for S3

Resolves #218

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Codecov Report

Merging #224 into master will decrease coverage by 0.33%.
The diff coverage is 71.42%.

@@            Coverage Diff             @@
##           master     #224      +/-   ##
==========================================
- Coverage   86.07%   85.74%   -0.34%
==========================================
  Files          69       69
  Lines        3606     3613       +7
  Branches      539      484      -55
==========================================
- Hits         3104     3098       -6
- Misses        409      416       +7
- Partials       93       99       +6

Impacted Files
Coverage Δ

setup.py
0% <ø> (ø)
:arrow_up:

petastorm/fs_utils.py
88.63% <71.42%> (-3.26%)
:arrow_down:

petastorm/pytorch.py
96.15% <0%> (-3.85%)
:arrow_down:

petastorm/workers_pool/tests/test_workers_pool.py
96.75% <0%> (-2.6%)
:arrow_down:

petastorm/tf_utils.py
79.13% <0%> (-1.74%)
:arrow_down:

petastorm/workers_pool/process_pool.py
93.56% <0%> (-1.17%)
:arrow_down:

petastorm/reader_impl/worker_loop.py
94.44% <0%> (-1.12%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 667a9f4...14fade1. Read the comment docs.

Tested as follows:
$ python examples/mnist/generate_petastorm_mnist.py
# Creates a dataset at `/tmp/mnist`

$ ...
# Uploads the `/tmp/mnist` folder to an S3 bucket named BUCKET_NAME

$ rm -rf /tmp/mnist
# Deletes the local dataset as a sanity check

$ python examples/mnist/pytorch_example.py --dataset-url s3://BUCKET_NAME/mnist
# Starts training successfully

Neat!
On Fri, Oct 26, 2018, 12:35 AM Vivek Panyam<EMAIL_ADDRESS>wrote:

Merged #224 https://github.com/uber/petastorm/pull/224 into master.
—
You are receiving this because your review was requested.
Reply to this email directly, view it on GitHub
https://github.com/uber/petastorm/pull/224#event-1928114112, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADKCDK_rKzi1xrACrkdZgqARo3P3W1ocks5uortdgaJpZM4X7bSo
.


Custom types for migrations

Do you plan on adding some sort of support for custom types for migrations?
I think just the ability to map a custom type to a CqlType would be the easiest work around, maybe a field attribute?
#[charybdis_model(...)] pub struct Example {     #[charybdis(cql_type = Varint)]     pub foo: MyCustomType,     #[charybdis(cql_type = Blob)]     pub bar: OtherCustomType, }

If I understand correctly, you would like to be able to create custom type that would be mapped to CQL type. Those types would need to implement FromCqlVal  and SerializeCql  traits in order to work.
Can you tell me what types you need?
I think it would make more sense if we try to cover each use case in ORM, rather than having support for custom types as we already support all native CQL types & UDTs . E.g. we could add enums that would convert to TinyInt, but if we have complex ones we would usually use UDTs.


Race condition between vixie-cron and cron resource

Description
Background
On vixie cron, when you update a user's crontab file, vixie-cron (ubuntu default) will only parse the sub-files when the folder /var/spool/cron/crontabs mtime is updated from last time it checked. Normally, the directory mtime is updated on the directory when an inode is either introduced or removed from that directory path, not when a file inside changes it's own mtime.
Issue
There exists an instant where if the last two cron resources of a chef run get added in the same second as when cron parses that users crontab, but one before the file descriptor is opened and one after, the last cron resource will never be read by the cron daemon unless it is restarted or the directory is touched.
Chef Version
Chef: 12.4.3
Platform Version
Ubuntu 14.04.5 LTS and Ubuntu 16.04.2 LTS
Replication Case
Have not figured out how to replicate this reliably yet since chef needs to time 2x updates at the exact second vixie-cron checks for updates. We have about 1-2 servers every day hitting this case though
Client Output
N/A
Stacktrace
N/A

Possible workaround to force crontab to reload database after a chef run.
execute "make-cron-spooldir-update-delayed" do
  command 'echo "touch /var/spool/cron/crontabs" | at now + 1 minute'
  action :nothing
end

execute "dummy-crontab-fix-caller" do
  command ":"
  action :run
  notifies  :run, "execute[make-cron-spooldir-update-delayed]", :delayed
end

so that's a bit weird because all the cron resource seems to be doing is using crontab -u to update the crontab.
kind of feels like a bug between the crontab command and vixie cron.
i was sort of suspecting that i'd find a bit of horrible code in the cron provider (it is very unloved) that did some old non-atomic manual file mangling and that the fix would be to have it construct a real file sub-resource to do the work, but having the provider call crontab -u seems to make sense and avoids us having to know where the crontabs are kept on different operating systems...
so i'm not sure what to change here...
i'm legitimately pretty puzzled.  the existing code looks a lot better than i feared and i don't see any of the obvious bugs that i was suspecting would be the root cause (doesn't mean there's no bug, but i don't see it right now).

my understanding of how crontab command works is it will create a copy of the current file, modify it, then copy it into place. This results in two very fast sequential updates on the same second updating directory mtime to the same value when the second one copied in.
The thinking I had was that cron resources could create one atomic update at the end of a run instead of many small updates throughout. This is a rather large change though to deal with a very small race condition.

yeah that would mean writing an accumulator to spool it up and run it at the end.  that is probably better done as an accumulator resource in a cookbook wrapping the cron resource.
so far we don't have any accumulated resources added to chef core, it'd definitely be intrusive to modify how the cron provider worked.
and it seems like any process which does rapid-fire updating of crontabs through crontab -u would trigger the same bug...

Agreed. Our workaround above is functional for the time being. I'll look into an accumulator wrapper though. Thanks for the suggestion 😀

if you want to do that the iptables_rule resource in the iptables cookbook is probably the only accumulator example that i'd recommend right now

so there's the substantially better workaround:
drop individual files into /etc/crond.d/whatever and make them completely idempotent.  you can have one-file-per-task that way you never have to mangle the same file.  the /etc/cron.d directory acts as the 'accumulator'.
generally we'd recommend using /etc/cron.d over using the cron resource in chef.  i think there's a cookbook out there with a cron_d resource, but i typically never bother with that since it boils down to just managing files, and i don't really need that abstraction layer.
if i understand the problem correctly that this is due to two updates to the same user's crontab file (and not two updates to two different crontab files) i think this would work around the problem (and might really be the recommended solution for numerous reasons that boil down to trying to avoid line based file editing wherever possible (and since the cron resource inherently does line based file editing, its a good thing to avoid).

That is indeed a substantially better solution that should be immune to this race condition. Ty for the follow up 😄

yeah i don't see any good way to fix cron.d barring turning it into an accumulator which isn't something we really want to do, and /etc/cron.d solves the whole problem a lot better.
still think this sounds like an upstream bug in vixie-cron though and should really get reported.  i think this is an identical problem to identifying "You have new mail" like BSD /usr/bin/mail does as new messages arrive in /var/spool/mail/user, which also has to deal with the problem of multiple messages arriving at the same time -- i'm sure there are $REASONS why its more complicated than i think though...


Add Github Actions

@kibettheophilus I would suggest I configure Github Actions so that we can have git pushes and PRs that are not failing

I will add you.


Feature Request: Accept WebSocket URLs with parameters.

At first thanks for this awesome crate :-)
As far as I can see, a WS url has this format:
ws://ip_address:port

But some servers require more complex urls like
ws://ip_address:port?parameter=value&parameter2=value ...

This is often used to submit auth tokens etc.
Would be great to have this feature, too.
Thank you very much,
Harald

Just wanted to add that there are also websocket URLs like
ws://subdomain.domain.tld/api/v1:8080?param1=value1&param2=value2 ....

So subdomain and URL paths should be supported too.
-- Harald

Hi @hschneider I'm glad you like message-io 😃
I think the feature you request is already done specifying the connection as RemoteAddr::Str(url), where you can specify the URL you want. Have you tried that?

You're the best :-)
let (server, _) = handler.network().connect(Transport::Ws, RemoteAddr::Str("ws://<IP_ADDRESS>:8080?test=1".to_string())).unwrap();

did the trick. Perfect!


--privileged should add explicit "rw" config for /sys mount

If you create a privileged container using the ctr client, it will explicitly mount /sys using the "rw" flag. See https://github.com/containerd/containerd/blob/9d9d1bc13c107a460212d12ed7ee2f422379a10f/oci/spec_opts_unix.go#L602
cri-containerd simply removes the "ro" flag for --privileged, assuming the container runtime will default to a RW mount, which appears to not be the case. See https://github.com/containerd/cri/blob/ed92befab78c1322ffc958e0fd2202afe5e1f937/pkg/server/container_create.go#L666
As a result, running kube-proxy using containerd 1.1 via cri-containerd results in the following error:
E0426 18:40:33.281905       5 conntrack.go:124] sysfs is not writable: {Device:sysfs Path:/sys Type:sysfs Opts:[ro nosuid nodev noexec relatime] Freq:0 Pass:0} (mount options are [ro nosuid nodev noexec relatime])

It seems fine in my cluster:
# crictl ps | grep kube-proxy
1b5e8f1282b81       sha256:bfc21aadc7d3e20e34cec769d697f93543938e9151c653591861ec5f2429676b   23 hours ago        CONTAINER_RUNNING   kube-proxy
# crictl exec -s 1b5e8f1282b81 /bin/sh -c "cat /proc/mounts | grep sysfs"
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
# crictl inspect 1b5e8f1282b81
...
        {
          "destination": "/sys",
          "type": "sysfs",
          "source": "sysfs",
          "options": [
            "nosuid",
            "noexec",
            "nodev"
          ]
        },
...

Are you sure you start kube-proxy with privileged set?

Thanks for helping me investigate @Random-Liu. I'm not using crictl, I'm creating a PodSandboxConfig and ContainerConfig, based on the v1.Pod specification for kube-proxy. I'm trying to figure out why I'm seeing this inconsistency. It could be something I've missed, but I'm struggling to see what it might be.
 OS is Debian Linux 4.9.0-6-amd64
It's interesting to note that the CRI spec is very clear about what Privileged means: https://github.com/kubernetes/kubernetes/blob/a38a02792b55942177ee676a5e1993b18a8b4b0a/pkg/kubelet/apis/cri/runtime/v1alpha2/api.proto#L541
// 3. Any sysfs and procfs mounts are mounted RW.
What's not clear though is whether runc should interpret Privileged in this way or whether that should be made explicit by the caller. That is indeed the case with ctr - it explicitly adds "rw" for Privileged before runc gets to see the spec.
If that's the answer, then that's the answer. The client needs to make the Privileged mounts explicit. If so, I'll modify my code to do that.

What's curious to me though @Random-Liu is that your mount spec looks just the same as mine as reported by inspect, but you have a different outcome. That potentially suggests an OS configuration difference.

@Random-Liu did you ever figure out this: https://github.com/moby/moby/issues/24000?

@Random-Liu did you ever figure out this: moby/moby#24000?

@corrieb No. We never able to figure that out. Can you update your code to explicitly set rw, and verify whether that fixes your problem? If it does, it's super welcome to send a PR, I'll review it.
I'd like to spend a little more time to figure out why there is such difference. It could be some kernel version difference, or something? Let me do a search.
But anyway, I do think we need your patch if that fixes the problem for you. :)

I'll do some research. If we find a default behavior difference caused by OS version or configure difference, we should fix docker as well.

I think it's pretty much guaranteed to fix it. When I run sudo ctr run -t --privileged docker.io/library/debian:latest myctr on the same system, /sys is mounted "rw" and this is reflected in the spec. This is because ctr adds it explicitly, per the code referenced above.

What this seems to boil down to is what happens if neither "ro" or "rw" are specified. cri-containerd makes sure to remove "ro", but doesn't add "rw". That ultimately makes it ambiguous

@corrieb Yeah, i mean cri-containerd does that because docker is doing so. But it seems that this may not work in some OS config/version.
Would you like to send a PR to add rw? Or I can send one as well if you don't have time. :)

It would be super helpful if you can verify the change, because I can't reproduce it. :)
You can simply make and scp the binary to your node, and restart containerd.

Absolutely. I'll need to go through our legal dept to get myself approved to submit code to containerd, but heck - that's a worthwhile think to do regardless.
@justincormack curious if you have an opinion on this.

@corrieb Is your host sysfs mounted ro?

@Random-Liu no - I checked that early on. That would be weird

@corrieb One last thing... If possible, can you exec into the kube-proxy pod to check whether cgroup is ro? cgroup mount and sysfs mount logic are almost the same, by checking this we know whether this is specific to sysfs specific problem.

@corrieb Thanks for helping me verify. The information is very useful for future debugging.
This seems specific to sysfs, and only happens on some node. Let's explicitly set rw for privileged.

@corrieb Do you have any problem? Probably I can send a fix for you? :)

@Random-Liu Thanks for doing this. I was at KubeCon last week, this was by far the fastest way to get it done.

@corrieb No problem. Did it for you. :)
Thanks a lot for finding and reporting the bug!


fix release flow

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 77.165%

Totals

Change from base Build<PHONE_NUMBER>:
0%

Covered Lines:
196

Relevant Lines:
254

💛  - Coveralls


Replace the QR code from pyFBS site to paper one

Replace the QR code from pyFBS site to paper one.

pre-commit.ci autofix


Request: Export to image (app library)

I would like to export all graph data to PDF and other forms and it would be nice to have the graph in the export as well. This would, presumably, require the image to be available in memory/local app image library, and be set in the document template dynamic image widget.
Can we do that?
For example, ChartJS exports to Base-64, that would be enough :)

Dear Sander.
At the moment it is not possible to save the image grammatically. User can save the image when the toolbar is enabled. see https://github.com/mendixlabs/charts/issues/14
Are you requesting browser side export or a server side export?
Its a much request feature, we will spend some research time on it. Though its hard to accomplish. if you have some suggestions, how we can do this, please let us know.
Cheers,
Andries

Most viable option is to use plotly.js services (REST)
https://api.plot.ly/v2/images
We research this option and if viable will document for Mendix usage

Dear Sander,
The plotly services REST end point is now added to the documentation.
https://docs.mendix.com/howto/extensibility/charts-plotly-images-rest
Please not that this an external service who will charge  use for usage. Though they have some free tier.
Cheers, Andries


drake_bazel_external warnings about reproducibility

While developing jenkins updates the job https://drake-jenkins.csail.mit.edu/job/RobotLocomotion/job/drake-external-examples/job/main/301/consoleFull says:
DEBUG: Rule 'drake' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "5d95cfd141662f42affe0cc50241b129b7a69ca0bcaae6b0ed180000e053c99d"
DEBUG: Repository drake instantiated at:
  /media/ephemeral0/ubuntu/workspace/ion_drake-external-examples_main/src/drake_bazel_external/WORKSPACE:32:13: in <toplevel>
Repository rule http_archive defined at:
  /media/ephemeral0/ubuntu/workspace/ion_drake-external-examples_main/_bazel_ubuntu/9a00cf05048ed9bdf126eba755fd527b/external/bazel_tools/tools/build_defs/repo/http.bzl:372:31: in <toplevel>

It's not clear to me if this is something that needs to be fixed or not.  Feel free to close without any fixes.  CC @jwnimmer-tri

This is typical behavior when using an unpinned dependency:
https://github.com/RobotLocomotion/drake-external-examples/blob/d538d621d8605253763b8e223f0f0ca96fb6bb23/drake_bazel_external/WORKSPACE#L5-L6


Map - profile megastudy/zero imputation-related methods

See if we can find some speedup for the floating visualizations in the megastudy.
Particularly affects sample-based visualisations.
Here's one I made earlier...
Site: qa.vectorbase.org
Study: megastudy
Filters:

Study.study_name LIKE 'Salt Lake' (e.g. filter with search box and select all)
Collection start date: 2012 to 2022 (just select around the pink in the histogram, doesn't have to be exact)
Donut marker overlay: Sample.species (default)
Floater: Timeline, x=collection start date, y=Sample.unbiased specimen count (but other Sample variables with categorical proportions seem equally slow)

For me it was taking around 20 seconds to make a plot. I changed variable in the plot settings rather than change filters to trigger a reload, so that map markers, timeline data etc weren't being requested at the same time.

we can make a separate ticket to profile plot.data, if needed. its been a while since thats been done.. but i think we should profile this first as its the more likely culprit and its never been done.


Directly render a string from database

Is there a way, to directly render a string with twig? So something like this:
$string = "Hello {{ name }}";
echo view($string, array("name" => "world"));

I wanna use TwigBridge for rendering templates from the database.

Oh, sorry! Just found it.
$string = "Hello {{ name }}";
$template = Twig::createTemplate($string);
echo $template->render($array("name" => "world"));


[Metrics] Use pytorch-like ignore_index instead of remove_bg for IoU

🚀 Feature
PL's implementation of the IoU metric has a remove_bg flag that allows to ignore the background class.
I propose changing it to be the same as PyTorch's loss functions,
that is an ignore_index argument that takes the index of the class to be ignored.

Motivation for the proposal
Currently when remove_bg is set to True, the background is expected to be associated to index 0.

This is not a constant among datasets, for example Cityscapes uses index 255 for its ignored semantic segmentation class.
Pytorch's loss functions for classification that allow to ignore a class do it with an ignore_index argument, like CrossEntropyLoss.

That means that when using those the way they are intended to be, you have to manually move the ignored class' channel at the beginning of the tensor in-between the loss and metric computation, and I don't like having that in the middle of my training step.

Pitch
I'd like to have the same interface as PyTorch's loss functions with the ignore_index argument.
On a side note, I also think that the documentation should display the expected shape of inputs, like PyTorch' loss functions, and not keep quiet about the automatic adaptation to cases when the prediction is given in one-hot encoding or as an index array.

Alternatives
Appart from modifying the data between loss an metric computation in the training step, it should be possible to do it at the data-set creation level, with custom transformations, and then use an ignore_index=0 for the loss function.
For the first method, this is what I add:
# set ignored class at pos 0
y_pred = torch.roll(y_pred, shifts=1, dims=1)
y = torch.add(y, 1)
y[y == 256] = 0

Actually, moving the channel after the loss computation is not even possible, since it would mess up the back-propagation.
So I am stuck with the alternative of doing it at the data-set level, which might not be the best thing regarding reproducibility.

@j-dsouza mind taking a look?

For information, I discussed a bit about IoU's usage in #2753 ; I 'm not sure, but I think the present issue is still relevant.

cc: @justusschock @SkafteNicki

I would gladly review a PR that implements this. The idea with the metric package is that it should be as close as possible to native torch.


Invoke Aspects in multi module project

Hi,
Does this plugin support weaving aspects in a multi module project? For example, if the modules in the project are libraries, how can we weave our aspects in those modules too? Currently, I see it working at app level only.
Please suggest.

Hello!
Sorry for late answer. If your aspects live in app module, it will have access to all your local modules due to appearing them in android's transformation stream and in classpath. It should work without any workarounds. But I did not test it this way, please try it in the simpliest way :)
If it won't. You can simply add in config closure in build.gradle parameter includeJar describing your sub-modules you want to augment with aspects.

Thanks @Archinamon , it solves the issue


Remove tuple-ith, just use Get

This PR removes tuple-ith, since with linearity issues resolved (soon) we can use Get instead.
This revealed an issue in saturating context rules, but only in tests. It turns out that adding infinite nestings of contexts, even if they are the same one, can result in the context rules not saturating (InLoop refers to a body, which may already have context).

Yeah
Ensuring saturation when you can keep making new contexts each iterations has been really painful, I'd like to return to the problem sometime


Use -- in pip install command

cc @MSanKeys963

Thanks, @jakirkham.

Sorry had this in a draft review comment.
Think it would be good to include the dev packages on conda-forge, but will update that a bit later


Definition of "serve" in base class "RoutablePageMixin" is incompatible with definition in base class "Page"

Issue Summary
Definition of "serve" in base class "RoutablePageMixin" is incompatible with definition in base class "Page"
def serve(self, request, view=None, args=None, kwargs=None):

vs
def serve(self, request, *args, **kwargs):

Steps to Reproduce
This is not an actual issue you can reproduce if you don't use mypy, but it's rather an annoying inconvenience for me at this point.
I consider this to be an issue because you are supposed to inherit both classes on a custom page class. That means you end up with one method with different signatures, which is not incorrect in Python, but in my humble opinion is at least a code smell. This will get in the way of making stubs for Wagtail.
It's not a massive inconvience but I must explicitly ignore type checking on those classes with mypy, e.g.
class BlogIndexPage(RoutablePageMixin, Page): # type: ignore

It would be nice to make the API consistent for both methods. I guess it'll be really annoying to make that kind of change to a contrib module that's so widely used.

Hi @tm-kn,
Can you give more details of what exactly mypy doesn't like about this setup? My take on what's supposed to be happening here is:

the base Page model doesn't care what extra args/kwargs are passed to serve - it will simply pass on whatever is returned as part of the result of route.
When you inherit RoutablePageMixin, you're allowing RoutablePageMixin to take control over the route / serve process. It takes advantage of the freedom that Page provides, and imposes a specific meaning to the args/kwargs: no args, and three keyword args named view, args and kwargs. RoutablePageMixin's definition of serve is therefore compatible, but stricter.

Perhaps mypy just gets confused by our non-standard use of the names args and kwargs? We could reasonably change them to view_args and view_kwargs if that helps. Alternatively, we could keep the signatures identical by doing something like:
class RoutablePageMixin:
    def serve(self, request, *args, **kwargs):
        view = kwargs.pop('view', None)
        view_args = kwargs.pop('args', None)
        view_kwargs = kwargs.pop('kwargs', None)
        if args or kwargs:
            raise TypeError("unexpected args passed to RoutablePageMixin.serve")

...but then we're basically doing type-checking in Python code, which seems like the opposite of what mypy is supposed to be doing :-)

Have reclassified this issue as "Cleanup/Optimisation" , as the code is valid and functioning as intended.

Have reclassified this issue as "Cleanup/Optimisation" , as the code is valid and functioning as intended.

Closing for now, but can revisit this if we get more details of what mypy is complaining about.

Closing for now, but can revisit this if we get more details of what mypy is complaining about.


Prepend sqllogictest explain result with line number

Which issue does this PR close?

Closes #9443.
Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

I think ci maybe happy after add feature avro.
I don't know whether INCLUDE_TPCH=true cargo test --test sqllogictests --feature avro or INCLUDE_TPCH=true cargo test --test sqllogictests --all-features  would works?

Hi @my-vegetable-has-exploded , could you help me review these small changes, conflict is most likely to happen but i know how to deal with it now ;)

Hi @my-vegetable-has-exploded , could you help me review these small changes, conflict is most likely to happen but i know how to deal with it now ;)

I am not sure the specifics of the issue at hand. For branch conflicts, I think it is okay to merge or rebase it regenerate the result.
Thanks for your work. @duongcongtoai

I am not sure the specifics of the problem at hand. For branch conflicts, I think it is okay to merge or rebase it and regenerate the sqllogicaltests.

The current generated output matches main now, i think it is good to review

Hi @alamb look like only you can merge it :D

Thanks again @duongcongtoai  and @my-vegetable-has-exploded  -- I merged up this PR from main and re-ran CI tests just to be sure we didn't introduce any logical conflicts and it all looks good to me. Thanks agian!


docs(transformer): add documentation for logical-assignment-operators plugin

follow-up #4881

#5014
#5013
#5012  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @Dunqing and the rest of your teammates on  Graphite


Server-side rendering error when imported into Angular Universal Starter

I'm submitting a ...  (check one with "x")
[x] bug report => search github for a similar issue or PR before submitting
[ ] feature request
[ ] support request => Please do not submit support request here, post on Stackoverflow or Gitter

(It might be a documentation upgrade request rather than a bug report. In which case, I do apologise. I sought advice on StackOverflow but go no response there after more than a week, so I'm turning this into a bug report as a last resort.)
Current behavior

When ngx-datatable is imported into otherwise-unmodified Angular Universal Starter app (the one recently upgraded for Angular 5.0), it throws the following runtime error when run with server-side rendering:
dist\server.js:128395
__metadata("design:paramtypes", [Event]),

ReferenceError: Event is not defined

The server.js code it's referencing is from the DataTableBodyRowComponent of ngx-datatable:
__decorate([
  core_1.HostListener('mouseenter', ['$event']),
  __metadata("design:type", Function),
  __metadata("design:paramtypes", [Event]),
  __metadata("design:returntype", void 0)
], DataTableBodyRowComponent.prototype, "onMouseenter", null);

Expected behavior

Angular Universal Starter is a standard, often-recommended seed for Angular Universal apps. ngx-datatable should either work with it out of the box, OR the docs should be updated to explain what other changes need to be made.
Reproduction of the problem

Clone Angular Universal Starter and install dependencies.

build and run with npm run build:ssr && npm run serve:ssr. It runs fine.

install ngx-datatable and import into app.module.ts as per the current documentation:
....
import { NgxDatatableModule } from '@swimlane/ngx-datatable';
....
imports: [
  ....
  NgxDatatableModule
],

run clientside with ng serve. It runs without errors.

build and run serverside with npm run build:ssr && npm run serve:ssr. It throws the above error.

What is the motivation / use case for changing the behavior?

Please tell us about your environment:

Windows 10, Node 8.9.1

Table version: 11.0.4, 11.0.3

Angular version: 5.0.1

Browser: Node Express server performing server-side rendering. Not seeing the same error client-side.

Language: TypeScript 2.4.2

Pushed 11.1.3, give that a try

That solves the issue, thanks. ngx-datatable now runs out of the box on a server-side implementation of Angular Universal Starter.


⚠ Bofrost.lu has degraded performance

In ce8f488, Bofrost.lu (https://www.bofrost.lu) experienced degraded performance:

HTTP code: 200
Response time: 1705 ms

Resolved: Bofrost.lu performance has improved in 9aa7f45 after 12 minutes.


Improved Windows Generation & Build Scripts

Describe the issue (if no issue has been made)
Windows build scripts in Scripts/Windows currently do not work when executed by double-clicking on them. MSBuild location is also tied to a specific version of VS (Community).
Changelist

Added vswhere.exe to locate MSBuild executable
Added common script to generate a vs project
Added common script to build a vs project (using vswhere.exe)
Updated GenerateVSX.bat scripts to use Common/GenerateVSProject.bat
Updated BuildWindow*.bat to use Common/BuildWindowsTarget.bat
Renamed GenerateVS.bat to GenerateVS17.bat for consistency

Additional context
Tested changes by running:
✅ GenerateVS17.bat
✅ GenerateVS19.bat
✅ GenerateVS22.bat
And building the runtime and editor using:
✅ BuildWindows.bat
✅ BuildWindows-Editor.bat
Test machine info:
Processor	AMD Ryzen 9 3900X 12-Core Processor 3.80 GHz
Installed RAM	64.0 GB
Edition	Windows 11 Pro
Version	 	23H2
OS build	22631.2792
Experience	Windows Feature Experience Pack 1000.22681.1000.0
MSBuild		17.8.3+195e7f5a3 for .NET Framework

vswhere.exe version: https://github.com/microsoft/vswhere/releases/tag/3.1.7

Nice! Thank you


Implementation of MCD capability descriptor and SA-Attestation Object (SAAO) as per ISO/IEC JTC 1/SC 17/WG 4 N 4566

Add module issuance::mcd with the structures for MCD capability descriptor and SA-Attestation Object (SAAO).

Closing this as we opened https://github.com/spruceid/isomdl/pull/97 because of some merge issues from some old code from coset integration


Properly document support for shared memory  (aka groupshared)    (was: implement too)

EDIT: Answer in https://github.com/EmbarkStudios/rust-gpu/issues/695#issuecomment-888448064  , but docs still need improvement.
Shared memory within a group (as annotated with groupshared in HLSL) doesn't seem to be documented at least, and I can't figure out how to use it.
Basically, this in HLSL:
groupshared float4 buffer[window_size];
where threads in a whole group can share data with each other (with careful use of group barriers).
The documentation for the barriers should also be linked from the documentation for this. Also, barriers need better documentation with examples. For example https://embarkstudios.github.io/rust-gpu/api/spirv_std/arch/fn.memory_barrier.html# , it really could use a copy-pastable line. It's not super clear what to pass into the two generics arguments.
So if support isn't there, it needs to be implemented, or if it is, it needs to be documented .

This appears to be referenced here:
FYI, while you might be used to HLSL nomenclature, this may

This appears to be referenced here:
https://github.com/EmbarkStudios/rust-gpu/issues/180
and here:
https://github.com/EmbarkStudios/rust-gpu/issues/8
But the actual implementation I can't find in issues.
Another problem is that "groupshared" is not standard nomenclature, and is HLSL specific, and probably should not be used as the name for this.

OpenCL calls this local memory.  This however is confusing because it conflicts with CUDAs concept of local memory which ironically isn't local.

SYCL uses the same nomenclature as OpenCL.

OneAPI  (DPC++) uses the same nomenclature as OpenCL.

CUDA calls this shared memory.  They are the people who coined this in the first place.

OpenGL calls this shared memory.

Vulkan calls this shared memory.

HIP/ ROCM calls this shared memory.

AFAIK the only reason HLSL calls this "groupshared" is because it screwed up the naming of "shared" to refer to something else before the advent of GPGPU computing (I still don't understand what " Mark a variable for sharing between effects; this is a hint to the compiler. " even means, and I can't even find an example of raw "shared" HLSL memory usage anywhere).
Heck HLSL even refers to this memory as "shared memory" in their own documentation:

HLSL enables threads of a compute shader to exchange values via shared memory. HLSL provides barrier primitives such as GroupMemoryBarrierWithGroupSync, and so on to ensure the correct ordering of reads and writes to shared memory in the shader and to avoid data races.

Good point, slightly changed the issue.

Shared memory needs to be specified in the entry point with the workgroup storage class. Besides that it works the same as a normal/generic slice.
#[spirv(compute(threads(256)))]
pub unsafe fn downsample_cs(
    ..
    #[spirv(workgroup)] scratch: &mut [[f32x4; 16]; 16],
) {

Ah, thanks! Then this issue is only about improving the documentation so that it's easy to find.

I like the workgroup naming as it matches the SPIR-V naming scheme.
Regarding barriers, the test cases for the arch instrinsics are a good starting point, just needs replacing of the scopes and semantics:
https://github.com/EmbarkStudios/rust-gpu/blob/main/tests/ui/arch/memory_barrier.rs
https://github.com/EmbarkStudios/rust-gpu/blob/main/tests/ui/arch/control_barrier.rs

@hrydgard Actually didn't know this either, and even though I suggested using shared, I think following spirv semantics in code is much more important, so I agree with @msiglreith that nothing should change codewise, though clearly there is an issue with finding if these features have been implemented or are being worked on or not.


Feature/hyperparameter tuning

added mlora_train_optuna.py script

Passed all local tests.
Code quality checks with flake8.
Isolated changes to only include mlora_train_optuna.py.

can we have a design doc for this pr, meanwhile, why we need to change lots of configuration in the training model

can we have a design doc for this pr, meanwhile, why we need to change lots of configuration in the training model

I've added a design to the PR description to outline the changes. Regarding the configuration changes, they were necessary to enable hyperparameter tuning using Optuna. This tuning process allows key training parameters such as learning rate, rank, alpha, and dropout to be optimized dynamically rather than being fixed. The goal was to improve the model's performance by letting the tuning process find the optimal configuration for each training trial automatically, thus adding flexibility to the training pipeline.

can you just only add a new file : mlora_train_optuna.py


add transpose_pad_fusion kernel for inference

Did you benchmark the performance improvement of the MR.
benchmark results are welcome in comparison with Effective transformers or Faster Transformers?

Did you benchmark the performance improvement of the MR. benchmark results are welcome in comparison with Effective transformers or Faster Transformers?

My implementation can achieve better performance in samples with larger seq_len variance (e.g. datasets in GLUE). I am going to make a more detailed evaluation tomorrow.

Did you benchmark the performance improvement of the MR. benchmark results are welcome in comparison with Effective transformers or Faster Transformers?

My implementation can achieve better performance in samples with larger seq_len variance (e.g. datasets in GLUE). I am going to make a more detailed evaluation tomorrow.

Thanks, post the result here if possible.


show meaningful content when crowdloan inactive

when the crowdloan

hasn't started yet
has ended

The visitor should be informed accordingly

don't care for hasn't started. only check if current block (head) > crowdloan.end


Add a regex implementation of Regexp to spinoso-regexp

I've had this branch on my local machine for a long time. pushing it up as WIP.
Initial work on this branch began on This work is squashed from Mon Dec 28 09:25:13 2020 -0800 in 5f0226dbafae3fd7b3a9653f99777207d0dd06b5.

cleaned up the commit history in a way that keeps the timestamps that show the history on this one. Almost two years old!