Clarification on x/y-grid direction in standard names

There are various standard names for x- and y-components of vector quantities. For instance: sea_water_x_velocity. The description always states: "x" indicates a vector component along the grid x-axis, positive with increasing x. I assume that this description stems from the idea that the computational grid is a simple Cartesian mesh in some coordinate reference system. The attribute name grid_mapping also refers to that assumption.
However, in my case the curvilinear grid is not Cartesian in any coordinate system, but the grid may have been created in some local coordinate system (e.g. a UTM zone), so the grid coordinates are stored using projection_x_coordinate and projection_y_coordinate variables and the grid_mapping describes (via WKT) the relationship between those x/y coordinates and global latitude/longitude. On this curvilinear grid a sea_water_x_velocity quantity is stored. How should it be interpreted? As

a vector component along a (curvilinear) grid direction (but which one)?
a vector component along the projection_x_coordinate direction?

If 1, how should we interpret the data if it results from an unstructured model without two grid directions?
If 2, can we remove the word "grid" from the description "x" indicates a vector component along the grid x-axis, positive with increasing x?
Best regards,
Bert

@hrajagers The axis is identified on a dimension coordinate variable using the axis attribute. In your case, the X axis is the projection_x_axis.

Hi Jim,
Thank you for the reply. In your first sentence you refer to a "dimension coordinate variable" ... I can't find that exact phrase in the CF conventions document. So, I assume that you refer to a one-dimensional "coordinate variable" for which dimension name and variable name are equal such as
float x(x) ;
  x:standard_name = "projection_x_coordinate"
  x:axis = "X"

However, I don't have a one-dimensional coordinate variable. My projection_x_coordinate is still a two-dimensional array, so an auxiliary coordinate variable in CF terms. For example,
float x(m,n) ;
  x:standard_name = "projection_x_coordinate"
  x:axis = "X"

Do I understand it corretly that in this case the sea_water_x_velocity is pointing in the projection_x_coordinate direction?
Is it necessary (or merely recommended) to add the axis attribute?
Best regards,
Bert

@hrajagers Yes, "dimension coordinate variable" is new terminology that we are currently developing to help reduce confusion.
The axis attribute may only be applied to dimension coordinate variables (not auxiliary coordinate variables). If you don't have an actual dimension coordinate variable, you rely on the standard_name values of projection_x_coordinate and projection_y_coordinate to identify the conceptual axis directions.
Your use case definitely makes the appropriate identification difficult. This is a good example of where our use of the word "grid" can make everything more confusing. I think it would be worth a proposal to change the language.

Hi Bert and Jim,
I think that it is now permitted to have the axis attribute on auxiliary coordinate variables (quite buried: sixth paragraph of  http://cfconventions.org/Data/cf-conventions/cf-conventions-1.8/cf-conventions.html#coordinate-system), and it can be set on N-d auxiliary coordinate variables, too. So
float x(m,n) ;
  x:standard_name = "projection_x_coordinate"
  x:axis = "X"

is OK.
My understanding(?) is that the direction "vector component along the grid x-axis" in the standard name definition is indeed assuming that there is a 1-d dimension coordinate variable of the form float x(x) ;, which you do not have.
Are the component directions in your case defined as unrotated east and north? If so, then perhaps the standard_name eastward_sea_water_velocity is better, with the x and y auxiliary coordinate variables defined as you already have them?
Thanks,
David

@davidhassell I was trying to find the discussion that led to the new language about the axis attribute. I vaguely remember it, but I can't find it anywhere. Is the intent to allow application to auxiliary coordinate variables that aren't 1D?

Hi Jim,
It took me while but I think the discussion can be found have a look at http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2017/thread.html (search for threads "axis attribute"). These exchanges also  refer back to discussions in late 2006/early 2007 (same thread name). Also https://cf-trac.llnl.gov/trac/ticket/8, modified by https://cf-trac.llnl.gov/trac/ticket/62.
I've only skimmed them so far, but there is a lot of discussion on the 2-d case. I was going on memory when I said that it's OK for 2-d variables, but I shall delve deeper ....

Hi @davidhassell : the conformance document still states that "The axis attribute is not allowed for auxiliary coordinate variables", but the CF checker appears to be updated to accept it in lien with the changes you refer to, which came into CF-1.6.

Thank you all for digging into this topic! If you all agree that the sea_water_x_velocity is the velocity component in the direction indicated by the axis = X attribute, and if that attribute can be used on an auxiliary coordinate variable (typically 2D for a curvilinear structured mesh or 1D for an unstructured mesh) then that is more or less in line with what we were hoping for. In that case, I would appreciate some guidance on -- as @JimBiardCics suggested -- how to best propose to change the language as it links to both the CF conventions document (related to the axis attribute on auxiliary coordinate variables) and the standard names table.
@davidhassell You asked whether the component directions in my case are defined as unrotated east and north? The original vector component in the numerical model are staggered and directed along the m and n dimensions of the curvilinear C-grid. The postprocessing routine transforms these vector components to co-located components along the x and y directions of the model domain which may be unrotated lat/lon (in which case we indeed write eastward_ and northward_sea_water_velocity) but usually are "arbitrary" Cartesian x/y for which we can't guarantee x to be eastward and y to be northward. However, since people usually select a standardized regional CRS, in most cases x will align with eastward and y with northward. So, then the question becomes ... Do I assume the alignment (and write eastward and northward components) and accept that the metadata in the NetCDF file may be incorrect when the assumption is not valid, or do I go for always correct metadata and merely indicate that the vector components are in the regional Cartesian x/y directions ... We made the latter choice, which triggers this discussion.

In a current project, we have a similiar use case as @hrajagers : our projected x/y coordinates depend on two dimensions. Our wind x_wind and y_wind blows along the coordinate axes (the 1d-ones).
int i(i) ;
   ...
int j(j) ;
   ...
double x(j, i) ;
   x:standard_name = "projection_x_coordinate" ;
   ...
double y(j, i) ;
   y:standard_name = "projected_y_coordinate" ;
   ...
double lon(j, i) ;
   lon:standard_name = "longitude" ;
   ...
double lat(j, i) ;
   lat:standard_name = "latitude" ;
   ...
float x_wind(j, iv) ;
   x_wind:standard_name = "x_wind" ;
   ...
float y_wind(jv, i) ;
   y_wind:standard_name = "y_wind" ;
   ...

I added the i(i) and j(j) variables so that the user will (hopefully) wonder if the x-wind is orientated along the axis with ...:standard_name = 'projected_x_coordinate' or along the 1d coordinate axis. Maybe, we should also rename our variable x and y.
New standard names like wind_along_dimension_one, wind_along_dimension_two, sea_water_velocity_along_dimension_one, ... would be the most simple solution for clearly indicating the situation described in item (1) of the initial post of Bert.
Similarly, one could add standard names like wind_along_coordinate_variable_one, ... to indicate that the wind is oriented as  described in item (2) of the initial post of Bert. In addition, one needed to specify the coordinates attribute properly and provide the coordinate axis in correct order (which is not necessary up till now).

Hi Daniel,
The reference to dimension _one and dmension_two is a bit fragile. When switching from C to FORTRAN the developer/user may easily make a mistake since the order of the dimensions would be switched. I would suggest to use a more neutral name for instance wind_along_dimension and add an attribute direction_dimension = 'i' or dimensionId in general to point to the relevant dimension.
Such an approach might also be used for vector quantities in a direction corresponding to a auxiliary coordinate variable (direction_coordinate = 'x'), but the earlier replies to my initial post indicate that the axis = 'x' attribute combined with standard_name = 'x_wind' is already available for that now.
Bert

Hi Bert,
sorry, I miss-read your post and the one of Martin. Thanks for the clarification.
Daniel

Thank you all for digging into this topic! If you all agree that the sea_water_x_velocity is the velocity component in the direction indicated by the axis = X attribute, and if that attribute can be used on an auxiliary coordinate variable (typically 2D for a curvilinear structured mesh or 1D for an unstructured mesh) then that is more or less in line with what we were hoping for. In that case, I would appreciate some guidance on -- as @JimBiardCics suggested -- how to best propose to change the language as it links to both the CF conventions document (related to the axis attribute on auxiliary coordinate variables) and the standard names table.

Some Ocean Models use a NEMO tripolar model grid in the horizontal plane.  This is not a projection.
The is represented in netCDF with 2 2D auxiliary coordinates, providing a WGS84 latitude longitude coordinate value pair at each grid point.
The axis attribute is used to indicate the x and y and to enable quantities which are solved on grid axes to be represented.
This is deemed reasonable for this data, even though the x and y directions vary with respect to the coordinate reference system across locations in the horizontal grid.
So, I think that the methodology described here is sound and has precedent.

@neumannd Do your dimensional coordinate axes i and j have any physical representation?

@JimBiardCics No: no physical representation. I just added them because the two wind components are oriented along these axes. The axis attribute should be attached to them as I learned above.
int i(i) ;
   i:axis = "X" ;
   ...
int j(j) ;
   j:axis = "Y" ;
   ...

How are the wind components oriented along non-physical axes?

How are the wind components oriented along non-physical axes?

Depends on the definition of a physical axis ... . The physical meaning depends on the values of  x(i,j) and y(i,j).

The wind flows along the grid points defined by x(i,j) and y(i,j). x and y span up an arbitrary grid.
x_wind(i=a, j=b) points from (x(i=a, j=b), y(i=a, j=b)) to (x(i=a+1, j=b), y(i=a+1, j=b)).
y_wind(i=a, j=b) points from (x(i=a, j=b), y(i=a, j=b)) to (x(i=a, j=b+1), y(i=a, j=b+1)).

A special case would we:
If x(i, j) = x0(i) for all i in {1, ..., I} and all j in {1, ... J} (where we start counting with 1; where I and J are the sizes of dimensions i and j, respectively), if y(i, j) = y0(j) for all [... as above ...] and if inverse_projection(x0(i), y0(j)) = (lon(i), lat(j)) with lon(i+1) > lon(i) and lat(j+1) > lat(j) for all [... as above ... except some special treatment for i=I and j=J], then x_wind would be eastward_wind and y_wind would be northward_wind.

@neumannd Thanks for taking the time to clarify.
This sounds a lot like satellite swath data. It seems to me that there must be a frame in which you have physical axes of a sort embedded in there somewhere, but if that information isn't available, it isn't. You also may not have any control over the wind component values, but it sure looks to me like the best solution would be to transform the wind vectors into the x/y frame.
It does look like your best hope is to use the axis attributes as you described above.

@JimBiardCics Thanks for the feedback. Actually, its urban atmospheric model data with resolved buildings. The grid is often aligned to the building structure. Therefore, the grid orientation considerably depends on the city or district in focus. But, I just help formatting/processing the data properly and don't have any deep understanding of the topic itself ;-) .

I believe this question has been answered so I'm closing the issue.


AUI-1876

Hey @mairatma,
Attached is an update for http://issues.liferay.com/browse/AUI-1876.
Please let me know if you have any questions.
Thanks!

Just started reviewing :) :octocat: Sent from GH.

Hey @jonmak08 and @westonhancock, thanks for working on this! This feature will be really nice to have.
I've added some inline comments, but besides those I think the behavior that was implemented is different from the one described on the issue. The issue wants the timepicker popover to scroll down to the time closest to the currently displayed time, not the current time on the user's clock. The title of the issue may be misleading, but its attached image explains it better. Besides, it does seem that it would be more helpful to show a time closer to the last picked time, instead of the current time.

Hey @mairatma
Thanks for the input. Did you want me to make those changes? Also, changing the functionality to scroll to the time closest to the currently displayed time will be no problem at all.

Yep, that would be great. Thanks again!

If there is no default time value displayed (i.e. "Meeting at hh:mm") should I have the picker scroll to the time nearest the users clock or not scroll at all?

I'm not sure what the best behavior would be in this case, but I think it makes sense to scroll to the current time, it's probably more useful to show that than to start at 12AM, so it's a good idea to try.

I'm closing this since it has been sent again as a new pr (https://github.com/mairatma/alloy-ui/pull/209).


Add BitMex price feed

Fixes #96.

Would you mind going one step further and wiring the price feed into the UIs as well?
We might want to add a separate price feed for that.
Alternatively could also create a follow up ticket :)

Would you mind going one step further and wiring the price feed into the UIs as well?
We might want to add a separate price feed for that.
Alternatively could also create a follow up ticket / PR:)

Would prefer a follow-up as this integration is functional it itself!

f.e. I think the integration will require to also let the CfdActor now about the price change so it might not just be an integration into the feed.


Quick remove the cut and paste error

Issue  cited by visitor "Akaskela" (thank you!) ... it was a cut and paste error that got undone in one of the updates.
Category

[x] Content fix
[ ] New article

Related issues:

mentioned in  #141

What's in this Pull Request?
While working on 141, comment came it citing the cut and paste error which can be fixed immediately.  Remove the errant lines
Guidance
Fix as noted thanks

Docs Build status updates of commit eef3392:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

Community/powerplatformlicensingforcitizendeveloper.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

@sympmarc not sure if this block is an issue?

Did you submit to master?

Looks like it was working as it should .. I see it going to live now, thanks all!


Improve performance of queries that result in many blocks

Is your feature request related to a problem? Please describe.
This request is related to a performance issue.
When clickhouse_connect.driver.client.Client.query_df results are sent back in a large number of separate blocks, the performance of clickhouse_connect.driver.npquery.NumpyResult.df_result suffers greatly. For example, if a query whose result contains 30k rows is returned in 5000 separate blocks of ~6 rows each (as can happen depending on the query and, I believe, outside of the client's control), the df_result method takes 10 seconds to construct the final DataFrame, because the generator returned by NumpyResult._df_stream() processes each block separately, unmarshalling it into a DataFrame in ~2ms and yielding it. Concatenating the resulting 5000 DataFrames into one by NumpyResult.close_df() then takes an additional 2-3s.
Describe the solution you'd like
An improved implementation of NumpyResult.df_result that performs well even if the result is returned by the server in a large number of small blocks.
I can propose the following simple alternative implementation of NumpyResult._df_stream.pd_blocks, which, in the example I gave above, improves the runtime of NumpyResult.close_df() by a factor of 3. Better solutions can probably be found with more time invested in testing and profiling.
from itertools import chain
...
class NumpyResult(Closable):
...
    def _df_stream(self) -> Generator:
...
        def pd_blocks():
            yield pd.DataFrame(dict(zip(self.column_names, [chain(*b) for b in zip(*block_gen)])))

Describe alternatives you've considered
I've considered somehow affecting/reducing the number of blocks returned in the response, but have not figured out a way to do it.
Additional context
clickhouse_connect==0.6.16

That seems like a very large number of blocks for the 30k rows response.  Are they very wide rows?  How large is the entire result set?
Your solution basically "unstreams" the result and pulls the entire HTTP response into memory at once, so while it makes sense for smaller result sets, it will hurt memory usage and otherwise defeat the purpose of streaming when blocks are "normal sized".
You might check the preferred_block_size_bytes setting on your server and play with sending it as a setting with your query.

In my proposal I was trying to have a minimal code impact. I will, of course, defer to you about what change would make most sense. I get the point about streaming - perhaps it would make sense to combine blocks into larger-sized chunks, but not into a single chunk as in my proposal (maybe add a itertools.batched around the chain(*b)?).
The performance issue, however, is real and it's currently causing us to consider circumventing the clickhouse_connect code path altogether.

Just to reiterate, I think your optimization makes perfect sense in the non-streaming case and I have no problem with implementing that in the next release.  It's a fairly small code impact to decouple df_result from df_stream which I'm happy to do for the performance gain.
It also may make sense to batch smaller blocks in the streaming case, but the difficulty comes with deciding how large the batches should be.
Getting back to the root cause, I think it would be very helpful to understand why you have so many blocks for such a small number of rows, and the better solution to the issue might be tuning the ClickHouse settings or the queries to return fewer blocks.

Figuring out why my query results in so many small blocks would, indeed, be great, but I haven't been successful at it.
I initially thought it was because my where clause included, in addition to selecting a single table partition, only one other field that occurs last in my lengthy order by list. However, when I change the where clause and replace that field with the very first field in the order by list, the result still comes in thousands of small pieces. My current hypothesis is that this is determined by the Clickhouse server implementation and table schema, and completely outside of the querying client's control.

@georgipeev - there's an attempted improvement in 0.7.2.  In order to maintain the correct dtypes, I had to use a somewhat different approach and I don't know how it compares with your fix since I don't have simple way to test dataframes with many small blocks at scale.  If you could test it out and report the results that would be appreciated.
You did mention looking at other alternatives to clickhouse-connect -- did you any further down that path and find anything better for the same queries?  Any other ideas of how to handle this would be helpful references.

Thanks!
In the two test cases I ran, the performance of 0.7.2 was indistinguishable from that of 0.6.16. There were some new type-related woes I had to take care of to get 0.7.2 to run properly too.

Ah, that's unfortunate.  Apparently the main performance hit is in concatenating the underlying pandas Series objects, not the DataFrame wrappers.  I think I'll need to construct a useful test case for small blocks to dig further into the bottleneck.
Just so I have some background, what exactly were the type problems you mentioned?  Are they pandas or Python types?  Did you use a different pandas version?

Some string columns that used to come back as dtype object/str now come as pandas.StringDtype().

Okay, I think that's the expected behavior if the use_extended_dtypes argument to query_df is set to True (which is the default).  And yes, that's probably been updated since 0.6.16.

@genzgd Can you revert the change then? The change is breaking the index behaviour of the resulting Dataframe. Prior to 0.7.2, the index of the resulting Dataframe was always contiguous, but after 0.7.2 the index of the resulting Dataframe now is only contiguous within each block.
If it's not possible to revert the change, then there should be a note about this Breaking Change in the documentation, I think.

Figuring out why my query results in so many small blocks would, indeed, be great, but I haven't been successful at it.

@georgipeev  Are you able to share you table schema and you query?

@cwegener I'll take a look -- do you know if there is there a way to fix the index by somehow either recalculating the index of the series that are being concatenated or recalculating the index of the resulting dataframe?  I'm not sure what the options are here, but concatenating dataframes seems expensive.
I do have ideas for other ways of combining larger blocks before they are converted to Pandas at all, but I've not yet had time to explore that route.

Hmmm ... I just had another look at the 0.7.2 change. I can see why the change has no effect on the problem reported by Georgi.
The change introduced in 0.7.2 simple replaces the one call to pd.concat() that takes all blocks and creates a new DataFrame with multiple calls to pd.concat() - one call per block .... which is just the same thing. The complexity before and after the change is the same (O(n))

Regardless of the fact that the 0.7.2 should not be expected to have any impact on Georgi's original problem as stated in my previous comment ...

do you know if there is there a way to fix the index

This should do the trick (can't test yet because I would need to also update my code with the same datetime precision type changes that Georgi alluded to)
diff --git a/clickhouse_connect/driver/npquery.py b/clickhouse_connect/driver/npquery.py
index 513c81a..5b47133 100644
--- a/clickhouse_connect/driver/npquery.py
+++ b/clickhouse_connect/driver/npquery.py
@@ -100,7 +100,7 @@ class NumpyResult(Closable):
         new_df_series = []
         for c in chains:
             new_df_series.append(pd.concat([pd.Series(piece, copy=False) for piece in c], copy=False))
-        self._df_result = pd.DataFrame(dict(zip(self.column_names, new_df_series)))
+        self._df_result = pd.DataFrame(dict(zip(self.column_names, new_df_series))).reset_index()
         self.close()
         return self

@cwegener The change is somewhat more subtle than that.  Yes, the complexity is still the same, I was just hoping that Series concat was more efficient than DataFrame concat, but apparently not.
@georgipeev's original improvement flattened the incoming data into a single array of Python native values, and then created a DataFrame from that array, but when I tried that the specific dtype was lost (so for example, anything with Nones became a dtype object, which seemed like even more of a breaking change).  That problem might be solved by inferring the specific Pandas dtype from the Clickhouse column type, and apply it to the raw data after the DataFrame is created from the raw Python objects, but that will require some extensive testing.
My other thought for an improvement is to combine the binary columns from multiple from ClickHouse blocks before any transformation, but that is moderately complex and requires tracking the size of the blocks coming in to know when to combine.
Finally the reset_index() seems like a straightforward fix to the non-consecutive index issue that should be fairly inexpensive, so if I don't have time in the near future for something better (which is unfortunately likely), I'll at least add that.

I did add a limited test to test small blocks, using the ClickHouse setting max_block_size, and my observations are:

The change in 0.7.2 actually does improve performance by about 40%
@georgipeev's suggested change of yield pd.DataFrame(dict(zip(self.column_names, [chain(*b) for b in zip(*block_gen)]))) actually significantly reduces performance
The index on the Dataframe under 0.7.7 looks contiguous/consecutive,

So to keep this open, I'll need some more information from @georgipeev -- in particular some kind of reproducible dataset.
@cwegener If you could open a new issue with a reproducible example of the broken behavior you're seeing, I'd definitely like to understand what's going on.
For reference, I've been testing under Python 3.11, Pandas 2.2.1, and using the following query to gauge rough performance numbers:
client.query_df('SELECT number, randomString(512) FROM numbers(5000000)',
                               settings={'max_block_size': 50}

That takes a little over 18.5 seconds on my mac, while the default max_block size takes about about 11.5 seconds.

If you could open a new issue with a reproducible example of the broken behavior you're seeing, I'd definitely like to understand what's going on.

I'll test and report results in a new issue.

I'm sorry, but I don't think I would be able to generate a sanitized set that reproduces the problem. Regarding the schema, the table is partitioned by day, has 6 additional order by columns, and about 50 other columns. As I mentioned earlier, the where clause that results in a heavily fragmented result selects a single partition and has a filter on just one more field from the order by list (which one seems to not be important).

@genzgd hello, I've done a test on a dataset with 7 million rows and 30 columns.  env: python==3.8 pandas==2.0.3 clickhouse-connect==0.7.16
query_df("select * from dataset") takes 26s;
pd.read_parquet(io.BytesIO(c.raw_query("select * from dataset format parquet"))) takes 10s;
I hope the performance of query_df should be no worse than the parquet workaround.

@Kevin9436 That's a great datapoint.  To achieve the same results with Native format in clickhouse-connect will be very challenging, since the parquet approach offloads data transformation (from ClickHouse format to the binary, columnar parquet format) to the ClickHouse server, and the parquet transformation to pandas in the client is essentially done entirely in C.
Some of that difference could be due to the datatypes used, since clickhouse-connect still uses pure Python for several transformations.  Any information about the dataset you used will be helpful to determine the potential opportunities for improvements.

Thanks for the explanation. Here is the ddl of my dataset, if it helps.

I got another problem with the parquet approach. I found the column with Date datatype will be transformed to int32, which is the count of days elapsed since epoch. Is there any way I can read the parquet properly without another query of the ddl of the table?

Thanks for the schema.   I'm fairly sure we could do some Pandas optimizations with either Arrow or low level numpy to improve the query_df for that kind of basic datatypes, but as usual I'm not sure when it will get prioritized.
For parquet queries I'm not sure what kind of options are available to converting it to the right datatypes.  There might be some suggestions in the main ClickHouse documentation or Github issues.

No additional information received about issues with performance with many small blocks, and as noted above, the proposed solution actually made performance worse.  Closing this unless there is more information that it is a significant issue.


params in hooks inside prefixed groups are incorrectly typed never

What version of Elysia is running?
1.1.11
What platform is your computer?
Darwin 23.6.0 arm64 arm
What steps can reproduce the bug?
Type-check the following code:
import Elysia, { t } from "elysia";

const idRoutes = new Elysia({ prefix: "/:id" })
  // .guard({ params: t.Object({ id: t.String() }) }) // Without this it doesn't work
  .onBeforeHandle(({ params }) => console.log(params.id))
  .get("/", ({ params }) => console.log(params.id));

What is the expected behavior?
The type should pass, i.e. params.id in .onBeforeHandle should be a string, like in the .get call, because the prefix already contains a dynamic param.
What do you see instead?
TypeScript error, as params in the .onBeforeHandle call is of type never.
Additional information
If you uncomment the .guard line, it works. But of course that's a workaround, and it should work without the workaround above.
Though, the Elysia type system is so good that even a typing workaround still looks really really neat.
Have you try removing the node_modules and bun.lockb and try again yet?
Yes

Fixed and published under 1.1.13 with test case included
see https://github.com/elysiajs/elysia/compare/1.1.12...1.1.13
If this doesn't fix the issue, feel free to tag me or reopen the issue.