speed to write files to s3 storage

https://github.com/pangeo-data/pangeo-eosc/blob/main/notebooks/SwiftXarrayTest.ipynb
@sebastian-luna-valero @guillaumeeb
I've tried to write the Zarr file following the example and I confirm it is very slow.
Is there any way to make it faster?

I can check with CESNET after @guillaumeeb confirms issues.

@tinaok could you point towards the code cell that you fill is slow? I have to admit I didn't go into details about performances, but this seemed reasonable to me when I did those tests.
If I remember correctly, this was between 0.5s and 2s per chunk, depending on chunk size.

Thank you @guillaumeeb and @sebastian-luna-valero
Here is the example I was making to teach everyone to write their Zarr file close to https://pangeo-clivar.vm.fedcloud.eu/jupyterhub/
https://github.com/pangeo-data/clivar-2022/blob/main/tutorial/examples/notebooks/save_to_cesnet_bucket_zarr_s3fs.ipynb

@guillaumeeb If you have better example than that, please place them at https://github.com/pangeo-data/clivar-2022/blob/main/tutorial/examples/notebooks/ and make pull request?

So, I've looked at the notebook and tried to reproduced. This is something I've experienced during my tests: when writing a Zarr file to S3 on CESNET, there is an non compressible time before the writes of chunks actually start. This time is about 30s.
From your example, we can see that it is not related to Dask. So this must be on Zarr or Xarray side. I'm not sure what is happening between the Python command and task graphs begin in Dask. Maybe it's the time it takes to create Metadata? But this seems really high. You can verify this by using a LocalCluster and watching the tasks stream.
Maybe it comes from CESNET, I don't know. Did you try the same code using another Cloud infrastructure?
What I can say, is that once the writes of chunks are started, the bandwidth seems reasonable, so bandwidth and latency of CESNET storage looks good when writing the chunks of data.
We can try simple writes from command line or with s3fs to see if it comes from CESNET, or from something internal to Xarray or Zarr, or maybe from a conjunction of both...

Hm, trying
ds.to_zarr("/tmp/myfile", mode='w', consolidated=True)

takes less than a second, so there is definitely something going on with Zarr and the object store.

I've also tried to sync the dataset from local storage to CESNET object store using aws s3 cli:
 time aws s3 --endpoint https://object-store.cloud.muni.cz --profile pangeoswift sync /tmp/myfile s3://tmp/guillaumeeb/
upload: ../../tmp/myfile/.zattrs to s3://tmp/guillaumeeb/.zattrs
upload: ../../tmp/myfile/lon/.zarray to s3://tmp/guillaumeeb/lon/.zarray
upload: ../../tmp/myfile/Tair/.zattrs to s3://tmp/guillaumeeb/Tair/.zattrs
upload: ../../tmp/myfile/Tair/.zarray to s3://tmp/guillaumeeb/Tair/.zarray
upload: ../../tmp/myfile/lat/0 to s3://tmp/guillaumeeb/lat/0
upload: ../../tmp/myfile/lat/.zarray to s3://tmp/guillaumeeb/lat/.zarray
upload: ../../tmp/myfile/.zgroup to s3://tmp/guillaumeeb/.zgroup
upload: ../../tmp/myfile/.zmetadata to s3://tmp/guillaumeeb/.zmetadata
upload: ../../tmp/myfile/lat/.zattrs to s3://tmp/guillaumeeb/lat/.zattrs
upload: ../../tmp/myfile/Tair/0.0.0 to s3://tmp/guillaumeeb/Tair/0.0.0
upload: ../../tmp/myfile/lon/0 to s3://tmp/guillaumeeb/lon/0
upload: ../../tmp/myfile/time/0 to s3://tmp/guillaumeeb/time/0
upload: ../../tmp/myfile/time/.zattrs to s3://tmp/guillaumeeb/time/.zattrs
upload: ../../tmp/myfile/time/.zarray to s3://tmp/guillaumeeb/time/.zarray
upload: ../../tmp/myfile/lon/.zattrs to s3://tmp/guillaumeeb/lon/.zattrs

real    0m1.989s
user    0m0.733s
sys     0m0.184s

Less than 2 seconds, so the problem does not come from CESNET per se. We're in the last situation, conjunction between using Zarr/Xarray and the object store. We should try the notebook on another system with S3 storage if possible. Else, maybe ask some help on Pangeo community side.

Thank you very much @guillaumeeb !!  I would like to start letting students use the object storage for writing their Zarr file.
I'll may be then warn student if it is small netcdf file, they should write it in netcdf format? And use Zarr only for big Zarr tmp file writing until we have definitive solution.
Do you have a way for avoiding one students deleting other student's data?

Did you try the same code using another Cloud infrastructure?

I have access to https://us-central1-b.gcp.pangeo.io/ but I do not have access to it's s3 storage so I can not bench.  May be @rabernat can help us?

Thank you @rabernat but I do not understand yet what I should change in this notebook to use Zarr.storage.FSStore ...
Also, we have one more question that we want to ask how US Pangeo is doing https://github.com/pangeo-data/pangeo-eosc/issues/17#issuecomment-1277768559

To profile this, here is what I am doing. From my notebook, I run
%prun -D zarr.prof ds.to_zarr(f'{SCRATCH_BUCKET}/air_temperature.zarr')

Then I download thezarr.prof file and visualize it locally with python -m snakeviz zarr.prof. This produces a view like this:

zarr.prof.gz (gzipped because github doesn't allow just .prof files.)
What I have learned from this is that an awful lot of time is being spent on calling __contains__ on the store. In other words, Zarr is checking, maybe too much, what data are already stored. If the list operation on your object store is slow, this will take even more time.
I would be very interested to see the same profiling for your object store. Feel free to run the same command and post your results here.
This seems like a good target for optimization within Zarr. I'm pinging @martindurant, and expert on all of these things, to see if he can see any low-hanging fruit.

I've tried switching from S3Map to FSStore with no luck, the time taken is about the same, between 40 and 45s!
So I profiled the call, an here is what I get:

The profiling file.
zarr.prof.gz
I'm not sure how to analyze this, it seems that every steps take longer time. I guess this is probably due to a higher latency of some commands?

@rabernat @guillaumeeb Thank you.
I'm still stuck with saving Zarr file FSStore
I tried
import zarr
access_key = !aws configure get aws_access_key_id
access_key = access_key[0]
secret_key = !aws configure get aws_secret_access_key
secret_key = secret_key[0]
your_name='tinaok'
s3_prefix =  "tmp/"+your_name
zarr_file_name= "newzarr"
uri = f"s3://{s3_prefix}/{zarr_file_name}"
storage_kwargs={
         "key":access_key,
         "secret":secret_key}
store = zarr.storage.FSStore(uri, **storage_kwargs)

@guillaumeeb How did you specify 'endpoint_url': 'https://object-store.cloud.muni.cz'  ?

Here's a useful breakdown of your profile.
A huge amount of time is being spent on __contains__ calls. I am quite certain we can optimize this. Next step is to start looking at the zarr source code more carefully.

ncalls
tottime
percall
cumtime
percall
filename:lineno(function)

1
7.953e-05
7.953e-05
46.81
46.81
~:0()

1
4.942e-05
4.942e-05
46.81
46.81
api.py:1511(to_zarr)

1
2.793e-05
2.793e-05
46.81
46.81
:1()

1
1.597e-05
1.597e-05
46.81
46.81
dataset.py:1935(to_zarr)

105
0.002647
2.521e-05
46.71
0.4448
asyn.py:60(sync)

105
0.001294
1.232e-05
46.71
0.4449
asyn.py:108(wrapper)

105
0.001647
1.569e-05
46.69
0.4446
threading.py:280(wait)

106
0.001035
9.767e-06
46.69
0.4404
threading.py:563(wait)

421
46.68
0.1109
46.68
0.1109
~:0(<method 'acquire' of '_thread.lock' objects>)

1
8.417e-05
8.417e-05
43.37
43.37
zarr.py:502(store)

1
8.858e-06
8.858e-06
43.37
43.37
api.py:1235(dump_to_store)

1
0.000419
0.000419
31.93
31.93
zarr.py:580(set_variables)

33
0.000313
9.483e-06
24.07
0.7295
storage.py:1411(contains)

33
0.0002367
7.173e-06
24.07
0.7294
mapping.py:171(contains)

4
0.0001056
2.64e-05
16.42
4.104
creation.py:19(create)

4
7.26e-05
1.815e-05
16.42
4.104
hierarchy.py:1034(_create_nosync)

4
6.31e-05
1.578e-05
16.42
4.104
hierarchy.py:786(_write_op)

4
3.062e-05
7.655e-06
16.42
4.104
hierarchy.py:1029(create)

4
7.684e-05
1.921e-05
14.97
3.743
storage.py:283(init_array)

19
0.0001746
9.191e-06
12.4
0.6524
storage.py:101(contains_array)

8
0.0001491
1.864e-05
11.71
1.463
hierarchy.py:367(contains)

18
0.0002274
1.263e-05
11.68
0.6487
storage.py:109(contains_group)

28
0.0003081
1.1e-05
10.26
0.3663
storage.py:1366(getitem)

28
0.0002857
1.02e-05
10.26
0.3663
mapping.py:133(getitem)

4
0.0002816
7.039e-05
9.202
2.3
storage.py:427(_init_array_metadata)

11
0.0003803
3.457e-05
8.249
0.7499
storage.py:1379(setitem)

4
7.747e-05
1.937e-05
6.561
1.64
common.py:149(add)

4
0.0002106
5.264e-05
6.56
1.64
core.py:1750(_set_selection)

4
4.959e-05
1.24e-05
6.56
1.64
core.py:1260(setitem)

I think the problem is, that a directory listing is cached by the s3fs instance, and makes further lookups much faster. But exists() (used by __contains__) in the absence of a cache will not call list but instead call head on the one item. This is much faster for the one item, but slower in the long run for many items. Also, while the _exists() method is async, and so many could be called concurrently, there is no simple API for checking the existence of many files in bulk - this does it serially. zarr isn't using the listdir method of the store, I'm not sure when that would get called.
A workaround would be to call .find() on the filesystem object before processing, which populates the directory cache all the way down, in a single call.
store.map.fs.find(rootdir);

A workaround would be to call .find() on the filesystem object before processing, which populates the directory cache all the way down, in a single call.

Do you mean before calling ds.to_zarr?
The thing here is that the store is cleared, either by doing it manually before writing the zarr dataset, or either by Zarr I guess?
Anyway, clearing the store manually does give speedup. And on a store cleared, store.map.fs.find(uri) just returns an empty list.
Sorry @martindurant, I'm not sure if I've correctly understood your suggestion.
In any case, as we are starting from a free bucket/uri, Zarr should not have to call __contains__ so many times, right?

the store is cleared

That's a good point! In this case, zarr should know that there are no keys available at all.
Actually, by writing to the directory, you invalidate the cache anyway, so my suggestion would only be true for reading, not writing.
I wonder whether s3fs should attempt to update update the relevant part of the directory cache when completing a write, rather than invalidating it. I have checked, and AWS doesn't quite give you the same information on finalizing a file as when querying it afterwards.

@martindurant @rabernat
would you like to have a connection to the cluster?
If yes plz
follow https://pangeo-data.github.io/clivar-2022/before/setup.html#running-the-tutorial-on-the-european-open-science-cloud-eosc
TLDR
click
https://aai-dev.egi.eu/registry/co_petitions/start/coef:290
and
https://aai.egi.eu/registry/co_petitions/start/coef:386
“statement for purpose” you can just say "hello"

No thanks! I am not going to log in your cluster. 🙃  But I am happy to interact here on GitHub around optimizing Zarr and fsspec to improve your use case.

Thank you @rabernat !

Hi @martindurant and @guillaumeeb
I tried with FSStore and  find (51 s) and with FSStore and with out find (51.3 s)
Compare to using s3fs (52.4 s)
So I could not observe the speed up.   May be I am doing something wrong?
Thank you

@tinaok this is also what I observed above, no speed up with these changes. So we've profiled the calls to try to understand why it takes such time on this object store.

I think the question is, when zarr wants to write a chunk, why does it first need to check if that key already exists? We mean to overwrite anyway, so it shouldn't need to know. If we were instead updating part of a chunk, we would need to first read the values from the part of the chunk that is not being changed - but not in this case.

when zarr wants to write a chunk, why does it first need to check if that key already exists?

Now we are asking the right questions. 👌
@tinaok & @guillaumeeb - this is how Pangeo has made progress over the past years: carefully identifying performance bottlenecks and working systematically to fix them upstream in the open source libraries. If you have the time and interest, I really encourage you to dig into the zarr source code and see if you can optimize this a bit. It's very satisfying! One behalf of the zarr developers, I can say we will gladly welcome a PR on this topic.
I also think there is something weird going on with your object store. Those list operations are going very slow compared to GCS or AWS S3. So perhaps there is a chance to optimize on that side as well.
If we can optimize both, I believe we can achieve a 10 - 100x speedup of this workflow.

Thank you @rabernat @martindurant

I also think there is something weird going on with your object store. Those list operations are going very slow compared to GCS or AWS S3. So perhaps there is a chance to optimize on that side as well.

CESNET is based on openstack, would it be other native protocols I should use to make it faster may be? (or do we see slow down due to that in the trace file?)

I went down a bit of a rabbit hole with this and learned some interesting things.
To understand what was happening better, I created a metaclass that would help me log everything the FSStore object was doing:

import logging
import sys
import os
from functools import wraps, partial
from types import FunctionType, MethodType, MethodWrapperType
from time import perf_counter

from zarr.storage import FSStore

def format_args(args):
    if not isinstance(args, tuple):
        args = (args,)
    return ', '.join((
            repr(a)[:32]
            if not isinstance(a, bytes)
            else f'{len(a)} BYTES'
            for a in args
        ))

def wrapper(method):
    @wraps(method)
    def _impl(self, *method_args, **method_kwargs):
        tic = perf_counter()
        method_output = method(self, *method_args, **method_kwargs)
        toc = perf_counter()
        elapsed = 1000*(toc - tic)
        arg_reprs = format_args(method_args)
        output_reprs = (' -> ' + format_args(method_output)) if method_output else ''
        self.logger.info(f"{method.__name__}({arg_reprs}){output_reprs} - {elapsed:2.5f} ms") # ({method_args}, {method_kwargs})")
        return method_output
    return _impl

class LoggingMetaclass(type):

    def __new__(cls, clsname, bases, attrs, methods_to_log=()):

        instance = super().__new__(cls, clsname, bases, attrs)
        #instance = type(clsname, bases, attrs)

        methods_to_overwrite = (
            method_name
            for method_name in dir(instance)
            if callable(getattr(instance, method_name, None))
            and method_name in methods_to_log
        )

        for method_name in methods_to_overwrite:
            setattr(instance, method_name, wrapper(getattr(instance, method_name)))

        logging.basicConfig(level=logging.INFO, stream=sys.stdout)
        instance.logger = logging.getLogger(clsname)
        instance.logger.info("initialized")

        return instance

class Foo:

    def __init__(self, a):
        self.a = a

    def get_a(self):
        return self.a

    def __getitem__(self, key):
        if key=="a":
            return self.a
        else:
            raise KeyError

class LoggingFoo(Foo, metaclass=LoggingMetaclass, methods_to_log=['get_a', '__getitem__']):
    pass

lf = LoggingFoo(a=1)
lf.get_a()
lf['a']

store_methods = [
    'getitems', '__getitem__', 'setitems', '__setitem__',
    'delitems', '__delitem__', '__contains__', 'keys'
]

class FinalMeta(type(FSStore), LoggingMetaclass):
    pass

class LoggedFSStore(FSStore, metaclass=FinalMeta, methods_to_log=store_methods):
    pass

With this I could watch the action as follows
import xarray as xr
import gcsfs

fs = gcsfs.GCSFileSystem()
SCRATCH_BUCKET = os.environ['SCRATCH_BUCKET']
print(SCRATCH_BUCKET)
try:
    # make sure scratch dir is empty
    fs.rm(SCRATCH_BUCKET, recursive=True)
except FileNotFoundError:
    pass

ds = xr.tutorial.open_dataset("air_temperature.nc").load()
encoding = {var: {"chunks": ds[var].shape} for var in ds.variables}
store = LoggedFSStore(f'{SCRATCH_BUCKET}/air_temperature.zarr')
ds.to_zarr(store, encoding=encoding, mode='w')

which gave me the following logs
INFO:LoggedFSStore:__setitem__('.zgroup', 24 BYTES) - 161.02744 ms
INFO:LoggedFSStore:__contains__('.zarray') - 134.78823 ms
INFO:LoggedFSStore:__getitem__('.zgroup') -> 24 BYTES - 43.74526 ms
INFO:LoggedFSStore:__contains__('lat/.zarray') - 76.33124 ms
INFO:LoggedFSStore:__contains__('lat/.zgroup') - 73.61064 ms
INFO:LoggedFSStore:__contains__('air/.zarray') - 78.77026 ms
INFO:LoggedFSStore:__contains__('air/.zgroup') - 78.51281 ms
INFO:LoggedFSStore:__contains__('lon/.zarray') - 79.81741 ms
INFO:LoggedFSStore:__contains__('lon/.zgroup') - 87.17914 ms
INFO:LoggedFSStore:__contains__('time/.zarray') - 78.06299 ms
INFO:LoggedFSStore:__contains__('time/.zgroup') - 78.81959 ms
INFO:LoggedFSStore:__setitem__('.zattrs', 307 BYTES) - 141.40278 ms
INFO:LoggedFSStore:__contains__('lat/.zarray') - 74.32906 ms
INFO:LoggedFSStore:__contains__('lat/.zgroup') - 72.73923 ms
INFO:LoggedFSStore:__contains__('.zarray') - 83.58116 ms
INFO:LoggedFSStore:__contains__('.zgroup') -> True - 59.21848 ms
INFO:LoggedFSStore:__contains__('lat/.zarray') - 77.10479 ms
INFO:LoggedFSStore:__contains__('lat/.zgroup') - 80.31471 ms
INFO:LoggedFSStore:__setitem__('lat/.zarray', 315 BYTES) - 149.82518 ms
INFO:LoggedFSStore:__getitem__('lat/.zarray') -> 315 BYTES - 44.97753 ms
INFO:LoggedFSStore:__setitem__('lat/.zattrs', 159 BYTES) - 143.53136 ms
INFO:LoggedFSStore:setitems({'lat/0': b'\x02\x013\x04d\x00\x) - 65.50918 ms
INFO:LoggedFSStore:__contains__('lon/.zarray') - 78.73831 ms
INFO:LoggedFSStore:__contains__('lon/.zgroup') - 69.94445 ms
INFO:LoggedFSStore:__contains__('.zarray') - 75.09015 ms
INFO:LoggedFSStore:__contains__('.zgroup') -> True - 82.24049 ms
INFO:LoggedFSStore:__contains__('lon/.zarray') - 78.93038 ms
INFO:LoggedFSStore:__contains__('lon/.zgroup') - 78.95320 ms
INFO:LoggedFSStore:__setitem__('lon/.zarray', 315 BYTES) - 141.58281 ms
INFO:LoggedFSStore:__getitem__('lon/.zarray') -> 315 BYTES - 30.87437 ms
INFO:LoggedFSStore:__setitem__('lon/.zattrs', 160 BYTES) - 148.20924 ms
INFO:LoggedFSStore:setitems({'lon/0': b'\x02\x011\x04\xd4\x0) - 69.60758 ms
INFO:LoggedFSStore:__contains__('time/.zarray') - 72.39811 ms
INFO:LoggedFSStore:__contains__('time/.zgroup') - 81.09918 ms
INFO:LoggedFSStore:__contains__('.zarray') - 82.42646 ms
INFO:LoggedFSStore:__contains__('.zgroup') -> True - 47.30075 ms
INFO:LoggedFSStore:__contains__('time/.zarray') - 75.26746 ms
INFO:LoggedFSStore:__contains__('time/.zgroup') - 72.75635 ms
INFO:LoggedFSStore:__setitem__('time/.zarray', 318 BYTES) - 141.51360 ms
INFO:LoggedFSStore:__getitem__('time/.zarray') -> 318 BYTES - 38.26456 ms
INFO:LoggedFSStore:__setitem__('time/.zattrs', 192 BYTES) - 140.15968 ms
INFO:LoggedFSStore:setitems({'time/0': b'\x02\x01!\x08@[\x00) - 68.22306 ms
INFO:LoggedFSStore:__contains__('air/.zarray') - 70.68645 ms
INFO:LoggedFSStore:__contains__('air/.zgroup') - 60.35751 ms
INFO:LoggedFSStore:__contains__('.zarray') - 78.56590 ms
INFO:LoggedFSStore:__contains__('.zgroup') -> True - 58.17347 ms
INFO:LoggedFSStore:__contains__('air/.zarray') - 81.18960 ms
INFO:LoggedFSStore:__contains__('air/.zgroup') - 70.30224 ms
INFO:LoggedFSStore:__setitem__('air/.zarray', 367 BYTES) - 163.85888 ms
INFO:LoggedFSStore:__getitem__('air/.zarray') -> 367 BYTES - 49.24272 ms
INFO:LoggedFSStore:__setitem__('air/.zattrs', 471 BYTES) - 150.49870 ms
INFO:LoggedFSStore:setitems({'air/0.0.0': b'\x02\x01!\x04 %\) - 242.37073 ms
INFO:LoggedFSStore:keys() -> <generator object FSMap.__iter__ - 24.50295 ms
INFO:LoggedFSStore:__getitem__('.zattrs') -> 307 BYTES - 35.04010 ms
INFO:LoggedFSStore:__getitem__('.zgroup') -> 24 BYTES - 27.86014 ms
INFO:LoggedFSStore:__getitem__('air/.zarray') -> 367 BYTES - 34.22666 ms
INFO:LoggedFSStore:__getitem__('air/.zattrs') -> 471 BYTES - 31.78257 ms
INFO:LoggedFSStore:__getitem__('lat/.zarray') -> 315 BYTES - 25.75439 ms
INFO:LoggedFSStore:__getitem__('lat/.zattrs') -> 159 BYTES - 45.25747 ms
INFO:LoggedFSStore:__getitem__('lon/.zarray') -> 315 BYTES - 38.31415 ms
INFO:LoggedFSStore:__getitem__('lon/.zattrs') -> 160 BYTES - 33.82616 ms
INFO:LoggedFSStore:__getitem__('time/.zarray') -> 318 BYTES - 27.00831 ms
INFO:LoggedFSStore:__getitem__('time/.zattrs') -> 192 BYTES - 35.61871 ms
INFO:LoggedFSStore:__setitem__('.zmetadata', 3989 BYTES) - 65.72519 ms
INFO:LoggedFSStore:__getitem__('.zmetadata') -> 3989 BYTES - 62.98140 ms

It is RIDICULOUS how much unnecessary listing of the same objects over and over is being done.
I am confident that we can pretty easily get a 10x improvement here by refactoring Zarr.

Out of curiosity, I tried implementing a very naive implementation of storing the data which bypasses most of xarray. This allows us to see whether xarray itself is responsible for these inefficient calls
import zarr
from xarray.backends.zarr import encode_zarr_attr_value

def naive_store(ds, store):
    group = zarr.open_group(store, mode="w")
    group.attrs.update(ds.attrs)
    for v in ds.variables:
        a = group.create(v, shape=ds[v].shape, chunks=ds[v].shape, dtype=ds[v].dtype)
        a[:] = ds[v].values
        attrs = {k: encode_zarr_attr_value(val) for k, val in ds[v].attrs.items()}
        attrs['_ARRAY_DIMENSIONS'] = ds[v].dims
        a.attrs.update(attrs)

%time naive_store(ds, LoggedFSStore(f'{SCRATCH_BUCKET}/air_temperature_naive.zarr'))

INFO:LoggedFSStore:__setitem__('.zgroup', 24 BYTES) - 126.42858 ms
INFO:LoggedFSStore:__contains__('.zarray') - 65.40666 ms
INFO:LoggedFSStore:__getitem__('.zgroup') -> 24 BYTES - 32.38567 ms
INFO:LoggedFSStore:__setitem__('.zattrs', 307 BYTES) - 190.39598 ms
INFO:LoggedFSStore:__contains__('.zarray') - 67.66061 ms
INFO:LoggedFSStore:__contains__('.zgroup') -> True - 40.53481 ms
INFO:LoggedFSStore:__contains__('lat/.zarray') - 86.53445 ms
INFO:LoggedFSStore:__contains__('lat/.zgroup') - 70.08551 ms
INFO:LoggedFSStore:__setitem__('lat/.zarray', 313 BYTES) - 136.86775 ms
INFO:LoggedFSStore:__getitem__('lat/.zarray') -> 313 BYTES - 33.09395 ms
INFO:LoggedFSStore:setitems({'lat/0': b'\x02\x013\x04d\x00\x) - 98.51932 ms
INFO:LoggedFSStore:__setitem__('lat/.zattrs', 159 BYTES) - 105.16701 ms
INFO:LoggedFSStore:__contains__('.zarray') - 73.29439 ms
INFO:LoggedFSStore:__contains__('.zgroup') -> True - 41.47805 ms
INFO:LoggedFSStore:__contains__('air/.zarray') - 70.57624 ms
INFO:LoggedFSStore:__contains__('air/.zgroup') - 76.24795 ms
INFO:LoggedFSStore:__setitem__('air/.zarray', 365 BYTES) - 161.10498 ms
INFO:LoggedFSStore:__getitem__('air/.zarray') -> 365 BYTES - 34.90320 ms
INFO:LoggedFSStore:setitems({'air/0.0.0': b'\x02\x01!\x04 %\) - 214.99064 ms
INFO:LoggedFSStore:__setitem__('air/.zattrs', 471 BYTES) - 116.65585 ms
INFO:LoggedFSStore:__contains__('.zarray') - 64.61574 ms
INFO:LoggedFSStore:__contains__('.zgroup') -> True - 43.48885 ms
INFO:LoggedFSStore:__contains__('lon/.zarray') - 78.02696 ms
INFO:LoggedFSStore:__contains__('lon/.zgroup') - 99.15074 ms
INFO:LoggedFSStore:__setitem__('lon/.zarray', 313 BYTES) - 152.40456 ms
INFO:LoggedFSStore:__getitem__('lon/.zarray') -> 313 BYTES - 33.64609 ms
INFO:LoggedFSStore:setitems({'lon/0': b'\x02\x011\x04\xd4\x0) - 60.73121 ms
INFO:LoggedFSStore:__setitem__('lon/.zattrs', 160 BYTES) - 135.19151 ms
INFO:LoggedFSStore:__contains__('.zarray') - 64.26887 ms
INFO:LoggedFSStore:__contains__('.zgroup') -> True - 44.14023 ms
INFO:LoggedFSStore:__contains__('time/.zarray') - 65.71949 ms
INFO:LoggedFSStore:__contains__('time/.zgroup') - 73.62353 ms
INFO:LoggedFSStore:__setitem__('time/.zarray', 319 BYTES) - 130.98015 ms
INFO:LoggedFSStore:__getitem__('time/.zarray') -> 319 BYTES - 35.68954 ms
INFO:LoggedFSStore:setitems({'time/0': b'\x02\x01!\x08@[\x00) - 93.36168 ms
INFO:LoggedFSStore:__setitem__('time/.zattrs', 105 BYTES) - 128.76703 ms
CPU times: user 434 ms, sys: 54.4 ms, total: 489 ms
Wall time: 3.59 s

This shows that Xarray is adding a few calls of its own, but mostly the problem exists at the Zarr level.

What does this all mean. It boils down to this
CPU times: user 571 ms, sys: 85.3 ms, total: 656 ms
Wall time: 5.31 s

The difference btw the 656ms of CPU time and the 5.31s of Wall time is time spent blocking on I/O. If we put all of these sequential I/O calls into an async event loop, I'm pretty sure we could eliminate most of that difference.
There is also absolutely no reason we need to be checking of .zarray and .zgroup over and over again...in series! 🤦  Nor do we need to be getting the data we have just set.

@tinaok & @guillaumeeb - this is how Pangeo has made progress over the past years: carefully identifying performance bottlenecks and working systematically to fix them upstream in the open source libraries. If you have the time and interest, I really encourage you to dig into the zarr source code and see if you can optimize this a bit.

Thanks so much @rabernat for all that.
Just wanted to say that I hope to do that soon, however I won't be available next week.

I am thinking something like
diff --git a/zarr/storage.py b/zarr/storage.py
index f5459990..0760d8d4 100644
--- a/zarr/storage.py
+++ b/zarr/storage.py
@@ -2858,7 +2858,11 @@ class ConsolidatedMetadataStore(Store):
         self.store = Store._ensure_store(store)

         # retrieve consolidated metadata
-        meta = json_loads(self.store[metadata_key])
+        try:
+            meta = json_loads(self.store[metadata_key])
+        except KeyError:
+            # new store for writing
+            meta = {'zarr_consolidated_format': 1}

         # check format of consolidated metadata
         consolidated_format = meta.get('zarr_consolidated_format', None)
@@ -2885,7 +2889,12 @@ class ConsolidatedMetadataStore(Store):
         raise ReadOnlyError()

     def __setitem__(self, key, value):
-        raise ReadOnlyError()
+        self.meta_store[key] = value
+
+    def commit(self):
+        out = {".zmetadata": self.meta_store}
+        out.update(self.meta_store)
+        del out["zarr_consolidated_format"]
+        self.store.setitems(out)

so that instead of calling consolidate_metadata, you use this store instead.

That's one option. However, I see it as a bit of a workaround for the fundamental problem that the store has to be created in this very sequential, procedural way.
What if Zarr had a fast path to create an entire hierarchy at once, i.e.
zarr.create_hierarchy(big_dict, store=store, mode="w")

and all the data were set using a single call to store.setitems(...)?
I feel like that would be a more elegant solution that would not require consolidated metadata. After all, the `aws s3' CLI test shows that it is possible to get all of this data in S3 (without consolidated metadata) in < 2s. Why shouldn't Zarr be able to do the same?

Certainly. I don't have an immediate evaluation as to the merits of adding top level functions to zarr versus making more versatile storage layers. They end up doing the same thing here. What does your big_dict contain?
setitems is (for now) only making any difference on FSStore compared to serial.

this very sequential, procedural way.

Some of this was also discussed in the async-zarr idea. That was about data access, possibly across multiple arrays (which zarr also can't normally do).
Note that fsspec instances also allow for "transactions" to defer finalising upload batches and make them semi-atomic. That wouldn't help with speed, but would with consistency, which any lazy/deferred collection of thing to write would need to worry about.

(ah, and my diff doesn't actually require consolidated metadata, I am merely hijacking the class, because it was already doing some of this work; any dict could be used as the metadata store for uploading in a batch)


Implement password policies to avoid weak passwords

Is your feature request related to a problem? Please describe.
The default spring security auth method does not have control set in password complexity. It is possible to add a user with a weak password.
Describe the solution you'd like
To add a secure password policy so that admin users must set complex passwords for users.

Hi @nobodyiam, is this feature completed? I'll add password validation to the user manage form and the backend api server. Verification specification: The password must contain uppercase and lowercase characters and digits, and the length is greater then or equals 8 characters.

@WillardHu This feature is not yet assigned, it'll be great if you could work on this! Shall I assign this issue to you?

@WillardHu This feature is not yet assigned, it'll be great if you could work on this! Shall I assign this issue to you?

Ok, thanks. @nobodyiam


Consolidate benchmarks

Follows on from https://github.com/apache/arrow-datafusion/pull/23.
There is now a single benchmark crate that can run against DataFusion or Ballista.
Integration tests pass.
Closes https://github.com/apache/arrow-datafusion/issues/6.

Codecov Report

Merging #34 (f212caa) into master (c365a4f) will increase coverage by 5.83%.
The diff coverage is 67.11%.

:exclamation: Current head f212caa differs from pull request most recent head 5b187fa. Consider uploading reports for the commit 5b187fa to get more accurate results

@@            Coverage Diff             @@
##           master      #34      +/-   ##
==========================================
+ Coverage   70.41%   76.24%   +5.83%
==========================================
  Files         123      134      +11
  Lines       21261    23023    +1762
==========================================
+ Hits        14970    17554    +2584
+ Misses       6291     5469     -822

Impacted Files
Coverage Δ

ballista/rust/client/src/context.rs
0.00% <0.00%> (ø)

ballista/rust/core/src/datasource.rs
0.00% <ø> (ø)

...lista/rust/core/src/serde/logical_plan/to_proto.rs
68.15% <0.00%> (+68.15%)
:arrow_up:

datafusion/src/optimizer/projection_push_down.rs
98.66% <ø> (ø)

datafusion/src/physical_plan/hash_utils.rs
100.00% <ø> (+2.89%)
:arrow_up:

benchmarks/src/bin/tpch.rs
35.15% <13.84%> (-3.19%)
:arrow_down:

datafusion/src/optimizer/constant_folding.rs
91.95% <33.33%> (-0.36%)
:arrow_down:

datafusion/src/logical_plan/plan.rs
82.01% <61.53%> (+1.62%)
:arrow_up:

datafusion/src/physical_plan/cross_join.rs
76.15% <76.15%> (ø)

datafusion/src/optimizer/hash_build_probe_order.rs
62.60% <88.88%> (+6.93%)
:arrow_up:

... and 63 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c365a4f...5b187fa. Read the comment docs.

@alamb @jorgecarleitao @Dandandan This is rebased and ready for review


jest-haste-map: Please update dependency on fsevents (currently 1.2.13)

🐛 Bug Report
npm WARN deprecated<EMAIL_ADDRESS>fsevents 1 will break on node v14+ and could be using insecure binaries. Upgrade to fsevents 2.
To Reproduce
$ npm install -D<EMAIL_ADDRESS>Expected behavior
No deprecated dependencies
Link to repl or repo (highly encouraged)

envinfo

It was upgraded in<EMAIL_ADDRESS>


Source Files in Linked Directories

see comment below for my point.

Using the readthedocs theme...
there are two issues for me:

If I place a tag in more than one source files: I still only see exactly one entry for this tag in the tagsindex.rst.
I do not see the tagsindex-toctree in the sidebar.

Any suggestions?

Hi Melissa (if that's your name).
I attached an very small example of what I think is unintended behaviour:
Expectations:

tag2 appearing 2 times in tag2.html
tag2 being counted twice in tagsindex.html

Observations:

tag2 appearing 1 time in tag2.html
tag2 being counted once in tagsindex.html

Tags.zip

Hi @kassbohm - thanks for opening the issue. You are right - let me try and investigate what is happening here, but looks like a bug.

Thanks for getting back. Take your time.

Hi @melissawm.
Should I use a workaround? Or will you adress this issue any time soon?

Hi @kassbohm - really sorry for the delay! Can you still reproduce this? I can't seem to anymore - I see tag2 being counted twice in _tagsindex.html and 2 pages listed under tag2.html. Let me know what you can observe!

Hello,
There is a directory called LinkedDir.
There is a difference between:

using this directory as is:
source/LinkedDir

using only a link to this directory, with the actual directory residing elsewhere, e.g.:
source/LinkedDir -> ../../LinkedDir/

Using 1) There are two files found with "tag2".
Using 2) There is only one file found with "tag2".

Got it! Let me try and see what's up. Thanks again!

Any news on this?

This solves the example I provided above. But I am afraid it does not solve the general problem with linked directories...
I have attached a new Tags.zip, which again leads to incorrect behaviour.
Tags.zip

Thanks - I list I have a clue now, will try to investigate!


Fix true being surrounded by [b] tags in documentation

Self-explanatory. Continuation of https://github.com/godotengine/godot/pull/95762, but I'd like to leave that untouched as it'll be merged later.

Thanks!


Why is tf.images.resize_bicubic different from misc.imresize with bicubic method

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): N
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Win 10
TensorFlow installed from (source or binary): binary
TensorFlow version (use command below): 1.6
Python version: 3.6
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version: 9.0  cudnn 7.0
GPU model and memory: GTX 1080ti
Exact command to reproduce: N
I consider the implement of tf.images.resize_bicubic is similar to opencv resize method, but it has problem.
The opencv resize has more noise and less soomth. If we use opencv to resize image and process image super-resolution, it will degrade the performance.
There are some visual results generated by tensorflow and scipy.misc:
tersorflow:

misc:

I wish the tensorflow can offer the implement of misc (or PIL, MATLAB)  resize method. Thanks!

Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks.
Have I written custom code
OS Platform and Distribution
TensorFlow installed from
TensorFlow version
Bazel version
CUDA/cuDNN version
GPU model and memory
Exact command to reproduce

@rmlarsen : Any idea why this doesn't match opencv, and whether we're open to a change in that direction?

Nagging Assignee @rmlarsen: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @rmlarsen: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @rmlarsen: It has been 45 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @rmlarsen: It has been 60 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @rmlarsen: It has been 77 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @rmlarsen: It has been 92 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @rmlarsen: It has been 107 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @rmlarsen: It has been 122 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @rmlarsen: It has been 137 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @rmlarsen: It has been 153 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @rmlarsen: It has been 168 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @rmlarsen: It has been 183 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @rmlarsen: It has been 198 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @rmlarsen: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @rmlarsen: It has been 29 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

This sounds more like a bug than a feature request.
tf.images.resize_bicubic is no longer supported in TensorFlow, and scipy.misc.imresize has been deprecated. Can someone reproduce with tf.image.resize(..., method=bicubic), and confirm that there is still a discrepancy with OpenCV?

@dynamicwebpaige Is tf.image.resize(..., method=bicubic) or any other interpolators intended to behaviour exactly the same as OpenCV?
As we know OpenCV's resize behaves differently with PIL or matlab. For a researcher's point of view, it's more likely to use PIL or matlab's image functions rather than those functions provided in TF or OpenCV.
Moreover, align_corners also affects the image quality and the output image is not actually "aligned" to the ground truth.

@linchuming We are checking to see if you still need help on this issue, as you are using an older version of tensorflow(1.x) which is officially considered as end of life. We recommend that you upgrade to 2.6 which is latest stable version of TF and let us know if the issue still persists in newer versions.Thanks!


[Lake] Add missing status include

What type of PR is this：

[x] bug
[ ] feature
[ ] enhancement
[ ] refactor
[ ] others

Which issues of this PR fixes ：

Fixes #
Problem Summary(Required) ：

run starrocks_admit_test


NameError: name 'ensure_punkt' is not defined

Hi again,
Getting the following error after pulling the most recent changes.
As such, I've reverted to the previous commit for now.
Is there something special I have to do, as punkt is already downloaded in my case?
Traceback (most recent call last):
File "C:\Users\daniel\Documents\epub2tts\epub2tts.py", line 419, in
main()
File "C:\Users\daniel\Documents\epub2tts\epub2tts.py", line 408, in main
mybook = EpubToAudiobook(source=args.sourcefile, start=args.start, end=args.end, skiplinks=args.skiplinks, engine=args.engine, minratio=args.minratio, model_name=args.model, debug=args.debug)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\daniel\Documents\epub2tts\epub2tts.py", line 67, in init
ensure_punkt()
^^^^^^^^^^^^
NameError: name 'ensure_punkt' is not defined

Ooof embarrassing!  Thanks for catching that, it should be fixed in main now.
Thanks again for letting me know!

No problem at all, cheers for fixing so quickly.


RE4: FoV option breaks cutscenes, attaché case, save menu, and others

Not sure if this is the intended behavior or not, but using the FoV override setting doesn't seem to override the value set in the game's settings, it instead just applies globally to the main camera.  This has the side effect of overriding the times when the game manually adjusts the Fov.  Some examples i've seen:

Cutscenes where the camera would be doing a close-up are shown from a wider FoV and this often leads to unintended things happening like parts of the level being invisible.  For example, Ashley is sitting on a bed at one point and as the camera angle changes, the bed disappears and re-appears each time it changes (pretty sure the times where it is invisible, the camera is supposed to be close enough where you wouldn't see it happening)
Sections in the game that have a binocular prompt don't work at all since the zoom function seems to be tied to the FoV.  It isn't game-breaking but you end up missing whatever it is that the game was supposed to be zooming into.
Some of the "press A to activate" overlays seem to be off-set a bit.  Not 100% sure if this is caused by the FoV change, but the only things I had enabled were DLSS and FoV.
The attaché case and the save menu appear to have a custom FoV as well, so if you set it higher than normal, it takes up less of the screen and sort of "pops" when you close it.

I found another mod on nexusmods called "Simple FOV for RE4 Remake" which is just a cheat-engine table that apparently doesn't have these issues (I'm guessing it's just affecting whatever value is changed in the settings menu).  Not sure if it's helpful but I wanted to point that out in case it was.

Use the ultrawide option to adjust FOV, it will work like you expect.

Ok, i'll try that.  Thanks!


Translate heading and subheading of setup/independent/high-availability/ in Japanese

This is a...

[x]  Feature Request
[ ]  Bug Report

Problem:
No heading and subheading of setup/independent/high-availability/ in Japanese.
Proposed Solution:
Translate heading and subheading of setup/independent/high-availability/ in Japanese.
Copy from en content and Update content/ja/docs/setup/independent/high-availability.md.
Page to Update:
https://kubernetes.io/ja/setup/independent/high-availability/

/assign

/close


Reusing TST files across multiple web projects

Hi,
just another question about Typewriter. I have created a piece of logic for serializing C# to TS typings in a single web project. However, the architecture I'm using utilizes plenty of web projects that form layers.
The core layer web project introduces some models that are used in the upper-layer web projects in server-side code and pretty much the same applies to TS code.
I know it's possible to reference .d.ts from core web project in upper web projects and that's cool. However, I'd like to be agile in reusing/referencing the TST files so that the same logic drives generating TS. Is it possible to achieve having single core TST implementation and reusing that in upper-layer web projects for generating its own TS files?

hi @rodro75 do you plan to create a Pr with your changes?

hi @agustinbcu01 , there is still work to do on both branches before they can be incorporated into master I think. I'll see what I can do, because I've been pretty busy on my regular job lately, but I will definitely try to finish either TemplateInclusion or AssemblyRef, or both if possibile, as I also need them for my project at work.

hi @agustinbcu01 , I've just submitted a PR for the AssemblyRef branch.

Nice.

On Jun 27, 2018, at 10:42 AM, Rodrigo<EMAIL_ADDRESS>wrote:
hi @agustinbcu01 , I've just submitted a PR for the AssemblyRef branch.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.


VMREAD RAX error

0: kd>  .scriptrun  "C:\dl\hvext.js"
JavaScript script successfully loaded from 'C:\dl\hvext.js'
"C:\dl\hvext.js" (line 46 (@ 9)): Error (0x80004005): No VMREAD RAX, RAX (0f 78 c0) found.
JavaScript script 'C:\dl\hvext.js' failed to execute main function!

Please post full output of vertarget on the session.

Going to close this because of inactivity. Free free to reopen as needed.


ShopifyAPI::DraftOrder where customer_id

Issue summary
So currently there is no way to query draft orders for specific customer I tried e.g.
where(customer_id: id)
and
where(customer: { id: id })
Expected behavior
It's possible to perform such a query
Actual behavior
It ignores this condition and returns full list
Steps to reproduce the problem

Create draft order for specific customer
Query draft orders with customer id
See result

Specifications

shopify_api version: 9.3.0
activeresource version: 6.1.3
Shopify API version used (example: 2020-07): 2020-10

So currently there is only status available for filtering which helps to reduce the scope but still, currently we need to while loop by 250 each time and then manually do Array#select which is definitely not a production ready solution

Unfortunately this is not something that the REST API supports. An alternative would be to look into what GraphQL allows.


GFG_Vowel_or_Not in c++

Hey! @aman247av
I would like to add a program of GFG checking vowel or not in C++
kindly assign me this issue

Sure go ahead.. assigning it to you... follow the README.md for instructions


Widgets not working

Seems to be the endpoints are not executing, especially the one to get access-token.
So widgets are just blank.

Have tried both with query param and path. Current theme is built on Elementor.

The /bcc-login/access-token endpoint often works after saving "permalinks" configuration in WP. Does that help?

Yes have tried that. Didn't help

Saw now there was an even newer version: 1.1.122 which works!
Thanks


Provide an example of usage on real project

Given a single library, say...  Symfony Console component, how phan should be used to perform code analysis correctly?
Analysing a single or a plain list of files is quiet easy to understand but when a huge project with autoloader managed by things like compose - it is hard to manually prepare file list with correct analys order

A how-to wiki page would make a lot of sense to me.

I hope we will not need the whole wiki page to scan a single project :)

I just added the wiki article https://github.com/etsy/phan/wiki/Tutorial-for-Analyzing-a-Large-Sloppy-Code-Base - I'm hoping it'll mature over time as more folks start doing real-world analyses.

Also, despite this being closed, it'd still be great to see an example of an actual project rather than tips for running on any project.

Would a good way of doing this be something like, linking to a pull request (or just a patch on a fork), and having that patch have all of the things needed to do to get phan up and running? And then link to these patches from that wiki page as concrete examples of what people are doing and how it works.

I hope we'll finish in running phan just like phan analyze -r src/ -r other_src/ -e vendor/ -p -o phan.out (and some other options for tune). I've already started some works on it using symfony console and other components to handle file aggregation process not involving bash.

I just added the -l CLI option for recursively scanning a directory for PHP files. https://github.com/etsy/phan/commit/5ac189326217ede8e9d1c57d1d8e27b086373889
Let me know if I can help with the Symfony console stuff.

@morria I can open a PR to get your thoughts on it. There is many things to do, as currently Cli is statically called among many classes (alongside the Log class)


Turbo frames events

Would be really helpful to add turbo frames events just like turbo drive has with the frame_id as a payload for that event.
document.addEventListener('turbo-frame:load', (event) => console.log('loaded frame ->', event.turboFrameId))

Would this be something helpful to explore?

Would https://github.com/hotwired/turbo/pull/59 address this? It doesn't introduce turbo-frame:: prefixed events or include the turboFrameId specifically, but it does include the frame as the target, so you could read it from there.

Would https://github.com/hotwired/turbo/pull/59 address this? It doesn't introduce turbo-frame:: prefixed events or include the turboFrameId specifically, but it does include the frame as the target, so you could read it from there.

I'd also go so far as to say that it would also be useful to have events that notify when Turbo streams modify a frame, even just something like a turbo-stream:received event so that a Stimulus controller can react to it (I.e.like incrementing the counter of replies on a post).
I'm currently using MutationObservers to achieve this, but I wonder whether it would be more lightweight/performant to have the event fired directly by Turbo.

I'd also go so far as to say that it would also be useful to have events that notify when Turbo streams modify a frame, even just something like a turbo-stream:received event so that a Stimulus controller can react to it (I.e.like incrementing the counter of replies on a post).
I'm currently using MutationObservers to achieve this, but I wonder whether it would be more lightweight/performant to have the event fired directly by Turbo.

@Sub-Xaero there is a turbo:before-stream-render that fires before rendering a <turbo-stream> element. According to past discussions, it is unlikely that there will be any other Stream events added in the future.

I wonder whether it would be more lightweight/performant to have the event fired directly by Turbo.

While the MutationObserver interface can be awkward to use at times, I'm not aware of any performance risks involved in using them. As I understand it, Stimulus is powered by tens (or hundreds?) of concurrently attached MutationObservers.

@Sub-Xaero there is a turbo:before-stream-render that fires before rendering a <turbo-stream> element. According to past discussions, it is unlikely that there will be any other Stream events added in the future.

I wonder whether it would be more lightweight/performant to have the event fired directly by Turbo.

While the MutationObserver interface can be awkward to use at times, I'm not aware of any performance risks involved in using them. As I understand it, Stimulus is powered by tens (or hundreds?) of concurrently attached MutationObservers.

@seanpdoyle Indeed, thank you, I had missed that. My only issue with that is that it fires at the document level, and the srcElement, target and path attributes all point to an element of type <turbo-stream> that I can only assume is either in the closed-mode ShadowDom of the <turbo-cable-stream-source> element or a dead reference to the <turbo-stream> that the <turbo-cable-stream-source> replaces as according to Chrome, Node cannot be found in the current page., so you cannot tell where the event originated.
I had imagined an event being fired on the frame itself and bubbling. But, following the precedent set in https://discuss.hotwire.dev/t/event-to-know-a-turbo-stream-has-been-rendered/1554/6 , it is probably a better convention to use MutationObservers as then it doesn't matter where the elements in the DOM come from.
Thanks for your help, I'll carry on my merry way.

@seanpdoyle Indeed, thank you, I had missed that. My only issue with that is that it fires at the document level, and the srcElement, target and path attributes all point to an element of type <turbo-stream> that I can only assume is either in the closed-mode ShadowDom of the <turbo-cable-stream-source> element or a dead reference to the <turbo-stream> that the <turbo-cable-stream-source> replaces as according to Chrome, Node cannot be found in the current page., so you cannot tell where the event originated.
I had imagined an event being fired on the frame itself and bubbling. But, following the precedent set in https://discuss.hotwire.dev/t/event-to-know-a-turbo-stream-has-been-rendered/1554/6 , it is probably a better convention to use MutationObservers as then it doesn't matter where the elements in the DOM come from.
Thanks for your help, I'll carry on my merry way.

Closed via https://github.com/hotwired/turbo/pull/59


build: introduce react-icons

This introduces react icons package and resolves #9.

adding react-icons library to the package.json does not create any effect as it is not used anywhere in the project.

I wasn't sure if the icons can already be used somewhere, can they be?

If it was used, then the library had been included. When we use any library then only we include it otherwise it affects performance of website.

There was an issue open for this, so I would assume the icons would be used very soon.

Okay


taglist: load taglist variants only once

Because Tlist_Get_Filenames() only exists within yegappan/taglist,
TlistUpdate was being envoked on every call when using other taglist
versions.(Like vim-scripts/taglist.) This caused visual artifacts and
slowdowns.
We now check for a function from both taglist variants to see if either
one has been loaded.
fixes vim-airline/#2470

thanks!


Request to add  Height property for text area component.

The form text area used in my application, has a little large content, so i want to display atleast  sizeable content without manually resizing it.

:tada: This issue has been resolved in version 1.7.1 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Stack querying should take role refinement into account

Either need a new way to query an OR relationships over the set of Roles, or need to do several queries for a Role and all its descendants.

This should work now.

Please can you make a new GitHub release?
I was demoing Designer today and couldn't find a jar for the latest version.
Sent from my iPhone
On Dec 8, 2016, at 19:53, Michael Zhang <notifications<EMAIL_ADDRESS>wrote:
This should work now.

You are receiving this because you were assigned.
Reply to this email directly, view it on GitHubhttps://github.com/SynBioDex/SBOLDesigner/issues/92#issuecomment-265837654, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AAbw75XT3Z776Qn8RZU3suGPq7dzFMLGks5rGGAYgaJpZM4KinhF.

We may not want an official release quite yet, so I’ve added a SNAPSHOT jar to github.  It is called SBOLDesigner-2.2-SNAPSHOT.jar
James: please grab this one and confirm that it works for you.
Chris

On Dec 8, 2016, at 1:40 PM, James Alastair McLaughlin<EMAIL_ADDRESS>wrote:
Please can you make a new GitHub release?
I was demoing Designer today and couldn't find a jar for the latest version.
Sent from my iPhone
On Dec 8, 2016, at 19:53, Michael Zhang <notifications<EMAIL_ADDRESS>wrote:
This should work now.

You are receiving this because you were assigned.
Reply to this email directly, view it on GitHubhttps://github.com/SynBioDex/SBOLDesigner/issues/92#issuecomment-265837654, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AAbw75XT3Z776Qn8RZU3suGPq7dzFMLGks5rGGAYgaJpZM4KinhF.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub https://github.com/SynBioDex/SBOLDesigner/issues/92#issuecomment-265849190, or mute the thread https://github.com/notifications/unsubscribe-auth/ADWD91C9rGG6P1F9nzSpoMCDIkwtmLUPks5rGGtDgaJpZM4KinhF.


urlopen error timed out

现在很容易爆超时错啊, 你们怎么处理的, 如果大量更新数据
是get_k_data报错

You can try to set retry_count more than 3. Such as
import tushare as ts
ts.get_k_data("000001", retry_count=10).
Another possibility is you call get_k_data in multi-process at the same time. Then the gtimg.cn might reject some requests if too many requests from the same IP address.

@bombhero thanks , that's a way to solve this.


feat: dimensions in the metric API

Implement dimensions in the metric API (the next PR will implement in the SQL API as well).
The way this works is simple. We annotate a column with the dimensions attribute, lke the user_id column in the core.comments node:
https://github.com/DataJunction/datajunction/blob/7ef9fc64607d5f9161bd5cbf4ebf46354916cbc9/examples/configs/nodes/core/comments.yaml#L1-L8
The metric core.num_comments now has more dimensions (from a curl http://localhost:8000/metrics/):
{
  "id": 3,
  "name": "core.num_comments",
  "description": "Number of comments",
  "created_at": "2022-04-15T00:53:54.852111",
  "updated_at": "2022-04-15T00:53:54.852119",
  "expression": "SELECT COUNT(*) FROM core.comments",
  "dimensions": [
    "core.comments.__time",
    "core.comments.count",
    "core.comments.id",
    "core.comments.text",
    "core.comments.timestamp",
    "core.comments.user_id",
    "core.users.age",
    "core.users.country",
    "core.users.full_name",
    "core.users.gender",
    "core.users.id",
    "core.users.preferred_language",
    "core.users.secret_number"
  ]
}

Note that all the columns of the core.users dimension are now dimensions of core.num_comments, in addition to the dimensions that come from the parent node (core.comments).
Now in the metric API we can filter and group by these dimensions. For example, this query:
% curl "http://localhost:8000/metrics/3/data/?d=core.users.gender&f=core.users.age>25"

Returns:
{
  "database_id": 3,
  "catalog": null,
  "schema": null,
  "id": "260d7b86-0261-426a-a525-e846004c4eed",
  "submitted_query": "SELECT count('*') AS count_1, \"core.users\".gender \nFROM (SELECT public.comments.id AS id, public.comments.user_id AS user_id, public.comments.timestamp AS timestamp, public.comments.text AS text \nFROM public.comments) AS \"core.comments\" JOIN (SELECT public.dim_users.id AS id, public.dim_users.full_name AS full_name, public.dim_users.age AS age, public.dim_users.country AS country, public.dim_users.gender AS gender, public.dim_users.preferred_language AS preferred_language \nFROM public.dim_users) AS \"core.users\" ON \"core.comments\".user_id = \"core.users\".id \nWHERE \"core.users\".age > 25 GROUP BY \"core.users\".gender",
  "executed_query": "SELECT count('*') AS count_1, \"core.users\".gender \nFROM (SELECT public.comments.id AS id, public.comments.user_id AS user_id, public.comments.timestamp AS timestamp, public.comments.text AS text \nFROM public.comments) AS \"core.comments\" JOIN (SELECT public.dim_users.id AS id, public.dim_users.full_name AS full_name, public.dim_users.age AS age, public.dim_users.country AS country, public.dim_users.gender AS gender, public.dim_users.preferred_language AS preferred_language \nFROM public.dim_users) AS \"core.users\" ON \"core.comments\".user_id = \"core.users\".id \nWHERE \"core.users\".age > 25 GROUP BY \"core.users\".gender",
  "scheduled": "2022-04-16T04:10:31.983073",
  "started": "2022-04-16T04:10:31.983234",
  "finished": "2022-04-16T04:10:32.076099",
  "state": "FINISHED",
  "progress": 1,
  "results": [
    {
      "sql": "SELECT count('*') AS count_1, \"core.users\".gender \nFROM (SELECT public.comments.id AS id, public.comments.user_id AS user_id, public.comments.timestamp AS timestamp, public.comments.text AS text \nFROM public.comments) AS \"core.comments\" JOIN (SELECT public.dim_users.id AS id, public.dim_users.full_name AS full_name, public.dim_users.age AS age, public.dim_users.country AS country, public.dim_users.gender AS gender, public.dim_users.preferred_language AS preferred_language \nFROM public.dim_users) AS \"core.users\" ON \"core.comments\".user_id = \"core.users\".id \nWHERE \"core.users\".age > 25 GROUP BY \"core.users\".gender",
      "columns": [
        {
          "name": "count_1",
          "type": "FLOAT"
        },
        {
          "name": "gender",
          "type": "STR"
        }
      ],
      "rows": [
        [
          6,
          "non-binary"
        ],
        [
          5,
          "male"
        ]
      ],
      "row_count": 2
    }
  ],
  "next": null,
  "previous": null,
  "errors": []
}

Note the query that was generated:
SELECT
  count('*') AS count_1,
  "core.users".gender
FROM (
  SELECT
    public.comments.id AS id,
    public.comments.user_id AS user_id,
    public.comments.timestamp AS timestamp,
    public.comments.text AS text
  FROM
    public.comments) AS "core.comments"
  JOIN (
    SELECT
      public.dim_users.id AS id,
      public.dim_users.full_name AS full_name,
      public.dim_users.age AS age,
      public.dim_users.country AS country,
      public.dim_users.gender AS gender,
      public.dim_users.preferred_language AS preferred_language
    FROM
      public.dim_users) AS "core.users" ON "core.comments".user_id = "core.users".id
WHERE
  "core.users".age > 25
GROUP BY
  "core.users".gender "

@agorajek this still needs more tests and some clean-up, but it's the idea I have for dimensions. There's no concept of unit_id, instead we simply add foreign keys defining 1:1 or n:1 relationships, and join on that key.

Beto, this lgtm. Have you considered adding/adjusting the ./docs in the same PRs? Otherwise adding them after the fact may be a tough project.

Yeah, that would be ideal... right now docs/ is empty and I have a ticket to write them (https://github.com/DataJunction/datajunction/issues/68). I've been focusing on code (and tests) at the moment to move faster, but I think I might start working on those docs. And once we have them it's easier to keep the documentation up to date.


Normalize namespaces

WIP

You've gotta normalize namespaces on all XAML files e.g. https://github.com/vasily-kirichenko/FSharpVSPowerTools/blob/c3affcbdccff2321fe13ebe6d69874462585f956/src/FSharp.Editing.VisualStudio/FolderNameDialog.xaml#L5-L19; otherwise, none of the XAML dialogs would work.

@dungpa regarding XAML (which I don't know anything of) any clue if Resharper would catch that as error? I can give a look at what Resharper says.

I'm not sure ReSharper works on XAML files in F# projects. You could give it a try.

From what I've seen, you'll have to hand-edit the xaml files to fix the
namespaces (and assembly names) in the headers.
On Thu, Jul 14, 2016 at 2:08 PM, Anh-Dung Phan<EMAIL_ADDRESS>wrote:

I'm not sure ReSharper works on XAML files in F# projects. You could give
it a try.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/fsprojects/VisualFSharpPowerTools/pull/1422#issuecomment-232793031,
or mute the thread
https://github.com/notifications/unsubscribe/AFCUmtkxgwtACWdLXSs_IELg-fKCHTfkks5qVqVggaJpZM4JMywS
.

Resharper at the rescue, it picks (at least the assembly) errors up.
@vasily-kirichenko do you want me to make a PR on your branch?

It also completes on namespaces it seems:

Yes, and if you remove the entire xmlns line, then go to the first error,
it should pick up what needs to be added and prompt to add it.  Still
needs to manually check each xaml, but it does help.
On Thu, Jul 14, 2016 at 3:02 PM, Gauthier Segay<EMAIL_ADDRESS>wrote:

It also completes on namespaces it seems:
[image: image]
https://cloud.githubusercontent.com/assets/87944/16857336/6aaf095e-4a1f-11e6-9e43-f30a2c51a2fc.png
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/fsprojects/VisualFSharpPowerTools/pull/1422#issuecomment-232805862,
or mute the thread
https://github.com/notifications/unsubscribe/AFCUmrQRFZl92r23O7y-2B9kt90zgmjcks5qVrH_gaJpZM4JMywS
.

https://github.com/vasily-kirichenko/FSharpVSPowerTools/pull/8

It's ready.

Great. Thanks.


Discussion: implement line separator service

I found this feature (turned off by default) in C# editor:

I'm not sure it's worth implementing though. Ideas?

I think that if it were to be implemented, it would likely be low priority.  I think it has less benefit to F# than to C#/VB (turned on by default in VB, actually) because F# is a lot less verbose, but I could see someone liking it.  I certainly wouldn't vote to reject a PR that implemented it.

I think there should be a setting to turn it off, which faces us with the old problem: we don't have a good settings dialog page.

Yeah, true.  In that case I think holding off on something like this is appropriate for now.  Once we have a good settings dialog with some ability to configure and tweak IDE features, this seems like something worth implementing.

👍
As usual, I think we should steal the dialog from C# editor (not it's content as is, but the way it works and the design).

Agreed!

I think the dialog is high priority. I'm already confused by all this aggressive code folding and block structure noise. It's really scary how much noise C#ers use. @vasily-kirichenko please steal the dialog and then we should discuss which things are really worth to be turned on by default

Just for the record: I'm not saying these things are not awesome new features. It's just that I think we should be really conservative with turning these on by default. 4 spaces indent is usually already a great visualization.

Unfortunately, as VisualFSharpOpenSource.vsix does not work for VFT solution, I don't use current muster at all and I don't have any opinions on what features should be turn on by default.
However, turning off features by default is somewhat dangerous because the vast majority of users never go to settings to examine what's available. That's the case with coloring in VFPT - I very rarely see that people turns any colorization except the default one (types only), which is unfortunate.
So I think all the features should be turned on by default. Maybe we should provide "tips" via that new annoying yellow bar, like "Do you know that you can turn the block structure helper in settings?".

Yeah that is a tough decision. I don't claim I have the perfect answer.

@forki I think the dialog isn't quite high priority (although I'd say it's a higher priority than this specific feature).  One of the awesome things about VS 2017 from an infrastructure standpoint is that the internal changes were made to allow for a much more rapid cadence for releases, allowing us to gradually move away from gigantic updates that change hundreds of things and potentially break people, and move more towards smaller updates that are more focused in certain areas.  They also don't have to be monolithic anymore - one Workload may be completely unaffected by an update to tools in another Workload, for example.  What that means is we're free to iterate a bit more rapidly (or slowly) and not be burdened with getting bundled up with a lot of unrelated stuff.
I think that because there's a lot of attention from the community on getting features from VFPT and trying to achieve "experience parity" with C#/VB (where it makes sense), I think it's best to focus on getting as many features in there that we can - provided that they're tested and aren't totally broken, of course.
If a dialog could get made that's simply some checkboxes, like VFPT, and done so in a week or two, I'd more ecstatic to have it in.  But I agree with @vasily-kirichenko that we should default to these new features being on, with the only exception being if a particular feature is really bad performance-wise compared to the rest.  Put another way, I'd much rather people complain about there being too many IDE features for F# than too little, and dial features back based on feedback if necessary.

I think there must be a way to disable features. There is no way around this since features can always be buggy in certain scenarios and we must give people a way out. Otherwise it will be a nightmare. (it can be manually editing regedit or worse but there has to be a way out)
some IDE features from C# just don't make any sense. We need to be careful

I can't think of many features that C#/VB have which don't make sense for F#, excluding language feature Analyzers (like suggesting the new is pattern over var y = x as T; if (x != null)).  Certainly any feature that makes working with .NET libraries and various .NET-isms easier we'll want to bring forward, and things like completion without having to hit ctrl-space are also great.
But I completely agree that we need to have a way to toggle things off and on, and moving forward, a way to tune them (like if we implemented predictive completions, being able to dial how "smart" it tries to be).
I think that block structuring is one of the more controversial features since it does add more noise to the IDE.  If we can hammer out a few more bugs in it, then I think we'll be able to evaluate if on or off by default is the best.  Right now, there are enough bugs there where I think turning off by default is the best choice.

I'm much happier with there being too many features [on by default] than too little features.

But only if they are all of sufficiently high quality!

My worry is that we now have very little control on toggle features on/off since all service initialization is done in Roslyn. For example, for https://github.com/Microsoft/visualfsharp/issues/1800, we even need some code change in Roslyn.
@Pilchie @CyrusNajmabadi Could you give some pointers on how Roslyn handle feature toggling via Text Editor options? Thanks.

@dungpa - we map text editor options to Roslyn options here


Lat couple of edits

@yujhongmicrosoft,
Thanks for your contribution as a Microsoft full-time employee or intern. You do not need to sign a CLA.
Thanks,
Microsoft Pull Request Bot

@yujhongmicrosoft : Thanks for your contribution to the Azure documentation! The author, @smichelotti, has been notified to review your proposed change.


fixed getting started example in book for version 0.3.0 and eos version 2.0

Change Description
Adapted the getting started example to work with the new version 0.3.0 and eos version 2.0
API Changes

[ ] API Changes

Documentation Additions

[x] Documentation Additions

Thanks!


Is there a separate library/api to instantiate functions in expressions?

Hello experts,
May I know if there is a separate library/api of azure logic app that we could use to instantiate functions in expressions?
We're using logic app for scheduling in our products, and would love to keep the same expression resolving behavior beyond scheduling scenario.
Below is an example to demonstrate two scenarios, and my question is not limited to the function itself.

scheduling scenario: in below workflow, we use function @addToTime(@utcNow(), 1, 'Day') for scheduling HTTP request.
single request without logic app: we have to implement the function resolving logic from scratch. This also makes the two scenarios different.

{
  'location': '{LogicAppRegion}',
  'properties': {
    'state': '{LogicAppState}',
    'definition': {
      '$schema': 'https://schema.management.azure.com/providers/Microsoft.Logic/schemas/2016-06-01/workflowdefinition.json#',
      'actions': {
        'HTTP': {
          'inputs': {
            'body': {"ParameterAssignments": {"time": "@addToTime(@utcNow(), 1, 'Day')"}},
            'method': 'POST',
            'uri': '{LogicAppUri}'
          },
          'runAfter': { },
          'type': 'Http'
        }
      },
      'contentVersion': '<IP_ADDRESS>',
      'outputs': { },
      'triggers': {
        'Recurrence': {
          'recurrence': {recurrence},
          'type': 'Recurrence'
        }
      }
    }
  }
}

Thanks in advance for your help! Also asked about the question on stackoverflow.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 03aa9e88-7fa0-56ab-f794-58f0c37b6d7a
Version Independent ID: 25c5ef5c-6ba4-f5ff-f94b-8f2818ec2feb
Content: Reference guide for functions in expressions - Azure Logic Apps
Content Source: articles/logic-apps/workflow-definition-language-functions-reference.md
Service: logic-apps
GitHub Login: @ecfan
Microsoft Alias: estfan

Thanks for the feedback! We are currently investigating and will update you shortly.

@chjinche Unfortunately, there is no library or API for you to leverage the same expressions outside of Logic Apps. For the exact scenario that you posed, you could approach it in the following way with 2 Logic Apps as follows

The first Logic App would be a HTTP Triggered one which would call the HTTP Action with the expression that you shared above
The second Logic App would use the recurrence trigger as you've shared but instead call the first Logic App

This way, the first Logic App can be called directly as well with the scheduled Logic App also triggered the first, thereby using the same expression.
Nonetheless, please feel free to open a feature request to expose the expression language outside of Logic Apps here.
Since there is no doc update required here now, we will now proceed to close this thread. If there are further questions regarding this matter, please tag me in your reply. We will gladly continue the discussion and we will reopen the issue.


Minor grammatical typo in capabilities response

smartthings capabilities
Error: could not find any namespaces for you account. Perhaps you haven't created any capabilities yet.
should read: "for your account"

This text is no longer used.


docs: update stacks.js pages

updates to Stacks.js pages
adds code tabs, will add more of these (e.g. npm VS yarn; stacks.js VS connect)
removes legacy pages

Reference:


Image recognition

Hi,
I'm using beyondAR to create a world and display some GeoObjects, and everything works just fine.
I'd like to know if it has the capability of image recognition, i.e. if I point a camera to a building, for which I have stored a picture of in my App, it can recognize it and display another picture on top of it. Not sure if it's clear what I'm trying to achieve.
Any help would be very appreciated.
Thank you

Unfortunately there is no image recognition in the SDK, you could use vuforia sdk for this


date_trunc function when connect to hive

Make sure these boxes are checked before submitting your issue - thank you!

[x] I have checked the superset logs for python stacktraces and included it here as text if there are any.
[x] I have reproduced the issue with at least the latest released version of superset.
[x] I have checked the issue tracker for the same issue and I haven't found one similar.

Superset version
0.28.1
Expected results
Actual results
errorCode=10011, errorMessage="Error while compiling statement: FAILED: SemanticException [Error 10011]: Line 6:9 Invalid function 'date_trunc'"), operationHandle=None)
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/superset/connectors/sqla/models.py", line 783, in query
    df = self.database.get_df(sql, self.schema)
  File "/usr/lib/python3.6/site-packages/superset/models/core.py", line 789, in get_df
    self.db_engine_spec.execute(cursor, sqls[-1])
  File "/usr/lib/python3.6/site-packages/superset/db_engine_specs.py", line 1268, in execute
    cursor.execute(query, **kwargs)
  File "/usr/lib/python3.6/site-packages/pyhive/hive.py", line 365, in execute
    _check_status(response)
  File "/usr/lib/python3.6/site-packages/pyhive/hive.py", line 495, in _check_status
    raise OperationalError(response)

Steps to reproduce
When I create time series bar chart, Superset uses 'date_trunc' function in hiveql group by clause and it raise exception.

Same here. I've checked the source code in incubator-superset/superset/db_engine_specs.py, from what I'm seeing, the reason this error happened is because they're using PrestoEngineSpec as HiveEngineSpec, as the code below:
`
class HiveEngineSpec(PrestoEngineSpec):
"""Reuses PrestoEngineSpec functionality."""

and hive doesn't support 'date_trunc' method. So if u can build from the source code, u can just change the code into some thing like this:
class HiveEngineSpec(BaseEngineSpec):
"""Reuses PrestoEngineSpec functionality."""

engine = 'hive'
max_column_name_length = 767

time_grain_functions = {
    None: '{col}',
    'PT1S': "from_unixtime(unix_timestamp({col}), 'yyyy-MM-dd HH:mm:ss')",
    'PT1M': "from_unixtime(unix_timestamp({col}), 'yyyy-MM-dd HH:mm:00')",
    'PT1H': "from_unixtime(unix_timestamp({col}), 'yyyy-MM-dd HH:00:00')",
    'P1D': "from_unixtime(unix_timestamp({col}), 'yyyy-MM-dd 00:00:00')",
    'P1W': "date_format(date_sub(nianxian, CAST(7-from_unixtime(unix_timestamp(nianxian),'u') as int)), 'yyyy-MM-dd 00:00:00')",
    'P1M': "from_unixtime(unix_timestamp({col}), 'yyyy-MM-01 00:00:00')",
    'P0.25Y': "date_format(add_months(trunc(nianxian, 'MM'), -(month(nianxian)-1)%3), 'yyyy-MM-dd 00:00:00')",
    'P1Y': "from_unixtime(unix_timestamp({col}), 'yyyy-01-01 00:00:00')",
    'P1W/1970-01-03T00:00:00Z': "date_format(date_add(nianxian, INT(6-from_unixtime(unix_timestamp(nianxian), 'u'))), 'yyyy-MM-dd 00:00:00')",
    '1969-12-28T00:00:00Z/P1W': "date_format(date_add(nianxian, -INT(from_unixtime(unix_timestamp(nianxian), 'u'))), 'yyyy-MM-dd 00:00:00')"
}

`
this should do the work.

@jeepxiaozi
thank for your reply :)

@jeepxiaozi
thank for your reply :)

sorry about the code part, I've edited a little bit so that u can see it clearly

@jeepxiaozi thank you for the fix! Could you create a pull request to the apache project please? :)

+1 for this one ;)

+1 for this one ;)

Remember you can override time grain functions in superset_config.py (see examples) without having to edit source and rebuild!


Delete PR directory

Github picks up the PR template from .github, not from this folder. Deleting this here.

Protecting branches that aren't main or develop is inconvenient to any developer. Once a PR is opened, I cannot add any more commits until the build passes and an approving review is given. This is extremely bad because if my PR had a bug that causes the tests to fail, I will never be able to update the PR and have no option but to close and open a new one.
I can confirm this is also a recent behavior change since the branch dual_proxy is not protected, but all the new branches with PR are. Please remove the rule/hook that causes this!

The name "maintenance" matches the regex for "main", hence the block :-)
If you can figure out a way to write a regex that protects only main and develop, please go ahead and change the regex.

Ah, yes, will do! I had not realized that the issue was in naming the branch. Just an exact match should be fine instead of pattern matching.


Added Description to Readme

Added description to let others understand the project more efficiently.

Thanks, but the description does not add more information in my opinion.


Unable to get geometry details in response

Describe the bug
Trying to fetch geometry details, but not receiving them.
GooglePlacesDetailsQuery prop has the value of fields: "geometry"
Reproduction - (required - issue will be closed without this)
Repo - https://github.com/varora1406/uber-clone
Start the app as per README instructions and try selecting any response from search field on top
Additional context

Library Version: ^2.4.1

React Native Version: https://github.com/expo/react-native/archive/sdk-42.0.0.tar.gz

[ ] iOS

[x] Android

[ ] Web

If you are using expo please indicate here:

[x] I am using expo

You left out the fetchDetails prop, and you have details = null.
https://github.com/varora1406/uber-clone/pull/1


marigold checkpoint

Hi,
Thank you for your great work and I appreciate your sharing.
BTW, I've got this error as follows:
Traceback (most recent call last):
File "/home/hyunsoocha/GitHub/MagicMan/inference.py", line 514, in
main()
File "/home/hyunsoocha/GitHub/MagicMan/inference.py", line 270, in main
ref_normal_pil = init_ref_normal(ref_rgb_pil, ref_mask_pil, method="marigold", device=device) # initilize reference normal map
File "/home/hyunsoocha/GitHub/MagicMan/inference.py", line 188, in init_ref_normal
pipe = MarigoldNormalsPipeline.from_pretrained(
File "/home/hyunsoocha/conda/envs/magicman/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
return fn(*args, **kwargs)
File "/home/hyunsoocha/conda/envs/magicman/lib/python3.10/site-packages/diffusers/pipelines/pipeline_utils.py", line 698, in from_pretrained
raise ValueError(
ValueError: The provided pretrained_model_name_or_path "/apdcephfs_cq10/share_1330077/hexu/NVS/pretrained_weights/marigold-normals-v0-1" is neither a valid local path nor a valid repo id. Please check the parameter.
How can we get this checkpoint?
Thanks in advance.

Hi, @HyunsooCha
We have fixed the wrong marigold checkpoint path. You can also download marigold checkpoint locally before running inference.py according to  https://huggingface.co/docs/diffusers/using-diffusers/marigold_usage

Hi, @HyunsooCha
We have fixed the wrong marigold checkpoint path. You can also download marigold checkpoint locally before running inference.py according to https://huggingface.co/docs/diffusers/using-diffusers/marigold_usage


See how many withdrawn/terminated projeccts

As an executive
I want to see all withdrawn and terminated Projects by why they failed
so that I can make informed decisions
Acceptance Criteria
Scenario 1:
GIVEN ... (you or your condition)
WHEN ... (what you do)
THEN ... (what you see)
Scenario 2:
xxxxxx
Validation Rules?
(If yes, list here)
Screenshot
(Link above will always be the most up-to-date version.)

drill down pie chart.
proportion -> grid view


Semantic Question

So I built this thing and ran it and I'm getting this message in the console:
$ dotnet watch run
[DotNetWatcher] info: Running dotnet with the following arguments: run
[DotNetWatcher] info: dotnet process id: 5428
Project DotNetCoreKoans (.NETCoreApp,Version=v1.0) was previously compiled. Skipping compilation.
AboutAsserts AssertTruth has damaged your karma.
The Master says:
  You have not yet reached enlightenment.
[DotNetWatcher] fail: dotnet exit code: -1
[DotNetWatcher] info: Waiting for a file to change before restarting dotnet...

I did a debug and stepped through the whole thing and it appears as though it's running through several completely empty method blocks and eventually hitting a program exit of -1. So indeed, it was programmed to behave like this. My question is, what exactly is the intended purpose here?

You are correct, those methods are stubs for functionality that the original Ruby Koans had for output.
There is a little bit to unpack in the output of the application at this point:

The application is just a console application that exits 0 or -1 depending on if you have completed all the koans. If a koan is failing, it exits -1. If all koans pass, you win and it exits 0.

We are using dotnet watch, which watches your source code and anytime a file changes it reruns the application. This lets the student, run a dotnet watch run and then just edit Koan files watching the output in their console window.

dotnet watch puts these goofy DotNetWatcher messages in the output, there is a issue #1 filed to upgrade to the latest version when it is available that will allow us to suppress these messages.

The "application" simply walks down the koans as defined in the path to enlightenment. It then executes each Step. If the step is successful, it executes the next step. It continues on till it hits a step that fails.

I am currently working on the Engine that does the walking and reporting. My goal is to get the output to look as much like the original Ruby Koans as I can.
The issues in this list are soliciting help in fleshing out the koans with good tutorials while I complete that work.

Awesome. Thanks so much for this, so basically, the student looks through the source files, completes them, and then when they're done, this little application will recognize that and change its output to 0 if I understand correctly. Meaning, the student must work through the different source files

Exactly.

Closing this issue.


Upgrade golang versions to 1.12.12

Version was selected by looking up k/k bazel build.

/lgtm
/approve


#285 - Rename template to ready and reorder lifecycle.

References discussion in #285. More specifically, it implements https://github.com/skatejs/skatejs/issues/285#issuecomment-133372496.
Before

bind events
define properties
call template
invoke created lifecycle on descendants
set initial property values
call created on host
execute queued event handlers

After

define properties and only initialise internal value to the init option, or the overridden property's value
bind event listeners
call created on host
invoke entire created lifecycle on all descendants
call ready on host

Notes

one step closer to #269: added experimental skate.property event that is emitted when a property changes
template was renamed to ready
ready is basically the new version of created
created is now invoked before descendant initialisation
both created and ready provide before and after hooks around descendant initialisation
the property get callback was removed
the property set callback was renamed to update
properties don't trigger update on initialisation
overridden property values now correctly set the initial property value for the new property
tests were added to cover common scenarios when using properties and templating

Oops mistapped a button...
Sorry @treshugart was this not previously exposed through prop.notify? Is this now prop.emit?

Just had a look at the source code and apparently notify was not implemented, sorry for the confusion.

It was being experimented with at some point, but it was never stable.


Cleanup operator codebase

This is just an umbrella issue to discuss potential refactorings and improvements to the operator codebase.

[ ] remove duplication between istioctl and in-cluster operator
we currently have a few areas, mostly IOP yaml merging and generation, that are slightly different between istioctl and operator, which is a potential source of hard-to-trace bugs. We should reuse as much code as possible between tools.

[ ] unify enablement code
there are multiple spots in the code where we check component enablement, e.g. during collection of charts (operator/pkg/controlplane) vs. generation of values (e.g. pkg/translate, which uses pkg/util and pkg/tpath)

[ ] repo merge artifacts
as part of merging the operator repo into istio/istio a lot of files have become obsolete. A lot of this has been tackled, but we should scan the operator/ directory for any leftover dead files.

fyi @ostromart @richardwxn @elfinhe @morvencao please add feedback, I'm sure I'm missing things here

@dgn As you mentioned in the 1st item, we have gaps about the marshaler/unmarshaler between istioctl and in-cluster operator. Basically, for all the commands integrated with istioctl, we're using jsonpb unmarshaler, while for in-cluster controller, we need to retrieve and store the IOP instance from kube-apiserver with go-client lib, it's using the yaml/json unmarshalers by default. Besides, to merge default profile with IOP spec in controller, we also need jsonpb unmarshaler.
AFAIK we need both of them, because currently the proto is the only source of truth for operator API.

I'd like to add the following:

[ ] Heavily prune back all the golden file outputs to retain only the resources relevant to the particular test.
[ ] Take a hard look at the controller framework (pkg/controller & helmreconciler) and see if we still need it because it introduces a lot of code and makes the flow harder to follow.
[ ] Take another stab at cleaning up the hacks we have in Go struct generation from proto.
[ ] I'm pretty sure we still have several similar implementations of tree traversals floating around the codebase. Unify these with one syntax / capability set (under tpath?)
[ ] get rid of most of pkg/manifest and use a common pkg to apply/prune objects (we already have such functionality in the controller, istioctl should use the same assuming it can be made performant enough)
[ ] fswatcher never took off so it can likely be removed.
[ ] general code tidiness and documentation. There are some parts left over after the pruning above that can use some cleanup and docs: quite a bit of the code under cmd/mesh and pkg/tpath.

add item for in-cluster operator:

[ ] add validating webhook for in-cluster operator so that we can validate the IOP instance before it is applied into apiserver.

[ ] remove initialisms all over the codebase e.g. https://github.com/istio/istio/pull/21412#discussion_r383492206

@morvencao actually I had a pending PR for that: https://github.com/istio/operator/pull/441 but we haven't justify the necessity for it so far based on our current validation need.

I added all items to the issue description, except the validation webhook. @morvencao I think that one is not necessarily cleanup-related and can go into a separate issue

@dgn Correct.

How much of this is expected to be completed for 1.6?

I think this is complete for 1.6. We've made some really important cleanups and I'm pretty happy with the state of the code to start the next release.

I think this can be considered as close to done as can be - all associated issues are closed.


rad init UI can be cut off on a small terminal

Bug information
Steps to reproduce (required)
Make your terminal pretty small.
Run rad init
Observed behavior (required)
This is what I see:

Desired behavior (required)
rad init UI works
Workaround (optional)
Resize terminal
AB#9829

Related: https://github.com/radius-project/radius/issues/6264

@vishwahiremat - please check if this is the same issue as https://github.com/radius-project/radius/issues/5467 and if so, please close #5467 as a duplicate

@vishwahiremat - please check if this is the same issue as #5467 and if so, please close #5467 as a duplicate

@willtsai  wanted to confirm this is a different issue, working on a fix for this.


Fixes #222: A memory leak in DashboardData.

Fixes #222, a memory leak in dashboard data supplier.
The leak was caused by DashobardData.Origin objects being continuously registered to the message bus, but never unregistered.

Note, this is required as a bug fix for styx 0.7.n branches only. We will not cherry pick to styx 1.0 branch as it will have a brand new implementation for dashboard data supplier.

Will cherry pick to 1.0 as the refactoring is likely to be delayed.


[Feature]: Flag missing settings that are required to run

Feature Request
When the AI node model is set to take an input but no input is provided, a TypeError throws in the graph node:

I'm guessing it flagged from one of the variations of this, which is how I eventually found the problem of a missing model:
const tiktokenModel = openaiModels[model].tiktokenModel;

Based on some of the other patterns, such as in text nodes, it seemed like the setting dropdown would specify a default:

I wasn't aware that the input toggle was toggled on so it took a moment to find the issue. Would be awesome if there was some error state, or use the selected one as a default!
Code of Conduct

[X] I agree to follow this project's Code of Conduct

Good point thanks!

Fixed in 828caa6705d324e907a26397dde0e83f56c27d0f

@all-contributors please add @Jkwok0714 for bug


Product needs max quantity option and/or option to make product non-physical (limited to 1 with no option to change quantity in cart)

Non physical doesn't necessarily mean just one though. Example: licenses. But I'm into making everything optional.

Non-physical is the wrong wording to use, just couldn't think of a better term.
Maybe it's just a case of setting a cart limit to 1, or something to mark a product a 'Service product', or 'No quantity option'..  you get what I mean though, for something like a membership product.

Yeah - how about a checkbox for "Fixed Quantity" and being able to define the quantity?

I think the main aspect of what I'm trying to describe is that not only would it restrict the quantity to one, but when it was checked it wouldn't show the quantity next to the cart item, or show the quantity change box, or complain if you try and add it again. A subscription or membership doesn't make sense to display a 'quantity: 1' next to it.

Right. I'm on the same page. I think maybe what wasn't clear about what I said is that defining the fixed quantity would be in the dashboard. I can't think of a reason why you would have a fixed quantity of anything other than 1, but that doesn't mean that someone else wouldn't.


Can't run the docs locally

I followed the steps in the contributing doc to set up a local docs server, but I get browser console errors when I try to access localhost:3000.

Material-UI: Master Branch
React: 15.3.0
Browser: Chrome
Node: 4.4.7
npm: 2.15.8

Have you seen #4758?

Material-UI: Master Branch
React: 15.3.0
Browser: Chrome
Node: 4.4.7
npm: 2.15.8

Cannot reproduce on master with:

node v6.3.1
npm 3.10.6
(other specs are the same)

Updating to node stable did the trick.  I'm guessing npm v2 does not work with the docs.  Here is my env that works:
~/D/r/m/docs ❯❯❯ node -v                                                                                master
v6.2.2
~/D/r/m/docs ❯❯❯ npm -v                                                                                master
3.9.5

This is maybe something to put in a FAQ.
Den 17.44 Tor 11. Aug 2016 skrev mziemer21<EMAIL_ADDRESS>
Closed #4936 https://github.com/callemall/material-ui/issues/4936.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/callemall/material-ui/issues/4936#event-752856429,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACBQBPr8vTzXZDlc00ZC8gHbkMxPmMUDks5qe0NngaJpZM4Jg85R
.


[ART-2336] Don't rebuild bundle if already exists for given NVR

Tries to solve the following scenario:

foo-operator-4.6-42 builds
foo-operator-metadata-4.6.42-1 builds
foo-operator-4.6-43 builds
foo-operator-metadata-4.6.43-1 builds
We decide to ship foo-operator-4.6-42 and attach it to an advisory
Now we want to attach the corresponding bundles, so we run olm_bundles against the advisory... which does see changes when it rebases back to 4.6-42, and so builds and attaches foo-operator-metadata-4.6.42-2.

Example:
$ doozer -g openshift-4.6 olm-bundle:rebase-and-build cluster-logging-operator-container-v4.6.0-202010201329.p0 2>/dev/null
cluster-logging-operator-metadata-container-v<IP_ADDRESS>010201329.p0-2

Just tested again with all modifications applied, it works (much faster, BTW)
$ ./doozer -g openshift-4.6 olm-bundle:rebase-and-build cluster-logging-operator-container-v4.6.0-202010201329.p0 2>/dev/null
cluster-logging-operator-metadata-container-v<IP_ADDRESS>010201329.p0-2

:triumph:


'Namespace' object has no attribute 'stop_timeout' in version 2.13.0

Describe the bug
After upgrading to 2.13.0, we started seeing the following error:
  File "/path/to/app/venv/lib/python3.9/site-packages/locust/env.py", line 84, in __init__
    self.stop_timeout = float(parsed_options.stop_timeout)
AttributeError: 'Namespace' object has no attribute 'stop_timeout'

We are using Locust as a library, and when creating the Locust Environment, we are passing in a Namespace object for parsed_options. In old versions of Locust, we did not need to pass stop_timeout, but now it appears that we do. Is this intentional?
Expected behavior
Existing code should continue to work after a minor version upgrade.
Actual behavior
An AttributeError exception is raised from within Locust.
Steps to reproduce
from argparse import Namespace
from locust.env import Environment
parsed_options = Namespace(timeout=42)
Environment(parsed_options=parsed_options)

^ the above works in 2.12.0 but fails in 2.13.0
Environment

OS: Linux
Python version: 3.9.11
Locust version: 2.13.0
Locust command line that you ran: n/a
Locust file contents (anonymized if necessary): n/a

Dont construct your Environment that way. Pass arguments explicitly: Environment(stop_timeout=42)


Added support for older python versions

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 91.61%. Comparing base (7364a53) to head (08f9fa6).

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##             main       #2      +/-   ##
==========================================
+ Coverage   91.58%   91.61%   +0.02%
==========================================
  Files          16       16
  Lines         297      298       +1
==========================================
+ Hits          272      273       +1
  Misses         25       25

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.